Storing data doesn’t always mean SQL
It’s a busy day in the office, business-as-usual. The customer is chasing the team for deliverables, which includes database schema design… The development team, counting on their previous experience, doesn’t take long to create a design using a well-known RDBMS. The lead ensures the schema is in good-old third normal form… Done!
Fast forward to six months down the line – the product phase 1 is released to the cloud, stabilized, users pour in… just then the customer raises an issue with the high bill amount. Bill break-up on the cloud billing page shows database consuming 80% of the costs!
What could be wrong? One needs to persist all real-time and historical data for so many reasons which everyone is aware of…
However, the missing point is – All storage is not SQL / RDBMS – there are other storage options worth considering. For example,
- The distant cousin of SQL – NoSQL – stores the data in unstructured JSON form. JSON is a friend of programmers since they are used to working with it. Let the data be stored in the same form.
- Data factories and data lakes – All cloud providers have thought through the problem of storing big data and give solutions that are performant and cost-effective.
- Blob storage – Yes, plain file format is also a storage option! There are cloud APIs available that let you work with data in blob storage similar to SQL storage.
- In-memory cache with persistence option- Consider this type for small data chunks required to be retrieved in real-time.
- Configuration files – Each tool/framework provides its own configuration file, plus custom configuration files can be created for additional, product-specific requirements. There is no need to use SQL tables for storing configurations.
Which of these storage options is your favorite?
