The current shift of bundling database services is enabling increased efficiency and AI/ML innovations at scale.
Once in a while database systems go through a phase of bundling and unbundling enabling a new set of use cases and value addition.
The DBMS bought the efficiency as a packaged system managing every aspect of data, enabling ACID properties and transactions, providing consistency guarantees compared with file-based data systems before, developed a niche area around OLTP applications, and became the backbone of ERP systems.
Then Hadoop happened.
Hadoop ushered in the era of unbundling of database management systems to address massive parallel processing of unstructured data.
The DBMS’s are composed of
c) Data files and
d) Standard SQL Interface
e) Meta store
services at high level. These services are bundled together as the database management system. Hadoop unpacked these four aspects. The compute is de-coupled from storage and makes scaling memory and compute easier. The storage is infinite and the data files are liberated from predefined schemas with no strict metadata adherence enabling structured querying and raw processing. A different set of security protocols are developed compared with GRANT and RBAC SQL commands. Hadoop systems enabled massive processing of data setting foundation for AI/ML use-cases.
Now we are witnessing another shift, back to the bundling of memory, compute, network, and client interfaces but with a serverless approach accelerated by cloud computing technologies, bringing in the efficiency in the form of zero operations and maintenance. Available cloud data warehouse define pricing in terms of units a combination CPU+RAM+Network rather than VMs.
The difference with the RDBMS systems is that cloud data warehouses are flexible in implementing ACID and relational structures and providing new data types arrays, structures, and de-normalized tables. The cloud data warehouse systems enable the required scale moving from the zone of terabytes to petabytes and capabilities for new use-cases.
Bundling and unbundling allows value to different categories of customers. It is a known fact that enterprises prefer bundled services, while startups prefer unbundled services. The value generation fueled by the shift to the bundled system is just started and this decade belongs to the vendors with serverless databases that make scaling decisions for you and enable AI/ML innovations without moving/making copies of data and with zero operations all in one package.