With more data created in the last couple years than in humankind’s entire history, the need to effectively manage, manipulate, and secure these information assets has never been more critical. This demand has traditionally been addressed by the leading database vendors. However, over the past decade, a myriad of challengers have entered the fray to bring order to chao vis-à -vis the ongoing data explosion.
Databases have subsequently gone through a dramatic evolution in recent years, with some flavors going the way of the floppy disk and others thriving to this day. Veteran DBAs will recall cutting their teeth on early Informix, SQL server, and Oracle DBMS offerings (the latter two are still dominant), while millennial developers reminisce about the open-source simplicity of MySQL/LAMP stack and PostgreSQL. Last but not least, today’s generation of DevOps engineers prefer the unstructured agility of NoSQL databases, like MongoDB and DynamoDB.
Databases Today
As it stands, most databases fall into one of two categories: relational database management systems (RDBMS) and the newer unstructured and/or special application databases. The former has been around since the 1970s and consists of related tables, which in turn are made up of rows and columns. Relational databases are manipulated using the structured query language (SQL), the de-facto standard language for performing create, read, update, delete (CRUD) functions.
The RDBMS is the dominant database type for enterprise computing and its SQL language, the lingua franca for communicating with databases. SQL-based RDBMS still make up 60.5% of databases in deployment, according to a recent survey by ScaleGrid.io. In fact, this continued popularity of the SQL language has resulted in big data offerings, like the fittingly named SQL-on-Hadoop and Apache Hive, to adopt the language, just to name a few.
The advent of the cloud saw data processing capabilities scale horizontally like never before, just in time to support the skyrocketing production of both structured and unstructured data brought on by the internet. With the latter gaining prominence, some posited that a new database paradigm was in order. Hence, NoSQL was born — a broad category that today includes all databases except those that use SQL as its main language. Because NoSQL databases have no set requirements in terms of schemas or structure, they are ideal for today’s software environments that utilize DevOps toolsets and CI/CD pipelines.
5 Trends in the Database Market
The global market for database management systems (DBMS) is estimated at nearly $63.1 billion for the year 2020 and is projected to reach $125.6 billion by 2026, growing at a CAGR of 12.4% over the period, according to Expert Market Research.
Here are the main trends driving the market growth for databases:
1. SQL Back on Top
A decade ago, the new NoSQL entrants seemed like formidable challengers to supplant the long-dominant SQL-based DBMS. These days, it’s more or less acknowledged that SQL will remain a cornerstone of DBMS for the foreseeable future. Even newer machine learning-based offerings, such as MindDB’s ML framework and AWS Redshift ML, have incorporated SQL as the default querying language.
2. ML-Driven Databases
Speaking of ML, the rising trend of integrating ML models where the data lives is becoming standard practice among vendors, with solutions such as Oracle Autonomous Database and Microsoft SQL Server Machine Learning Services on the enterprise side and the aforementioned MindsDB and SingleStore startup offerings.
3. Microservice Integration
Today’s modern software engineering teams design and build applications using a microservices approach. That is, they architect applications as a series of smaller, API-driven services. This improves scalability and agility, but can prove problematic for organizations with pre-existing data stored in traditional, monolithic databases. Fortunately, many of the newer database offerings, most notable NoSQL vendors, such as MongoDB and AWS DynamoDB, provide the schema flexibility, redundancy/scalability requirements, and serverless architecture pattern support required for microservices.
4. In-memory Databases
Today’s mission-critical software solutions require minimal database latency for optimal performance. Unfortunately, traditional DBMS rely on sluggish disk read/write operations for storing data to media (e.g., hard disk drives, solid-state drives). For this reason, in-memory databases have become strong alternatives for these critical use cases: because records are stored and retrieved directly from memory (RAM), faster and more reliable performance is possible. Additionally, popular solutions such as Redis support more data structure types and custom access patterns, allowing for the simplification of software code (read: no data structure conversion/serialization necessary).Â
5. Stronger Database Security Layers
With cyber attacks and data breaches continuing to dominate the technology world headlines, more focus than ever before has been placed on securing the data layer of the software application. More vendors are augmenting their offerings with stronger, baked-in security features. For example, Oracle now integrates always-on encryption and automated patching at the database level, while Amazon RDS includes a built-in firewall (i.e., security groups) for rules-based database access.
Conclusions
Regardless of type or flavor, databases will continue to function as the linchpin of modern internet applications, enabling the processing and storage of large amounts of data reliably and efficiently. Granted, the definition of large has changed over the years.
In general, data sets that are unmanageable via traditional spreadsheets are ideal for DBMS. And with the ever-increasing demand for databases supporting specialized use cases, such as time-series and geospatial applications, you can expect to see a myriad of burgeoning features from both new and traditional DBMS offerings on the near horizon. Â
