Dan Vesset, group vice president at IDC, said, “After years of traversing the adoption S-curve, big data and business analytics solutions have finally hit mainstream. [Big data and business analytics] as an enabler of decision support and decision automation is now firmly on the radar of top executives. This category of solutions is also one of the key pillars of enabling digital transformation efforts across industries and business processes globally.” The company projects particularly strong growth for non-relational analytic data stores and cognitive software platforms over the next few years. It believes that by 2020 enterprises will be spending $70 billion on big data software.
Surveys of IT leaders and executives also lend credence to the idea that enterprises are spending substantial sums on big data technology. The NewVantage Partners Big Data Executive Survey 2017, found that 95 percent of Fortune 1000 executives said their firms had invested in big data technology over the past five years. In some cases, those investments were large, with 37.2 percent of respondents saying their companies had spent more than $100 million on big data projects, and 6.5 invested more than $1 billion.
And the IDG Enterprise 2016 Data & Analytics Research found that this spending is likely to continue. Among those surveyed, 89 percent expected that within the next 12 to 18 months their companies would purchase new solutions designed to help them derive business value from their big data.
If you’re in the market for a big data solution for your enterprise, read our list of the top big data companies
Big Data Technologies: Investment by Industry
Nearly every industry has begun investing in big data analytics, but some are investing more heavily than others. According to IDC, banking, discrete manufacturing, process manufacturing, federal/central government, and professional services are among the biggest spenders. Together those industries will likely spend $72.4 billion on big data and business analytics in 2017, climbing to $101.5 billion by 2020.
The fastest growth in spending on big data technologies is occurring within banking, healthcare, insurance, securities and investment services, and telecommunications. It’s noteworthy that three of those industries lie within the financial sector, which has many particularly strong use cases for big data analytics, such as fraud detection, risk management and customer service optimization.
“Outside of financial services, several other industries present compelling opportunities,” Jessica Goepfert, a program director at IDC, said. “Within telecommunications, for instance, big data and analytics are applied to help retain and gain new customers as well as for network capacity planning and optimization. Meanwhile, the media industry has been plagued by massive disruption in recent years thanks to the digitization and massive consumption of content. Here, big data and analytics can help firms make sense of and monitor their readers’ habits, preferences, and sentiment. Vendors targeting the big data and analytics opportunity would be well-served to craft their messages around these industry priorities, pain points, and use cases.”
From a geographic perspective, most of the spending will occur in the United States, which will likely account for about 52 percent of big data and analytics spending in 2017. Western Europe is the second biggest regional market with nearly a quarter of spending. However, the fastest growth is occurring in Latin America and the Asia/Pacific region.
The bulk of the spending on big data technologies is coming from enterprises with more than 1,000 employees, which comprise 60 percent of the market, according to IDC. Still, SMBs aren’t letting the trend pass them by, as they account for nearly a quarter of big data and business analytics spending.
So what Big Data technologies are these companies buying?
Non-relational analytics systems is a favored area for Big Data technology investment, as is cognitive software.
15 Big Data Technologies to Watch
The list of technology vendors offering big data solutions is seemingly infinite. Many of the big data solutions that are particularly popular right now fit into one of the following 15 categories:
1. The Hadoop Ecosystem
While Apache Hadoop may not be as dominant as it once was, it’s nearly impossible to talk about big data without mentioning this open source framework for distributed processing of large data sets. Last year, Forrester predicted, “100% of all large enterprises will adopt it (Hadoop and related technologies such as Spark) for big data analytics within the next two years.”
Over the years, Hadoop has grown to encompass an entire ecosystem of related software, and many commercial big data solutions are based on Hadoop. In fact, Zion Market Research forecasts that the market for Hadoop-based products and services will continue to grow at a 50 percent CAGR through 2022, when it will be worth $87.14 billion, up from $7.69 billion in 2016.
Key Hadoop vendors include Cloudera, Hortonworks and MapR, and the leading public clouds all offer services that support the technology.
2. Spark
Apache Spark is part of the Hadoop ecosystem, but its use has become so widespread that it deserves a category of its own. It is an engine for processing big data within Hadoop, and it’s up to one hundred times faster than the standard Hadoop engine, MapReduce.
In the AtScale 2016 Big Data Maturity Survey, 25 percent of respondents said that they had already deployed Spark in production, and 33 percent more had Spark projects in development. Clearly, interest in the technology is sizable and growing, and many vendors with Hadoop offerings also offer Spark-based products.
3. R
R, another open source project, is a programming language and software environment designed for working with statistics. The darling of data scientists, it is managed by the R Foundation and available under the GPL 2 license. Many popular integrated development environments (IDEs), including Eclipse and Visual Studio, support the language.
Several organizations that rank the popularity of various programming languages say that R has become one of the most popular languages in the world. For example, the IEEE says that R is the fifth most popular programming language, and both Tiobe and RedMonk rank it 14th. This is significant because the programming languages near the top of these charts are usually general-purpose languages that can be used for many different kinds of work. For a language that is used almost exclusively for big data projects to be so near the top demonstrates the significance of big data and the importance of this language in its field.
4. Data Lakes
To make it easier to access their vast stores of data, many enterprises are setting up data lakes. These are huge data repositories that collect data from many different sources and store it in its natural state. This is different than a data warehouse, which also collects data from disparate sources, but processes it and structures it for storage. In this case, the lake and warehouse metaphors are fairly accurate. If data is like water, a data lake is natural and unfiltered like a body of water, while a data warehouse is more like a collection of water bottles stored on shelves.
Data lakes are particularly attractive when enterprises want to store data but aren’t yet sure how they might use it. A lot of Internet of Things (IoT) data might fit into that category, and the IoT trend is playing into the growth of data lakes.
MarketsandMarkets predicts that data lake revenue will grow from $2.53 billion in 2016 to $8.81 billion by 2021.
5. NoSQL Databases
Traditional relational database management systems (RDBMSes) store information in structured, defined columns and rows. Developers and database administrators query, manipulate and manage the data in those RDBMSes using a special language known as SQL.
NoSQL databases specialize in storing unstructured data and providing fast performance, although they don’t provide the same level of consistency as RDBMSes. Popular NoSQL databases include MongoDB, Redis, Cassandra, Couchbase and many others; even the leading RDBMS vendors like Oracle and IBM now also offer NoSQL databases.
NoSQL databases have become increasingly popular as the big data trend has grown. According to Allied Market Research the NoSQL market could be worth $4.2 billion by 2020. However, the market for RDBMSes is still much, much larger than the market for NoSQL.
MonboDB is one of several well-known NoSQL databases.
6. Predictive Analytics
Predictive analytics is a sub-set of big data analytics that attempts to forecast future events or behavior based on historical data. It draws on data mining, modeling and machine learning techniques to predict what will happen next. It is often used for fraud detection, credit scoring, marketing, finance and business analysis purposes.
In recent years, advances in artificial intelligence have enabled vast improvements in the capabilities of predictive analytics solutions. As a result, enterprises have begun to invest more in big data solutions with predictive capabilities. Many vendors, including Microsoft, IBM, SAP, SAS, Statistica, RapidMiner, KNIME and others, offer predictive analytics solutions. Zion Market Research says the Predictive Analytics market generated $3.49 billion in revenue in 2016, a number that could reach $10.95 billion by 2022.
7. In-Memory Databases
In any computer system, the memory, also known as the RAM, is orders of magnitude faster than the long-term storage. If a big data analytics solution can process data that is stored in memory, rather than data stored on a hard drive, it can perform dramatically faster. And that’s exactly what in-memory database technology does.
Many of the leading enterprise software vendors, including SAP, Oracle, Microsoft and IBM, now offer in-memory database technology. In addition, several smaller companies like Teradata, Tableau, Volt DB and DataStax offer in-memory database solutions. Research from MarketsandMarkets estimates that total sales of in-memory technology were $2.72 billion in 2016 and may grow to $6.58 billion by 2021.
8. Big Data Security Solutions
Because big data repositories present an attractive target to hackers and advanced persistent threats, big data security is a large and growing concern for enterprises. In the AtScale survey, security was the second fastest-growing area of concern related to big data.
According to the IDG report, the most popular types of big data security solutions include identity and access controls (used by 59 percent of respondents), data encryption (52 percent) and data segregation (42 percent). Dozens of vendors offer big data security solutions, and Apache Ranger, an open source project from the Hadoop ecosystem, is also attracting growing attention.
9. Big Data Governance Solutions
Closely related to the idea of security is the concept of governance. Data governance is a broad topic that encompasses all the processes related to the availability, usability and integrity of data. It provides the basis for making sure that the data used for big data analytics is accurate and appropriate, as well as providing an audit trail so that business analysts or executives can see where data originated.
In the NewVantage Partners survey, 91.8 percent of the Fortune 1000 executives surveyed said that governance was either critically important (52.5 percent) or important (39.3 percent) to their big data initiatives. Vendors offering big data governance tools include Collibra, IBM, SAS, Informatica, Adaptive and SAP.
10. Self-Service Capabilities
With data scientists and other big data experts in short supply — and commanding large salaries — many organizations are looking for big data analytics tools that allow business users to self-service their own needs. In fact, a report from Research and Markets estimates that the self-service business intelligence market generated $3.61 billion in revenue in 2016 and could grow to $7.31 billion by 2021. And Gartner has noted, “The modern BI and analytics platform emerged in the last few years to meet new organizational requirements for accessibility, agility and deeper analytical insight, shifting the market from IT-led, system-of-record reporting to business-led, agile analytics including self-service.”
Hoping to take advantage of this trend, multiple business intelligence and big data analytics vendors, such as Tableau, Microsoft, IBM, SAP, Splunk, Syncsort, SAS, TIBCO, Oracle and other have added self-service capabilities to their solutions. Time will tell whether any or all of the products turn out to be truly usable by non-experts and whether they will provide the business value organizations are hoping to achieve with their big data initiatives.
11. Artificial Intelligence
While the concept of artificial intelligence (AI) has been around nearly as long as there have been computers, the technology has only become truly usable within the past couple of years. In many ways, the big data trend has driven advances in AI, particularly in two subsets of the discipline: machine learning and deep learning.
The standard definition of machine learning is that it is technology that gives “computers the ability to learn without being explicitly programmed.” In big data analytics, machine learning technology allows systems to look at historical data, recognize patterns, build models and predict future outcomes. It is also closely associated with predictive analytics.
Deep learning is a type of machine learning technology that relies on artificial neural networks and uses multiple layers of algorithms to analyze data. As a field, it holds a lot of promise for allowing analytics tools to recognize the content in images and videos and then process it accordingly.
Experts say this area of big data tools seems poised for a dramatic takeoff. IDC has predicted, “By 2018, 75 percent of enterprise and ISV development will include cognitive/AI or machine learning functionality in at least one application, including all business analytics tools.”
Leading AI vendors with tools related to big data include Google, IBM, Microsoft and Amazon Web Services, and dozens of small startups are developing AI technology (and getting acquired by the larger technology vendors).
12. Streaming analytics
As organizations have become more familiar with the capabilities of big data analytics solutions, they have begun demanding faster and faster access to insights. For these enterprises, streaming analytics with the ability to analyze data as it is being created, is something of a holy grail. They are looking for solutions that can accept input from multiple disparate sources, process it and return insights immediately — or as close to it as possible. This is particular desirable when it comes to new IoT deployments, which are helping to drive the interest in streaming big data analytics.
Several vendors offer products that promise streaming analytics capabilities. They include IBM, Software AG, SAP, TIBCO, Oracle, DataTorrent, SQLstream, Cisco, Informatica and others. MarketsandMarkets believes the streaming analytics solutions brought in $3.08 billion in revenue in 2016, which could increase to $13.70 billion by 2021.
13. Edge Computing
In addition to spurring interest in streaming analytics, the IoT trend is also generating interest in edge computing. In some ways, edge computing is the opposite of cloud computing. Instead of transmitting data to a centralized server for analysis, edge computing systems analyze data very close to where it was created — at the edge of the network.
The advantage of an edge computing system is that it reduces the amount of information that must be transmitted over the network, thus reducing network traffic and related costs. It also decreases demands on data centers or cloud computing facilities, freeing up capacity for other workloads and eliminating a potential single point of failure.
While the market for edge computing, and more specifically for edge computing analytics, is still developing, some analysts and venture capitalists have begun calling the technology the “next big thing.”
14. Blockchain
Also a favorite with forward-looking analysts and venture capitalists, blockchain is the distributed database technology that underlies Bitcoin digital currency. The unique feature of a blockchain database is that once data has been written, it cannot be deleted or changed after the fact. In addition, it is highly secure, which makes it an excellent choice for big data applications in sensitive industries like banking, insurance, health care, retail and others.
Blockchain technology is still in its infancy and use cases are still developing. However, several vendors, including IBM, AWS, Microsoft and multiple startups, have rolled out experimental or introductory solutions built on blockchain technology.
Blockchain is distributed ledger technology that offers great potential for data analytics.
15. Prescriptive Analytics
Many analysts divide big data analytics tools into four big categories. The first, descriptive analytics, simply tells what happened. The next type, diagnostic analytics, goes a step further and provides a reason for why events occurred. The third type, predictive analytics, discussed in depth above, attempts to determine what will happen next. This is as sophisticated as most analytics tools currently on the market can get.
However, there is a fourth type of analytics that is even more sophisticated, although very few products with these capabilities are available at this time. Prescriptive analytics offers advice to companies about what they should do in order to make a desired result happen. For example, while predictive analytics might give a company a warning that the market for a particular product line is about to decrease, prescriptive analytics will analyze various courses of action in response to those market changes and forecast the most likely results.
Currently, very few enterprises have invested in prescriptive analytics, but many analysts believe this will be the next big area of investment after organizations begin experiencing the benefits of predictive analytics.
The market for big data technologies is diverse and constantly changing. But perhaps one day soon predictive and prescriptive analytics tools will offer advice about what is coming next for big data — and what enterprises should do about it.
Most experts expect spending on big data technologies to continue at a breakneck pace through the rest of the decade. According to IDC’s Worldwide Semiannual Big Data and Analytics Spending Guide, enterprises will likely spend $150.8 billion on big data and business analytics in 2017, 12.4 percent more than they spent in 2016. And the firm forecasts a compound annual growth rate (CAGR) of 11.9 percent for the market through 2020, when revenues will top $210 billion.