Big data analytics is quickly gaining adoption. Enterprises have awakened to the reality that their big data stores represent a largely untapped gold mine that could help them lower costs, increase revenue and become more competitive. They don’t just want to store their vast quantities of data, they want to convert that data into valuable insights that can help improve their companies.
As a result, investment in big data analytics tools is seeing remarkable gains. According to IDC, worldwide sales of big data and business analytics tools are likely to reach $150.8 billion in 2017, which is 12.4 percent higher than in 2016. And the market research firm doesn’t see that trend stopping anytime soon. It forecasts 11.9 percent annual growth through 2020 when revenues will top $210 billion.
Clearly, the trend toward big data analytics is here to stay. IT professionals need to familiarize themselves with the topic if they want to remain relevant within their companies.
What is Big Data Analytics?
The term “big data” refers to digital stores of information that have a high volume, velocity and variety. Big data analytics is the process of using software to uncover trends, patterns, correlations or other useful insights in those large stores of data.
Data analytics isn’t new. It has been around for decades in the form of business intelligence and data mining software. Over the years, that software has improved dramatically so that it can handle much larger data volumes, run queries more quickly and perform more advanced algorithms.
The market research firm Gartner categories big data analytics tools into four different categories:
- Descriptive Analytics: These tools tell companies what happened. They create simple reports and visualizations that show what occurred at a particular point in time or over a period of time. These are the least advanced analytics tools.
- Diagnostic Analytics: Diagnostic tools explain why something happened. More advanced than descriptive reporting tools, they allow analysts to dive deep into the data and determine root causes for a given situation.
- Predictive Analytics: Among the most popular big data analytics tools available today, predictive analytics tools use highly advanced algorithms to forecast what might happen next. Often these tools make use of artificial intelligence and machine learning technology.
- Prescriptive Analytics: A step above predictive analytics, prescriptive analytics tell organizations what they should do in order to achieve a desired result. These tools require very advanced machine learning capabilities, and few solutions on the market today offer true prescriptive capabilities.
Source: Gartner and others
Benefits of Big Data Analytics
Organizations decide to deploy big data analytics for a wide variety of reasons, including the following:
- Business Transformation In general, executives believe that big data analytics offers tremendous potential to revolution their organizations. In the 2016 Data & Analytics Survey from IDGE, 78 percent of people surveyed agreed that over the next one to three years the collection and analysis of big data could fundamentally change the way their companies do business.
- Competitive Advantage In the MIT Sloan Management Review Research Report Analytics as a Source of Business Innovation, sponsored by SAS, 57 percent of enterprises surveyed said their use of analytics was helping them achieve competitive advantage, up from 51 percent who said the same thing in 2015.
- InnovationBig data analytics can help companies develop products and services that appeal to their customers, as well as helping them identify new opportunities for revenue generation. Also in the MIT Sloan Management survey, 68 percent of respondents agreed that analytics has helped their company innovate. That’s an increase from 52 percent in 2015.
- Lower Costs In the NewVantage Partners Big Data Executive Survey 2017, 49.2 percent of companies surveyed said that they had successfully decreased expenses as a result of a big data project.
- Improved Customer Service Organizations often use big data analytics to examine social media, customer service, sales and marketing data. This can help them better gauge customer sentiment and respond to customers in real time.
- Increased Security Another key area for big data analytics is IT security. Security software creates an enormous amount of log data. By applying big data analytics techniques to this data, organizations can sometimes identify and thwart cyberattacks that would otherwise have gone unnoticed.
Big data analytics can offer key advantages across many verticals.
Big Data Analytics Challenges
Implementing a big data analytics solution isn’t always as straightforward as companies hope it will be. In fact, most surveys find that the number of organizations experiencing a measurable financial benefit from their big data analytics lags behind the number of organizations implementing big data analytics. Several different obstacles can make it difficult to achieve the benefits promised by big data analytics vendors:
- Data Growth One of the biggest challenges of big data analytics is the explosive rate of data growth. According to IDC, the amount of data in the world’s servers is roughly doubling every two years. By 2020, those servers will likely hold 44 zettabytes of digital information. To put that in perspective, that is enough data to fill a stack of iPads stretching from the earth to the moon 6.6 times. Big data analytics solutions must be able to perform well at scale if they are going to be useful to enterprises.
- Unstructured Data Must of the data stored in an enterprise’s systems doesn’t reside in structured databases. Instead, it is unstructured data, such as email messages, images, reports, audio files, videos and other types of files. This unstructured data can be very difficult to search—unless you have advanced artificial intelligence capabilities. Vendors are constantly updating their big data analytics tools to make them better at examining and extracting insights from unstructured data.
- Data Siloes Enterprise data is created by a wide variety of different applications, such as enterprise resource planning (ERP) solutions, customer relationship management (CRM) solutions, supply chain management software, ecommerce solutions, office productivity programs, etc. Integrating the data from all these different sources is one of the most difficult challenges in any big data analytics project.
- Cultural Challenges Although big data analytics is becoming commonplace, it hasn’t infiltrated the corporate culture everywhere yet. In the NewVantage Partners Survey, 52.5 percent of executives said that organizational hurdles like lack of alignment, internal resistance or lack of
Big Data Analytics Trends
What’s coming next for the big data analytics market? Experts offer a number of predictions.
Open Source
As big data analytics increases its momentum, the focus is on open-source tools that help break down and analyze data. Hadoop, Spark and NoSQL databases are the winners here. Even proprietary tools now incorporate leading open source technologies and/or support those technologies. That seems unlikely to change for the foreseeable future.
Market Segmentation
Plenty of general-purpose big data analytics platforms have hit the market, but expect even more to emerge that focus on specific niches, such as security, marketing, CRM, application performance monitoring and hiring. Analytics tools are also being integrated into existing enterprise software at a rapid rate.
Artificial Intelligence and Machine Learning
As interest in AI has skyrocketed, vendors have rushed to incorporate machine learning and cognitive capabilities into their big data analytics tools. According to Gartner, by 2020, almost every new software product, including big data analytics, will incorporate AI technologies. In addition, the company says, “By 2020, AI will be a top five investment priority for more than 30 percent of CIOs.”
Prescriptive Analytics
Fueled by this rush to AI, expect companies to become more interested in prescriptive analytics. Seen by many as the “ultimate” type of big data analytics, these tools will not only be able to predict the future, they will be able to suggest courses of action that might lead to desirable results for organizations. But before these types of solutions can become mainstream, vendors will need to make advancements in both hardware and software.
Refocusing on the Human Decision-Making?
As machine learning improves and becomes a table stakes feature in analytics suites, don’t be surprised if the human element initially gets downplayed, before coming back into vogue.
Two of the most famous Big Data prognosticators/pioneers are Billy Beane and Nate Silver. Beane popularized the idea of correlating various statistics with under-valued player traits in order to field an A’s baseball team on the cheap that could compete with deep-pocketed teams like the Yankees.
Meanwhile, Nate Silver’s effect was so strong that people who didn’t want to believe his predictions created all sorts of analysis-free zones, such as Unskewed Polls (which, ironically, were ridiculously skewed). Many think of Silver as a polling expert, but Silver is also a master at Big Data analysis.
In each case, what mattered most was not the machinery that gathered in the data and formed the initial analysis, but the human on top analyzing what this all means. People can look at polling data and pretty much treat them as Rorscharch tests. Silver, on the other hand, pours over reams of data, looks at how various polls have performed historically, factors in things that could influence the margin of error (such as the fact that younger voters are often under-counted since they don’t have landline phones) and emerges with incredibly accurate predictions.
Similarly, every baseball GM now values on-base percentage and other advanced stats, but few are able to compete as consistently on as little money as Beane’s A’s teams can. There’s more to finding under-valued players than crunching numbers. You also need to know how to push the right buttons in order to negotiate trades with other GMs, and you need to find players who will fit into your system.
As Big Data analytics becomes mainstream, it will be like many earlier technologies. Big Data analytics will be just another tool. What you do with it, though, will be what matters.
Big Data Analytics Tools
Big data analytics has become so trendy that nearly every major technology company sells a product with the “big data analytics” label on it, and a huge crop of startups also offers similar tools. Cloud-based big data analytics have become particularly popular. In fact, the 2016 Big Data Maturity Survey conducted by AtScale found that 53 percent of those surveyed planned to use cloud-based big data solutions, and 72 percent planned to do so in the future. Open source tools like Hadoop are also very important, often providing the backbone to commercial solution.
The lists below are not exhaustive, but do include a sampling of some of better known big data analytics solutions.
