What’s your Big Data strategy? You do have one, right? You better, or prepare to be left behind your competition. Companies are constantly seeking a competitive edge and are looking to data for that edge. From the right data you can gain valuable customer insights, predict future trends and activities, and create new business opportunities.
The question is how to get that knowledge from data, which in and of itself is valueless. You face technical obstacles, both in maximizing your existing systems vs. the need to buy new ones, as well as extracting the necessary knowledge from the data. Data might be siloed, and you might not have the right tools.
Then there are the internal conflicts between the technology, business, and executive departments. Everyone has their own interests and will often look out for them over the greater good.
It may fall to one person, or at the very least one department, to devise a solid Big Data strategy. To determine the technological requirements. Decide what data is gathered. How that data is analyzed and processed. And how to act upon the findings. The first thing you need, then, is a proper framework.
Big Data Framework
There are some analysts who believe there is no need for a separate Big Data strategy because it has matured and become a part of overall business strategy, so why create a separate strategy?
“Big Data is over its hype. In fact, we don’t even publish a hype cycle anymore,” said Frank Buytendijk, research fellow with Gartner. “The dots are distributed across the hype cycles around information management, information infrastructure and analytics. This doesn’t mean that Big Data failed and dropped. In fact it is the opposite, it has made its way to ‘business as usual’ very quickly.”
As a consequence, there is less or no need to create a separate Big Data strategy, he argues. “Organizations need an overall ‘data and analytics’ strategy. And even that strategy is dependent. Data and analytics in the end are just a set of capabilities in service of the wider digital platform and digital strategy,” he said.
However you structure your strategy, you have to have a plan before you begin to gather the data, otherwise you won’t know what data to gather and what data to ignore. You won’t know how much you need of the different types of data, or how to process it.
“A common mistake is collecting data without having a purpose for it,” said Andi Mann, chief technology advocate at Splunk, a developer of operational intelligence software. “ATM data can be hundreds of gigabytes. If you don’t know what you are doing with it then collecting it doesn’t help.”
An effective and competitive Big Data strategy balances a diverse array of variables, and needs to be constantly updated as these variables changes.
Data Value
Another similar problem is not managing data by value. “Some is high value some is low value. Collecting all of this essentially situation normal sensor data and storing it in redundant environments is paying too much for what it’s worth, as opposed to collecting customer data and saving it for a day,” said Mann.
Some data sources, like IoT and sensor data, generates huge amounts of data that is disposable. Machinery, for example, keeps sending “everything is fine” signals in milliseconds. Do you need to retain gigabytes of data saying there are no problems with a piece of equipment?
On the flip side is customer data transaction data. All kinds of information can be gleaned from that, and for years to come as well when looking for buying patterns.
Big Data Models
The next step is your analytic models, for things like data optimization or predictions. A plan must identify how the models will create additional business value and who will use them, and how they will use them. So your models need to include the end goal.
“In terms of data usage, the sort of things you can apply it to is incorporating a really good understanding of your customer and all the activity your customers are going through. That 360-degree view of the customer involves being able to understand the customer experience, from the journey to connect to you to the buying experience to their post-sale experience. Are they enjoying it and telling their friends good things?” said Mann.
Tools and Technologies
The next step is proper tools. This can be a bit of a challenge because many tools are only usable by data scientists as they involve highly esoteric and complex algorithms. This can result in two bad scenarios: a poor search or evaluation of the data, or extracting the wrong data or information. The wrong output leads to the wrong decisions.
“Most tools require skilled experts with a data science or similar skillset to use them properly,” said Mann. “Part of the reason is that humans are notoriously bad at working with large numbers; it is also because large numbers actually behave differently; but mostly it is because Big Data analytics is still a new area of technology, so there is still value in the experimentation, iteration, and ultimately innovation that deep specialization allows.”
Because Big Data is often about exploration of the data, seeing what you can find, the tools need the flexibility to change their query style and structure. That in turn means complexity. Tools that focus on a specific source type or use case are usually much more user-friendly, said Mann.
Data Analysis Skills
One thing proper models, data, and tools have in common is the skills to utilize them. Right now it’s a highly esoteric skill often requiring people with Master’s or Doctorate’s degrees, but education is catching up. One data scientist predicted that in five to 10 years, people with a Bachelor’s degree and specialized training will do the work of PhDs.
Companies need a road map for assembling the right talent pool. They need a mix of skills, knowledge, and specialization. You don’t want a bunch of technologists with little to no understanding of business if you are trying to maximize your business opportunities. Likewise, you don’t want a bunch of MBAs trying to build a Big Data system. There needs to be a mix of data scientists, analytics modelers, and business specialists.
Big Data: Planning and More Planning
A proper Big Data strategy means planning. Lots of planning. Know what you are trying to accomplish and what will be the sources of data before you gather one kilobyte of data or you will find yourself drowning in information you cannot use. An ounce of preparation here is worth a week of processing.
