Data fabrics are a new type of networking based on a very familiar design concept.
If you haven’t heard of the concept of a “data fabric,” you will – the trend is growing rapidly. Allied Market Research projects the data fabric market will grow to $4.5 billion by 2026, with much of that in the North American market, due to hyperscale cloud growth.
Data fabrics are emerging as an approach to help organizations better deal with fast growing volumes of data, ever-changing application requirements and distributed processing scenarios.
Think of data fabric as a web stretched over a large network that connects multiple locations, types, and sources of data, both on-premises and in the public cloud, with a variety of methods for accessing that data to process, move, manage, and store the data within the confines of the fabric.
To be a “fabric,” it must have redundancy of pathways and not be dependent upon a single point to point connection, so if one connection is overloaded with data or otherwise unavailable, there are other pathways to the destination.
Does this sound familiar? It should, that’s how the Internet operates. The original Internet, Darpanet, was designed by the military to create a resilient and redundant network capable of surviving a nuclear attack but shifting from traditional hub-and-spoke network design to multiple pathways. Data fabrics work in the same manner.
Elements of a Data Fabric
For the longest time, apps had their own unique approach to storing and retrieving data. Unifying all of this data can be quite a challenge, since every application stores the data in different formats. Data is stored in many places around the network and in different application silos. Sometimes the data is redundant and deduplication is required. All of this adds to the burden of unifying data.
A data fabric, then, crosses all the data stores and brings together the right data for the right application. We have touched on what makes a data fabric, but now let’s go into detail. What is needed to constitute a data fabric includes the following:
- Creates a unified data environment. A data fabric is not a connection to a storage array or database. It is a holistic connection of multiple and disparate data sources and should not omit any data source.
- Combine data from multiple systems. It should pull data from everything from a mainframe to an AWS S3 storage repository.
- Support multiple locations. It should support the on-premises data center, edge networks, and cloud computing environments.
- Provide high availability and reliability. The fabric must be available at all times, resilient to high loads and even self-healing when there is a problem such as an unexpected outage.
- Connect to any data source via connectors and components, eliminating the need for hard coding connections.
- Provide seamless data ingestion and integration capabilities between the different data sources.
Data Fabrics and Big Data
Digital transformation is a major strategic agenda in most organizations, and that means tapping into all resources, legacy and modern, across a variety of formats. The goal is to create a converged platform that supports the storage, processing, analysis and management of disparate forms of data. This data can be drawn from a variety of sources, including files, database tables, data streams, objects, images, sensor data and even container-based applications.
In short, it’s a network of Big Data sources all tied together through a high-speed, redundant network of interfaces such as Network File System (NFS), POSIX (portable operating system interface), a REST API, Hadoop distributed file system (HDFS), ODBC, and Apache Kafka.
To truly engage in digital transformation and make the most of all of your data, you need to access all of it from different resources, and not be overwhelmed by the sheer volume of data. This requires a data-centric organization not tied to one vendor or protocol.
Why Data Fabrics Matter to Business
It’s often said that “data is the new oil.” Data drives competitive advantage for every business. Organizations need to deliver data quickly to serve business and customer needs. If you do not want rapid data access from all parts of your enterprise, from the mainframe to AWS stores, you aren’t serious about being competitive.
IT systems are becoming more complex than ever before and need the ability to work across complex environments using both legacy applications and data while also embracing new microservice-based applications. A data fabric is the versatile, powerful way to achieve this important goal.
Data Fabrics Vendors
A data fabric is a software solution with significant hardware connectivity and networking requirements. Most of the big names in networking have an offering of some kind:
