To realize the promise of the Internet of Things (IoT), companies cannot continue to use legacy technologies. Legacy data centers process discrete files and objects, but the constant flow of data from IoT devices represent something new: data streams.
Data streams don’t have a beginning or an end to provide natural starting and stopping points for processing. Legacy data processing technology that depends upon batch processing cannot expand quickly enough to avoid throughput bottlenecks, and the resulting delays affect the critical functions of ingestion, processing, and analysis.
To make practical use of real-time data streams from sensors, security cameras, medical devices, and other time-critical information flows, the data must be ingested and processed quickly. Dell’s Streaming Data Platform provides a scalable solution to rapidly accommodate high volumes of IoT data and make it available for real-time analysis.
See below to learn all about Dell Streaming Data Platform and where it stands in the storage sector:
Dell Streaming Data Platform and the Storage Market
Dell’s Streaming Data Platform competes in the global next-generation data storage market, which focuses on file and object-based storage and block storage solutions. While this category includes physical hard drives, it also encompasses cloud-based storage solutions as well.
Grand View Research’s 2019 estimate of $53.1 billion dollars for the global next-generation data storage market size includes a compound annual growth rate of 12.5% from 2019 to 2025 and an ultimate market size of $118.22 billion. KBV Research’s 2019 analysis projected similar numbers with a CAGR of 12.9% and an eventual market size of $106.3 billion by 2024.
Competitors in this market consist of various manufacturers: including Hewlett Packard Enterprise (HPE); Hitachi Vantara; IBM; Inspur; Micron Technology; NetApp; Netgear; Pure Storage; and Western Digital.
The Streaming Data Platform also competes within the software-defined storage market, which can provide additional insight into potential product growth and revenue.
In 2016, Markets and Markets forecasted a $4.72 billion software-defined storage (SDS) market growing at 36.7% that would reach $22.56 billion by 2021. The market did not quite reach these aggressive growth projections, and according to Maximize Market Research, the market revenue hit $12.92 billion in 2021 and, with a projected CAGR of 25.8%, will reach $51.21 billion by 2027.
The competitors in the software-defined storage market includes: Citrix; FalconStor Software; Fujitsu; HPE; IBM; Microsoft; NetApp; OpenText; Oracle; Seagate; StarWind Software; and VMware.
Dell Streaming Data Platform key features
- Enterprise-ready software platform
- Broad infrastructure support
- Supports on-premises ecosystems
- Supports Kubernetes (K8s), multicloud, and edge resources
- K8s available as Kubespray for edge development or OpenShift by RedHat for core development
- Process millions of data streams from multiple sources
- Unified and scalable data streams
- Ingests all types of data types, static and streaming
- Historical files convert to bounded data streams
- Low latencies and high availability
- Manages stream ingestion and storage
- Elastic tiered storage
- Loosely coupled long-term storage to enable unbound digital video recorder of all streaming data sources
- Hosts and enables development of analytic applications for real-time processing
- Real time stream analysis through embedded analytics engine
- Unified analysis of historical and real-time streaming data
- APIs (application programming interfaces) included in the distribution
- Web portal supports app development and artifact storage
- Embedded open-source analytic engines, including Apache Flink, Apache Spark, Pravega Search (PSearch), and GStreamer
- Dell’s proprietary management platform provides dynamic and automatic management
- Distributes data processing and analytical jobs over available infrastructure
- Sales resources to satisfy processing requirements in real time
- Web portal configures stream properties, displays metrics, runs applications, and displays job status
- Actionable alerts
- Enables notification of enterprise alerting tools
- Output feeds into third-party virtualization tools
- Open source: Built on Pravega and with Bookkeeper, InfluxDB, Grafana, Zookeeper
Dell Streaming Data Platform key benefits
Customers that pursue solutions such as the Dell Streaming Data Platform hope to gain the following key benefits.
Consolidate infrastructure
Dell’s Streaming Data Platform not only enables rapid data flows that keep up with IoT information sources, it also can expand to encompass and replace existing legacy infrastructure. The expandability of a single platform eliminates the need to maintain and secure multiple technologies.
Edge data control
The Dell Streaming Platform solution installs into local or cloud data centers completely under the control of the organization. The solution directly ingests data feeds and provides a single solution incorporating analysis and alerting. After processing, the Streaming Data Platform can directly feed the data into local data centers or whatever cloud storage provider might be desired for long-term storage.
Maximize IoT information insights
The surging quantities of information from IoT devices continues to grow in number and in richness. However, if an organization cannot keep up with the pace of the information it will fail to receive information on a timely basis and be able to act.
Adopting the Dell Streaming Data Platform enables an enterprise to continuously match the growing needs of the IoT feeds for ingestion, processing, and analysis. If needed, the system can also scale to incorporate historical data flows to help catch up to information backlogs and provide the maximum information for alerts and artificial intelligence (AI) analysis and to minimize the delays for intelligent reactions.
Simplified deployment
Enterprises always have the option to build their own stream solution, but these do-it-yourself (DIY) solutions consume enormous development time for integration, vulnerability testing, debugging, and deployment. The true value to the company comes from the information extracted from the data stream, so deploying a packaged solution enables companies to extract value more efficiently and much faster.
Dell Streaming Data Platform use cases
RWTH Aachen University
The scientists, mathematicians, and software developers collaborating in the Laboratory for Machine Tools for RWTH Aachen University (WZL) seek new insights from machine, product, and manufacturing data. Various sensors in WZL’s custom metal fineblanking incubators generate more than 1 million data points per second, and small localized data processing created huge delays in making necessary adjustments.
“We wanted to use high-frequency data to help manufacturers analyze changes in their processes, monitor output and process quality, and make adjustments in real time,” explained Philipp Niemietz, head of digital technologies for WZL.
Adopting Dell’s Streaming Data Platform provided the lab with the capabilities to make AI-guided adjustments based on real-time analysis of the sensor data.
“No matter how many sensors we use, once we set up the analytics pipeline and the data streams, we don’t have to address any load-balancing issues,” said Niemietz.
Other use cases
Itzik Reich, VP technologist of ISG at Dell, published a blog in which he illustrates real-world test cases to which the Dell Streaming Data Platform was successfully applied to replace batch and real-time processes. These use cases cover a broad variety of applications such as:
- Video feeds from a fleet of drones monitor the real time health of cattle.
- A sheet metal manufacturer uses real-time temperature sensor data streams to replace ineffective real-time and variance batch pipelines to make automated adjustments and maintain temperature within optimal thresholds.Â
- Real-time video footage of airplane take-offs and landings can be used to automatically detect abnormalities that might require maintenance.
The faster data processing of the Streaming Data Platform permits more responsive preventative measures and subtle adjustments than legacy solutions.
While the blog does not quote specific customers, Reich notes that customers often enjoy a reduction in their physical infrastructure and management time required for software and stack operations. The blog also focuses on the infrastructure savings from simplified IT architecture and the elimination of redundant data storage.
Dell Streaming Data Platform differentiators
When selecting the Dell Streaming Data Platform over other possible edge processing solutions, an organization will typically do so because of the following differentiators.
Dell management platform
Dell’s proprietary management software for the Streaming Data Platform uses dynamic and automatic management of systems to turbo-charge the open-source foundation of the platform. The management software:
- Distributes data processing and analytical jobs over the available infrastructure
- Satisfies processing requirements in real time
- Configures data stream properties
- Controls application deployment
- Provides critical information such as operations metrics and job status
Deployment options
Streaming Data Platform is available in various versions for flexible deployment:
- SDP Core: On-premises data center implementation that enables real-time data ingestion and feeds from other SDP deployments. Supports 3–12 nodes.
- SDP Edge: Edge deployment implementation with local processing, filtering, and enriching capabilities. Deployable on single (no high availability requirement) or three physical nodes (high availability requirement).
- SDP Micro: Lightweight version of SDP Edge specifically for low-volume data from sensors without gateways. Can only be installed on a single virtual or physical node.
Open-source with support
Built on open-source community built-and-tested components, Dell’s Streaming Data Platform provides the flexibility to avoid future vendor lock-in. However, the solution is also backed by Dell ProSupport Plus as well as Dell’s enormous sales and support network. This support network delivers the expert consulting needs an organization may want for an effective implementation.
Unified platform
Organizations can create their own mix of technologies to duplicate the capabilities of the Dell Streaming Data Platform. Yet, Dell’s unified solution provides turn-key availability for rapid deployment and integrated features that eliminate the need for tedious operational and security testing.
Conclusions
Not all enterprises have a need to process real-time data streams. However, as IoT and operational technology (OT) continues to evolve and become network connected, more businesses and governments will find themselves developing the need to handle streaming data sources.
Real-time processing and analysis of streaming data can enable nimble and accurate adjustments of processes or rapid detection of issues that could save a fortune — if addressed early. The Dell Streaming Data Platform enables scalable support for data streams of all types and should be evaluated as a potential solution for in-house data streaming needs.
