Workforce woes have been with us for a year or more. A great many entry-level positions are going unfilled, most retail shops are posting job vacancies, and of course, pay rates are rising. Countries such as Japan saw the writing on the wall some time ago and have been among the leaders in harnessing artificial intelligence (AI) to bring about higher levels of automation. The goal — to do a lot more with fewer people.
Here are some of the top trends in AI automation:
AI Training and Data Quality
Data engineers who are working on AI training are getting serious about data quality processes, according to Ruben Orduz, Senior Developer Advocate of the Great Expectations open data quality platform. The problem is that real-world data isn’t neat or predictable. It can suffer from quality afflictions like missing, truncated, or invalid data, unexpected duplication, and anomalies.
“When you use messy data to train an algorithm, the results can be catastrophic,” he said. “Machine Learning and AI algorithms that rely on making inferences and adjustments based on incoming data are especially susceptible to bad data conditions.”
For example, if your company routes package deliveries, you might receive a set of orders from a supplier that includes columns for name, address, and the number of items to be delivered. The AI takes in that data and uses it to learn and plan delivery routes. If deliveries usually include 1-30 packages and the data suddenly displays thousands in the number of packages column, not only are you hiring too many delivery drivers, but your model is also skewed. That’s why data engineers are working hard to understand the shape of their data and using data quality platforms to identify outliers and afflicted data. It’s easier than ever to do this now. Engineers can automate discovery, running pipeline tests as the data streams in and catch outliers before they get to the AI training process.
AI Automation and Networking
A number of players such as Cisco, Juniper Networks, Gluware, and Splunk are investing in the automation of AI capabilities to apply them to network troubleshooting and performance diagnostics. Network operations traditionally require a highly involved human effort. AI and data can automate a lot of that effort.
Networks, after all, are complex systems, embodying a plethora of technologies, architectures and overlays resulting in many operational points where failures and performance issues can occur. The human aspect of networks is equally challenging. Despite some efforts to centralize operations to network controllers, network management is still largely a manual, human effort that requires a lot of expertise. Moreover, the effort is mostly exploratory, where engineers try to manually derive the sources of issues without having insight or data ahead of time. As a result, the cost of operating networks in a traditional manner is high both in service downtime or degradation and in human effort to reach solutions.
“There is a huge opportunity to leverage AI automation for analytics and data,” said Stanislav Miskovic, Vice President of AI, Gluware, a company that is building a platform which will leverage and unify real data across the networking stack.
“There are many ways AI automation can help reduce the operational costs related to networking: performing root cause analysis and localizing sites, devices, and protocols that are problematic; autonomously baselining infrastructure as a whole and all its components; ranking the relevance of identified problems; and identifying supporting evidence and prescribing remediation actions to network engineers for example. But most of these functions are still in their infancy in the world of networks.”
AI in Network Security
Just as networking in general needs automation via more AI, so does network security. AI is capable of finding that needle in a haystack — that’s what security is all about. As threats are becoming more sophisticated, the perimeter of enterprise networks is becoming more and more blurry. Especially with transitions to the cloud, the amounts of data that need to be analyzed are far outside the capabilities of human inspection.
“AI and analytics are the key enablers of enhanced security,” said Miskovic. “Security needs to cover a much wider footprint today, which cannot be done without the help of AI automation. The number of attack surfaces is too great and data volume too huge for inspection without AI assistance.”
This has given rise to the development of User and Entity Behavior Analytics, a purely AI driven security field. Another AI driven security approach is detecting zero-day attacks or unknown-unknows, which cannot be done without AI automating baseline profiling and anomaly detection. Finally, AI automation helps security response teams by ranking security alerts, reducing alert fatigue and suggesting corrective actions.
Process Automation
Even with all the products that help monitor, manage, operate and secure company infrastructure, the number of alerts they produce and the number of steps engineers need to take to address them are becoming intractable. For this reason AI-driven process automation is becoming an integral part of network operations, security operations, and ticket management.
“A number of AI-driven solutions have been developed to automate various aspects of the alert and problem-solving tasks,” said Miskovic. “These AI automation solutions provide customizable playbooks that can perform a number of logistical or remediation tasks without human intervention. The system can autonomously learn a number of playbook steps that need to be done to solve a problem by identifying patterns in what engineers were doing to solve similar problems. These playbooks can also be further customized by engineers.”
AI as a Service
AI engines and platforms are complex. Thus, it was inevitable that others would take on the heavy lifting and offer AI as a Service as an alternative to AI as a platform.
“Niche AI that requires deep knowledge delivered as AI as a Service will accelerate as firms and their data science teams who have already leveraged themselves through AI platforms, begin to reach to specialized teams to realize additional value in AI,” said Frans Cronje, CEO and Co-founder of DataProphet.
