Machine Learning Will Cause Paradigm Shift in Information Technology Operations Analytics (ITOA) and Automation

ds_ml_solve1I have been a thought leader and “loud voice” for machine learning  and Predictive Analytics for many years.  I have always thought that the very narrow and specific use of Machine Learning for Predictive Coding within eDiscovery was just the beginning of the many uses for Machine Learning throughout the entire enterprise.  As an example, I published an article on November 30, 2015 entitled, Predictive Analytics across the Enterprise: From eDiscovery to DoJ Second Requests to Proactive Protection of Intellectual Property“, in which I stated, “If these predictive analytic technologies prove to be successful during the identification and preservation phases of the eDiscovery lifecycle, then knowledge workers throughout any enterprise with the task of analyzing any large amounts of ESI (i.e. Big Data) should be able to successfully utilize predictive analytics.”

As I expanded my analytic and project work into the Information Technology Automation / Workload Management and more specifically into the Information Technology Operations Analytics (ITOA) market,  I have been very encouraged to discovery that the “best and brightest” in the ITOA market also see the value of Machine Learning.

There is no doubt that Machine Learning is going to drive a major paradigm shift in the Information Technology Operations Analytics (ITOA) and Automation market just like it is doing in the eDiscovery and Information Governance markets.

In an article by Boštjan Kaluža, PhD, the Chief Data Scientist for Evolven, Dr. Kaluža, PhD states that, “Machine learning can be applied to solve really hard problems, such as credit card fraud detection, face detection and recognition, and even enable self-driving cars! For today’s IT Big Data challenges, machine learning can help IT teams unlock the value hidden in huge volumes of operations data, reducing the time to find and diagnose issues. Enterprises can automatically process large quantities of data in ways that were previously unattainable to improve IT operations, prevent breakdowns, and enhance support for critical business services.”

Dr. Kaluža, PhD goes on to discuss different types of Machine Learning:

  • Supervised learning
  • Unsupervised learning
  • Reinforcement learning

 

Supervised Learning

Supervised learning is the type of learning that takes place when the training data are labelled with the correct outcome, which gives the learning algorithm examples for learning. This is like having a supervisor who can show different objects and tell what they represent. The task of the learning algorithm is to learn the relation.

Example
Supervised learning can address credit card fraud detection, where the learning algorithm is presented with credit card transactions marked as normal or suspicious. The learning algorithm produces a decision model that marks unseen transactions as normal or suspicious.

 

Unsupervised Learning

On the other hand, unsupervised learning is harder because there is no supervisor telling you what the objects represent; instead, the learning algorithm should figure that out which objects go together by itself. Unsupervised learning algorithms do not assume any outcome labels Y, since they focus on grouping similar inputs X into clusters. Unsupervised learning can hence discover hidden patterns in data as well as similar items in the dataset.

Example
Unsupervised learning can enable an item-based recommendation system, where the learning algorithm discovers similar items bought together, for example like how Amazon looks at the people who bought book A also bought book B.

 

Reinforcement Learning

Reinforcement learning addresses the learning process, allowing machines and software agents to automatically determine the ideal behavior within a specific context, in order to maximize performance. Simple reward feedback is required for algorithms to learn behavior.

Reinforcement learning assumes that an agent, which can be a robot, a bot or a computer program, interacts with a dynamic environment to achieve a specific goal. The environment is described with a set of states and the agent can take different actions to move from one state to another. Some states are marked as goal states and if the agent achieves that state, it receives a large reward. In other states, the reward is smaller, non-existing or even negative.

The goal of reinforcement learning is to find an optimal policy, that is, a mapping function that specifies what action to take in each of the states without a supervisor explicitly telling whether this leads to the goal state or not.

Example
Reinforcement learning can make possible for a program to drive a vehicle, where the states correspond to driving conditions; for example, current speed, road segment information, surrounding traffic, speed limits, obstacles on the road, and actions could be driving maneuvers such as turn left/right, stop, accelerate, continue. The learning algorithm produces a policy that specifies what action to take in specific configuration of driving conditions.

In IT operations, reinforcement learning enables a self-healing system that learns what actions need to do to recover from an incident, increase data flows, and optimize operations.

Machine Learning Can Solve Key IT Operations Problems

Today’s IT operations, struggle daily to cope with—and derive value from—huge amounts of data generated in dynamic infrastructures and applications. Due to the complexity of the underlying systems, human and policy driven management is, basically, unable to react fast enough and realize value from large amounts of data statistics and patterns.

Machine learning can help IT operations teams to analyze IT performance issues, and provide insights to maintain high levels of availability for critical business systems and applications.

Machine learning relies on these different types of data analysis:

  • Descriptive (data mining): Looks at data and analyzes past events for insight for how to approach the future, quantifying data relationships. Using anomaly detection (supervised and unsupervised learning approach), IT operations can locate problematic behavior changes hidden in huge volumes of operations data, so IT operations can know what happened and get to a root cause faster.
  • Predictive (forecasting): Turns data into valuable, actionable information by using data to predict (supervised learning approach) when problems will occur, given past behavior, analyzing frequent operational patterns that could lead to incidents.
  • Prescriptive (optimization): Automatically synthesizes big data and other inputs to make predictions about what could go wrong and suggest decision options for taking steps to prevent issues.

To automatically identify and isolate disruptions and failures, IT operations needs to be able to identify and predict anomalies and detect risk in IT environments.

Dr. Kaluža, PhD goes on to summarize that due to the complexity of IT systems, machine learning is best geared to automatically and quickly analyze tremendous volumes of data distributed across disparate data stores, identifying patterns for detecting anomalies, and revealing performance and security risks.

Machine learning can help IT managers to not only isolate errors, but also gain valuable insight in real-time into those data anomalies that create system errors and failures. Automated analysis of the data created by IT systems is critical for seeing the clues as to why applications and systems fail. By clearly seeing the associated causes of issues and correlating them to a specific error, operations managers can better maintain peak operational efficiency for their IT infrastructures, reduce the mean-time-to-resolution within support organizations and provide end users with a near error-free experience.

Although I am not necessarily endorsing Evolven, I have recommended them to several of my Global 1000 clients and we are currently in the process of evaluation.

As way of background, Evolven’s Blended Analytics is a game changing IT Operation Analytics (ITOA) solution that correlates and analyzes cross silo data sources to deliver unparalleled IT operations insights.   They have done a brilliant job of integrating all relevant data sources across IT silos (including performance, log, network, deployment automation, service desk and CMDB), and correlates them with changes – the true root causes of performance and availability issues. Powerful analytics that rely on machine learning, anomaly detection and domain specific heuristics turn this data into actionable insights for slashing mean time to resolution, cutting the number of incidents and improving DevOps and Audit. Evolven is a recognized IT Operations Analytics (ITOA) leader and was selected by Gartner as a Cool Vendor in IT Operations. Evolven is also the winner of the Red Herring Top 100 North America. TiE 50 Top Startup, 20 Most Promising Data Center Solution Providers, Banking CIO Outlook and ITOA50 awards

Evolven is a privately held, venture-backed company headquartered in the U.S., with presence in Europe and the Middle East. Evolven’s executive team and advisory board include world-renowned experts in IT management and enterprise software. Evolven’s investors are leading venture capital firms Pitango Venture Capital (www.pitango.com) and Index Ventures (www.indexventures.com).

If you and your organization are interested in the next generation of analytics for your IT Organization (ITOA) and really want to “kick start” the process, I would recommend that you have Evolven on your short list of vendors.  And, I would also recommend that you add questions about support and use of Machine Learning to your list of criteria for choosing ITOA vendors.

 

 

About Charles Skamser
Charles Skamser is an internationally recognized technology sales, marketing and product management leader with over 25 years of experience in Information Governance, eDiscovery, Machine Learning, Computer Assisted Analytics, Cloud Computing, Big Data Analytics, IT Automation and ITOA. Charles is the founder and Senior Analyst for eDiscovery Solutions Group, a global provider of information management consulting, market intelligence and advisory services specializing in information governance, eDiscovery, Big Data analytics and cloud computing solutions. Previously, Charles served in various executive roles with disruptive technology start ups and well known industry technology providers. Charles is a prolific author and a regular speaker on the technology that the Global 2000 require to manage the accelerating increase in Electronically Stored Information (ESI). Charles holds a BA in Political Science and Economics from Macalester College.