Building a Large Scale, Machine Learning-Based Anomaly Detection System - Part 1: Design Principles
January 17, 2017
It has become a business imperative for high-velocity online businesses to analyze patterns of data streams and look for anomalies that can reveal something unexpected. Automated anomaly detection is a technique of machine learning, and it is a complex endeavor. Companies that use the right models can detect even the most subtle anomalies. Those that do not apply the right models can suffer through storms of false positives or, worse, fail to detect a significant number of anomalies, leading to lost revenue, dissatisfied customers, broken machinery, or missed business opportunities.
The techniques described within this three-part white paper series are grounded in data science principles and have been adapted or utilized extensively by the mathematicians and data scientists at Anodot, which was founded in 2014 with the purpose of creating a commercial system for real-time analytics and automated anomaly detection. The veracity of these techniques has been proven in practice across hundreds of millions of metrics from Anodot's large customer base. A company that wants to create its own automated anomaly detection system would encounter challenges similar to those described within this white paper series.
This white paper, part one, covers the design principles of creating an anomaly detection system. It will explore various types of machine learning techniques, along with the main design principles that affect how that learning takes place.