TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The Primary Principles of Business-Incident Detection for BI

Is your enterprise making the most of incident and anomaly detection?

• By David Drai
• January 3, 2017

The emergence of business intelligence was supposed to finally enable businesses to truly utilize the massive amounts of data running through their systems. When the field initially began, offering visualizations or basic parameter settings to detect business incidents, the overwhelming feeling was optimistic that from these beginnings -- however humble -- the BI sector could begin to unlock new truths hiding right under the nose of the average analyst.

However, the reality (and now the status quo) is that the initial offering has been the only offering. Massive leaps forward based on innovative concepts have failed to materialize and progress in other sectors quickly lapped what the leading BI players could produce.

As BI accommodates increasingly large volumes of data, the technologies that facilitate the collection, analysis, visualization, and reporting of business data must become more advanced. From simple analytics to advanced machine learning and anomaly detection, it is imperative for thought leaders and decision makers to understand how BI leverages the exploration of data to discover potential anomalies and strategic business opportunities.

The Expectations for Existing Solutions

There are many industries (such as e-commerce, fintech, and adtech) that find their success in the assessment of their collected data. The ability to apply predictive analytics relies on analyzing immense amounts of data from a variety of origins to learn the normal behavior of the data. This builds patterns to help understand and produce predictive analytics, leading to anomaly discovery.

What sets apart businesses providing BI solutions is timing and accuracy. The collection and analysis of data results and analytics visualizations can be output at varying frequencies, such as every minute, once an hour, or daily.

When an e-commerce business sees conspicuous buying behavior, it needs to understand the cause. Amazon noticed an unusual decrease in sales in July of 2016 during Amazon Prime Day -- second in revenue generation only to Cyber Monday. The company addressed the situation on social media, stating that the "add to cart" feature had a technical glitch, preventing customers from finalizing purchases.

Amazon needed to immediately locate the root cause and notify technicians about the detected anomaly. Delivering results quickly can be the deciding factor in making millions or losing millions. Today's current expectation is that the speed of assessment should match the quality of results.

Staying Alert for Success

The primary challenge for any automated business intelligence service is to scale the number of KPI alerts and identify the significant ones. Business-incident detection relies on KPIs such as transactions, transaction volume, clicks, conversions, page views, and the millions of possible metric combinations that divide into alert parameters.

There is no limitation to data points or flexibility for setting alert notifications, and by connecting a series of alerts, an enterprise can significantly increase the quality of insights in areas that require attention. Many businesses try to build in-house anomaly detection but suffer from alert storms, making it difficult to differentiate redundant anomalies from significant ones -- a trade secret that separates the competition. In some instances, an enterprise must combine multiple KPIs to identify trends and patterns.

For example, IoT-connected smart homes and city infrastructure rely on an elaborate array of sensors in constant communication. If a water pipe bursts repeatedly in the same location, accumulated data may alert technicians that the water control systems are distributing too much water at certain times, forcing excessive pressure on a specific area.

The correlation between different results provides the clearest picture of the root of an error and what solution will resolve the issue. Connecting multiple correlations leads to limitless possibilities for anomaly identification and resolution. It all begins with one alert.

The Application of Incident Detection

Business incident and anomaly detection through machine learning is a major step forward for all online industries, which is why Gartner lists intelligence as one of the top technology trends of 2017. Machine-learning algorithms get more intelligent with increased use, making the continuous streaming of raw data for BI serve both the quality of intelligence and the technology itself -- enabling unprecedented insights and identification of the sources for business incidents.

Any industry, enterprise, or business has a multitude of simple or complex data that can be assessed to locate typical and atypical behavior. E-commerce, fintech, adtech, IoT -- every industry seeks to perform at its peak, and machine learning can immediately identify revenue leaks or glitches that undermine business. In the past, that was simply too complex, but the future has arrived through machine learning.

A marketing campaign's success can be difficult to measure, but with data and digital analytics, the underlying results are full of new learning opportunities. The potential for applying the techniques of business-incident detection is only limited by the amount of data collected -- the room to expand is immeasurable.

As enterprises continue to accelerate growth and accumulate data, it becomes increasingly important to leverage data to create more efficient and cost-effective businesses. Without these innovative solutions, companies will simply stay in the dark -- they will miss out on promising opportunities and be unable to prevent many potential challenges.