Why Analytics Must Focus on Outliers and Idiosyncrasies
Traditional analytic best practices are slowly being supplanted by what might be called the analytics of the particular.
- By Stephen Swoyer
- September 15, 2010
Author and data warehousing (DW) consultant Mark Madsen, a principal with DW consultancy Third Nature, is a big proponent of bottom-up disruption.
In his keynote address at TDWI's World Conference in Las Vegas earlier this year, for example, Madsen used historical examples to illustrate some of the ways in which bottom-up innovations get subsumed by -- or wrested under the thumb of -- top-down management efforts. In a recent whitepaper, Madsen revisits this theme, outlining several ways the Age of Big Data has transformed an already analytics-hungry retail industry.
Although his conclusions are specific to retail, Madsen's broader points -- that traditional, top-down analysis only gets one so far and that shops should consider seemingly idiosyncratic behaviors that often indicate emerging or prevailing trends, -- have universal applicability.
"Analytic models and data in retail today are designed around the 'average customer' and a single channel," Madsen writes. "Planning using the assumption of an average customer ignores what we now know about how customers drive profitability. Effective retail requires more sophisticated customer segmenting strategies as well as understanding behavior across a growing set of channels."
For most of the last 20 years, Madsen points out, retailers have used analytics to identify or fine-tune techniques -- e.g., to better understand demand behaviors or to optimize product assortment and pricing schemes. In this model, analysis tended to focus on the big picture -- i.e., on identifying trends or behaviors that were believed to be common to customers, partners, or suppliers, among others -- at the expense of the particular. In this scheme, the rest really was "noise."
Analysis of this kind is inherently pragmatic; it gives business decision-makers an incomplete picture. This used to be considered an acceptable trade-off, chiefly because of the capacity of the technological cutting-edge.
In the Age of Big Data, that's no longer the case, Madsen says. "Through the 1990s, retailers developed techniques to understand demand and to optimize assortment and pricing. The era of category management brought both efficiency and profitability, but treated the customer as a mass or a handful of large, generalized segments. In recent years, the retail go-to-market strategy has been shifting to shopper marketing. The core idea is for retailers and manufacturers to work together to market and merchandise products with the goal of providing a better shopper experience."
In the Age of Big Data, Madsen argues, what analysts used to dismiss as "noise" actually has value. "Noise," in this context, describes the idiosyncratic, the anomalous, the outlying -- the non-average-able.
An analytic effort that focuses on "noise" is an explicitly bottom-up innovation, Madsen argues. "Top-down analysis involving product sales is now a bottom-up exercise, teasing out aggregate behaviors from individual idiosyncrasies," he explains. "Store formats and assortment should be driven by these details, necessitating changes to planning and analysis. Marketing and merchandising optimization need to incorporate the behavioral differences derived from customer and channel analysis. In order to compete effectively, retailers must operationalize these new customer- and channel-driven practices."
Madsen is writing specifically about the retail industry, but many believe his observations can be extrapolated to other verticals, too.
Take Darren Taylor, vice president of integrated business systems with Blue Cross and Blue Shield of Kansas City (Blue KC). Back in 2004, Blue KC initiated a top-to-bottom overhaul of its data management infrastructure, in tandem with HP BI Solutions, the business intelligence (BI) services arm of Hewlett-Packard Co. (HP). In the process, Blue KC effectively backsourced almost a dozen wellness programs, consolidating its health management information into a single centralized data warehouse. As a result, Blue KC now has bigger ambitions on the wellness front.
Two decades ago, for example, its health management program chiefly focused on customers who suffered from catastrophic illnesses. Over time, Blue KC extended this focus to include chronic-care patients, too. "They're a huge cost component, and there are things that you can do to have them [lead] a better lifestyle," he explains.
Taken together, catastrophic and chronic suffers account for most of Blue KC's cost outlays. At the same time, both categories comprise a fraction of Blue KC's overall subscriber base. "You have 10 percent of the people in your plan who are driving 90 percent of your cost, so the other 90 percent of people [traditionally] were handled differently," says Taylor.
Thanks to its data management overhaul, Blue KC now has the DW and analytic capacity to -- in Madsen's terms -- focus on the idiosyncrasies. After all, the healthy subscriber of today could be the chronic sufferer of tomorrow. "If you look at a healthy population, there are actually these ticking time bombs, these people with undiagnosed issues or with potential [health issues] that -- if they were diagnosed [proactively] -- could be handled much more effectively, so our focus is on touching and managing 100 percent of [our customers]. It's on getting outside of that 10 percent [which consumed most healthcare spending] and getting to that other 90 percent," he explains.
At this level, says Taylor, it's all about idiosyncrasies. Identifying idiosyncrasies, of course, involves collecting and analyzing information about customers -- or subscribers, in Blue KC's case -- across several different domains.
In the retail vertical, Madsen explains, this involves integrating data from marketing, merchandising, inventory planning, and store operations. In Blue KC's case, it involves not just integrating data from health or wellness management programs, but -- to the degree that "subscribers" are also customers -- sales, marketing, and other domains, as well. To put it another way, "subscribers," just like customers, want to be cosseted, too.
"Sometimes it might be a simple mailing that says you can get a health club discount. Only maybe a person doesn't want a mailing -- maybe their preferred contact [method] is via e-mail or text. That's [an area] we're starting to explore, identifying things like how [a person] prefers to be contacted and [incorporating that] into the program," Taylor explains.
Blue KC's experience underscores a hard reality, at least for some shops: just as traditional analytic practices are insufficient to the task of selling and marketing in the 21st century, so, too, is the analytic infrastructure of yesteryear.
"Market and the necessary adjustments to retail strategy require new and deeper analytic support. Most organizations have the information necessary to address current operations, but not the detailed data or infrastructure to support new retail practices," Madsen writes, adding that "the challenge all organizations face is the creation of a platform to pull all the required information together and deliver it when and where it is needed. This applies whether the reporting and analysis is done internally or through outsourced operations."