TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

In Search of…Smarter Information Analysis

The convergence of data mining and artificial intelligence technologies has some folks screaming Big Brother—perhaps with good reason.

• By Stephen Swoyer
• November 16, 2005

At this week’s Institute for Operations Research and the Management Sciences (INFORMS) annual meeting, members of the INFORMS data mining and artificial intelligence (AI) caucuses are expected to vote on a particularly tendentious proposition—namely, to merge their two sections into one.

As INFORMS member and SAS Institute Inc. marketing director Mary Crissey concedes, this proposal is almost certain to excite impassioned debate on either side of the aisle. At the same time, however, it’s a reflection of the natural convergence of these two largely autonomous technology domains—fueled, in part, by the disruptive events of September 11, 2001.

The impetus? Smarter data mining and information analysis tools.

And as business intelligence and data warehousing guru Mike Schiff points out, there’s a lot of commonality between what folks like Crissey call “operations research” (OR) and AI. If nothing else, a lot of the insights that data mining and information analysis tools unearth can seem spooky—even quasi-intelligent. “Operations research is really just a superset of data mining. But it includes the statistical analysis piece, so the convergence of [OR and AI] would make sense. It definitely should be an exchange of information between the two, because basically they’re both used to solve problems,” says Schiff.

As a graduate of MIT’s Sloan School of Management, Schiff knows a thing or two about OR. And to the extent that OR practitioners can tap some of the techniques developed by AI researchers—e.g., the use of rules and inferencing algorithms, genetic learning capabilities—to improve (or in some cases automate) the actionable insights that are unearthed as part of the data mining or information analysis processes, he says he smells an opportunity. “Companies need to be able to both analyze and operate—to take the fruits of this [analysis] and turn it into something actionable,” he argues. “I think there’s a lot of commonality to begin with [between data mining and AI], in terms of sort of the pattern recognition and learning techniques, so the idea is to make these tools even smarter.”

Naturally, the convergence of data mining and AI technologies has some folks screaming Big Brother. One eerie example was the Defense Advanced Research Project Agency’s (DARPA) Total/Terrorist Information Awareness (TIA) project, which was itself the brainchild of DARPA’s Information Awareness Office. TIA proposed to amass information about everything—e.g., personal Web surfing habits and Internet activity, credit card purchasing histories, tax returns, airline ticket purchases, car rentals, medical records, utility bills, and any other available data—and sift through it to uncover insights.

But TIA was positioned as more than just a data mining tool. DARPA researchers proposed a range of AI-like complementary services, including effective affordable (EAR) reuse-to-text capabilities; “Genoa,” a program designed to automate decision-making in order to “deal with and adjust to dynamic crisis management;” human identification at a distance capabilities; and an asymmetric war-gaming AI. (A successor program, Genoa II, purports to improve—if not automate—collaboration between and among government agencies.)

TIA ignited a firestorm of controversy when its existence was first disclosed. So much so that DARPA ultimately killed funding for it in September of 2003. (It’s still possible that TIA technologies could resurface in other government departments and organizations, of course.)

Many data mining and OR advocates—not to mention AI researchers—are extremely uncomfortable with the kind of “intelligence” that’s embodied in the TIA research. SAS’ Crissey, for example, says she is all for smarter data mining tools—inasmuch as such tools are explicitly designed to help improve interactive human use. As for automation via quasi-intelligence, Crissey says, “I would really want a human being [involved]. I think most of our customers expect that [the output of] analysis will be interpreted by a human being. The point is that the machines are not running by themselves. I am strongly in favor of the human participant being a valuable part of the decision-making process.”