QL2 Tackles Complex, Unstructured Data
QL2 specializes in indexing and querying the most complex unstructured data
- By Stephen Swoyer
- June 25, 2008
Unstructured data can take a variety of forms. Just ask QL2 Software Inc., a business intelligence (BI) start-up that specializes in indexing and querying what you might otherwise think is difficult if not impossible: data from the Web and other sources.
Scot Madill, QL2's director of product management, says these tasks present problems that most organizations are already grappling with -- one way or another.
"There has always been this sampling of people [inside any organization], just grabbing information and throwing it in a spreadsheet. It's one of those informal processes that nobody talks about … [but that] just happens all the time," he comments.
What kind of information are these users working with? Unstructured information -- but not just garden-variety e-mail, Word documents, PDF files, instant messaging transcripts, and the like. He's also talking about an emerging category of unstructured information -- so-called "auxiliary data."
This is information that business users want and need to consume, but which -- for a variety of reasons (many of them having to do with data management best practices or simple logistics) -- IT is unwilling or unable to bring into an enterprise data warehouse (EDW). Madill lumps data from Web services providers (such as any of the Google services -- e.g., Google Maps or Google Weather) or subscription services (such as those maintained by the airline or financial services industries).
Business users want to use this data for a variety of reasons, Madill indicates, but -- owing to the strictures of enterprise data management -- must frequently resort to copying and pasting it into Excel spreadsheets if they're to consume it. Madill and QL2 tout what they say is a better way, using either an on-premises or software-as-a-service (SaaS) model.
"In the past, you couldn't really get the information at [a] scale [necessary] to be able to make operational-level decisions, but now that we can do it at such a large scale [with Web or subscription services] -- I think we're getting 50-70 million records a day for some of our flights customers -- it begins to make sense to put it inside a data warehouse," he explains.
QL2's on-premises offering is powered by QL2 Server, which (like a similar product from start-up competitor ChartSearch Inc.) uses a virtual database layer to help mitigate (by abstraction) the complexities of data source, file format, and network protocol Babel (see http://www.tdwi.org/News/display.aspx?ID=8995). QL2 Server exposes a Web services-standard interface (via the Simple Object Access Protocol), but can also consume queries from its business analytic counterpart, QL2 Studio.
QL2 Server speaks in terms of WebQL, a proprietary spin on SQL that enables it (or any of QL2's SaaS offerings) to parse and query unstructured or semi-structured data. WebQL is an extension of ANSI-standard SQL, according to Madill. "It's our extension to SQL that basically allows developers to use the Internet as a source of data for applications. What WebQL has been optimized to do is to grab information from the Web and deliver that to revenue management systems, BI systems, and the like, and allow market data to be included in operational decisions."
On the analytic side, the company markets QL2 Studio, a sophisticated query and design environment geared toward business analysts. It boasts a drag-and-drop interface and intuitive flowchart features (similar to the ubiquitous Visio model); what's more, Madill says, QL2 Studio can support a range of output options, including spreadsheets, custom file layouts (which can be directly imported into a database), or real-time feeds.
QL2, like ChartSearch, and (to a lesser extent) Composite Software (which markets its own structured search-and-analysis appliance, Composite Discovery), is attracting interest directly the line-of-business users, according to Madill. Business users are hungry for a laissez-faire technology such as QL2, he notes, because it helps them attack a bucket list of analytic scenarios.
"We used to go in to places and people were like, 'My gosh, I need to have that data. I need to have it. Get it to me.' Often, especially in retail, they're like, 'What do we really do with this? Now that we have it, we've never been able to get information like this, so how do we create the business processes to consume it? Their initial reaction is to grab as much data as possible, but a good sampling is probably more prudent than trying to get everything on the site everyday," he comments.
It's for this reason that Madill and QL2 stress the importance of IT buy-in: business users tend to see an agglomeration of data and want to consume it all; IT helps them not only decide how much of this data to grab (and how often to refresh it), but how to encapsulate it within sound data management processes. For the most part, Madill indicates, IT usually comes around to QL2, too.
"IT will know more opportunities than the number of business users that we can talk to. The feedback we get [from IT], once they understand how [QL2] works, how they can fit [it] into their existing [data management] processes, is positive," he concludes, adding that QL2's range of delivery options -- its SaaS services include PriceTrack (a price comparison service), ProductMix, MarketScape (a market trend and consumer behavior- tracking service), and MarketVoice (a sentiment tracking service) -- also makes it a more palatable proposition, from IT's perspective.
"They think putting this kind of technology in their data warehouse stack or their BI stack is a great addition."