TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Bit Stew: When Conventional Data Integration Can't Keep Up

Traditional data integration isn't up to integrating data from non-traditional sources. Nor can it keep pace with self-serving users. Upstart vendor Bit Stew touts an "intelligent" approach to data integration to address a range of contemporary use cases.

• By Steve Swoyer
• October 31, 2016

Conventional data integration was designed for a batch-oriented data warehouse paradigm, notes Andrew Miller, senior sales manager with "intelligent" data integration specialist Bit Stew Systems.

Traditional DI isn't up to the task of integrating data from non-traditional sources such as RESTful services or sensors and connected devices. Nor can traditional DI keep pace with the needs and priorities of self-serving information consumers. Give self-service users access to the data they need and they'll create their own insights. The problem is this isn't happening; self-service users are still starved for data because IT can't ingest, profile, cleanse, and integrate data at scale.

Remove the Gatekeeper to Improve Self-Service

"If you enable as many people as possible in an organization to turn data into information, you're exponentially increasing the information you produce -- information that's available to everybody," Miller says. "Business users understand their part of the business better than anybody else. They want access to their data. Access seems like a pretty basic thing, but often it isn't. IT is the gatekeeper. They're forced into the role. They don't want to be the gatekeeper but they are."

One of Bit Stew's value-adds, Miller claims, is that it helps to free IT from this unasked-for and unwanted gatekeeper role. Bit Stew's "intelligent" DI platform accelerates the rate at which data can be ingested, profiled, cleansed, and managed for consumption by self-service users, he says.

"The number of information points goes up considerably when you arm an entire company [to integrate and explore data themselves] versus just an IT organization," Miller argues.

"What we enable is decentralization of the work [of data integration]. Whether it's our platform or [self-service], both [paradigms] are all around moving work and development effort away from IT -- in the case of self-service, down to the business teams that really understand what they're doing."

Better Data Integration Through Automation and Machine Learning

What, precisely, makes Bit Stew an "intelligent" DI platform? Miller says it uses machine learning (ML) to automate the process of discovering, profiling, cleansing, and integrating data from non-traditional sources.

Bit Stew's core market is IIoT -- the Industrial Internet of Things. The challenge of Industrial IoT is a problem of both data deluge and diversity. The complexity of ingesting, profiling, transforming, analyzing, and (if necessary) persisting IIoT data at scale far outstrips the capabilities of conventional data integration technologies, Miller argues.

In order to support IIoT and IoT analytics, self-service analytics, data discovery, and many other emergent use cases, data integration must become both more automated and smarter.

"Intelligent data integration is automating many of those very manual, very expensive processes around discovering relationships [in] the data, even take it one step back around data quality, being able ... to automate data quality, automate reconciliation and discovery, automate mapping from source systems to targets. That's intelligent data integration," Miller explains.

"We've built this [intelligence] into the platform. [Bit Stew] automatically analyzes your data and sources, [determines] how they relate, what the [appropriate source-to-target] mappings are, and how the data should be ingested," he continued. "Under the covers, we're leveraging combinations of machine learning algorithms. The libraries we include in the product out of the box -- there's no secret sauce algorithms we've come up with, however. We've just taken a wide variety of open source algorithms and we've figured out ways to combine them together and apply them [for data integration]."

After Automation Ends, Algorithms Still Useful

Bit Stew can't automate everything, Miller concedes, invoking the ubiquitous 80/20 rule. Even in cases where Bit Stew's ML algorithms can't completely profile, cleanse, and model data, they help developers or analysts accelerate these tasks, Miller claims.

"The algorithms go through a voting process where you [a human being] can control the voting weights of specific groups of individual algorithms. They [the algorithms] will get you 80 percent of the way [there]. You can interact [with this process] along the way. Part of the self-service story is that it's easier for IT or developers to interact [with the tool] to fix this stuff," he says.

Integration Roadblock Preventing Self-Service

Organizations can and should be enfranchising a much larger proportion of information consumers, Miller concludes. Data integration is the stubborn bottleneck keeping them from doing so.

Conventional approaches to DI can't and don't scale. They couldn't scale in the data warehouse-driven BI paradigm, when turnaround times of weeks to months were common. They especially can't scale in the context of IoT, IIoT, social analytics, data discovery, and other emergent paradigms.

"Data integration is a foundation. You have to accelerate the rate [at which] you ingest and integrate data. The quality doesn't have to be perfect for all users. You just have to make [data] available -- for self-service, for discovery," he notes. "If you make it available [to people] in a format they can use, [self-service] users will turn it into information that's meaningful [to them]."