December 1, 2011
Delivering Business Value with Data Integration in 2012
Topic: Data Integration
People often say that data integration should function like plumbing: hidden from view and delivering data wherever it needs be, without incident. Of course, I have yet to meet anyone with data integration so unseen and unproblematic; come to think of it, I've never met anyone who has plumbing that's that good either.
Whether due to mergers and acquisitions, sales territory realignment, or regulatory compliance needs, changes in business requirements are always major drivers behind data integration projects. Homeowners usually don't change the shape of their house every few months. Businesses shift priorities all the time, which means that most data integration projects are rarely finished and are always vulnerable to change.
Thus, although data integration can be a highly technical topic that most organizations try to keep out of discussions with business users, data warehouse and data integration managers should never lose sight of the business purposes behind data integration -- not to mention the business value that can be achieved when projects are successful. In nearly every industry, decision-making is limited by the inability to gain a single view of all relevant data.
High costs and inefficiency often stem from poor information flow between process steps or across functions. Customer satisfaction can also take a hit if sales, service, and support personnel do not have the right data in front of them during an interaction. Data integration plays a key role in addressing these types of business challenges even if business users are not aware of it.
Trends for 2012
As we close out this year and look ahead to 2012, here are four important trends influencing business requirements for data integration projects:
Trend #1: Organizations will protect sensitive data
Most organizations have sensitive data, such as personally identifiable information (PII), scattered in silos throughout their enterprises. This data could exist in application databases as well as spreadsheets, documents, and customer interaction records. Organizations need to discover where this data exists physically and whether it is circulating inside and outside the organization.
Data managers may be unaware of older, hand- coded integration routines or extract, transform, and load (ETL) processes that touch PII. Yet, businesses could be exposed to regulatory or privacy law violations if they are not managing this data properly. Thus, managers need to know what data they are integrating, not just how to integrate it.
Trend #2: Master data management projects will require better profiling and discovery
In 2012, master data management (MDM) will continue to be an important trend in data integration. By focusing on business objects and higher-level domains, MDM can help businesses integrate a wider variety of information than traditional data integration projects can achieve.
However, MDM projects will realize objectives only with good knowledge of underlying data relationships. "Data discovery" is a term perhaps better known in BI circles, but it is also used in data management to describe the effort to locate and document metadata across data sources and uncover relationships between data elements. Without this knowledge, MDM projects fall short of being comprehensive. In 2012, organizations that are pursuing MDM objectives will need to evaluate their data discovery, profiling, and matching capabilities to determine whether they can uncover all the information about the data that MDM needs to integrate information at a higher level.
Trend #3: Advanced analytics will influence organizations to reduce data movement
Predictive analytics, data mining, and other types of advanced forms of analysis thrive when they can operate on big data: that is, terabytes of structured and unstructured data, often in raw, detailed form. Already under stress to support BI initiatives, data integration and ETL processes that involve heavy data movement could hit the breaking point as organizations ramp up advanced analytics.
Organizations will need to evaluate data federation and virtualization, which can give users comprehensive views of data in place rather than physically moving it into data marts or specialized data stores. Analytic applications that need a steady, frequently updated flow of data from multiple business functions can be good matches for data federation.
Trend #4: Business users will get ready for real time
Business users want a steady flow of timely data, whenever they need it. Leading organizations want to respond to trends in data and events as soon as possible, knowing that if they can eliminate delays, they can turn their information prowess into a competitive advantage. They want to deploy analytic models that can operate on data in real time so that decisions and actions anticipate shifts in markets. In addition, growth in BI and analytics on mobile devices will increase demands from executives and frontline sales and service personnel for comprehensive access to timely data.
In 2012, data integration projects will come under pressure to deliver views of data in real time. BI directors and data managers will need to set user expectations carefully, including explaining the concept of "right time" to calibrate exactly how timely data updates need to be.
A New Year
As organizations grow to rely on information to serve customers, improve business processes, and launch products and services into the marketplace, data integration projects will play an increasingly vital role. The New Year should bring excitement to the mundane, unheralded task of integrating data. Who knew plumbing could be so interesting?
David Stodder is director of research for business intelligence at The Data Warehousing Institute (TDWI). David can be reached at firstname.lastname@example.org
Copyright 2011. TDWI. All rights reserved.