Anchored but Agile: The New Vision for Data Management
After experiencing the problems of data chaos, many organizations want to learn how they can balance self-service and big data with centralized data and information management.
- By David Stodder
- April 30, 2018
In an age when user freedom as well as self-service BI and analytics are prized, it can be difficult to make the case for modernizing shared data and information management. Why not just throw all the data into a humongous data lake, set a few simple rules, and let everyone fish for what they need? Consistency is for cowards. If marketing and sales want to own their data, let them!
Alas, after much experimentation, organizations are experiencing some of the downsides of letting self-service BI and analytics run loose. Many are now refocusing on how they can modernize centralized data and information management. Ironically, organizations are finding that they need strong shared resources to support the independence desired by business users, analysts, and data scientists.
This year, TDWI's Chicago Conference (May 6-11) and Leadership Summit (May 7-8) will examine how to establish a better balance between the freedoms of self-service and big data and the necessities of data and information management. There's no doubt that many IT departments must change their approach to enable greater agility, easier integration with new data, and innovative data analytics. Even as governance and regulatory adherence pressures mount, "no" cannot be the answer to what personnel want to do. Fortunately, newer technologies and methods such as artificial intelligence (machine learning), graph databases, and semantic modeling and integration are giving organizations better options for improving data and information management, including through automation.
Chief Data Officers (Or Someone Like Them)
TDWI is tracking the rise of the chief data officer (CDO), a relatively new title being bestowed upon the executive or manager responsible for increasing and protecting the value of data assets. Our research does not yet indicate that large numbers of organizations have CDOs, but with the rising importance of data and ambitious goals to be data-driven, we see many organizations trying to centralize authority over data assets.
Key areas of focus for CDOs and those entrusted with CDO-like responsibilities include governance and regulatory adherence, expanding the data realm with new assets, identifying opportunities to monetize data and analytics, and making sure users, analysts, and data scientists have the quality, relevant data that they need.
Of course, in many organizations, chief information officers (CIOs), chief analytics officers (CAOs), and the heads of BI and data warehousing may already hold some CDO duties. Titles often overlap when organizations are changing; to avoid turf wars, it's up to the CEO or COO to clarify who is chief of what and who reports to whom. (At both the Conference and Leadership Summit, speakers will discuss how to craft the right strategy for realizing greater value from data assets, including who should be in charge.)
Spotlight on Data Catalogs and Metadata
To manage diverse data assets more effectively and make it easier for personnel to find and access data, organizations need to provide resources such as data catalogs, metadata management, and master data management. Recent TDWI research finds that about a fifth of organizations surveyed currently have a central data catalog or marketplace and nearly half plan to have one within three years. (For more on this topic, see TDWI Pulse Report: Reducing Inefficiency and Increasing the Value of Analytics and Business Intelligence.)
The marketplace concept takes a data catalog a step further by assembling access to a selection of trusted and curated sources (sometimes also new and less well-understood ones), data services, and a knowledge base about the sources and services so that personnel can select those relevant to their projects.
It's not easy to keep these systems and practices up to date when organizations move in a self-service direction and invest in less well-manicured data platforms such as data lakes and cloud storage, but it's worth it. Along with managing the data inventory, analyzing data relationships, and making access easier and faster, data catalogs, metadata management, and master data management are critical to governance. It's hard to effectively govern data when you have no systematic way to find the data, understand how different items are related, and track how it flows through the organization's data pipelines.
Historically, one of the biggest problems with these resources has been the manual effort required to create and maintain them. In 2018, however, data catalogs and related systems have become a hot technology area as vendors offer solutions that implement machine learning, search, natural language processing, graph databases, and visualization. (At TDWI Chicago, you can learn how organizations are modernizing their data catalogs and related resources.)
Three Key Objectives
Organizations across industries have many reasons for improving data and information management, but there are three drivers TDWI sees often:
Customer 360. This expression has become shorthand for gaining a single view of the customer's (or segments of similar customers') journey -- from their first interaction with your organization to the decision to buy or not buy a product or service and any interactions afterwards. Today, these engagements occur across numerous channels. When organizations can gather and relate diverse data from these channels, they can analyze it for patterns, gain predictive insights, and then be proactive in how they engage customers.
Organizations require single views of information for a variety of reasons, but a single view of customer data is a competitive advantage. At the Chicago Leadership Summit, Andrew McIntyre, VP of technology, and Chris Brummett, manager of DW and BI, will describe how they built 360-degree views of customers and achieved critical operational insights for the Chicago Cubs.
Monetization. Organizations can't create data-driven services for customers and partners without strong and agile data and information management. At the Leadership Summit, Aaron Garnett, senior director of Global Enterprise Analytics at Discovery Communications, will discuss how his entertainment firm gave executives and managers a unified view of content so they could make more informed investment, production, and programming decisions.
Governance. As I mentioned, governance improves with complete and accurate knowledge about the data. Many speakers at the TDWI Conference and Leadership Summit will be touching on governance as part of data and information management strategy. The conference will feature an in-depth seminar, "Data Governance Fundamentals: Adapting for Agile, Big Data, and Cloud," by analytics consultant and educator Richard Hines. In his Modern Data Strategy & Architecture Bootcamp, Evan Levy, VP of business consulting at SAS, will discuss how governance methods fit into a successful data strategy.
Anchored but Agile
It's not easy to provide a solid core around which self-service BI and analytics, explorations into big data, and other adventures can thrive along with more staid activities such as governance. It takes creative thinking, investment by both business and IT, and use of the latest that technology solutions have to offer. In Chicago, it might also require discussion over deep dish pizza, but that's just a suggestion.