Executive Summary | Achieving Scalable, Agile, and Comprehensive Data Management and Data Governance
This TDWI Best Practices Report focuses on understanding current challenges to data governance, integration, and quality.
- By David Stodder
- September 6, 2023
The data explosion continues to accelerate across distributed landscapes with data on premises and on multiple cloud data platforms. Organizations face challenges as well as tremendous potential for increasing the value of data assets, including through data monetization—potential that can go untapped without good data management and governance. Most organizations have a democratized spectrum of users, creating ever greater demand for data inside and outside their organizations. This drives the need for diverse types of data and different levels of data timeliness, as well as differing requirements for data governance, integration, and quality.
This TDWI Best Practices Report focuses on understanding current challenges and providing best practices insights for modernizing processes and deploying technologies to solve them. Analytics workloads augmented with AI/ML are critical to competing in every industry. Data-driven business initiatives depend on scalable, agile, and comprehensive data management and governance. The latest applications embed sophisticated analytics using AI/ML capabilities that must be provisioned with continuous, integrated, curated data to deliver insights to all users. Flexibility is key to keeping pace with business demand and unanticipated events.
Organizations are advancing with AI/ML through easier, automated capabilities such as autoML. Some are investigating large language models (LLMs) and generative AI. To move forward, organizations need to modernize data management, integration, and governance and align investments with evolving business requirements. Legacy technologies and practices often force data scientists, data and business analysts, and business users to spend too much time acquiring, integrating, and preparing data. We discuss how AI-infused automation in data integration and preparation processes are maturing to enable users to focus more time on solving business challenges and achieving data-driven innovation.
Our research finds that most organizations have only isolated success in managing and governing data to meet objectives. Accelerating growth in data volume, workloads, and users across distributed and disparate data landscapes generates pressure that can lead to chaos and higher costs. Our report discusses how organizations can improve the balance between enterprise data governance and the agility required for self-service user empowerment.
Limited data access is a problem when organizations are trying to develop new insights about concerns such as customer behavior, supply chains, public health, operational cost drivers, and business performance. Distributed data dispersed across silos is a major challenge to gaining complete views of all data about these concerns. It also presents challenges to holistic data governance and management. Research in this report shows many enterprises now have experience with or plans for bridging distributed data through data virtualization, data mesh, and data fabric architectures or consolidating disparate data into a unified cloud data platform.
All data architectures today rely on technology modernization to capture and manage metadata and other knowledge about all the data, including data lineage. Organizations are expanding use of data catalogs and additional data intelligence and semantic layer systems. This report discusses current satisfaction with data catalogs, business glossaries, and metadata management systems and where organizations need to improve to increase satisfaction.
The report concludes with a discussion of how modern technologies and practices are coming together to create unified data environments. It discusses the importance of making this unity flexible rather than restrictive. Finding the right balance enables organizations to empower teams to maximize the value of enterprise data assets. We close the report with 10 recommended best practices for success.
Alation, Denodo, Dataiku, erwin by Quest, Hitachi Vantara, SAP, and Snowflake sponsored the research and writing of this report.
About the Author
David Stodder is senior director of TDWI Research for business intelligence. He focuses on providing research-based insights and best practices for organizations implementing BI, analytics, data discovery, data visualization, performance management, and related technologies and methods and has been a thought leader in the field for over two decades. Previously, he headed up his own independent firm and served as vice president and research director with Ventana Research. He was the founding chief editor of Intelligent Enterprise where he also served as editorial director for nine years. You can reach him by email ([email protected]), on Twitter (twitter.com/dbstodder), and on LinkedIn (linkedin.com/in/davidstodder).