Executive Summary | Reducing Time to Insight and Maximizing the Benefits of Real-Time Data
This TDWI Best Practices Report emphasizes reducing insight times by addressing data quality and integration as well as by leveraging modern technologies for real-time data utilization.
- By David Stodder
- March 13, 2024
Reducing speed to insight has long been an important objective, but many users are frustrated by delays in getting access to the right data and converting it into insights. Challenges are only growing as data gets faster, more varied, and more voluminous and organizations seek to deploy complex workloads that include artificial intelligence (AI) models and algorithms.
Decision-makers are at a disadvantage when they are forced to work with stale dashboards and analytics. Opportunities to use AI to personalize marketing, sales, and service at the point of engagement to increase customer loyalty are lost if data is old and irrelevant. Modern healthcare organizations depend on complete and accurate insights to improve patient care and understand population health. Financial services and insurance firms need timely insights to innovate with AI-driven products, evaluate risks, and detect fraud.
Faster insights can only be achieved if delays and bottlenecks throughout data life cycles are addressed using better practices and modern technologies. This TDWI Best Practices Report focuses on current challenges and how organizations can move forward to reduce time to insight and maximize the benefits of real-time data. Reducing time to insight involves a range of practices, data systems, and solutions to meet varied user and application requirements. The report discusses trends in AI-driven automation and augmentation, including generative AI.
Our research has found that poor data quality is a significant and common impediment. Seeing the relationship between data quality and data latency, organizations are putting a high priority on processes for improvement. They are also focusing on investment in data intelligence. This category of technologies and practices is critical to data quality and to streamlining data governance, which users often view as a hindrance to achieving data-driven goals. It includes data catalogs, metadata management integrated with data platforms, and master data management (MDM). These and related tools are vital to assembling knowledge that makes it easier to find relevant data, know about its quality, and track sensitive data for governance and security.
Organizations need to reduce bottlenecks and improve automation in developing and deploying data integration and preparation processes. Problems such as complex manual coding, redundancy, and excessive data movement are often major contributors to data latency. The research also identifies the problem of already numerous and growing data silos. Silos make it difficult to streamline data integration and ensure complete data quality and governance. The report discusses strategies such as data fabrics and data silo consolidation for solving data silo challenges.
Real-time data is growing as most machines, vehicles, systems, and equipment become connected data sources. Organizations want to tap the value of real-time data from sensors, mobile devices, telemetry tracking systems, and logs that record customer activity on websites. To ensure cost-effective value, data streaming and processing need to be part of a broader strategy for deploying the right technology to fit use cases and workloads. This report discusses the essentials for developing a successful strategy, including using data observability to gain a holistic perspective of how systems and processes are contributing to desired business outcomes.
Among this comprehensive report’s key findings:
- Fewer than one in five of those surveyed reported being successful in enabling users to gain faster insights and having confidence in their ability to meet future challenges; in contrast, 42% said they were somewhat successful currently but were concerned about their ability to respond to future challenges
- Improving trust in data quality, accuracy, and completeness was one of the top priorities for respondents, cited by 57%; nearly the same percentage (56%) said expanding self-service data visualization, BI, and analytics was a priority
- 45% of organizations work with near-real-time data (e.g., data available after a small delay); this delay could be seconds, minutes, or even hours, depending on requirements, curation, and the speed of processing
- Half of respondents said poor-quality and incomplete data were the main obstacles to timely insights for their users; nearly as many (47%) said the lack of a single view of relevant data was their main obstacle
The report concludes with 10 best practice recommendations for reducing time to insight and maximizing the benefits of real-time data.
This research was sponsored by Denodo, SAP, Snowflake, and ZoomInfo.
About the Author
David Stodder is senior director of TDWI Research for business intelligence. He focuses on providing research-based insights and best practices for organizations implementing BI, analytics, data discovery, data visualization, performance management, and related technologies and methods and has been a thought leader in the field for over two decades. Previously, he headed up his own independent firm and served as vice president and research director with Ventana Research. He was the founding chief editor of Intelligent Enterprise where he also served as editorial director for nine years. You can reach him by email ([email protected]), on Twitter (twitter.com/dbstodder), and on LinkedIn (linkedin.com/in/davidstodder).