Executive Summary | Faster Insights from Faster Data
Executive Summary for the TDWI Best Practices Report: Faster Insights from Faster Data
- By David Stodder
- December 20, 2019
Organizations today place a high priority on fact-based, data-driven decision making. This makes speed to insight a competitive advantage. If a retailer can analyze data to uncover a trend in customer preferences before other retailers in the marketplace, they can gain an edge, potentially delivering higher market share, customer loyalty, and profitability. If an insurer or government healthcare agency can use predictive models to detect a fraud scheme before it has a chance to do significant damage, it can save costs and avoid public embarrassment.
There are many more cases where faster insights can be beneficial. However, achieving faster insights can only happen if delays and bottlenecks that exist throughout data life cycles are addressed using better practices and modern technologies. This TDWI Best Practices Report examines where organizations are coming up against barriers to getting relevant data from sources into the right condition for analytics, for developing artificial intelligence (AI) programs such as machine learning to discover insights, and for delivery to the array of users who need insights in time to solve business problems.
Some of the challenges relate to how organizations put together project development teams and determine deliverables. Setting project objectives is a challenge; TDWI research finds that just 9% of organizations surveyed regard themselves as very successful in identifying value measures and quantifiable objectives (see Figure 4 in this report). As projects move toward deliverables, less than half of those surveyed regard their organizations as either good or excellent at testing prototypes and developing proofs of concept. To address these weaknesses, organizations are implementing agile, DataOps, and other methods to help them better organize projects and move faster to create value.
Technology advances are also key. This report discusses how organizations can reduce latency in data preparation, transformation, and development of data pipelines. It details how organizations could be using data catalogs, metadata repositories, and data virtualization more effectively, including for governance. With expense and scalability identified by research participants as two main issues they face, it is not surprising that cloud-based data management, integration, transformation, and development are popular. This report discusses how moving to the cloud solves some problems but spotlights other issues, such as governance and finding the right balance between centralization and self-service environments.
Data itself is getting faster as organizations begin to analyze new sources including streaming data coming from sensors, websites, mobile devices, geolocations, and more. The report finds that some organizations use streaming, real-time analytics and AI to automate decisions and deliver actionable recommendations to users. TDWI recommends that organizations focus on well-defined objectives and also devote attention to their big-picture strategy to avoid letting complexity slow innovation.
Ascend.io, Denodo, Matillion, SAS, and Wyn Enterprise by GrapeCity sponsored the research and writing of this report.
About the Author
David Stodder is senior director of TDWI Research for business intelligence. He focuses on providing research-based insights and best practices for organizations implementing BI, analytics, data discovery, data visualization, performance management, and related technologies and methods and has been a thought leader in the field for over two decades. Previously, he headed up his own independent firm and served as vice president and research director with Ventana Research. He was the founding chief editor of Intelligent Enterprise where he also served as editorial director for nine years. You can reach him by email ([email protected]), on Twitter (twitter.com/dbstodder), and on LinkedIn (linkedin.com/in/davidstodder).