TDWI Research Report Examines Improving Data Preparation for Business Analytics
Report examines experiences with data preparation, discusses goals and objectives, and looks at important technology trends reshaping and enhancing data preparation processes.
SEATTLE, WA, July 11, 2016—TDWI Research has released its newest Best Practices Report, Improving Data Preparation for Business Analytics. Business users want the power of analytics—but analytics can only be as good as the data. The biggest challenge nontechnical users are encountering is the same one that has been a steep challenge for data scientists: slow, difficult, and tedious data preparation. This original, survey-based research offers organizations guidance on taking advantage of the rapidly evolving technology and self-service capabilities that allow business users to realize insights from data faster without IT hand-holding.
Data preparation matters—in fact, it is one of the hottest topics in the technology industry and is important to all organizations making decisions and taking action informed by data. Preparation processes include determining what the data is and improving its quality, collecting and consolidating it, and taking steps to make it useful for reporting and analysis.
Faster data preparation enables faster insights that can move a company ahead of the competition, yet the biggest overall complaint about data preparation is that it takes too long. Users across the spectrum deal with data chaos every day, from business users and analysts bogged down with spreadsheets and manual data cleansing to IT burdened with poor, ill-defined data and hand coding of preparation and transformation routines.
Suggesting significant room for improvement, this report’s survey reveals that more than a third (37%) of respondents indicated dissatisfaction with how easily they can find relevant data and understand how to use it appropriately for BI and analytics projects. Other barriers to improving how data is prepared include convincing executive management to support projects, a lack of skilled personnel or training, and difficulty accessing and integrating data across system data silos.
The quest for competitive advantage and the increasing volume, variety, and velocity of data have placed more pressure on organizations to rethink traditional methods of preparing data for reporting, analysis, sharing, and use. New data preparation tools offer solutions that join disparate data sets and enable more nontechnical business users to drive data transformation and integration themselves through self-service and visual analytics tools.
David Stodder, senior director of TDWI Research for business intelligence, explains that “with improved practices and technologies for data preparation, organizations can better deal with current data troubles and prepare for future challenges arising from new data and user requirements…. Most important, executives, line-of-business managers, and the rest of an organization’s personnel will be able to get the necessary data faster and use it with greater confidence for strategic, operational, and financial decisions.”
Report Highlights
This comprehensive report reveals the following best practices:
- Make shortening the time to achieving business insight a data preparation improvement priority. Apply new technologies and methods that trim out delays in getting users from data to insight.
- Focus on reducing how long data preparation takes to deliver valuable data. Eliminate unnecessary routines and develop strategies for increasing the use of automation and standardizing processes for integrating new data.
- Use new technologies and methods to achieve higher levels of repeatability. Evaluate how you can apply technologies and adjust processes so that you can reuse scripts and workflows, and encourage sharing by adopting a collaboration framework.
- Develop shared data catalogs, glossaries, and metadata repositories as part of data preparation processes. Technologies in the marketplace can automate development of these resources to make it easier for users to find data that is relevant and collaborate with others using the same resources.
- Integrate self-service data preparation with self-service BI and visual analytics. New technologies are increasing options for enabling nontechnical users to select, blend, and otherwise prepare data themselves. Evaluate technologies that improve this integration.
Stodder examines experiences with data preparation, discusses goals and objectives, and looks at important technology trends reshaping data preparation processes. He also outlines best practices for reducing the time it takes to prepare data and for integrating data preparation with governance programs.
This research was sponsored by Alation, Alteryx, Attivio, Datameer, Looker, Paxata, Pentaho (a Hitachi Group company), RedPoint Global, SAP/Intel, SAS, Talend, Trifacta, Trillium Software, and Waterline Data.
About the Author
David Stodder is senior director of TDWI Research for business intelligence (BI). He focuses on providing research-based insight and best practices for organizations implementing BI, analytics, performance management, data discovery, data visualization, data preparation, and related technologies and methods. He is the author of TDWI Best Practices Reports and Checklist Reports on visual analytics, customer analytics, BI/DW agility, mobile BI, and information management. He has chaired TDWI conferences on customer analytics, big data analytics, and other topics. Stodder has provided thought leadership for over two decades. He has served as vice president and research director with Ventana Research, and he was the founding chief editor of Intelligent Enterprise. You can reach him by email ([email protected]), on Twitter (@dbstodder), and on LinkedIn.
About TDWI
For 20 years, TDWI has provided individuals and teams with a comprehensive portfolio of business and technical education and research about all things data. The in-depth, best-practices-based knowledge TDWI offers can be quickly applied to develop world-class talent across your organization’s business and IT functions to enhance analytical, data-driven decision making and performance. TDWI advances the art and science of realizing business value from data by providing an objective forum where industry experts, solution providers, and practitioners can explore and enhance data competencies, practices, and technologies. TDWI offers five major conferences as well as topical seminars, onsite education, membership, certification, live webinars, resourceful publications, industry news, and in-depth research. See tdwi.org or follow us on Twitter @TDWI.
About 1105 Media
1105 Media, Inc., is a leading provider of integrated information and media in targeted business-to-business markets, including specialized sectors of the information technology community; industrial health, safety, and compliance; security; environmental protection; and home healthcare. 1105's offerings span print and online magazines, journals, and newsletters; seminars, conferences, and trade shows; training courseware; and Web-based services. 1105 Media is based in Chatsworth, California, with offices throughout the U.S.
Media Contact:
Christin Jamieson
Senior Marketing Manager
425.277.9182
[email protected]