The Road Ahead: Demand for Data-Driven Business Agility Will Continue to Rise
Enterprises have long dreamed of delivering the right information to the right people at the right time. In 2021, three developments may help make that dream come true.
- By David Stodder
- December 21, 2020
With the difficult, challenging, and tragic 2020 behind us, the coming year promises new twists and turns as organizations try to determine "the new normal" as each month unfolds. Customer channel preferences, supply chains, logistics, and service demands are sure to change as more people are vaccinated and the economic environment hopefully turns more positive.
Change accentuates the importance of empowering users and applications with access to quality, timely, and relevant data and information. Thanks to technology innovations such as artificial intelligence, massively scalable cloud data platforms, and advances in data virtualization and integration, the longstanding dream of delivering the right information to the right people at the right time is closer to reality. Here are three areas where developments will be key in 2021.
2021 Trend #1: Intelligent business analytics applications become active partners to users
Data democratization is driving demand for business analytics applications that go beyond traditional, "passive" relationships with users. Traditional BI systems deliver scheduled, carefully prepared reports and dashboards and then wait for users to initiate a query or search.
Although this mode may work for business and data analysts who are experienced at querying and exploring data – and, of course, increasingly intuitive, self-service tools are helpful to these analysts – less-experienced users need more "active" solutions. Technology trends are delivering solutions that offer more active relationships with users, something that is especially important as nontechnical users seek deeper interaction with data.
New solutions are flipping the traditional paradigm to deliver answers to users proactively based on the system's knowledge of the data and user patterns and requirements. AI techniques such as machine learning and natural language processing, combined with automated delivery options, can augment human intelligence by automatically finding and delivering contextual data insights without waiting for the user to intentionally search or query the data.
Augmenting human intelligence with AI techniques and alerts and notifications on all devices (triggered by key events, time schedules, or threshold values) can take self-service BI and analytics to a different level, where the system becomes an active partner rather than a passive system.
As users move through a process that might include forms, text, reports, or other application interfaces, AI-driven systems can anticipate users' needs and provide them with timely information within the context of their work. This proactive approach can save time and improve the accuracy and relevance of the data users are working with, whether they are implementing standalone business analytics solutions or embedded functionality within business applications.
TDWI research finds that organizations are highly interested in the potential of intelligent, AI-driven applications for daily decisions. In our Q3 2020 Best Practices Report (Evolving from Traditional Business Intelligence to Modern Business Analytics), more than two-thirds (68 percent) of those surveyed say that they would like to see dashboards and KPIs integrated with AI-driven insights. With the use of mobile devices now ubiquitous, leading-edge intelligent applications will be able to supply AI-driven insights, personalized visualizations, and recommendations to users on the go based on their location and the equipment or other resources they are working with, and customers, partners or other employees with whom they are engaged.
2021 Trend #2: Self-service data preparation and transformation take advantage of processing power
Data preparation steps such as cleansing, transformation, integration, and enrichment are essential to creating usable data for reporting, visualizations (such as dashboards), and analytics. However, delays, errors, and inconsistencies remain rampant within and between data preparation steps. To increase time to insight, organizations need to apply modern technologies and practices to improve and streamline data preparation. In 2021, we will see continued advances in the use of AI and automation to reduce manual work and speed up data preparation steps.
Self-service data preparation and pipeline development is an important trend for enabling all types of users to do more on their own and be less dependent on IT, both for technical expertise and developers' and administrators' time. The same Best Practices Report found that nearly a third (31 percent) of organizations surveyed by TDWI say that their users would like to have more self-service data preparation capabilities in the near future. Nearly half (48 percent) say that their users can already blend, consolidate, join, and link records in a self-service fashion. About the same percentage (47 percent) can create calculated fields, dimensions, or aggregations for multidimensional analysis on their own; when we surveyed organizations in 2016, only 14 percent said users could perform activities with self-service tools.
Extract, transform, and load (ETL) routines and data pipelines are becoming performance and management headaches in many organizations. Hundreds, if not thousands, of them might be running to support various reports, metrics, dashboards, and analytics. As organizations migrate more of their data management and preparation to the cloud, they should use the opportunity to modernize rather than just carry over existing processes.
This can include shifting some ETL routines to ELT: that is, extract, load (typically into a cloud storage-based data lake), and then transform. This can allow organizations to take advantage of powerful and scalable massively parallel processing (MPP) database engines for faster and more reliable performance of complex data transformation and cleansing rather than have the additional step of moving data to a separate staging area. As organizations shift to ELT, they can eliminate ETL routines that are no longer necessary, such as routines that are serving legacy reports that users no longer value.
In 2021, we will see technologies advance in self-service functionality and automation, enabling different types of users to develop and run ETL and ELT routines and data pipelines with better performance. These advances are important to faster insights and data-driven business agility.
2021 Trend #3: Data catalogs make it easier to locate, govern, and manage diverse data
"Data cataloging, metadata management, and semantic data integration will advance" was one of my top trends for 2020. Indeed, the buzz about data catalogs was loud throughout the year as organizations sought solutions that would make it easier and faster to locate, integrate, and manage diverse and growing data.
Governance pressures also brought attention to data catalogs as a place to gather important knowledge about data lineage: that is, the data's origin, who is responsible for it, and what has happened to it during its life cycle. Solution providers responded with an array of data catalog, business glossary, and governance access control technologies and services.
TDWI research indicates that this area remains a work in progress for most organizations. Less than a quarter (23 percent) of research participants in that Q3 2020 Best Practices Report voiced satisfaction with how well people could use their current data catalog to find data, understand data relationships, and tap it as a knowledge asset about data and its lineage. About half (49 percent) of research participants said they need a major data catalog upgrade, and almost a third (30 percent) say they need at least some improvement.
Traditionally, creating data catalogs, glossaries, and metadata repositories has been manually intensive, which often leaves them incomplete or out of date. Modern solutions are applying AI and automation to reduce manual effort and enable organizations to gather metadata and additional information about data from larger and more complex volumes such as the contents of a data lake. With enabling users to find data more easily a key goal, advances in using natural language search will be important to success with data catalogs in 2021.
Data catalogs can play an important role in data preparation, data virtualization, and data pipeline development. Modern data catalogs can actively supply information about data into the user's business analytics workflow as they explore data. For example, catalogs can notify users of the data's quality, age, and privacy and governance constraints at the point of data use rather than requiring them to go to a separate system or consult administrators. Catalogs can be set up to provide users with previews of contextual information about the data relevant to them as authorized by predefined compliance policies. Crowdsourcing capabilities can enable users to add annotations about data sets and queries to improve collaboration.
Building on the advances of the past year, organizations should find it easier in 2021 to set up and gain value from data catalogs and make them core data integration resources for single projects, departments, or for the entire enterprise.