Ease of Use Meets AI: BI and Business Analytics Trends in 2019
TDWI analyst David Stodder looks at the major trends of the year and offers 3 BI trends worth watching in 2019.
- By David Stodder
- December 20, 2018
Business intelligence (BI), visual data discovery, and business analytics solutions continue to evolve toward enabling nontechnical users to interact in more ways with more types of data. More than two decades ago, BI started as a package intended for querying and reporting; then, as solutions added online analytical processing (OLAP) features, data visualization, and mobile functionality, pressure grew to make these complex solutions easier to use. Along with more intuitive graphical user interfaces, search and natural language processing have become part of the solution portfolio. Now, data preparation capabilities are joining the mix, making it easier to select, transform, and improve the quality of data without waiting for IT.
Today, organizations can choose to deploy solutions on premises to keep them behind firewalls and closer to existing data warehouses and production data sources, or they can deploy them in the cloud to save time and expense and to put them closer to the growing data platforms and storage residing there. Our research finds that cloud-based services are growing, but most organizations will have both on-premises and cloud-based solutions for the foreseeable future.
What happened in 2018, and what should we watch for in BI and business analytics 2019? I offer three key trends that were important during the past year and three that will be in the coming year.
Top Trends of 2018
Direct BI Access and Native BI Capabilities on Data Lakes Grew
With organizations consolidating different types of data into either on-premises or cloud-based data lakes, many are eager to enable users to perform BI workloads on them. Historically, organizations have had to extract, transform, and load data from data lakes into data warehouses first for querying, reporting, and data exploration. In 2018, we saw growth and maturation of SQL-on-Hadoop engines and solutions that provide native BI functionality inside data lakes. These options reduce the need to move data to an intermediate platform first.
Search capabilities are also important for easier and faster access to the contents of data lakes given the diversity of data types. We expect organizations to increase their implementation of SQL-on-Hadoop, native BI functionality, and search to access data lakes and pull insights from them into interactive dashboards and other visualizations.
Organizations Pushed for Faster and Fresher Data
Coming into 2018, we anticipated that streaming data and real-time analytics would become a strategic priority for a greater number of organizations, especially those that have deployed Internet of Things (IoT) devices as part of operational technology and Industrial Internet strategies. Streaming data and real-time analytics are already fairly well established for use in cybersecurity, fraud detection, and other areas where immediate awareness is essential.
However, adoption of these technologies is really only one facet of a broader effort by organizations to reduce the latency between the creation and ingestion of data and its availability for reporting, alerting, and analytics. Organizations are pursuing a range of technologies to provide BI and analytics users machine learning algorithms, and automated decision management systems with fresher data and frequent updates. Many have long deployed operational data stores, which today may use Apache Hadoop clusters. Organizations are also using change data capture (CDC) technology to identify and capture changes to data and data structures as they occur and informing users of those changes in an efficient way.
Speed is a competitive advantage in many scenarios, so we expect that the pursuit of near or true real-time insights will continue to be a top objective in 2019. The challenge will be for organizations to correctly match technology options with their workloads requirements.
Technologies and Methods Evolved to Support Greater Agility
Enabling users to explore and analyze a broader and deeper selection of data is great, but it also puts pressure on organizations to improve flexibility and agility. As users explore data, it changes their conception of what they need in dashboards and in BI and analytics applications. If technologies and development methodologies make it difficult for users and developers to institute changes, the end result will be frustration.
Organizations have been implementing agile methods to improve how users and developers collaborate, including to produce incremental releases that enable teams to make adjustments along the way toward final applications and systems. DevOps methods have increased speed and flexibility as well. More recently, organizations have been adopting Design Thinking methods to help teams unleash creativity in developing requirements so they capture more than just standard functionality needs.
Self-service technologies fit well with agile methods and Design Thinking because they make it possible for users to test applications, visually explore data along the way to discover new areas, and then inform development teams about what works and what needs changing. Organizations made strides with agile, DevOps, and Design Thinking methods in 2018 and we expect to see more implementations in the New Year.
Anticipated Top Trends in 2019
More Organizations Will Create a Chief Data Officer Position
The vital importance of data -- the “lifeblood” as many call it -- is driving the establishment of the chief data officer (CDO) function. TDWI research does not yet show that this is a common title. However, we are encountering it more frequently and find that companies are planning to either appoint a CDO or assign “chief of data” responsibility to their CIO, chief analytics officer (CAO), or head of BI and data warehousing.
The problem with assigning CDO responsibilities to other titles is that these executives already have multiple duties. CIOs in particular are in enmeshed in difficult strategic decisions such as determining how to allocate the budget, managing the cost of networks and systems, and moving applications, databases, and other IT assets to the cloud. Organizations need leadership that is focused on the data itself.
Major responsibilities for the CDO are to improve the trust and usefulness of the data, oversee the protection of data assets, and increase the value of these assets. As an enterprise-level position, CDOs can take a cross-functional view of issues such as data quality and governance, including adherence to regulations; in many cases, the CDO’s primary responsibility is governance and regulatory adherence.
CDOs could also champion development of resources that capture and centralize knowledge about data and information such as catalogs, business glossaries, master data management, and semantic data integration.
Finally, we see organizations envisioning the CDO as the person who identifies and drives opportunities to monetize data assets. As data rises as a subject of strategic focus, we expect more organizations to establish a CDO position or in other ways centralize chief of data responsibilities.
User Empowerment, Not Self-Service, Becomes the End Game
The explosion in self-service BI and analytics solutions has given users preferable alternatives to using spreadsheets, reporting tools, and waiting in the IT backlog queue for developers to create more substantial applications for them. Self-service solutions allow users freedom to do more on their own, including data selection, preparation, blending, and visualization. Users are better able to start with the business questions they want to answer and make those the context for data analysis and visual discovery rather than have their experience constrained by the limits of what they know about SQL and how the data is organized.
However, just giving users tools and leaving them on their own is not enough. Users still need to work with IT to expand, protect, govern, and sustain what they may have achieved with self-service tools on a limited scale. In our research, we find that organizations are concerned that democratization through self-service BI and analytics will introduce data chaos. Problems with the data, including lack of trust in its quality, can push organizations further from data-driven goals, not closer.
Thus, in many organizations, the self-service objective is turning into a focus on user empowerment. In this vision, rather than view them as opposing forces, governance and self-service fit together in a single strategy. IT embraces self-service technology solutions to give users the power to personalize how they explore, access, interact with, and visualize data. Users know best what they need. Then, IT -- perhaps led by the CDO if the organization has one -- creates an environment where users are provisioned with trusted, governed data. IT provides stewardship to guide users in selecting data sources, reusing preparation processes, and choosing appropriate visualizations such as the right dashboards.
Technology vendors are responding to the focus on empowerment rather than pure self-service. In 2019, we will see further integration between governance and self-service, with solutions coming to market that offer sophisticated capabilities to enable IT or CDO guidance.
AI Will Further Augment Business Intelligence
As my TDWI colleagues have noted, AI is finding its way into every facet of analytics, BI, data integration, and data management. AI for BI will have a significant impact on the nontechnical user experience. Organizations will no doubt use AI to devise algorithms that automate repeatable decisions, but the chief aim of AI for BI will be to augment human decision making.
The self-service BI trend has been critical to enabling users to become fluent in the use of visualizations and data so that analytics insights are a natural part of decision making and collaboration. The augmented intelligence provided by AI will improve self-service capabilities further to let nontechnical users engage in more relevant data interaction.
Machine learning, deep learning, and natural language processing will also expand the scale and speed with which users are able to gain insights from larger volumes of data and content. AI-infused BI systems will become active partners in decision making by delivering timely insights and prescriptive, in-context recommendations tailored to users’ needs. In 2019, we expect to see continued innovation in using AI to augment BI and analytics for nontechnical users.
BI and Analytics: Getting Faster and Smarter
Making BI and analytics tools and applications easier to use has been a key goal in the evolution of self-service technologies. Now, with AI capabilities embedded in solutions, BI and analytics for nontechnical users should become not just easier but smarter and faster.
However, AI is not magic, and technologies alone cannot drive better use of data. In 2019, organizations will also need to institute development methods, governance, stewardship, and shared best practices if they are to increase user satisfaction and move closer to achieving their ultimate data-driven objectives.