Smarter and Faster: BI and Visual Analytics Trends in the New Year
Two trends from this year enabled organizations to get more value faster from their BI and visual analytics projects. What's ahead for 2018?
- By David Stodder
- December 14, 2017
Excitement continues to grow about how organizations can realize the potential of data through powerful, democratized business intelligence and visual analytics. Executives and managers in lines of business and departmental functions want to gain comprehensive situation awareness and an understanding of predictive patterns by analyzing diverse data.
Traditionally, organizations have had to be patient with the slow pace of development and deployment of BI, analytics, and supporting data management solutions. However, smarter, faster, and more automated technologies -- along with flexible development methods -- are helping to reduce delays.
What happened in 2017 to enable organizations to speed the path to value with BI and visual analytics? What should we watch for in 2018? I offer two key trends that were important during the past year and four that will be in the coming year.
Top Trends of 2017
2017 Trend #1: Innovation took "ease of use" to a new level
For most business users, the easiest way to interact with data wasn't through reports and code but via their natural language. Last year, as a future trend for 2017, I prognosticated that "natural language search will have a bigger role in BI and analytics." As the year unfolded, the vogue demo at several vendors' annual conferences featured executives asking, by voice, Amazon's Echo device and Alexa voice assistant platform for answers to business questions such as, "What is the percentage change in our market share in the last six months?" Behind the scenes and within seconds, the vendor's solution would receive the natural language query and return a result, which Echo and Alexa would speak. Cool demo!
Of course, this would likely not work for many types of BI and analytics, particularly where questions lead to more questions and understanding the context is important. However, it was a significant step in the ongoing movement toward greater ease of use.
Vendors took other steps to make their solutions smarter and faster, including enhancing internal capabilities for pre-building queries and using machine learning and other artificial intelligence (AI) techniques to speed data preparation. Some of the newer functionality using these techniques included automated guidance that provided recommendations for data sources and model development as well as the ability to launch algorithms in response to natural language queries. We will see more developments along these lines in 2018, to be seen, no doubt, in even cooler demos with Alexa, Echo, Apple Siri, and Microsoft Cortana.
2017 Trend #2: Growth in self-service BI and analytics drove demand for better metadata
A second trend I anticipated in 2017 was a strong focus on managing metadata more effectively. TDWI finds that organizations are concerned that as they democratize BI and analytics through user-driven deployment of self-service BI and visual analytics tools, they will increase data chaos. With the potential for more data silos and increasingly inconsistent and conflicting data definitions, expansion in self-service BI and visual analytics could make it more difficult for users to access and analyze data, not less.
Vendors responded to this concern. In 2017, many competed on which platform solution offered the strongest capabilities for managing metadata and automating the development of metadata repositories and business glossaries. To relieve business users -- not to mention IT -- of the often heavily manual tasks involved in creating these facilities, TDWI expects organizations will evaluate and deploy solutions with these capabilities. The trend toward AI-driven automation of metadata management, business glossary development, and master data management will continue strong in 2018.
Anticipated Top Trends of 2018
Here are four trends in BI and visual analytics to watch in 2018.
2018 Trend #1: Technologies enable delivery of incremental data insights for operational BI and analytics
In TDWI's 2017 Best Practices Report, Accelerating the Path to Value with Business Intelligence and Analytics, we found that most organizations are moderately pleased at best with the time it takes for BI and analytics projects to deliver intended value. To shorten this time, many organizations are using agile and DevOps methods as alternatives to traditional waterfall cycles. With these methods, users typically get to test applications and realize some incremental value sooner. Organizations are looking for technologies and methods that enable users to begin working with data and building visual analytics right away, both in a prototyping fashion to guide development of complete applications and to simply get value from the data sooner.
A key trend is toward more packaged and integrated data management and delivery platforms that ease data access. Many of these platforms run in the cloud, which is important for organizations that are storing volumes of data on cloud platforms as well as on top of data lakes implementing Hadoop clusters on premises. Some vendors are incorporating tools that enable users to interact with data as it is flowing through pipelines into data lakes and data warehouses so they do not have to wait until all the terabytes are loaded and for other refinement processes to finish. Data discovery becomes a learning process as the software uses machine learning to understand users' preferences and users can determine more quickly whether the data is useful for analysis.
In recent years, we've seen advancements in "fast" data interaction provided through vendors' implementations of Apache open source technologies and frameworks for SQL-on-Hadoop, in-memory computing, and in-database processing. Other recent advances include capabilities for continuously updating analytics and dashboards from data flowing into Hadoop clusters, not just traditional data warehouses. These technologies will enable users of BI and visual analytics applications to tap Hadoop data lakes and data hubs for daily, operational requirements rather than having to wait for batch-oriented cycles to finish. We expect continued technology developments in this area in 2018.
2018 Trend #2: Business applications get smarter through embedded visualization and analytics
Embedded BI and analytics functionality plays an important role in enabling users to progress faster with data-driven decision making. First, it is still rare for most users in an organization to have their own standalone BI and analytics tools; TDWI research finds that on average, less than one-quarter have their own tools. Second, users may be more comfortable with implementing BI and analytics functionality including dashboards and analytic visualizations from within their familiar business application or cloud-based SaaS solution. These solutions are typically designed to fit the context of the user's function, operation, or vertical industry.
Providers of business applications, including for ERP, CRM, and sales management systems are bringing forth analytics and AI-driven capabilities as embedded "platforms" for business applications, processes, and workflows. Independent providers of BI and analytics tools and applications are using open APIs, connectors, and other technologies to make it easier to snap together components and embed dashboards, predictive models, and more in applications. In 2018, we will see the wall between BI and analytics and business applications further erode.
2018 Trend #3: Visualization and analysis of streaming "data in motion" goes mainstream
BI and analytics have traditionally feasted on data "at rest"; that is, data that is only available for access, reporting, and analysis once it has landed in a database, data warehouse, data lake, spreadsheet, or other persistent storage system. Users interact with historical data at rest in these sources. Real-time, streaming analytics has been an exotic pastime, but that is changing.
Now, especially as the Internet of Things (IoT) heats up, the technology focus is turning toward solving problems of analyzing, managing, and protecting data in motion. Analytics is moving to the edge, to perform computations on data as sensors and websites are capturing it rather than waiting for it to come to rest in a data store.
Instead of just specialized applications in telecommunications or energy industries, there are now growing use cases across industries where real-time, streaming analytics is relevant. Organizations want to manage logistics or even mobile personnel. They want to analyze customer (or fraudster) behavior as it is happening so they can react in real time. "What the customer did in the last two minutes is more predictive of what they will do than what you can learn from historical data," an executive at one of the data platform vendors explained to me. "Latency between systems of engagement and the analytics platforms is too long."
Apache open source projects for streaming are maturing. To grapple with the complexity of IoT networks, data platform vendors are implementing technologies such as Nifi for orchestrating data flow between sensors and managing security. As the ability to manage streaming data platforms matures, we will see technologies that enable front-end BI and visual analytics applications to access the data for real-time reporting and analysis. Look for developments in this area in 2018.
2018 Trend #4: Governance gets hot
Can the staid, rules-and-regulations topic of governance ever really be hot? The answer is yes.
Organizations need solid governance as they democratize BI and analytics. Along with protecting sensitive data, governance rules and policies are expanding to address data lineage so decision makers can validate analytics insights. Many organizations are taking a broader view of governance to cover data stewardship, data curation, and quality improvement. Metadata and master data management are highly useful for governance, so organizations are seeking solutions that tighten integration between these initiatives and governance.
At TDWI Conferences and Leadership Summits, governance is always top of mind among attendees. Technology providers are addressing governance needs through application of AI techniques such as machine learning and other advanced analytics. Platform providers are beginning to compete on how well their solutions automate governance processes where possible and in general make it easier for organizations to scale governance and data stewardship as BI and analytics democratize and data management duties grow in size and complexity. In the next 12 months, we will see interesting developments for making governance smarter and more automated.
Dodging Silver Bullets
Exciting developments lie ahead in 2018 as technology platforms infuse AI and advanced analytics to provide better support for democratizing BI and visual analytics. However, as always, organizations need to make prudent decisions and not be tempted by silver bullets that offer a lot of hype but not enough benefit. Organizations need to test new technologies to make sure they are as "smart" as they claim to be. They should reassess the roles human IT managers and administrators need to play to train AI-infused technologies so the organization can be confident in using them. If the technologies fulfill their potential, organizations will be in a better position to expand and extend BI, visualization, and analytics for data-informed decisions and actions.