Executive Summary: Emerging Technologies For Business Intelligence, Analytics, and Data Warehousing
- By David Stodder, Fern Halper, Ph.D., Philip Russom, Ph.D.
- September 28, 2015
Part of the fun of being in business intelligence (BI), analytics, data warehousing (DW), and data management (DM) is the constant stream of new and exciting technologies, vendor tools, team structures, development methods, user best practices, and new sources of big data. TDWI refers to these collectively as emerging technologies and methods (ETMs).
For example, tools for data visualization are the most hotly adopted ETMs in BI in recent years. In addition to visualization, most of these tools also support other emerging techniques, including data exploration and discovery, data preparation, analytics, and storytelling. ETMs for analytics involve advanced techniques, such as predictive analytics, stream mining, graph, and text analytics, which are progressively applied to emerging data sources such as social media sites, machines, clouds, and the Internet of Things. A number of emerging data platforms have entered DW environments, namely Hadoop, MapReduce, columnar database management systems (DBMSs), and real-time platforms for event and stream data. The most influential emerging methods are based on agile development or collaborative team structures (e.g., competency centers).
According to this report’s survey, the leading general benefits of ETMs (in survey order) are improvements in competitiveness, decision making, responses to business change, business performance, and innovation. These benefits are being realized today because two-thirds of organizations surveyed are already using ETMs and the vast majority consider ETMs an opportunity.
Despite the benefits, a number of barriers stand in the way of adopting ETMs. Many people feel held back by their IT team’s lack of skills, staffing, infrastructure, and buy-in. Others have trouble seeing the business value of leading-edge technologies. Some work in risk-averse organizations that lack a culture of innovation for either IT or the business. Yet both business and technical respondents report working through these issues to adopt ETMs.
Some ETMs are more like tool features that are emerging in a variety of tool types. The most pervasive is self-service functionality, which is found in tools for reporting, analytics, data prep, analytics, and so on. The point is to give certain classes of users tools so easy, intuitive, and integrated with common data sources that they can use them with little or no setup or assistance from IT. More than half of users surveyed consider themselves successful with IT-free self-service. Other BI ETMs are also progressive, namely mobile BI and agile BI.
Hadoop (whether from Apache or a software vendor), tools and frameworks associated with it (MapReduce, Spark, Hive, HBase), and similar data platforms (NoSQL databases) have emerged from their Internet-company roots and are now being adopted by mainstream enterprises. These ETMs are examples of how influential open source software (OSS) has become for innovative products. Interfaces to these platforms’ data are also common emerging features in vendor-supplied tools for data integration, data prep, data exploration, reporting, and analytics. Hadoop is infamous for weak security, which a new class of ETMs is addressing.
All these OSS-based or OSS-inspired ETMs are now entering data warehouse environments, along with slightly older ETMs such as DW appliances, analytic DBMSs, and columnar DBMSs. This emergence has driven a trend toward multi-platform data warehouse environments, where the core relational warehouse is joined by a long list of standalone data platforms, most of them ETMs.
This report informs the reader about the ETMs that apply to BI, analytics, and DW, plus how ETMs can support innovation in business processes, customer management, competitiveness, IT best practices, and the business leverage of data.
HP Security Voltage, HP Vertica, MicroStrategy, Qlik, Snowflake Computing, Striim, and Trifacta sponsored the research and writing of this report.
About the Authors
David Stodder is senior director of TDWI Research for business intelligence. He focuses on providing research-based insights and best practices for organizations implementing BI, analytics, data discovery, data visualization, performance management, and related technologies and methods and has been a thought leader in the field for over two decades. Previously, he headed up his own independent firm and served as vice president and research director with Ventana Research. He was the founding chief editor of Intelligent Enterprise where he also served as editorial director for nine years. You can reach him by email ([email protected]), on Twitter (twitter.com/dbstodder), and on LinkedIn (linkedin.com/in/davidstodder).
Fern Halper, Ph.D., is vice president and senior director of TDWI Research for advanced analytics. She is well known in the analytics community, having been published hundreds of times on data mining and information technology over the past 20 years. Halper is also co-author of several Dummies books on cloud computing and big data. She focuses on advanced analytics, including predictive analytics, text and social media analysis, machine-learning, AI, cognitive computing, and big data analytics approaches. She has been a partner at industry analyst firm Hurwitz & Associates and a lead data analyst for Bell Labs. Her Ph.D. is from Texas A&M University. You can reach her by email ([email protected]), on Twitter (twitter.com/fhalper), and on LinkedIn (linkedin.com/in/fbhalper).
Philip Russom, Ph.D., is senior director of TDWI Research for data management and is a well-known figure in data warehousing, integration, and quality, having published over 600 research reports, magazine articles, opinion columns, and speeches over a 20-year period. Before joining TDWI in 2005, Russom was an industry analyst covering data management at Forrester Research and Giga Information Group. He also ran his own business as an independent industry analyst and consultant, was a contributing editor with leading IT magazines, and a product manager at database vendors. His Ph.D. is from Yale. You can reach him by email ([email protected]), on Twitter (twitter.com/prussom), and on LinkedIn (linkedin.com/in/philiprussom).