Best Practices Report | Emerging Technologies For Business Intelligence, Analytics, and Data Warehousing
October 1, 2015
Part of the fun of being in business intelligence (BI), analytics, data warehousing (DW), and data management (DM) is the constant stream of new and exciting technologies, vendor tools, team structures, development methods, user best practices, and new sources of big data. TDWI refers to these collectively as emerging technologies and methods (ETMs).
For example, tools for data visualization are the most hotly adopted ETMs in BI in recent years. In addition to visualization, most of these tools also support other emerging techniques, including data exploration and discovery, data preparation, analytics, and storytelling. ETMs for analytics involve advanced techniques, such as predictive analytics, stream mining, graph, and text analytics, which are progressively applied to emerging data sources such as social media sites, machines, clouds, and the Internet of Things. A number of emerging data platforms have entered DW environments, namely Hadoop, MapReduce, columnar database management systems (DBMSs), and real-time platforms for event and stream data. The most influential emerging methods are based on agile development or collaborative team structures (e.g., competency centers).
According to this report’s survey, the leading general benefits of ETMs (in survey order) are improvements in competitiveness, decision making, responses to business change, business performance, and innovation. These benefits are being realized today because two-thirds of organizations surveyed are already using ETMs and the vast majority consider ETMs an opportunity.
Despite the benefits, a number of barriers stand in the way of adopting ETMs. Many people feel held back by their IT team’s lack of skills, staffing, infrastructure, and buy-in. Others have trouble seeing the business value of leading-edge technologies. Some work in risk-averse organizations that lack a culture of innovation for either IT or the business. Yet both business and technical respondents report working through these issues to adopt ETMs.
Some ETMs are more like tool features that are emerging in a variety of tool types. The most pervasive is self-service functionality, which is found in tools for reporting, analytics, data prep, analytics, and so on. The point is to give certain classes of users tools so easy, intuitive, and integrated with common data sources that they can use them with little or no setup or assistance from IT. More than half of users surveyed consider themselves successful with IT-free self-service. Other BI ETMs are also progressive, namely mobile BI and agile BI.
Hadoop (whether from Apache or a software vendor), tools and frameworks associated with it (MapReduce, Spark, Hive, HBase), and similar data platforms (NoSQL databases) have emerged from their Internet-company roots and are now being adopted by mainstream enterprises. These ETMs are examples of how influential open source software (OSS) has become for innovative products. Interfaces to these platforms’ data are also common emerging features in vendor-supplied tools for data integration, data prep, data exploration, reporting, and analytics. Hadoop is infamous for weak security, which a new class of ETMs is addressing.
All these OSS-based or OSS-inspired ETMs are now entering data warehouse environments, along with slightly older ETMs such as DW appliances, analytic DBMSs, and columnar DBMSs. This emergence has driven a trend toward multi-platform data warehouse environments, where the core relational warehouse is joined by a long list of standalone data platforms, most of them ETMs.
This report informs the reader about the ETMs that apply to BI, analytics, and DW, plus how ETMs can support innovation in business processes, customer management, competitiveness, IT best practices, and the business leverage of data.