TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Advanced Analytics: A Look Back at 2018 and What’s Ahead for 2019

TDWI analyst Fern Halper explains why 2019 will be the start of a digital transformation journey for many organizations.

• By Fern Halper
• December 19, 2018

In 2018, advanced analytics maintained its momentum in the enterprise. AI was the big buzzword and machine learning was top of mind for many organizations. Other technologies including open source continued to pique interest. Use of the cloud became more mainstream, along with a move to a serverless model to support analytics. TDWI expects these technologies to continue to grow in importance. We also anticipate other advanced analytics hotspots in 2019.

For Further Reading: Hybrid Cloud Analytics: Plan for These Challenges Reimagining the Analytics CoE 3 Signs of a Good AI Model

2018 Trends

AI/Machine Learning

Machine learning -- the science of getting computers to act without being explicitly programmed -- was hot this year (and will continue to be in 2019). Many vendors use the term AI when they are referring to machine learning, which was kicked into high gear in 2018. One area that was especially hot was the infusion of machine learning into commercial analytics products to create “smart tooling” (also known as machine intelligence, augmented intelligence, and automated intelligence) across the analytics life cycle. Machine learning was infused in products to help identify data quality issues in data preparation, to populate metadata in data catalogs, to suggest insights in visual analytics products, and to build models automatically.

Although this trend is still nascent, it will continue to be important as part of a company’s digital transformation, which itself will be a hot topic in 2019 (see below).

Open Source

Open source continued its momentum in 2018, especially for big data and data science. During 2018, Python, an interpreted, interactive, easy-to-read, object-oriented scripting language continued to gain steam with new users. Projects such as TensorFlow for deep learning also grew in popularity this year. For example, in a 2018 TDWI survey, over 20% of respondents were using TensorFlow! Chainer, Caffe, and others also built momentum.

Commercial software vendors continued to embrace open source by making their platforms more open. For instance, many vendors extended their platforms to enable the use of models developed in packages such as R and Python to help deploy these models into production.

The Cloud as Part of a Data Strategy

The cloud has been hyped as the go-to analytics platform for years. Yet, at TDWI we saw resistance to it from a number of quarters, mostly based on security concerns. Resistance seemed to slow down in 2017 and by 2018 we saw large numbers of organizations using the cloud as an important part of their data infrastructure; cloud data warehouses continued to gain in popularity, for instance. In 2018, we saw the emergence of the “serverless computing” model as part of the reimaging of the data warehouse. In this cloud-computing model, the cloud provider dynamically manages the allocation of machine resources, which frees up the organization to focus on more important matters, such as data analysis.

2019’s Anticipated Hot Spots

The trends we saw in 2018 above will continue to be hot trends in 2019. However, there are other emerging trends also worth noting.

The Growth of AI as Part of a Digital Transformation Wave

Vendors spent a lot of 2018 hyping AI (whether they had the technology or not). In 2019, we expect that to continue, often in the context of “digital transformation.” AI/ML/NLP are core technologies in digital transformation, which changes how businesses operate. For instance, the Internet/ecommerce boom of the 1990s and 2000s is an example of a digital transformation wave. This wave is about transforming organizations, processes, and products with data and analytics. The majority of organizations we survey state that digital transformation is an imperative for their company if it is to succeed. AI/ML/NLP is at the core of this transformation.

Here’s where we’ll see growth in 2019:

• AI/ML/NLP for use in analysis. ML will continue to be hot into 2019; in fact, the majority of respondents we survey state that demand for machine learning is increasing. We also expect NLP projects to gain more steam as organizations begin to analyze more unstructured data or want new ways to interact with customers (e.g., chatbots).


• AI/ML/NLP for use in analytics products. As mentioned above, the use of augmented intelligence across the analytics life cycle was fairly nascent in 2018. We expect to see more users deploy tools that make it easier for them to gain insights. Vendors will continue to promote products that are infused with “AI.” One area of growth will be products that have natural language interfaces so users can interact with an analytics system in a more interactive way. Another growth area will be discovery tools that suggest insights.


• Intelligent product development. In addition to using machine intelligence in analytics products and for insight, more organizations will look to embed machine learning and other technologies into consumer and business apps in 2019, often driven by open source (see below).

Open Source for Analytics

Open source will continue its momentum in the organization for analytics and especially for advanced analytics such as predictive analytics and machine learning. As mentioned, projects such as Tensorflow for deep learning are already gaining many users. Open source is especially popular with developers who are building analytics apps.

Additionally, we expect to see more open source projects in less glamorous areas of the analytics life cycle such as model deployment and management, especially as organizations look to put models into production. Some projects (such as Anaconda) are beginning to provide some of this functionality. However, this is an area that needs more attention from the open source community.

Analytics Governance

We expect that 2019 will be a year of reckoning for all companies looking to deploy more advanced analytics (such as machine learning models) into production whether they are using open source or commercial products. It is one thing to manage a few models, but once organizations start to build dozens of models (or more), they will need to employ version control and better keep track of them.

Metadata about models will also be important -- when they were created, who created them, what they are being used for, and so on. Additionally, once a model is put into production, it will need to be monitored to make sure it doesn’t get stale. As organizations mature in their analytics journey, they will need to start planning for some sort of model governance process.

Another aspect of model governance is compliance with regulations. The GDPR got considerable attention in 2018; however, more regulations are coming down the pike such as the CCPA (California Consumer Product Act) which will impact machine learning and AI. There are a number of aspects to this including data privacy, data erasure, and explainability. We expect the impact of privacy regulations on more advanced analytics such as machine learning and AI to be a hot topic in 2019.

The Bottom Line

Next year will build on the advances made in 2018 as analytics is infused into products to make them easier to use. At the same time, open source will need to harden itself to support advanced analytics. Additionally, models will need to be better governed and the impact of regulations better understood. TDWI will be following these trends as well as a number of others.