TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Advanced Analytics: A Look Back at 2019 and What’s Ahead for 2020

Practical concerns will (and should) dominate 2020.

• By Fern Halper
• December 20, 2019

The Year in Review

In 2019, advanced analytics continued to build momentum in the enterprise. AI was still the buzzword du jour and machine learning (ML) dominated the analytics landscape. The majority of respondents we surveyed in 2019 stated that demand for machine learning was increasing. In a recent TDWI Best Practices survey, the number one use case for AI technologies was building predictive models using tools such as machine learning. This beat out other use cases by almost 2:1.

Much of the momentum around machine learning was fueled by open source, which continued to grow in 2019, especially for big data and data science use cases. In a 2019 TDWI survey, for instance, 91 percent of respondents stated that open source was critical to their advanced analytics efforts. During 2019, Python continued to gain steam with new users, in some cases outpacing the adoption of R. Organizations made greater use of deep learning, using projects like TensorFlow. Commercial software vendors continued to embrace open source by making their platforms more open. Other vendors extended their platforms to enable the use of models developed in packages such as R and Python to help deploy these models in production.

One area that vendors hyped was the infusion of machine learning into commercial analytics products to create “smart tooling” (other names include machine intelligence, augmented intelligence, automated intelligence) across the analytics life cycle. Here, machine learning is infused in products to help identify data quality issues in data preparation; to populate metadata in data catalogs; to suggest insights in visual analytics products; and to build models automatically. Although this trend was nascent in 2019 (with less than 20 percent of respondents to our surveys using these tools), it will continue to be important as part of a company’s digital transformation strategy, and will be a hot spot in 2020 (see below).

Finally, in 2019, we saw many organizations continue to embrace the cloud and especially cloud data warehouses to support analytics. About a third of the companies we surveyed were using some sort of cloud data warehouse; some were using data warehouses architected for the cloud (e.g., cloud-native). More were planning to move to the cloud.

2020 Anticipated Hot Spots

These trends will continue to be hot in 2020. However, as the dust settles around machine learning, enterprises will face practical considerations they’ll need to act on. Enterprises will need to tackle several important questions.

1. Is improving the skills of business analysts the right strategy?

As machine learning moves into the mainstream, organizations will continue to assemble data science teams to support model building. TDWI research indicates that a majority of organizations building models feel they need more data scientists. However, given that data scientists are an expensive resource and often in short supply, many organizations are planning to “upskill” their business analysts to build (at least some part of) the models. In fact, this is often cited as a top strategy for becoming more advanced with analytics. In 2020, enterprises will begin to reconcile whether improving skills can work for them and what will be needed to succeed with this strategy.

2. Is augmented intelligence friend or foe?

Many analytics vendors are offering tools that help business analysts and even business users construct machine learning models. In some of the tools, all the user needs to do is specify the outcome or target variable of interest, along with the attributes believed to be predictive. The software picks the best model. Other tools are even more automated. Although the use of augmented intelligence tooling is still relatively new, a majority of respondents to our surveys state they will be using these tools in 2020 and 2021. Many view these tools as a way to put more complex analytics (such as ML) in the hands of business analysts. In other words, these tools are being looked at, in part, to help to close the talent gap.

On the practical side, organizations will need to determine how much these tools can help them and what will be involved in using them. We’ve recommended that users should have the skills to verify the insights produced by these tools; it is important for organizations to choose tools that are transparent so users know what is happening behind the scenes. Additionally, organizations need a process to validate the models before they are put into production. The next few years will tell whether these tools will be useful for organizations in building models.

3. Do we need DataOps?

Organizations using ML will realize that it isn’t just about building models. Much of the real work comes in putting those models into production and making sure that they stay current. This need will necessitate new roles including DataOps as part of the machine learning process. DataOps (in some organizations called ModelOps, MLOps, or DevOps) is the team responsible for model management: validating, registering, deployment, monitoring, and retraining models in production. This role will continue to become increasingly important as organizations scale their ML efforts.

In 2019, the majority of organizations we spoke to and surveyed had only a few models in production. As organizations begin to scale their efforts, they will realize the criticality of DataOps. DataOps teams are skilled in Java, R, or Python; they understand how to deploy models using APIs or even packaging the models into containers. Next year will be one of reckoning for many organizations that were focused only on the front end of the model-building process.

The Bottom Line

In 2020, we’ll see even more technology advances made, but organizations will also address the practical aspects associated with utilizing advanced analytics and ML. They will determine whether they can improve the skills of their business analysts; evaluate whether they will be able to use some of the new tooling on the market; and realize that they need a DataOps team if they want to succeed in advanced analytics at scale.