TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Advanced Analytics: A Look Back at 2020 and What’s Ahead for 2021

Organizations will continue to digitally transform to both survive and thrive in the new normal.

• By Fern Halper
• December 23, 2020

Introduction

In 2020, both COVID-19 and its economic impact turned the world upside down. Many organizations had to pivot -- and pivot fast -- to survive. In some organizations, data and analytics became even more essential to help gain insights and plot a new strategy.

For Further Reading: Don’t Forget the Back End of the Machine Learning Process Why Data Literacy is Critical to Higher Satisfaction and Productivity The Machine Learning Data Dilemma

TDWI research at the start of the COVID crisis indicated that many data and analytics professionals were being asked to collect new kinds of data, do more analyses, and generally provide more insight. Digital transformation strategies accelerated, too, especially when it came to moving to the cloud.

Current data suggests that in 2021, the same patterns may hold. Some portion of organizations will hire new people to accelerate digital transformation while others will move along with existing staff to address new efforts. Organizations will build new products and develop new ways of using data.

Aside from COVID-19 (or as part and parcel of it), self-service and advanced analytics (such as machine learning) continued to gain steam in 2020. The majority of respondents we surveyed this year stated that demand for machine learning was increasing. The talent needed for machine learning is being addressed by organizations hiring data scientists as well as planning to upskill their business analysts to become data scientists.

Vendors continue to make their products easier to use by offering tools such as autoML (automated machine learning) to help improve data scientist productivity (by providing features including automated feature extraction and hyperparameterization) and make it easier for business analysts to build models. The jury is still out as to whether upskilling will be successful. Early data suggests it can be. We will know more in 2021.

Another area that vendors hyped during 2020 was unified platforms, especially in the cloud. The idea here is to support analytics, especially more advanced analytics that is iterative and compute-intensive (such as AI technologies) with trusted data at scale. Many organizations are looking to the cloud as part of their unified platform strategy because of its scalability and flexibility.

2021 Anticipated Hot Spots

These 2020 trends will continue to be hot in 2021. However, as organizations make their way out of the pandemic, they will also have their eyes on the following three trends.

Hot Spot #1: Machine learning and self-service continue to be priorities

According to TDWI research, demand for machine learning and self-service continues to grow. Companies want to become more advanced in analytics to better compete. On the practical side, organizations will need to determine how much these tools can help them and what will be involved in using them. For self-service tooling, data literacy will be key and this is an area where organizations need to progress. TDWI research indicates that many organizations are still not data literate and this impacts the adoption of self-service analytics. Data literacy will be a priority in 2021.

In terms of machine learning, there is growing interest in automated tools, as I mentioned. We’ve recommended that users should have the skills to verify the insights produced by these tools; it is important for organizations to choose tools that are transparent so users know what is happening behind the scenes because they will have to understand and defend their analysis regardless of how a model was derived. Explainability features will continue to be important as part of vendor solutions in 2021. Work in helping to identify data bias will also get underway in earnest.

Hot Spot #2: Unified platform

Getting data rapidly into the hands of business users and analytics applications has never been more important. Yet, as organizations look to generate faster and better insights, they are faced with a dizzying array of data sources and slow, complicated steps to prepare and transform the data for analytics, operational dashboards, and other types of data consumption. Traditional data warehousing processes such as extract, transform, and load (ETL) often end up being too slow and complicated to keep up with data demands.

An emerging alternative is a unified data and analytics platform -- a tightly integrated suite of tools that provide data management, analytics, and other services. More often these platforms are found in the cloud. More often, too, they utilize a microservices architecture. As organizations make the move to the cloud in 2021, expect to hear more about cloud data and analytics platforms.

Hot Spot #3: Containers

As organizations begin to build more machine learning models and want to put them into production, they are often looking to containers to help them. Containerization involves packaging up software code and all its dependencies in a container so the software can run on practically any infrastructure.

Containers are definitely becoming popular. In a recent TDWI survey, for example, approximately one-fourth of respondents were already using containers (according to unpublished TDWI research). Machine learning models -- including the code, dependencies, tools, libraries, and configuration files -- can all be packaged up into a container. Data scientists can then use containers to share their work and put it into production. This also means that others can re-run/reproduce the work. In the cloud, the container can be replicated to run across a cluster. Expect 2021 to be the year of the container for machine learning deployment.

Along with this, as organizations scale the number of models in production, they will need new roles including MLOps or ModelOps. This is the highly technical team responsible for model management and monitoring. They are responsible for validating, registering, deploying, monitoring, and retraining models in production. Although many data scientists are responsible for this today, expect organizations to begin to look more seriously into hiring for MLOps in 2021 as they scale the number of models in production.

The Bottom Line

In 2021, companies will emerge from the COVID-19 crisis. Some companies that have survived will become stronger and more nimble, in part due to the role of data and analytics in their organization. Look for 2021 to be a shake-out year for organizations as they adapt to the new normal and modernize accordingly.