TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Advice for Automating Machine Learning and Predictive Analytics

AutoML can solve many of the problems businesses face, but not all. Democratization of machine learning can make a difference. Adam Carrigan, co-founder and COO of MindsDB, explains.

• By James E. Powell
• April 5, 2021

Machine learning and predictive analytics present a massive opportunity for businesses looking to improve existing processes, become more data-driven, and enhance the customer experience. Many companies have instituted some ML projects and are ready to take the next step with more widespread adoption.

For Further Reading: The Machine Learning Data Dilemma Data Requirements for Machine Learning The Rise of Automated Machine Learning

Unfortunately, this is one area where companies run into challenges. The expertise and investment required to make ML and predictive analytics successful aren't readily available. The demand for data scientists continues to snowball. According to LinkedIn, the data science field saw 37 percent job growth last year.

How can companies deal with this shortage of data science resources to take full advantage of AI and predictive analytics? We talk with Adam Carrigan, co-founder and COO of MindsDB, to learn about adding automation to machine learning.

Upside: What are the current challenges with adopting machine learning?

Adam Carrigan: Now that the use of machine learning and predictive analytics is more widespread and investment in this technology is increasing, companies are finding it difficult to scale the benefits to the entire organization. The department that oversees machine learning (often the data science team) has, in many companies, now become a bottleneck within the process to implement these solutions.

Data science teams are usually small, and the process of researching, testing, and deploying machine learning can take anywhere from days to months depending on the complexity of the data. This results in a long list of backlogged projects, where specific projects take priority and others are pushed back or never implemented. This challenge is even more significant for SMBs, which typically don't have the resources to invest in highly in-demand data scientists.

AutoML now attempts to automate specific tasks in the ML workflow to reduce the strain on data scientists, but this comes with limitations. Many of the solutions available today still require a data scientist to play a significant role in the process. Although many aim to automate the training and testing, this leaves much of the process untouched, including the crucial deployment stage.

How should companies approach automating machine learning?

There are two approaches to automating machine learning inside an organization. The first is automating more of the process for the data scientist, freeing up capacity for them to solve more problems faster. This increases capacity and at least temporarily solves the bottlenecks I talked about earlier. This approach doesn't help those who don't yet have a data science team and is primarily a Band-Aid solution.

The second approach, and the one I believe will have a more meaningful impact, is to equip existing employees with the capabilities to solve most machine-learning problems. This eliminates the need to go back and forth between teams because ultimately the team trying to solve the problem is the one that knows it best. This is great for companies without a data science team to begin with. This approach also has an added benefit for those companies already with a data science team. It frees up their time and resources for the more complex and challenging problems. Ultimately, AutoML can solve many of the problems businesses face, but not all. In many instances, data scientists are still essential.

This approach requires a fundamental shift in the way we approach machine learning. Instead of thinking about it as an element within the application layer, it becomes useful to think about it as a data-layer problem. This opens up some exciting opportunities.

When you consider ML as simply other representations of your data, it makes it easier to run models and predictions at the data layer. AI tables -- automated ML models as native data tables inside databases -- let users execute models by merely running a data query. Instead of using AutoML to streamline the data scientist's tasks, this approach puts ML into the hands of the end-user of the data.

How can companies educate the end users of predictive analytics?

This is a crucial component to the democratization of machine learning but shouldn't necessarily be treated any differently than any other method of improving the skills of the enterprise's employees. These users still need basic education to use those tools. Essential skills include running queries, analyzing data, and understanding how to use ML data to support the human decision-making process.

Now that machine learning is simple enough for the average database user to leverage, these users must now understand the potential strengths and weaknesses in the models they produce. Teaching end users how to determine a level of trust and confidence in the models using explainable AI (XAI) is even more important than the operational education. Thankfully, more AutoML solutions are including explainability as a standard feature. Armed with this information and the generated predictions, users become even more powerful in making impactful changes within the organization.

About the Speaker

Adam Carrigan, co-founder and COO of MindsDB, is an entrepreneur and former management consultant at Deloitte. After completing his studies at the Australian National University and the University of Queensland, Adam completed his dissertation at the University of Cambridge, specializing in the use of NLP to predict equity pricing. Adam has significant operational experience in various industries and extensive knowledge in finance, marketing, and strategy, Adam co-led his newest venture MindsDB through the YCombinator accelerator.