Comet’s New Survey Highlights AI’s Latest Challenges: Too Much Friction, Too Little ML
Majority of machine learning professionals admit to abandoning 40-60 percent of their experiments.
Note: TDWI’s editors carefully choose press releases related to the data and analytics industry. We have edited and/or condensed this release to highlight key information but make no claims as to its accuracy.
Comet, provider of a development platform for enterprise machine learning (ML) teams, has released the results of its recent survey of machine learning professionals. Hundreds of enterprise ML practitioners were asked about their experiences and the factors that affected their teams’ ability to deliver the level of business value their organizations expected from ML initiatives. Rather than attaining desired outcomes, however, many survey respondents revealed that they lack the right resources or that the resources they have are often misaligned. As a result, many AI initiatives have been far less productive than they could be.
Issues include the fact that many experiments are abandoned because some part of the data science life cycle was mismanaged. This is due in large part to the manual tracking processes that organizations often put in place, which hinder effective team collaboration and are not adaptable, scalable, or reliable. Of models actually deployed into production, nearly one quarter failed in the real world for more than half (56. 5 percent) of the companies surveyed. As such, the full business value of machine learning is rarely captured.
By the Numbers
According to survey respondents:
- Significant time, resources, and budgets are being wasted. Although teams expect to run, adjust, and rerun experiments as part of model development, 68 percent of respondents admit to scrapping a whopping 40 to 80 percent of their experiments altogether. This is due to breakdowns that occur throughout the machine learning life cycle <em>outside</em> of the normal iterative process of experimentation.
- There is a serious lag in model deployment. Only 6 percent of teams surveyed have been able to take a model live in under 30 days. By contrast, 47 percent of ML teams require four to six months to deploy a single ML project, and another 43 percent take up to three months. This can cause delays in delivering value to the respective lines of business.
- Budgets for tools that could address issues are woefully inadequate. Despite the enthusiasm for ML overall, 88 percent of respondents have an annual budget of less than $75,000 for machine learning tools and infrastructure. This is far less than the average salary for a single data scientist and dwarfed by the opportunity cost resulting from under-investment.
- Without funds for automation, ML teams must track experiments manually. In fact, 58 percent of machine learning teams track at least some piece of their experiments manually. This places an enormous strain on workers, causes projects to take far longer to complete, creates challenges for team collaboration and model lineage tracking, hinders model auditability, and leads to mistakes.
- Companies are not intentionally withholding budgets or misallocating ML resources. They simply “don’t know what they don’t know” because ML is still an emerging discipline. Of survey respondents, 63 percent said their organizations would increase ML budgets for 2022, but whether these funds will be devoted to the right tools and resources remains to be seen.
State of Enterprise ML Today
ML has demonstrated that it can deliver outsized business value and exciting technological innovation. As such, more companies are seeking to apply it, only to find the tools and processes to be nascent, disconnected, and complex. This makes it difficult to collaborate among teams and uncover the insights that drive business forward. Even though many organizations are good at identifying ML use cases and initiating projects, they fall short in investing in ML operations best practices and the tools and resources needed to make these machine learning initiatives as clear, efficient, and scalable as possible.
Developing effective ML models requires a lot of experiments. These experiments can involve changing the model itself or tweaking its hyperparameters. They may utilize different data sets or involve changing code to evaluate how the algorithms behave differently. When developing and training an ML model, all these changes happen repeatedly, sometimes with only minute differences each time. This makes it difficult to keep track of which experiments and which parameters produced which results -- including details such as scripts, the runtime environment, configuration files, data versions, hyperparameters, metrics, weights, and more. Poor experiment management leads to the inability to reproduce results accurately and consistently, and it can throw an entire project off the rails, wasting countless hours of a team’s work.
“Even though companies are prepared to allocate more money and resources to ML programs, they must address some core operational issues first if they want to see a positive return on their investment,” said Gideon Mendels, CEO and co-founder of Comet.. “If teams are maxed out and struggling with visibility, reproducibility and cost-efficiency today, it will be difficult for them to add more models, experiments, and deployments this year, as they expressed the desire to do. Successful ML outcomes depend on people; and with the right tools, teams can avoid burnout."
View the complete report: 2021 ML Practitioner Survey