Data Scientists Reveal Roadblocks to Digital Transformation
SAS research also identifies strategies to capitalize on this pivotal moment and empower data scientists and organizations.
Note: TDWI’s editors carefully choose press releases related to the data and analytics industry. We have edited and/or condensed this release to highlight key information but make no claims as to its accuracy.
Digital transformation has accelerated significantly due to the COVID-19 pandemic, but the extra demands on data scientists have revealed significant barriers to effective working and high levels of job dissatisfaction in some areas. For example, around four in 10 are dissatisfied with their company’s model deployment and use of analytics. Research also revealed more than 20 barriers to effective working, according to a survey of data scientists commissioned by SAS.
However, the work of data scientists has grown in importance, with many organizations accelerating digital transformation projects by using technology to improve business operations. More than 90 percent of respondents indicated the importance of their work was the same or greater compared to before the pandemic.
To delve deeper into the state of data science, the report assesses the impact of the pandemic, challenges faced, overall satisfaction with the analytics environment, and more. The research showed the pandemic upended standard business practices, shifting the assumptions and variables in models and predictive algorithms, and causing a ripple effect of adaptations in processes, practices, and operating parameters.
More than two-thirds of respondents were satisfied with the outcomes from analytics projects. However, 42 percent of data scientists were dissatisfied with their company’s model deployment and use of analytics, suggesting a problem with how analytics insights are used by organizations to inform decision making. This was backed up by the 42 percent who said data science results were not used by business decision makers, making it one of the main barriers faced.
The survey also highlighted some specific skills gaps. Less than a third of respondents reported having advanced or expert proficiency in program-heavy skills such as cloud management and database administration. This is an issue given that use of cloud services is up significantly, with 94 percent saying they experienced the same or greater use of the cloud since COVID-19 struck.
The research also identified gaps in consistent organizational emphasis on AI ethics, with 43 percent of respondents indicating that their organization does not conduct specific reviews of its analytics processes with respect to bias and discrimination, and only 26 percent of respondents reporting that unfair bias is used as a measure of model success in their organization.
When it comes to the challenges identified to ensure fair and unbiased decision making, Dr. Sally Eaves, an industry expert, said, “Rather than sometimes hoping they are given appropriate, clean data and relying too much on the technology to drive fair outcomes, data scientists can play an active role in putting the right guidelines and checks in place at each stage of the analytical process to try and eliminate bias. Having a transparent and explainable flow from data to decision is obviously key to this.”
The research revealed positive outcomes from the global disruption of the pandemic. Nearly three-quarters (73 percent) said they are just as productive or more productive since the pandemic, while a similar proportion (77 percent) revealed they had the same or greater collaboration with colleagues. This suggests many of the challenges highlighted were in existence, possibly to a greater degree, before the pandemic.
Other challenges experienced were the amount of time spent on data preparation versus model creation. Respondents are spending more of their time (58 percent) than they would prefer gathering, exploring, managing, and cleaning data.
Additional findings and discussion points, including strategies for data scientists to improve processes and outcomes, are available in the full report.