Executive Summary | Responsible Data and Analytics
The executive summary for the TDWI Best Practices Report: Responsible Data and Analytics
This TDWI Best Practices Report examines where organizations are today in terms of responsible data and analytics and what work they still need to do. It also addresses organizational imperatives and technologies to help organizations become more responsible.
- By Fern Halper, Ph.D., James G. Kobielus, David Stodder
- December 22, 2022
Enterprises are under increasing pressure to address social, environmental, and other responsible outcomes in their digital operations. To stay ahead of the curve, data management and analytics professionals have begun to address such topics as ethics, equity, fairness, safety, and sustainability in their strategic planning and operational practices.
For years, TDWI Research has covered such responsible data and analytics topics as data compliance, quality, privacy, security, protection, and bias mitigation. In order to manage data and analytics responsibly, enterprises often leverage platforms, tools, skills, and techniques that have proven useful in data compliance, data governance, and other established practices. At the same time, organizations are finding they need to evolve and extend these practices in order to serve a different set of stakeholders and use cases.
Responsible data and analytics considers the ethical, societal, compliance, legal, and environmental ramifications of using data in a wide variety of applications and processes. This TDWI Best Practices Report discusses the diverse range of use cases under the heading of responsible data and analytics. It defines the scope of this paradigm, discusses key business drivers, and baselines the current state of enterprise practices, platforms, and tools for addressing responsible data and analytics use cases.
This report uses findings from a survey of 214 data management and analytics professionals worldwide. It illustrates how enterprises are addressing the multifaceted requirements included under the paradigm of responsible data and analytics:
- Data quality, privacy, security, and compliance are the highest priorities. Data management and analytics professionals continue to place the highest priority on established requirements, such as data governance, trustworthiness, privacy, security, and regulatory compliance.
- Data transparency and explainability are important for enterprises to focus on. Transparency—the requirement that average humans be able to easily review and understand why predictive and other analytics models behave as they do—is a clear priority in a majority of organizations. Over half of those respondents using machine learning (ML) reported that their data and analytics organizations were comfortable explaining AI/ML algorithms.
- Data ethics and equity considerations are on the enterprise radar, but still take lower priority. Practitioners are starting to address newer imperatives such as data ethics, but rank these as less important drivers of their responsible data and analytics initiatives. Matters of data equity—such as whether analytics application designs support a range of disabilities and preferences—are not yet a high priority in enterprises. They are being addressed in the enterprises of around one-third of respondents, with a majority either not knowing or stating outright that these aren’t being addressed.
- Data sustainability is not yet a substantial practitioner focus. Environmental sustainability concerns, such as the carbon footprint from the nonrenewable energy used in training machine learning models, are still a low priority with enterprise data and analytics practitioners. Sustainability was cited as a priority by less than 25% of respondents. Nevertheless, companies have begun collecting data on the energy consumption, carbon footprint, and resource sustainability of their business and technical operations. Slightly less than half of respondents stated that their companies were either collecting this data or had plans to do so in the near future as part of their company-wide environmental, social, and governance initiatives.
To some degree, the enterprise journey to responsible data and analytics depends on building awareness, knowledge, and skills on these matters among both business and technical professionals. However, the survey showed that knowledge and training in responsible data and analytics knowledge is still weak, scattered, and spotty in many enterprises. Less than one in five respondents reported that data professionals in their organizations have training or knowledge about a full range of responsible data and analytics practices, including data privacy, data ethics, and data bias mitigation. Slightly under half reported their organizations’ data practitioners have some knowledge on a smattering of these topics. Less than one-third report that their data practitioners have knowledge on data privacy, and less than one in five organizations are providing training on data ethics.
This TDWI Best Practices Report examines these issues. It looks at where organizations are today in terms of responsible data and analytics and what work they still need to do. It also addresses organizational imperatives and technologies to help organizations become more responsible.
Denodo, SAP, and Snowflake sponsored the research and writing of this report.
About the Authors
Fern Halper, Ph.D., is vice president and senior director of TDWI Research for advanced analytics. She is well known in the analytics community, having been published hundreds of times on data mining and information technology over the past 20 years. Halper is also co-author of several Dummies books on cloud computing and big data. She focuses on advanced analytics, including predictive analytics, text and social media analysis, machine-learning, AI, cognitive computing and big data analytics approaches. She has been a partner at industry analyst firm Hurwitz & Associates and a lead data analyst for Bell Labs. Her Ph.D. is from Texas A&M University. You can reach her by email ([email protected]), on Twitter (twitter.com/fhalper), and on LinkedIn (linkedin.com/in/fbhalper).
James Kobielus is senior director of research for data management at TDWI. He is a veteran industry analyst, consultant, author, speaker, and blogger in analytics and data management. At TDWI he focuses on data management, artificial intelligence, and cloud computing. Previously, Kobielus held positions at Futurum Research, SiliconANGLEWikibon, Forrester Research, Current Analysis, and the Burton Group. He has also served as senior program director, product marketing for big data analytics for IBM, where he was both a subject matter expert and a strategist on thought leadership and content marketing programs targeted at the data science community.
David Stodder is senior director of TDWI Research for business intelligence. He focuses on providing research-based insights and best practices for organizations implementing BI, analytics, data discovery, data visualization, performance management, and related technologies and methods and has been a thought leader in the field for over two decades. Previously, he headed up his own independent firm and served as vice president and research director with Ventana Research. He was the founding chief editor of Intelligent Enterprise where he also served as editorial director for nine years. You can reach him by email ([email protected]), on Twitter (twitter.com/dbstodder), and on LinkedIn (linkedin.com/in/davidstodder).