What the Modern Data Team Looks Like and Where It's Headed
To tackle the challenges and advanced use cases ahead, companies need to rethink the structure of their data teams.
- By Ed Thompson
- September 1, 2020
The cloud has become the cornerstone of data innovation as enterprises move their data and analytics effort to cloud platforms to expedite the time to insights for business decisions. Most companies are opting for a cloud-first approach to cloud data warehouses for their flexible and scalable architecture. Companies are also pursuing hybrid and multicloud strategies. Growing data volumes -- and increasing data complexity -- make scaling a data management strategy nearly impossible unless teams can implement a hybrid or cloud-first approach and enable self-service data inside the organization.
These trends are driving changes in today's data teams. The highly technical coding skills and other hands-on tasks that were in high demand even a few years ago just to keep workflows going are giving way to low-code and no-code tools. Data engineers will always be a critical part of any modern data team, but the kind of hand-coding that was once routine is nearly impossible with today's data volumes.
Who's Who on a Modern Data Team
To tackle the challenges ahead, companies need to rethink the structure of their data teams.
What roles and responsibilities make up the modern data team? The job titles will vary depending on the business and industry, but each team member's responsibilities fall into one of the following five categories.
1. A technical role -- a data engineer or an ETL developer -- builds workflows. This person will be responsible for making sure data pipelines and ETL jobs are running and colleagues have the data access they need for their projects. This team member oversees data migration, orchestration, and transformation and helps develop applications and systems that drive advanced analytics use cases, including AI and machine learning.
2. Data scientists are necessary to derive value and insight from data. Although data scientists are normally situated inside IT, nowadays it's normal to find them in other parts of the business. Data scientists find innovative ways to work with data and help teams achieve a rapid ROI on analytics efforts using methods including data curation or advanced search, matching, and recommendation algorithms. Data scientists need access to the highest quality of data and large amounts of computing resources to extract deeper insights. Using the data scientists' time for sourcing, preparing, and checking data in the warehouse is wasteful.
3. A data analyst queries and reports on data in the data lake or cloud data warehouse. Analysts can use these findings to build interactive charts and dashboards for business user reporting, diagnostics, and decision making.
4. Citizen data professionals (CDPs) are increasingly important members of the team as demand for data scientists is often unmet due to a skills shortage. A CDP is a data-savvy knowledge worker who sits outside the IT department and wants to tackle data problems using intuitive, simple tools to load and centralize data sources for analytics and innovation. CDPs do not replace but rather complement technical team members in solving business challenges because they know their data sources better than anyone and can work with a technical counterpart to troubleshoot and enhance the use of data and insights.
5. An executive team member -- be it a CTO, CIO, or CDO -- keeps the data team on track and shapes the companywide data management strategy. This person will work across disciplines to ensure that every department's initiatives are aligned and business logic, governance, and security are properly managed.
New Directions
Collectively, the data team provides the essential capabilities an enterprise needs: SQL understanding to manage cloud data warehouses, the ability to deploy and secure cloud infrastructure, familiarity with data orchestration and pipelines, and a strong understanding of applying business logic through transformations. These tasks remain the foundation of data management.
Once those needs are met, enterprises can look ahead to new, advanced use cases. Here are several of the exciting avenues data teams can explore to improve established processes and start new initiatives.
- Enable data democratization. Business professionals need to become data literate and data-driven, and they'll be expected to acquire and demonstrate basic competency with technical tools. With that knowledge, they can accomplish tasks they would previously have asked IT to perform for them, such as preparing a data set for analytics.
- Improve data governance. Building a modern data team is a great opportunity to structure better data governance. This team can work together to establish data governance processes to maintain data security, such as creating an audit trail for data access and ensuring that only approved data is loaded into the data warehouse, data lake, and downstream tools.
- Perform advanced analytics. With centralized data and faster data transformation workflows, enterprises can launch advanced projects such as augmented analytics for data preparation, data discovery, and data science to unlock actionable insights.
- Deliver a better customer experience. Empowering business users with customer experience data such as sentiment analysis or telemetry information can help organizations determine new ways to delight customers, detect dissatisfaction, and create new products based on customer insights.
A Final Word
Every day, the modern data team must keep data flowing, governed, and accessible, so collaboration and data literacy are important skills for the team members. Their advice and expertise will shape their company's future growth, so they must be able to organize data for cost-effective analysis and reporting that leads to intelligent conclusions.
As business and IT data tasks increasingly overlap, the ultimate goal of the data team will be to create a strategic vision for data and provide the self-service access to insights that help the company achieve that vision.
About the Author
Ed Thompson is CTO and co-founder of Matillion. He started his career as an IBM software consultant and spent 11 years consulting for some of the premier blue-chip companies in the UK. Along with CEO Matthew Scullion, he launched Matillion in 2011 and set about building a team of data integration experts and software engineers. He and his team launched Matillion's flagship ETL product in 2014, which has driven the company’s growth ever since. Ed’s strength is his ability to bring together best-in-class technologies from across the software ecosystem and apply them to solving the deep and complex requirements of modern businesses in new and disruptive ways.