TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

How to Fix Data Engineer Burnout

Data engineer burnout has reached crisis levels. We explore four issues and suggest how to address them.

• By Satish Jayanthi
• February 21, 2023

A recent study yielded some dismal findings that won’t shock anyone who has worked in data engineering. Of the 600 data engineers surveyed, nearly all (97 percent) are experiencing burnout. In addition, 70 percent reported being likely to leave their current job in the next 12 months and 80 percent said they have considered leaving their career entirely. 

For Further Reading: 33 Ways Data Workers Can Beat Job Burnout How to Prevent Data Pipeline Engineering Burnout The Great Resignation: 10 Lessons for Data Teams

These overwhelmingly negative numbers point to pervasive pain points, including the sheer volume of data many engineers are obliged to work with and the pressure to stay up-to-date on a constantly evolving set of tools and technologies. So much so that according to the survey, 78 percent of data engineers said they wish their jobs came with a therapist to help them manage stress. 

At the root of that stress is a problem that has become pervasive in our industry: the lack of communication between business leadership and engineering teams, which often creates an unbalanced relationship and a lack of appreciation for what data engineering contributes to a business. 

Given the importance of data to organizations -- we can all agree that becoming truly data-driven is now a universal goal for business -- addressing engineer burnout should be a top priority.

I believe there are four main reasons for engineer burnout, and I’ll explain how companies can address and eliminate them.

Problem #1: Poor communication

Not many organizations will admit to this, but we’ve all seen it: too often, a company creates a culture that doesn’t support free and open communication, both within teams and among teams. If data engineers aren’t encouraged (or allowed) to express opinions and preferences -- or debate the best way to solve problems -- they can’t do their best work no matter what tools they have. Poor communication also leads to misunderstandings and a lack of alignment between engineers and business leaders. 

Business teams often bombard data engineers with requests without taking the time to understand what those requests entail and what results they can reasonably expect. This creates a world of moving targets that are difficult for engineers to prioritize logically, let alone completely. The high volume of requests leads to data engineers spending too much time on call and sometimes managing crises that aren’t within their official job scope. 

Solution: Enable and encourage open communication

To improve the likelihood of both sides getting the results they want, engineering and business leaders need to have an always-open door to a regular flow of thoughts, concerns, and ideas. In practice, it will often fall on the shoulders of engineers in a director role or similar to establish this rapport.

One best practice is to identify a champion in leadership on the business side who appreciates the contributions of the data engineers. This person can influence the thinking of other key executives, raise standards, and even help establish a data governance committee.

Problem #2: Lack of data literacy

Many organizations still lack -- or do not prioritize -- data literacy from the top down. This creates problems with data access, confusion about how data will be used, or even a misinterpretation or misrepresentation of data. When two business users or executives can look at the same data and understand it differently, miscommunication with the engineering team will lead to wasted work cycles and wasted time. The clear remedy to this disorientation is to invest in and adhere to data governance, but data governance is still often seen as a bottleneck instead of an opportunity to standardize data usage patterns.

Solution: Evangelize data literacy

Companies can foster an appreciation of data engineering teams by encouraging greater data literacy. I’ve seen tremendous improvement sparked by even small initiatives, such as setting aside five minutes at company-wide meetings to illuminate the work involved in data engineering. This education will establish greater trust and appreciation for what data engineers do and help leadership understand the importance of investing in programs that automate processes that allow engineers to do more high-level, creative work.  

Problem #3: Data quality has been neglected too long

Data quality has taken a back seat to speed of product delivery in recent years. With the rush to get products to market, many organizations are amassing significant tech debt, which is compounded when processes that ensure data is sound and relevant (such as data modeling) are skipped in favor of short-term ROI. These bad habits create data silos, general mistrust of data, and a further lack of understanding and appreciation for data engineering. 

Solution: Get back to the fundamentals

Create time for data modeling, which is foundational but has fallen by the wayside in favor of accelerating release dates in recent years. (This is another manifestation of the “slow down to speed up” idea.) What’s more, data modeling processes build in time for engineers to flex their creativity and determine how to execute the project properly to create more flexible, durable code -- improving their overall satisfaction.

Problem #4: Data engineering work is too often grunt work

More than half of all data engineers say they spend too much time on finding and fixing errors, maintaining data pipelines and manual processes, and playing catch-up with requests that come in at too fast a pace. This leaves little to no time to perform innovative or interesting work or to focus on strategy and developing long-term solutions. It’s no surprise that, as a result, engineers feel unappreciated, unproductive, and underutilized.

Solution: Automate as much as possible

Today, platforms and tools exist to automate nearly every stage in the life of data: from ingestion to transformation, and reporting and visualization. As budgets will likely continue to tighten in 2023 and businesses will be less willing to grow their engineering teams, automating data processes that were once done manually will enable data teams to not only do more with less, but focus on the work that generates business value.

Additional Steps

In addition to the four solutions offered above, I would like to offer two additional things a company can do.

First, abandon short-termism. Set the right expectations and have a “slow down to speed up” ethos. This sounds simple but can be revolutionary. This means prioritizing what is most valuable and breaking multi-year projects into short-term milestones. This can be hard for organizations that have emphasized a fast development, but prioritizing one or two projects and thoroughly understanding the implications of the data and how to integrate it is ultimately more effective.

Second, slow down -- really. If you’re reading this and you’re a data engineer, you should definitely take more breaks. Working more hours on more projects may accelerate your career in the short term, but you run the risk of getting burned out. It’s better to slow down, communicate with your team, and be part of gradual but meaningful changes that support lasting transformation and have an impact.