TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Data Engineers Suddenly Very Much In Demand

What does it take to be a data engineer? A background in software engineering doesn't hurt. Although the number of data engineers doubled from 2013 to 2015, that growth rate far outstripped that of data scientists.

• By Steve Swoyer
• November 4, 2016

What's it take to be a data engineer? A background in software engineering doesn't hurt. You could use an expert grasp of SQL, Java, and Python. R ninjas need not (necessarily) apply.

These are just a few of the nuggets from "The State of Data Engineering," a recent report from Stitch, an intriguing start-up that bills itself as an "ETL service built for software developers."

How Many Data Engineers?

Stitch's report is based on an analysis of self-reported data from LinkedIn.com. It tallied only profiles in which the term "data engineer" is listed as a current title.

Stitch also counted only data engineers affiliated with known or identifiable companies.

"The State of Data Engineering" makes for fascinating reading. For example, Stitch discovered that the number of data engineers doubled from 2013 to 2015. It tallied 6,500 self-described data engineers on LinkedIn. That figure is probably just the tip of the metaphorical iceberg.

"Six-thousand five-hundred is not a big number. In fact, we were a little surprised at just how small it is. For comparison: as of this writing, there are 6,600 data engineering job postings on Indeed.... just in the San Francisco Bay area," the Stitch authors observe.

They also note that, although there are about two times as many self-described data scientists on LinkedIn (approximately 11,400) "the growth rate of data engineers is much faster than anything the data scientist job market ever experienced: In this same period [2013-2015], the number of data scientists grew by a little over 50 percent."

Salary Range and Career Path

"Salary data also confirms that data engineers are in demand. Anecdotally, top data engineering positions at tech giants [such as] Facebook, Amazon, and Google can exceed $500[,000]."

Stitch notes that Indeed.com's salary range for data engineering-related job postings in San Francisco runs a pretty wide gamut: from $85,000 on the low end -- barely enough to cover the cost of renting a nice used shipping container to call home -- to more than $130,000.

Software engineering is the crucible in which the overwhelming majority of data engineers are forged. According to Stitch's analysis, 42 percent of data engineers have a background in software engineering. The next most prominent role ("analyst") isn't even close; it was associated with only about 7 percent of all profiles.

Other gateway roles include "consultant" (about 6 percent of all data engineers), along with business analyst and data architect (both less than 5 percent).

Common Skills for Data Engineers

SQL is the most important -- or, at any rate, most common -- data engineering-related skill. Nearly 40 percent of the data engineers in Stitch's tally claim to be SQL savvy. Right behind SQL was Java: about 37 percent of data engineers claim to be skilled in Java.

SQL and Java are the two skills heavyweights. Number three is Python, which is claimed by about 27 percent of data engineers. Hadoop-related skills are cited by just over a quarter (approximately 26.5 percent) of data engineers. Next on the list are Linux-related skills, which account for 18.5 percent.

"SQL, a declarative language that most software engineers think of as little more than something to wrap up in an [Object Relational Mapping], is the most common skill for data engineers. This is big," Stitch notes. "For years, SQL was a bit of an ugly duckling within data tech with the ascendancy of NoSQL approaches. However, SQL isn't going anywhere -- in fact, it's enjoying a renewal as SQL-based interfaces for unstructured data (e.g., Impala, Drill, Hive, and Presto) increase in popularity."

R expertise didn't even crack the top 20, according to Stitch.

A Position with Solid History

Stitch's report is well worth reading. It's a wonder data engineering isn't a better known discipline. After all, "data engineer" -- unlike the newer "big data engineer" -- is an established, indeed venerable, job role.

The Institute of Electrical and Electronics Engineers (IEEE) has long recognized data engineering. Since 1977, IEEE has published the "Data Engineering Bulletin," a quarterly journal focusing on the "design, implementation, modeling, theory, and application of database systems and their technology."

Data engineer is, then, a solid, stolid position. The kind of thing you can brag to your prospective father- or mother-in-law about.

If recent trends are any indication, it will very likely become even more brag-tastic in future.