Move Over Data Scientists, Analytics Engineers Have the Sexiest Job
Why the analytics engineer will displace the data scientist as the world’s sexiest job, and how the modern data stack plays a part in this trend.
- By Cindi Howson, Michael Ritchie
- February 28, 2022
Picture it: a major snowstorm has created travel chaos. The de-iced plane that had been waiting on the tarmac for clearance to take off is now ordered to return to the gate. Some passengers and crew will unexpectedly need a hotel; others will rough it and sleep on the floor, the chairs, or whatever is available in the terminal, and everyone might have to fend for themselves. However, this airline has a super power: an analytics engineer. Rather than disgruntled passengers fighting queues and making phone calls to rebook everything, the analytics engineer has created a new pipeline to pull together data sources from:
To Unleash Data Potential, Enterprises Need to Fully Embrace the Cloud
- Customer loyalty programs to identify VIP customers and home addresses
- Passenger and crew list
- Partner hotels in the area with room availability
Text messages are sent to out-of-town passengers and crew with provisional hotel bookings.
Sound impossible? This kind of agility is increasingly possible with a modern cloud ecosystem and the rise of the world’s new sexiest role: the analytics engineer.
The rise of the analytics engineer reflects a confluence of forces in the industry in terms of skills, data stack modernization, and business pressures.
The Hunt for Talent
It was 2012 when Tom Davenport and DJ Patil first dubbed the data scientist the sexiest job of the 21st century in their seminal HBR article. Many flocked to gain skills in this area, attracted by the desire for impact and high-end salaries. Universities and online boot camps responded by establishing data science programs, but many missed the mark in meeting the needs of business. Newly minted graduates excelled in coding in Python and R but lacked business acumen and communication and data storytelling skills. Impact remained elusive as an estimated 85 percent of data science and AI projects failed to deliver ROI.
Furthermore, getting to useful data can be both painful and time-consuming. A data analyst is at the mercy of a centralized data team, waiting months for new data to be loaded into a centralized data warehouse. Alternatively, the data scientist spends too much time gathering and cleansing the data themselves and too little time on insights.
The demand for data and analytics skills remains high, outstripping supply, as businesses know being data-driven is not just a competitive advantage but the key to business success in today’s digital world. The data and analytics industry needs a modern approach to getting insights faster from all available data to all business users. Enter the modern data stack.
The Rise of the Modern Data Stack
Gartner estimated that in 2016, less than 10 percent of data was stored in a cloud data platform. They now predict that cloud data platforms will account for 50 percent of data this year. The shift of data to the cloud has been happening for years through cloud business applications. Maturing cloud platforms offer additional benefits over on-premises data platforms with elastic compute, data sharing, and agility. Lengthy data ingestion processes and aggregations forced by limits in on-premises capacity also mean that ETL has shifted to ELT. Enterprises can now bring in all their granular data and transform it in the cloud data platform in days or weeks instead of months.
The industry may debate approaches in the design of lakehouses or data meshes and data fabrics, but one thing is clear: businesses need faster insights on all their new digital data. The data mesh concept lends itself well to an analytics engineer role (see more on this trend here and here). Let the analytics engineer access the distributed data, but rather than build a non-scalable one-off, leverage platforms such as dbt to create an auditable model complete with documentation. A centralized data warehouse may continue to play a role in this, but it is no longer a bottleneck or the only source for data.
Now, for example, when the marketing team decides to implement a SaaS platform, the analytics engineer in marketing can rapidly respond with new pipelines and insights rather than rely on only the centralized data team to first load the data into a data warehouse. Further, with the greater ease in leveraging external data and the importance of this for leading indicators (another top trend), the same analytics engineer can pull in third-party data sources such as human mobility and customer demographics.
The analytics engineer will use a combination of no-code and low-code platforms to create a reusable asset. For example, he or she may use dbt within Snowflake or BigQuery to transform data and create complex calculations, employing an analytics platform scripting language to create an analytics application that contains pointers to the dbt models, includes Snowflake tables, and features customizable visualizations. Operational analytics that allows insights to be taken from an analytics platform (such as the list of stranded VIP passengers) and written back into an operational system is also enabled by cloud and open APIs. The combination of cloud applications and open APIs make insight to action a reality.
Although our flight-delay story was fictionalized, JetBlue is one such organization leveraging the modern data stack for greater agility and faster insight. As Ben Singleton, director of data science and analytics at JetBlue, explains, "Legacy, on-premises data warehouse and transformation tools were not designed for the explosion of data we've experienced over the last five years. We need to evolve and invest to meet the expectations of our data analysts and decision-makers."
Agility: A Business Imperative
Organizations were digitizing before the pandemic, but the pandemic accelerated most organizations' plans. Digitizing is one thing, but transforming and deriving the benefits of a digital world means leveraging data.
Consider a business that skyrocketed in the last two years: food delivery. If you think of the analog pizza delivery days, a restaurant would only know if an order was lost when the customer called to complain. Now, you can track the movement of your order in real time. The courier's mobile device is generating data. Your online order is generating data. Taking an item off a shelf is increasingly generating data such that data-savvy restaurants and food delivery organizations can say down to the food item, hour, and customer if the service was good.
However, the gap has further widened in the last two years between analytics leaders and analytics laggards. According to Accenture, analytics leaders now have two to three times the revenue growth of the laggards. Research from Kearny shows that laggards could improve their profitability by 81 percent if they were analytically mature. Part of this maturity is about technology modernization and culture, but it is also about the pervasiveness and timeliness of data-driven decisions. A recent HBR survey identified self-service analytics as a top priority to empower front-line decision makers. A static report based on stale, aggregated data is simply not fast enough or detailed enough to be actionable in today’s digital economy.
Becoming an Analytics Engineer
Knowledge of SQL, domain expertise, and software engineering practices are some of the key skills for an analytics engineer. This role may grow out of a data scientist role or a BI or data analyst. Good communication and problem-solving skills, plus curiosity, will be in their DNA.
Unlike report developers of the past, analytics engineers are not “order takers” or interested in one-off report requests. They will want to problem-solve for the business and consider the range of data sources available to answer a question, with a mindset of ensuring the solution scales.
Writing analytics code that scales to millions or billions of rows (especially semistructured rows) is challenging. It can take a long time to test changes and can feel quite different from the roles an analytics engineer had in the past. Beefing up your programming skills and remaining patient while you learn is crucial to a successful transition into this new role.
According to the Harnham 2021 Salary Survey, the salary of an analytics engineer is higher than that of a BI analyst but lower than that of a data scientist or data engineer. These higher salaries, the opportunity to contribute to higher business value, and the ability to scale results are the trifecta attracting data professionals to this role.
The Future with an Analytics Engineer
Back to our plane in the snowstorm. Without that analytics engineer, a BI analyst would have had to run a report or create a dashboard to show how many passengers were impacted by customer segment. The data scientist might have predicted the likelihood the flight would be canceled, but the analytics engineer could turn data insights into action.