Modern Data Engineering with Matt Florian
Matt Florian, partner and cloud analytics practice director with Comerit, discusses modern data engineering and key tips for data engineers.
- By Upside Staff
- January 26, 2024
In this recent “Speaking of Data” podcast, Matt Florian discusses data engineering for tomorrow’s enterprise landscape, including important tips for data engineers. Florian is partner and cloud analytics practice director with Comerit and a TDWI faculty member. [Editor’s note: Speaker quotations have been edited for length and clarity.]
Florian began by talking about some of the changes he’s seen over his 30 years in the data field.
“One particular thing that’s changed,” he began, “is the love/hate relationship data engineers have had with data modeling.” He explained that when he was teaching modeling in the 1990s, the model would be developed in some high-brow academic fashion, but once it was handed over to the engineers, they would toss it out the window and do whatever they wanted.
However, now the data model is more often developed by product owners who are creating data products with a fuller knowledge of what people want to do with them. As a result, the model is more applicable to the final product and, as a result, is more useful to the engineers.
“Of course, we’ve also seen a lot of change in roles and titles,” Florian noted, “even though they’re doing essentially the same work. For example, what today’s ‘data engineers’ are doing used to be done by ‘ETL developers.’ The tasks are often the same though.”
In talking about the course he will be teaching at TDWI Transform 2024 in Las Vegas, Florian laid out five clear pieces of advice that he gives to all the early-career data engineers he’s worked with over the years:
- Keep it simple. Don’t try to develop an overly complicated solution. Start simple and build up from there.
- Let your solution evolve on its own. Don’t start out by trying to hammer out every last detail up front. Although you will start from a set of requirements, you’re developing a product for the business. The data and its users will naturally show you where you’ll need to go.
- Don’t over-optimize too early. You won’t know where your constraints actually are until you deploy your code and see it in action, so be patient.
- Don’t code for yourself. This applies to developers at any stage of their careers. At some point, someone else will have to read and support all those lines of code that you’ve written, so be mindful of those who will come after you.
- Code with empathy for your consumer. Whether you’re an employee or a consultant, always keep your consumer in mind -- their needs, their goals, and the business process they’re engaged in.
Florian went on to explain what it takes to succeed in data engineering, comparing it to a winning baseball team.
“You don’t get to the World Series by hitting the most home runs,” he said. “You get there by hitting the most singles and doubles.” In the same way, Florian said, success in data engineering is not a matter of duplicating what the giants such as Facebook and Amazon are doing -- it’s a matter of consistent, smaller-scale victories over time.