Revisiting the Analytics Trends Pendulum for 2022
Advice about the three most important trends or technologies data/analytics professionals should pay attention to in 2022 and why.
- By Rob Armstrong
- December 7, 2021
After spending over 35 years in the analytics industry, I have seen the pendulum swing back and forth a few times now. The coming year seems to be a time when we get another swing and will see some familiar trends in new lights. With a bit of hindsight, we can all be better prepared from them this time around.
Trend #1: Cloud: Is this the answer?
How many times have we seen it before: the silver bullet has finally arrived! This happened with data lakes, big data, agile development, and many other trends. All of these had positive elements that survived, but unfortunately they were often implemented without the foundational principles to ensure long-term success. The cloud is clearly a game changer, but it is not the only game. Oh, and there is not just one cloud, so keep in mind that they all have their own idiosyncrasies.
Cloud environments bring lots of positive capabilities such as quick instantiation and scale, consumption pricing, and lots of toolsets integrated with their platforms and networks. However, they also bring egress charges, network traffic, and shared infrastructure that may not be as optimized for a singular community. The cloud also must fit in a world where data privacy and security are also becoming more critical to a company's overall business.
People like the idea of cloud but also like the control they have over on-premises systems. Although "the cloud" is good, it needs to be balanced with hybrid models. In a hybrid model. some aspects of the architecture are within corporate data centers for security (or the perception of it), data gravity, operationalization at scale, or manageability and other systems are in the cloud for "fail fast exploration" and independent model development. Learn the lesson of the past and balance the cloud benefits correctly. Remember, too much of anything can be bad for you.
Trend #2: Data mesh: Decentralized development needs enterprise oversight
With data coming from so many places and in so many formats, it does not make sense to try and centralize all development. Add to this the multitude of platforms (thanks again, clouds) and advances made in network bandwidth, it is now feasible to allow different data staging architectures and embrace cross-platform analytics. So that is all good.
However…
The data mesh is a way to leverage resources and scale data management, data ingestion, and data ownership. Additionally, across your corporation, there is also the need to have the autonomy for flexible data models. Keep in mind that using the data mesh approach does not alleviate the need to have a consistent and unified view of your data. My advice if you're going down the data mesh path is that you also need to have oversight of the key data elements that tie your corporation together -- elements such as customer and product identifiers, location codes, and corporate metrics.
These are the data elements that unify and codify your business. Without consistency across these types of elements, you will end back in the mess of "which data is where, for what purpose, and how is it represented?" Individuals will not be able to run cross-area analytics without an onerous task of data reconciliation and transformation. Data mesh is a good future direction for quick development but learn the lesson of the past and make sure it has some oversight across the teams.
Trend #3: Governance: The goal is agility without anarchy
Speaking of oversight, governance is once again rearing its ugly head. I say that because when most people hear "governance" they envision an overbearing and cumbersome set of edicts with little room for innovation or exploration. This is not the case. In short, governance should drive empowerment while minimizing confusion.
Two of my favorite sayings on this topic are (1) "Rules are for people who can't, won't, or shouldn't think; all others get governed" and (2) "Great governance demands an involved and educated populace." Taken together, this means that executive governance bodies should work with the IT and business communities to lay the guardrails that encourage exploration but also ensure safety.
Governance is not a set of rules that cover every possible and conceivable option but rather a set of principles that your company adheres to as it moves forward (think the Ten Commandments, not the U.S. tax code).
For example, one point of governance may be that you want to minimize data replication and maximize data reuse, and thereby also improve data quality. Another may be to incrementally deliver only the level of governance each business initiative needs, thereby creating a cumulative effect of multiple business needs over time. A final piece of advice: undertake no big bang efforts that are not grounded in delivering business-driven value.
Governance covers a lot of territory, much more than can be discussed here. There are many good articles available for you to explore. [Editor's note: Upside has consolidated articles about data governance and compliance on this page.] At a high level you need to ensure that you have governance at the data side, the architecture side, and the financial side. You must have the elements in a flexible infrastructure that is not only cost effective but is balanced out by the value derived from user analytics that drive favorable business outcomes.
Parting Thoughts
The thing about a pendulum is that while it likes to swing, and it does not like to stay at the ends very long. Take the lesson that anything at the extremes will be bad and ensure you have a strong counterbalance -- one that provides oversight to keep your company centered by balancing the data, the architecture, and the governance of a complete analytics ecosystem.
About the Author
Rob Armstrong is an ecosystem architect at Teradata. Since 1987, he has contributed in virtually every aspect of the analytical arenas with some of the largest and most complex corporations. Today he helps companies not only create the foundation but also incorporate the principles of a modern data architecture into their overall analytical processes. He can be reached via email or LinkedIn.