TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The Year of Resilience in a Data-Powered World

These three trends will help mold how we interact with and leverage data in 2022.

• By Max Liu
• January 21, 2022

The onset of the pandemic served as an eye-opening experience for those in the data and analytics industry. With IDC projecting an increase in the total amount of data being consumed to as much as 181 zettabytes in 2025 and the acceleration in digital transformation we have seen in the past two years, multiple concerns need to be addressed as we aim to thrive in a digitally transformed world.

For Further Reading: The True Cost of Moving to the Cloud Managing Data in the Cloud: The Challenge of Complex Environments for Real-Time Applications How 3 Major IT Shifts Will Impact Data Management and Storage Solutions in 2021

Organizations continue to face data, privacy, and resilience challenges as they adapt to new and more demanding realities. As we navigate the new year, there will be an increased focus on making processes agile, scalable, and fully available, moving from traditional databases into the future of data.

Here are a few of the trends and technology milestones that will mold how we interact with and leverage data in the upcoming year.

Trend #1: Democratization of chaos engineering will begin

In distributed computing, workload cluster failures have been top of mind for organizations. However, until now this has been a concern for large enterprises only, those with a massive web presence serving real-time transactions. As the pandemic accelerated digital transformation, it pushed more consumers to an online setting. Now that all organizations rely on distributed computing deployments, we'll see a shift into chaos engineering -- a Netflix-pioneered concept that better identifies vulnerabilities and resilience in these highly agile environments.

Chaos engineering is a process for testing a highly distributed computing platform's ability to withstand random disruptions and ultimately improving its reliability and resilience. It has been adopted by other major web companies, but it hasn't caught on with organizations running sub-hyperscale deployments, who lack the resources to leverage it -- that is, until now. Unlike centralized systems, distributed systems are more complex, and traditional software quality control approaches -- such as standard unit testing and integration testing -- will be difficult to ensure the stability of the system. We'll need to shift from deterministic testing to enhance the probability of abnormal states as quickly as possible to expose the problem.

In 2022, chaos engineering will be democratized through a new concept that is beginning to gain steam: chaos-as-a-service (CaaS). This concept will provide a quick, simple method to run chaos experiments and test systems' resiliency. CaaS will give access to a valuable but complex technology, eventually enabling organizations that aren't running at the scale of Netflix or Facebook to leverage chaos engineering and boost their infrastructure's durability.

Trend #2: Data begins to become a utility

Data powers the modern world. Organizations of all types depend on massive, rapidly growing and evolving data sets to deliver more intelligent services and achieve business growth. Given its importance and pervasiveness, data should be treated as a utility -- just like water, gas, and electricity. This means it must be made readily available and refined for easy and structured access.

Data can only become a utility with support from open source databases, data integration, and modern data management tools. Open source technology, specifically, will play a vital role, helping democratize and lower the barrier to entry for using these technologies while improving quality and reliability. Much like energy, with data's power being largely invisible, leaders will have to start by understanding their business needs and then leverage the benefits of open source and choose technologies that simultaneously fulfill those needs, whether they are scalability, availability, security, or a combination of these needs.

Trend #3: The rise of cloud-native databases

Although accelerating digital transformation continues to be top of mind, many organizations are still facing extensive big data challenges, including the need to extract real-time, relevant data at scale and at a faster pace. In the past few years, we have seen how traditional database systems have struggled to keep up with all the requests and the new data that has flooded in.

On the other hand, as applications move to the cloud, databases as the serving infrastructure are following the lead. In 2022, organizations will increasingly turn to cloud-native databases for enhanced data migration efficiency, intelligent data retrieval, and real-time insights, which will help provide greater data agility, reliability, scalability, and availability compared to traditional databases. The adoption of cloud-native databases will pick up, particularly among enterprises in the e-commerce and finance sectors, which support massive numbers of customer transactions and must rapidly expand data volumes while creating new apps to deliver new services.

More leaders will bet on cloud-native databases to mitigate complexities introduced to their environment by conducting data integration themselves or through third-party data management tools. The business results will shine through as they will be able to extract data much quicker and lower their operational costs while taking further steps toward their digital transformation goals and business growth initiatives.

Specifically, for cloud-native databases in 2022, there are a few trends worth our attention:

• The separation of computational and storage resources, or even computational and memory resources, will enable better hardware resource scheduling.
• More cloud-native databases will be built upon standard cloud services such as S3, EBS, EC2, and Lambda. The high availability, by-design elasticity and relatively lower cost of the cloud storage services naturally fit in the design and implementation of cloud-native databases.
• The composition of the database will be highly available as a microservice. Pluggable storage engine and serverless-based SQL engine will become the trend of cloud databases.
• Multitenancy will become a norm for cloud databases.

Here's to a Smarter, More Innovative Future

We're already living in a digitally transformed world. In 2022, organizations will increasingly implement newer technologies that will help them endure and thrive in the post-pandemic world.