TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

How 3 Major IT Shifts Will Impact Data Management and Storage Solutions in 2021

As global creation of -- and access to -- data grows, these three IT trends will have a distinct impact on data storage.

• By Paul Speciale
• January 6, 2021

In 2021, three major shifts in IT will impact data management and storage solutions:

• Enterprise IT apps and infrastructure moving to the new cloud-native model
• Data being generated everywhere, globally
• Increasingly rapid evolution in platforms, APIs, and cloud services

Data storage has morphed significantly in the past decade to address the well-known avalanche of data that enterprises and consumers create and consume. The scale of data storage growth is truly astonishing. In the 1990s, one gigabyte of data seemed massive and a terabyte of data was a distant promise on the horizon. Today, we see companies of all sizes managing petabytes of data and beyond, even in industries where digital transformation arrived late, such as healthcare.

For Further Reading: Future-Proofing Your Data Storage Executive Q&A: Software-Defined Storage Security Considerations for Hybrid Cloud Data Management and Storage

This explosion of data pressed the data storage industry to rethink the scale of systems and enabled solutions that we refer to as “scale out,” meaning the ability to grow across multiple physical resources. New ways of protecting data have also developed because traditional mechanisms (such as RAID) were simply outgrown and would have resulted in significant data loss if depended upon for petabyte-scale data.

Twenty years ago, only the wisest prognosticators could have foreseen video streaming services to personal mobile devices or the presence of ubiquitous and low-cost, high-speed connectivity. Storage providers have been forced to think about global 24x7 access to storage for thousands, even millions, of simultaneous users.

As global access continues, we see these three IT trends having similar influences on data storage.

Trend #1: Cloud-native computing will shape data storage deployment and access

Every 10 years or so, the world of IT faces a significant change in infrastructure that affects applications and the underlying resources they depend upon. Twenty years ago this change was driven by server virtualization, and just over 10 years ago we saw the emergence of public cloud services greatly impact how businesses leveraged IT. Both of these trends had an impact on data storage, and we see a similar trend emerging now as IT moves to a new model called cloud-native.

Cloud-native will cause a shift in enterprise IT applications and infrastructure, which will also transform data storage. The changes we foresee are in how storage will be deployed as well as in how it will be accessed and consumed by applications. Deployment in cloud-native computing environments relies on building blocks of software services called containers, which are managed in container orchestration environments such as Kubernetes. According to IDC, by 2023 -- just two years from now -- over 500 million digital apps and services will be developed and deployed using cloud-native approaches. That’s the same number of apps developed in the last 40 years.

These applications will naturally consume storage over APIs such as the popular AWS S3 API for object storage. Moreover, new storage automation methods for container environments are emerging, including the Container Storage Initiative (CSI), the COSI project from the Cloud Native Computing Foundation, and the SODA Foundation project within the Linux Foundation. Both deployment and access to storage will, therefore, be shaped significantly by cloud-native computing.

Trend #2: The “data is created everywhere” dynamic will necessitate global data management

Another significant industry trend we see is the increasing decentralization of data creation. We have seen this before when we went through several evolutions of centralization (mainframe), local distribution (client/server), and globalization of companies (multiple data centers and remote office access).

Now, with enterprises having global data centers, using public clouds and an emerging edge-tier, it is clear that “data is created everywhere” is a dynamic that we must confront from a data management standpoint. This will drive new data management solutions that can view data globally through unified namespaces and can perform intelligent searches on data no matter where it is located.

Trend #3: Acceleration in new technology adoption will bridge data to the future

Finally, we see a macro trend that has been occurring for decades but is accelerating: an increasingly rapid evolution of the underlying technology that businesses depend on to store and manage their data.

Consider that just 10 years ago most servers had 40 to 60 gigabyte (GB) hard disk drives, making a server with 2TB of storage capacity “cutting edge.” Today we have 16TB hard disk drives creating over a petabyte of storage capacity in a single server. Now emerging are flash disks that can hold tens of terabytes of data, promising fast access at lower cost.

The access protocols for storage have also rapidly changed from block to remote file system access (NFS and SMB) and now to object APIs (AWS S3). This rapid evolution will create a need for data storage and management solutions in software that can bridge across and transcend these technology changes, while preserving existing investments and, most important, preserving critical enterprise data well into the future.