5 Data Management Lessons Learned from the Front Lines of COVID-19
Across industries, COVID-19 has exposed common data and data management challenges. These five key lessons are shaping post-COVID-19 expectations of data management and analytics.
- By Ben Sharma
- June 19, 2020
In April, a Gartner survey revealed that 18 percent of companies said they were highly prepared for the impact of coronavirus. That’s a small percentage which now seems optimistic. It has become clear in the weeks since that the data and analytics challenges facing most industries are critical and urgent -- and have catalyzed rapid shifts in priorities.
Whether logistics for medical supplies, virtual customer engagement, pandemic response centers, insurance processing, real-time outbreak monitoring, or PPE distribution, companies that met these data challenges with a mature data operations pipeline are delivering faster, better solutions.
The shifts in data management priorities are here to stay as companies seek to meet current needs and ensure they are prepared for unknown future crisis scenarios. In going through this process, several priorities have taken on greater importance:
- Efficiently moving data through the entire data supply chain has become table stakes
- An agile DataOps platform is the key to swift data scaling and ingestion of new sources
- Unifying siloed data for better AI and ML-based analytics is performance-critical
- Data collaboration features must improve both speed and quality of insights
For Zaloni’s enterprise data management customers, who are largely from regulated industries such as banking, insurance, pharmaceuticals, and life sciences, these priorities have risen as they swiftly adopt solutions for novel complex data challenges including quickly processing PPP loan applications, COVID-19 testing initiatives, and accelerated virus treatment trials.
In building solutions to accomplish these initiatives and support shifting priorities, here are five lessons we’ve learned that result in data management success.
Lesson #1: Agility is key and it requires a strong foundation
The CARES Act was signed into law on March 27, 2020. Banks were asked to accept Paycheck Protection Program (PPP) loan applications starting April 3, just one week later. Many banks were unprepared to accept the flood of data in the form of digital applications, supporting documents, and financial histories, much less process them in a way that allowed efficient analysis.
Companies that successfully dealt with time-pressured COVID-19 challenges, such as Bank of America’s processing of over 60,000 PPP loans on the first day of application acceptance [], were able to do so because they had an agile data operations foundation. This means these banks:
- Had a loan origination platform in place capable of ingesting massive amounts of new types of data
- Could add new data sources to their catalog promptly
- Were able to provide self-service access to the new data for their analysts
- Had extensible connections across the data supply chain so the new data could easily be quality checked algorithmically and analyzed with BI tools to begin discovering valuable patterns
Agile data operations was the key to their ability to move forward with coronavirus initiatives and will be fundamental for future crisis-ready data management.
Lesson #2: Streamlined DataOps delivers scalability
The immense amounts of data accompanying efforts such as outbreak monitoring or virus testing have been a primary priority-shifting challenge. Even for companies accustomed to large-scale data processing, the enormity of new data requirements has given an advantage to businesses capable of nimbly scaling their data management environments.
Our work with a leading medical testing business offers an example. This global healthcare diagnostics company recognized the need for millions, if not billions, of fast-acting, easily distributable COVID-19 tests. To accomplish this quickly, their analysts had to work across new real-time sets of data, using advanced analytics to determine the factors required for effective testing. They also had to analyze operations data to determine how to develop, manufacture and distribute them.
Having a data platform capable of ingesting vast amounts of near-real-time data at scale has been critical to their ongoing success.
Lesson #3: Controlled governance with an end-to-end view reduces risk
Many of the significant COVID-19 initiatives involve sensitive data, from patient data to personal banking data. There are great risks to data security posed by the swift influx of these new data sources into a company’s data ecosystem. Protecting customer data and excellence in data governance are primary priorities for a bank we work with, priorities that have only grown in the time of coronavirus.
Key to governance success is having a platform that tracks and visually re-creates the lineage of data from ingestion all the way to consumption. On top of showing who has had access to the data, when, and what was done to it, adding data quality workflows, masking, and tokenization as automated features increases data security and accuracy; it also offers compliance-readiness for current and future unknown obstacles.
We have repeatedly seen that customers who started the crisis with these functions already in place were better able to add new projects speedily without endangering their data security.
Lesson #4: Collaborative data features accelerate analytics
The heightened need for faster analytics has been one of the salient priority shifts catalyzed by the coronavirus. Perhaps no sector has felt this pressure more than companies tasked with discovering treatments for the virus.
To analyze data sets efficiently, facilitate needed data collaboration, and analyze vast amounts of research, healthcare companies need streamlined data pipelines that can unify relevant data in their catalog, work across research teams to analyze real-time results, and ultimately accelerate their development timeline.
A collaborative data catalog allows analysts to tag, recommend, and share data sets, patterns, and results, enabling faster collective insights. Healthcare companies that began the crisis with a collaborative, efficient DataOps platform were prepared to run analytics models as soon as the data became available.
Lesson #5: ESG matters and will only grow in importance
Both consumers and businesses were placing increased priority on environmental, social, and governance (ESG) scores, data, and analytics before COVID-19. The virus has heightened their importance. Companies that score well for ESG metrics, and those that provide the analytics around such scores, are consistently rated higher for trustworthiness and known as responsible investing leaders.
In a time of economic uncertainty, these trust metrics are key differentiators for driving new revenue growth from customer acquisition. We helped an investment firm build an ESG data and analytics solution for each company in their portfolio, enabling them to offer ESG scores to their customers so they could both grow customer loyalty and offer a differentiated product with a competitive advantage. Having their own ESG scoring platform and derivative products has benefited revenue and their reputation as a leader in trustworthiness in the time of COVID-19.
Implementing DataOps
Successful DataOps enables several key pillars necessary to the modernized data supply chain that make it agile enough to cope with current and future crises. DataOps also enables cost and time efficiencies needed to grow revenue in challenging periods.
In the end, implementing a DataOps solution streamlines the data supply chain, saving companies IT and related costs, accelerating data-based revenue projects, facilitating better analytics through collaboration and transparency, and ultimately creating a scalable foundation for future, unknown challenges.