To Maximize Your DataOps Future, Take Advantage of These Four Trends
As the big data landscape matures, take advantage of these four trends to advance your analytics efforts this year.
- By Mark Marinelli
- February 28, 2020
It’s 2020 and the challenges associated with enterprise data continue to intensify along with enterprises’ demands for easily accessible, unified data. Now more than ever, organizations are looking for ways to spend less time unifying, prepping, and cleansing data and more time analyzing it and using the insights it provides to generate strategic advantage.
As the big data landscape matures, here are four trends that enterprises can capitalize on to advance their efforts this year.
1. Smart, Automated Data Analysis Will Rule
Internet of Things (IoT) devices will generate enormous volumes of data that must be analyzed if organizations want to gain insights -- such as when crops need water or heavy equipment needs service. John Chambers, former CEO of Cisco, declared that there will be 500 billion connected devices by 2025at’s nearly 100 times the number of people on the planet.
IoT devices are just one factor driving this massive increase in the amount of data enterprises can use. There’s also the influx of clickstream data from social channels, cybersecurity attack information generated by machines, and other sources.
The result of all of this new data is that its management and analysis will become more difficult and will continue to strain or break traditional data management processes and tools. Only through increased automation via AI and machine learning (ML) can this diverse and dynamic data be harnessed.
2. Custom Solutions from Purpose-Built Components
The explosion of new types of data in great volumes has demolished the erroneous assumption that enterprises can master big data through a single platform. The truth is, no single vendor can keep up with the ever-evolving landscape of tools to build enterprise data management pipelines and package the best ones into a unified solution.
Organizations will need to turn to both open source and commercial components that can address the complexity of the modern data supply chain. These components will need to be integrated into end-to-end solutions. Fortunately, they have been built to support the interoperability necessary to make them work together.
3. Increased Approachability of Advanced Tools
The next few years of data analysis will require a symbiotic relationship between human knowledge and technology. With more data in a variety of formats to deal with, organizations will need to take advantage of advancements in automation (AI and ML) to augment human talent.
Simultaneously, knowledge workers will need to improve their technical skills to bridge the gaps that technology cannot fill completely. Only through the combination of automation and increased human knowledge can organizations solve the problem of getting the right data to the right users so they can make smarter decisions.
Thankfully, the supply of data-proficient workers is increasing. As we see more data management tools packaging AI/machine learning with cleaner user interfaces, we’ll see lower barriers to adoption of these newer techniques.
4. Subject Matter Experts Will Become Data Curators and Stewards
Organizations will need to think about crowdsourcing when it comes to data discoverability, maintenance, and quality improvement. Ultimately, the people required to make data unification truly effective are not data engineers but rather highly contextual recommenders -- subject matter experts -- who, if directly engaged in the unification process, can enable a new level of productivity in data delivery.
Data consumers (non-technical users) know customer, sales, HR, and other data by heart. They can assess the quality of the data and contribute their expertise to projects to improve data integrity. However, they are too busy to devote their time to the focused tasks of data curation and stewardship. There will be a huge opportunity for improvement as more people are allowed to work with the data they know best and provide feedback about whether it is accurate and valuable from within their existing tools and workflows.
Incorporating this data feedback systematically instead of having it locked up in email or possibly never provided at all will produce dramatic gains in the ability to focus data quality efforts on the right problem sets, to correct issues with source data, and (ultimately) to prevent bad data from entering the enterprise in the first place.
A Final Word
Traditional data management techniques are adequate when data sets are static and relatively few, but they break down in environments of high volume and complexity. This is largely due to their top-down, rules-based approaches, which often require significant manual effort to build and maintain. These approaches are becoming extinct quickly.
The future is inevitable -- more data, technology advancements, and an increasing need for curation by subject matter experts. Data unification technology will help by connecting and mastering data sets through the use of human-guided machine learning. The future is bright for organizations that embrace this new approach.
About the Author
Mark Marinelli is head of product with Tamr, which builds innovative solutions to help enterprises unify and leverage their key data. A 20-year veteran of enterprise data management and analytics software, Mark has held engineering, product management, and technology strategy roles at Lucent Technologies, Macrovision, and most recently at Lavastorm, where he was chief technology officer.