Putting the Data Revolution in Perspective
Advances in analytics and data management have practical effects on the world around us. These improvements are driving innovation in fast-changing industries such as healthcare and agriculture.
- By David Stodder
- July 21, 2016
As professionals immersed in this industry, it is easy to get so focused on technology details that we lose sight of the big picture. Our world is changing dramatically due to advances in analytics and data management combined with networking and processing power.
Telemedicine and Big Data Changing World Health
I was reminded of these changes recently when I picked up The Wall Street Journal and read "How Telemedicine is Transforming HealthCare," by Melinda Beck (Journal Report, June 27, 2016).
In a case example, the article describes how Doctors Without Borders "relays questions about tough cases from its physicians in Niger, South Sudan, and elsewhere to its network of 280 experts around the world and back again via the Internet." Doctors located at Mercy health system's Virtual Care Center outside of St. Louis, Missouri, are able to examine data remotely and can use cameras to zoom in to see conditions that require attention.
Although there is still debate about whether telemedicine is the best way for doctors to care for patients where one-on-one visits are feasible, it is clearly valuable for providing medical care and expert advice to those in remote areas where healthcare is inadequate.
Increasingly, doctors will determine treatments with the help of analytics and data access to study population health, defined by Dr. Kenneth Kizer in the Journal as "the health status or health outcomes of a group of people who share one or more common characteristics," which could include genetics, clinical conditions, age, or other factors.
Doctors will need near real-time access to data and analytics about the "evidence" -- that is, the body of knowledge about conditions and treatments relevant to their medical decisions. With telemedicine, this could be provided globally from medical hubs to doctors and patients in remote, less well-served areas.
Healthcare is changing fast due to a number of factors, including new payment models and policies, but the growing role and impact of analytics and improved data access is clear. However, all firms in this industry continue to run into data integration obstacles. Effective use of data is stymied by poor data definitions, data models, and other issues that make integration difficult.
Part of the problem may be that older, traditional modes of data integration and querying cannot overcome the difficulties presented by so many disparate and inconsistent data silos. The healthcare industry is ripe for new ways to pull data together and analyze multistructured information. In fact, we are already beginning to see progress made in healthcare using technologies such as advanced search, machine learning, and cognitive computing to assemble heterogeneous data and discover insights in new ways.
Data Drives the Human Food Chain
Healthcare is hardly the only industry that is becoming more data driven. Agriculture is also undergoing dramatic changes with the availability of new sources of data -- coming from satellites, sensors in the ground and on equipment, laboratories, and elsewhere.
A recent article in The Economist Technology Quarterly (June 11, 2016) described data-driven irrigation at Tom Rogers' almond farm in California's Central Valley: "Moisture sensors planted throughout the nut groves keep track of what is going on in the soil. They send their results to a computer in the cloud to be crunched. The results are passed back to the farm's irrigation system.... [Rogers] uses 20 percent less water than he used to."
Other articles in the Technology Quarterly, which was dedicated to the future of agriculture, described the new world of smart, "precision" farming using geospatial analytics to tailor seeding to local conditions, satellite tracking of soil conditions, and predictive modeling of livestock fertility.
As part of my research for the upcoming TDWI Best Practices Report: Improving Data Preparation for Business Analytics (Third Quarter, 2016), I interviewed Farm Market iD, an innovative provider of data and marketing solutions to firms in the agriculture industry.
Farm Market iD has built up a large, proprietary database of facts about nearly all the farmland in the U.S., including data about farm owners, what is being grown, and geospatial data about field boundaries. As it begins to analyze this data along with massive, externally available streams of data about weather, soil conditions, water tables, and other geospatial data, Farm Market iD will leverage this big data gold mine to realize one of the agriculture industry's biggest dreams: projecting yield.
"If you can start projecting yield, it becomes unbelievably powerful," said Steve Rao, CEO of Farm Market iD. "You can really start to address challenges and disruptions to the food supply."
To scale to handle the growing volume and variety of data, the company had to upgrade its data preparation to be faster and less repetitive. "We felt like we were running through oatmeal, manually having to touch everything that went through the pipeline and doing a lot of the same fundamental data standardization, deduplication, format checking, and quality steps over again for each client," Rao said. "The speed of the data coming in and the speed with which we needed to push it out to clients were getting faster, but we were slowed using disparate and unintegrated software."
The company has been working with RedPoint Data Management to automate, standardize, and integrate data preparation tasks to produce deliverables that it could not provide before, such as its Dealer Insights report, which requires 60 different data preparation processes.
Data Advances Benefit the Future
Data preparation, which involves a spectrum of processes from data ingestion to transformation, data quality improvement, and cataloging, will be a critical area of improvement for organizations striving to realize the potential of data.
This is particularly true in industries that have outgrown spreadsheets and single-application databases but are relatively new to using data extensively for decisions and driving operations. Getting the data "food chain" moving smoothly across systems and divisions so that analytics and reporting are properly nourished will determine which companies succeed or fail in fast-changing industries such as healthcare and agriculture.
To build the future, data professionals will be deeply immersed in technology, data modeling, data analysis, and data architecture details. Every once in a while, they should step back and consider the dramatic changes that their work will bring, hopefully to the benefit of many.
About the Author
David Stodder is director of TDWI Research for business intelligence. He focuses on providing research-based insight and best practices for organizations implementing BI, analytics, performance management, data discovery, data visualization, and related technologies and methods. He is the author of TDWI Best Practices Reports on mobile BI and customer analytics in the age of social media, as well as TDWI Checklist Reports on data discovery and information management. He has chaired TDWI conferences on BI agility and big data analytics. Stodder has provided thought leadership on BI, information management, and IT management for over two decades. He has served as vice president and research director with Ventana Research, and he was the founding chief editor of Intelligent Enterprise, where he served as editorial director for nine years.