TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The Future of Machine Data

More types of data are being collected every day. Circonus CEO Bob Moul spoke with Upside recently about how enterprises can gain new insights from machine data.

• By James E. Powell
• February 14, 2020

Upside: What technology or methodology must be part of an enterprise’s data strategy if it wants to be competitive today? Why?

Bob Moul: The volume of machine data in our world today is literally exploding and the importance of that data is rising sharply. We believe machine data intelligence platforms that can process billions of metric streams and also analyze that data in real time are essential in today’s “Internet of Everything” economy.

As more of our world becomes computerized, digitized, and virtualized, the growth of machine data is exploding. All machines emit data, technically called telemetry data, including resource utilization, location data, settings, metrics, measurements, and readings. Sources of machine data range from our thermostats and security cameras to IT infrastructures that keep our favorite movies streaming and mission-critical industrial applications that keep our energy grids up, oil wells flowing, and manufacturing lines running.

For Further Reading: IoT and the ML Connection The Intelligent Edge: Making Digital Transformations a Reality Why Synthetic Data Could Be the Ultimate AI Disruptor

All of this machine data is a virtually untapped source of extraordinary business insights. The key to unlocking this enormous potential is the ability to harness and make sense of the tremendous amount of machine data already being generated in enterprises. To date, conventional monitoring and analytics tools have always had limits on volume, frequency, storage, and scale -- but that is no longer the case. Machine data intelligence platforms can gather and analyze vast amounts of machine-generated data, including data from sensors, systems, and connected devices, to achieve new levels of insight that drive smarter operations, better decision making, and new business opportunities.

As the number of things we want to monitor and measure grows and sensors proliferate, enterprises have the opportunity to seize new levels of insight to drive competitive advantage and change the trajectory of their products, operations, IT infrastructure, and business performance.

What one emerging technology are you most excited about and think has the greatest potential? What’s so special about this technology?

Data compression technology is game-changing when it comes to data ingestion rates. This technology makes it possible to ingest a virtually unlimited amount of high-frequency data, allowing enterprises to securely collect and process trillions of measurements per second from a variety of data sources. Enterprises will gain more in-depth, accurate data than they ever have before, allowing them to make the best possible business decisions with more clarity and confidence.

What is the single biggest challenge enterprises face today? How do most enterprises respond (and is it working)?

The biggest challenge enterprises face today is getting the most value out of their data. Even getting their arms around the infrastructure they have and the metrics available to them can be challenging. Many are unsure what data streams they have and therefore can’t see all of the opportunities before them.

The reality is that most enterprises struggle to collect, aggregate, and store their machine data, let alone to derive actionable insights from that data.

Handling large volumes of machine data is a very difficult technical problem -- especially if you want to derive insights in real-time. Doing so requires custom platforms that are purpose-built to handle the data ingestion and storage and then deliver real-time analytics to drive growth.

When properly deployed, machine data intelligence platforms enable organizations to efficiently plan capacity, perform preventive maintenance before catastrophic failures, and forecast business needs and growth potential. They also enable more granular activities, such as notifying the business when certain thresholds have been crossed -- to ensure the safety of their employees and investments or to alert key staff when certain manual activities must be done in real time.

Is there a new technology in data and analytics that is creating more challenges than most people realize? How should enterprises adjust their approach to it?

Harnessing machine data is not a new challenge but rather an evolving one. Technology never stands still. We’re moving quickly beyond virtualization to containers and serverless, for example. Architectures are becoming far more complex. Add to that next-generation IoT sensors, intelligent devices, robots, and beyond. Does your enterprise have a strategy to stay on top of that proliferation? Moreover, can you harness that data for competitive advantage? Although machine data offers immense business value, it’s extremely challenging to manage, store, process, and analyze as the underlying IT landscape continues to shift and evolve.

Where do you see analytics and data management headed in 2020 and beyond? What’s just over the horizon that we haven’t heard much about yet?

As compute, device, and network costs continue to drop, you will see two tectonic shifts accelerate -- the implementation of intelligent operations in the physical world and digital transformation of services in the virtual world. The current distinction between IT infrastructure and IoT devices will blur. With that, you’re going to see exploding data volumes. Theo Schlossnagle, founder and CTO of Circonus, estimates that machine data is growing at the rate of 1×10 to the 12th power every 10 years, making the possibilities for machine data intelligence boundless. We believe that enterprises, especially market leaders, will finally begin to derive real value from this data such as optimizing operations, innovating products and services, and creating entirely new revenue streams-. As a result, we’ll see machine data intelligence surpass business intelligence as the next frontier of competitive advantage.

Describe your solution and the problem it solves for enterprises.

Circonus provides a machine data intelligence platform capable of handling billions of metric streams in real time to drive business insight and value.

Our connected world is unleashing an explosion of data that is outpacing the ability for organizations to process and analyze in a way that provides valuable business insights. Legacy information systems struggle or fail to keep pace with today’s data volumes.

Circonus processes and analyzes machine data at scale, helping our customers unlock transformative insights that allow them to lower costs, mitigate downtime, generate new revenue streams, and prevent problems before they happen.