TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The Age of Instrumentation Enables Better Business Outcomes

The benefits of instrumentation are massive but only if you start now and invest in the right tools.

• By Evan Kaplan
• May 30, 2018

We hear the phrase "Work smarter, not harder" all the time. Does this mean we should hire Ph.D.s and MBAs to help our companies make better business decisions? Of course not. It means we need to use technology wisely to gather and analyze relevant information at the right time to help us make (and act on) intelligent decisions in time for a real bottom-line impact.

For Further Reading: The Importance of Being IoT Creating an Effective Information Strategy Machine Learning, Predictive Analytics, and the Modern Enterprise

Fortunately, it's getting easier to work smarter, not harder, with the judicious use of technology now that industry trends have moved us into the Age of Instrumentation.

By instrumentation, I mean putting a sensor on something to take a measurement or detect the occurrence of an activity -- an "interesting business moment," if you will. This bit of data is time-stamped and collected in a database where it joins millions of similar data points that collectively tell a story either over time or at a specific point in time.

Almost anything that is capable of giving off a measurable metric or recorded event can be instrumented, in the virtual world as well as the physical world. Metrics commonly involve the measurement of things such as temperature, pressure, heart rate, salinity, volume, speed, and countless other characteristics. An event can be anything that happens -- a sale transaction is finalized, an account is created, a file is accessed, a valve opens, a virtual machine is spun up, or a switch turns off.

The placement of sensors on practically every available surface in the material world -- from machines to humans -- is a reality today. The Internet of Things provides full instrumentation to enable precise control of everything from utilities and critical infrastructure to home automation and self-driving cars. Healthcare is another important industry that is highly instrumented, as human health and wellness are being measured and monitored in every possible manner.

In the virtual world, instrumentation is used for monitoring and controlling the software components that drive our business processes, such as transaction engines, applications, microservices, containers, software-defined networks, and operating systems. These things are ephemeral; they come and go as needed to support modern computing infrastructures. Instrumentation in software is critical for visualizing what these systems are doing -- what activities are happening and precisely when -- and how well the applications and services are performing. From there they can be tweaked and optimized to get the best bang for the computing buck.

Naturally, there is a business justification that has taken us into the Age of Instrumentation. Companies want to become more data-driven, applying data insights to identify business opportunities and threats. Data-driven predictions are more effective than relying on historical information or gut instinct. When vast amounts of data points are collected, correlated, and analyzed, the organization can find insights that help identify emerging opportunities and competitive advantages and show the company where to invest resources for maximum benefit. Letting the data insights show the way forward is a very real way to "work smarter, not harder."

Technical Imperatives in the Age of Instrumentation

Placing sensors and collecting data are just the beginning steps on the path to being data-driven. There are many technical imperatives to heed as you instrument your physical and virtual worlds.

Enable decisions that matter in the moment. Every business wants to be "real-time" -- that is, operating as quickly as possible so that decisions based on incoming data can have a meaningful impact on what happens next. On the technology front, this requires the ability to ingest vast amounts of data very quickly and to store it in a database that can accommodate continuous real-time queries and other immediate actions on the data. Some decisions need split-second timing.

Consider, for example, a collision avoidance system designed to prevent or reduce the severity of a collision in a modern vehicle. It correlates data coming from GPS, radar, LIDAR, and camera systems to detect an imminent crash and either warn the driver or even take autonomous action to apply the brakes or steer the vehicle in another direction. All of this happens in milliseconds. The decisions this system makes matter in the moment, determining whether a crash can be avoided.

Enable observability of your systems. When everything within a system is instrumented, you have the power of "observability" -- the ability to gain situational awareness of what's happening inside that system that would otherwise be a mystery. Beyond simple monitoring, observability includes gaining insight across systems, observing behavior of the system components, and using machine learning to predict what will likely happen next. For example, predicting when various machine parts will wear out or fail so they can be replaced just prior to failure, thus reducing downtime and the potential for costly repairs.

Enable actions, not just reports. Monitoring what is happening within a system is not enough. The real power comes from using that data to control what happens next. With the right understanding of the right data in real time, you can make well-informed decisions that lead to beneficial actions. What's more, humans don't necessarily have to be the decision makers and action takers; it's possible to automate these steps to create even more value. This is the principle behind driverless cars that act on their own according to current conditions, self-healing systems that detect problems and restore themselves to normalcy, and autoscaling computer applications that expand or contract resources according to current demand.

Embrace new solutions. Just as the Age of Instrumentation holds the promise of new opportunities, it also requires technologies that are purpose-built to collect, store, analyze, act on, and divest vast amounts of data. A general-purpose database can't accommodate the volume and the real-time nature of time-stamped data. Likewise, the data collection agents must be tailored to the numerous sources of metric or event data. Machine learning technologies are essential for finding those "business moments" in the data. Companies that want to get the most from their instrumented systems must embrace new data tools.

Meeting the Requirements

The technology requirements to leverage the Age of Instrumentation are a tall order, but new purpose-built platforms can now deal with these specific metrics and events, otherwise known as time-series workloads, and provide situational awareness to the business. These platforms support ingesting millions of data points per second, are able to scale both horizontally and vertically, are designed from the ground up to support real-time insights, and have strong machine learning and anomaly detection functions to find interesting business moments. In addition, they are resource-aware, applying compression and down-sampling functions to optimize resource use, and are built to support faster time to market with minimal dependencies and storage required.

The Age of Instrumentation is here and data-driven organizations thoroughly understand that instrumentation provides the foundation for real-time control and adaptive systems that allow businesses to thrive in a rapidly changing world. Where are you on the instrumentation journey? The benefits to your organization are massive, but only if you start now and invest in the right tools.