TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Under the Covers: Databases in Devices

We break down the basics and benefits of embedded databases.

• By William McKnight
• December 9, 2019

Embedded databases have become a popular use of database technologies. Aggregating all the uses of databases would show embedded databases to be just as popular (or more so) than the more extrinsic approach now that there are 25 billion connected devices, growing to a projected 75 billion by 2025. We've all used them, knowingly or not. Now, because many enterprises have become quasi software factories churning out applications, building mobile applications, and supporting IoT, enterprises have jumped into embedded databases in a big way.

For Further Reading: The Intelligent Edge: Making Digital Transformations a Reality Q&A: Cutting-Edge Analytics Technologies Are At the Edge  Why Continuous Intelligence Should Be a Key Priority for Business Leaders

Enterprise embedded databases can be found, for example, in mobile airline applications that feature online check in, boarding pass retrieval, flight status checking, and real-time flight information retrieval.

Enterprises using IoT can use embedded databases to copy aggregated IoT sensor data to a back-end database when online. This brings the value of data directly to operations. At the same time, data from all the IoT devices is being developed in the back-end database in order to develop analytics to advance the business.

Embedded databases are built into the software application, transparent to the user and maintenance-free. This has become the new preference over client-server approaches that use database servers accessed by client applications. Embedded databases are installed with software silently, fine-tuned for the small footprint of a small device, and built for a range of devices.

Purposes for an Embedded Database

The main goals are to get the data to the devices and collect data on the edge. The purpose of collecting data at the edge has shifted from purely device control and monitoring to improving various service capabilities through real-time analysis.

There are many reasons to embed a database in your application endpoints instead of just pushing data out to the device -- the same reasons you use a database for an unconnected application. Additionally, embedded databases are low cost, embedded in the application installation, and DBA- and administration-free -- with disaster recovery, encryption, and security built-in. They accelerate development and work across multiple platforms (Windows, Mac, iOS, Android, and others). In addition, modern IoT development tools with databases have made the development process more accessible.

Flat files at the endpoints lack data portability, lack a single API portability across programming languages (such as NoSQL and SQL), and have more unsolvable data integrity problems. Furthermore, indexing, filtering, SQL, metadata, synchronization, security, reporting, and auditing have to be built.

Capabilities for an Embedded Database

When choosing your embedded database, just like when choosing an enterprise operational database, one area that truly matters is write speed. This is the essential performance metric for IoT data and can be viewed with and without synchronization of data to a server.

Your chosen solution needs to be able to automatically synchronize -- in real time -- between a remote device and a server without ETL. This capability is critical for today's needs and uses because the potential number of mobile devices could easily run into the thousands, and all that information may need to funnel into a core database on a server.

Many database vendors now have a mobile, or "lite," version of their database for just this purpose. Many are time series databases because much of IoT data is time series data. These databases are like extrinsic databases, but possibly without some potentially unneeded features such as sharding and replication, storage engine configuration, database authentication, encryption/SSL, change streams, server-side Javascript execution, and transactions.

Mobile databases are built for software developers, not end users. Functions that may be needed, but may not be included, include full create, read, update, and delete applications, ACID, the capability to run on multiple platforms and with multiple languages, the ability to move to more centralized databases seamlessly, and true zero maintenance with DBA independence.

Data requirements for IoT devices differ from traditional data requirements due to volume and the speed at which data is collected from various streaming sources, sensors, or generated by algorithms, then passed through IoT edge devices and gateways -- both steps which can compound the demands on a system. Sensors also have greater precision today due to modern transponders' frequencies being higher than before. Edge devices also need enough resources -- memory, persistent storage, and CPU power -- to fully analyze the data on their own while fulfilling the devices' core purpose.

Additionally, edge nodes' physical connectivity is often unpredictable so the architecture must contend with nodes when they are offline. The database management system should be able to adjust its data replication patterns automatically based on various application-defined criteria.

Finally, as with much development, data can be a primary weak link in embedded development. You can have all the data you want placed in real time at the exact spot you want it, but if it's low-quality data, you might as well not.