Streamlining real-time data management, time series databases are enhancing performance and storage for IoT and sensor data across industries. How are they shaping the future?
As industries push the limits of scale and speed in the digital age, a new breed of databases is quietly powering the next wave of innovation. For instance, MHI Vestas Offshore Wind Company of Denmark uses VictoriaMetrics to ingest and visualise sensor data from offshore wind turbines. The reason is its efficient storage and ability to backfill. The company is running the cluster version of VictoriaMetrics on Kubernetes, utilising Helm charts to achieve maximum efficiency. With an active time series of 270K, an ingestion rate of 70K samples per second, 850 billion data points, 800 GiB data size on disk and a retention period of three years, VictoriaMetrics is providing a dramatic shift to sensor data.
Another company, Groove X, is engaged in designing and producing robotics solutions. Promisingly, the company needs a monitoring solution for device (robot and charge station) health monitoring. At first, it took over the Prometheus server, and then migrated to Thanos. But it was challenging to manage the Thanos cluster and faced a performance issue (long latency on request). Presently, it feels comfortable with VictoriaMetrics because it features low latency.
These are other eye-catching examples as well.
It is a new dawn for time series data
Data that consists of successive measurements of something over a time interval is time series data. It is also called timestamped data. It is recorded sequentially over time and is indexed by time. With the modernisation of financial trading and advancements in the Internet of Things (IoT), artificial intelligence (AI), and machine learning (ML), time-series databases are becoming increasingly essential.
Stock and cryptocurrency prices fluctuate constantly. Data provided by various sensors in a research lab changes frequently. To manage such changing data and perform analysis on it, we need an expert system for storing and retrieving data.
A time-series database (TSDB) utilises timestamp or time-series data. A time series is an orderly arrangement of data points where each point consists of two parts: a timestamp and a numeric value. A TSDB stores a separate time series for each metric, allowing it to query and graph the values over time.
Time series data can be classified into two categories:
- Regular time series data: They occur at regular intervals of time( also called as metrics). Daily stock prices, quarterly profits, annual sales, weather data, river flow rates, atmospheric pressure, heart rate, and pollution data falls under this.
- Irregular time series data: They occur at irregular intervals of time ( also called events ). ATM withdrawals, account deposits, seismic activity, logins or account registrations, and content consumption are a few examples.
Time series DB: a new way of data life
Timestamped data can be effectively indexed and efficiently written using a time series database . You can query here much faster than you would be using relational or NOSQL DB.
With the unprecedented penetration of IoT devices in our lives, the data generated by IoT devices is increasing every day. May it be the diagnostics of your car, the acoustic data sensed by microphones located deep in the sea, or data delivered by smoke sensors attached to trees in the forests, IoT devices are everywhere.
IoT devices are made to do one thing and one thing only. Capture information through the sensors of the device and send it to the server for storage. As the existing communication protocols were too complex for this kind of lightweight, high-frequency streaming data.
It is pretty obvious that data is being collected at an astonishing and rapidly increasing rate. We’re collecting more data on more systems and across more industries than ever before in human history. Keeping up with that flow of data is one of the major challenges in the IT industry today.
Characteristics of time series database
Important features of a time series database include:
- Efficient data life cycle management: The process of managing the entire lifecycle of data from collection and ingestion to aggregation, processing, and termination is very effective.
- Attractive presentation: The practice of presenting data through flexible queries, transformations, visualisations, and dashboards is very meaningful here.
- Specialised software: Such software utilises purpose-built compression, indexing, and spatial generalisation algorithms that enable users to quickly write, query, and visualise millions of points.
- Database sharding: It is possible in TSDB nodes, which permits greater scaling.
- Database replication: Storing data in multiple instances on multiple nodes is possible.
| Factor | Relational database | Time series database |
| Nature of data | Structured | Unstructured or semi-structured |
| Indexing | Uses B tree | Uses an LSM tree |
| Data Compression | Data compression algorithm is limited | Optimum data compression algorithm |
| Storage Cost | More | Less |
| Schema | Adding or removing columns requires database migration | New fields can be added quickly and easily. |
| Application | Can be used for any type of application | Works exclusively with time series data |
| Examples | MySQL, Oracle, Microsoft SQL Server, PostgreSQL | Influx DB, TimeScale DB, Prometheus, QuestDB, Amazon TimeStream, DataStax, Graphite |
Popular time series databases
InfluxDB
It is an open-source TSDB written in Rust, using Apache Arrow, Apache Parquet, and Apache DataFusion as its foundational building blocks. InfluxDB focuses on providing a real-time buffer for observational data of all kinds (metrics, events, logs, and traces) that is queryable via SQL or InfluxQL, and persisted in bulk to object storage as Parquet files, which can be used by other third-party systems. It can run large data workloads at high volumes globally. It has a series cardinality and high output and can continuously ingest and transform hundreds of millions of time series data per second. It can ingest and join data from millions of sources. It has flexible storage to manage high-fidelity and sampled data.
Timescale DB
It is engineered to handle demanding workloads, like time series, vector, events, and analytics data. It is built on PostgreSQL, with expert support at no extra charge. It can process queries very quickly, taking only 100 ms for a table with approximately 1.4 billion rows. It provides automatic partitioning across time and space (partitioning key), as well as full SQL support.
Prometheus
It makes use of time series data in order to generate ad-hoc graphs, tables, and alerts. Prometheus has multiple modes for visualising data: a built-in expression browser, Grafana integration, and a console template language. Scaling is achieved here by functional sharding and federation.
Graphite
It is an enterprise-ready monitoring tool that runs equally well on cheap hardware or cloud infrastructure. It can track the performance of websites, applications, business services, and networked servers.
With IOT devices increasing at an alarming rate, there is a lot of urgency to develop better infrastructure to handle sensor data. Designed to handle large volumes of data and optimised for query performance, time series databases offer improved performance. Time series analysis can be useful to see how a given asset, security, or economic variable changes over time.
References
- “The Landscape of Time Series Databases” by Kovid Rathee
- “The value of time series data and TSDBs” by Anais Dotis-Georgiou
- InfluxDB by Github.com
- Websites of timescale.com, prometheus.io, and graphite.org
- “Case studies and talks” by docs.victoriametrics.com
Authored By: Vinayak Ramachandra Adkoli. The author holds a BE degree in Industrial Production and has been a lecturer in three different polytechnics for ten years. He is also a freelance writer and cartoonist.
