Telemetry Data Collection

ThingsBoard provides a rich set of features related to time-series data:

  • Collect data from devices using various protocols and integrations;
  • Store time series data in SQL (PostgreSQL) or NoSQL (Cassandra or Timescale) databases;
  • Query the latest time series data values or all data within the specified time range with flexible aggregation;
  • Subscribe to data updates using WebSockets for visualization or real-time analytics;
  • Visualize time series data using configurable and highly customizable widgets and dashboards;
  • Filter and analyze data using flexible Rule Engine;
  • Generate alarms based on collected data;
  • Forward data to external systems using External Rule Nodes (e.g. Kafka or RabbitMQ Rule Nodes).

This guide provides an overview of the features listed above, and some useful links to get more details.

Data points #

ThingsBoard internally treats time-series data as timestamped key-value pairs. We call single timestamped key-value pair a data point. Flexibility and simplicity of the key-value format allow easy and seamless integration with almost any IoT device on the market. Key is always a string and is basically a data point key name, while the value can be either string, boolean, double, integer or JSON.

Time-series data upload API #

You may use built-in transport protocol implementations:

Most of the protocols above support JSON, Protobuf or own data format. For other protocols, please review “How to connect your device?” guide.

Data visualization #

We assume you have already pushed time-series data to ThingsBoard. Now you may use it in your dashboards. We recommend dashboards overview to get started. Once you are familiar how to create dashboards and configure data sources, you may use widgets to visualize either latest values or real-time changes and historical values. Good examples of widgets that visualize latest values are digital and analog gauges, or cardsCharts are used to visualize historical and real-time values and maps to visualize movement of devices and assets.

You may also use input widgets to allow dashboard users to input new time-series values using the dashboards.

Data storage #

System administrator is able to configure ThingsBoard to store time-series data in SQL (PostgreSQL) or NoSQL (Cassandra or Timescale) databases. Using SQL storage is recommended for small environments with less than 5000 data points per second. Storing data in Cassandra makes sense when you have either high throughput or high availability requirements for your solution.

See SQL vs NoSQL vs Hybrid for more information.

Data retention #

Data retention policy and configuration depends on the chosen storage.

Cassandra supports time-to-live(TTL) parameter for each inserted row. That is why, you may configure default TTL parameter on a system level, using ‘TS_KV_TTL’ environment variable. You may overwrite the default value in the “Save Timeseries” rule node or using “TTL” metadata field of your message. This allows you to optimize storage consumption.

PostgreSQL and Timescale does not support time-to-live(TTL) parameter for each inserted row. That is why, you may only configure periodic time-series data cleanup routine using ‘SQL_TTL_*’ environment variables.

Data durability #

The device that sends message with time-series data to ThingsBoard will receive confirmation once the message is successfully stored into the Rule Engine Queue that is configured for particular device profile.

As a tenant administrator, you may configure processing strategy for the queue. You may configure the queue either to reprocess or ignore the failures of the message processing. This allows granular control on the level of durability for the time-series data and all other messages processed by the rule engine.

Rule engine #

The Rule Engine is responsible for processing all sorts of incoming data and event. You may find most popular scenarios of using attributes within rule engine below:

Generate alarms based on the logical expressions against time-series values

Use alarm rules to configure most common alarm conditions via UI or use filter nodes to configure more specific use cases via custom JS functions.

Modify incoming time-series data before they are stored in the database

Use message type switch rule node to filter messages that contain “Post telemetry” request. Then, use transformation rule nodes to modify a particular message.

Calculate delta between previous and current time-series value

Use calculate delta rule node to calculate power, water and other consumption based on smart-meter readings.

Fetch previous time-series values to analyze incoming telemetry from device

Use originator telemetry rule node to enrich incoming time-series data message with previous time-series data of the device.

Fetch attribute values to analyze incoming telemetry from device

Use enrichment rule nodes to enrich incoming telemetry message with attributes of the device, related asset, customer or tenant. This is extremely powerful technique that allows to modify processing logic and parameters based on settings stored in the attributes.

Use analytics rule nodes to aggregate data for related assets

Use analytics rule nodes to aggregate data from multiple devices or assets.

Useful to calculate total water consumption for the building/district based on data from multiple water meters.