InfluxDB and Grafana for sensor time series

I’ve started using InfluxDB for storing my sensor data as time series. The main reason for this is that it allows me to use Grafana for analyzing the data. This blog post is an introduction to my setup with these tools on a Raspberry Pi.

Introduction

All sensor data in my IoT-home setup are transformed to MQTT messages that are published to a self-hosted MQTT broker (mosquitto). I have different services that subscribes to the MQTT topics:

To get data for a data source in Grafana, I simply add an additional MQTT subscriber that stores all messages in time series in an InfluxDB database. My setup will look like this:

mqtt_environment_ha
My self-hosted MQTT system. 433MHz and z-wave signals are bridged to MQTT messages. WiFi-enabled devices produce MQTT messages directly. Multiple subscribers collect the data.

An alternative solution for InfluxDB and Grafana is to let HomeAssistant export all state changes to InfluxDB:

https://home-assistant.io/components/influxdb/

I prefer not to use this approach as I don’t want HomeAssistant to be the central hub in my system. I really like HomeAssistant, but I want it to be a subscriber node that no other node has to depend on.

InfluxDB

InfluxDB is a time series database. It is optimized for queries in the time domain, e.g. for graphs with data points measured at different points in time. The database contains named “measurements”, e.g. “temperature_from_sensor_1”, and each measurement contains a set of data points that represents samples of the sensor data. A datapoint has a measurement name, a time stamp and a value. An additional set of values and arbitrary key-value tags can be attached to a datapoint (I don’t use this feature in my setup though).

Creating and using a database with measurement, value and time can look like this:

If a time stamp is not provided in the insert statement, InfluxDB will create one from the current time. The time stamps are UTC-based so you have to convert to local time when viewing your data. And if inserting time stamps manually, time should be in UTC.

InfluxDB has a HTTP API that can be used for queries. This is what Grafana uses.

Grafana

Grafana is a web-based data visualizing tool that can connect to InfluxDB and some other data sources. With Grafana, you can setup custom dashboards, alerts and notifications and you can zoom in- and out of the data sets. Grafana has a web front end that is very responsive and cool looking.  It’s open-source and very easy to use and self-host. I’m really impressed with this application.

Installation & setup

I have installed InfluxDB and Grafana on a Raspberry Pi (version 3 model B).

I followed this guide:

https://sites.google.com/site/tgmaxx/energy-enviro-monitoring/influxdb-grafana

Integration with my existing system

As mentioned in the introduction, all sensor data from my IoT-nodes are transformed to MQTT messages that are published to a locally hosted mosquitto broker.

Integrating InfluxDB is as simple as adding an additional MQTT subscriber that takes a received sensor message and store it as a value in an InfluxDB database. I use a Python 3 script (as a service) with paho-mqtt for subscribing to MQTT topics and the influxdb-python library for writing the values to the InfluxDB database.

I use the MQTT topic as measurement type for the data. This lets me get for example “Home/Outdoor/Temperature” as a time series from InfluxDB. In this implementation I set the time property of the data that is uploaded (have to use UTC-time!). If the time property was excluded, InfluxDB would have created a time stamp from the current time.

Grafana configuration

With Grafana installed, you can log in as an admin and add an InfluxDB database as a new data source where you specify the http settings for InfluxDB’s http api:

influxdb_datasource
InfluxDB configuration in Grafana

Setting the access to “proxy” will make Grafana access the database from the server side. You have to use this when using Grafana outside your LAN (with port forwarding) while InfluxDB is only exposed within the LAN network. The default access is “direct”, which means that the browser front end will fetch the data over http (that’s why it will not work outside your LAN unless both Grafana and InfluxDB has been port-forwarded).

With the data source defined, you can create a new dashboard that can be filled with rows and panels. On each panel you select one or several metrics from your datasource.

grafana_metric_definition
Graph panel configuration in Grafana

You can use the drop downs to build up the query or switch to raw mode where you write the query in plain text. Grafana is really easy to use and as the result of a change is shown immediately one can just play around with the settings to learn how it works.

Results

My current main dashboard contains an overview of climate sensors and detected events:

Grafana_dashboard2_small.png

I have also added system monitoring measurements from my three Raspberry Pi:s (temperature, CPU usage, memory usage, disk usage):

grafana_rpi_graphs

Conclusions

Adapting my system to store the data as time series was easy as all data was available as MQTT topics. With the data in place in InfluxDB, connecting it to Grafana was a breeze. I’m really impressed with the ease-of-use, flexibility and performance of Grafana. I think the Grafana dashboards will be my new favourite tool for analyzing my collected data.

Both InfluxDB and Grafana works great on a Raspberry Pi without using much resources.

 

Advertisements

2 thoughts on “InfluxDB and Grafana for sensor time series

Add yours

  1. I am running mosquitto broker on rpi3 and python script from this article also on rpi3. It creates “dht” database in influx but it does not wirite any measurements from topic “out/#”.

    InfluxDB version 1.2.2
    Python 3.4

    Any clues how to deal with it? 🙂

    Like

    1. Hi Marcin! The script in this blog post subscribes to topics starting with “Home”, e.g. “Home/Outdoor/Value” or “Home/MyValue”. Have you modified the script to subscribe to “out/#”? What are your published MQTT topics? What values? The script only stores values that can be converted to floats. You can add additional print statements in the on_message callback function to see how far you get. Here is a Gist with more print tracing https://gist.github.com/a0ceb7c3576fd0c9018456561787377a
      /Lars

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: