I’ve started using InfluxDB for storing my sensor data as time series. The main reason for this is that it allows me to use Grafana for analyzing the data. This blog post is an introduction to my setup with these tools on a Raspberry Pi.
All sensor data in my IoT-home setup are transformed to MQTT messages that are published to a self-hosted MQTT broker (mosquitto). I have different services that subscribes to the MQTT topics:
- HomeAssistant. Stores changes of measurements as events. Displays the data and creates automation based on changes in the data.
- A subscriber for MongoDB. Stores all data in a MongoDB database. Used for a home made graph app that probably will be redundant with InfluxDB and Grafana in place.
- A proxy for Adafruit IO. Transforms a subset of the messages to Adafruit IO feeds and publishes them to an external cloud service.
To get data for a data source in Grafana, I simply add an additional MQTT subscriber that stores all messages in time series in an InfluxDB database. My setup will look like this:
An alternative solution for InfluxDB and Grafana is to let HomeAssistant export all state changes to InfluxDB:
I prefer not to use this approach as I don’t want HomeAssistant to be the central hub in my system. I really like HomeAssistant, but I want it to be a subscriber node that no other node has to depend on.
InfluxDB is a time series database. It is optimized for queries in the time domain, e.g. for graphs with data points measured at different points in time. The database contains named “measurements”, e.g. “temperature_from_sensor_1”, and each measurement contains a set of data points that represents samples of the sensor data. A datapoint has a measurement name, a time stamp and a value. An additional set of values and arbitrary key-value tags can be attached to a datapoint (I don’t use this feature in my setup though).
Creating and using a database with measurement, value and time can look like this:
If a time stamp is not provided in the insert statement, InfluxDB will create one from the current time. The time stamps are UTC-based so you have to convert to local time when viewing your data. And if inserting time stamps manually, time should be in UTC.
InfluxDB has a HTTP API that can be used for queries. This is what Grafana uses.
Grafana is a web-based data visualizing tool that can connect to InfluxDB and some other data sources. With Grafana, you can setup custom dashboards, alerts and notifications and you can zoom in- and out of the data sets. Grafana has a web front end that is very responsive and cool looking. It’s open-source and very easy to use and self-host. I’m really impressed with this application.
Installation & setup
I have installed InfluxDB and Grafana on a Raspberry Pi (version 3 model B).
I followed this guide:
Integration with my existing system
As mentioned in the introduction, all sensor data from my IoT-nodes are transformed to MQTT messages that are published to a locally hosted mosquitto broker.
Integrating InfluxDB is as simple as adding an additional MQTT subscriber that takes a received sensor message and store it as a value in an InfluxDB database. I use a Python 3 script (as a service) with paho-mqtt for subscribing to MQTT topics and the influxdb-python library for writing the values to the InfluxDB database.
I use the MQTT topic as measurement type for the data. This lets me get for example “Home/Outdoor/Temperature” as a time series from InfluxDB. In this implementation I set the time property of the data that is uploaded (have to use UTC-time!). If the time property was excluded, InfluxDB would have created a time stamp from the current time.
With Grafana installed, you can log in as an admin and add an InfluxDB database as a new data source where you specify the http settings for InfluxDB’s http api:
Setting the access to “proxy” will make Grafana access the database from the server side. You have to use this when using Grafana outside your LAN (with port forwarding) while InfluxDB is only exposed within the LAN network. The default access is “direct”, which means that the browser front end will fetch the data over http (that’s why it will not work outside your LAN unless both Grafana and InfluxDB has been port-forwarded).
With the data source defined, you can create a new dashboard that can be filled with rows and panels. On each panel you select one or several metrics from your datasource.
You can use the drop downs to build up the query or switch to raw mode where you write the query in plain text. Grafana is really easy to use and as the result of a change is shown immediately one can just play around with the settings to learn how it works.
My current main dashboard contains an overview of climate sensors and detected events:
I have also added system monitoring measurements from my three Raspberry Pi:s (temperature, CPU usage, memory usage, disk usage):
Adapting my system to store the data as time series was easy as all data was available as MQTT topics. With the data in place in InfluxDB, connecting it to Grafana was a breeze. I’m really impressed with the ease-of-use, flexibility and performance of Grafana. I think the Grafana dashboards will be my new favourite tool for analyzing my collected data.
Both InfluxDB and Grafana works great on a Raspberry Pi without using much resources.