A web app for IoT data visualization

As described in some earlier posts, I have a setup at home with IoT devices that publish measurement messages to a Raspberry Pi via MQTT. The RPi stores the data in a database and also forwards the messages to a cloud service (Adafruit IO).

In this post I have made a self-hosted data visualization web app that can be accessed from any browser-enabled device.

Overview

For the web front end and back end, I use my current favorite combination of frameworks for Raspberry Pi for web development – Flask and AngularJS. As I will involve a database and some charts, the total setup includes:

• Flask as a microframework for a REST API and for hosting the initial SPA html page.
• pymongo for accessing MongoDB via Python
• angular-chart.js for AngularJS directives with Chart.js
• AngularJS for creating the Single-Page Application

I have hosted the environment on a Raspberry Pi 3 with Raspbian OS, but any OS that has Python support should work.

The complete code for this project can be fetched from https://github.com/LarsBergqvist/IoT_Charts. To try this project out, fetch the code, install the JavaScript dependencies, install the Python dependencies and start the web server. Then, browse to http://<RASPBERRY_IP>:6001/ChartData.

About the project setup

Flask requires a certain folder setup:

• / – the root folder where the Flask web server Python script is placed
• /templates – the folder where html templates are placed (only one html file is needed for a SPA)
• /static – the folder where the AngularJS controllers, CSS and third-party JavaScript dependencies are placed

As Flask will render html pages with Jinja2 that uses {{}} for placeholders, a workaround for AngularJS has to be done by changing the default {{}}-binding syntax to something else, [[]] in my case.

The application uses angular-chart.js that depends on AngularJS and Chart.js. I will use npm for installing these libraries, so the first step is to install npm on the RPi (if not already done):

sudo apt-get install npm

Then, create a folder named static, move to this folder and fetch angular-char.js, AngularJS and Chart.js:

npm install angular-chart.js --save
npm install angular --save
npm install chart.js --save

A subfolder to static called node_modules will be created where the JavaScript libraries are downloaded.

Creating the back end with Flask

Flask is used for creating a REST API. If not installed for Python 3.* already, use pip to install it:

sudo pip3 install flask

As pymongo also will be needed, we need to install that library as well:

sudo pip3 install pymongo

Now, in the root folder, create a new Python script:

	#!/usr/bin/env python3
	from flask import Flask, render_template, jsonify, request
	from datetime import datetime, timedelta
	import sys
	import data_fake
	import data_mongodb
	app = Flask(__name__)
	def get_labels_and_values_for_topic(topic_name, numdays):
	if (numdays < 1):
	numdays = 1
	if app.config['FAKE'] == False:
	repo = data_mongodb.MongoDBRepository
	else:
	repo = data_fake.FakeRepository
	return repo.get_data(repo,topic_name,numdays)
	@app.route("/ChartData")
	def index():
	return render_template('index.html')
	@app.route('/ChartData/api/<string:location>/<string:measurement>')
	def get_measurements_as_labels_and_values(location,measurement):
	numdays = request.args.get('numdays', default=1, type=int)
	topic = "Home/" + location + "/" + measurement
	# Get all measurements that matches a specific topic from the database
	# Fetch data from today and numdays backwards in time
	# The measurements are split into two arrays, one with measurement times (=labels)
	# and one with the actual values.
	labels, values = get_labels_and_values_for_topic(topic,numdays)
	return jsonify({"measurements":{'labels':labels,'values':values}})
	if __name__ == "__main__":
	for arg in sys.argv:
	if arg.lower() == "–fake":
	print("Using fake data")
	app.config['FAKE'] = True
	else:
	app.config['FAKE'] = False
	app.run(host='0.0.0.0', port=6001,debug=False)

view raw

charts_server.py

hosted with ❤ by GitHub

There is a main entry <IP>:6001/ChartData that serves up the initial html template for the Single Page application. The REST API URI:s are built up from <IP>:6001/ChartData/api/<location>/<measurement>. This categorization is based on how I have defined my IoT data (see my previous MQTT IoT posts). For example:

ChartData/api/Outdoor/Temperature - gets the data from the temperature sensor located outdoors.

ChartData/api/GroundFloor/Humidity - gets the data from the humidity sensor on the ground floor.

The data can be fetched from MongoDB (how this data is stored is described in https://larsbergqvist.wordpress.com/2016/06/26/a-self-hosted-mqtt-environment-for-internet-of-things-part-3/) or from a fake repository (just to get some data for the front end development without a complete database).

The repository implementations are defined in two separate files:

	from datetime import datetime, timedelta
	from repository_base import Repository
	class FakeRepository(Repository):
	def get_data(self,topic_name, numdays):
	today=datetime.today()
	# Create some test data
	# Three values from yesterday and three values from today
	dataPoint = (
	(20.1,today-timedelta(1.8)),
	(19.1,today-timedelta(1.4)),
	(22.1,today-timedelta(1.2)),
	(23.3,today-timedelta(0.9)),
	(19.1,today-timedelta(0.5)),
	(30.1,today-timedelta(0.1))
	)
	values = []
	labels = []
	for value,time in dataPoint:
	if (time > (today – timedelta(numdays))):
	values.append(value)
	labels.append(super(FakeRepository,self).date_formatted(time))
	return labels, values

view raw

data_fake.py

hosted with ❤ by GitHub

	import pymongo
	from datetime import datetime, timedelta
	from repository_base import Repository
	class MongoDBRepository(Repository):
	def get_data(self,topic_name, numdays):
	mongoClient=pymongo.MongoClient()
	db=mongoClient.SensorData
	yesterday=datetime.today() – timedelta(numdays)
	cursor = db.home_data.find({"topic":topic_name,"time":{"$gte":yesterday}}).sort("time",pymongo.ASCENDING)
	values = []
	labels = []
	for r in cursor:
	values.append(r['value'])
	labels.append(super(MongoDBRepository,self).date_formatted(r['time']))
	return labels, values

view raw

data_mongodb.py

hosted with ❤ by GitHub

These classes are based on an abstract base class that defines the common interface and the shared base method:

	from datetime import datetime, timedelta
	class Repository:
	"""Abstract base class for the repositories.
	The sub classes should implement the get_data method."""
	def date_formatted(date):
	return date.strftime('%d %b %H:%M')
	def get_data(self,topic_name, numdays):
	raise NotImplementedError( "Should have implemented this" )

view raw

repository_base.py

hosted with ❤ by GitHub

With these back end files in place, we can test the REST API. First start the Flask server:

python3 charts_server.py --fake

The –fake argument makes the implementation use the fake repository so that you don’t need a database with data to test the API. When browsing to http:<IP>:6001/ChartData/api/Outdoor/Temperature, you will get JSON back:

{
measurements: {
labels: [
'09 Jul 22:24',
'10 Jul 08:00',
'10 Jul 17:36'
],
values: [
23.3,
19.1,
30.1
]
}
}

As you can see, the measurements are divided into an array of values and an array of labels. This is because the api provides ChartData that will fit the front end charts easily.

The query parameter numdays can be used for fetching data older than 24 hours, e.g. <IP>:6001/ChartData/api/Outdoor/Temperature?numdays=3.

Creating the front end

The front end consists of a html page in the templates folder and a JavaScript controller in the static folder:

	<!DOCTYPE html>
	<html ng-app="app" lang="en">
	<head>
	<meta charset="utf-8" />
	<title>Sensor data</title>
	<script src="/static/node_modules/angular/angular.js"></script>
	<script src="/static/node_modules/chart.js/dist/Chart.min.js"></script>
	<script src="/static/node_modules/angular-chart.js/dist/angular-chart.min.js"></script>
	<script src="/static/app.js"></script>
	</head>
	<body>
	<h1>Sensor data</h1>
	<div ng-controller="ChartCtrl">
	<label>Number of days:
	<input type="number" min="1" ng-change="numdaysChanged()" ng-model="numdays" />
	</label>
	<div ng-repeat="item in data">
	<h2 ng-bind="titles[$index]"></h2>
	<canvas id="line" class="chart chart-line" chart-data="data[$index]"
	chart-labels="labels[$index]" chart-series="series" chart-options="options"" chart-dataset-override="item.datasetOverride" chart-click="onClick">
	</canvas>
	</div>
	</div>
	</body></html>
	Status

view raw

index.html

hosted with ❤ by GitHub

	var myApp = angular.module('app', ["chart.js"])
	.config(function($interpolateProvider) {
	$interpolateProvider.startSymbol('[[').endSymbol(']]');
	});
	myApp.controller("ChartCtrl", function ($scope,$http) {
	$scope.numdaysChanged = function() {
	requestNewData();
	};
	var lastVal = function(array, n) {
	if (array == null)
	return void 0;
	if (n == null)
	return array[array.length – 1];
	return array.slice(Math.max(array.length – n, 0));
	};
	var getData = function(index,topic,numdays) {
	$http({
	method: 'GET',
	url: "/ChartData/api/" + topic + "?numdays=" + numdays
	}).then(function successCallback(response) {
	values = response.data.measurements.values;
	labels = response.data.measurements.labels;
	$scope.data[index] = [values];
	$scope.labels[index] = labels;
	title = topic + ": " + lastVal(values,1) + " (" + lastVal(labels,1) + ")";
	$scope.titles[index] = title;
	}, function errorCallback(response) {
	});
	};
	$scope.numdays = 1;
	var requestNewData = function() {
	$scope.data = [];
	$scope.labels = [];
	$scope.titles = [];
	$scope.series = ['Sensor'];
	$scope.datasetOverride = [{ yAxisID: 'y-axis-1' }];
	$scope.options = {
	scales: {
	yAxes: [
	{
	id: 'y-axis-1',
	type: 'linear',
	display: true,
	position: 'left'
	}
	]
	}
	};
	getData(0,'Outdoor/Temperature',$scope.numdays);
	getData(1,'GroundFloor/Temperature',$scope.numdays);
	getData(2,'Garage/Temperature',$scope.numdays);
	getData(3,'Outdoor/Humidity',$scope.numdays);
	getData(4,'GroundFloor/Humidity',$scope.numdays);
	getData(5,'Garage/Humidity',$scope.numdays);
	};
	requestNewData();
	});

view raw

app.js

hosted with ❤ by GitHub

The controller makes asynchronous http requests to the REST API and updates the bound $scope properties with the result when the data has been fetched.

With the front end in place and using the fake data, you will get a highly responsive SPA app that display 3 data points for today and yesterday for each sensor that is defined (the same fake is used for each sensor).

With real data, you will get graphs like these, with a data point per recorded sensor value:

For each graph, there is a title that contains the sensor topic plus the latest value and its recording time.

If you want to play with real data, I have made a JSON export from my MongoDB database. It can be downloaded from:

https://github.com/LarsBergqvist/IoT_Charts/tree/master/DataBaseExport

Addendum, March 2017

I’ve started storing my sensor data in an InfluxDB time series database. For the chart app I have made an additional repository that fetches the data from this source. InfluxDB is now the default source for the charts app, but MongoDB or a fake repository can still be selected with startup arguments for the app.

The code is updated in GitHub:

https://github.com/LarsBergqvist/IoT_Charts