A self-hosted MQTT environment for Internet of Things – Part 1

MQTT (MQ Telemetry Protocol) is a lightweight machine-to-machine communication protocol that works on top of TCP/IP and it is very much suited for usage in an Internet-of-Things scenario. MQTT uses the publish-subscribe pattern where the clients subscribe to topics and publish messages through a broker. There are many different framework implementations of MQTT as well as complete cloud services that use MQTT (Adafruit IO, CloudMQTT, ThingMQ etc). The cloud services are nice and easy to use, but if you don’t want your data uploaded to a third party storage, you can host an MQTT environment yourself and have full control of the gathered data. Even if you decide to upload data to the cloud, the local MQTT environment can work as a gateway where data goes out to the Internet only from one single place on the Intranet. This makes it easier to control and change the outgoing data flow.

In this post I will show how I have self-hosted an MQTT environment with the help of a Mosquitto broker running on a Raspberry Pi and let two different ESP8266 microcontrollers publish sensor data through the broker. I will also use the Raspberry Pi as a subscriber for the messages, and these data will be persisted in a MongoDB database. Another subscriber will route the messages to a cloud service.

As this is quite a lot of information, I have divided it up into three parts:

• Part 1 – Mosquitto up-and-running and creating a Python subscriber client with paho-mqtt
• Part 2 – Set up the ESP8266 boards for publishing data with MQTT
• Part 3 – Storing measurements locally and in the cloud

 

Part 1 – Mosquitto up-and-running and creating a Python subscriber client

In this first part, I will install Mosquitto, test it via the command line and then create a Python client with paho-mqqt that subscribes to messages.

I will use the simplest MQTT environment possible without encryption or passwords as all communication happens within my LAN only. MQTT supports different Quality-of-Service levels that guarantees that messages are not lost. For my setup I use QoS 0, i.e. “fire-and-forget”. As I only send non-critical sensor data it does not matter if a measurement is missed or duplicated. For more details on MQTT I really recommend this tutorial by HiveMQ:

http://www.hivemq.com/blog/mqtt-essentials-wrap-up

Installing and testing Mosquitto

I’m using Raspbian as OS on the Raspberry Pi, and to test Mosquitto I first need to install the broker. The default Raspbian package repository has a very old version of Mosquitto. To setup the repository with a newer Mosquitto, follow the instructions in this link:

http://mosquitto.org/2013/01/mosquitto-debian-repository/

sudo apt-get install mosquitto

This will install and activate a mosquitto broker service. You can check that it is running with:

service mosquitto status

When testing the setup it can be better to start mosquitto manually and get the print outs directly to the terminal window. To do this we need to stop the mosquitto service and start it manually with verbose output:

sudo service mosquitto stop
mosquitto -v

To try out Mosquitto via the bash I install the Mosquitto cmd line clients:

sudo apt-get install mosquitto-clients

Now, to test publish and subscribe I start up two ssh sessions to the Pi from my laptop (if you operate the Pi directly via the Raspbian desktop you can just start two terminal windows).

In the first ssh session/terminal, I start a subscriber for messages with topics that match either of two patterns:

mosquitto_sub -h 192.168.1.16 -t 'Hellos/+' -t 'Goodbyes/+'

192.168.1.16 is the IP where the broker is running, i.e. the IP of my Raspberry on my LAN. When running the clients on the same host as the broker, the -h host option is not needed, but I have specified it for clarity.

The mosquitto_sub command starts a subscription to topics where the first parts are “Hellos” or “Goodbyes”. The sub command will echo any messages that are received. You can use the -v option on mosquitto_sub to echo the topic of the message that is received.

In MQTT you can have any number of topic levels separated by forward slashes that together form a topic. The topic levels are case-sensitive and can have any form that (preferably) describes the data being sent. For example:

Home/TopFloor/Temperature
Home/GroundFloor/Temperature
Home/GroundFloor/Humidity

When subscribing to a topic, you can use wildcards:

+ matches any single topic level, like Home/+/Temperature
# matches several topic levels at the end, like Home/#

To see if the broker and the subscriber are working, let’s publish some messages. In the second ssh session/terminal window:

mosquitto_pub -h 192.168.1.16 -t 'Hellos/Pi3' -m 'Hello from Pi3 via MQTT'
mosquitto_pub -h 192.168.1.16 -t 'Hellos/Pi3' -m 'Hello again from Pi3 via MQTT'
mosquitto_pub -h 192.168.1.16 -t 'dummysubject' -m 'Is someone listening to me?'
mosquitto_pub -h 192.168.1.16 -t 'Goodbyes/Pi3' -m 'Goodbye from from Pi3 via MQTT'

If everything is setup correctly, the messages will be echoed in the subscriber terminal.

Publish 1,2 and 4 will be received by the subscriber. Publish 3 uses a subject that the subscriber does not listen to, so it is not pushed from the broker to the subscriber.

Creating a Python subscriber client

To do something useful with the received messages, we can use Python and the paho-mqtt client library. To get it installed for Python 3.*:

sudo pip3 install paho-mqtt

We can check that it works with Python 3 by starting the Python 3 REPL and making an import:

import paho.mqtt.client as mqtt

You use the paho-mqtt client in this way:

• Create a client instance:
  client = mqtt.Client()
• Connect to a broker using one of the connect() functions:
  client.connect([IP of broker])
• Call one of the loop() functions to maintain the connection with the broker. The simplest one would be:
  client.loop_forever()
• Use subscribe() to subscribe to a topic and receive messages:
  client.subscribe([TOPIC])
• Use publish() to publish messages to the broker:
  client.publish([Topic])
• Use disconnect() to disconnect from the broker:
  client.disconnect()

Here is a complete example:

	import paho.mqtt.client as mqtt
	def on_connect(client, userdata, flags, rc):
	print("Connected with result code "+str(rc))
	client.subscribe("Hellos/+")
	def on_message(client, userdata, msg):
	print(msg.topic+" "+str(msg.payload))
	client = mqtt.Client()
	client.on_connect = on_connect
	client.on_message = on_message
	client.connect("192.168.1.16", 1883, 60)
	client.loop_forever()

view raw

subscriber.py

hosted with ❤ by GitHub

The script creates a client instance and attaches handlers for on_connect and on_message callbacks. It then connects to the broker and enters the loop where notifications will cause callbacks to the defined methods. By having the subscription statements in the on_connect method, the client will re-subscribe to the desired topics if the broker goes down and comes back again.

If you start subscriber.py in one terminal window, you can use the mosquitto_pub calls from another terminal to check that the mosquitto publisher, the mosquitto broker and the paho mqtt client are wired correctly.

Next step

The next step is to program the ESP8266 boards so that they publish sensor data to the Mosquitto broker. This is described in the next post:

https://larsbergqvist.wordpress.com/2016/06/24/a-self-hosted-mqtt-environment-for-internet-of-things-part-2/