Using mtqq to create a notification network: mosquitto, mqttwarn, hare, and hared

As you read this post, keep in mind that my particular use case of notification on ssh login is not for everyone. It may not appeal to you. In fact, you might find this to be an absolutely ridiculous thing to do. I respect that. I suggest that somewhere within your network there is at least one type of error condition, one urgent situation, one thing that you would like pushed to your cellphone / pager / etc. It might be a failed HDD for example.

That is what I ask you to keep in mind as you read this. Think of the possibilities.

One last proviso: I elected to distribute my notifications via but there are many services which mqttwarn supports. Pick the ones you like. Have a read of the list of plugins, there are at least 38 others to choose fromt.

The background: it started at $WORK and moved to Twitter

This project started off small. A coworker suggested that we try a login notification system. Each time we logged in, we’d get a notice to our mobile devices. For us, it was an easy solution: add a curl statement to ~/.bash_profile and we would hear about every time our account was used to login.

That curl statement would invoke and we would get a notice on my watch, cellphone, web browser. If someone logged into our accounts, we’d know. We were using NFS for our home directories on each server, so we just had one file to maintain.

Then I started to think if there was another way to do this, other than via shell hooks, something that would also catch scp, etc. I tweeted my idea and goals. Allan Jude suggested PAM. Michael Lucas suggested pam_exec and a shell script.

An evil plan was born. I started playing and learning about pam_exec. The next day, I had played with PAM and created something rather short. It worked. It worked well.

Michael Lucas rightly said:

Me: “pam_exec is scary, avoid it. Especially with shell scripts.”

Dan: “How about I use pam_exec to call a shell script, but add a call to curl(1)?”

Excuse me. I’ll be in the corner, sobbing piteously for the future of humanity.

Enter Jan-Piet MENS with code. He created a small C program which sends off a UDP packet. This code is invoked via /etc/pam.d/sshd, just like my code was, but that is where the similarity stops.

This is where it might sound complex, but it’s not. There are several moving parts, and that’s the way it should be. Small programs, each doing something very well.

I immediately took a liking to it, mostly because of what I could do with MQTT for other things, not just this login notification obsession.

The moving parts

Here is what happens, more or less, when someone logs in with this solution:

  1. successful login occurs. This line in /etc/pam.d/sshd is invoked:
    session    optional     /usr/local/sbin/hare
  2. hare is a small C program. By small, I mean about 100 lines of code. It sends a small udp packet to on port 8035. That packet looks something like this:
    $ sudo tcpdump -A -ni ix0 port 8035
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on ix0, link-type EN10MB (Ethernet), capture size 262144 bytes
    20:16:23.165360 IP > UDP, length 145

    You will notice that is a JSON string. Very flexible.

    This is me logging into dbclone from my MacBook air. What is at hared

  3. That packet is received by hared, a small Go program. The original was a very short Python program. hared composes a message and deposits it into mqtt.
  4. The mqtt implementation I am using is mosquitto. Mosquitto accepts this message and processes it like any other.
  5. mqttwarn, a multiple-purpose mqtt-client, accepts that message, and notifies
  6. A notification arrives on my cellphone and/or web browser informing me of that login.

What do those notifications look like? You can adjust the icon, but here is what I have chosen:

iPhone Notification
This is what the notification looks like on my iPhone. application on iPhone
This is a cropped screen shot of the application on my iPhone.
Apple Watch Notification
This is the notification I saw on my Apple Watch. Web client
This is the web client for notifications

Here is a diagram which outlines what just happened:

Component diagram
The flow of the message as it goes from PAM to

Where do I get this code?

  • hare & hared are on GitHub. I maintain the FreeBSD ports for both hare and hared (written in Go). There is also a python version of hared in FreeBSD but I use the Go version because it has supports TLS.
  • The MQTT broker (that’s the terminology for what Mosquitto does) I am using is net/mosquitto (I love that category for that port name). Mosquitto is an Eclipse Project
  • mqttwarn is a GitHub project available as sysutils/py-mqttwarn in the FreeBSD ports tree.
  • It is recommended by the author that mqttwarn be run via supervisord, available as sysutils/py-supervisor in the FreeBSD ports tree. I follow that advice and also run hared via supervisord.

About the configuration

In this configuration, I am using just one host. Everything is installed on that host. My home installation differs. At home I have one instance each of hared, mqttwarn, mosquitto and supervisord which services all the servers and jails at home. My servers in other other locations run one instance of hared and one mosquitto (which forwards messages to the mosquitto instance at home). This is very flexible.

I do much of the following with an Ansible script, but I’ll show you the manual commands.


hare – code that gets invoked on login. Install on each host/jail you want to monitor

hared – parses udp packet and tosses it into MQTT. Install on one host in your local network. Mine is at listening on port 8035.

mosquitto – Message broker. Install on one host in your local network. Mine is at listening on port 8883.

mqttwarn – Processes the message and initiates the notifications via third-party plugins. Install on one host in your local network. Mine is at

supervisord – Processing management tool. Runs code which can’t background itself. Install it where ever you run hared and/or mqttwarn. Mine is at

Installing hare

To monitor a host for ssh-logins, install hare on that host:

pkg install hare

Add the following entry to /etc/pam.d/sshd:

session         optional             /usr/local/sbin/hare

Installing hared

pkg install hared

This is the configuration file I use:

$ sudo cat /usr/local/etc/hared.ini
verbose    = True
udpHost    =

mqttURI    = tls://
mqttClient = hared
mqttTopic  = logging/hare

mqttUser   = hared
mqttPass   = passwordForHare

mqttCAfile = /usr/local/etc/ssl/cert.pem

Details of the above:

  • udpHost – the IP address which hared should listen on. It should match the address in /etc/pam.d/sshd.
  • mqttURI – the MQTT instance to which hared should send messages. Since I am using TLS, and you should too, I use the FQDN of the host. By convention, mosquitto listens on port 8883 when using TLS.
  • mqttClient – the name used when hared connects to MQTT. It shows up on log entries. It is not used for authorization/authentication.
  • mqttTopic – the MQTT message topic which we will be sending. This will match something in the mqttwarn configuration.
  • mqttUser & mqttPass – the authentication used to connect to mosquitto. We will use this later when configuring mosquitto using mosquitto_passwd.
  • mqttCAfile – the CA file which hared should used for verifying the certificate presented by the MQTT broker (i.e.mosquitto).

Starting hared will be covered under supervisord configuration.

Installing mosquitto

To install mosquitto, issue this command:

pkg install mosquitto

This is the configuration file I use for mosquitto:

$ sudo cat /usr/local/etc/mosquitto/mosquitto.conf
pid_file /var/run/
port 8883
user mosquitto

allow_anonymous false

certfile /usr/local/etc/ssl/
keyfile  /usr/local/etc/ssl/
cafile   /usr/local/etc/ssl/cert.pem


password_file /usr/local/etc/mosquitto/mosquitto.passwd

connection_messages true
log_dest            syslog

The only item worth mentioning: allow_anonymous prevents non-authenticated connections. That is, only clients with valid entries as found in /usr/local/etc/mosquitto/mosquitto.passwd can connect.

When adding using mosquitto_passwd be cautious with the -c option: it will overwrite any existing file.

Here is me adding a password entry for hared:

sudo mosquitto_passwd -c /usr/local/etc/mosquitto/mosquitto.passwd hared
Reenter password: 

Remember to enable mosquitto:

$ sudo sysrc mosquitto_enable="YES"
mosquitto_enable:  -> YES

Then start mosquitto:

$ sudo service mosquitto start
Starting mosquitto.

Not shown above is other logging, handled by supervisord.

Installing mqttwarn

To install mqttwarn:

pkg install mqttwarn

This is my configuration file:

$ sudo cat /usr/local/etc/mqttwarn/mqttwarn.ini
hostname  =
port      = 8883
username  = mqttwarn
password  = mqttwarn_password

ca_certs    = '/usr/local/etc/ssl/cert.pem'
tls_version = 'tlsv1'

libdir    = '/usr/local/lib/python2.7/site-packages/mqttwarn/lib'

loglevel  = debug

functions = '/usr/local/etc/mqttwarn/'

; name the service providers you will be using.
launch	 = file, pushover, log

append_newline = True
targets = {
    'sshd_log'     : ['/var/log/mqttwarn/sshd.log']


targets = {
    'info'   : [ 'info' ],
    'warn'   : [ 'warn' ],
    'crit'   : [ 'crit' ],
    'error'  : [ 'error' ]

targets = file:sshd_log, log:error

targets = {
    'pam'       : ['userkey1', 'appkey1'],

targets = pushover:pam, log:info, file:sshd_log
alldata = moredata()
title: {service} login on {hostname}
filter = logging_filter()
format = User {user} logged into {hostname} from {rhost} at {tstamp}

My notes for the above configuration:

  • tls_version – I use tlsv1 because the other option, sslv3, has known issues.
  • libdir – was needed for earlier versions of

    on FreeBSD, but not after 0.10.0.

  • sshd_log – I also log everything to a file. Yes, this duplicates existing logs on the hosts in question, but I figure, if you can log, you should log. Space is cheap. Time is not. This can be useful when debugging. So I log.
  • userkey1 & appkey1 are obtained from your account.

You will need to set a password for mqttwarn so it can connect to mosquitto:

mosquitto_passwd /usr/local/etc/mosquitto/mosquitto.passwd mqttwarn

Not shown above is other logging, handled by supervisord.

Installing supervisord

Why supervisord? Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. It is ideal for this situation where the code you want to run does not background itself.

I install supervisord, I did this:

pkg install py27-supervisor

These are the setting I have in /etc/rc.conf for supervisord:

$ grep super /etc/rc.conf

supervisord, in it’s current FreeBSD port, expects its configuration file to be at /usr/local/etc/supervisord.conf, but there are a number of configuration files associated with supervisord, I prefer to keep then all in one directory /usr/local/etc/supervisord.

This is the configuration file I use for supervisord:

$ cat /usr/local/etc/supervisord/supervisord.conf
file=/var/run/supervisor/supervisor.sock   ; (the path to the socket file)

logfile=/var/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=debug               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisor/ ; (supervisord pidfile;default
nodaemon=false              ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=200                ; (min. avail process descriptors;default 200)


supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket

files = /usr/local/etc/supervisord/*.inc

Of note:

  • childlogdir – this is in the default file, and I still keep it, but read on for where I put child logs.
  • files – other files to include within this configuration. I will explain this later. Great idea from Jan-Piet MENS.

There are the include files I have installed, one for each of mqttwarn and hared:

[dan@mqtt01:/usr/local/etc/supervisord] $ ls -l *.inc
-rw-r-----  1 root  wheel  154 Apr 10 22:29
-rw-r-----  1 root  wheel  286 Apr 13 19:16
[dan@mqtt01:/usr/local/etc/supervisord] $ 

These files instruct supervisord how to run each of those daemons. Those programs themselves do not background nicely like most programs do. This is why supervisord is used.

The configuration for hared looks like this:

$ cat 
command = /usr/local/bin/hared
user = hared

You might want to mkir /var/log/hared.

The include file for mqttwarn:

$ cat
command = /usr/local/bin/mqttwarn
user    = mqttwarn

As above, you might want to do this:

mkdir /var/log/mqttwarn
chown mqttwarn:mqttwarn /var/log/mqttarn

The mqttwarn port will create the mqttwarn user.

Of note above: MQTTWARNINI and MQTTWARNLOG are environment variables which instruct the configuration and logging files, respectively for mqttwarn.

To start supervisord:

service supervisord start

What does it all look like?

There are all the processes running on my host at home:

[dan@mqtt01:~] $ ps auwwx | egrep -e 'hared|mqtt|mosq'
mosquitto 28869  0.9  0.0  38500 19208  -  SsJ  22:22   0:53.71 /usr/local/sbin/mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf -d
hared     30341  0.0  0.0  15472  9964  -  IJ   22:32   0:00.22 /usr/local/bin/hared
mqttwarn  30342  0.0  0.1 132884 33068  -  SJ   22:32   0:00.95 /usr/local/bin/python2.7 /usr/local/bin/mqttwarn
dan       42814  0.0  0.0  10732  1848  2  R+J  23:32   0:00.00 egrep -e hared|mqtt|mosq
[dan@mqtt01:~] $ 

I also create a hared user myself. Perhaps the port should create that user.

That’s it

That’s it. That’s everything. It should work.

If it doesn’t, here are the main things to check:

  • Is hare installed on the host you are logging into?
  • Is hare configured in /etc/pam.d/sshd on that host?
  • Is the right IP address specified in /etc/pam.d/sshd on that host?
  • Is hared listing on that IP address?
  • Is hared talking to the right MQTT instance? i.e. mosquitto?
  • Is mosquitto running?
  • Is mqttwarn running?

There are several moving parts. Check the logs, make sure each step is working.

For debugging mosquitto, you can run this command see stuff moving into MQTT:

$ mosquitto_sub -v -t 'logging/hare' -u USER -P PASSWORD -p 8883 -h mqtt01  --cafile /usr/local/etc/ssl/cert.pem  --insecure
logging/hare logging/hare {"hostname":"","remote":"","rhost":"","service":"sshd","tst":1523838071,"tty":null,"user":"dan"}

That shows the message going into MQTT.

Here is that same message recorded by hared:

*** /var/log/hared/hared-stdout.log ***

This is the mqttwarn logging:

2018-04-16 00:21:11,062 DEBUG    [mqttwarn.core            ] Message received on logging/hare: {"hostname":"","remote":"","rhost":"","service":"sshd","tst":1523838071,"tty":null,"user":"dan"}
2018-04-16 00:21:11,063 DEBUG    [mqttwarn.core            ] Section [logging/hare] matches message on logging/hare. Processing...
2018-04-16 00:21:11,065 DEBUG    [mqttwarn.core            ] Message on logging/hare going to pushover:pam
2018-04-16 00:21:11,065 DEBUG    [mqttwarn.core            ] New `pushover:pam' job: logging/hare
2018-04-16 00:21:11,065 DEBUG    [mqttwarn.core            ] Message on logging/hare going to log:info
2018-04-16 00:21:11,065 DEBUG    [mqttwarn.core            ] Processor #0 is handling: `pushover' for pam
2018-04-16 00:21:11,065 DEBUG    [mqttwarn.core            ] New `log:info' job: logging/hare
2018-04-16 00:21:11,067 DEBUG    [mqttwarn.core            ] Message on logging/hare going to file:sshd_log
2018-04-16 00:21:11,067 DEBUG    [mqttwarn.core            ] New `file:sshd_log' job: logging/hare
2018-04-16 00:21:11,067 DEBUG    [] *** MODULE=/usr/local/lib/python2.7/site-packages/mqttwarn/services/pushover.pyc: service=pushover, target=pam
2018-04-16 00:21:11,068 DEBUG    [] Sending pushover notification to pam [{'priority': 0, 'message': 'User dan logged into from at 2018-04-16 00:21:11', 'retry': 60, 'expire': 3600, 'title': 'sshd login on'}]....
2018-04-16 00:21:11,070 DEBUG    [urllib3.connectionpool   ] Starting new HTTPS connection (1):
2018-04-16 00:21:11,258 DEBUG    [urllib3.connectionpool   ] "POST /1/messages.json HTTP/1.1" 200 None
2018-04-16 00:21:11,260 DEBUG    [] Successfully sent pushover notification
2018-04-16 00:21:11,300 DEBUG    [mqttwarn.core            ] Job queue has 2 items to process
2018-04-16 00:21:11,300 DEBUG    [mqttwarn.core            ] Processor #0 is handling: `log' for info
2018-04-16 00:21:11,302 DEBUG    [    ] *** MODULE=/usr/local/lib/python2.7/site-packages/mqttwarn/services/log.pyc: service=log, target=info
2018-04-16 00:21:11,303 INFO     [    ] User dan logged into from at 2018-04-16 00:21:11
2018-04-16 00:21:11,303 DEBUG    [mqttwarn.core            ] Job queue has 1 items to process
2018-04-16 00:21:11,303 DEBUG    [mqttwarn.core            ] Processor #0 is handling: `file' for sshd_log
2018-04-16 00:21:11,305 DEBUG    [   ] *** MODULE=/usr/local/lib/python2.7/site-packages/mqttwarn/services/file.pyc: service=file, target=sshd_log
2018-04-16 00:21:11,306 DEBUG    [mqttwarn.core            ] Job queue has 0 items to process

What use is this?

I can think of two other notifications I would want:

  • The power has gone off.
  • Backups are out of space.

I get email notices regarding those issues, but they are urgent enough to be pushed to my phone.

I’m sure you can find interesting uses for MQTT apart from login notifications.

Future posts

In a future post, I will describe how I install these tools on remote hosts. I have only one instance of mqttwarn. Each remote host runs its own hared and mosquitto. That instance of mosquitto sends its messages to my main MQTT instance.

Website Pin Facebook Twitter Myspace Friendfeed Technorati Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top