As you read this post, keep in mind that my particular use case of notification on ssh login is not for everyone. It may not appeal to you. In fact, you might find this to be an absolutely ridiculous thing to do. I respect that. I suggest that somewhere within your network there is at least one type of error condition, one urgent situation, one thing that you would like pushed to your cellphone / pager / etc. It might be a failed HDD for example.
That is what I ask you to keep in mind as you read this. Think of the possibilities.
One last proviso: I elected to distribute my notifications via Pushover.net but there are many services which mqttwarn supports. Pick the ones you like. Have a read of the list of plugins, there are at least 38 others to choose fromt.
The background: it started at $WORK and moved to Twitter
This project started off small. A coworker suggested that we try a login notification system. Each time we logged in, we’d get a notice to our mobile devices. For us, it was an easy solution: add a curl statement to ~/.bash_profile and we would hear about every time our account was used to login.
That curl statement would invoke pushover.net and we would get a notice on my watch, cellphone, web browser. If someone logged into our accounts, we’d know. We were using NFS for our home directories on each server, so we just had one file to maintain.
Then I started to think if there was another way to do this, other than via shell hooks, something that would also catch scp, etc. I tweeted my idea and goals. Allan Jude suggested PAM. Michael Lucas suggested pam_exec and a shell script.
An evil plan was born. I started playing and learning about pam_exec. The next day, I had played with PAM and created something rather short. It worked. It worked well.
Michael Lucas rightly said:
Me: “pam_exec is scary, avoid it. Especially with shell scripts.”
Dan: “How about I use pam_exec to call a shell script, but add a call to curl(1)?”
Excuse me. I’ll be in the corner, sobbing piteously for the future of humanity.
Enter Jan-Piet MENS with code. He created a small C program which sends off a UDP packet. This code is invoked via /etc/pam.d/sshd, just like my code was, but that is where the similarity stops.
This is where it might sound complex, but it’s not. There are several moving parts, and that’s the way it should be. Small programs, each doing something very well.
I immediately took a liking to it, mostly because of what I could do with MQTT for other things, not just this login notification obsession.
The moving parts
Here is what happens, more or less, when someone logs in with this solution:
- successful login occurs. This line in /etc/pam.d/sshd is invoked:
session optional pam_exec.so /usr/local/sbin/hare 10.25.0.10
- hare is a small C program. By small, I mean about 100 lines of code. It sends a small udp packet to 10.25.0.10 on port 8035. That packet looks something like this:
$ sudo tcpdump -A -ni ix0 port 8035 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ix0, link-type EN10MB (Ethernet), capture size 262144 bytes 20:16:23.165360 IP 10.25.0.40.14990 > 10.25.0.10.8035: UDP, length 145 E.......@... 7.. 7. :..c..Z.{"tst":1523823383,"hostname":"dbclone.int.unixathome.org","user":"dan","service":"sshd","rhost":"air01.startpoint.vpn.unixathome.org","tty":null}You will notice that is a JSON string. Very flexible.
This is me logging into dbclone from my MacBook air. What is at 10.25.0.10? hared
- That packet is received by hared, a small Go program. The original was a very short Python program. hared composes a message and deposits it into mqtt.
- The mqtt implementation I am using is mosquitto. Mosquitto accepts this message and processes it like any other.
- mqttwarn, a multiple-purpose mqtt-client, accepts that message, and notifies pushover.net.
- A notification arrives on my cellphone and/or web browser informing me of that login.
What do those notifications look like? You can adjust the icon, but here is what I have chosen:
Here is a diagram which outlines what just happened:
Where do I get this code?
- hare & hared are on GitHub. I maintain the FreeBSD ports for both hare and hared (written in Go). There is also a python version of hared in FreeBSD but I use the Go version because it has supports TLS.
- The MQTT broker (that’s the terminology for what Mosquitto does) I am using is net/mosquitto (I love that category for that port name). Mosquitto is an Eclipse Project
- mqttwarn is a GitHub project available as sysutils/py-mqttwarn in the FreeBSD ports tree.
- It is recommended by the author that mqttwarn be run via supervisord, available as sysutils/py-supervisor in the FreeBSD ports tree. I follow that advice and also run hared via supervisord.
About the configuration
In this configuration, I am using just one host. Everything is installed on that host. My home installation differs. At home I have one instance each of hared, mqttwarn, mosquitto and supervisord which services all the servers and jails at home. My servers in other other locations run one instance of hared and one mosquitto (which forwards messages to the mosquitto instance at home). This is very flexible.
I do much of the following with an Ansible script, but I’ll show you the manual commands.
Glossary
hare – code that gets invoked on login. Install on each host/jail you want to monitor
hared – parses udp packet and tosses it into MQTT. Install on one host in your local network. Mine is at 10.25.0.10 listening on port 8035.
mosquitto – Message broker. Install on one host in your local network. Mine is at 10.25.0.10 listening on port 8883.
mqttwarn – Processes the message and initiates the notifications via third-party plugins. Install on one host in your local network. Mine is at 10.25.0.10
supervisord – Processing management tool. Runs code which can’t background itself. Install it where ever you run hared and/or mqttwarn. Mine is at 10.25.0.10
Installing hare
To monitor a host for ssh-logins, install hare on that host:
pkg install hare
Add the following entry to /etc/pam.d/sshd:
session optional pam_exec.so /usr/local/sbin/hare 10.25.0.10
Installing hared
pkg install hared
This is the configuration file I use:
[defaults] $ sudo cat /usr/local/etc/hared.ini verbose = True udpHost = 10.25.0.10 mqttURI = tls://mqtt01.int.unixathome.org:8883 mqttClient = hared mqttTopic = logging/hare mqttUser = hared mqttPass = passwordForHare mqttCAfile = /usr/local/etc/ssl/cert.pem
Details of the above:
- udpHost – the IP address which hared should listen on. It should match the address in /etc/pam.d/sshd.
- mqttURI – the MQTT instance to which hared should send messages. Since I am using TLS, and you should too, I use the FQDN of the host. By convention, mosquitto listens on port 8883 when using TLS.
- mqttClient – the name used when hared connects to MQTT. It shows up on log entries. It is not used for authorization/authentication.
- mqttTopic – the MQTT message topic which we will be sending. This will match something in the mqttwarn configuration.
- mqttUser & mqttPass – the authentication used to connect to mosquitto. We will use this later when configuring mosquitto using mosquitto_passwd.
- mqttCAfile – the CA file which hared should used for verifying the certificate presented by the MQTT broker (i.e.mosquitto).
Starting hared will be covered under supervisord configuration.
Installing mosquitto
To install mosquitto, issue this command:
pkg install mosquitto
This is the configuration file I use for mosquitto:
$ sudo cat /usr/local/etc/mosquitto/mosquitto.conf pid_file /var/run/mosquitto.pid port 8883 user mosquitto allow_anonymous false certfile /usr/local/etc/ssl/mqtt01.int.unixathome.org.fullchain.cer keyfile /usr/local/etc/ssl/mqtt01.int.unixathome.org.key cafile /usr/local/etc/ssl/cert.pem ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4 password_file /usr/local/etc/mosquitto/mosquitto.passwd connection_messages true log_dest syslog
The only item worth mentioning: allow_anonymous prevents non-authenticated connections. That is, only clients with valid entries as found in /usr/local/etc/mosquitto/mosquitto.passwd can connect.
When adding using mosquitto_passwd be cautious with the -c option: it will overwrite any existing file.
Here is me adding a password entry for hared:
sudo mosquitto_passwd -c /usr/local/etc/mosquitto/mosquitto.passwd hared Password: Reenter password:
Remember to enable mosquitto:
$ sudo sysrc mosquitto_enable="YES" mosquitto_enable: -> YES
Then start mosquitto:
$ sudo service mosquitto start Starting mosquitto.
Not shown above is other logging, handled by supervisord.
Installing mqttwarn
To install mqttwarn:
pkg install mqttwarn
This is my configuration file:
$ sudo cat /usr/local/etc/mqttwarn/mqttwarn.ini
[defaults]
hostname = mqtt01.int.unixathome.org
port = 8883
username = mqttwarn
password = mqttwarn_password
ca_certs = '/usr/local/etc/ssl/cert.pem'
tls_version = 'tlsv1'
libdir = '/usr/local/lib/python2.7/site-packages/mqttwarn/lib'
loglevel = debug
functions = '/usr/local/etc/mqttwarn/my-functions.py'
; name the service providers you will be using.
launch = file, pushover, log
[config:file]
append_newline = True
targets = {
'sshd_log' : ['/var/log/mqttwarn/sshd.log']
}
[config:log]
targets = {
'info' : [ 'info' ],
'warn' : [ 'warn' ],
'crit' : [ 'crit' ],
'error' : [ 'error' ]
}
[test/+]
targets = file:sshd_log, log:error
[config:pushover]
targets = {
'pam' : ['userkey1', 'appkey1'],
}
[logging/hare]
targets = pushover:pam, log:info, file:sshd_log
alldata = moredata()
title: {service} login on {hostname}
filter = logging_filter()
format = User {user} logged into {hostname} from {rhost} at {tstamp}
My notes for the above configuration:
- tls_version – I use tlsv1 because the other option, sslv3, has known issues.
- libdir – was needed for earlier versions of
mqttwarn
on FreeBSD, but not after 0.10.0.
- sshd_log – I also log everything to a file. Yes, this duplicates existing logs on the hosts in question, but I figure, if you can log, you should log. Space is cheap. Time is not. This can be useful when debugging. So I log.
- userkey1 & appkey1 are obtained from your Pushover.net account.
You will need to set a password for mqttwarn so it can connect to mosquitto:
mosquitto_passwd /usr/local/etc/mosquitto/mosquitto.passwd mqttwarn
Not shown above is other logging, handled by supervisord.
Installing supervisord
Why supervisord? Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. It is ideal for this situation where the code you want to run does not background itself.
I install supervisord, I did this:
pkg install py27-supervisor
These are the setting I have in /etc/rc.conf for supervisord:
$ grep super /etc/rc.conf supervisord_enable="YES" supervisord_config="/usr/local/etc/supervisord/supervisord.conf"
supervisord, in it’s current FreeBSD port, expects its configuration file to be at /usr/local/etc/supervisord.conf, but there are a number of configuration files associated with supervisord, I prefer to keep then all in one directory /usr/local/etc/supervisord.
This is the configuration file I use for supervisord:
$ cat /usr/local/etc/supervisord/supervisord.conf [unix_http_server] file=/var/run/supervisor/supervisor.sock ; (the path to the socket file) [supervisord] logfile=/var/log/supervisord.log ; (main log file;default $CWD/supervisord.log) logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB) logfile_backups=10 ; (num of main logfile rotation backups;default 10) loglevel=debug ; (log level;default info; others: debug,warn,trace) pidfile=/var/run/supervisor/supervisord.pid ; (supervisord pidfile;default supervisord.pid) nodaemon=false ; (start in foreground if true;default false) minfds=1024 ; (min. avail startup file descriptors;default 1024) minprocs=200 ; (min. avail process descriptors;default 200) childlogdir=/tmp [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket [include] files = /usr/local/etc/supervisord/*.inc
Of note:
- childlogdir – this is in the default file, and I still keep it, but read on for where I put child logs.
- files – other files to include within this configuration. I will explain this later. Great idea from Jan-Piet MENS.
There are the include files I have installed, one for each of mqttwarn and hared:
[dan@mqtt01:/usr/local/etc/supervisord] $ ls -l *.inc -rw-r----- 1 root wheel 154 Apr 10 22:29 hared.inc -rw-r----- 1 root wheel 286 Apr 13 19:16 mqttwarn.inc [dan@mqtt01:/usr/local/etc/supervisord] $
These files instruct supervisord how to run each of those daemons. Those programs themselves do not background nicely like most programs do. This is why supervisord is used.
The configuration for hared looks like this:
$ cat hared.inc [program:hared] command = /usr/local/bin/hared user = hared stdout_logfile=/var/log/hared/hared-stdout.log stderr_logfile=/var/log/hared/hared-stderr.log
You might want to mkir /var/log/hared.
The include file for mqttwarn:
$ cat mqttwarn.inc [program:mqttwarn] command = /usr/local/bin/mqttwarn user = mqttwarn environment=MQTTWARNINI="/usr/local/etc/mqttwarn/mqttwarn.ini",MQTTWARNLOG="/var/log/mqttwarn/mqttwarn.log" stdout_logfile=/var/log/mqttwarn/mqttwarn-stdout.log stderr_logfile=/var/log/mqttwarn/mqttwarn-stderr.log
As above, you might want to do this:
mkdir /var/log/mqttwarn chown mqttwarn:mqttwarn /var/log/mqttarn
The mqttwarn port will create the mqttwarn user.
Of note above: MQTTWARNINI and MQTTWARNLOG are environment variables which instruct the configuration and logging files, respectively for mqttwarn.
To start supervisord:
service supervisord start
What does it all look like?
There are all the processes running on my host at home:
[dan@mqtt01:~] $ ps auwwx | egrep -e 'hared|mqtt|mosq' mosquitto 28869 0.9 0.0 38500 19208 - SsJ 22:22 0:53.71 /usr/local/sbin/mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf -d hared 30341 0.0 0.0 15472 9964 - IJ 22:32 0:00.22 /usr/local/bin/hared mqttwarn 30342 0.0 0.1 132884 33068 - SJ 22:32 0:00.95 /usr/local/bin/python2.7 /usr/local/bin/mqttwarn dan 42814 0.0 0.0 10732 1848 2 R+J 23:32 0:00.00 egrep -e hared|mqtt|mosq [dan@mqtt01:~] $
I also create a hared user myself. Perhaps the port should create that user.
That’s it
That’s it. That’s everything. It should work.
If it doesn’t, here are the main things to check:
- Is hare installed on the host you are logging into?
- Is hare configured in /etc/pam.d/sshd on that host?
- Is the right IP address specified in /etc/pam.d/sshd on that host?
- Is hared listing on that IP address?
- Is hared talking to the right MQTT instance? i.e. mosquitto?
- Is mosquitto running?
- Is mqttwarn running?
There are several moving parts. Check the logs, make sure each step is working.
For debugging mosquitto, you can run this command see stuff moving into MQTT:
$ mosquitto_sub -v -t 'logging/hare' -u USER -P PASSWORD -p 8883 -h mqtt01 --cafile /usr/local/etc/ssl/cert.pem --insecure
logging/hare logging/hare {"hostname":"dbclone.int.unixathome.org","remote":"10.5.0.14","rhost":"air01.startpoint.vpn.unixathome.org","service":"sshd","tst":1523838071,"tty":null,"user":"dan"}
That shows the message going into MQTT.
Here is that same message recorded by hared:
*** /var/log/hared/hared-stdout.log ***
{"tst":1523838071,"hostname":"dbclone.int.unixathome.org","user":"dan","service":"sshd","rhost":"air01.startpoint.vpn.unixathome.org","tty":null}
{"hostname":"dbclone.int.unixathome.org","remote":"10.5.0.14","rhost":"air01.startpoint.vpn.unixathome.org","service":"sshd","tst":1523838071,"tty":null,"user":"dan"}
This is the mqttwarn logging:
2018-04-16 00:21:11,062 DEBUG [mqttwarn.core ] Message received on logging/hare: {"hostname":"dbclone.int.unixathome.org","remote":"10.5.0.14","rhost":"air01.startpoint.vpn.unixathome.org","service":"sshd","tst":1523838071,"tty":null,"user":"dan"}
2018-04-16 00:21:11,063 DEBUG [mqttwarn.core ] Section [logging/hare] matches message on logging/hare. Processing...
2018-04-16 00:21:11,065 DEBUG [mqttwarn.core ] Message on logging/hare going to pushover:pam
2018-04-16 00:21:11,065 DEBUG [mqttwarn.core ] New `pushover:pam' job: logging/hare
2018-04-16 00:21:11,065 DEBUG [mqttwarn.core ] Message on logging/hare going to log:info
2018-04-16 00:21:11,065 DEBUG [mqttwarn.core ] Processor #0 is handling: `pushover' for pam
2018-04-16 00:21:11,065 DEBUG [mqttwarn.core ] New `log:info' job: logging/hare
2018-04-16 00:21:11,067 DEBUG [mqttwarn.core ] Message on logging/hare going to file:sshd_log
2018-04-16 00:21:11,067 DEBUG [mqttwarn.core ] New `file:sshd_log' job: logging/hare
2018-04-16 00:21:11,067 DEBUG [mqttwarn.services.pushover] *** MODULE=/usr/local/lib/python2.7/site-packages/mqttwarn/services/pushover.pyc: service=pushover, target=pam
2018-04-16 00:21:11,068 DEBUG [mqttwarn.services.pushover] Sending pushover notification to pam [{'priority': 0, 'message': 'User dan logged into dbclone.int.unixathome.org from air01.startpoint.vpn.unixathome.org at 2018-04-16 00:21:11', 'retry': 60, 'expire': 3600, 'title': 'sshd login on dbclone.int.unixathome.org'}]....
2018-04-16 00:21:11,070 DEBUG [urllib3.connectionpool ] Starting new HTTPS connection (1): api.pushover.net
2018-04-16 00:21:11,258 DEBUG [urllib3.connectionpool ] https://api.pushover.net:443 "POST /1/messages.json HTTP/1.1" 200 None
2018-04-16 00:21:11,260 DEBUG [mqttwarn.services.pushover] Successfully sent pushover notification
2018-04-16 00:21:11,300 DEBUG [mqttwarn.core ] Job queue has 2 items to process
2018-04-16 00:21:11,300 DEBUG [mqttwarn.core ] Processor #0 is handling: `log' for info
2018-04-16 00:21:11,302 DEBUG [mqttwarn.services.log ] *** MODULE=/usr/local/lib/python2.7/site-packages/mqttwarn/services/log.pyc: service=log, target=info
2018-04-16 00:21:11,303 INFO [mqttwarn.services.log ] User dan logged into dbclone.int.unixathome.org from air01.startpoint.vpn.unixathome.org at 2018-04-16 00:21:11
2018-04-16 00:21:11,303 DEBUG [mqttwarn.core ] Job queue has 1 items to process
2018-04-16 00:21:11,303 DEBUG [mqttwarn.core ] Processor #0 is handling: `file' for sshd_log
2018-04-16 00:21:11,305 DEBUG [mqttwarn.services.file ] *** MODULE=/usr/local/lib/python2.7/site-packages/mqttwarn/services/file.pyc: service=file, target=sshd_log
2018-04-16 00:21:11,306 DEBUG [mqttwarn.core ] Job queue has 0 items to process
What use is this?
I can think of two other notifications I would want:
- The power has gone off.
- Backups are out of space.
I get email notices regarding those issues, but they are urgent enough to be pushed to my phone.
I’m sure you can find interesting uses for MQTT apart from login notifications.
Future posts
In a future post, I will describe how I install these tools on remote hosts. I have only one instance of mqttwarn. Each remote host runs its own hared and mosquitto. That instance of mosquitto sends its messages to my main MQTT instance.
