Testing email delivery – Dan Langille's Other Diary

The solution

Here is the solution. You can stop reading now:

./check_email_delivery --smtp-server smtp.example.org --mailto dan@example.org \
 --mailfrom dan@example.org \
 --body 'test, please ignore' --imapssl \
 --imap-server imap.example.org --username deltest --password secret

That will test both delivery and receipt.

There. Be gone. You don’t need the rest of this article. Thanks.

Background

As pointed out, this isn’t testing for an email loop. This is testing delivery. I took the term from the phrase email delivery loop as found in the documentation for this code.

Sometimes your mail stops working, and you never notice. That happened to me yesterday. I noticed only because my Delivered queue on my spam defeater showed an incoming message that I had not read. I went looking and found the problem, which had started 2 or 3 hours earlier.

While I do have Nagios checks to verify that mail queues are under control, I do not have a check to verify that mail can be sent.

I first thought about doing something like this a few months back, and I went looking. I failed to find anything, I think, because I was searching for the wrong terms. Then, earlier this week, I was searching for something else and found Check Email Delivery, and there is already a FreeBSD port for it.

I did a quick proof-of-concept for this last night and it worked without much effort. I like that in a project. Make it easy for beginner and it’s easier to get more people using your code. Make it too hard, and people will go elsewhere. Practical examples, my personal blogging mantra, are key to this. Providing a simple easily reproduced example goes a long way to helping newbies.

Side note: Logstash has the right attitude: “If a newbie has a bad time, it’s a bug.” <-- Take note. The concept of Check Email Delivery is:

send an email
see if it’s been received

Sounds simple, right?

That’s what some people said about FreshPorts. Code is easy, sure. Handling exceptions, that’s hard.

This is why I always prefer to use something already written instead of coding it myself. Reuse.

Items of concern

The full list of available utilities are:

check_smtp_send (check SMTP)
check_imap_receive (check IMAP)
check_email_delivery (check mail loop)
check_imap_quota (check IMAP QUOTA)
imap_ssl_cert (download IMAP server’s SSL certs)

I’ve shown check_email_delivery above, and examples of check_smtp_send and check_email_delivery will appear below.

There are a few items of concern here, so please keep them in mind while you doing this.

The code appears abandoned. There have been no updates since Feb 2012 (nearly three years ago). Most of links under Websites at Software Projects no longer exist.
There is potential for password exposure on the command line, if you do it wrong.
The dependencies for this software are not clearly listed, but missing items are clearly announced when running.
You may not want to use your real IMAP account for this.

I don’t mind item 3. One might argue: why install ports you aren’t going to use.

The code seems robust enough, but as with any code, lack of updates might be a concern. I don’t see the author’s repo anywhere, so knowing the history is not possible.

How does it work

First: pkg install net-mgmt/nagios-check_email_delivery

I started by using the examples in the plugin configuration cookbook. I installed from the ports tree but my first attempt failed because of missing dependencies:

$ cd /usr/local/libexec/nagios
$ ./check_smtp_send -H mail.example.org --mailto dan@example.org --mailfrom dan@example.org \
  --body 'hello, world!' --header 'Subject: test message'
Missing perl modules: Net::DNS Email::Address
$

NOTE: I plan to update the FreeBSD port soon. If you do not encounter this error, it’s been fixed.

After installing mail/p5-Email-Address-List and dns/p5-Net-DNS, I was able to proceed:

$ ./check_smtp_send -H mail.example.org --mailto dan@example.org  --mailfrom dan@example.org \
  --body 'hello, world!' --header 'Subject: test message'
SMTP SEND OK - 0 seconds | elapsed=0s;15;30

I checked my incoming mail, and yes, there was the message. OK, step 1: completed.

Now, let’s check the mail from the command line. The recipe for this is:

$ ./check_imap_receive -H imap.example.org --ssl -U dan@example.org -P password -s SUBJECT -s 'test message'

Umm, I do not like having password on the command line. But I found that you could place your password in a environment variable, by doing something like this:

$ ./check_imap_receive -H imap.example.org --ssl -U dan@example.org -P $MYPASS \
  -s SUBJECT -s 'test message'

But don’t do that in the Nagios configuration. I’ll use a entry in the resource configuration. See below.

So here we go:

$ ./check_imap_receive -H imap.example.org --ssl -U dan@example.org -P $MYPASS \
  -s SUBJECT -s 'test message'
IMAP RECEIVE OK - 3 seconds, 1 found, 0 captured, 1 deleted | elapsed=3s found=1messages captured=0messages

I see some spaces could be added to the output, but all looks good.

Testing the loop

I ran the above tests while evaluating the suitability of the tool. Tonight, I’ll be testing the full loop and implementing it in my Nagios installation.

I started looking at check_email_delivery and I soon found a problem which is fixed with the following patch:

$ diff check_email_delivery~ check_email_delivery
64c64
< my $libexec = "/usr/local/nagios/libexec";
---
> my $libexec = "/usr/local/libexec/nagios";

Without that patch, you will encounter this error:

I created a new command:

define command{
command_name    check_email_delivery
command_line    $USER1$/check_email_delivery --smtp-server $ARG1$ --mailto $ARG2$ --mailfrom $ARG3$ --body '$ARG4$' --wait 0 --imapssl --imap-server $ARG5$ --username $ARG6$ --password  $USER2$
}

$USER2$ is user defined variable, usually placed in resource.cfg.

I might adjust the –wait value over time.

I created a new Host Group, Email Testing.

define hostgroup{
        hostgroup_name  email testing
        alias   This host runs standard email testing
}

I create one service for this host group:

define service {
        use local-service
        service_description From cliff to dan
        check_command check_email_delivery!smtp.example.org!dan@example.org!dan@example.org!test, please ignore!imap.example.org!deltest!/usr/local/etc/nagios/MailJailPassword
        hostgroup_name email testing   
        contact_groups admins
}

As you can see, I moved that password to /usr/local/etc/nagios/MailJailPassword.

I added one host to that hostgroup. The test runs just fine:

The incoming email looks like this:

To: dan@example.org
From: dan@example.org
Subject: Nagios Message SMTP smtp.example.org ID 1421116472.
Date: Tue, 13 Jan 2015 02:34:32 +0000 (UTC)
Message-ID: <1421116472.h3aela9m9htgiej7.checksmtpsend@nagios.example.org>

Nagios Email Delivery Plugin
test, please ignore

As you can see, the Subject line has a number on it. Based on observation, that is a unix epoch time value, demonstrated by converting it to a date value:

$ date -r 1421116472
Tue Jan 13 02:34:32 UTC 2015

which corresponds rather nicely to the Date value in the message shown above.

Other stuff you need to know

I found that I could run check_imap_receive only once for a given SUBJECT value. A second attempt would give:

$ ./check_imap_receive -H imap.example.org --ssl -U dan@example.org -P secret \
  -s SUBJECT -s 'test message'
IMAP RECEIVE CRITICAL - found less than 1; 50 seconds, 0 found, 0 captured, 0 deleted | elapsed=50s found=0messages captured=0messages

If I send another email, with a different subject, and then search for that new subject, it succeeds. Then I learned about –nodelete. Adding that option retains the message for another check. However, for testing like this, deleting after success make good sense.

Not using my real email account

I decided not to use my personal mail account for doing the receive check. I will be sending mail to my regular email address, but using filtering, I will redirect the incoming mail to another IMAP account. This will test both my mail filtering and delivery.

I’ve got this idea for testing mail to my FreeBSD and Gmail accounts to verify those delivery routes are working too. I haven’t figured that solution out yet, but if I do, I’ll include it here.

NOTE: You do not want to put your primary Gmail credentials in Nagios, especially considering how important these accounts can be. I created a second Gmail account and forward the incoming test messages from my primary account to this other account. Nagios then checks the second account.

Problems I solved

If you get this, there is an issue with your certificate on the IMAP server. In my case, the certificate was issued by StartSSL, and the ssl_CA specified in dovecot.conf was that of intermediate.startssl.com.pem. What was required instead was ca.pem from http://www.startssl.com/certs/ but your solution will vary.

In addition, I had ssl_cert I had just my host certificate. When resolving this issue I created a server.pem file by concatenating my certificate first, and the StartCom Class 2 Primary Intermediate Server CA cert after it.

Here is the symptom:
$ ./check_imap_receive -H imap.example.org --ssl -U dan@example.org -P secret \ -s SUBJECT -s 'test message' IMAP RECEIVE CRITICAL - Could not connect to imap.example.org port 993: IO::Socket::INET6 configuration failed (if you get this only when using both --ssl and --ssl-ca-file, but not when using just --ssl, the server SSL certificate failed validation) at ./check_imap_receive line 138.

A good tool for checking your certificate configuration is:
openssl s_client -connect imap.example.org:993

I was getting this (I’m showing only one line from the output will would usually see):

Verify return code: 21 (unable to verify the first certificate)

After I fixed it, I saw:

Verify return code: 20 (unable to get local issuer certificate)

Compare your results with what you get from imap.gmail.com.

House trained

I found that not only does check_imap_receive clean up after itself, check_email_delivery does too. In fact, it cleans up things it might have missed on an earlier check. I will explain.

I had two mail checks set up, one for foo@example.com, and another for bar@example.net. These two accounts were hosted on different IMAP servers. Mail to foo was also CC’d to bar. Thus, test messages for both foo and bar wound up in bar‘s mailbox. They did not accumulate. I thought I would have to manually delete them, but check_email_delivery cleaned them all up. I’m not sure what criteria is used for deleting messages.