named: error sending response: host unreachable

Every day for weeks I’ve been seeing these entries in my logs:

May 12 03:03:36 toiler named[89425]: client 10.55.0.117#62049: error sending response: host unreachable
May 12 03:03:51 toiler named[89425]: client 10.55.0.117#57916: error sending response: host unreachable
May 12 03:03:58 toiler named[89425]: client 10.55.0.117#57916: error sending response: host unreachable
May 12 03:04:14 toiler named[89425]: client 10.55.0.117#33183: error sending response: host unreachable
May 12 03:04:21 toiler named[89425]: client 10.55.0.117#33183: error sending response: host unreachable
May 12 03:04:36 toiler named[89425]: client 10.55.0.117#32906: error sending response: host unreachable
May 12 03:04:43 toiler named[89425]: client 10.55.0.117#32906: error sending response: host unreachable
May 12 03:04:58 toiler named[89425]: client 10.55.0.117#23841: error sending response: host unreachable
May 12 03:05:05 toiler named[89425]: client 10.55.0.117#23841: error sending response: host unreachable
May 12 03:05:20 toiler named[89425]: client 10.55.0.117#41946: error sending response: host unreachable
May 12 03:05:27 toiler named[89425]: client 10.55.0.117#41946: error sending response: host unreachable
May 12 03:05:43 toiler named[89425]: client 10.55.0.117#35243: error sending response: host unreachable
May 12 03:05:50 toiler named[89425]: client 10.55.0.117#35243: error sending response: host unreachable
May 12 03:06:05 toiler named[89425]: client 10.55.0.117#47116: error sending response: host unreachable
May 12 03:06:12 toiler named[89425]: client 10.55.0.117#47116: error sending response: host unreachable
May 12 03:06:27 toiler named[89425]: client 10.55.0.117#64226: error sending response: host unreachable
May 12 03:06:34 toiler named[89425]: client 10.55.0.117#64226: error sending response: host unreachable
May 12 03:06:49 toiler named[89425]: client 10.55.0.117#52291: error sending response: host unreachable
May 12 03:06:56 toiler named[89425]: client 10.55.0.117#52291: error sending response: host unreachable
May 12 03:07:11 toiler named[89425]: client 10.55.0.117#64713: error sending response: host unreachable
May 12 03:07:12 toiler named[89425]: client 10.55.0.117#15531: error sending response: host unreachable
May 12 03:07:12 toiler named[89425]: client 10.55.0.117#49433: error sending response: host unreachable
May 12 03:07:12 toiler named[89425]: client 10.55.0.117#50315: error sending response: host unreachable
May 12 03:07:18 toiler named[89425]: client 10.55.0.117#64713: error sending response: host unreachable
May 12 03:07:19 toiler named[89425]: client 10.55.0.117#15531: error sending response: host unreachable
May 12 03:07:19 toiler named[89425]: client 10.55.0.117#49433: error sending response: host unreachable
May 12 03:07:19 toiler named[89425]: client 10.55.0.117#50315: error sending response: host unreachable

I could ignore them, but that’s not in my nature. I want to know why this is occurring and fix it.

Attempting to find the source, I started a tcpdump on the host:

$ sudo tcpdump -ni em0 host 10.55.0.117
Password:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:41:34.032681 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:41:37.232692 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:41:40.432692 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:41:43.632693 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:41:46.832693 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:41:53.032703 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:42:05.232680 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
11:42:29.432680 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
12:11:31.034674 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
12:11:34.034692 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
12:11:37.234691 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
12:11:40.434684 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28
12:11:43.634689 ARP, Request who-has 10.55.0.117 tell 10.55.0.74, length 28

That wasn’t much help. I think I should have logged it to a file.

After seeing the log messages again today, and while a little bit of unexpected time on my hands, I think I figured it out.

I had a jail:

   JID  IP Address      Hostname                      Path
    36  10.55.0.117     ansible-test                  /usr/jails/ansible-test

The jail was largely unused.

The problem: no such IP address was allocated to the jail host:

[dan@knew:~] $ ifconfig | grep 10.55.0.117
[dan@knew:~] $

I stopped the jail and deleted it:

[dan@knew:~] $ sudo ezjail-admin stop ansible-test
Stopping jails: ansible-test.
[dan@knew:~] $ sudo ezjail-admin delete -w ansible-test
[dan@knew:~] $

I hope that’s the cause. Otherwise, I have more detective work to do.