This is mostly for my future reference, for when this happens the next time. I hope it saves me 10-15 minutes of pondering.
I noticed this problem a few days ago. I was clicking on links in an email from Disney. The hostname didn’t resolve.
I passed it off as: well, they got problems…
It happened again today. Hold on, this isn’t likely. I tried the link again, from my phone, with wifi switched off (so as to not be using my home DNS). The link worked.
It took me a few looks around to figure out was going on.
From my laptop:
[16:59 pro05 dvl ~] % host t.visit.disneydestinations.com Host t.visit.disneydestinations.com not found: 3(NXDOMAIN)
I went to my gateway / firewall host (gw01) and looked around.
named logs
This is what I found in the logs:
17-Aug-2025 21:08:04.050 client @0x37a55c21bc90 10.8.1.200#49498 (t.visit.disneydestinations.com): query: t.visit.disneydestinations.com IN A + (10.55.0.1) *** /var/log/named/default.log *** 17-Aug-2025 21:08:04.050 client @0x37a55c21bc90 10.8.1.200#49498 (t.visit.disneydestinations.com): rpz QNAME NXDOMAIN rewrite t.visit.disneydestinations.com/A/IN via t.visit.disneydestinations.com.rpz
OK, that’s clearly something local. rpz? That sounds familiar.
Nothing in here:
[21:01 gw01 dvl /var/db] % sudo grep -r t.visit.disneydestinations.com adguardhome
I searched my email for rpz, because I’m sure I now this. I found an email thread with Morgan Davis. I implemented dns-blackhole back in April 2025. That led me to crontabs.
Crontab?
Let’s look over here:
[21:01 gw01 dvl /var/db] % cd /usr/local/etc/cron.d [21:01 gw01 dvl /usr/local/etc/cron.d] % ls -l total 10 -rw-r--r-- 1 root wheel 68 2025.01.28 17:12 dma -rw-r--r-- 1 root wheel 441 2025.08.01 12:26 dns-blocker -rw-r--r-- 1 root wheel 557 2025.03.29 16:41 sanoid [21:01 gw01 dvl /usr/local/etc/cron.d] % cat dns-blocker # Ansible managed. Template: /usr/local/etc/ansible/roles/named/templates/dns-blocker.crontab.j2 # # mail any output to `dan', no matter whose crontab this is MAILTO=dan@langille.org PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin # #minute hour mday month wday who command # 15 4 * * * root /usr/local/etc/dns-blackhole/dns-blackhole.sh update 2>&1 | mail -s "update DNS blackhole zone" root
I remember this, I recently enabled it because it was commented out. I’m getting daily emails about it.
What does this file do?
It’s part of https://github.com/morganwdavis/dns-blackhole but it’s not part of a package. I’m not sure why. I think I’ll have to create one. If you look at the source https://github.com/morganwdavis/dns-blackhole/blob/main/dns-blackhole.sh you’ll find references to rpz.
I think I’m onto something. What files are over here:
[21:36 gw01 dvl /usr/local/etc/dns-blackhole] % ls -l total 52 drwxr-xr-x 2 root bind 20 2025.04.28 12:32 OLD/ -rw-r--r-- 1 root wheel 132 2025.08.17 21:15 allowed_hosts -rw-r--r-- 1 root wheel 875 2025.08.17 21:21 dns-blackhole.conf -rw-r--r-- 1 root wheel 1045 2025.04.29 00:28 dns-blackhole.conf.84849.2025-08-17@21:21:02~ -rwxr-xr-x 1 root wheel 4726 2025.04.28 23:14 dns-blackhole.sh -rwxr-xr-x 1 root bind 6476 2025.04.29 00:28 dns-blackhole.sh.mine -rw-r--r-- 1 root wheel 1404 2025.04.28 12:32 local_blocked_hosts
All, look at that, allowed_hosts. Let’s add to that:
[21:36 gw01 dvl /usr/local/etc/dns-blackhole] % cat allowed_hosts # Ansible managed. Template: /usr/local/etc/ansible/roles/named/templates/allowed_hosts.j2 apple.com t.visit.disneydestinations.com
Once I added my entry, I ran the update script:
[21:16 gw01 dvl /usr/local/etc/dns-blackhole] % sudo /usr/local/etc/dns-blackhole/dns-blackhole.sh update Fetching master host list... Optimizing hosts list... Excluding allowed hosts... Building enabled RPZ zone file... Building included zone file... Cleaning up... Stopping named. Waiting for PIDS: 95489. Starting named.
Now it works:
[17:08 pro05 dvl ~] % host t.visit.disneydestinations.com t.visit.disneydestinations.com is an alias for wdpro-mid-prod1-cpgnreq-124527-1158115363.us-west-2.elb.amazonaws.com. wdpro-mid-prod1-cpgnreq-124527-1158115363.us-west-2.elb.amazonaws.com has address 54.68.12.18 wdpro-mid-prod1-cpgnreq-124527-1158115363.us-west-2.elb.amazonaws.com has address 35.155.230.251
Thank you for coming to my TED talk.
