In the interests of maintaining Michael W Lucas in the lifestyle to which he has become accustomed, I am creating this blog post. Although Mr Lucas was the first to post, he is not solely to blame for my burdensome workload.
Jan-Piet Mens and Evan Hunt also have much to answer for. Their misdeeds include mentioning newer BIND tools which necessitated an update to an older blog post.
The worst of them all, however, is John-Mark Gurney, for he is the one who pointed out a more restrictive method granting TXT update permissions. That useful information is what has prompted me to write this blog post, early on a cold Friday winter morning. I was also told to mention Allan Jude.
Well, that, and that my Let’s Encrypt challenges failed last night because I implemented that change without any testing of the permissions.
In this post:
- bind916-9.16.6_1
- 12.2-RELEASE-p1
- acme.sh-2.8.7
It is assumed you are familiar with Let’s Encrypt, dns-01 challenges, and BIND.
Background
I use a centralized location to generate Let’s Encrypt certificates. The process uses DNS-01 challenge to prove I control the domains for which the certificates are being generated.
At the root of my DNS structure is a hidden DNS master (is it not listed in any domain and has no public IP addresses and is accessible only via VPN), aptly named dns-hidden-master. This server runs BIND and the named configuration grants permission for a nsupdate key to modify TXT records.
This blog post will show the changes made to the configuration so that one only specific TXT record can be modified in each domain. It will also show how I messed up and broken my certificate upgrades and how I fixed it.
The change
This change is the first non-trivial change to this infrastructure since it was implemented in mid-2017. What could go wrong?
This is the change recommedation:
@@ -204,7 +204,7 @@
type master;
file "zones/langille.org.db";
allow-transfer { AllowZoneTransfer; };
- update-policy { grant certs.int.unixathome.org zonesub TXT;
+ update-policy { grant certs.int.unixathome.org. name _acme-challenge.langille.org TXT;
grant dan.dns.hidden.master zonesub any;
grant git.langille.org self git.langille.org. A; };
notify yes;
Line 5 is the original grant, and allows the key (certs.int.unixathome.org, not shown) to update any TXT record in this domain.
Line 6 restricts this key to only the _acme-challenge.langille.org record.
I like the difference.
The consequences
As I went to bed last night, I checked my email.
From: noc@example.org
To: dan@example.org
Content-Type: text/plain; charset=utf-8
Message-Id: <20201218031901.86BC04B4E3@certs.int.example.org>
Date: Fri, 18 Dec 2020 03:19:01 +0000 (UTC)
Error certs:
bin.langille.org
bsdcan.com
devgit.freshports.org
fedex.int.unixathome.org
[snip]
unixathome.com
wwwgit.freshports.org
In total, it was all 15 certificates up for renewal that night which failed.
Checking the other certificate related email, I found:
update failed: REFUSED [Fri Dec 18 03:18:03 UTC 2020] error updating domain [Fri Dec 18 03:18:03 UTC 2020] Error add txt for domain:_acme-challenge.bin.langille.org [Fri Dec 18 03:18:03 UTC 2020] Please check log file for more details: /var/log/acme.sh.log [Fri Dec 18 03:18:03 UTC 2020] Error renew bin.langille.org.
This makes me think it is DNS.
Looking in /var/log/acme.sh.log as instructed, I found this:
[Fri Dec 18 03:18:03 UTC 2020] adding _acme-challenge.bin.langille.org. 60 in txt "8G5oBjEfKnrgiajj6Ly83TN5Rir7sbCHGvheaHRqmyU" [Fri Dec 18 03:18:03 UTC 2020] error updating domain [Fri Dec 18 03:18:03 UTC 2020] Error add txt for domain:_acme-challenge.bin.langille.org
Next, let’s check the named logs, specifically, my /var/log/named/queries.log file.
18-Dec-2020 03:18:03.237 client @0x83185e168 10.55.0.112#61976/key certs.int.unixathome.org (_acme- challenge.bin.langille.org): query: _acme-challenge.bin.langille.org IN SOA -S (10.55.0.53)
It took me a while, but I soon realized my grants were too restrictive.
I’m allowing _acme-challenge.langille.org but the challenge is for _acme-challenge.bin.langille.org.
I don’t need one GRANT per zone/domain.
I need one GRANT per certificate.
Getting a list of certificates from acme.sh
This sounds very straight forward.
I logged into my certs server where I run acme.sh, and became the acme user.
[dan@certs:~] $ sudo su -l acme
$ bash
[acme@certs ~]$ ls certs | wc -l
164
[acme@certs ~]$
That’s 164 certificates. That’s 164 grants I need to add into my zone files…
But wait, there’s more
These certificates can have more than one host in them, for example:
- langille.org
- www.langille.org
I have to look inside the cert. The directory listing is not sufficient.
Here is how I started:
[acme@certs ~]$ openssl x509 -text -in ~/certs/langille.org/langille.org.cer | grep -i dns
DNS:langille.org, DNS:www.langille.org
[acme@certs ~]$
Challange records will be required for host of those names.
The script
I created list-certs.
I grabbed the output into a file:
[acme@certs ~]$ ~/bin/list-certs | sort -u > certs-list [acme@certs ~]$
I copied that file off to my laptop, and renamed it to certs. I wanted to create a semi-automatic way to get all the hostnames used in certificates for a given zone file.
An example:
[dan@air01:~] $ grep -i langille.org ~/Documents/certs | sort -u | xargs -n 1 -I % echo " grant certs.int.unixathome.org. name _acme-challenge.% TXT;"
grant certs.int.unixathome.org. name _acme-challenge.bin.langille.org TXT;
grant certs.int.unixathome.org. name _acme-challenge.dan.langille.org TXT;
grant certs.int.unixathome.org. name _acme-challenge.git.langille.org TXT;
grant certs.int.unixathome.org. name _acme-challenge.langille.org TXT;
grant certs.int.unixathome.org. name _acme-challenge.www.langille.org TXT;
Then I can copy/paste that right into the zone file.
20 minutes later, which isn’t that long when you’re adding so many records, I had completed the changes and named was restarted without error.
By that time, 11:06 PM (04:06:20 UTC), acme.sh had already been run and produced errors, just like last night.
I changed the crontab and waited.
It’s running now… Waiting.
Done. Only two certs failed.
Let’s just run again and see. I checked the zone files for those two names, and it seemed fine.
Only one failed that time.
Run it again!
I think I should bump my pause time between add a TXT record and waiting.
Ahh, the last domain was an error in the grant:
- update-policy { grant certs.int.unixathome.org. name _acme-challenge.pgcon.pgcon.net TXT; };
+ update-policy { grant certs.int.unixathome.org. name _acme-challenge.pgcon.net TXT; };
Not sure about this
I’m not sure about this.
Each time I want a new cert, I need to first update my grants.
Some domains now have 80 new TXT records. I’m guessing the volume won’t be an issue.
Certs are public record, but now they are all conveniently available from my zone file.
What about delegation
Instead of having to add a new GRANT clause for each certificate creation, what about delegating all this? Let’s pretend I am going to use the dns-01.example.org domain for all DNS-01 challenges. How do I configure that?
Surprise! I’ve already written about this in early 2019 and I forgot about it. Please allow me to rephrase that here.
Let’s say I am requesting a new certificate for www.langille.org, langille.org, and bin.langille.org, all in one cert. I will need the following CNAME records in the langille.org zone file:
_acme-challenge.www.langille.org 600 CNAME _acme-challenge.dns-01.example.org. _acme-challenge.langille.org 600 CNAME _acme-challenge.dns-01.example.org. _acme-challenge.bin.langille.org 600 CNAME _acme-challenge.dns-01.example.org.
This means I have just one challenge key: _acme-challenge.dns-01.example.org.
Everything points at it.
I have just one grant, this time in the dns-01.example.org zone file:
update-policy { grant certs.int.unixathome.org. name _acme-challenge.dns-01.example.org. zonesub TXT; };
I can either do:
- lots of GRANTs
OR
- lots of CNAMES
- one GRANT
Even with delegation, I still need to first add a new record to the zone file for that new hostname.
I have some more pondering to do. In that original delegation post, I talked about the various security considerations and zone / churn.
Please attribute blame to those mentioned at the top of this post.
