Let’s Encrypt’s wildcard certificates ^
Now that Let’s Encrypt can issue wildcard TLS certificates I found some time to look into that.
I already use a Lua script with haproxy which takes care of automatically answering http-01 ACME challenges, but to issue/renew a wildcard certificate you need to answer a dns-01 challenge. A different client/setup would be needed.
dns-01 ACME challenges ^
Most of the clients that support ACME v2 offer a range of integrations for DNS providers, plus a manual mode that prints out the DNS record that you need to add and then waits for you to indicate that you’ve done it. I run my own DNS infrastructure so the thing to do would be RFC2136 dynamic DNS updates.
One wrinkle here is that currently none of my DNS zones have dynamic updates enabled. At the moment I manage them as zone files (some are automatically generated by scripts though). After looking at a few of the client options I found that acme.sh supports an “alias zone”.
Basically, in your main zone you create a CNAME for the challenge record that points at another zone, and then enable dynamic updates in that other zone. The other zone is dedicated for this purpose, so the only updates which will be happening will be for the purpose of answering dns-01 ACME challenges. I made my dynamic zone a sub-zone of my main one:
strugglers.net zone file content ^
These records need to be added to the main zone for this to work.
.
.
.
; sub-zone purely used for dns-01 ACME challenges.
acmesh NS a.authns.bitfolk.co.uk.
NS b.authns.bitfolk.com.
NS c.authns.bitfolk.com.
; Alias the dns-01 challenge record into the dedicated zone.
_acme-challenge CNAME _acme-challenge.acmesh.strugglers.net.
.
.
. |
acmesh.strugglers.net zone file content ^
Initially this just needs to be an empty zone with only SOA and NS records, so this is the entire content of the file.
$ORIGIN .
$TTL 86400 ; 1 day
acmesh.strugglers.net IN SOA a.authns.bitfolk.co.uk. hostmaster.bitfolk.com. (
2018031905 ; serial
14400 ; refresh (4 hours)
7200 ; retry (2 hours)
1209600 ; expire (2 weeks)
43200 ; minimum (12 hours)
)
NS a.authns.bitfolk.co.uk.
NS b.authns.bitfolk.com.
NS c.authns.bitfolk.com. |
DNS server configuration ^
The DNS server needs to know a key by which it will authenticate acme.sh‘s updates, and also needs to be told that the new zone is a dynamic zone. I use BIND, so it goes as follows.
Generate a key for dynamic DNS updates ^
Use the dnssec-keygen command to generate a key suitable for authenticating DNS updates.
$ dnssec-keygen -r /dev/urandom -a HMAC-SHA512 -b 512 -n HOST DDNS_UPDATE |
This creates two files named like Kddns_update.+165+14059.key and Kddns_update.+165+14059.private.
Put the key in the BIND config ^
Look in the private file and take the key from the line that starts “Key:”. Put that in some config file that you will load into your BIND like this:
key "strugglers" {
algorithm hmac-sha512;
secret "Sb8nvwpO8bhiU4haPB+NiJKoMO6vVJumrr29Bj3daSuB8hBoTKoqPKMBKTYLRUv12pbKPwJATgdsU6BtL4Hmcw==";
}; |
The thing in quotes after “key” is a symbolic name for this key and can be anything that makes sense to you. The “secret” is the key from the private file. You can delete the two Kddns_update.+165+14059.* files now.
Put the new zone into the BIND config ^
The config for the zone itself looks something like this:
zone "acmesh.strugglers.net" {
type master;
file "/path/to/acmesh.strugglers.net";
allow-update {
key "strugglers";
};
}; |
Reload the DNS server ^
Once BIND has been reloaded the log file should indicate that the acemsh.strugglers.net zone was loaded correctly, and in my case that triggers DNS NOTIFY to my secondary servers which automatically begin zone transfers.
Check things out with nsupdate ^
At this point it might be worth using the nsupdate command to check that you can do dynamic DNS updates.
Just type the nsupdate line in the shell, the > is a prompt at which you will type the updates you wish to send. We’ll add a trivial TXT record. The -k argument is the path to the file containing the key.
$ nsupdate -k /path/to/strugglers.key -v > server a.authns.bitfolk.co.uk > debug yes > zone acmesh.strugglers.net. > update add foo.acmesh.strugglers.net. 86400 TXT "bar" > show Outgoing update query: ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0 ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0 ;; ZONE SECTION: ;acmesh.strugglers.net. IN SOA ;; UPDATE SECTION: foo.acmesh.strugglers.net. 86400 IN TXT "bar" > send Sending update to 85.119.80.222#53 Outgoing update query: ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 19987 ;; flags:; ZONE: 1, PREREQ: 0, UPDATE: 1, ADDITIONAL: 1 ;; ZONE SECTION: ;acmesh.strugglers.net. IN SOA ;; UPDATE SECTION: foo.acmesh.strugglers.net. 86400 IN TXT "bar" ;; TSIG PSEUDOSECTION: strugglers. 0 ANY TSIG hmac-sha512. 1521454639 300 64 dPndp1/ZyqzmSEn0AKIsGR62HrsplJBhntWioM4oBdPlNXUIAwg7Jwpg DGSM2S3kY+5hfGTleNqwXZrMvnBhUQ== 19987 NOERROR 0 Reply from update query: ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 19987 ;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1 ;; ZONE SECTION: ;acmesh.strugglers.net. IN SOA ;; TSIG PSEUDOSECTION: strugglers. 0 ANY TSIG hmac-sha512. 1521454639 300 64 NfH/78kvq6f+59RXnyJwC6kfFRLGjG6Rh9jdYRId7UjH0jwIbtRVpqCu xx4HToGmlJrDTUqpgbYZq2orUOZlkQ== 19987 NOERROR 0 > [Ctrl-D] |
And to verify it really got added (though the status of NOERROR should be confirmation enough):
$ dig +short -t txt foo.acmesh.strugglers.net "bar" |
That it; you can do dynamic DNS updates.
acme.sh ^
I’m going to assume you’ve installed acme.sh according to one of its supported installation methods. Personally I am not into curl | sh so I:
- Create a system user that can’t log in.
git clonethe source.acme.sh --installit as that user.
acme.sh doesn’t have to be run on the primary DNS server, because it’s going to use a dynamic DNS update to do all the DNS things. It just needs access to the dynamic DNS update key file. Either you can install acme.sh on each host that will need to generate/renew certificates and copy the DNS key there, or else do all the certificate generation/renewal in one place and copy the certificate files around.
However you manage it, make sure that the user you’re going to run acme.sh as can read the dynamic DNS update key file.
Issuing the first wildcard certificate ^
The first time you issue the certificate you need to set NSUPDATE_KEY and NSUPDATE_SERVER in your environment. After the first successful issuance acme.sh will store these variables in its configuration for use in the automated renewals.
$ NSUPDATE_SERVER=a.authns.bitfolk.co.uk NSUPDATE_KEY=/path/to/strugglers.key ./acme.sh --issue -d strugglers.net -d '*.strugglers.net' --challenge-alias acmesh.strugglers.net --dns dns_nsupdate [Mon 19 Mar 09:19:00 UTC 2018] Multi domain='DNS:strugglers.net,DNS:*.strugglers.net' [Mon 19 Mar 09:19:00 UTC 2018] Getting domain auth token for each domain [Mon 19 Mar 09:19:03 UTC 2018] Getting webroot for domain='strugglers.net' [Mon 19 Mar 09:19:03 UTC 2018] Getting webroot for domain='*.strugglers.net' [Mon 19 Mar 09:19:04 UTC 2018] Found domain api file: /path/to/acmesh/dnsapi/dns_nsupdate.sh [Mon 19 Mar 09:19:04 UTC 2018] adding _acme-challenge.acmesh.strugglers.net. 60 in txt "WmenhbXRtenhpNLYLOBjznyHcVvFk-jjxurCVTrhWc8" [Mon 19 Mar 09:19:04 UTC 2018] Found domain api file: /path/to/acmesh/dnsapi/dns_nsupdate.sh [Mon 19 Mar 09:19:04 UTC 2018] adding _acme-challenge.acmesh.strugglers.net. 60 in txt "fwZPUBHijOQkJJaoOF_nIn3Z_FtuVU9R635NDVz_hPA" [Mon 19 Mar 09:19:04 UTC 2018] Sleep 120 seconds for the txt records to take effect |
At this point a DNS update has been crafted and sent so you should see your zone update and zone transfer happen to any secondary servers. If that doesn’t happen within 120 seconds then when Let’s Encrypt tries to verify the challenge it might query a DNS server that doesn’t yet have the record. Your zone transfers need to be reliable.
[Mon 19 Mar 09:21:08 UTC 2018] Verifying:strugglers.net [Mon 19 Mar 09:21:12 UTC 2018] Success [Mon 19 Mar 09:21:12 UTC 2018] Verifying:*.strugglers.net [Mon 19 Mar 09:21:15 UTC 2018] Success [Mon 19 Mar 09:21:15 UTC 2018] Removing DNS records. [Mon 19 Mar 09:21:15 UTC 2018] removing _acme-challenge.acmesh.strugglers.net. txt [Mon 19 Mar 09:21:16 UTC 2018] removing _acme-challenge.acmesh.strugglers.net. txt [Mon 19 Mar 09:21:16 UTC 2018] Verify finished, start to sign. [Mon 19 Mar 09:21:18 UTC 2018] Cert success. -----BEGIN CERTIFICATE----- MIIFETCCA/mgAwIBAgISAz4ZQV27n1FgemVAEhIqiUZnMA0GCSqGSIb3DQEBCwUA MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD . . . NeAmr5I= -----END CERTIFICATE----- [Mon 19 Mar 09:21:18 UTC 2018] Your cert is in /path/to/acmesh/.acme.sh/strugglers.net/strugglers.net.cer [Mon 19 Mar 09:21:18 UTC 2018] Your cert key is in /path/to/acmesh/.acme.sh/strugglers.net/strugglers.net.key [Mon 19 Mar 09:21:18 UTC 2018] The intermediate CA cert is in /path/to/acmesh/.acme.sh/strugglers.net/ca.cer [Mon 19 Mar 09:21:18 UTC 2018] And the full chain certs is there: /path/to/acmesh/.acme.sh/strugglers.net/fullchain.cer |
Examining a certificate ^
Just for peace of mind…
$ openssl x509 -text -noout -certopt no_subject,no_header,no_version,no_serial,no_signame,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux -in /path/to/acmesh/.acme.sh/strugglers.net/strugglers.net.cer
Validity
Not Before: Mar 19 08:21:17 2018 GMT
Not After : Jun 17 08:21:17 2018 GMT
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
BF:C7:8E:F5:87:05:D0:6E:15:AC:7B:37:9F:82:05:C3:E3:11:B7:32
X509v3 Authority Key Identifier:
keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
Authority Information Access:
OCSP - URI:http://ocsp.int-x3.letsencrypt.org
CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
X509v3 Subject Alternative Name:
DNS:*.strugglers.net, DNS:strugglers.net
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
User Notice:
Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/ |
From the Subject Alternative Name we can see it is a wildcard certificate.
Great solution. Great guide. Thanks!
Hi thanks for the tutorial,
but i’m getting an error when doing the “send” command in nsupdate that says SERVFAIL
Reply from update query:
;; ->>HEADER<<- opcode: UPDATE, status: SERVFAIL, id: 25863
;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1
;; ZONE SECTION:
i have no idea what this is about
can you please help me out with this
Fixed this by putting the zone file into the /var/named/dynamic/ directory
Tried issueing the wildcard certificate with the default dnssleep timer but that did not work for me
Doubled it to 240 seconds fixed this problem for me
Thank you for this guide
Yes, I have generally only seen this problem when bind can’t write to the dynamic zone file, either because of permissions or mandatory access control like apparmor or selinux.
Andy – would you please email me at bw@w5gfe.org ? I have written an article which might be considered a followup to this one,
and I would appreciate your comments, and permission to publish my own. 73 de Bill W5GFE
Hi Bill, you don’t need my permission to write an article of your own. 🙂
Mine elaborates on yours, using split horizon bind, and perhaps a few more words. I made use of your suggestions from the article above, almost verbatim though. I’ve credited you with it, and referred to your work explicitly, but I really would like to be sure you don’t regard it as “too close” for comfort. It’s about 8 pages, and in LaTeX. May I send it to you ?
Here is the introduction:
I run a personal server in my home. That machine offers services to the external world through email and web, and also provides services to my own internal network, which includes security cameras, several terminals, an internal wireless arrangement, a couple of dedicated servers, and an internal web with pages whose purpose differs from those pages intended for external consumption.
By necessity, my installation demands “split horizon” domain name service (DNS), a need which is easily accomodated by running “bind” on a server which faces both the external world and the internal network. This presents one “view” of my network to the external world, and another entirely different “view” (ie different IP’s for the same domain) to the internal network.
In keeping with my personal desire to engage in “best practices” (whatever those are!) I wish to employ DNSSEC. I also have no intention of engaging any service for which I have to pay.
Here is a list of what I need:
DNSSEC
split horizon DNS
LetsEncrypt wildcard certificates
Acquiring all of these things at the same time required a surprisingly complicated effort. This article is intended to let you know how it was managed.
where did file /path/to/strugglers.key come from and its contents are unclear
It is the bind configuration snippet for loading a TSIG key and will look something like this:
key "some-key" { algorithm hmac-sha512; secret "…"; };The content of the
secretpart is obtained however you would normally create a TSIG key, for example:I didn’t go into it in depth because creating TSIG keys is a BIND-specific thing generally used for zone transfers and dynamic updates, not specific to ACME DNS-based authentication.
See https://si.w5gfe.org/howto/ for another take
Thanks! I had the same confusion as Cthulhu with the “Put the key in the BIND config” section but figured it out thanks to his question and your response.
Also was getting a TSIG BADKEY error but I just regenerated the key and then it worked.
Overall, super helpful and a lifesaver!
Hi Andy,
I have following error when using dnssec-keygen:
root@ns1:/etc/bind# dnssec-keygen -a HMAC-SHA256 -b 2048 -n HOST letsencrypt_wildcard.
dnssec-keygen: fatal: unknown algorithm HMAC-SHA256
Version: 9.16.1-Ubuntu
Available algorithms:
RSASHA1 | NSEC3RSASHA1 |
RSASHA256 | RSASHA512 |
ECDSAP256SHA256 | ECDSAP384SHA384 |
ED25519 | ED448 | DH
If I use RSASHA then -n HOST gives me “dnssec-keygen: fatal: invalid DNSKEY nametype host”.
Any help appreciated.