lematt is Matt's Let's Encrypt Automation

Matt’s Let’s Encrypt Automation

Welcome to Infrastructure Week July 2018! New articles and tools every day this week.

I made a kickass certificate automation platform called lematt and this page is all about it!

What Is It?

lematt is a self-contained certificate management system allowing you to automatically:

  • provision RSA and EC keys
  • generate RSA and EC CSRs (up to 100 domains per certificate)
  • generate and renew RSA and EC LE certificates
  • copy certs and keys to multiple places when renewed
  • reload services based on the domain(s) inside renewed certs
  • run certificate updates as a dedicated certificate-maint user
  • continuously rotate keys and certificates as fast as every 3.5 days
  • end-to-end test your configuration, copy, pre-sign, and post update actions using the LE staging endpoint with isolated test-specific keys, CSRs, and certs so you don’t burn through production rate limits or overwrite production keys and certs with test data.

lematt is designed for multi-key, multi-cert, multi-domain, multi-host, multi-service continual provisioning and renewals.

lematt does not change any part of your system outside of writing new keys, CSRs, and signed certificates. You must specify all copy actions and service reload triggers based on your own system needs.

All you need to get started is already having a web server where LE can discover verification challenges under the URI /.well-known/acme-challenge/.

Why Is It?

Consider this a “paying off technical debt” project. My original LE automation was a 40 line shell script looping over domains to generate RSA keys, CSRs, certs from LE, then copying keys/certs and reloading services. The 40 line shell script worked great for two and a half years, but now it has been upgraded to a 500 650 700 800 870+ line Python program (though, 200 of those are comments) with improved reliability, enhanced functionality, plus general usability across different installations through better config management and stable update triggers.

A goal of lematt is to be relatively simple. Other LE requesting systems like ‘certbot’ are over 64,000 lines of python with manuals exceeding 35 pages, plus they do unexpected things like rewrite web server configs, support 300 combinations of things you’ll never use, and even have options to self-install OS package dependencies to get them running. Seems overkill.

Other systems also don’t seem to be designed around centralized issuance of multi-system usage. They expect you to run one web server on one host and maintain all services locally. Or, they expect you to run independent certificate requests across multiple servers.

lematt lets you have one place to manage all keys and certs and service reloads across your entire environment without needing to wrap layers of shell scripts around unsatisfactory automation tools.

Where is It?

It should be at mattsta/lematt.

Running

Run lematt by giving lematt.py one argument of either --test or --prod along with a config path if you aren’t using the default location of conf/:

lematt.py --config conf/lematt.conf --test

or, if you have all your permissions and users set correctly (as described in Advanced Usage):

sudo -H -u certmaint lematt.py --test

Keys, CSRs, and certs will be written to one of:

  • test mode: conf/test/{key,csr,cert}
  • production mode: conf/prod/{key,csr,cert}

lematt uses the conf/{prod,test}/cert directory to determine when to renew existing certificates, so you don’t remove files from there unless you are abandoning the certificate (or if you want to force an unconditional renewal). Use update actions to copy specific keys and certs from the lematt key and certificate cache to other directories and servers automatically.

Configuring

lematt has three configuration files:

  • lematt.conf describes global options for:
    • how many days before expiration to renew certs
    • optionally, how many days between renewing certs
    • your LE account key location
    • directory to place LE challenge verification files
    • how many bits to use for your RSA keys (default: 2048)
    • which curve to use (default: prime256v1 (also known as secp256r1))
[config]
# Renew existing certificates this number of days before they expire
reauthorizeDays: 15

# Or, renew certs based on how many days old they are:
# (Warning: do not set this lower than 3.5 or you'll hit per-domain rate limits)
# Obviously some numbers are equivalent like:
#   reauthorizeDays: 60
#         equals
#   generateNewCertsAfterDays: 30
# because LE certs have a 90 day lifetime.
# generateNewCertsAfterDays: 7

# Directory on this file system where your web server
# serves the path .well-known/acme-challenge/
# (for _all_ domains you are requesting certs for)
challengeDropDir: /srv/web/challenges/

# Your private key for requesting all certs
# openssl genrsa 4096 > account.key
accountKey: /etc/ssl/private/lets-encrypt-account.key

# Bits for your RSA certificate and which EC curve to use
# (openssl aliases secp256r1 to prime256v1)
# Currently secp256r1/prime256v1 is the only widely supported
# EC usable over the public internet for common browsers.
# Using RSA size larger than 2048 is not recommended because
# it will increase client computation by 6x-8x (encryption isn't
# free! it takes a lot of CPU cycles! also, it'll burn even more
# mobile battery life needlessly) without significantly more security.
keyBitsRSA: 2048
curve: prime256v1

# If alwaysGenerateNewKeys is true, new keys will be generated
# for _every_ certificate renewal, giving you the ability to
# rotate keys as often as you rotate certificates.
# Combine this with 'generateNewCertsAfterDays' to generate completely
# new keys and certs as often as every 3.5 days.
alwaysGenerateNewKeys: no
  • domains describes which domains to manage:
    • each line will generate a new key and new certificate
    • each line must start with a FQDN the LE server can contact
    • you can add SAN/SNI/UCC domains by just listing them on the same line separated by spaces
    • as shorthand, if you just list a subdomain without any ‘.’, the first domain on the line will be appended to the subdomain (e.g. “mysite.com www” will make one SAN cert for “mysite.com” with an altSubjectName field of mysite.com,www.mysite.com)
mydomain.com
dev.mydomain.com
example.com www mail test otherexample.com thirdexample.org
  • actions.conf describes commands to run before and after requesting certs:
    • all commands run under a shell, so globs and variable expansion works as expected
    • section [default] applies to any domain without a specific override
    • section [every] applies to all domains after default or override actions
      • useful for creating rsync actions to upload all modified keys and certs back to a central config management system
    • override sections create custom commands for a list of domain names.
      • name your section anything except [default] or [every]
      • overrides have a space-separated domains entry
      • any domain updated in the domains entry will trigger your override actions
        • domains: mysite.com mail.mysite.com othersite.org
    • actions for [default], [every], and override sections are:
      • update - after a certificate is updated, run these commands
        • update: ["service nginx reload", "ssh mailserver-reload"]
        • DOMAINS_CN in your commands will be replaced with a string of domain CNs (the first name in a cert) for each updated cert
        • DOMAINS_ALL will be replaced with all domains updated (including SAN names)
      • prepare - before requesting the LE cert, run these commands (useful for starting a temporary web server or opening firewall ports temporarily; command will be killed after cert is issued)
        • prepare: ["ssh mailserver-openport http://central.validator.mysite.com"]
        • DOMAIN in your commands will be replaced with the one domain name being prepared
      • uploadCerts - runs when certs are updated or created.
        • uploadCerts: ["rsync -avz CERTS certmaint@mailserver:/etc/ssl/"]
        • CERTS in your commands will be replaced with shell glob patterns (e.g. conf/prod/cert/mydomain.com* conf/prod/cert/myotherdomain.net*)
      • uploadKeys - also runs when certs are updated or created.
        • uploadKeys: ["rsync -avz KEYS certmaint@mailserver:/etc/ssl/private/"]
        • KEYS in your commands will be replaced with shell glob patterns (e.g. conf/prod/key/mydomain.com* conf/prod/key/myotherdomain.net*)
# Commands to run after if no override exists
[default]
update: ["sudo service nginx reload"]

# Note: CERTS and KEYS get expanded to multiple glob patterns.
#       CERTS becomes multiple arguments:
#           conf/cert/mail.domain.com* conf/cert/web.domain.com*
#       Same with KEYS — it can also expand to multiple arguments.
uploadCerts: ["rsync -avz --chmod=F644 CERTS /etc/ssl/"]
uploadKeys: ["rsync -avz --chmod=F640 KEYS /etc/ssl/private/"]

[every]
# Perform post processing uploads to our configuration management system for
# all generated keys and all updated certs
uploadCerts: ["rsync -avz --chmod=F644 CERTS confserv:~/repos/ansible/files/tls/"]
uploadKeys: ["rsync -avz --chmod=F640 KEYS confserv:~/repos/ansible/files/tls/private/"]

# Override commands for specific certs
# You can use an unlimited number of override sections with unique names.
# ---
# In this example, 'mail' and 'dev' are NOT on the same machine
# were lematt.py is running.
# These are all remote actions, remote copies, and remote updates.
# We accomplish this by using HTTP 301 redirects on the remote servers
# so they point back to the machine running lematt.py (because that's
# where the LE http-01 challenges have been dropped).
[override-mail]
# Space separated list of domains (quotes, commas, brackets not required)
domains: mail.matt.sh dev.matt.sh

prepare: ["ssh -t mailmash-forward http://matt.sh"]

uploadCerts: ["rsync -avz --chmod=F644 CERTS mailmash-upload:/etc/ssl/"]
uploadKeys: ["rsync -avz --chmod=F640 KEYS mailmash-upload:/etc/ssl/private/"]

update: ["ssh mailmash-postfix", "ssh mailmash-dovecot"]

And that’s it! The above is all you need to get started, clocking in around 3 pages of docs instead of 35 pages with certbot. Sure, we don’t have an option for DNS challenges yet (so wildcards can’t be requested using lematt right now, but we can always add that later), and lematt won’t mutilate your web server configs, but if you don’t understand your server config files enough to write them yourself, you have bigger problems.

Over all, the tiny act of requesting certificates is just 1/10th of the process of maintaining a live TLS infrastructure. Hopefully lematt can help you get the rest of the way there.

Got Apples?

Want more infrastructure content?

join me over on twitter as I catalog the downfall of civilization when I’m not busy creating technical content lots of people and companies get value out of without ever paying anything for. ¯\_(ツ)_/¯

Advanced

Thirsty for more? This section is just for you.

Let’s cover:

  • common TLS terms
  • running cert updates and service reloads as a non-root user
  • updating remote systems and reloading remote services
  • launching pre-issue commands for remote systems (redirectors, firewall punchers)

We’ll use a recent Ubuntu system for examples below.

Our system layout looks like:

  • certs go in: /etc/ssl/
  • private keys go in: /etc/ssl/private/

Terms

Brief (very brief) one line description per term:

  • LE: Let’s Encrypt - CA issuing free DV certs, subject to rate limits
  • CA: Certificate Authority - an issuer/signer of certificates
  • DV: Domain Validated - just verifies you can control hosting and/or email
  • SAN: subjectAltName - how one certificate supports multiple domain names
  • SNI: Server Name Indication - TLS virtual hosting by giving clients SANs
  • TLS: Transport Layer Security - the “s” in “https” allowing encryption
  • UCC: Unified Communications Certificate - X.509 TLS certificate with SANs
  • X.509: an archaic, but sadly universal, file format for certificates
  • CSR: Certificate Signing Request - how CAs sign public keys and domains
  • PEM: “Privacy-Enhanced E-Mail” - a file format for base64 encoded data
  • RSA: historically standard Internet-wide public key encryption system
  • EC: Elliptic Curve - a more modern public key encryption system
  • OCSP: Online Certificate Status Protocol - realtime CRL; signed responses
  • Staple: include CA-signed OCSP status with your cert when clients connect
  • CRL: Certificate Revocation List - a way to check if certs are revoked

Read Private Keys as Non-Root

Ubuntu ships with a special group called ssl-cert whose entire purpose is to allow non-root users to read private keys.

If your service needs to use TLS certificates, just add the service user to group ssl-cert then point them to the private key in /etc/ssl/private/[domain].pem.

This of course requires all keys in /etc/ssl/private to be owned by group ssl-cert with a mode of 0640.

Now you can read private keys as a non-root user! But, what about creating and updating certs?

Write Private Keys as Non-Root

The private key storage directory /etc/ssl/private is owned by root:ssl-cert with 0770 permissions, so we need to be a user with group ssl-cert to write to the directory.

Create:

  • a new group called certmaint
  • a user called certmaint

Then:

  • make ssl-cert the primary group for user certmaint
    • this is important so uploaded certs can be read by other members of group ssl-cert
  • manually add user certmaint to group certmaint

You may want to chown -R certmaint:ssl-cert /etc/private/ssl/ so your certmaint user can replace any already existing keys too (why? existing keys may be owned by root with only read access for ssl-cert, and if you go to renew the certs, copying as certmaint would fail since certs are probably owned by root with 0640 permissions).

Checkpoint Now your system should look like:

  • /etc/ssl/ - owned by certmaint:ssl-cert with permissions 0755
  • /etc/ssl/*.pem - owned by certmaint:ssl-cert with permissions 0644 (certs are always public)
  • /etc/ssl/private/ owned by certmaint:ssl-cert with permissions 0710 (or 0770 or 0750 depending on how much access you want to hand out)
  • /etc/ssl/private/*.pem owned by certmaint:ssl-cert with permissions 0640
  • now certmaint has permission to create and replace private keys other services can read

Sidenote: ACLs

For more advanced usage, create ACLs with setfacl so multiple users can have read/write access to files instead of being limited to classic user:group semantics.

You can check if ACLs are enabled with:

> tune2fs -l $(mount |grep " on / " |awk '{print $1}') |grep "Default mount"
> mount |grep " on / " |grep acl

(why both? if acl is enabled as a “default” on your FS, it won’t be listed as an enabled mount option, even though it is actually enabled. thanks, linux.)

One of those two commands should return some ACL description. If not, you can:

> mount -o remount,acl /

Then update fstab with acl options as necessary.

Reload Services as Non-Root

Our certmaint user has permission to create and replace private keys, but now it also needs permission to reload services.

Modern distros have /etc/sudoers.d/ so we can just add some files there.

Let’s allow any user in the certmaint group to reload services without a sudo password prompt (if you want to restrict to just user and not group, remove the ‘%’).

Create /etc/sudoers.d/certmaint_reload and allow only the reload action of specific services:

%certmaint ALL = (root) NOPASSWD: /usr/sbin/service postfix reload,
                                  /usr/sbin/service dovecot reload,
                                  /usr/sbin/service haproxy reload,
                                  /usr/sbin/service nginx reload

Checkpoint Now:

Enable rsync Of Certs And Remote Service Reloads

Our remote servers will use the same certmaint user/group and /etc/ssl/private/ setup described previously with the addition of some SSH magic. Create the certmaint user using all the steps above on all remote systems you want to cert-admin, including fixing ownership/permissions in /etc/ssl/ and /etc/ssl/private/. (You are using an idempotent config management system, right? You should be able to script the user creation and permission updates as well as everything below to be easily repeatable across unlimited numbers of systems.)

We’re going to add custom SSH keys for each action we want to perform remotely.

Why so many individual keys? We’ll use authorized_keys to limit each key to only one command when used on a connection.

First, on the lematt certificate downloading host, create keys with ssh-keygen -t ed25519 for each service you want to restart remotely on other servers plus one more for uploading:

for service in postfix dovecot rsync; do
    ssh-keygen -t ed25519 -N '' -C $service -f ~certmaint/.ssh/$service
done

Now create a local authorized_keys file to automatically run a command when a specific key connects:

for service in postfix dovecot; do
    command="command=\"sudo service $service reload\""
    protection=no-port-forwarding,no-x11-forwarding,no-agent-forwarding
    pubkey=$(cat ~certmaint/.ssh/$service.pub)
    printf "$command,$protection $pubkey\n" >> ~certmaint/.ssh/authorized_keys
done

For rsync, just add another line directly to ~certmaint/.ssh/authorized_keys:

command="command=\"/usr/local/bin/ssh_copy_only.sh\""
protection=no-port-forwarding,no-x11-forwarding,no-agent-forwarding
pubkey=$(cat ~certmaint/.ssh/rsync.pub)
printf "$command,$protection $pubkey\n" >> ~certmaint/.ssh/authorized_keys

What’s the ssh_copy_only.sh all about? We can only restrict SSH keys to one exact command, but such a configuration won’t work for rsync or scp. Instead of an exact command, we can use a “check command” to verify if we’re allowed to continue running or not.

Just create /usr/local/bin/ssh_copy_only.sh on all remote hosts you want to rsync (or scp) into with the following contents then make it executable. This will restrict your rsync key to only allow rsync or scp without allowing any other commands or a real login shell.

#!/usr/bin/env bash

# Only allow ssh commands starting with 'scp' or 'rsync'
case $SSH_ORIGINAL_COMMAND in
    scp*)
        $SSH_ORIGINAL_COMMAND ;;
    rsync*)
        $SSH_ORIGINAL_COMMAND ;;
    *)
        echo "Not allowed with this key: $SSH_ORIGINAL_COMMAND" ;;
esac

Now your authorized_keys file is complete (for the sample services provided, adjust to taste), and you can copy it to ~certmaint/.ssh/authorized_keys on all remote systems where you’ve created your certmaint user.

Checkpoint Now you should have:

  • on your lematt certificate download machine:
    • private and public ssh keys in ~certmaint/.ssh/
    • authorized_keys file (describing which keys map to which commands) copied to all machines needing remote certificate management
  • on your remote cert upload machines:
    • A copy of authorized_keys in ~certmaint/.ssh/
    • Your rsync upload restriction script ssh_copy_only.sh with execute permissions

What’s left? There was some sudo up there in your authorized_keys lines, so we need to enable service reload sudo on your remote machines too.

As with your certificate downloading machine, create /etc/sudoers.d/certmaint_reload and allow only the reload action of services you run:

%certmaint ALL = (root) NOPASSWD: /usr/sbin/service postfix reload,
                                  /usr/sbin/service dovecot reload,
                                  /usr/sbin/service haproxy reload,
                                  /usr/sbin/service nginx reload

Checkpoint Now you should be able to:

  • ssh -i ~certmaint/.ssh/nginx certmaint@remotehost and it will automatically reload nginx then disconnect. No commands necessary on your part since the connect itself triggers the authorized_keys command for the given key!

Alternatives

systemd directly with a common reload key

Instead of service for reloading you could obviously use systemctl reload if you are being oppressed by systemd as well.

Also, instead of using individual service reload keys, you could make one “reload remote services” ssh key and give it an authorized_keys command entry of command="sudo systemctl reload $SSH_ORIGINAL_COMMAND", then use ssh -i reload certmaint@remotehost nginx.service.

Any arguments given to ssh are presented to as $SSH_ORIGINAL_COMMAND, and here they will get appended as arguments to sudo systemctl reload (note: but the full exact variable-substituted command line still must be in sudoers too).

Centralize SSH Key-to-Service Mapping

Needing to type ssh -i ~certmaint/.ssh/nginx certmaint@remotehost is ugly and gets repetitive after a while.

On your certificate download machine, you can create key-bound aliases for all your actions in ~certmaint/.ssh/config:

# 'ControlMaster' is a performance optimization allowing sharing of
# sessions on one machine across the same connection
# without needing to re-establish it every time.
Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

Host remotehost
    User certmaint
    HostName 192.168.122.122

Host remotehost-rsync
    IdentityFile ~/.ssh/rsync
    HostName remotehost

Host remotehost-reload-dovecot
    IdentityFile ~/.ssh/dovecot
    HostName remotehost

Host remotehost-reload-postfix
    IdentityFile ~/.ssh/postfix
    HostName remotehost

Host remotehost-reload-nginx
    IdentityFile ~/.ssh/nginx
    HostName remotehost

Host remotehost-forward-challenges
    IdentityFile ~/.ssh/forward-challenges
    HostName remotehost

Now you can just use ssh remotehost-reload-nginx everywhere.

But Wait, There’s More!

Redirect Remote Servers Back To You For LE Challenges

We have one feature left to cover: forwarding remote challenge requests back to the certificate download server.

When you request a certificate (using the default http-01 method lematt supports), LE gives you a file to put on a web server so it can verify you control the server, then LE requests the file from your server to verify you placed it correctly.

For verifying remote hosts, you have two options:

  • copy and load
    • start a webserver on the remote host (if they don’t have one running)
    • copy the LE challenge file to the remote server
    • wait for success
  • forward
    • redirect all remote requests back to your certificate download server
      • LE supports following redirects for challenges

Let’s take the second approach.

lematt ships with a tiny python script called leforward.py which replies to all valid requests with a 301 redirect.

Steps:

  • copy leforward.py to your remote machines at /usr/local/bin/
  • create ssh key for forward-challenges
ssh-keygen -t ed25519 -N '' -C fwd -f ~certmaint/.ssh/forward-challenges
  • add ssh public key to authorized_keys with a command of leforward.py --baseurl
printf "command=\"/usr/local/bin/leforward.py --baseurl \$SSH_ORIGINAL_COMMAND\",no-port-forwarding,no-x11-forwarding,no-agent-forwarding $(cat ~certmaint/.ssh/forward-challenges.pub)\n" >> ~certmaint/.ssh/authorized_keys
  • the actual baseurl argument will be passed on the ssh command line

Now, on your certificate download machine, you can set a lematt override for the domains where remote forwarding is required in actions.conf:

[remoteA]
domains: myother.remoteserver.com third.remoteserver.com

# ssh configuration 'remotehost-forward-challenges' was created in ~/.ssh/config
prepare: ["ssh -t remotehost-forward-challenges http://prod.server.com"]

uploadCerts: ["rsync -avz --chmod=F644 CERTS remotehost-rsync:/etc/ssl/"]
uploadKeys: ["rsync -avz --chmod=F640 KEYS remotehost-rsync:/etc/ssl/private/"]

update: ["ssh remotehost-reload-postfix", "ssh remotehost-reload-dovecot"]

Now, when lematt requests a certificate for myother.remoteserver.com, it will first launch the prepare commands which will redirect http://myother.remoteserver.com/.well-known/acme-challenge/* to http://prod.server.com/.well-known/acme-challenge/*

After the certificate is issued, lematt will kill the prepare commands, which in turn stops the remote web servers too (note: you must run as ssh -t for the remote web server to exit when the ssh session disconnects).

One Caveat

Obviously leforward.py needs to listen on port 80 to redirect http requests, so you can either:

  • set up a sudo entry to run leforward.py as root
  • or, preferred, run/persist:
sysctl net.ipv4.ip_unprivileged_port_start=0
  • Linux 4.11 (2017-04-30) added a tunable allowing any process to listen on any port without the decades old restricted port restriction.

…and that’s it! You are now a certified remote certificate administration professional. Congrats.

Renewals

Now you just need to run your configured lematt setup nightly via cron or by timers.

You can safely run lematt manually as well when you need to add new domains (as long as you remain under the LE production rate limits). lematt will only request new certificates for existing domains when they are within the configured reauthorizeDays expire period (note: adding or removing SAN domains on an existing cert will cause lematt to generate entirely new keys and certs for the new SAN configuration).

You can easily add lematt to cron for your certmaint user or set up systemd timer services with proper User=certmaint configurations.

Conclusion

Basically:

  • stop updating your certs as root
  • stop writing shell scripts around your cert renewal mechanisms
  • use RSA and EC keys and certs for all public facing services

Simple enough?

-Matt — ☁mattsta

personal
howto c (2016)py structure (2024)think you can const?aboutresume