My plan for two-stage usage of Certbot when installing web server hosts
Let me start with our problem. When you request TLS certificates through Certbot, you must choose between standalone authentication, where Certbot runs an internal web server to handle the Let's Encrypt HTTP challenge, or webroot authentication, where Certbot puts files in a magic location under the web server's webroot. You can only choose one, which is awkward if you want a single universal process that works on all your hosts, and this choice is saved in the certificate's configuration; it will automatically be used on renewal by default. The final piece is that Apache refuses to start up if there are missing TLS certificates.
All of this creates a problem when installing a host that runs Apache. What you would like to do is perform the install (including the your specific Apache configuration), request the TLS certificates using standalone authentication since Apache can't start yet, and then start Apache and switch to webroot authentication for certificate renewals (so that Certbot can actually renew things now that Apache is using port 80). This would be trivial if Certbot provided a command to change the configured renewal method for a certificate, but as far as I can see they don't. While you can specify the authentication method when you ask for a certificate renewal, this doesn't by itself update the configuration; instead, Certbot only changes the renewal method when you actually renew the certificate.
This means that one way around this would be to request our TLS certificates with standalone authentication, then once Apache was up and running, immediately renew them using webroot authentication purely for the side effect of updating the certificate's configuration. The problem with this (at least in our environment) is that we risk running into Let's Encrypt rate limits, although perhaps not as much as I thought. However, there is a trick we can play to avoid that, because we don't need the first certificate to be trusted. It only exists to bootstrap Apache, and Apache doesn't validate the certificate chain of your certificates. This means that we can ask Certbot to get test certificates instead of real Let's Encrypt ones (using standalone authentication), start Apache, then immediately 'renew' them as real Let's Encrypt certificates using webroot authentication, which will as a side effect update the certificate's configuration.
(Of course in many real situations the actual procedure is 'restore
/etc/letsencrypt from the current production machine'.)
This is not as smooth and fluid a process as
acmetool offers, and
you have to ask for the certificates twice, with different magic
command line options. I'm not certain it's worth writing a cover
script to simplify this a bit, but perhaps it is, since we also
need magic options for registration.
(With appropriate work in the script, you wouldn't even need to list all of the hostnames a second time, just tell it to renew everything as a real certificate now.)
PS: Realizing this trick and working this out makes me feel a fair bit happier about using Certbot. This particular problem was the largest, most tangled obstacle I could see, so I'm glad to have gotten past it.