You don't have to authorize a machine for Let's Encrypt from the machine
A commentator on yesterday's entry brought up the issue of authorizing internal-only machines, ones that are in DNS but that aren't otherwise reachable from the Internet. Although we haven't actually done this, in general it's possible to do Let's Encrypt's authorization for a particular machine on an entirely different machine, even without using the DNS-based authorization method. All you need is that HTTP requests from the Internet go somewhere where you can handle them in something you control.
If the internal host has a public IP, this is going to take a firewall with some redirection rules (and a suitable other host). But you probably have that already. If the internal host has a private IP address, you probably have 'split horizon' DNS so in your Internet-visible DNS you can assign it a public IP that goes to the suitable other host. As far as I know, most Let's Encrypt clients are perfectly happy in this situation; they don't try to check that the host you're running them on is the host <X> that you're requesting a certificate for.
(If you're unlucky enough to have private IP addresses in public DNS (which can happen for odd reasons), well, then you're out of luck for that host.)
This does leave you with the job of transporting the new TLS certificate to the internal host and handling any daemon notifications needed there, but there are lots of solutions for that. 'Propagate file to host <X> and do something if it's changed' is not hard to automate and generally there's a lot of already mature solutions for it (some of which you may already be using). Some Let's Encrypt clients let you run custom scripts on 'certificate updated' events, so you could use this to immediately push the new certificate to the target host.
In the specific case of acmetool, you have a lot of options if you're willing to do some scripting. Acmetool supports running scripts to handle both challenges and 'certificate updated' events. If you want to run acmetool on your internal host, you could have it push the HTTP challenge files to the bastion host that will expose them to Let's Encrypt; if you want to run it on the bastion host, you could have it propagate the new TLS certificates to the internal host either directly or indirectly (by storing them into some internal data store, which the internal system then pulls from).
Sidebar: Clever tricks with the ACME protocol
As I found out, Let's Encrypt's ACME protocol splits up authorizing machines from issuing certificates. This means that it's technically possible to authorize a host from one machine (say, your bastion machine or your DNS server) and then later obtain a certificate for that host from a second machine (say, the internal machine itself, provided it can talk to the Let's Encrypt servers). The two machines involved have to use a common Let's Encrypt account in order to share the authorization, but that's just a matter of having the same account information and private keys on both (although this has some security implications).
However, as far as I know clients don't generally support performing these steps separately, either doing only authorization and then stopping or doing certificate requests and aborting if Let's Encrypt tells you that it requires authorization. An ideal client for this would also track authorization and certificate timeouts separately, so your bastion host or DNS server could run something to make sure that all authorizations were current and then internal hosts would never wind up reporting 'need authorization' errors.
(You might also want to associate different authorizations with different Let's Encrypt accounts and keys, to limit your exposure if an internal host is compromised. With the bastion host, well, you're on your own unless you build something really complicated.)
We've switched over to using Let's Encrypt as much as possible
Over the years, we've used a whole collection of different TLS CAs. We've preferred free ones where we could, for good reasons, which meant that we've used both ipsCA (until they exploded) and StartSSL (aka StartCom), but we've also paid for TLS certificates when we had to; modern TLS certificates are pretty affordable even for us if we don't go crazy. And these days we even have access to free TLS certificates through the university's central IT. However, we've now switched over to using Let's Encrypt as much as possible; basically it's the first CA we attempt to use, and if it doesn't work for some reason we'd probably turn to the free TLS certificates from central IT (both because they're free and because the process of getting one isn't too painful).
Our main reason for switching to Let's Encrypt isn't that it's free (it's not our only current source of free certificates); instead, as with my personal use, it's become all about the great automation. With Let's Encrypt, getting an initial certificate just requires running a command line program, and once we've worked out how to handle any particular program (since LE's good for more than web servers), we can completely stop worrying about certificate renewals. It just quietly happens and everything works and we don't notice a thing. The LE client that we've wound up using all of the time is Hugo Landau's acmetool, which is what I settled on myself. Acmetool has proven to be reliable and easy to tweak so it supports various programs like Dovecot and Exim.
(Our current approach to satisfying Let's Encrypt challenges is to let HTTP from the Internet through to any machine that needs a TLS certificate, whether or not it normally runs a web server. Acmetool will automatically run its own web server while a challenge is active, if necessary.)
Using acmetool or any other suitable Let's Encrypt client is not the only way of automating TLS certificate updates, but it has the great advantage for us that it comes basically ready to go. In our environment there's almost nothing to build to support new TLS-using programs and almost nothing special to do to set acmetool up on any particular machine (and we have canned directions for the few steps required). People with existing modern automation infrastructure may already have this solved, and so may find Let's Encrypt less compelling than we do.
Almost two years ago I wrote about how we couldn't use Let's Encrypt for production due to rate limits. What's changed since then is that Let's Encrypt's current rate limits specifically exempt certificate renewals from their 'certificates per registered domain' limit. This means that if we can get an initial certificate for a host, we're basically sure to be able to renew it, which is the important thing for us. If the initial issuance fails, that's when we can turn to alternate CAs (but for the names we want it almost never does).
PS: Since automation is such a big motivation for us, what sold us is not Let's Encrypt by itself but acmetool. In a real sense, we're indifferent to what TLS certificate provider is behind acmetool, and if we could get free certificates from central IT just as easily (perhaps literally using LE's ACME protocol), we'd be happy to do just that. But at least for now, Let's Encrypt itself and ACME are conjoined together.