Wandering Thoughts

How we're dealing with our expiring OpenVPN TLS root certificate

Recently I wrote about my failure to arrange a graceful TLS root certificate rollover for our OpenVPN servers. This might leave you wondering what we're doing about this instead, and the answer is that we've opted to use a brute force solution, because we know it works.

Our brute force solution is to set up a new set of OpenVPN servers (we have two of them for redundancy), using a new official name and with it a new TLS root certificate that is good for quite a while (I opted to be cautious and not cross into 2038) and with it a new host certificate. With the new servers set up and in production, we've updated our support site so use the new official name and the new TLS root certificate, so people who set up OpenVPN from now onward will be using the new environment.

Since these servers are using a new official name, they and the current (old) OpenVPN servers can operate at the same time. People with the new client configuration go through our new servers; people with the old client configuration and old TLS certificate go through our old servers. There's no flag day where we have to change the TLS root certificate on the old servers, and in fact they won't change; we're going to run them as-is right up until the TLS root certificate expires and no one can connect to them any more.

This leaves us with all of the people who are currently using our old OpenVPN servers with the expiring TLS root certificate. We're just going to have to contact all of them and ask them to update (ie change) their client configuration, changing the OpenVPN server name and getting and installing the new TLS root certificate. This is not quite as bad as it might sound, because we were always going to have to contact the current people to get them to update their TLS root certificate. So they only have to do one extra thing, although that extra thing may be quite a big pain.

(Some environments have nice, simple OpenVPN configuration systems. But on some platforms, the configuration is 'open a text editor and ...', and one of them is probably not one you're thinking of.)

Doing the change this way 'costs' us two extra servers for a while, which we have to spare, and more importantly it meant that we needed a new official name for our OpenVPN service. This time around this was acceptable, because our old official name was in retrospect perhaps not the best option. If we have to do this again, we may have a harder time coming up with a good new name, but hopefully next time around we'll be able to roll over the TLS root certificate instead of having to start all over from scratch.

(From my perspective, the most annoying thing about this is that I just rebuilt the OpenVPN servers in January in order to update them to a modern OpenBSD. If I'd known all of this back then, we could have gone straight to our new end state and saved one round of building and installing machines.)

TLS certificate durations turn out to be complicated and subtle

The surprising TLS news of the time interval is that Let's Encrypt made a systematic mistake in issuing all of their TLS certificates for years. Instead of being valid for exactly 90 days, Let's Encrypt certificates were valid for 90 days plus one second. This isn't a violation of the general requirements for Certificate Authorities on how long TLS certificates can be, but it was a violation of Let's Encrypt's own certificate policy.

TLS certificates have a 'not before' and 'not after' times. For ages, Let's Encrypt (and almost everyone else) has been generating these times by taking a start time and adding whatever duration to it. You can see an example of this in some completely unrelated code in my entry on how TLS certificates have two internal representations of time, where the certificate starts and ends on the same hour, minute, and second (19:40:26 in the entry). However, it turns out that the TLS certificate time range includes both the start and the end times; it's not 'from the start time up to but not including the end time'. Since this includes both the second at the start and the second at the end, a simple 'start time plus duration' is one second too long.

(A properly issued literal 90 day certificate from Let's Encrypt now has an ending seconds value that's one second lower than it starts, for example having a not before of 2021-06-10 15:31:37 UTC and a not after of 2021-09-08 15:31:36.)

This is already a tricky issue but the Mozilla bug gets into an even more tricky one, which is fractional seconds. If a certificate has a 'not after' of 15:31:36, is it valid right up until 15:31:37.000, or does it stop being valid at some time after 15:31:36.000 but before 15:31:37.000? The current answer is that it's valid all the way up to but not including 15:31:37.000, per Ryan Sleevi's comment, but there's some discussion of that view in general and it's possible there will be a revision to consider these times to be instants.

(People are by and large ignoring leap seconds, because everyone ignores them.)

All of this careful definition of not before and not after is in the abstract of RFCs and requirements for Certificate Authorities, but not necessarily in what actual software does. Some versions of OpenSSL apparently treat both the not before and not after times as exclusive when validating TLS certificates (cf); the time must be after the not before time and before the not after time. Other software may have similar issues, especially treating the not after time as the point where the certificate becomes invalid. I would like to say that it also doesn't matter in actual practice, but with TLS's luck someone is eventually going to find an attack that exploits this. Weird things happen in the TLS world.

PS: Let's Encrypt's just updated CPS deals with the whole issue by simply saying they will issue certificates for less than 100 days.

PPS: Some certificate reporting software may not even print the seconds for the not before and not after fields. I can't entirely blame it, even though that's currently a bit inconvenient.

Early notes on using the new python-lsp-server (pylsp) in GNU Emacs

When I started with LSP-based Python editing in GNU Emacs, the Python LSP server was pyls. However, pyls is apparently now unmaintained and the new replacement is python-lsp-server, also known as 'pylsp'. I noticed this recently when I looked into type hints a bit, and then when I was editing some Python today, lsp-mode or some sub-component nagged me about it:

Warning (emacs): The palantir python-language-server (pyls) is unmaintained; a maintained fork is the python-lsp-server (pylsp) project; you can install it with pip via: pip install python-lsp-server

The first thing to note about pylsp is that it supports Python 3 only (it says Python 3.6+). It works to some degree if you edit Python 2 code, but I don't fully trust it, so I'm keeping around my current Python 2 version of the older pyls. Pyls may be unmaintained, but at the moment it appears to work okay.

Because of the Python 3 versus 2 issue, I already had a front end 'pyls' script to try to figure out which Python's version of pyls I needed to run. Fortunately pyls and pylsp are currently invoked with the same (lack of) arguments, so I cheated by renaming this script to 'pylsp' and having it run the real pylsp for Python 3 and fall back to pyls for Python 2.

(I opted to uninstall the Python 3 version of pyls before I installed pylsp, because I didn't want to find out if they had conflicting version requirements for some shared dependencies.)

Lately I've been running pyls under PyPy, so I started out by installing pylsp this way too. Pylsp (like pyls before it) has some useful looking third party plugins and since I was installing from scratch now I decided to install some of them, including mypy-ls. This is when I found out that unfortunately mypy doesn't run under PyPy. So I switched to installing pylsp using CPython with my usual 'pip3 install --user'. This worked for pylsp itself and mypy-ls, but pyls-memestra had issues due to memestra basically having to be installed in a virtualenv or other personal install of CPython or PyPy. I dealt with this by removing pyls-memestra; it might be nice but it's not essential.

(Memestra attempts to make a directory under sys.prefix, which is owned by root if you're running the system CPython or PyPy.)

The result appears to work fine and has no more warning messages in various internal Emacs LSP buffers than I expect, but I haven't used it extensively yet. I'm not sure I'll keep mypy-ls yet, because it does add some extra warnings in some situations. The warnings are valid ones if you're using type annotations, but a potential problem if you're not. Probably it's good for me to get the warnings and maybe start fixing them.

The modern web design aesthetic of hiding visited links

Every since I started aggressively overriding websites so I could see what links I'd visited, it's become really clear to me that modern web design apparently hates making visited links look different and goes out of its way to not do so. This is most visible on the quite a lot of text focused websites that still stick to more or less the original web colours of black text on a white background with blue links. In the original basic web colour scheme, visited links would be a purple shade. But although all of these websites stick to the basic black, white, and blue colours, they wipe away the purple of visited links.

Some websites opt to style their links in some way other than with colours (often underlines), and for those websites not styling visited links differently is understandable (even though I disagree with it). These sites have decided to have text and text decorations in a single colour, and you can't significantly restyle visited links these days (for good reasons). So, for example, if you use a solid underline for unvisited links, you can't use a dashed one for visited ones (as far as I know). Other websites opt to significantly change some or many of the colours for whatever reasons, so not styling visited links differently avoids coming up with an additional colour (and colours are hard if you take it seriously).

(Also, if I'm understanding CSS right, if you set a colour for <a> elements it applies to them in all states. Once you start setting link colours, you have to go out of your way to colour visited, active, focused, or hovered links different than standard (unvisited) ones.)

However, quite a lot of text focused sites stick to the basic colours for text content but more or less deliberately wipe out any special colour for visited links. These sites could easily continue to let visited links be as they are when unstyled, but they don't. Whether this is done deliberately or is simply casually accepted, it's clearly part of today's web design aesthetic (and probably has been for a while).

I'm sure I noticed this subconsciously before, but actually creating site style after site style in Stylus has rubbed my nose in just how many of the sites I wanted to fix use the standard black, white, and blue colour scheme. It's also made me aware of how common a basic scheme of black, white, and underlined links is (it's probably the second most common one I alter).

My failure to arrange a graceful TLS root certificate rollover with OpenVPN

Generally, what I write about here is discoveries, questions, and successes. This presents a somewhat misleading picture of what my sysadmin work is like, so today I'm going to talk about a TLS issue that I spent a day or two failing at recently.

(I wouldn't say that failure is a routine event in system administration, but sometimes you can't solve a problem, and it can happen to anyone.)

We have some OpenVPN servers for our users, running on OpenBSD using the OpenBSD packaged version of OpenVPN. When you run OpenVPN, you normally establish a private Certificate Authority, with your own root certificate. This is used to authenticate your OpenVPN server to users, by them verifying that your OpenVPN server presents a host certificate that's ultimately signed by your CA, and it can also be used to sign user certificates that are used to authenticate users. Of course to do this your users have to manually tell their OpenVPN client about your root CA. We do this by providing a copy of our local CA root certificate that they need to download and install in their client.

Almost ten years ago, in August of 2011, we set up the first instance of our modern OpenVPN server infrastructure, and generated its root CA certificate. The default expiry time on this CA certificate was ten years, and so it runs out at the end of August of 2021, which is to say in a couple of months. Since we can't assume that all OpenVPN clients will ignore the expiry time of the CA root certificate, we need to do something about this. The simple thing to do is to generate a new CA root certificate (with a long expiry time) and a new host certificate and start using them, but this creates a flag day where all of our OpenVPN users have to download the new CA certificate from us and switch to it; if they don't switch the CA certificate in time, they stop being able to connect to our OpenVPN servers.

We would like to do better, and I wound up with two ideas for how to do it. My first attempt was to create a new cross-signed CA root certificate (and a new host certificate signed by it). One version of the new root certificate was signed by the current root; our OpenVPN servers would provide this in a certificate chain until the old CA root expired. The other version was self-signed, and would be downloaded by people who'd switch to it in advance. The server host certificate would verify through either certificate chain.

Cross signed root certificates are a reasonably common thing in web TLS, and once I fumbled my way through some things the resulting certificate chains passed verification in OpenSSL and another tool I tend to use. But I couldn't get my test OpenVPN client to validate the host certificate using the new CA certificate.

My second attempt was more brute force. I took the keypair from our existing CA root certificate and used it to create a new version of the CA root certificate with the same keypair, the same Subject Name, and a longer expiry. Since this used the same keypair and Subject Name as our existing root certificate, in normal TLS certificate verification it's a perfect substitute for our current expiring CA. My verification tools said it was the same and would verify the current host certificate, and after some work 'openssl asn1parse' said that the two certificates had the same low-level content except the serial number, the validity dates, and the signature. But my test OpenVPN client would not accept the new CA certificate no matter what I did. I even generated and signed a new (test) server host certificate using this new version of the CA certificate and had my test OpenVPN server provide the new CA certificate and the new host certificate while my client was using the new CA certificate. It didn't work.

At this point I'm out of clever ideas to avoid significant pain for our users. Unless something changes in the situation, the best we can do for people is avoid a flag day as much as possible.

(This sort of elaborates on some tweets of mine. My test OpenVPN client was my Fedora 33 laptop; Fedora's OpenVPN client may be a bit atypical, but we have both Fedora and Ubuntu OpenVPN users, so if our work-around doesn't work with them some of our users will have a bad time.)

PS: Official TLS certificates for our OpenVPN servers wasn't really an option back in 2011, and it's probably still not one for various reasons. I made some tests to see if I could make it work in a test setup (hence my use of Let's Encrypt on OpenBSD) but failed there too, although I didn't investigate very deeply.

TLS certificates have at least two internal representations of time

TLS certificates famously have a validity period, expressed as 'not before' and 'not after' times. These times can have a broad range, and there are some TLS Certificate Authority root certificates that already have 'not after' times relatively far in the future (as I mentioned here). All TLS certificates, including CA root certificates, are encoded in ASN.1. Recently I was both generating long-lived certificates and peering very closely into them in an attempt to figure out why my new certificates weren't working, and in the process of doing so I discovered that ASN.1 has at least two representations of time and what representation a TLS certificate uses depends on the specific time.

Most TLS certificates you will encounter today encode time in what 'openssl asn1parse' calls a UTCTIME. If you have a TLS certificate with a sufficiently far in the future time, it will instead be represented as what OpenSSL calls a GENERALIZEDTIME. Somewhat to my surprise, both of these turn out to be strings under the covers and the reason that TLS switches from one to the other isn't what I thought it was. I'll start by showing the encoding for a not before and a not after date (and time) for a certificate I generated:

UTCTIME         :210531194026Z
GENERALIZEDTIME :20610521194026Z

This certificate is valid from 2021-05-31 19:40 UTC to 2061-05-21 19:40 UTC. The Z says this is in UTC, the '194026' is 19:40:26, and the '0531' and '0521' are the month and day. The difference between the two time formats is at the front; the UTCTIME starts with '21' while the other starts with '2061'.

When I started looking into the details of this, I assume that the choice between one or the other form was because of the year 2038 problem. This is not the case, since UTCTIME is not represented as any sort of Unix epoch timestamp and has no such limits. Instead, UTCTIME's limitation is that it only uses a two-digit year. As covered in RFC 5280, if the two year digits are 00 to 49, the year is 20yy, and for 50 to 99, it is 19yy. This means that a UTCTIME can only represent times up to the end of 2049. The certificate I generated is valid past that, so it must use the more general version.

In theory, all code that deals with TLS certificates should be able to deal with both forms of time. This is a low level concern that the ASN.1 parsing library should normally hide from programs, and both forms have been valid since RFC 2459 from 1999. In practice, I suspect that there's almost no use of the second time format in certificates today, so I suspect that there's at least some software that mishandles them. For general use, we have years to go before this starts to be an issue (starting with CA root certificates that push their expiry date into 2050 and beyond).

For our own use, I think I'm going to limit certificate validity to no later than 2049. The more cautious approach is to assume that there's a Unix timestamp somewhere in the chain of processing things and stick to times that don't go beyond the year 2038 boundary.

(I think that these are the only two ASN.1 time representations that are considered valid in TLS certificates on the Internet, but I haven't carefully gone through the RFCs and other sources of information to be sure. So I'm being cautious and saying that TLS certificates have 'at least' two representations of time.)

The case of the very old If-Modified-Since HTTP header

Every so often I look at the top IP sources for Wandering Thoughts. Recently, I noticed that one relatively active IP was there because it was fetching my Atom syndication feed every few minutes, and on top of that it was always getting a HTTP 200 reply with the full feed. Usually my assumption is that these requests aren't using HTTP conditional GET at all, but I keep looking because I might find something like the Tiny Tiny RSS problem (which I can theoretically fix Tiny Tiny RSS). To my surprise, something a bit interesting is happening.

This feed fetcher was sending an If-Modified-Since HTTP header, but it had a rather striking value of 'Wed, 01 Jan 1800 00:00:00 GMT'. Naturally this doesn't match any Last-Modified value my feed has ever provided, and it wouldn't help if I used a time based comparison since all syndication feeds in the world have been changed since 1800.

Any time I see a very old timestamp like this, I suspect that there's code that has been handed an un-set zero value instead of an actual time (here, a Last-Modified time). Syndication feed fetchers are perhaps especially prone to this; they start out with no Last-Modified time when they fetch a feed for the first time, and then if they ever fail to parse a Last-Modified time properly they might write back an unset value. However, 1800 is a somewhat unusual zero value for time; I'm more used to Unix timestamps, where the zero value is January 1st 1970 GMT.

This feed fetcher identifies itself as 'NextCloud-News/1.0'. If that is this NextCloud application (also), it's written in PHP and is probably setting If-Modified-Since here using a PHP DateTime (or maybe it uses feed-io, I don't actually know PHP so I'm just grep'ing the codebase). I can't readily find any documentation on what the zero value for a DateTime is, or if it's even possible to wind up with one. Neither MySQL, PostgreSQL, nor SQLite appear to use 01 Jan 1800 as a zero value either. So on the whole I'm still lost.

(In passing I'll note that this user-agent value is not all that useful. To be useful, it should include the actual version number of the NextCloud-News release (they're up to 15.x, with 16.0.0 coming soon) and some URL for it, so I can be confident I have identified the right NextCloud News thing.)

PS: If this is a NextCloud-News code issue, correcting it would be nice (and please don't treat Last-Modified as a timestamp), but it would be better to use ETag and If-None-Match.

(This elaborates on a Twitter thread.)

HTTP/3 needs us (and other people) to make firewall changes

The other day, I had a little realization:

Today I realized that the growing enabling of HTTP/3 means that we need to allow UDP 443 through our firewalls (at least outbound), not just TCP 443. Although in the mean time, blocking it shields our users from any HTTP/3 issues. (Which happen.)

Like many places, our network layout has firewalls in it, in fact quite a lot of them. We have a perimeter firewall, of course, then we have firewalls between our internal subnets, our wireless network has a firewall, and our VPN servers have their own set of firewall rules. All of our firewalls have restrictions on outbound traffic, not just inbound traffic.

For obvious reasons, all of our firewalls allow outbound traffic to TCP port 443 (and port 80, and a number of others). However, some of them don't allow outbound traffic to UDP port 443, because there's been no protocol that used that. Until now. HTTP/3 uses QUIC, which runs over UDP, and so it thus generates traffic to UDP port 443. Right now any such traffic is probably not getting through.

Google's Chrome has enabled HTTP/3 (and QUIC) for some time, Firefox enabled HTTP/3 by default in Firefox 88, and Microsoft Edge has also had it for a while (Apple's Safari has yet to enable it by default). All of those browsers will now be sending traffic to UDP port 443 under the right circumstances, or at least trying to; while our firewalls block that traffic, they're not getting very far. I don't know how HTTP/3 implementations behave here, but I wouldn't be surprised if this creates at least a little bit of a slowdown.

(Of course this may shield people from a great deal of slowdown if HTTP/3 appears to work more.)

We're not the only places who are going to need to update firewalls to enable outbound UDP port 443, of course. But I suspect that Google (the originators of the whole QUIC idea) has studied this and determined that there are fewer firewall blocks in the way than I might expect.

Eventually we may also want to enable inbound UDP to port 443, so that people can run web servers that support HTTP/3. But that will probably take much longer, because server support is apparently rather lacking right now (based on the Wikipedia list). So far most of the web servers we run don't even have HTTP/2 enabled yet, for various reasons.

New versions of Bash (and readline) default to special handling of pasting into the shell (or other programs)

Our standard habit on our OpenBSD machines is to use their packaged version of Bash as the root shell, instead of the default of OpenBSD's version of ksh. When I set up an OpenBSD 6.9 machine recently and started to paste in install steps from our standard instructions, an odd thing happened: the line I'd just pasted in was highlighted in reverse video and Bash just sat there, instead of doing anything. After some fiddling around, I discovered that I had to hit Return in order get things to go (and at that point Bash would act on all of the input without further prompts, even if I'd pasted in multiple lines).

People who are more familiar with Bash and readline than I was already know what this is; this is the Bash readline setting of enable-bracketed-paste. OpenBSD 6.9 is the first Unix I've encountered where this was turned on by default. This isn't because OpenBSD did anything special; instead, it's because OpenBSD 6.9 is the first Unix I've used that has Bash 5.1. As mentioned in the detailed Bash list of changes, bracketed paste mode was enabled by default starting in bash-5.1-alpha. The reverse video behavior of it is also new there; in 5.0 and before, nothing special shows in bracketed paste mode to signal that something unusual is going on.

Bash 5.1 will be rolling out over time to more Unixes, so I suspect that more people will be running into this behavior unless their particular Unix opts to disable it in one way or another. If I had updated to Fedora 34 by now, it's possible I'd have already encountered this (I believe that Fedora 34 has Bash 5.1, but I don't know how Fedora opted to set bracketed paste).

This change to the default of bracketed paste is also in GNU Readline 8.1 (according to its CHANGES). However, Readline has a configuration time option that can change this, so different Unixes may opt to build Readline differently. On a Unix with Readline 8.1+ and bracketed paste enabled by default, I believe that all programs using GNU Readline will automatically have this behavior.

(This directly affects me because these days I build my alternate shell using GNU Readline. If I do nothing, it will someday inherit this behavior on new versions of Fedora and Ubuntu.)

If you decide that you don't want bracketed paste mode, the safest change is to set this in either your $HOME/.inputrc or globally in /etc/inputrc. You would do this with:

set enable-bracketed-paste off

This will (or should) cover Bash and anything else that starts using Readline 8.1 on a Unix that builds 8.1 with this enabled. Adding this to your .inputrc today is harmless if you have Readline 7.0 and Bash 4.4 or later (the versions where this setting was introduced).

If you just want to turn this off only in Bash (at least for now), I think that you want to set up a $HOME/.bashrc that has in it:

bind 'set enable-bracketed-paste off'

You can set this in your .profile, but then it won't be turned on in subshells.

How to set this bind up globally for Bash depends on how your Unix's version of Bash was built and may not be possible. Ubuntu builds Bash so that there's a global /etc/bash.bashrc you can put this bind into, but Fedora and OpenBSD don't. Fedora provides a starting .bashrc for new accounts that source /etc/bashrc, so you can put the bind there and probably get most people. Since Bash is an add-on in OpenBSD, it has nothing like this and people are on their own to disable it one by one.

Fedora co-mingles its source packages with Red Hat Enterprise Linux

In the RPM package management world, the source RPM is what packages are built from and the source RPM's specfile is what describes how to build the binary RPMs (in addition to other package metadata). For a specific and relevant example, the Fedora specfile for the kmod RPM is what says (or used to say) that the weak-modules script should be added to the binary RPM, although it's not in the upstream source and this created a DKMS problem.

However, this is a somewhat incomplete and misleading description of the real problem. The underlying issue was that calling it 'the Fedora kmod specfile' is not really accurate. In reality, the Fedora source RPM for the kmod RPM (and thus the specfile) was also the (future) Red Hat Enterprise Linux source RPM and specfile for kmod. The weak-modules script comes from RHEL and is (probably) useful there, but because Fedora uses the same specfile, it was blindly shipped in Fedora as well. The actual fix to remove weak-modules doesn't remove it from the specfile entirely but adds more complexity to make it conditional.

(Since it's specifically excluded only on Fedora, a Fedora-derived distribution will have to modify this specfile. If the distribution notices, and if it doesn't already pretend to be Fedora for package building purposes, since this probably isn't the only such specfile.)

This kmod issue illustrates at least one potential problem with this dual usage of specfiles. Fedora and Red Hat Enterprise Linux are not the same thing, so there are undoubtedly other places where RHEL packaging needs diverge from Fedora packaging ones. At the moment it's up to package maintainers to manually identify such places and deal with them in some way. If they don't, one or the other of Fedora and RHEL will have issues. If people are lucky, those will be immediate and obvious and will be fixed. Otherwise, you get something like my problem.

This situation is unfortunately probably unavoidable because in practice, Fedora is not an independent distribution; instead it's an arm of Red Hat Enterprise Linux. Red Hat (now IBM) originated what became Fedora, uses Fedora as the base for RHEL, significantly funds Fedora, and still hosts important resources for it (Fedora bugs are reported to bugzilla.redhat.com, for example). Red Hat has no interest in spending the effort to maintain two independent sets of source RPMs and RPM specfiles.

(This situation isn't the same as Debian and Ubuntu. Ubuntu uses Debian's packaging work, but Ubuntu's changes for their own needs don't automatically wind up in Debian.)

PS: I believe it's relatively rare that the latest RHEL and Fedora will have the same version of a package, so it's probably usually not the case that the source RPM is the same on both. Instead, the RHEL package most likely started off as some older version of the Fedora package, so we can informally say that they use the same source RPM and specfile.

Simple use of Let's Encrypt on OpenBSD is pleasantly straightforward (as of 6.8)

For reasons beyond the scope of this entry, I recently needed to get a Let's Encrypt TLS certificate for testing on an OpenBSD machine, which isn't something I've done before. On a relatively modern OpenBSD (6.8), it was pleasantly straightforward and easy, with the programs necessary already installed in a full base install (which is what we normally do on our OpenBSD machines, since a full install is so small).

OpenBSD's standard Let's Encrypt client is acme-client, which has to be configured through /etc/acme-client.conf and then invoked (for example) as 'acme-client -v yourhost' to start the process of getting your TLS certificate. As the OpenBSD documentation tells you, a sample acme-client.conf is in /etc/examples and is easy to edit into shape to list the names you want Let's Encrypt certificates for. I opted to add the optional contact option to the 'letsencrypt' authority in my acme-client.conf, although in retrospect it's pointless for a test server where I don't care if the certificate expires after I'm done.

In the OpenBSD tradition, acme-client is very minimal. Unlike more all encompassing Let's Encrypt clients like Certbot, acme-client doesn't do either post-renewal actions or answering the HTTP challenges.

(Certbot and many other heavy-weight clients have their own little internal HTTP server that they'll run for you for the duration of a challenge, if you ask. This is Certbot's standalone mode.)

To handle the HTTP side of things, the easiest approach is to run OpenBSD's standard httpd server at least temporarily. OpenBSD ships with a sample httpd.conf in /etc/examples that I was able to use with very few changes. Because I wanted to be able to test my new certificate, I left the HTTPS version of my host in my httpd.conf (although it wasn't serving anything), but you could remove it to have a HTTP server that's just there to answer the Let's Encrypt challenges. Pleasantly, OpenBSD httpd will still start if you have a HTTPS site configured but the TLS certificates for it are missing. This lets you leave your HTTPS site configured before you've gotten the Let's Encrypt certificate for it.

(The default OpenBSD httpd.conf redirects all of your HTTP site to your theoretical HTTPS site, which nicely makes it serve nothing except answers to Let's Encrypt challenges.)

Because I was getting my Let's Encrypt TLS certificate for something other than serving a web site, I didn't permanently enable the HTTP server. I then had to start httpd more forcefully than usual, with 'rcctl -f start httpd'; otherwise it reported:

/etc/rc.d/httpd: need -f to force start since httpd_flags=NO

(This is unlike the Prometheis host agent package, which can be started temporarily with 'rcctl start node_exporter' before you decide to permanently enable it. This is probably routine to regular OpenBSD users.)

Once I'd started httpd, acme-client succeeded without problems. Unlike Certbot and some other clients, acme-client has no separate Let's Encrypt account registration process; it registers an account if and when necessary to get your certificate. I opted to get both the certificate ('domain certificate ...' in acme-client.conf) and the default full chain ('domain full chain certificate ...'). This isn't strictly necessary, since you can always manually extract each individual certificate from the full chain file, but I wasn't sure how I was going to use the certificate so I opted to save myself time.

(Since this was a quick test setup, I haven't tried to automate any renewal, but the acme-client manpage has an example cron entry. You need a separate cron entry for every separate certificate you have on the machine; unlike Certbot, there is no 'try to renew anything necessary', even though all of your certificates are listed in your acme-client.conf.)