Wandering Thoughts archives

Web CGI programs aren't particularly slow these days

I recently read Reminiscing CGI scripts (via), which talked about CGI scripts and in passing mentioned that they fell out of favour for, well, let me quote:

CGI scripts have fallen out of favor primarily due to concerns related to performance and security. [...]

This is in one sense true. Back in the era when CGIs were pushed aside by PHP and other more complex deployment environments like Apache's mod_perl and mod_wsgi, their performance was an issue, especially under what was then significant load. But this isn't because CGI programs are intrinsically slow in an absolute sense; it was because computers in the early 00s were not very powerful and might even be heavily (over-)shared in virtual hosting environments. When the computers acting as web servers couldn't do very much in general, everything less you could get them to do could make a visible difference, including not starting a separate program or two for each request.

Modern computers are much faster and more powerful than the early 00s servers where PHP shoved CGIs aside; even a low end VPS is probably as good or better, with more memory, more CPU, and almost always a much faster disk. And unsurprisingly, CGIs have gotten a lot faster and a lot better at handling load in absolute terms.

To illustrate this, I put together a very basic CGI in Python and Go, stuck them in my area on our general purpose web server, and tested how fast they would run. On our run of the mill Ubuntu web server, the Python version took around 17 milliseconds to run and the Go version around four milliseconds (in both cases when they'd been run somewhat recently). Because the CGIs are more or less doing nothing in both cases, this is pretty much measuring the execution overhead of running a CGI. A real Python CGI would take longer to start because it has more things to import, but even then it's not necessarily terribly slow.

(As another data point, I have ongoing numbers for the response time of Wandering Thoughts, which is a rather complex piece of Python normally running as a CGI. On fairly basic (virtual) hardware, it seems to average about 0.17 seconds for the front page (including things like TLS overhead), which is down noticeably from a decade ago.)

Given that CGI scripts have their attractions for modest scale pages and sites, it's useful to know that CGI programs are not as terrible as they're often made out to be in old sources (or people working from old sources). Using a CGI program is a perfectly good deployment strategy for many web applications (and you can take advantages of other general web server features).

(Yes, your CGI may slow down if you're getting a hundred hits a second. How likely is that to happen, and if it does, how critical is the slowdown? There are some environments where you absolutely want and need to plan for this, but also quite a number where you don't.)

What /.well-known/ URL queries people make against our web servers

WebFinger is a general web protocol for obtaining various sorts of information about 'people' and things, including someone's OpenID Connect (OIDC) identity provider. For example, if you want to find things out about 'brad@example.org', you can make a HTTPS query to example.org for /.well-known/webfinger?resource=acct%3Abrad%40example.org and see what you get back. WebFinger is on my mind lately as part of me dealing with OIDC and other web SSO stuff, so I became curious to see if people out there (ie, spammers) were trying to use it to extract information from us.

As we can see, WebFinger is just one of a number of things that use '/.well-known/<something>'; another famous one is Let's Encrypt's HTTP based challenge (HTTP-01), which looks for /.well-known/acme-challenge/<TOKEN> (over HTTP, not HTTPS, although I believe it accepts HTTP to HTTPS redirects). So I decided to look for general use of /.well-known/ to see what came up, and to my surprise there was rather more than I expected.

The official registry for this is Well-Known URIs at IANA. On the web server for our normal email domain (which is not our web server), by far the common query was for '/.well-known/carddav', documented in RFC 6764. After that I saw some requests for '/.well-known/openpgpkey/policy', which is covered here and less clearly here, but which isn't an officially registered thing yet. Then there were a number of requests for '/.well-known/traffic-advice' from "Chrome Privacy Preserving Prefetch Proxy". This too isn't officially registered and is sort of documented here (and here), in this question and answers, and in this blog entry. Apparently this is a pretty recent thing, probably dating from August 2023. Somewhat to my surprise, I couldn't see any use of WebFinger across the past week or so.

On our actual web server, the picture is a bit different. The dominant query is for '/.well-known/traffic-advice', and then after that we get what look like security probes for several URLs:

/.well-known/class.api.php
/.well-known/pki-validation/class.api.php
/.well-known/pki-validation/cloud.php
/.well-known/pki-validation/
/.well-known/acme-challenge/class.api.php
/.well-known/acme-challenge/atomlib.php
/.well-known/acme-challenge/cloud.php
/.well-known/acme-challenge/
/.well-known/

(Although '/.well-known/pki-validation' is a registered Well-Known URI, I believe this use of it is as much of a security probe as the pokes at acme-challenge are.)

There was a bit of use of '/.well-known/assetlinks.json' and '/.well-known/security.txt', and a long tail of other things, only a few of them registered (and some of them possibly less obviously malicious than people looking for '.php' URLs).

(We did see some requests for Thunderbird's '/.well-known/autoconfig/mail/config-v1.1.xml', which perhaps we should support, although writing and validating a configuration file looks somewhat complicated.)

There weren't that many requests overall, which isn't really surprising given that we HTTP 404'd all of them. What's left is likely to be the residual automation that blindly tries no matter what and some degree of automated probes from attackers. I admit I'm a bit sad not to have found any for WebFinger itself, because it would be a bit nifty if attackers were trying to mine that (or we had people probing for OIDC IdPs, or some other WebFinger use).

Seeing how fast people will probe you after you get a new TLS certificate

For reasons outside the scope of this entry I spent some time today setting up a new Apache-based web server. More specifically, I spent some time setting up a new virtual host on a web server I'd set up last Friday. Of course this virtual host had a TLS certificate, or at least was going to once I had Let's Encrypt issue me one. Some of the time I'm a little ad-hoc with the process of setting up a HTTPS site; I'll start out by writing the HTTP site configuration, get a TLS certificate issued, edit the configuration to add in the HTTPS version, and so on. This can make it take a visible amount of time between the TLS certificate being issued, and thus appearing in Certificate Transparency logs, and there being any HTTPS website that will respond if you ask for it.

This time around I decided to follow a new approach and pre-write the HTTPS configuration, guarding it behind an Apache <IfFile> check for the TLS certificate private key. This meant that I could activate the HTTPS site pretty much moments after Let's Encrypt issued my TLS certificate. I also gave this new virtual host it's own set of logs, in fact two sets, one for the HTTP version and one for the HTTPS version. Part of why I did this is because I was curious how long after I got a TLS certificate it would be before people showed up to probe my new HTTPS site.

(It's well known by now that all sorts of people monitor Certificate Transparency logs for new names to probe. These days CT logs also make new entries visible quite fast; it's easily possible to monitor the logs in near real time. My own monitoring, which is nowhere near state of the art, was mailing me less than five minutes after the certificate was issued.)

If you've ever looked at this yourself, you probably know the answer. It took roughly a minute before the first outside probes showed up (from a 'leakix.org' IP address). Interestingly, this also provoked some re-scans of the machine's first HTTPS website, which had been set up Friday (and whose name was visible in, for example, the IP address's reverse mapping). These scans were actually more thorough than the scans against the new HTTPS virtual host. The HTTP versions of both the base name and the new virtual host were also scanned at the same time (again, the base version more thoroughly than the new virtual host).

Our firewall logs suggest that the machine was getting hit with a higher rate of random connections than before the TLS certificate was issued, along with at least one clear port scan against assorted TCP ports. This clear port scan took a while to show up, only starting about twenty minutes after the TLS certificate was issued (an eternity if you're trying to be the one who compromises a newly exposed machine before it's fixed up).

At one level none of this is really surprising to me; I knew this sort of stuff happened and I knew it could happen rapidly. At another level there's a difference between knowing it and watching your logs as it happens live in front of you.

Mapping out my understanding of (web-based) single sign-on systems

Suppose, not entirely hypothetically, that you want to use some systems (perhaps external systems) that wants you to have a 'single sign on' (SSO) system that it can use to authenticate you and your users. There are a number of good reasons for both sides to want this; you get better control and the outside system gets to outsource all of the hassles of managing authentication to you. To create this SSO setup, there are a number of pieces, and here is how I currently understand them.

The thing you want to end up with is an Identity Provider (IdP). Typical IdPs have two roles; they challenge users to authenticate (generally through a web browser) and perhaps approve giving this authentication information to other systems, and they provide authenticated identity information to other systems. They typically do their single sign on trick by putting a cookie in the browser to mark you as already authenticated, so when a system sends you to the IdP to get authenticated you just bounce right through without getting challenged. A garden variety IdP does all of this with HTTP(S) transactions, some of them from people's web browsers and some of them from systems to API endpoints (or from the IdP to other people's API endpoints).

An IdP needs to speak some protocol to systems that are getting authentication information from it. Two common protocols are SAML and OIDC (OpenID Connect) (also). Different IdP implementations speak different protocols; for example, SimpleSAMLphp primarily speaks SAML (as you might expect from the name), although now I look it can apparently also speak OIDC through an OIDC module. By contrast, Dex is purely an OIDC and OAuth2 IdP, while Keycloak will support all of SAML, OIDC, and OAuth2.

Naturally people have built bridges that do protocol translation between SAML and OIDC, so that if you have a SAML IdP already, you can provide OIDC to people (and perhaps vice versa). You can also 'bridge' between the same protocol, so (for example) Dex can use another OIDC IdP for authentication. I believe one reason to do this in general is to filter and winnow down the upstream IdP's list of users. Dex's documentation suggests another reason is to reformat the answers that the upstream OIDC IdP returns to something more useful to the systems using your IdP, and I'm sure there are other reasons.

(One obvious one is that if your IdP is basically an OIDC proxy for you, you don't have to register all of the systems and applications using your IdP with the upstream IdP. You register your IdP and then everything hides behind it. Your upstream IdP may or may not consider this a feature.)

An OIDC or SAML IdP needs to find out what users you have (and perhaps what administrative groups they're part of), and also authenticate them somehow. Often one or both can be outsourced to what Dex calls connectors. A popular source of both user information and user authentication is a LDAP server, which you may already have sitting around for other reasons. An IdP can also supplement outsourced authentication with additional authentication; for example, it might do password authentication against LDAP or Active Directory and then have an additional per-user MFA challenge that it manages in some database of its own.

(Some IdPs can manage users and their authentication entirely internally, but then you have to maintain, update, and protect their user and authentication database. If you already have an existing source of this, you might as well use it.)

OIDC also has a discovery protocol that is intended to let you find the OIDC IdP URLs for any particular user on any particular domain, so that a user can tell some system that they're 'me@example.org' and the system can find the right OIDC IdP from there. This discovery protocol is part of WebFinger, which means that to really run an OIDC IdP, you need something to answer WebFinger requests and provide the necessary data from RFC 7033. WebFinger isn't specific to OIDC (it's used on the Fediverse, for example) and some of the data people may want you to provide for them is user specific, like their avatar.

(I believe that OIDC IdPs usually or always require clients to be registered and assigned a client secret, so this discoverability has some limits to what you can use it for.)

PS: It's possible to set up a very janky testing OIDC IdP, but there are ways that are almost as easy and somewhat more industrial and workable.

(I'm trying to understand this whole area for work reasons.)