Chris's Wiki :: blog/web/ModernSimpleWebServerFeatures

Thinking about what you'd want in a modern simple web server

May 21, 2025

Over on the Fediverse, I said:

I'm currently thinking about what you'd want in a simple modern web server that made life easy for sites that weren't purely static. I think you want CGI, FastCGI, and HTTP reverse proxying, plus process supervision. Automatic HTTPS of course. Rate limiting support, and who knows what you'd want to make it easier to deal with the LLM crawler problem.

(This is where I imagine a 'stick a third party proxy in the middle' mode of operation.)

What I left out of my Fediverse post is that this would be aimed at small scale sites. Larger, more complex sites can and should invest in the power, performance, and so on of headline choices like Apache, Nginx, and so on. And yes, one obvious candidate in this area is Caddy, but at the same time something that has "more scalable" (than alternatives) as a headline features is not really targeting the same area as I'm thinking of.

This goal of simplicity of operation is why I put "process supervision" into the list of features. In a traditional reverse proxy situation (whether this is FastCGI or HTTP), you manage the reverse proxy process separately from the main webserver, but that requires more work from you. Putting process supervision into the web server has the goal of making all of that more transparent to you. Ideally, in common configurations you wouldn't even really care that there was a separate process handling FastCGI, PHP, or whatever; you could just put things into a directory or add some simple configuration to the web server and restart it, and everything would work. Ideally this would extend to automatically supporting PHP by just putting PHP files somewhere in the directory tree, just like CGI; internally the web server would start a FastCGI process to handle them or something.

(Possibly you'd implement CGI through a FastCGI gateway, but if so this would be more or less pre-configured into the web server and it'd ship with a FastCGI gateway for this (and for PHP).)

This is also the goal for making it easy to stick a third party filtering proxy in the middle of processing requests. Rather than having to explicitly set up two web servers (a frontend and a backend) with an anti-LLM filtering proxy in the middle, you would write some web server configuration bits and then your one web server would split itself into a frontend and a backend with the filtering proxy in the middle. There's no technical reason you can't do this, and even control what's run through the filtering proxy and what's served directly by the front end web server.

This simple web server should probably include support for HTTP Basic Authentication, so that you can easily create access restricted areas within your website. I'm not sure if it should include support for any other sort of authentication, but if it did it would probably be OpenID Connect (OIDC), since that would let you (and other people) authenticate through external identity providers.

It would be nice if the web server included some degree of support for more or less automatic smart in-memory (or on-disk) caching, so that if some popular site linked to your little server, things wouldn't explode (or these days, if a link to your site was shared on the Fediverse and all of the Fediverse servers that it propagated to immediately descended on your server). At the very least there should be enough rate limiting that your little server wouldn't fall over, and perhaps some degree of bandwidth limits you could set so that you wouldn't wake up to discover you had run over your outgoing bandwidth limits and were facing large charges.

I doubt anyone is going to write such a web server, since this isn't likely to be the kind of web server that sets the world on fire, and probably something like Caddy is more or less good enough.

(Doing a good job of writing such a server would also involve a fair amount of research to learn what people want to run at a small scale, how much they know, what sort of server resources they have or want to use, what server side languages they wind up using, what features they need, and so on. I certainly don't know enough about the small scale web today.)

PS: One reason I'm interested in this is that I'd sort of like such a server myself. These days I use Apache and I'm quite familiar with it, but at the same time I know it's a big beast and sometimes it has entirely too many configuration options and special settings and so on.

Comments on this page:

By Ricky at 2025-05-22 08:48:30:

Hi! Does Lighttpd meet your requirements? I use it because it supports CGI, and I see FastCGI and reverse proxy options (although no process supervision), but it's been fairly useful for my light-weight needs?

By Ian Z aka nobrowser at 2025-05-22 11:31:11:

I use nginx -- or today freenginx -- because of its much cleaner (to me) configuration syntax, compared to either apache or lighty. Of course YMMV. And I would definitely consider an alternative of the kind you imagine, were it to magically appear.

By cks at 2025-05-22 17:19:27:

I've used lighttpd in the past but I don't really favor it any more for various reasons, and it doesn't really match what I'm looking for. Lighttpd does show the difficulties of this sort of thing, since you kind of do want a web server that is keeping up with the evolution of TLS and HTTP (although you may or may not want HTTP/3).

(Lighttpd may be more up to date these days than it used to be; I haven't looked.)

By Alcides Fonseca at 2025-05-26 02:37:21:

Have you looked into Passenger?

It replaces fastcgi/cgi with some custom Ruby & Python handling. touch tmp/restart.txt in your website folder, and it will reload the process. Minimal config needed.

I have used it forever for handling my Django websites, and I don't want anything else.

By jrs@outband.net at 2025-05-28 17:35:48:

I am heavy into the OpenBSD ecosystem, mainly for this reason, while openbsd does have a lot of downsides when you try and use it in large performance critical roles, I find it great for little infrastructure boxes. Almost everything is in the base system and their internal services tend to be simple and easy to use. In particular I like the obsd web server.

I do have one beef, their relayd. it is sort of like the obsd version of haproxy, and in this role it works pretty great, with that characteristic obsd well designed config language. However it also has the ability to act as a http reverse proxy. And I found this setup to be confusing and cryptic. I got it working in the end. but unlike everything else in obsd was not able to figure it out it via their man pages. I mean the information is all there, but I was unable to put it together to make a working http dispatcher. I think I just don't think the same way as the author.

A hint for future generations, it is not hard, I just found it confusing the first time.
   table <static_host> { 192.168.2.4 }
   table <dynamic_host> { 192.168.2.4 }
   http protocol dispatch {
     return error
     pass path "/*" forward to <static_host>
     pass path "/app/*" forward to <dynamic_host>
     }
   relay http_proxy {
     listen on 192.168.2.4 port 443 tls
     protocol dispatch
     forward to <static_host> port 8080 check http "/check" code 200
     forward to <dynamic_host> port 8081 check http "/check" code 200
     }
-- JRS

From 24.207.52.143 at 2025-06-19 17:01:41:

I would want X.509 client authentication as the other type of authentication (in my opinion, it is better than the other methods, in many ways). I would also want to not automatically redirect HTTP to HTTPS except for files that require authentication to access them. Configuration of MIME types is also helpful (and it should not require the character set to be UTF-8 or any other one; it should be definable).

Written on 21 May 2025.

«	The five platforms we have to cover when planning systems

Fedora's DNF 5 and the curse of mandatory too-smart output