Wandering Thoughts archives

The fading out of tcpwrappers and its idea

Once upon a time, tcpwrappers were a big thing in (Unix) host security. Plenty of programs supported the original TCP Wrapper library by Wietse Venema, and people wrote their own takes on the idea. But nowadays, tcpwrappers is clearly on the way out. It doesn't seem to be used very much any more in practice, fewer and fewer programs support it at all, and of the remaining ones that (still) do, some of them are removing support for it. This isn't exclusive to Wietse Venema's original version; the whole idea and approach just doesn't seem to be all that popular any more. So what happened?

I don't know for sure, but I think the simple answer is 'firewalls and operating system level packet filtering'. The core idea of tcpwrappers is application level IP access filtering, and it dates from an era where that was your only real choice. Very few things had support for packet filtering, so you had to do this in the applications (and in general updating applications is easier than updating operating systems). These days we have robust and well developed packet filtering in kernels and in firewalls, which takes care of much of the need for tcpwrappers stuff. In many cases, maintaining packet filtering rules may be easier than maintaining tcpwrappers rules, and kernel packet filtering has the advantage that it's centralized and so universally 'supported' by programs; in fact programs don't have any choice about it.

(Kernel packet filters can't do DNS lookups the way that tcpwrappers can, but using DNS lookups for anything except logging has fallen out of favour these days. Often people don't even want to do it for logging.)

Having written some code that used libwrap, I think that another issue is that the general sort of API that Venema's tcpwrappers has is one that's fallen out of favour. Even using the library, what you get is basically a single threaded black box. This works sort of okay if you're forking for each new connection, but it doesn't expose a lot of controls or a lot of information and it's going to completely fall down if you want to do more sophisticated things (or control the DNS lookups it does). Basically Venema's tcpwrappers works best for things that you could at least conceive of running out of inetd.

(It's not impossible to create an API that offers more control, but then you wind up with something that is more complex as well. And once you get more complex, what programs want out of connection matching becomes much more program-specific; consider sshd's 'Match' stuff as contrasted with Apache's access controls.)

Another way of putting it is that in the modern world, we've come to see IP-level access control as something that should be handled outside the program entirely or that's deeply integrated with the program (or both). Neither really fits the tcpwrappers model, which is more 'sitting lightly on top of the program'.

(Certainly part of the decline of tcpwrappers is that in many environments we've moved IP access controls completely off end hosts and on to separate firewalls, for better or worse.)

Upgrading machines versus reinstalling them

Yesterday I mentioned that we would be 'upgrading' the version of OmniOS on our fileservers not by using the OmniOS upgrade process but by reinstalling them. While this was partly forced by an OmniOS problem, it's actually our approach in general. We tend to take this for two reasons.

The first reason is that it leads to either simpler install instructions or more identical machines if you have to rebuild one, depending on how you approach rebuilding upgraded machines. If you upgraded a machine from OS version A to OS version B, in theory you should reinstall a replacement by going through the same process instead of directly installing OS version B. If you directly install OS version B, you have a simpler and faster install process but you almost never get an exactly identical machine.

(In fact until you actually do this as a test you can't be sure you even wind up with a fully functional replacement machine. It's always possible that there's something vital in your current build instructions that only gets set up right if you start from OS version A and then upgrade.)

The second reason is that customizations done on OS version A are not always still applicable or necessary on OS version B. Sometimes they've even become counterproductive. If you're upgrading, you have to figure out how to find these issues and then how to fix them up. If you're directly (re)installing OS version B, you get a chance to start from scratch and apply only what you need (in the form you now need it in) on OS version B, and you don't have to deal with juggling all sorts of things during the transition from version A to version B.

(Relatedly, you may have changed your mind or simply learned better since your install of OS version A. Doing a from-scratch reinstall is a great opportunity to update to what you feel is the current best practice for something.)

Mind you, there are machines and situations where in-place upgrades are less disruptive and easier to do than complete reinstalls. One of them is when the machine has complex local state that is hard to fence off or back up and restore; another is if a machine was heavily customized, especially in ad-hoc ways. And in-place upgrades can involve less downtime (especially if you don't have surplus machines to do complex juggling). This is a lot of why I do in-place live upgrades of my workstations.

I don't think I'm interested in containers

Containers are all the rage in system administration right now, and I can certainly see the appeal. So it feels more than a bit heretical to admit that I'm not interested in them, ultimately because I don't think they're an easy fit for our environment.

What it comes down to is two things. The first is that I think containers really work best in a situation where the 'cattle' model of servers is a good fit. By contrast, our important machines are not cattle. With a few exceptions we have only one of each machine today, so in a container world we would just be turning those singular machines into singular containers. While there are some wins for containers I'm not convinced they're very big ones and there are certainly added complexities.

The second is that we are pretty big on using different physical machines to get fault independence. As far as we're concerned it's a feature that if physical machine X dies for whatever reason, we only lose a single service. We co-locate services only infrequently and reluctantly. This obviously eliminates one of the advantages of containers, which is that you can run multiple containers on a single piece of hardware. A world where we run a base OS plus a single container on most servers is kind of a more complicated world than we have now and it's not clear what it gets us.

I can sort of imagine a world where we become a container based environment (even with our present split of services) and I can see some advantages to it. But it's clear that it would take a lot of work to completely redo everything in our environment as a substrate of base OS servers and then a strata of ready to go containers deployed on top of them, and while we'd get some things out of such a switch I'm not convinced we'd get a lot.

(Such a switch would be more like a green field rebuild from total scratch; we'd probably want to throw away everything that we do now. This is just not feasible for us for various reasons, budget included.)

So the upshot of all of this is that while I think containers are interesting as a technical thing and I vaguely keep track of the whole area, I'm not actually interested in them and I have no plans to explore them, try them out, and so on. I feel oddly embarrassed by this for reasons beyond the comfortable scope of this entry, but there it is whether I like it or not.

(I was much more optimistic a few years ago, but back then I was just theorizing. Ever since then I've failed to find a problem around here where I thought 'yes, containers will make my life simpler here and I should advocate for them'. Even my one temptation due to annoyance was only a brief flirtation before sense set in.)

One speed limit on your ability to upgrade your systems

One of the responses on Twitter to Ted Unangst's long term support considered harmful was this very interesting tweet:

[...] it's not "pain" - it just doesn't happen. At 2 weeks of planning + testing = 26 systems per year

This was eye-opening in a 'I hadn't thought about it that way before now' way. Like many insights, it's blindingly obvious in retrospect; of course how fast you can actually do an upgrade/update cycle determines how many of them you can do in a year (given various assumptions about manpower, parallelism, testing, and so on). And of course this limit applies across all of your systems. It's not just that you can only upgrade a given system so many times a year; it's that you get only so many upgrades in a year, period, across all of your systems.

(What the limit is depends very much on what systems you're trying to upgrade, since the planning, setup, and testing process will take different amounts of time for different systems.)

To upgrade systems more frequently, you have two options. First, you can reduce the time an upgrade cycle takes by speeding up or doing less planning, building, testing, and/or the actual deployment. Second, you can reduce the number of upgrades you need to do creating more uniform systems, so you amortize the time a cycle takes across more systems. If you have six special snowflakes running completely different OSes and upgrading each OS takes a month, you get twelve snowflake upgrades in a year (assuming you do nothing else). But if all six run the same OS in the same setup, you now get to upgrade all six of them more or less once a month (let's optimistically assume that deployment is a snap).

I see this as an interesting driver of uniformity (and at all levels, not just at the system level). Depending on how much pre-production testing you need and use, it's also an obvious driver of faster, better, and often more automated tests.

(Looking back I can certainly see cases where this 'we can only work so fast' stuff has been a limiting factor in our own work.)