Wandering Thoughts archives

I should keep and check notes even on my own little problems

I mentioned yesterday that I had a serious issue when I installed a VMWare Workstation update, going from 12.1.0 to 12.1.1. I wound up being very grumpy about it, disrupted the remaining part of my work day, filed a support request with VMWare, and so on, and eventually VMWare support came through with the cause and a workaround..

It turns out that I could have avoided all of that, because I ran into this same problem back when I upgraded to Fedora 23. At the time I did my Internet research, found the workaround, and applied it to my machine. This means that I could have proceeded straight to re-doing the workaround if I'd remembered this. Or, more likely, if I'd kept good notes on the problem then and remembered to read them this time.

We try to make and keep good notes for problems on our production systems, or even things that we run into in testing things for our work environment; we have an entire system for it. But I generally don't bother doing the same sort of thing for my own office workstation; when I find and fix problems and issues I may take some notes, but they're generally sketchy, off the cuff, and not centrally organized. And partly because of this, I often don't think to check them; I think I just assume I'm going to remember things about my own workstation (clearly this is wrong).

So, stating the obvious: I would be better off if I kept organized notes about what I had to do to fix problems and get various things going on my workstation, and put the notes into one place in some format (perhaps a directory with text files). Then I could make it a habit to look there before I do some things, or at least when I run into a problem after I do something.

Also, when I make these notes I should make them detailed, including dates and versions of what they're about. It turns out that I actually had some very sketchy notes about this problem from when I upgraded to Fedora 23 (they were some URLs that turned out to be discussions about the issue), but they didn't have a date or say 'this applied when I upgraded to Fedora 23 with VMWare 12' or anything like that. So when I stumbled over the file and skimmed it, I didn't realize that the URLs were still relevant; I skipped that because I assumed that of course it had to be outdated.

(I'm sure that when I wrote the note file in the first place I assumed that I'd always remember the context. Ha ha, silly me, I really should know better by now. Especially since I've written more than one entry here about making just that assumption and being wrong about it.)

How to get Unbound to selectively add or override DNS records

Suppose, not entirely hypothetically, that you're using Unbound and you have a situation where you want to shim some local information into the normal DNS data (either adding records that don't exist naturally or overriding some that do). You don't want to totally overwrite a zone, just add some things. The good news is that Unbound can actually do this, and in a relatively straightforward way (unlike, say, Bind, where if this is possible at all it's not obvious).

You basically have two options, depending on what you want to do with the names you're overriding. I'll illustrate both of these:

local-zone: example.org typetransparent
local-data: "server.example.org A 8.8.8.8"

Here we have added or overridden an A record for server.example.org. Any other DNS records for server.example.org will be returned as-is, such as MX records.

local-zone: example.com transparent
local-data: "server.example.com A 9.9.9.9"

We've supplied our own A record for server.example.com, but we've also effectively deleted all other DNS records for it. If it has an MX record or a TXT record or what have you, those records will not be visible. For any names in transparent local-data zones, you are in complete control of all records returned; either they're in your local-data stanzas, or they don't exist.

Note that if you just give local-data for something without a local-zone directive, Unbound silently makes it into such a transparent local zone.

Transparent local zones have one gotcha, which I will now illustrate:

local-zone: example.net transparent
local-data: "example.net A 7.7.7.7"

Because this is a transparent zone and we haven't listed any NS records for example.net as part of our local data, people will not be able to look up any names inside the zone even though we don't explicitly block or override them. Of course if we did list some additional names inside example.net as local-data, people would be able to look up them (and only them). This can be a bit puzzling until you work out what's going on.

(Since transparent local zones are the default, note that this happens if you leave out the local-zone or get the name wrong by mistake or accident.)

As far as I know, there's no way to use a typetransparent zone but delete certain record types for some names, which you'd use so you can do things like remove all MX entries for some host names. However, Unbound's idea of 'zones' don't have to map to actual DNS zones, so you can do this:

local-zone: example.org typetransparent
local-data: "server.example.org A 8.8.8.8"
# but:
local-zone: www.example.org transparent
local-data: "www.example.org A 8.8.8.8"

By claiming www.example.org as a separate transparent local zone, this allows us to delete all records for it but the A record that we supply; this would remove, say, MX entries. Since I just tried this out, note that a transparent local zone with no data naturally doesn't blank out anything, so if you want to totally delete a name's records you need to supply some dummy record (eg a TXT record).

(We've turned out to not need to do this right now, but since I worked out how to do it I want to write it down before I forget.)

Unbound illustrates the Unix manpage mistake with its ratelimits documentation

Our departmental recursive nameservers are based on OpenBSD, which has recently switched from BIND to Unbound and NSD. As a result of this, we've been in the process of setting up a working Unbound configuration. In the process of this we ran into an interesting issue.

A relatively current unbound.conf manpage has this to say about (some) ratelimiting options (I'm excerpting here):

ratelimit: <number or 0>

Enable ratelimiting of queries sent to the nameserver for performing recursion. If 0, the default, it is disabled. [...] For example, 1000 may be a suitable value to stop the server from being overloaded with random names, and keeps unbound from sending traffic to the nameservers for those zones.

ratelimit-for-domain: <domain> <number qps>

Override the global ratelimit for an exact match domain name with the listed number. [...]

So you set up an Unbound configuration that contains the following:

# apparent good practice
ratelimit: 1000

# but let's exempt our own zones from it,
# just in case.
ratelimit-for-domain: utoronto.ca 0

Congratulations, on at least the OpenBSD version of Unbound you have just blown your own foot off; you'll probably be unable to resolve anything in utoronto.ca. If you watch the logs sufficiently carefully, you can eventually spot a little mention that your query for say the A record of www.utoronto.ca has been ratelimited.

(If you're writing a moderately complicated Unbound configuration for the first time, it may take you some time to reach this point instead of suspecting that you have screwed something up in other bits of the configuration.)

What has happened is that you have not read the manpage with the necessary closeness for a true Unix manpage. You see, the manpage does not come out and actually say that ratelimit-for-domain treats a ratelimit of 0 as unlimited. It just looks like it should, because ratelimit-for-domain is a more specialized version of plain ratelimit so you'd certainly assume that they treat their number argument in the same way. And of course that would be the sensible thing to do so you can do just what we're trying to do here.

This may or may not be an Unbound bug in either Unbound itself or in the unbound.conf manpage. Unix's minimalistic, legalistic 'close reading' history of both reading and writing manpages makes it impossible to tell, because this could be both intended and properly documented.

(In my opinion it is not well documented if it is intended, but that is a different argument. Classical style Unix manpages take specification-level terseness far too far for my tastes, partly for historical reasons. However this is not a winning argument to make with someone who likes this extreme terseness and 'pay attention to every word, both present and absent' approach; they will just tell you to read more carefully.)

There's a spectrum of 'pets versus cattle' in servers

One of the memes in modern operations is that of pets versus cattle. I've written about this before, but at the time I accepted the usual more or less binary pet versus cattle split that's usually put forward. I've now shifted to feeling that there is a spectrum along the line between pets and cattle, so today I'm going to write down four spots I see on that line.

Total pets (classical pets) are artisanal servers, each one created and maintained completely by hand. You're lucky if there's any real documentation on what a machine's setup is supposed to be; there probably isn't. Losing a server probably means restoring configuration files from backups in order to get it back into service. This is the traditional level that a small or disorganized place operates at (or at least is stereotyped to operate at).

At one step along the line you have a central, global store of all configuration information and build instructions; for instance, you have the master copy of all changed configuration files in that central place, and a rule that you always modify the master version and copy it to a server. However, you build and maintain machines almost entirely by hand (although following your build documents and so on). You can recreate servers easily but they are still maintained mostly by hand, you troubleshoot them instead of reinstalling, and users will definitely notice if one suddenly vanishes. Possibly they have local state that has to be backed up and restored.

(This is how we build and maintain machines.)

Moving one more step towards cattle is when you have fully automated configuration management and more or less fully automated builds, but you still care about specific servers. You need to keep server <X> up, diagnose it when it has problems, and so on; you cannot simply deal with problems by 'terminate it and spin up another', and people will definitely notice if a given server goes down. One sign of this is that your servers have names and people know them.

Total cattle is achieved when essentially all servers can be 'fixed' by simply terminating them and spinning up another copy made from scratch, and your users won't notice this. Terminate and restart is your default troubleshooting method and you may even make servers immutable once spun up (so maintaining a server is actually 'terminate this instance and spin up an updated instance'). Certainly maintenance is automated. You never touch individual servers except in truly exceptional situations.

(Total cattle is kind of an exaggeration. Even very cattle-ish places seem to accept that there are situations where you want to troubleshoot weird problems instead of trying to assume that 'terminate and restart' can be used to fix everything.)

The three types of challenges that Let's Encrypt currently supports

I've recently been working to understand Let's Encrypt a bit better, and in particular to understand the different sorts of challenges (ie, ways of proving that you control a hostname) that they currently support.

(In general, Alex Peattie's A guide to creating a LetsEncrypt client from scratch has a great overview of the overall flow of the challenge process.)

Right now, there are three challenges; 'HTTP', 'DNS', and what is called 'TLS-SNI'.

• in the DNS challenge, you add a specific TXT record with specific contents (specified by Let's Encrypt) to your DNS zone, proving control over the DNS for the host that you want a certificate for. One of the drawbacks of this challenge is that DNS information doesn't necessarily update immediately, so it may take some time before LE sees your new DNS TXT record and can issue your certificate.

• in the 'TLS-SNI' challenge, you run a server on a specific port (currently only port 443, HTTPS) that has a self-signed certificate with a specific set of Server Name Indicator records; you can read the gory details in the ACME draft.

  The drawback of TLS-SNI is that it is difficult to handle the challenge through an existing HTTPS server (although not impossible). You'd basically need to configure a new virtual host for the special self-signed certificate you need to use for the TLS-SNI challenge, and it has a special magic name.

  As far as I know, the TLS-SNI challenge doesn't require you to be running a real web server on port 443. The challenge simply requires a TLS server that will return the special certificate.

• in the 'HTTP' challenge (the most well-known one), you put a magic file in /.well-known/acme-challenge/ on your web server and LE fetches it. At the moment the initial request from LE is always made via HTTP, but LE will accept a redirection to the HTTPS version of the URL so you can still do a universal HTTP to HTTPS redirection (I don't know if it verifies the HTTPS certificate in this case, but you should probably assume that it does).

  The HTTP challenge is the easiest challenge to satisfy without disrupting existing services, since you just need to serve a file at a new URL. General purpose web servers can usually do this without any reconfiguration at all; you just need a couple of new directories in the filesystem.

Under normal circumstances, only the DNS challenge can easily be done from a different machine than the one you're getting the TLS certificate for; HTTP and TLS-SNI both want to talk to the actual machine itself. However, if you're willing to play evil firewall games with port redirections, it's probably possible to satisfy HTTP and TLS-SNI challenges from another machine. You simply redirect inbound port 80 and/or port 443 traffic to the normal machine off to your challenge-handling machine, and then make sure you're running a LE client that is willing to cooperate with this.

I suspect that client support for HTTP challenges is higher than support for TLS-SNI. Thus if you only want to allow one port through your firewall to your target machine, the easiest one is probably HTTP. Note that there is no need to run a general purpose HTTP server on your machine to handle LE challenges; there are any number of clients that just run their own HTTP server for the duration of the challenge, including the official LE client.

In theory the TLS-SNI challenge could be extended to work against other TLS ports, such as IMAPS, POP3S, or SMTPS; this might allow your existing IMAP server or MTA to handle a LE challenge without needing additional ports opened or additional services running (some of the time) on the machine. In practice I suspect that this is not a Let's Encrypt priority and is unlikely to happen any time soon.

(It would also require SNI support in your IMAP or SMTP server, and having them reconfigured more or less on the fly to serve the magic TLS-SNI certificate to people who ask for the right server name.)

Let's Encrypt certificates can be used for more than HTTPS

The Let's Encrypt website basically only talks about using its certificates for (HTTPS) websites, and their FAQ is a little bit silent on this. So let me say it out loud:

Let's Encrypt certificates can be used for pretty much any TLS service, not just HTTPS websites.

In particular, you can absolutely use Let's Encrypt certificates for IMAP servers and MTAs (for SMTP). The LE documentation won't tell you how to set this up, the official client doesn't have any support for it as far as I know, and the LE 'prove that you control this host' challenge process doesn't have any provisions for doing it through IMAP or SMTP servers, but it can certainly be done. And if you already have a certificate issued to a host for HTTPS, you can also use that certificate for your IMAP server, your SMTP server, and so on.

Based on my brief experience, the thing that may give you the most annoyance is wrangling certificate chain issues. Web browsers are used to filling in the blanks on their own and web servers are generally willing to accept just about any old set of certificates as your certificate chain. Other server software can be much pickier (such as insisting on only necessary certificates and in the correct order), and things like IMAP clients may be less willing to fetch intermediate certificates on their own. Complicating this is how LE has multiple certificate chains (or at least they used to, right now you may just use their X3 intermediate certificate).

(I didn't take notes the last time I had to do this, so I don't have any specific directions for things like Dovecot or Exim.)

Of course, just as with web servers you'll need to arrange to handle the relatively rapid LE certificate rollovers. Some servers are nice enough to automatically notice new certificates and just start using them; others will require restarting or signalling, which you'll need to connect up to whatever system you're using for this (I have my own opinions here). If you're counting on the official client's magical handling of this for some web servers, well, now you get to do some work.

(In time I'm sure that third party clients will start supporting various non-HTTPS servers, both generating the certificate setups they require and knowing how to restart them. I suppose the support may even appear in the official client.)

My initial experience of using NSD as a simple authoritative DNS server

I've been running djb's tinydns for a long time, but for a while I've been planning to replace it someday. The direct reason for replacing tinydns is that I want support for IPv6, for new DNS record types, and for best practices things like rate limiting; the indirect reason is that all of djb's software has been abandoned and on the Internet, unmaintained software rots. When I found myself adding an AAAA record using raw bytes, my irritation started rising slowly and eventually pushed me over the edge.

After ruling out PowerDNS for entirely personal reasons, my two primary options were Knot DNS and NSD. I looked a bit at both and wound up settling on NSD for now, partly because OpenBSD has switched to NSD and we run OpenBSD nameservers at work, so I'm going to wind up having to work with it someday.

(Interested parties can see eg the Wikipedia comparison of DNS server software, which I found helpful in making sure I wasn't missing anything obvious.)

NSD on Fedora 23 is pleasingly easy to configure and get operating, with configuration split across multiple files so that I can drop in my customizations without having to change any standard files. NSD's configuration syntax is simple, straightforward, and minimal, which makes it easy to write. Anyways, I only needed to set a very few things (like what IP address to listen on and then basic information about my one zone). NSD uses Bind format zone files, which meant that I had to rewrite my existing tinydns zone file, but that's not a particularly big job with a small zone.

(Had I wanted to I could have set up NSD as a secondary and done a zone transfer to get a Bind-format zonefile, but I decided I wanted to hand-write a clean zone file. I did wind up doing AXFRs from both tinydns and NSD to verify that my new rewritten version of the zone was the same apart from some unimportant things, like SOA refresh times.)

So far I've been running NSD instead of tinydns for a few days and basically haven't noticed anything different. Things continue to work as they did before without any problems or fuss.

NSD does use (much) more memory than tinydns ever did, but I'm resigned to that. RAM is at least inexpensive these days and a resident set size of 20 to 30 Mbytes is small beans, even if tinydns used to have a total memory size of only a couple of Mbytes and a RSS in the hundreds of kilobytes. It's possible I could tune NSD to use less memory if I wanted to spend the time on it, which I don't particularly.

PS: based on scanning over the Knot documentation, I suspect that setting it up would have been equally easy (and I may try doing so someday, just out of curiosity). It's possible that Knot is easier if you want to do DNSSEC, which I don't particularly.

Why I don't think upgrading servers would save us much power

In a comment on this entry, Ben Cotton noted:

The other consideration for upgrading low-utilization servers is power (and thus cooling) efficiency. [...]

Although I haven't gone out and metered this with our current server fleet, my impression is that we wouldn't save very much here, if anything. Of course my impression may be wrong, so it's time to go exploring.

Almost all of our potentially upgradeable servers are basic entry level 1U servers (all except the one or two remaining Dell 2950s). My strong impression is that like 1U server prices and hard disk prices, 1U server power usage has basically stayed flat for the past relatively decent amount of time. What's changed is instead how much computation you got for your N watts.

(To a large extent this isn't too surprising, since a lot of the power usage is driven by the CPUs and at least Intel has been on fixed power budgets for their CPUs for some time. Of course, as time goes on you can get more performance from the lower powered CPUs instead of needing the high-power ones, assuming they're offered in servers you're interested in.)

Am I right about this? Well, the evidence is mixed. Our primary server generations today are SunFire X2100s and X2200s versus Dell R210 IIs and Dell R310 IIs. The SunFires are so old that it's hard to find power usage information online, but various sources seem to suggest that they idle at around 88 watts (and back in 2006 I measured one booting as using 130 watts). A Dell R210 II is claimed to idle at a measured 43.7 watts, which would be about half the power of a SunFire. On the other hand, an R310 seems to idle at almost 80 watts (per here, but that may not match our configuration), very close to the SunFire power at idle.

(I care about power at idle because most of our servers are idling most of the time.)

All of this assumes a one for one server replacement program, where we change an old server into a new, perhaps more power efficient server. Of course if you really want to save power, you need to consolidate servers. You can do this through virtualization or containers, or you can just start co-locating multiple services on one server; in either case, you'll win by converting some number of idle servers (all with their individual baseline idle power draws) into one almost idle server with a power draw hopefully only somewhat higher than a single old server.

Could we do this? Yes, to some extent, but it would take a sea change in our opinions about service isolation and management. Right now we put different services on different physical services both to isolate failures and to make our lives easier by decoupling things like server OS upgrade cycles for different services. Re-joining services (regardless of the means) would require changing this, although things like containers could somewhat deal with the server OS upgrade cycle issue.

There's a relationship between server utilization and server lifetime

In yesterday's entry on how old our servers are, I said that one reason we can keep servers for so long is that our CPU needs are modest, so old servers are perfectly fine. There is another way to put this, namely that almost all of our servers have low CPU utilization. It is this low CPU utilization that makes it practical and even sensible to keep using old, slow servers. Similarly, almost all of our servers have low RAM utilization, low disk IO utilization, and so on.

This leads me to the undoubtedly unoriginal observation that there's an obvious relationship between capacity utilization and how much pressure there is to upgrade your servers. If your servers are at low utilization in all of their dimensions (CPU, RAM, IO bandwidth, network bandwidth, etc), any old machine will do and there is little need to upgrade servers. But the more your servers are approaching their capacity limits, the more potential benefit there is of upgrading to new servers that will let you do more with them (especially if you have other constraints that make it hard to just add more servers, like power or space limits or a need for more capacity in a single machine).

It follows that anything that increases server utilization can drive upgrades. For example, containers and other forms of virtualization; these can take N separate low-utilization services, embody them all on the same physical server, and wind up collectively utilizing much more of the hardware.

(And the inverse is true; if you eschew containers et al, as we do, you're probably going to have a lot of services with low machine utilizations that can thus live happily on old servers as long as the hardware stays reliable.)

In one way all of this seems obvious: of course you put demanding services on new hardware because it's the fastest and best you have. But I think there's a use for taking a system utilization perspective, not just a service performance one; certainly it's a perspective on effective server lifetimes that hadn't really occurred to me before now.

(Concretely, if I see a heavily used server with high overall utilization, it's probably a server with a comparatively short lifetime.)

How old our servers are (as of 2016)

I was recently asked how old our servers are. This is a good question because, as noted, universities are cheap; as a result we probably run servers much longer than many people do. In fact we can basically wind up running servers into the ground, and we actually are somewhat doing that right now.

The first necessary disclaimer is that my group here only handles general departmental infrastructure on the non-undergraduate side of things (this split has its roots way back in history and is another story entirely). Also, while we try to have some reasonably powerful machines for people who need compute servers, we don't have the budget to have modern, big, up to date ones. Most professors who need heavy duty computation for their research programs buy machines themselves out of grant funding, and my understanding is that they turn over such compute servers much faster than we do (sometimes they then donate the old servers to us).

Over my time here, we've effectively had three main generations of general servers. When I started in 2006, we were in the middle of a Dell PowerEdge era; we had primarily 750s, 1650s, and 2950s. Most of these are now two generations out of productions, but we still have some in-production 2950s, as well as some 1950s that were passed on to us later.

(It's worth mentioning that basically all of these Dells have suffered from capacitor plague on their addon disk controller boards. They've survived in production here only because one of my co-workers is the kind of person who can and will unsolder bad capacitors to replace them with good ones.)

Starting in 2007 and running over the next couple of years we switched to SunFire X2100s and X2200s as our servers of choice, continuing more or less through when Oracle discontinued them. A number of these machines are still in production and not really planned for replacement soon (and we have a few totally unused ones for reasons beyond the scope of this entry). Since they're all at least five or so years old we kind of would like to turn them over to new hardware, but we don't feel any big rush right now.

(Many of these remaining SunFire based servers will probably get rolled over to new hardware once Ubuntu 16.04 comes out and we start rebuilding our remaining 12.04 based machines on 16.04. Generally OS upgrades are where we change hardware, since we're (re)building the machine anyways, and we like to not deploy a newly rebuilt server on ancient hardware that has already been running for N years.)

Our most recent server generation is Dell R210 IIs and R310 IIs (which, yes, are now sufficiently outdated that they're no longer for sale). This is what we consider our current server hardware and what we're using when we upgrade existing servers or deploy new ones. We still have a reasonable supply of unused and spare ones for new deployments, so we're not looking to figure out a fourth server generation yet; however, we'll probably need to in no more than a couple of years.

(We buy servers in batches when we have the money, instead of buying them one or two at a time on an 'as needed' basis.)

In terms of general lifetime, I'd say that we expect to get at least five years out of our servers and often we wind up getting more, sometimes substantially more (some of our 2950s have probably been in production for ten years). Server hardware seems to have been pretty reliable for us so this stuff mostly keeps running and running, and our CPU needs are usually relatively modest so old servers aren't a bottleneck there. Unlike with our disks, our old servers have not been suffering from increasing mortality over time; we just don't feel really happy running production servers on six or eight or ten year old hardware when we have a choice.

(Where the old servers actually tend to suffer is their RAM capacity; how much RAM we want to put in servers has been rising steadily.)

Some of our servers do need computing power, such as our departmental compute servers. There, the old servers look increasingly embarrassing, but there isn't much we can do about it until (and unless) we get the budget to buy relatively expensive new ones. In general, keeping up with compute servers is always going to be expensive, because the state of the art in CPUs, memory, and now GPUs keeps turning over relatively rapidly. In the mean time, our view is that we're at least providing some general compute resources to our professors, graduate students, and researchers; it's probably better than nothing, and people do use the servers.

PS: Our fileservers and their backends were originally SunFires in the first generation fileservers (in fact they kicked off our SunFire server era), but they're now a completely different set of hardware. I like the hardware they use a lot, but it's currently too expensive to become our generic commodity 1U server model.

Sidebar: What we do with our old servers

The short answer right now is 'put them on the shelf'. Some of them get used for scratch machines, in times when we just want to build a test server. I'm not sure what we'll do with them in the long run, especially the SunFire machines (which have nice ILOMs that I'm going to miss). We recently passed a number of our out of service SunFires on to the department's undergraduate computing people, who needed some more servers.