Wandering Thoughts archives

My initial experience of using NSD as a simple authoritative DNS server

I've been running djb's tinydns for a long time, but for a while I've been planning to replace it someday. The direct reason for replacing tinydns is that I want support for IPv6, for new DNS record types, and for best practices things like rate limiting; the indirect reason is that all of djb's software has been abandoned and on the Internet, unmaintained software rots. When I found myself adding an AAAA record using raw bytes, my irritation started rising slowly and eventually pushed me over the edge.

After ruling out PowerDNS for entirely personal reasons, my two primary options were Knot DNS and NSD. I looked a bit at both and wound up settling on NSD for now, partly because OpenBSD has switched to NSD and we run OpenBSD nameservers at work, so I'm going to wind up having to work with it someday.

(Interested parties can see eg the Wikipedia comparison of DNS server software, which I found helpful in making sure I wasn't missing anything obvious.)

NSD on Fedora 23 is pleasingly easy to configure and get operating, with configuration split across multiple files so that I can drop in my customizations without having to change any standard files. NSD's configuration syntax is simple, straightforward, and minimal, which makes it easy to write. Anyways, I only needed to set a very few things (like what IP address to listen on and then basic information about my one zone). NSD uses Bind format zone files, which meant that I had to rewrite my existing tinydns zone file, but that's not a particularly big job with a small zone.

(Had I wanted to I could have set up NSD as a secondary and done a zone transfer to get a Bind-format zonefile, but I decided I wanted to hand-write a clean zone file. I did wind up doing AXFRs from both tinydns and NSD to verify that my new rewritten version of the zone was the same apart from some unimportant things, like SOA refresh times.)

So far I've been running NSD instead of tinydns for a few days and basically haven't noticed anything different. Things continue to work as they did before without any problems or fuss.

NSD does use (much) more memory than tinydns ever did, but I'm resigned to that. RAM is at least inexpensive these days and a resident set size of 20 to 30 Mbytes is small beans, even if tinydns used to have a total memory size of only a couple of Mbytes and a RSS in the hundreds of kilobytes. It's possible I could tune NSD to use less memory if I wanted to spend the time on it, which I don't particularly.

PS: based on scanning over the Knot documentation, I suspect that setting it up would have been equally easy (and I may try doing so someday, just out of curiosity). It's possible that Knot is easier if you want to do DNSSEC, which I don't particularly.

Why I don't think upgrading servers would save us much power

In a comment on this entry, Ben Cotton noted:

The other consideration for upgrading low-utilization servers is power (and thus cooling) efficiency. [...]

Although I haven't gone out and metered this with our current server fleet, my impression is that we wouldn't save very much here, if anything. Of course my impression may be wrong, so it's time to go exploring.

Almost all of our potentially upgradeable servers are basic entry level 1U servers (all except the one or two remaining Dell 2950s). My strong impression is that like 1U server prices and hard disk prices, 1U server power usage has basically stayed flat for the past relatively decent amount of time. What's changed is instead how much computation you got for your N watts.

(To a large extent this isn't too surprising, since a lot of the power usage is driven by the CPUs and at least Intel has been on fixed power budgets for their CPUs for some time. Of course, as time goes on you can get more performance from the lower powered CPUs instead of needing the high-power ones, assuming they're offered in servers you're interested in.)

Am I right about this? Well, the evidence is mixed. Our primary server generations today are SunFire X2100s and X2200s versus Dell R210 IIs and Dell R310 IIs. The SunFires are so old that it's hard to find power usage information online, but various sources seem to suggest that they idle at around 88 watts (and back in 2006 I measured one booting as using 130 watts). A Dell R210 II is claimed to idle at a measured 43.7 watts, which would be about half the power of a SunFire. On the other hand, an R310 seems to idle at almost 80 watts (per here, but that may not match our configuration), very close to the SunFire power at idle.

(I care about power at idle because most of our servers are idling most of the time.)

All of this assumes a one for one server replacement program, where we change an old server into a new, perhaps more power efficient server. Of course if you really want to save power, you need to consolidate servers. You can do this through virtualization or containers, or you can just start co-locating multiple services on one server; in either case, you'll win by converting some number of idle servers (all with their individual baseline idle power draws) into one almost idle server with a power draw hopefully only somewhat higher than a single old server.

Could we do this? Yes, to some extent, but it would take a sea change in our opinions about service isolation and management. Right now we put different services on different physical services both to isolate failures and to make our lives easier by decoupling things like server OS upgrade cycles for different services. Re-joining services (regardless of the means) would require changing this, although things like containers could somewhat deal with the server OS upgrade cycle issue.

There's a relationship between server utilization and server lifetime

In yesterday's entry on how old our servers are, I said that one reason we can keep servers for so long is that our CPU needs are modest, so old servers are perfectly fine. There is another way to put this, namely that almost all of our servers have low CPU utilization. It is this low CPU utilization that makes it practical and even sensible to keep using old, slow servers. Similarly, almost all of our servers have low RAM utilization, low disk IO utilization, and so on.

This leads me to the undoubtedly unoriginal observation that there's an obvious relationship between capacity utilization and how much pressure there is to upgrade your servers. If your servers are at low utilization in all of their dimensions (CPU, RAM, IO bandwidth, network bandwidth, etc), any old machine will do and there is little need to upgrade servers. But the more your servers are approaching their capacity limits, the more potential benefit there is of upgrading to new servers that will let you do more with them (especially if you have other constraints that make it hard to just add more servers, like power or space limits or a need for more capacity in a single machine).

It follows that anything that increases server utilization can drive upgrades. For example, containers and other forms of virtualization; these can take N separate low-utilization services, embody them all on the same physical server, and wind up collectively utilizing much more of the hardware.

(And the inverse is true; if you eschew containers et al, as we do, you're probably going to have a lot of services with low machine utilizations that can thus live happily on old servers as long as the hardware stays reliable.)

In one way all of this seems obvious: of course you put demanding services on new hardware because it's the fastest and best you have. But I think there's a use for taking a system utilization perspective, not just a service performance one; certainly it's a perspective on effective server lifetimes that hadn't really occurred to me before now.

(Concretely, if I see a heavily used server with high overall utilization, it's probably a server with a comparatively short lifetime.)

How old our servers are (as of 2016)

I was recently asked how old our servers are. This is a good question because, as noted, universities are cheap; as a result we probably run servers much longer than many people do. In fact we can basically wind up running servers into the ground, and we actually are somewhat doing that right now.

The first necessary disclaimer is that my group here only handles general departmental infrastructure on the non-undergraduate side of things (this split has its roots way back in history and is another story entirely). Also, while we try to have some reasonably powerful machines for people who need compute servers, we don't have the budget to have modern, big, up to date ones. Most professors who need heavy duty computation for their research programs buy machines themselves out of grant funding, and my understanding is that they turn over such compute servers much faster than we do (sometimes they then donate the old servers to us).

Over my time here, we've effectively had three main generations of general servers. When I started in 2006, we were in the middle of a Dell PowerEdge era; we had primarily 750s, 1650s, and 2950s. Most of these are now two generations out of productions, but we still have some in-production 2950s, as well as some 1950s that were passed on to us later.

(It's worth mentioning that basically all of these Dells have suffered from capacitor plague on their addon disk controller boards. They've survived in production here only because one of my co-workers is the kind of person who can and will unsolder bad capacitors to replace them with good ones.)

Starting in 2007 and running over the next couple of years we switched to SunFire X2100s and X2200s as our servers of choice, continuing more or less through when Oracle discontinued them. A number of these machines are still in production and not really planned for replacement soon (and we have a few totally unused ones for reasons beyond the scope of this entry). Since they're all at least five or so years old we kind of would like to turn them over to new hardware, but we don't feel any big rush right now.

(Many of these remaining SunFire based servers will probably get rolled over to new hardware once Ubuntu 16.04 comes out and we start rebuilding our remaining 12.04 based machines on 16.04. Generally OS upgrades are where we change hardware, since we're (re)building the machine anyways, and we like to not deploy a newly rebuilt server on ancient hardware that has already been running for N years.)

Our most recent server generation is Dell R210 IIs and R310 IIs (which, yes, are now sufficiently outdated that they're no longer for sale). This is what we consider our current server hardware and what we're using when we upgrade existing servers or deploy new ones. We still have a reasonable supply of unused and spare ones for new deployments, so we're not looking to figure out a fourth server generation yet; however, we'll probably need to in no more than a couple of years.

(We buy servers in batches when we have the money, instead of buying them one or two at a time on an 'as needed' basis.)

In terms of general lifetime, I'd say that we expect to get at least five years out of our servers and often we wind up getting more, sometimes substantially more (some of our 2950s have probably been in production for ten years). Server hardware seems to have been pretty reliable for us so this stuff mostly keeps running and running, and our CPU needs are usually relatively modest so old servers aren't a bottleneck there. Unlike with our disks, our old servers have not been suffering from increasing mortality over time; we just don't feel really happy running production servers on six or eight or ten year old hardware when we have a choice.

(Where the old servers actually tend to suffer is their RAM capacity; how much RAM we want to put in servers has been rising steadily.)

Some of our servers do need computing power, such as our departmental compute servers. There, the old servers look increasingly embarrassing, but there isn't much we can do about it until (and unless) we get the budget to buy relatively expensive new ones. In general, keeping up with compute servers is always going to be expensive, because the state of the art in CPUs, memory, and now GPUs keeps turning over relatively rapidly. In the mean time, our view is that we're at least providing some general compute resources to our professors, graduate students, and researchers; it's probably better than nothing, and people do use the servers.

PS: Our fileservers and their backends were originally SunFires in the first generation fileservers (in fact they kicked off our SunFire server era), but they're now a completely different set of hardware. I like the hardware they use a lot, but it's currently too expensive to become our generic commodity 1U server model.

Sidebar: What we do with our old servers

The short answer right now is 'put them on the shelf'. Some of them get used for scratch machines, in times when we just want to build a test server. I'm not sure what we'll do with them in the long run, especially the SunFire machines (which have nice ILOMs that I'm going to miss). We recently passed a number of our out of service SunFires on to the department's undergraduate computing people, who needed some more servers.

My views on clients for Lets Encrypt

To use Let's Encrypt, you need a client, as LE certificates are available only through their automated protocol. I can't say I've checked out all the available clients (there are a lot of them), but here are the three that I've actively looked at and explored. I stopped exploring clients after three because these meet my needs and pretty much work the way I want.

The official Let's Encrypt client is, well, the official client. It's big and has many features and can be kind of a pain (or at least slightly scary) to get going, depending on if it's already available as a system package. Its advantage is that it is the official client; it'll probably have the most up to date and complete protocol support, it's pretty much guaranteed to always work (LE isn't going to let their official client be broken for very long), and there are tons of people who are using it and can help you if you run into problems. If you don't really care and just want a single certificate right now, it's probably your easiest option. And it has magic integration with an ever growing collection of web servers and so on, so that it can do things like automatically reload newly-renewed certificates.

The drawback of the official client is that it's big and complicated. One of my major use cases for LE is certificates for test machines and services, where what I just want a cert issued with minimal fuss, bother, flailing around, and software installation. My client of choice for 'just give me a TLS certificate, okay?' cases like this is lego. You turn off the real web server on your test machine (if any), you run lego with a few obvious command line options, and you have a certificate. Done, and done fast. Repeat as needed. As a Go program, the whole thing is a single executable that I can copy around to test machines as required. Lego doesn't really automate certificate renewal, but this is not an issue for test usage.

(You do have to remember to (temporarily) allow HTTP or HTTPS in to your test machine, which is something that I've forgotten a few times.)

My other usage case for LE is as the regular TLS certificates for my personal site. Here I very much wanted a client that had a clear story for how to do automated certificate renewals in an unusual environment with a custom setup, and that I could understand, control, and trust. Perhaps the official client could have done that if I studied it enough, but I felt that it was more focused towards doing 'just trust us' magic in standard setups. The client I settled on for this is Hugo Landau's acmetool, which is refreshingly free of magic. The simple story of how you get automatic certificate renewals and service reloads is that acmetool supports running arbitrary scripts after a certificate has been renewed. Just set up a script that does whatever you need, put an appropriate acmetool command in a once-a-day crontab entry, and you're basically done. One of the reasons that I like acmetool so much is that I think its way of handling the whole process is the correct approach. As its README says, it's intended to work like 'make' (which is a solidly proven model), and I think the whole approach of running arbitrary scripts on certificate renewal is the right non-magic way to handle getting services to notice the renewed certificates.

(Acmetool also has some clever and handy tricks, but that's something for another entry.)

Unsurprisingly, acmetool requires a certain amount of work to set up and configure (unlike lego, which is 'run and done'). But after that, so far it has been something I can completely ignore. I rather look forward to being able to not think about TLS certificate renewal on my personal site at all, instead of having to remember it once a year or so.

(The necessary disclaimer is that it hasn't yet been 60 days since I started using LE and acmetool, so I haven't had it go through the certificate renewal process. If I was more patient, I'd have waited longer to write this entry. But as it is, I think acmetool's fundamental model is sound so I'm fairly confident that everything's going to be fine.)

Some notes on OpenSSH's optional hostname canonicalization

As I mentioned in my entry on how your SSH keys are a potential information leak, I want to stop offering my ssh public keys to all hosts and instead only offer them to our hosts. The fundamental reason that I wasn't doing this already is that I make heavy use of short hostnames, either entirely without a domain or with only our local subdomain (ie, hostnames like apps0 or comps0.cs). When you use short hostnames, OpenSSH's relatively limited 'Host ...' matching power means that it's easiest to just say:

Host *
  IdentityFile ....

This has the effect that you offer your public keys to everything.

There are two ways to deal with this. First, you can use relatively complex Host matching. Second, you can punt by telling ssh to canonicalize the hostnames you typed on the command line to their full form and then matching on the full domain name. This has a number of side effects, of course; for instance, you'll always record the full hostnames in your known_hosts file.

Hostname canonicalization is enabled with 'CanonicalizeHostname yes'. This can be in a selective stanza in your .ssh/config, so you can disallow it for certain hostname patterns; for instance, you might want to do this for a few crucial hosts so that you aren't dependent on ssh's canonicalization process working right in order to talk to them. CanonicalDomains and CanonicalizeMaxDots are well documented in the ssh_config manpage; the only tricky bit is that the former is space-separated, eg:

CanonicalDomains sub.your.domain your.domain

The CanonicalizePermittedCNAMEs setting made me scratch my head initially, but it has to do with (internal) hostname aliases set up via DNS CNAMEs. We have some purely internal 'sandbox' networks in a .sandbox DNS namespace, and we have a number of CNAMEs for hosts in them in the internal DNS view of our normal subdomain, for both convenience and uniformity with their external names. In this situation, if I did 'ssh acname', OpenSSH would normally fail to canonicalize acname as a safety measure. By setting CanonicalizePermittedCNAMEs, I can tell OpenSSH that hosts in our subdomain pointing to .sandbox names is legitimate and expected. So I set up:

CanonicalizePermittedCNAMEs *.sub.our.dom:*.sandbox,*.sub.our.dom

I don't know if explicitly specifying our normal subdomain as a valid CNAME target is required. I threw it in as a precaution and haven't tested it (partly because I didn't feel like fiddling with our DNS data just to find out).

Although it's not documented, OpenSSH appears to do its hostname canonicalization by doing direct DNS queries itself. This will presumably bypass any special nsswitch.conf settings you have for hostname lookups. Note that although OpenSSH is using DNS here, it only cares about the forward lookup (of name to IP), not what the reverse lookup of the eventual host's IP is.

I've been experimenting with having OpenSSH do this hostname canonicalization for a few weeks now. So far everything seems to have worked fine, and I haven't noticed any delays or hiccups in making new SSH connections (which was one of the things I was worried about). Of course we haven't had any DNS glitches or failures over that time, either (at least none we know about).

Sidebar: Why OpenSSH cares about CNAMEs during canonicalization (I think)

I assume that this is because if OpenSSH was willing to follow CNAMEs wherever they went, an attacker with a certain amount of access to your DNS zone could more or less silently redirect existing or new names in your domain off to outside hosts. You would see the reassuring message of, say:

Warning: Permanently added 'somehost.sub.your.domain' (RSA) to the list of known hosts.

but your connection would actually be going to otherhost.attacker.com because that's where the CNAME points.

You still get sort of the same issue if you don't have hostname canonicalization turned on (because then the system resolver will presumably be following that CNAME too), but then at least the message about adding keys doesn't explicitly claim that the hostname is in your domain.