Some reasons to combine systemd-resolved with your private DNS resolver

September 26, 2023

Probably like many people, we have some machines that are set up as local DNS resolvers. Originally we had one set for everyone, both our own servers and other people's machines on our internal networks, but after some recent issues we want to make DNS resolution on our own critical servers more reliable and are doing that partly by having a dedicated private DNS resolver for our servers. Right now all of our servers do DNS in the old fashioned way, with a nsswitch.conf that tells them to use DNS and an /etc/resolv.conf that points to our two (now three on some servers) DNS resolvers. One of the additional measures I've been considering is whether we want to consider using systemd-resolved on some servers.

Systemd-resolved has two features that make it potentially attractive for making server DNS more reliable. The obvious one is that it normally has a small cache of name resolutions (the Cache= configuration directive). Based on 'resolvectl statistics' on a few machines I have that are running systemd-resolved, this cache doesn't seem to get very big and doesn't get very high a hit rate, even on machines that are just sitting there doing nothing (and so are only talking to the same few hosts over and over again). I certainly don't think we can count on this cache to do very much if our normal DNS resolvers stop responding for some reason.

The second feature is much more interesting, and it's that systemd-resolved will rapidly switch to another DNS resolver if your initial one stops responding. In situations where you have multiple DNS servers (for a given network link or global setting, because systemd-resolved thinks in those terms), systemd-resolved maintains a 'current DNS server' and will send all traffic to it. If this server stops responding, resolved will switch over and then latch on whichever of your DNS servers is still working. This makes the failure of your 'primary' DNS server much less damaging than in a pure /etc/resolv.conf situation. In normal resolv.conf handling, every program has to fail over itself (and I think some runtime environments may always keep trying the first listed 'nameserver' and waiting for it to time out).

The generally slow switching of nameservers listed in your resolv.conf means that you really want the first DNS resolver to stay responsive (whatever it is). Systemd-resolved makes it much less dangerous to add another DNS resolver along side your regular ones, as long as you can trust it to not give wrong answers. If it stops working, those systems using it will switch over to other DNS resolvers fast enough that very little will notice.

(Unfortunately getting those systems to switch back may be annoying, but in a sense you don't care whether or not they're using your special private DNS resolver that's just for them or one of your public DNS resolvers. If your public DNS resolvers get flooded by other people's traffic and stop responding, systemd-resolved will switch the systems back to your private DNS resolver again.)

PS: Of course there are configuration issues with systemd-resolved that you may need to care about, but very little is flawless.

Written on 26 September 2023.
« Splitting our local DNS resolvers apart to serve different audiences
Uncertainties over adding IP ratelimits to your local DNS resolvers »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Sep 26 23:19:50 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.