2023-09-26
Some reasons to combine systemd-resolved with your private DNS resolver
Probably like many people, we have some machines that are set up
as local DNS resolvers. Originally we had one set for everyone,
both our own servers and other people's machines on our internal
networks, but after some recent
issues we want to make DNS resolution on our own critical servers
more reliable and are doing that partly by having a dedicated
private DNS resolver for our servers.
Right now all of our servers do DNS in the old fashioned way, with
a nsswitch.conf that tells them to use DNS and an /etc/resolv.conf
that points to our two (now three on some servers) DNS resolvers.
One of the additional measures I've been considering is whether we
want to consider using systemd-resolved
on some servers.
Systemd-resolved has two features that make it potentially attractive
for making server DNS more reliable. The obvious one is that it
normally has a small cache of name resolutions (the Cache=
configuration directive). Based on 'resolvectl statistics' on a
few machines I have that are running systemd-resolved, this cache
doesn't seem to get very big and doesn't get very high a hit rate,
even on machines that are just sitting there doing nothing (and so
are only talking to the same few hosts over and over again). I
certainly don't think we can count on this cache to do very much
if our normal DNS resolvers stop responding for some reason.
The second feature is much more interesting, and it's that
systemd-resolved will rapidly switch to another DNS resolver if
your initial one stops responding. In situations where you have
multiple DNS servers (for a given network link or global setting, because systemd-resolved thinks in those
terms), systemd-resolved maintains a 'current DNS server' and will
send all traffic to it. If this server stops responding, resolved
will switch over and then latch on whichever of your DNS servers
is still working. This makes the failure of your 'primary' DNS
server much less damaging than in a pure /etc/resolv.conf situation.
In normal resolv.conf handling, every program has to fail over
itself (and I think some runtime environments may always keep trying
the first listed 'nameserver' and waiting for it to time out).
The generally slow switching of nameservers listed in your resolv.conf means that you really want the first DNS resolver to stay responsive (whatever it is). Systemd-resolved makes it much less dangerous to add another DNS resolver along side your regular ones, as long as you can trust it to not give wrong answers. If it stops working, those systems using it will switch over to other DNS resolvers fast enough that very little will notice.
(Unfortunately getting those systems to switch back may be annoying, but in a sense you don't care whether or not they're using your special private DNS resolver that's just for them or one of your public DNS resolvers. If your public DNS resolvers get flooded by other people's traffic and stop responding, systemd-resolved will switch the systems back to your private DNS resolver again.)
PS: Of course there are configuration issues with systemd-resolved that you may need to care about, but very little is flawless.