Things that systemd-resolved is not for (as of systemd 251)

February 10, 2023

Today, in a burst of enthusiasm, I converted my office and home Fedora desktops to using systemd-resolved (as foreshadowed). The result has taught me a lot, and the general thing I've learned is that systemd-resolved is much more narrowly scoped than I thought it was. It is not a general system for handling the potential complexity of name resolution in a multi-faceted environment; instead, it's focused on managing a world where network connections come and go, and each network connection may come with a DNS server you're supposed to use on it and some DNS names it's good for.

This leaves a lot of things that systemd-resolved is not for. Here is my current list, as of Fedora's systemd 250 and 251.

  • it's not for arbitrary mappings of names to DNS servers, independent of network interfaces. If you simply use a random interface as an attachment point to map a set of names to a DNS server, you may get an unpleasant surprise, because under some circumstances resolved will insist on trying to reach the DNS server through that interface. This includes if the listed DNS server is 127.0.0.1:53. This makes sense in resolved's narrow scope of 'this network comes with a DNS server (for some names)'.

    This means that if you have a local resolving DNS server, for example to implement some local names, the only safe place to specify it and its domains is in the global DNS and Domains settings in resolved.conf.

    (Possibly you can also attach them to 'lo', the loopback interface.)

If you just want to add some local names, your best option is probably to put them in /etc/hosts. Resolved will turn this into synthetic DNS data for you without the contortions you need with, eg, Unbound. If you need to steer a bunch of names and zones to a bunch of different servers, I believe you're probably going to have to configure a local resolving DNS server (like Unbound) and then point resolved at it in your resolved.conf.

  • it's not for having any particular order to what's in your resolv.conf search directive (or what order resolved uses in its native DBus interface). Resolved can list domains there if you ask it to, but there's no specific control to what order they're in.

    (If you have multiple search domains where the same name can be in more than one, you may have strong opinions about what order they should be tried in.)

  • it's not for specifically adding search domains separate from how names are resolved. Resolved only adds search domains through their association with some specific DNS server (and thus generally with some specific interface that DNS server will be reached over), and afterward that DNS server will (or may) be used for them, no matter if you want to divert resolution to another DNS server when it becomes available.

If you want to specifically control your /etc/resolv.conf search outside of resolved's constraints, you need to manually construct a resolv.conf to your tastes and then change your nsswitch.conf to not use the 'resolve' mechanism (which goes straight to resolved and its unpredictable search order).

  • it's not for exactly duplicating traditional resolv.conf behavior in name resolution. In particular, as more or less documented in resolved.conf, the search order is only used for names with no dots (a 'single-label' hostname). In a normal DNS environment, if you have 'search example.org' and you do a DNS lookup for 'fred.bar', it will eventually look for 'fred.bar.example.org'. Not in resolved; the Domains are specifically only for single-label hostnames.

Using the resolv.conf 'search' for labels with dots has fallen out of favour lately (especially as new top level domains have proliferated), but in some environments it's historical practice that people may be rather attached to. However, at least today if you talk to resolved only via DNS, things still work, because the libraries still have the traditional behavior. Only the resolved DBus interface behaves this way, so now you may have two reasons to remove 'resolve' from the hosts: line in your nsswitch.conf.

  • it's not for environments where networks are giving you DNS resolution information you want or need to partially ignore (or override). Resolved has no particularly strong idea of priorities if two sources claim the same name and is not built to let you say things like 'don't actually add this name to the search list despite the network asking for it'. It's your job to fix that sort of thing upstream in the program that's sending new settings to resolved.

    (I think resolved generally assumes 'the last update wins', which is perfectly reasonable for its narrow focus. If you bring up a VPN, you probably want its opinions on DNS resolution to win.)

Systemd-resolved's narrow focus is perfectly fine in general. Having your DHCP client, your N VPN clients, and so on all fighting over your resolv.conf was a real problem, and systemd-resolved has solved it. I can't blame systemd-resolved for not being the all consuming center of handling DNS resolution complexity that I wish it was, and so far I can work around the things it's not for.

(And in some ways the resolved experience is better and easier to manage than my previous set of Unbound manipulations; putting static local names in /etc/hosts is simple and straightforward, for example.)

Sidebar: My home search domain case as an illustration

On my home desktop, I almost always have a Wireguard tunnel to work up, but I might not (for example, maybe the remote end of the tunnel is unavailable). Because we have a split horizon DNS setup, when the tunnel is up I need queries for our subdomain to be sent to our internal resolvers. But when the tunnel is down, some of our machines are still accessible and I want to still be able to reach them by their short names, so I can keep on typing 'ssh apps0' and 'ping apps0' instead of switching to their long forms.

What this means is that I want a search domain set all of the time, but I only want DNS queries diverted some of the time. Systemd-resolved isn't for this, because it assumes I should have the search domain bolted to DNS query redirection being possible (ie, the tunnel is up).


Comments on this page:

By Jason at 2023-02-11 09:41:00:

I recently started using systemd-resolved on my home network as well, and at first I had some issues with it not picking up the local mappings that I specified on my dnsmasq server. For some reason it would ignore the local mapping, follow the public CNAME, and return the IPv4 for its target address. I ended up figuring out that it was actually IPv6 that was throwing it off. In dnsmasq where I specified "address=/host.example.com/192.168.1.5", I also needed to specify "address=/host.example.com/::". After that, systemd-resolved started handling my local mappings correctly.

instead, it's focused on managing a world where network connections come and go, and each network connection may come with a DNS server you're supposed to use on it and some DNS names it's good for.

So if you're running a server farm, either physical systems or virtual machines/instances, is there any point in it?

I've found that a lot of OS changes in recent years appear to be software (systemd-resolved, netplan, cloud-init, etc) that assumes that you have a "dynamic" environment of some kind. Perhaps it's just the places I've worked, but I've never needed all the cleverness that comes with this software: after the system boots the first time, and gets it's initial configuration (e.g., Ansible), things don't change. Using the setup of yore (plain resolve.conf, interfaces(5) under Debian/Ubuntu) would be fine.

At most I'd might need cloud-init for first boot (e.g., under OpenStack), but the network part then becomes static until the system is retired.

All this cleverness does nothing to improve my life as a sysadmin, and sometimes makes things more difficult because things can change behind your back.

By cks at 2023-02-11 17:49:47:

I think this is potentially useful for static servers; you can use it to set up a split horizon DNS environment without needing any actual DNS servers that understand and serve both views. You have one set of standard resolving servers and another set that answer your internal names, then use systemd-resolved to steer internal names to the internal servers and everything else to the standard resolvers. This potentially beats the old way where you had to set up internal resolving DNS servers that knew how to steer internal names to your internal DNS servers.

(You can also use this to implement an 'overlay' DNS namespace, as Tailscale sort of does.)

From 193.219.181.219 at 2023-02-12 02:49:40:

But when the tunnel is down, some of our machines are still accessible and I want to still be able to reach them by their short names, so I can keep on typing 'ssh apps0' and 'ping apps0' instead of switching to their long forms.

I think you can still achieve this by setting a search domain in the global config (i.e. resolved.conf). I'm a bit unclear about whether it is special-cased in any other way besides not being tied to an interface. (But I generally avoid split-view DNS as much as I possibly can, though – I'd rather have the same host under two suffixes, e.g. its internal address under an internal domain that's always routed via the WireGuard interface.)

(I think resolved generally assumes 'the last update wins', which is perfectly reasonable for its narrow focus. If you bring up a VPN, you probably want its opinions on DNS resolution to win.)

Regarding priorities and "last update wins": Each interface also has a DefaultRoute flag which somehow affects this, on top of the "routing-only domains".

https://blogs.gnome.org/mcatanzaro/2020/12/17/understanding-systemd-resolved-split-dns-and-vpn-configuration/

By cks at 2023-02-12 12:09:58:

As far as I can tell you can't set just a resolv.conf search domain. Setting Domains= in the global resolved.conf also sends queries for those domains to your global DNS servers, especially if you also set the global DNS servers as '~.', the search default. If you have a specific network that wants to send queries for that domain to its associated DNS server (and thus has 'Domans=' or the resolvectl equivalent set for it), resolved will sometimes (or maybe always) ignore that and give the queries to your global DNS server.

It may be possible to avoid this by setting up your configuration just right, but since we have to go to significant contortions it's clearly fragile. Systemd-resolved doesn't guarantee any of this, so their heuristics could change in the next release and break the configuration, which means breaking resolution for some names. In my view it's not worth it; if you need a speccific search path all the time, you need to hard-code it in a static /etc/resolv.conf.

Written on 10 February 2023.
« I'm considering giving in to the systemd-resolved wave
Learning about Linux fwmark masks »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Feb 10 23:16:15 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.