Why systemd-resolved can give weird results for nonexistent bare hostnames

December 13, 2023

Suppose, not hypothetically, that you use systemd-resolved and you have a long standing practice of specific DNS search path so that people can use short domain names. In this environment you probably need to use systemd-resolved purely through /etc/resolv.conf, and if you do this you may experience an oddity:

$ ping nosuchname
ping: nosuchname: Temporary failure in name resolution

If you try 'resolvectl query nosuchname' it will tell you that the name is not found, but if you directly query the systemd-resolved DNS server at 127.0.0.53 you will see that you get a DNS SERVFAIL response for the bare name:

$ dig a nosuchname. @127.0.0.53
[...]
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52471
[...]

(You will wind up querying for the bare name when you've exhausted all of the domains in your DNS search path.)

This is not what a normal DNS server like Unbound will return for the same query; Unbound will return NXDOMAIN for this query, which will cause programs like ping to tell you 'Name or service not known', which is probably what you want. If you know what's going on you can translate, but why worry yourself about the possibility that something is really going wrong.

What is going on here is systemd-resolved's interpretation of how to behave for DNS queries if ResolveUnicastSingleLabel is unset in your resolved.conf. How the documentation describes it is:

Takes a boolean argument. When false (the default), systemd-resolved will not resolve A and AAAA queries for single-label names over classic DNS. [...]

Since ping's attempts to find the IP address of 'nosuchname' eventually wind up making a single-label name query to systemd-resolved, with this setting in its default state systemd-resolved will not try to resolve this query by sending it to an upstream DNS resolver (where it would fail). When queried as a DNS server, resolved's interpretation of 'will not (try to) resolve' is to return SERVFAIL instead of NXDOMAIN. This is in some sense technically correct, but it's usually not as useful as returning NXDOMAIN would be (and it's not how Unbound or Bind behave).

If you have local DNS resolvers that systemd-resolved on your systems is pointing to, you can safely set ResolveUnicastSingleLabel=yes to work around this. Systemd-resolved will dutifully send these queries to your local DNS resolvers, your local DNS resolvers will NXDOMAIN them, and systemd-resolved will pass this NXDOMAIN back to you so that ping tells you there's no such host. I'm probably going to do this on my desktops (and any of our machines that wind up using systemd-resolved).

(A lot of my understanding of this comes from finding and reading Ubuntu systemd bug #2024320 and systemd issue #28310.)

Sidebar: Some thoughts on SERVFAIL versus NXDOMAIN here

If you have upstream DNS servers that will actually return something for A and AAAA queries for single-label names for some (local) reason, systemd-resolved returning SERVFAIL and ping reporting it as a 'temporary' failure in name resolution is probably doing you a favour because it's signalling that something weird is going on in your (DNS) name resolution. Systemd-resolved returning NXDOMAIN might lead you to suspect that your upstream DNS servers didn't have the data you expected them to.

However, this is a rare case. A much more usual case is going to be what we saw here; you have a DNS search path, you type a name that you implicitly expect to be in your local domain or not present at all, and instead of 'name or service not known' because it's not in your local domain you get some odd 'temporary failure' (which doesn't happen if you use resolvectl to theoretically check directly).


Comments on this page:

By Ian Z aka nobrowser at 2023-12-14 12:53:28:

There are loose ends here, and I definitely don't know the answers. Please clear up my confusion if you can :)

- In the dig query you ask for nosuchname. -- with a trailing dot. So DNS is looking up in the root zone, which will of course fail, systemd or no systemd. But the ping command had no dot, so the resolver should have applied your search path before asking resolved. Why hasn't it? Or it has, and you're testing apples with ping and oranges with dig?

- Does the trailing dot even make a difference for dig? Being a low-level DNS only program, doesn't it bypass the resolver completely and interpret single label names as belonging to the root?

- The systemd documentation snippet refers to "classic DNS". I originally thought the subtle distinction here might be not between resolved and other DNS servers, but rather between global DNS and mDNS, ie. avahi in the context on Unix/Linux. But trying dig myself with an avahi-bound name it looks like mDNS names are not supposed to be passed to a DNS server at all. Where are they handled, then? In the resolver, or in each application separately?

By cks at 2023-12-14 14:05:41:

When I do 'ping nosuchname', ping will (in normal configurations) try it with every /etc/resolv.conf search path domain and then make a final try as a bare name. This is spelled out rather oddly in the 'search' section of resolv.conf(5):

[...] Finally, if the hostname does not contain a '.', the root domain is assumed as the local domain name.

So the 'dig a nosuchname. @127.0.0.53' query is what ping will wind up eventually doing as its final attempt to resolve 'nosuchname'. You can see this in action with, among other tools, dnspeep.

As for the trailing '.' on the name in dig, whether or not you need that depends. Here, I do; 'dig +showsearch a nosuchname @127.0.0.53' (with no trailing dot) shows that dig actually makes a succession of lookups for everything in my /etc/resolv.conf DNS search path. I generally stick the '.' on the end of everything I'm looking up with dig, because I always want the absolute lookup and this way makes sure.

I'm not sure where and how mDNS and other such things are handled in systemd-resolved, as I don't make any use of them myself.

Written on 13 December 2023.
« We've switched (back) to using Bind for our local DNS resolvers
Partially emulating #ifdef in Go with build tags and consts »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Wed Dec 13 22:37:59 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.