Why systemd-resolved can give weird results for nonexistent bare hostnames

December 13, 2023

Suppose, not hypothetically, that you use systemd-resolved and you have a long standing practice of specific DNS search path so that people can use short domain names. In this environment you probably need to use systemd-resolved purely through /etc/resolv.conf, and if you do this you may experience an oddity:

$ ping nosuchname
ping: nosuchname: Temporary failure in name resolution

If you try 'resolvectl query nosuchname' it will tell you that the name is not found, but if you directly query the systemd-resolved DNS server at 127.0.0.53 you will see that you get a DNS SERVFAIL response for the bare name:

$ dig a nosuchname. @127.0.0.53
[...]
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52471
[...]

(You will wind up querying for the bare name when you've exhausted all of the domains in your DNS search path.)

This is not what a normal DNS server like Unbound will return for the same query; Unbound will return NXDOMAIN for this query, which will cause programs like ping to tell you 'Name or service not known', which is probably what you want. If you know what's going on you can translate, but why worry yourself about the possibility that something is really going wrong.

What is going on here is systemd-resolved's interpretation of how to behave for DNS queries if ResolveUnicastSingleLabel is unset in your resolved.conf. How the documentation describes it is:

Takes a boolean argument. When false (the default), systemd-resolved will not resolve A and AAAA queries for single-label names over classic DNS. [...]

Since ping's attempts to find the IP address of 'nosuchname' eventually wind up making a single-label name query to systemd-resolved, with this setting in its default state systemd-resolved will not try to resolve this query by sending it to an upstream DNS resolver (where it would fail). When queried as a DNS server, resolved's interpretation of 'will not (try to) resolve' is to return SERVFAIL instead of NXDOMAIN. This is in some sense technically correct, but it's usually not as useful as returning NXDOMAIN would be (and it's not how Unbound or Bind behave).

If you have local DNS resolvers that systemd-resolved on your systems is pointing to, you can safely set ResolveUnicastSingleLabel=yes to work around this. Systemd-resolved will dutifully send these queries to your local DNS resolvers, your local DNS resolvers will NXDOMAIN them, and systemd-resolved will pass this NXDOMAIN back to you so that ping tells you there's no such host. I'm probably going to do this on my desktops (and any of our machines that wind up using systemd-resolved).

(A lot of my understanding of this comes from finding and reading Ubuntu systemd bug #2024320 and systemd issue #28310.)

Sidebar: Some thoughts on SERVFAIL versus NXDOMAIN here

If you have upstream DNS servers that will actually return something for A and AAAA queries for single-label names for some (local) reason, systemd-resolved returning SERVFAIL and ping reporting it as a 'temporary' failure in name resolution is probably doing you a favour because it's signalling that something weird is going on in your (DNS) name resolution. Systemd-resolved returning NXDOMAIN might lead you to suspect that your upstream DNS servers didn't have the data you expected them to.

However, this is a rare case. A much more usual case is going to be what we saw here; you have a DNS search path, you type a name that you implicitly expect to be in your local domain or not present at all, and instead of 'name or service not known' because it's not in your local domain you get some odd 'temporary failure' (which doesn't happen if you use resolvectl to theoretically check directly).

Written on 13 December 2023.
« We've switched (back) to using Bind for our local DNS resolvers
Partially emulating #ifdef in Go with build tags and consts »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Dec 13 22:37:59 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.