We have traditionally configured our Ubuntu servers to have an /etc/resolv.conf that points at our central recursive DNS resolvers. People in research group sandbox networks have generally done likewise, partly because it's usually been the easiest thing to do. Machines have to consult our local resolvers in order to correctly look up other local machines, and once you're doing that you might as well not add any extra layers (which have generally taken extra work to add). But there's a downside to this configuration.

Every so often someone either writes or runs a program that does a lot of hostname lookups. Often this is as part of making a lot of connections, for example to fetch a bunch of external resources. Very few programming languages and standard libraries cache the results of those lookups even if they are all of the same hostname (and for good reason, especially in a world where the IP associated with a hostname can change rapidly). But in our environment, this results in a flood of requests to our local resolvers, a flood that would be drastically reduced by even a little bit of local caching. Local caching would also make the responses faster, since even on the same network, an over the network DNS query is slower than querying a daemon on your own machine.

Adding an extra layer of DNS caching does create some operational issues, especially if it caches negative answers. These issues can be reduced if DNS answers are only cached for a very short amount of time, but that generally takes extra configuration (if it's even possible). It's also traditionally taken an extra setup step and extra configuration in general, which is part of our bias against doing it. However, systemd is on its way to changing that with systemd-resolved, although there are plenty of questions about how it will work in an environment like ours and whether Ubuntu will ever adopt it as a standard part of server installs.

So far, we've been aggressive about disabling systemd-resolved in our install system (and haven't set up any other local caching resolver). However I'm starting to wonder if we should change that, especially if Ubuntu switches to normally wanting systemd-resolved on (so that, for example, netplan is unhappy with you if resolved isn't running).

(To really answer this question we should probably get fine grained query statistics from our DNS servers, or at least packets per second statistics. But that's a longer term project for various reasons.)