Systemd's DynamicUser feature is (currently) dangerous
Yesterday I described how timesynd couldn't be restarted on one of our Ubuntu 18.04 machines, where the specific thing that caused the failure was timesyncd attempting to access /var/lib/private/systemd/timesync and failing because /var/lib/private is only accessible by root, not the UID that timesyncd was running as. My diagnostic efforts left me puzzled as to how this was supposed to work at all, but Trent Lloyd (@lathiat) pointed me to the answer, which is in Lennart Poettering's article Dynamic Users with systemd, which introduces the overall system, explains the role of /var/lib/private, and covers how timesyncd is supposed to get access through an inaccessible directory. I'll quote the explanation for that:
[Access through /var/lib/private] is achieved by invoking the service process in a slightly modified mount name-space: it will see most of the file hierarchy the same way as everything else on the system ([...]), except for
/var/lib/private, which is over-mounted with a read-onlytmpfsfile system instance, with a slightly more liberal access mode permitting the service read access. [...]
Since timesyncd is not able to get access through /var/lib/private,
you might guess that something has gone wrong in the process of
setting up this slightly modified mount namespace. Indeed this
turned out to be the case. The machine that this happened on is an
NFS client and (as is usual) its UID 0 is mapped to an unprivileged
UID on our fileservers. On this
machine there were some FUSE mounts in the home directories of users
who have their $HOME not world readable (our default $HOME
permissions are owner-only, to avoid accidents). When systemd was
setting up the 'slightly modified mount name-space' it attempted
to access these FUSE mounts as part of binding them into the
namespace, but it failed because UID 0 had no permissions to look
inside user home directories.
This failure caused systemd to give up attempting to set up the
namespace. However, systemd did not abort unit activation or even
log an error message. Instead it continued on to try to start
timesyncd without this special namespace, despite the fact that
timesyncd uses both DynamicUser and StateDirectory and so
starting it normally was essentially absolutely guaranteed to fail.
(Although my initial case was dangling FUSE mounts, it soon developed that any FUSE mounts would do it, for example a sshfs or smbfs mount in a user's NFS mounted home directory when the home directory isn't world-accessible.)
Systemd's failure to handle errors in setting up the namespace here
has been raised as systemd issue 9835. However, merely
logging an error or aborting the unit activation would not actually
fix the core problem; it would merely let you see exactly why your
timesyncd or whatever service is failing to start. The core problem
is that systemd's current design for DynamicUser intrinsically
blows up if systemd and UID 0 don't have full access to every mount
that's visible on the system.
(Well, DynamicUser plus StateDirectory, but the idea seems to be that pretty much every service using dynamic users will have a systemd managed state directory.)
In my opinion, this makes using DynamicUser surprisingly dangerous.
A systemd service that is set to use it can't be reliably started
or restarted on all systems; it only works on some systems, some
of the time (but those happen to be the common case). If there's
ever a problem setting up the special namespace that each such
service requires, things fail. Machines that are NFS clients are
the obvious case, since the client's UID 0 often has limited
privileges, but I believe that there are likely to be others.
(And of course services can be restarted for random and somewhat unpredictable reasons, such as package updates or other services being restarted. You should not assume that you can always control these circumstances, or completely predict the state of the system when they happen.)
Comments on this page:
|
|