Wandering Thoughts archives

Some places where I think that Unix is incomplete or imperfect

A commentator on Unix needs to grow asked for some examples of what I had in mind and, well, the resulting comment got away from me.

This is a very incomplete list written off the top of my head (and is in no particular order). All of these issues are from the perspective of 'traditional Unix', the Unix on static servers that most everyone agrees is more or less fully Unixy. While modern Unixes have dealt with a number of them it's often not in ways that people really think are Unixy, so I consider them open problems that still need work.

• hotplug hardware (on basically everything), including issues like how /dev entries for new devices show up.

• on multi-user workstations (and then desktops), making it so that only the user on the console could use various bits of local hardware like the audio device. Pranking people by ssh'ing in to their workstation and then playing loud music is all well and good, but let's not.

  (A genuine multi-head environment makes this even less clear.)

• allowing the console user to access things like CD-ROMs, DVDs, USB memory sticks, and so on that they plug in to the hardware. Even apart from the inconvenience, 'su to root and run mount' is not a viable answer because in a multi-user workstation environment the console user may not have the root password. This needs to be controllable (some environments don't want to allow this) and is made more complicated because there are many different sorts of access that are needed beyond just mounting a filesystem (eg burning a DVD).

• dealing with the variable networking environment that a laptop or other mobile device sees as it moves around. Again, 'su to root' is a terrible answer here. This is more than just basic changes in network devices, IP addresses, and so on; it ripples through into user programs, DNS resolution, and any number of other areas.
• in general, handling various sorts of 'on-demand' and transient network connections.

• suspending and unsuspending machines, especially since there are often user-space things you want to do both on suspend and on unsuspend. This used to be just a laptop issue but is now also interesting for things like virtual machines.

• many aspects of networking in general. For example, what is a Unixy approach to firewalling, NAT, and so on, or does Unix have nothing to say about the right way to approach this? What is the right Unixy API for programs to use the network?

  (Note that the Bell Labs answer to this looks nothing like the BSD sockets API.)

• password and key management at both the system and user layer. This is especially important if encrypted filesystems start becoming pervasive. Current solutions are ad-hoc large blobs (Plan 9 had some ideas here). This interacts with things like networking, especially on shared machines; if you hand your laptop to your friend to use temporarily (with a guest account or a just-created account), you probably don't want them to be able to bring up your VPN connection to work.

• coordinating system and user choices for things like 'how do you view a web page' or 'what shows pictures'.
• can you make a general (program) configuration and options system that feels Unixy and is easily manipulated in a Unixy way, or are we doomed to things that look uncannily like the Windows registry?

  (I sure hope the answer is not 'yes'. Note that dotfiles are not the answer; they are ad-hoc single program solutions.)

• general structured interprocess communication. Many of the other problems involve programs learning about changes to the world or telling the world about changes and some of them have programs talking to other programs. They can all do this in a giant anarchy of program and problem specific mechanisms, but shouldn't there be some more general, more Unixy approach to the whole issue? Ideally one that made it easy to write little programs and shell scripts that plugged into the whole system.

• how do you process structured data in Unix pipelines and store it in 'flat' text files? Are there general mechanisms or conventions that Unix should adopt or are we doomed to a bunch of ad-hoc solutions for specific problems (such as 'a grep for json')?

• the graphical environment equivalent of the shell and shell scripts.

• what I called a good Unix glue language, something that is better than shell scripts for what shell scripts are good for while feeling right and being just as convenient. A better shell is not necessarily the right model for this.

• fair share resource allocation on multi-user or multi-use machines.

• a better permissions and restrictions model. It's become obvious that the current basic Unix model of separating and confining things by UID is not a great fit for the modern world, especially with the current set of Unix access restrictions. What is a better Unixy way to both confine processes and to give them extra permissions?

  (For example, it's absurd that you must run Apache as root in order for it to bind to port 80 and 443 and equally absurd that Apache needs to either keep root permissions or run all CGIs/etc as the same UID that is running the main web server.)

  A really well done system would allow programs to take advantage of this for handling things that they want access restricted. For example, it would be nice if the printing system didn't need its own ad-hoc private system for managing who can manipulate particular printer queues.

• integrating Unix machines in clusters and other larger network systems. My view is that none of the existing components really feel all that Unixy (NIS, LDAP, automounter, etc).

• if one of the big Unix ways of handling things is 'virtual filesystems', Unix needs to make them easier. Plan 9 argues that you can do a lot with very Unixy results but also that you need a way to make them accessible to users in order to tap their full power. What does a Plan 9 like approach look like on Unix, if it can be made to work at all? How do you integrate user-owned, user-mounted filesystems with the Unix permissions model and also not make a multi-user system explode when everyone is mounting flaky filesystems?

• /tmp on multi-user machines. This could be an entire entry by itself but /tmp is both a source of problems and a useful feature. How do you get rid of the problems while keeping the good stuff?

• what is the Unixy way to handle disk storage feature sets like ZFS and btrfs? The current approach is to shove everything from RAID on up into one huge blob, which is not really the traditional Unix way. Does Unix need some sort of fundamental shift in the underlying model of how filesystems interact with disk storage, one that will enable splitting ZFS, btrfs, etc up into nice logical multi-layered things?

  (This is more than just ZFS/btrfs. For example, I haven't seen a logical volume management system that actually feels Unixy.)

Many of these are large, abstract problems. I don't think any of them have easy or obvious answers; for some of them it's not even clear how you'd start dealing with the problem or if it's actually possible to have a 'Unixy' solution. Where current Unix systems deal with these, people often find the answers unsatisfying and non-Unixy (as I mentioned).

Unix needs to grow and a consequence of that

I hope it will not be news to people that Unix is neither perfect nor complete. By this I mean that there are things that a traditional Unix system doesn't do ideally and that can be improved as well as any number of things that it doesn't do that people really do want. I'm not talking about minor technology additions on the side (IPv6, say); I'm talking about significant additions and changes.

(If you feel that Unix is only a server OS you may feel that it is more or less complete as it stands. I disagree; my Unix is general and in many non-server environments a traditional Unix system is clearly incomplete and lacks important features. And even as a server OS only Unix is still not perfect (eg try to tell me with a straight face that SysV init is the best init system possible).)

So Unix needs to grow and change. But there is a consequence of this, one having to do with people's attitudes about changes to Unix.

To put it simply: any number of people will say that such growth is fine but only as long as the growth is (perfectly) 'Unixy', that it fits beautifully into the spirit of Unix. Unfortunately the practical consequence of saying this is that you are actually saying that Unix can't grow.

(Let us set aside for now the chances of getting a roomfull of people to agree in detail on just what the Unix spirit is and whether or not something is sufficiently Unixy.)

The reality is that it has always taken time and experimentation to work out just how to do something new in the Unixy way and the first attempts were often not very good. Even Bell Labs, the very home of the Unix way, took years and multiple attempts to get things like window systems and networking to really fit right into Unix. Other core sources of Unix development did (much) worse. Things that we think of as big parts of the correct Unix way are often relatively late additions, displacing much older and cruder solutions.

(For instance, virtual filesystems like /proc seem obviously the right way to do things but they appeared only in the late 1980s and required a whole string of previous innovations from various places. And you really don't want to know how programs like ps got information about processes before /proc.)

If you insist that people get changes to Unix correct right from the start, on the very first attempt, you are demanding that they be even more insightful about Unix than Bell Labs was. This is more or less impossible so what you are really saying is 'you can't change Unix'.

(This elaborates on two tweets of mine.)

My Unix is a general purpose operating system

When you start thinking about the present and future of Unix, one of the questions you are confronted with is what Unix is for.

One vision of Unix is that its focus is text-mode or headless servers that have a basically static location (in both physical and network terms) and exist to run daemons and services; websites, databases, disk storage nodes, fileservers, and so on. Let me be blunt: this is a very popular thing to do with Unix and is probably the dominant use of Unix today. It runs all the way from a devops organization virtualized in the cloud to one person running a single do-it-all server machine under their desk in a small organization.

My view of Unix is broader than this. For me, Unix is a general purpose operating system, one that is not just for servers but also for simple graphical environments (such as mine), full blown fancy desktops, laptops whether minimal or fancy, little simple machines, and many other places. I keep an open mind and feel that what is fundamentally Unix is capable of wide scope and applicability. In short, it scales in many directions.

(I will skip trying to summarize my reasons for why, but part of it is I feel that Unix has turned out to be a pretty good framework for interacting with computers.)

By Unix I mean more than the kernel and the core APIs. I mean, well, the metaphors and the framework and in general everything that makes Unix a familiar environment for general use. A machine can run a Unix kernel and have Unix-like APIs without being Unix; where the line is between Unix and non-Unix is ultimately one of feel (and varies from person to person).

There are environments where it's not clear if Unix fits. For example, I'm not sure much of the Unix framework really works in smartphones and it may not work well with tablets; we're going to have to see. Part of this is that the Unix framework is a general framework and is not necessarily a good fit to a very specialized, narrow device. Part of it is that the Unix metaphors may not be a good fit for some environments.

Sidebar: my views on some not-quite-Unixes

From what I know of it so far, Android seems clearly not a Unix in this sense although it uses a Unix kernel (yes, Linux is a Unix). Since I have low exposure to Android I may be wrong (and I'm open to having my mind changed).

Mac OS X is a fuzzy case but I consider it mostly not a Unix. If you use OS X as Apple intends you to, its Unix is simply a substrate for what it really is (in the same way its kernel uses Mach as a substrate without actually being Mach in any meaningful way). You can use an OS X box as a Unix machine but in some ways doing so seems to be swimming upstream.

(Part of this perception is based on the history of Apple and of Mac OS. To put it one way, I don't think that Apple has any interest in making Unix machines with a nice Apple desktop; that they arguably do is just a side effect, not a goal.)

The fundamental problem faced by user-level NFS servers

In a comment on yesterday's entry, Perry Lorier wrote in part:

[...] I know that I don't like userspace NFS servers, because last time I tinkered with one it went horribly horribly wrong. [...]

User-level NFS servers basically always explode; they are essentially intrinsically doomed unless they get very unusual support from the host Unix system. The core problem is our old friend NFS filehandles, which are the only identification the NFS server gets for what filesystem object the client wants to do something to. To create good ones, a NFS server needs some unique identifier for filesystem objects that has three properties: it is a stable identifier that stays with the file even if arbitrary things get renamed, it can be used to efficiently and rapidly access the file itself, and it must be invalidated when the file is deleted. Let's handwave the third property for now.

The problem is that Unix doesn't actually have an identifier with the first two properties that's generally available to user level. Filenames can be used to rapidly access an object but they aren't stable across renames, while inodes are stable across renames but there is no (general) 'open by inode' system call so they can't be opened rapidly in the general case. In the absence of such an identifier user-level NFS servers have no choice but to fake it in various ways, and their fakes break down every so often. When that happens you get some sort of explosion.

(You can keep a cache of inode to filename mappings but your cache entry may be missing or invalid, at which point you have to search the entire filesystem to find the file with the right inode.)

By the way: I'm ignoring the size of this identifier because I'm giving the user-level NFS server a persistent, arbitrary-sized database that it uses to keep track of filehandle to identifier mappings. If you don't want to have such a database, the identifier also needs to be relatively small.

(I feel that such a database is feasible under most circumstances. Most filesystems have only a few tens of thousands or hundreds of thousands of filesystem objects; this is not a big database these days. Even a few million objects is feasible, especially since indexing is easy.)

Kernel NFS servers don't have this problem because they have 'open by inode' (or really 'access by inode'). Well, usually they don't have this problem; they run into this exactly when they're trying to export a filesystem that doesn't use an inode-like identifier for its files. In Unix-like filesystems, generally either the filesystem's stable internal identifier is too large or the filesystem makes up its short 'inode numbers' in a way that doesn't let it look up an object by this number.

(In non-Unix-like filesystems, there may be no equivalent of an inode number at all; perhaps the file's name is the only identifier for it.)

PS: authors of new, sophisticated filesystems are not infrequently very grumpy about this inode number requirement and wish that it, along with NFS, would just go away. Sometimes they refuse to support inode numbers this way and then their new filesystem is not NFS exportable and a bunch of people become irritated with them. My view is that their time would be better spent implementing and advocating for a rich enough system call interface that a quality user-level NFS server is actually possible.

(Note that the 'open by stable identifier' system call would probably only be usable by root, which eliminates a whole class of security concerns.)