Unix doesn't normally do short write()s to files and no one expects it to

November 16, 2020

A famous issue in handling network IO on Unix is that write() may not send all of your data; you will try to write() 16 KB of data, and the result will tell you that you only actually wrote 4 KB. Failure to handle this case leads to mysteriously lost data, where your sending program thinks it sent all 16 KB but of course the receiver only saw 4 KB. It's very common for people writing network IO libraries on Unix to provide a 'WriteAll' or 'SendAll' operation, or sometimes make it the default behavior.

(Go's standard Write() interface requires full writes unless there was an error, for example.)

In theory the POSIX specification for write() allows it to perform short writes on anything (without an error), not just network sockets, pipes, and FIFOs. In particular it is allowed to do them for regular files, and POSIX even documents some situations where this may happen (for example, if the process received a signal part way through the write() call). In practice, Unixes do not normally do short write()s to files without an error occurring, outside of the special case of a write() being interrupted by a signal that doesn't kill the process outright.

(If the process dies on the spot, there is no write() return value.)

In theory, because it's possible, every Unix program that write()s to a file should be prepared to handle short writes. In practice, since it doesn't really happen, many Unix programs are almost certainly not prepared to handle it. If you (and they) are lucky, these programs check that the return value of the write() is the amount of data they wrote and error out otherwise. Otherwise, they may ignore the write() return value and cheerfully sail on with data lost. Of course they don't actually error out or lose data in practice, because short write()s don't really happen on files.

(Some sorts of programs are generally going to be okay because they are already very careful about data loss. I would expect any good editor to be fine, for example, or at least to report an error.)

This difference between theory and practice means that it would be pretty dangerous to introduce a Unix environment that did routinely have short writes to files (whether it was a new Unix kernel or, say, a peculiar filesystem). This environment would be technically correct and it would be uncovering theoretical issues in programs, but it would probably not be useful.

PS: Enterprising parties could arrange to test this with their favorite programs through a loadable shared library that intercepts write() and shortens the write size. I suspect that you could get an interesting undergraduate Computer Science paper out of it.

Written on 16 November 2020.
« I don't expect to have an ARM-based PC any time soon
POSIX write() is not atomic in the way that you might like »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Nov 16 00:12:50 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.