2018-08-25
The history of file type information being available in Unix directories
The two things that Unix directory entries absolutely have to have
are the name of the directory entry and its 'inode', by which we
generically mean some stable kernel identifier for the file that
will persist if it gets renamed, linked to other directories, and
so on. Unsurprisingly, directory entries have had these since the
days when you read the raw bytes of directories with read()
, and for a long time that was all they had; if you
wanted more than the name and the inode number, you had to stat()
the file, not just read the directory. Then, well, I'll quote
myself from an old entry on a find
optimization:
[...], Unix filesystem developers realized that it was very common for programs reading directories to need to know a bit more about directory entries than just their names, especially their file types (
find
is the obvious case, but also consider things like 'ls -F
'). Given that the type of an active inode never changes, it's possible to embed this information straight in the directory entry and then return this to user level, and that's what developers did; on some systems,readdir(3)
will now return directory entries with an additionald_type
field that has the directory entry's type.
On Twitter, I recently grumbled about Illumos not having this
d_type
field.
The ensuing conversation wound up with me curious about exactly where
d_type
came from and how far back it went. The answer turns out to
be a bit surprising due to there being two sides of d_type
.
On the kernel side, d_type
appears to have shown up in 4.4 BSD.
The 4.4 BSD /usr/src/sys/dirent.h
has a struct dirent
that has a d_type
field, but the field
isn't documented in either the comments in the file or in the
getdirentries(2)
manpage; both of those admit only to the traditional
BSD dirent
fields. This 4.4 BSD d_type
was carried through
to things that inherited from 4.4 BSD (Lite), specifically FreeBSD,
but it continued to be undocumented for at least a while.
(In FreeBSD, the most convenient history I can find is here, and the d_type
field
is present in sys/dirent.h as far back as FreeBSD 2.0, which
seems to be as far as the repo goes for releases.)
Documentation for d_type
appeared in the getdirentries(2)
manpage in FreeBSD 2.2.0, where the manpage itself claims to have
been updated on May 3rd 1995 (cf).
In FreeBSD, this appears to have been part of merging 4.4 BSD
'Lite2', which seems to have been done in 1997. I stumbled over a
repo of UCB BSD commit history,
and in it the documentation appears in this May 3rd 1995 change,
which at least has the same date. It appears that FreeBSD 2.2.0 was
released some time in 1997, which is when this would have appeared
in an official release.
In Linux, it seems that a dirent structure with a d_type
member
appeared only just before 2.4.0, which was released at the start
of 2001. Linux took this long because the d_type
field only
appeared in the 64-bit 'large file support' version of the
dirent
structure, and so was only return by the new 64-bit
getdents64()
system call. This would have been a few years after
FreeBSD officially documented d_type
, and probably many years
after it was actually available if you peeked at the structure
definition.
(See here for an overview of where to get ancient Linux kernel history from.)
As far as I can tell, d_type
is present on Linux, FreeBSD,
OpenBSD, NetBSD, Dragonfly BSD, and Darwin (aka MacOS or OS X).
It's not present on Solaris and thus Illumos. As far as other
commercial Unixes go, you're on your own; all the links to manpages
for things like AIX from my old entry on the remaining Unixes appear to have rotted away.
Sidebar: The filesystem also matters on modern Unixes
Even if your Unix supports d_type
in directory entries, it
doesn't mean that it's supported by the filesystem of any specific
directory. As far as I know, every Unix with d_type
support has
support for it in their normal local filesystems, but it's not
guaranteed to be in all filesystems, especially non-Unix ones like
FAT32. Your code should always be prepared to deal with a file type
of DT_UNKNOWN
.
(Filesystems can implement support for file type information in directory entries in a number of different ways. The actual on disk format of directory entries is filesystem specific.)
It's also possible to have things the other way around, where you have a filesystem with support for file type information in directories that's on a Unix that doesn't support it. There are a number of plausible reasons for this to happen, but they're either obvious or beyond the scope of this entry.
2018-08-22
Why ed(1)
is not a good editor today
I'll start with my tweet:
Heretical Unix opinion time: ed(1) may be the 'standard Unix editor', but it is not a particularly good editor outside of a limited environment that almost never applies today.
There is a certain portion of Unixdom that really likes ed(1), the 'standard Unix editor'. Having actually used ed for a not insignificant amount of time (although it was the friendlier 'UofT ed' variant), I have some reactions to what I feel is sometimes overzealous praise of it. One of these is what I tweeted.
The fundamental limitation of ed is that it is what I call an
indirect manipulation interface,
in contrast to the explicit manipulation interfaces of screen editors
like vi and graphical editors like sam (which
are generally lumped together as 'visual' editors, so called because
they actually show you the text you're editing).
When you edit text in ed, you have some problems that you don't
have in visual editors; you have to maintain in your head the context
of what the text looks like (and where you are in it), you have to
figure out how to address portions of that text in order to modify
them, and finally you have to think about how your edit commands
will change the context. Copious use of ed's p
command can help
with the first problem, but nothing really deals with the other
two. In order to use ed, you basically have to simulate parts of
ed in your head.
Ed is a great editor in situations where the editor explicitly presenting this context is a very expensive or outright impossible operation. Ed works great on real teletypes, for example, or over extremely slow links where you want to send and receive as little data as possible (and on real teletypes you have some amount of context in the form of an actual printout that you can look back at). Back in the old days of Unix, this described a fairly large number of situations; you had actual teletypes, you had slow dialup links (and later slow, high latency network links), and you had slow and heavily overloaded systems.
However, that's no longer the situation today (at least almost all of the time). Modern systems and links can easily support visual editors that continually show you the context of the text and generally let you more or less directly manipulate it (whether that is through cursoring around it or using a mouse). Such editors are easier and faster to use, and they leave you with more brainpower free to think about things like the program you're writing (which is the important thing).
If you can use a visual editor, ed is not a particularly good editor to use instead; you will probably spend a lot of effort (and some amount of time) on doing by hand something that the visual editor will do for you. If you are very practiced at ed, maybe this partly goes away, but I maintain that you are still working harder than you need to be.
The people who say that ed is a quite powerful editor are correct; ed is quite capable (although sadly limited by only editing a single file). It's just that it's also a pain to use.
(They're also correct that ed is the foundation of many other things in Unix, including sed and vi. But that doesn't mean that the best way to learn or understand those things is to learn and use ed.)
This doesn't make ed a useless, vestigial thing on modern Unix, though. There are uses for ed in non-interactive editing, for example. But on modern Unix, ed is a specialized tool, much like dc. It's worth knowing that ed is there and roughly what it can do, but it's probably not worth learning how to use it before you need it. And you're unlikely to ever be in a situation where it's the best choice for interactive editing (and if you are, something has generally gone wrong).
(But if you enjoy exploring the obscure corners of Unix, sure, go for it. Learn dc too, because it's interesting in its own way and, like ed, it's one of those classical old Unix programs.)
2018-08-19
Why I'm mostly not interest in exploring new fonts (on Unix)
Every so often, an enthusiasm for some new font or set of fonts goes around the corners of the Internet that I pay attention to. It's especially common for monospaced fonts (programmers seem to have a lot of opinions here), but people get enthused by proportional ones as well. I used to be reasonably interested in this area and customized my fonts and sometimes my font rendering, but these days I find that I generally take at most a cursory look at new fonts and tune out.
The first reason I tune out is limited Unicode glyph coverage. As
I found out when I dug into it, xterm
doesn't require your
primary font to have CJK glyphs, but as far
as I know it does require your font to have most anything else,
including Cyrillic, Arabic, and Hebrew glyphs, if you're not going
to see just 'glyph missing' boxes. I don't necessarily look at stuff
in these languages in my xterm
s all that often, but it comes up
every so often and I'd like to have things available in any new font,
since I pretty much do currently.
(I have to admit that the most common place these character sets come up is when I'm looking at spam messages, email and otherwise.)
For proportional fonts, things are less clear to me. Firefox apparently still uses glyphs from multiple fonts as necessary (based on eg this bug), even with modern XFT/FreeType fonts. I believe that Gnome and probably KDE have some sort of system for this, which sometimes matters for programs like Liferea (syndication feeds) and Corebird (Twitter), but I have no idea how it all works or is controlled. If all of this works, it means that my nominal main proportional font wouldn't need complete glyph coverage, although I might have to do a bunch of configuration tweaking.
(Back in the pre-XFT days, Firefox used to have a quite complex and arcane system for finding glyphs to render, since X fonts often had very small coverage. If things went wrong this could produce rather bizarre results; Firefox was perfectly capable of switching fonts around in the middle of lines or words, even in nominally European text, if it decided that for some reason your primary font didn't have the necessary character and it had to fish it out from some other, very different font. All of this is fortunately behind us and I've mercifully forgotten most of the details.)
All of this matters because most alternate fonts don't have a very wide glyph coverage. This isn't surprising; drawing a lot of glyphs is a lot of work, and creating glyphs for non-Latin characters requires its own set of type design expertise in how those characters are supposed to look. In practice most artisanal new fonts are likely to be pretty much confined to Latin and near-Latin characters.
The bigger reason that I tune out these days is that standard Unix XFT fonts and font display are now pretty good. Back in the old pre-XFT days, X fonts (bitmapped or TrueType) were generally pretty limited and often not of particularly high quality, and rendering decisions could be questionable or oriented at hardware that you didn't have. This could make it very worthwhile to seek out an alternate font, because it could significantly improve your reading experience. I used an alternate bitmapped font in Firefox for years, and even after that stopped being a good idea I switched to an alternate TrueType font (one or the other is visible in this view of my desktop, although I no longer remember if it was the bitmapped 'Garamond' font or TTF Georgia).
The modern XFT world is much different. The current fonts that are mostly used and shipped by major desktop environments and Linux distributions are pretty well designed and have increasingly good glyph coverage (primarily this is DejaVu), and if you don't like them, there are alternates (eg Google's Noto). I've also come around to finding that the rendering decisions are generally good, which is a change from the past for me.
Given that the default fonts are already pretty good, the incremental improvement I might gain through fiddling around with my fonts doesn't appear to be worth the effort and the uncertainty. I could spend a lot of time tinkering, trying out various fonts, and basically wind up with something that was subtly worse because, to be honest, I don't know what I'm actually doing here (and I know enough to know that type design and typography is complex). So the easy path is just to use the defaults.
(This entry is sparked by reading vermaden's FreeBSD Desktop – Part 15 – Configuration – Fonts & Frameworks (via), and thinking about why I had no interest in doing something similar.)
Sidebar: Why Unix XFT fonts and font rendering got good
My impression is that part of it is simply the progress of time creating a slow but steady improvement, but a lot of it is that Unix/Linux font rendering became important enough that people spent money on it. Google obviously spent the money to develop Noto, but they're far from the only people. Sometimes this funding came explicitly, by commissioning font work and so on, and sometimes it came passively, where companies like Red Hat and Canonical hired people and had them spend (work) time on Unix font rendering.
(Android probably helped motivate Google and other parties here.)
2018-08-03
Firefox now implements its remote control partly over D-Bus
On Unix, Firefox has had a long standing feature where you could remote control a running Firefox instance. This has traditionally worked through X properties (with two generations of protocols), which has the nice advantage that it works from remote machines as well as your local one, provided that you're forwarding X. Since I read my mail through exmh that's running on one of our servers, not my desktop, this is pretty useful for me; I can click on a link in mail in exmh, and it opens in my desktop Firefox. However, working through X properties also has the disadvantage that it naturally doesn't work at all on Wayland. Since Wayland is increasingly important, last November or so the Mozilla people fixed this by adding a new D-Bus based protocol (it landed in bug 1360560 and bug 1360566 but has evolved in various ways since then).
On current versions of Firefox, you will find this service on the session bus under the name org.mozilla.firefox.<something>, where the <something> is often 'ZGVmYXVsdA__'. In general this weird thing is the base64 encoded name of your Firefox profile with a few special characters turned into _, and that particular name is, well:
; echo -n default | base64 ZGVmYXVsdA==
Because this directly encodes the profile name in something that
you have to get right, the D-Bus based version of Firefox remote
control will reliably restrict itself to talking to a running Firefox
that's using the same profile; the X properties based version doesn't
always (or didn't always, at any rate). You can force a new Firefox
to not try to talk to an existing Firefox by using --new-instance
,
as before.
(One case where you might need this is if you're testing an
alternate version of Firefox by manually setting your $HOME
to, eg, /tmp/ffox-test
.)
It turns out that which protocol Firefox uses when is a bit tangled.
If Firefox is built with D-Bus support, a running Firefox on X will
be listening for incoming requests using both D-Bus and the X
properties based protocol; you can talk to this Firefox with either.
In the current Firefox code, if you built with both D-Bus and Wayland
support, the client Firefox always uses D-Bus to try to talk to the
running 'server' Firefox; it doesn't fall back to X properties if
there's no D-Bus available. If you built Firefox without Wayland
support, it always uses the X properties based protocol (even if
you built with D-Bus, and so the running Firefox is listening there).
You can see this sausage being made in StartRemoteClient()
here.
This logic was introduced in the change for bug 1465371. Before then Firefox tried to use the X properties based remote control if it was running on X, and fell back to the D-Bus protocol otherwise. In thinking about it I've come to believe that the logic here is sound, because in a Wayland session you may have some programs that think they're running in X and then pass this view on to things run from them. D-Bus is more session type agnostic, although it only works on the local machine.
Note that this implies that you can no longer use Firefox itself
as a client on a second machine, at least not if your second machine
Firefox is a modern one that was built with Wayland support; it'll
try to talk D-Bus and fail because your running Firefox isn't on
that machine. If you want to remote control Firefox from a second
machine, you now want a dedicated client like my ffox-remote
program.
(Hopefully Mozilla will leave the X properties based protocol there for many years to come, so my cross-machine remote control will still keep working.)
Sidebar: some D-Bus protocol details
The D-Bus object path is /org/mozilla/firefox/Remote, which has one
org.mozilla.firefox method, OpenURL()
, all of which you can see
by using a D-Bus browsing program such as d-feet. In the Firefox source code,
what you want to look at is widget/xremoteclient/DBusRemoteClient.cpp
(the client side, ie the firefox
command you just ran that is
going to pass your URL or whatever to the currently running one)
and toolkit/components/remote/nsDBusRemoteService.cpp (the server
side, ie the running Firefox).
Despite the fact that D-Feet will tell you that the argument to
OpenURL()
is a string, in actuality it's an entire command line
encoded in the same annoying binary encoding that is used in the
current X property based protocol, which you can read a concise
description of in nsRemoteService.cpp.
Presumably this minimizes code changes, although it's not the most
natural D-Bus interface. This encoding does mean that you're going
to need some moderately tangled code to remote-control Firefox over
D-Bus; you can't fire up just any old D-Bus client program for it.
The client code for this is in toolkit/xre/nsAppRunner.cpp,
in the StartRemoteClient()
function.