Wandering Thoughts archives

2018-08-25

The history of file type information being available in Unix directories

The two things that Unix directory entries absolutely have to have are the name of the directory entry and its 'inode', by which we generically mean some stable kernel identifier for the file that will persist if it gets renamed, linked to other directories, and so on. Unsurprisingly, directory entries have had these since the days when you read the raw bytes of directories with read(), and for a long time that was all they had; if you wanted more than the name and the inode number, you had to stat() the file, not just read the directory. Then, well, I'll quote myself from an old entry on a find optimization:

[...], Unix filesystem developers realized that it was very common for programs reading directories to need to know a bit more about directory entries than just their names, especially their file types (find is the obvious case, but also consider things like 'ls -F'). Given that the type of an active inode never changes, it's possible to embed this information straight in the directory entry and then return this to user level, and that's what developers did; on some systems, readdir(3) will now return directory entries with an additional d_type field that has the directory entry's type.

On Twitter, I recently grumbled about Illumos not having this d_type field. The ensuing conversation wound up with me curious about exactly where d_type came from and how far back it went. The answer turns out to be a bit surprising due to there being two sides of d_type.

On the kernel side, d_type appears to have shown up in 4.4 BSD. The 4.4 BSD /usr/src/sys/dirent.h has a struct dirent that has a d_type field, but the field isn't documented in either the comments in the file or in the getdirentries(2) manpage; both of those admit only to the traditional BSD dirent fields. This 4.4 BSD d_type was carried through to things that inherited from 4.4 BSD (Lite), specifically FreeBSD, but it continued to be undocumented for at least a while.

(In FreeBSD, the most convenient history I can find is here, and the d_type field is present in sys/dirent.h as far back as FreeBSD 2.0, which seems to be as far as the repo goes for releases.)

Documentation for d_type appeared in the getdirentries(2) manpage in FreeBSD 2.2.0, where the manpage itself claims to have been updated on May 3rd 1995 (cf). In FreeBSD, this appears to have been part of merging 4.4 BSD 'Lite2', which seems to have been done in 1997. I stumbled over a repo of UCB BSD commit history, and in it the documentation appears in this May 3rd 1995 change, which at least has the same date. It appears that FreeBSD 2.2.0 was released some time in 1997, which is when this would have appeared in an official release.

In Linux, it seems that a dirent structure with a d_type member appeared only just before 2.4.0, which was released at the start of 2001. Linux took this long because the d_type field only appeared in the 64-bit 'large file support' version of the dirent structure, and so was only return by the new 64-bit getdents64() system call. This would have been a few years after FreeBSD officially documented d_type, and probably many years after it was actually available if you peeked at the structure definition.

(See here for an overview of where to get ancient Linux kernel history from.)

As far as I can tell, d_type is present on Linux, FreeBSD, OpenBSD, NetBSD, Dragonfly BSD, and Darwin (aka MacOS or OS X). It's not present on Solaris and thus Illumos. As far as other commercial Unixes go, you're on your own; all the links to manpages for things like AIX from my old entry on the remaining Unixes appear to have rotted away.

Sidebar: The filesystem also matters on modern Unixes

Even if your Unix supports d_type in directory entries, it doesn't mean that it's supported by the filesystem of any specific directory. As far as I know, every Unix with d_type support has support for it in their normal local filesystems, but it's not guaranteed to be in all filesystems, especially non-Unix ones like FAT32. Your code should always be prepared to deal with a file type of DT_UNKNOWN.

(Filesystems can implement support for file type information in directory entries in a number of different ways. The actual on disk format of directory entries is filesystem specific.)

It's also possible to have things the other way around, where you have a filesystem with support for file type information in directories that's on a Unix that doesn't support it. There are a number of plausible reasons for this to happen, but they're either obvious or beyond the scope of this entry.

DirectoryDTypeHistory written at 00:31:39; Add Comment

2018-08-22

Why ed(1) is not a good editor today

I'll start with my tweet:

Heretical Unix opinion time: ed(1) may be the 'standard Unix editor', but it is not a particularly good editor outside of a limited environment that almost never applies today.

There is a certain portion of Unixdom that really likes ed(1), the 'standard Unix editor'. Having actually used ed for a not insignificant amount of time (although it was the friendlier 'UofT ed' variant), I have some reactions to what I feel is sometimes overzealous praise of it. One of these is what I tweeted.

The fundamental limitation of ed is that it is what I call an indirect manipulation interface, in contrast to the explicit manipulation interfaces of screen editors like vi and graphical editors like sam (which are generally lumped together as 'visual' editors, so called because they actually show you the text you're editing). When you edit text in ed, you have some problems that you don't have in visual editors; you have to maintain in your head the context of what the text looks like (and where you are in it), you have to figure out how to address portions of that text in order to modify them, and finally you have to think about how your edit commands will change the context. Copious use of ed's p command can help with the first problem, but nothing really deals with the other two. In order to use ed, you basically have to simulate parts of ed in your head.

Ed is a great editor in situations where the editor explicitly presenting this context is a very expensive or outright impossible operation. Ed works great on real teletypes, for example, or over extremely slow links where you want to send and receive as little data as possible (and on real teletypes you have some amount of context in the form of an actual printout that you can look back at). Back in the old days of Unix, this described a fairly large number of situations; you had actual teletypes, you had slow dialup links (and later slow, high latency network links), and you had slow and heavily overloaded systems.

However, that's no longer the situation today (at least almost all of the time). Modern systems and links can easily support visual editors that continually show you the context of the text and generally let you more or less directly manipulate it (whether that is through cursoring around it or using a mouse). Such editors are easier and faster to use, and they leave you with more brainpower free to think about things like the program you're writing (which is the important thing).

If you can use a visual editor, ed is not a particularly good editor to use instead; you will probably spend a lot of effort (and some amount of time) on doing by hand something that the visual editor will do for you. If you are very practiced at ed, maybe this partly goes away, but I maintain that you are still working harder than you need to be.

The people who say that ed is a quite powerful editor are correct; ed is quite capable (although sadly limited by only editing a single file). It's just that it's also a pain to use.

(They're also correct that ed is the foundation of many other things in Unix, including sed and vi. But that doesn't mean that the best way to learn or understand those things is to learn and use ed.)

This doesn't make ed a useless, vestigial thing on modern Unix, though. There are uses for ed in non-interactive editing, for example. But on modern Unix, ed is a specialized tool, much like dc. It's worth knowing that ed is there and roughly what it can do, but it's probably not worth learning how to use it before you need it. And you're unlikely to ever be in a situation where it's the best choice for interactive editing (and if you are, something has generally gone wrong).

(But if you enjoy exploring the obscure corners of Unix, sure, go for it. Learn dc too, because it's interesting in its own way and, like ed, it's one of those classical old Unix programs.)

EdNoLongerGoodEditor written at 01:17:53; Add Comment

2018-08-19

Why I'm mostly not interest in exploring new fonts (on Unix)

Every so often, an enthusiasm for some new font or set of fonts goes around the corners of the Internet that I pay attention to. It's especially common for monospaced fonts (programmers seem to have a lot of opinions here), but people get enthused by proportional ones as well. I used to be reasonably interested in this area and customized my fonts and sometimes my font rendering, but these days I find that I generally take at most a cursory look at new fonts and tune out.

The first reason I tune out is limited Unicode glyph coverage. As I found out when I dug into it, xterm doesn't require your primary font to have CJK glyphs, but as far as I know it does require your font to have most anything else, including Cyrillic, Arabic, and Hebrew glyphs, if you're not going to see just 'glyph missing' boxes. I don't necessarily look at stuff in these languages in my xterms all that often, but it comes up every so often and I'd like to have things available in any new font, since I pretty much do currently.

(I have to admit that the most common place these character sets come up is when I'm looking at spam messages, email and otherwise.)

For proportional fonts, things are less clear to me. Firefox apparently still uses glyphs from multiple fonts as necessary (based on eg this bug), even with modern XFT/FreeType fonts. I believe that Gnome and probably KDE have some sort of system for this, which sometimes matters for programs like Liferea (syndication feeds) and Corebird (Twitter), but I have no idea how it all works or is controlled. If all of this works, it means that my nominal main proportional font wouldn't need complete glyph coverage, although I might have to do a bunch of configuration tweaking.

(Back in the pre-XFT days, Firefox used to have a quite complex and arcane system for finding glyphs to render, since X fonts often had very small coverage. If things went wrong this could produce rather bizarre results; Firefox was perfectly capable of switching fonts around in the middle of lines or words, even in nominally European text, if it decided that for some reason your primary font didn't have the necessary character and it had to fish it out from some other, very different font. All of this is fortunately behind us and I've mercifully forgotten most of the details.)

All of this matters because most alternate fonts don't have a very wide glyph coverage. This isn't surprising; drawing a lot of glyphs is a lot of work, and creating glyphs for non-Latin characters requires its own set of type design expertise in how those characters are supposed to look. In practice most artisanal new fonts are likely to be pretty much confined to Latin and near-Latin characters.

The bigger reason that I tune out these days is that standard Unix XFT fonts and font display are now pretty good. Back in the old pre-XFT days, X fonts (bitmapped or TrueType) were generally pretty limited and often not of particularly high quality, and rendering decisions could be questionable or oriented at hardware that you didn't have. This could make it very worthwhile to seek out an alternate font, because it could significantly improve your reading experience. I used an alternate bitmapped font in Firefox for years, and even after that stopped being a good idea I switched to an alternate TrueType font (one or the other is visible in this view of my desktop, although I no longer remember if it was the bitmapped 'Garamond' font or TTF Georgia).

The modern XFT world is much different. The current fonts that are mostly used and shipped by major desktop environments and Linux distributions are pretty well designed and have increasingly good glyph coverage (primarily this is DejaVu), and if you don't like them, there are alternates (eg Google's Noto). I've also come around to finding that the rendering decisions are generally good, which is a change from the past for me.

Given that the default fonts are already pretty good, the incremental improvement I might gain through fiddling around with my fonts doesn't appear to be worth the effort and the uncertainty. I could spend a lot of time tinkering, trying out various fonts, and basically wind up with something that was subtly worse because, to be honest, I don't know what I'm actually doing here (and I know enough to know that type design and typography is complex). So the easy path is just to use the defaults.

(This entry is sparked by reading vermaden's FreeBSD Desktop – Part 15 – Configuration – Fonts & Frameworks (via), and thinking about why I had no interest in doing something similar.)

Sidebar: Why Unix XFT fonts and font rendering got good

My impression is that part of it is simply the progress of time creating a slow but steady improvement, but a lot of it is that Unix/Linux font rendering became important enough that people spent money on it. Google obviously spent the money to develop Noto, but they're far from the only people. Sometimes this funding came explicitly, by commissioning font work and so on, and sometimes it came passively, where companies like Red Hat and Canonical hired people and had them spend (work) time on Unix font rendering.

(Android probably helped motivate Google and other parties here.)

MyFontDisinterest written at 01:09:47; Add Comment

2018-08-03

Firefox now implements its remote control partly over D-Bus

On Unix, Firefox has had a long standing feature where you could remote control a running Firefox instance. This has traditionally worked through X properties (with two generations of protocols), which has the nice advantage that it works from remote machines as well as your local one, provided that you're forwarding X. Since I read my mail through exmh that's running on one of our servers, not my desktop, this is pretty useful for me; I can click on a link in mail in exmh, and it opens in my desktop Firefox. However, working through X properties also has the disadvantage that it naturally doesn't work at all on Wayland. Since Wayland is increasingly important, last November or so the Mozilla people fixed this by adding a new D-Bus based protocol (it landed in bug 1360560 and bug 1360566 but has evolved in various ways since then).

On current versions of Firefox, you will find this service on the session bus under the name org.mozilla.firefox.<something>, where the <something> is often 'ZGVmYXVsdA__'. In general this weird thing is the base64 encoded name of your Firefox profile with a few special characters turned into _, and that particular name is, well:

; echo -n default | base64
ZGVmYXVsdA==

Because this directly encodes the profile name in something that you have to get right, the D-Bus based version of Firefox remote control will reliably restrict itself to talking to a running Firefox that's using the same profile; the X properties based version doesn't always (or didn't always, at any rate). You can force a new Firefox to not try to talk to an existing Firefox by using --new-instance, as before.

(One case where you might need this is if you're testing an alternate version of Firefox by manually setting your $HOME to, eg, /tmp/ffox-test.)

It turns out that which protocol Firefox uses when is a bit tangled. If Firefox is built with D-Bus support, a running Firefox on X will be listening for incoming requests using both D-Bus and the X properties based protocol; you can talk to this Firefox with either. In the current Firefox code, if you built with both D-Bus and Wayland support, the client Firefox always uses D-Bus to try to talk to the running 'server' Firefox; it doesn't fall back to X properties if there's no D-Bus available. If you built Firefox without Wayland support, it always uses the X properties based protocol (even if you built with D-Bus, and so the running Firefox is listening there). You can see this sausage being made in StartRemoteClient() here.

This logic was introduced in the change for bug 1465371. Before then Firefox tried to use the X properties based remote control if it was running on X, and fell back to the D-Bus protocol otherwise. In thinking about it I've come to believe that the logic here is sound, because in a Wayland session you may have some programs that think they're running in X and then pass this view on to things run from them. D-Bus is more session type agnostic, although it only works on the local machine.

Note that this implies that you can no longer use Firefox itself as a client on a second machine, at least not if your second machine Firefox is a modern one that was built with Wayland support; it'll try to talk D-Bus and fail because your running Firefox isn't on that machine. If you want to remote control Firefox from a second machine, you now want a dedicated client like my ffox-remote program.

(Hopefully Mozilla will leave the X properties based protocol there for many years to come, so my cross-machine remote control will still keep working.)

Sidebar: some D-Bus protocol details

The D-Bus object path is /org/mozilla/firefox/Remote, which has one org.mozilla.firefox method, OpenURL(), all of which you can see by using a D-Bus browsing program such as d-feet. In the Firefox source code, what you want to look at is widget/xremoteclient/DBusRemoteClient.cpp (the client side, ie the firefox command you just ran that is going to pass your URL or whatever to the currently running one) and toolkit/components/remote/nsDBusRemoteService.cpp (the server side, ie the running Firefox).

Despite the fact that D-Feet will tell you that the argument to OpenURL() is a string, in actuality it's an entire command line encoded in the same annoying binary encoding that is used in the current X property based protocol, which you can read a concise description of in nsRemoteService.cpp. Presumably this minimizes code changes, although it's not the most natural D-Bus interface. This encoding does mean that you're going to need some moderately tangled code to remote-control Firefox over D-Bus; you can't fire up just any old D-Bus client program for it.

The client code for this is in toolkit/xre/nsAppRunner.cpp, in the StartRemoteClient() function.

FirefoxDBusRemoteControl written at 23:00:04; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.