Chris's Wiki :: blog/unix/XNetworkTransparencyFailure

X's network transparency has wound up mostly being a failure

January 26, 2018

I was recently reading Mark Dominus's entry about some X keyboard problems, in which he said in passing (quoting himself):

I have been wondering for years if X's vaunted network transparency was as big a failure as it seemed: an interesting idea, worth trying out, but one that eventually turned out to be more trouble than it was worth. [...]

My first reaction was to bristle, because I use X's network transparency all of the time at work. I have several programs to make it work very smoothly, and some core portions of my environment would be basically impossible without it. But there's a big qualification on my use of X's network transparency, namely that it's essentially all for text. When I occasionally go outside of this all-text environment of xterms and emacs and so on, it doesn't go as well.

X's network transparency was not designed as 'it will run xterm well'; originally it was to be something that should let you run almost everything remotely, providing a full environment. Even apart from the practical issues covered in Daniel Stone's slide presentation, it's clear that it's been years since X could deliver a real first class environment over the network. You cannot operate with X over the network in the same way that you do locally. Trying to do so is painful and involves many things that either don't work at all or perform so badly that you don't want to use them.

In my view, there are two things that did in general X network transparency. The first is that networks turned out to not be fast enough even for ordinary things that people wanted to do, at least not the way that X used them. The obvious case is web browsers; once the web moved to lots of images and worse, video, that was pretty much it, especially with 24-bit colour.

(It's obviously not impossible to deliver video across the network with good performance, since YouTube and everyone else does it. But their video is highly encoded in specialized formats, not handled by any sort of general 'send successive images to the display' system.)

The second is that the communication facilities that X provided were too narrow and limited. This forced people to go outside of them in order to do all sorts of things, starting with audio and moving on to things like DBus and other ways of coordinating environments, handling sophisticated configuration systems, modern fonts, and so on. When people designed these additional communication protocols, the result generally wasn't something that could be used over the network (especially not without a bunch of setup work that you had to do in addition to remote X). Basic X clients that use X properties for everything may be genuinely network transparent, but there are very few of those left these days.

(Not even xterm is any more, at least if you use XFT fonts. XFT fonts are rendered in the client, and so different hosts may have different renderings of the same thing, cf.)

What remains of X's network transparency is still useful to some of us, but it's only a shadow of what the original design aimed for. I don't think it was a mistake for X to specifically design it in (to the extent that they did, which is less than you might think), and it did help X out pragmatically in the days of X terminals, but that's mostly it.

(I continue to think that remote display protocols are useful in general, but I'm in an usual situation. Most people only ever interact with remote machines with either text mode SSH or a browser talking to a web server on the remote machine.)

PS: The X protocol issues with synchronous requests that Daniel Stone talks about don't help the situation, but I think that even with those edges sanded off X's network transparency wouldn't be a success. Arguably X's protocol model committed a lesser version of part of the NeWS mistake.

Comments on this page:

By Opk at 2018-01-26 06:13:17:

I also use X 's network tranparency all the time. What X got right is the convenience with the DISPLAY environment variable and the fact that it works well for single applications and single windows. It worries me that I'll lose this with Wayland. Alternatives like vnc and rdesktop mostly only work with full desktops and even then they are not exactly convenient. VNC was meant for desktop sharing of single desktop PCs so the port number is not dynamic which doesn't work well when you have 70 potential users on a server.

X gets criticised in part because it isn't being used in the way it was originally designed. With modern GUI toolkits, all drawing, font rendering etc is done on the client and dumped to the server as a pure bitmap. There's good reasons for this and Wayland is probably a good thing but I really wish that there was a usable replacement to X's network transparency.

By Ade at 2018-01-26 16:06:24:

When I first started in university IT, we had a whole lab of Sun 3s converted to X11 consoles. (Mosaic hadn't come along at this point, and even when it did I used to run it remotely on more powerful servers - but then, it was still used mainly for hyperTEXT.) Lol, remember when it was going to be thin clients all the way?

By David Magda at 2018-01-26 18:41:31:

I think it's a matter of perspective. Network transparency worked quite well for many decades, and it's only been more recently with graph-heavy stuff that it became less effective.

If we say that X was first released in 1985, and was still good up until about 2005 or so, that's twenty years. Not a bad run.

Even now it's still useful for many applications, even if web browsers aren't one of them. Emacs, Matlab, R(studio), etc., are still used in many places. Throw in xpra and and you can have a location-agnostic workflow that handles network hiccups quite well.

By Mark Dominus at 2018-01-27 14:31:08:

I use emacs and xterm remotely with X all the time, but I long ago gave up on the idea of having X handle the network part for me. Instead, I invariably use ssh to connect to the remote host, run inside a local terminal. The remote process is then a shell or a non-X-capable emacs session running under SSH, sending character data back to be rendered locally.

I've tried many times to use X's network connectivity instead (and I do have my X connection tunneled through my ssh connection, just in case) but the payoffs seem much smaller than the costs. Principally: the event latency is to poor to do anything that isn't trivial.

Another significant problem is that the software environment is less uniform, because on each host I have to use whatever emacs, terminal installation, fonts, local config, etc are available there, and I never quite know what it is going to be. So, for example, on my local machine, I can use Konsole, and get the fonts and the local configuration I expect, and do the tab management locally. In a remote login to machine P, I can't use Konsole at all, because it's not installed. The remote machine has xterm, but xterm looks like crap. I could install Konsole on the remote machine, dragging in an additional 125MB of KDE libraries, but for what? Clumsy mouse copies and slow scrollback? Editing that only works over a fast connection?

I notice that Chris's example of `xrun` only has it running terminals and editors on the remote machine. `xrun` may be an X client, but it doesn't have to be. Instead of running remotely, getting the button click over the network, and running an X client remotely, it could just as easily run locally, get the button click locally, and then run `xterm -e ssh $remote command args` to pop up the new window. If there's a benefit to using the x protocol here, I don't see what it is.

So I'm not persuaded.

By cks at 2018-01-27 17:19:24:

The advantage of xrun is that it exists (on the remote machine) in an environment where my login session and shell has already been fully initialized. This makes starting new xterms more like telling screen or tmux to open up a new screen and less like actually logging in. However, this is a difference that matters much less today, now that machines are much faster than they used to be and logging in is not such a slow process any more.

(My home version of my X environment uses sshterm, which does ssh to a host in an xterm, and in practice I often can't tell it apart from my work version in terms of speed.)

I also routinely run genuinely graphical X programs remotely, even if they are only using those graphics to display text, things like the sam editor and exmh (and occasionally things like kdiff3 or gitk). But as Mark Dominus implicitly mentions, this is an unusual usage case today.

David Magda: I have to disagree about the time range when remote X worked really great. I'm pretty certain I could definitely percieve a lag on 10 MBit Ethernet and I think even still on 100 Mbit to some degree (especially if the remote machine was not on the local subnet). Only once 1G Ethernet became pervasive did things hit the point where it didn't matter for X GUI programs that dealt with text.

(To some degree I think this was a product of what display depth you were working at and how the programs you used did things. I moved to colour displays fairly early, so I was paying extra over the people who were on black & white ones.)

This also points out something that may be affecting Mark Dominus more than it does me: the further away your remote machines are, the more remote X degrades. I live very 'close' to the machines I run X on remotely, either on the same 1G subnet or only a few hops away over the university 1G+ network (I believe our backbone is at least 10G). Many people now work with remote machines that are further away than that and with slower-speed networking between them and those machines (for example, servers in various clouds, or even just servers in off-premise colocation facilities). High latencies and lower bandwidth bites remote X much harder than it bites plain SSH.

By Walex at 2018-01-31 14:59:25:

«Lol, remember when it was going to be thin clients all the way?»

But that actually happened: with web-based services the overwhelming mode of computer access is "thin client", like tablets and cellphones, running web browsers. Consider the popular Google mail and office applications, what's that if not running a thin client UI to a remote server? We are talking about over 2 billion users here...

By Aristotle Pagaltzis at 2018-02-01 02:29:03:

A contemporary web browser is an incredibly obese kind of thin client.

I suppose you can counter that a Google data center is an unfathomably obese kind of fat server too.

I guess the upshot should be that 30-year-old terminology is no match for current realities?

By Graham Lee at 2018-02-06 04:01:23:

I went to a talk on the history of the UK academic network, JANET, probably in 2014 at its 30th anniversary. One of the former directors mentioned there that the "killer application" for internet protocol on that network, which was formerly X.25-based, was the X window system. Obviously that's a network with a very specific audience and a lot of resources, which led to X's success in that environment.

By Burz at 2018-03-20 11:07:40:

Agree with most of this, but author overlooks significant developments by Apple and MS (as contrast to X) and the NX protocol, the latter being a great improvement over regular X network transparency on Unix that didn't take off.

The lesson here is there's only one corner of the Unix world that is focused on consumer needs (such as Internet conferencing/window sharing) and that is OS X. The rest (Linux distros et al) treat anything graphical as stuff that gets in the way of their 1337 command line experience.

By Burz at 2018-03-20 11:14:25:

The comment by Walex, BTW, is an example of not getting the point. Remote clients are important and the world developed a set of technologies that maximize parallelism on the client.

Written on 26 January 2018.

«	What the Linux `rcu_nocbs` kernel argument does (and my Ryzen issues again)

A misbehaving SMTP sender can fool me about malware volume