Wandering Thoughts archives

Some notes on how xdg-open picks which web browser to use

Let's start with my tweet:

It sure would be convenient if xdg-open documented how it picked which browser to use, so you could maybe change that to one you like.

XDG here is what used to be called the 'X Desktop Group' and is now freedesktop.org (per wikipedia). As you might then guess, xdg-open the program is the canonical KDE/GNOME/etc agnostic way of opening (among other things) an URL in whatever is nominally your preferred browser. Since providing a web interface as your package's UI is an increasingly common thing to do these days, there are any number of programs and scripts and so on that want to do this and do it by just running xdg-open, which gives you whatever strange browser xdg-open decided to inflict on you today.

If you read the xdg-open manpage, it's remarkably uninformative about how it decides what your default or preferred web browser is. On the one hand xdg-open is currently a shell script so you could just read it to find this out; on the other hand, it's a 900+ line shell script full of stuff, which makes it not exactly clear 'documentation'. So here are some notes on the current behavior, boiled down to the useful bits.

(Because none of this is documented, freedesktop.org is free to change it at any time if they feel like it. They probably won't, though.)

Broadly, xdg-open does the following when you ask it to open an URL:

• first, it tries to figure out if you're running a desktop environment that it knows about. If you are, it uses the desktop's own mechanism to open URLs if there is one (these are things like kde-open, gvfs-open, and exo-open). The important thing here is that this should automatically respect your normal desktop browser preference, since it's using the same mechanism.

• if $BROWSER is set, it's used immediately. I'll discuss what it can be set to later.

• in some but not all versions of xdg-open, it will attempt to determine the correct program to handle the nominal MIME type of 'x-scheme-handler/http' (or https, or ftp, or whatever) and then run this program. This determination is made by xdg-mime; how it works is complicated enough to require another entry.

  In versions of xdg-open that support this, it's extremely likely that this will produce a result and xdg-open will never go on to its final option.

• if all else fails, xdg-open tries to run a whole bunch of possible browsers (via the $BROWSER mechanism). The most important thing about this is that the first default browser xdg-open tries is x-www-browser.

Some but not all Linux distributions have a /usr/bin/x-www-browser. If yours has one, it may or may not point to something useful or what you want, and it may vary from version to version (and on what packages you wind up installing). For example, on our Ubuntu 12.04 and 14.04 machines x-www-browser points to arora by default, which is not what I consider a particularly good choice on machines with Firefox installed (it's certainly likely to puzzle users who wind up running it). On 16.04 it points to Chromium, which is at least a plausible default choice.

On current Fedora and Ubuntu 16.04, the full list of (graphical) browsers, in order, is:

x-www-browser firefox iceweasel seamonkey mozilla epiphany konqueror chromium-browser google-chrome

The useful thing here is that xdg-open searches for all of these using your $PATH, so you can create a personal x-www-browser symlink that points to whatever you want without having to risk unintended side effects from eg setting $BROWSER.

If your xdg-open uses xdg-mime, you can determine what program it will use (more or less) by running eg:

$ xdg-mime query default x-scheme-handler/http
firefox.desktop

Xdg-mime will let you set this stuff as well as query it, but of course doing so may have more consequences than just changing xdg-open's behavior. However at that point there is no really good solution; if xdg-open is consulting xdg-mime, I think all you can do is either set $BROWSER or set xdg-mime's defaults.

(If you're setting scheme handlers in xdg-mime, remember to get at least http and https, and perhaps ftp as well.)

While the xdg-open manpage will refer you to the xdg-settings manpage, this is potentially quite misleading. If you aren't using a desktop environment, there's no guarantee that what xdg-settings reports as your default browser is what xdg-open will necessarily run. The two scripts use different code and as far as I can tell, neither makes any particular attempt to carefully duplicate each other's logic.

(If you are running a desktop environment that xdg-settings knows about, well, you're at the mercy of how well it can read out your DE's default browser. I wouldn't hold my breath there.)

Sidebar: What $BROWSER can be set to

As xdg-open interprets it, $BROWSER is a colon separated list of commands, not just program names (ie you can include things like command line switches). A command may contain a %s, which will be substituted with the URL to be opened; otherwise the URL is just tacked on the end of the command. Xdg-open does not exactly have sophisticated handling of quoting and argument splitting or anything, so I would do nothing tricky here; I wouldn't even really trust the %s substitution. If you think you need it, use a cover shell script.

How xdg-mime searches for MIME type handlers (more or less)

XDG is what was once called the X Desktop Group and is now freedesktop.org. Part of their work is to create various cross-desktop and desktop agnostic standards and tools for how to do things like determine what program should handle a particular MIME type. This includes special 'scheme' MIME types, like x-scheme-handler/ftp, which means 'whatever should handle FTP' (technically this is 'FTP urls'). The XDG tool for mapping MIME types to programs (actually .desktop files) is xdg-mime. Like all of the XDG tools, xdg-mime uses a collection of environment variables, which will normally have the default values covered in the XDG base directory specification.

In theory there are two sorts of data that xdg-mime looks at. The first, found in files called <desktop>-mimeapps.list and mimeapps.list, is supposed to be a set of specifically chosen applications (whether configured by the user or the system). The second is a general cache of MIME type mapping information based on what program can handle what MIME types (as listed in each program's .desktop file); these are found in files called defaults.list and mimeinfo.cache. In practice, a system mimeapps.list file may have just been slapped together based on what the distribution thought was a good idea, and it may not correspond to what you want (or even what you have installed).

(Not all Linux distributions ship a system mimeapps.list. Fedora does; Ubuntu doesn't.)

Xdg-mime searches for mimeapps.list in any of $XDG_CONFIG_HOME, $XDG_CONFIG_DIRS, and the 'applications' subdirectories of $XDG_DATA_HOME and $XDG_DATA_DIRS. It searches for the other two files only in the applications subdirectories of $XDG_DATA_HOME and $XDG_DATA_DIRS.

Now, we get to an important consequence of this. $XDG_DATA_DIRS is normally two directories, /usr/local/share and /usr/share, each of which has MIME type information only for itself, and they are checked in order. The normal result of this is that things installed into /usr/local steal MIME types from things installed in /usr because xdg-mime will check the /usr/local/share/applications files first.

(I discovered this when I installed Google Chrome on my laptop and it promptly stole URL handling from Firefox, which I did not want, because the RPM put bits into /usr/local instead of /usr. Actually finding the files that controlled this was remarkably hard.)

Normally, nothing automatically generates or updates the system mimeapps.list; on Fedora, it's part of the shared-mime-info RPM (and kde-settings for the kde-mimeapps.list version). The mimeinfo.cache files are maintained by update-desktop-database, which will normally be automatically run on package installs, removals, and probably upgrades.

Now, xdg-mime does not give you an actual program to run. What it does is give you the name of a .desktop file, eg firefox.desktop. In order to use this to run a program, you must look through the right magic places to find the .desktop file and then parse it to determine the command line to run. Probably you don't want to do this yourself, but I don't know what your alternative is; as far as I know, there is no XDG tool script to say 'run this .desktop command with the following arguments'.

(Note that the .desktop file is not necessarily in the same directory as the MIME mapping file that caused it to be the handler application. For example, your $HOME/.local/share/applications might just have various MIME type overrides you've set for what system application should handle what.)

This explanation is somewhat dependent on what exact version of the XDG tools and scripts you have installed. It's also not necessarily totally complete, because I am reading through undocumented shell scripts here and I've left a few things out. See the Arch wiki entry on default applications for much more information.

PS: If you're trying to track down why xdg-mime is giving you some strange result, set $XDG_UTILS_DEBUG_LEVEL to at least '2'. This will tell you just what files it looked at when, although I don't think it ever reports what directories it looked at but didn't find any files in.

There are (at least) two sorts of DNS blocklists

Here is a trite and obvious thing that I never the less feel like writing down: in practice, there are (at least) two sorts of DNS anti-spam blocklists. Since I want to use value neutral terms here, let us call these 'simple' and 'complex' blocklists.

The operation of a simple DNSBL is, well, simple. If it sees spam from an IP, it lists the IP (or if it sees whatever is the DNSBL's idea of 'bad stuff'). Usually the IP gets automatically delisted after a while, but in some DNSBLs the listing lasts forever unless someone takes action to have it get cleared, appealed, or whatever.

A complex DNSBL attempts to have a more complex balancing criteria for adding listings than simple presence of spam; for instance, it may somehow assess how much apparently legitimate traffic it's seen from the source IP as well as spam volume. A complex DNSBL is sometimes going to be slower to list an IP than a simple DNSBL.

A simple DNSBL does not have 'false positives' as such (assuming that it's honestly run), but that's because a listing means something very narrow; it means that the IP did a bad thing within the time horizon. People who reject email based on a listing in a simple DNSBL may have false positives in that rejection, though, because an IP doing a bad thing once doesn't necessarily mean that it will do it every time. Complex DNSBLs can have false positives because they're fundamentally intended to assert that the email you're getting is probably bad. Good operators of complex DNSBLs attempt to minimize such false positives.

To give an example of each, the Spamhaus SBL is a complex DNSBL (or at least generally it is). The CBL is a simple DNSBL, but one that (theoretically) uses a very narrow listing criteria that is very strongly correlated with sending only spam.

Unfortunately not all DNSBLs make it clear what sort of DNSBL they are in their description (or sometimes they wave their hands about it a bit). At least at the moment, one quite strong signal that you are dealing with a simple DNSBL is if it ever lists one of GMail's outgoing mail servers.

(I feel that rejecting email based on a simple DNSBL is not necessarily a mistake, but the sidebar attempting to explain this got long enough that it's going to be another entry.)

Some notes on adding exposed statistics to a (Go) program

As a slow little project for the past while, I have been adding some accessible statistics to my sinkhole SMTP server, using Go's expvar package. This has resulted in me learning lessons both about expvar in specific and the process of adding statistics in general.

My big learning experience is going to sound fairly obvious and trite: I only really figured out what statistics I wanted to expose through experimentation. I started out with the idea that counting some obvious things would be interesting (and to a certain extent they were), but I created many of the lot of stats by a process of looking at the current set and realizing that there was information I wanted to know or questions that I wanted answered that were not covered by existing things I was exposing. Sometimes trying to use the initial version of statistic showed me that it was too broad or needed some additional information in order to be useful.

The corollary to this is that what statistics you'll want depends in large part on what questions are interesting and informative for you, which depends on how you're using the program. A lot of my stats are focused on anti-spam related issues, because that's how I'm using my sinkhole SMTP server. Someone using it to collect email from a collection of nodes and tests might well want a significantly different set of statistics. This does make adding stats to a theoretically general program a somewhat tricky thing; I have no good answers to this currently.

(I have not tried to be particularly general in my current set of stats. Since this has been an experiment to play around with the idea, I've focused on making them interesting to me.)

Just exporting statistics from a program is less general than pushing events and metrics into a full time series based metrics system, but Go's expvar package and a few other tools like jq makes it much easier to do the former (for a start, I don't need a metrics system). Exporting statistics is also not as comprehensive as having an event log or the like. Since I do sort of have an event log, I've chosen to view my expvar stats as being an on-demand summary of it, one that I can look at without having to actively parse the log to count things up.

And on another obvious note, putting counters and so on in a hierarchical namespace is quite helpful for keeping things comprehensible and organized. To some extent a good hierarchy can substitute for not being able to come up with great names for individual statistics. And sometimes you have data with unpredictable names that has to be confined to a namespace.

(For instance, I track DNS blocklist hit counts. The names of DNS blocklists are essentially arbitrary, so I put the whole set of stats into a dnsbl_hits namespace. And because the expvar package automatically publishes some general Go stats on things like your program's memory usage, I put all of my stats under a top-level name so it's easy to pick them out.)

I accept that someday I'll give up MH and move to IMAP mail clients

My current email tooling is strongly built around MH, using both command line tools and exmh. MH assumes a traditional Unix mail environment where your inbox can be accessed through the filesystem, and more than that it fundamentally assumes that it entirely owns your email. As many people who try it out find out to their regret, MH's only interaction with the regular Unix mail ecosystem is to get your mail out of your Unix inbox as fast as possible.

So far I've been able to use and keep on using MH because I've worked (and had my personal email) on Unix systems that handled email in the traditional Unix way, with your inbox directly accessible through the filesystem in /var/mail and so on. However, these are a vanishing breed, for reasonably good reasons, and in the modern world the generic way you get at your email is IMAP. IMAP is not very Unixy, but it's what we've got and it's better than being stuck with proprietary network mail protocols.

MH and IMAP not so much don't get along as don't interact with each other. As far as I know, MH has no protocol support for IMAP, which is not surprising; IMAP is designed to keep all of your email on the IMAP server, which is completely opposite to how MH operates. It might be nice to have an 'IMH' system that was a collection of command line programs to manipulate IMAP mail and folders, but no such thing exists that I know of and it's unlikely that anyone will ever write one.

Some day I will have to use a mail system that can only be accessed over IMAP. In theory I could deal with this by using a program to pull all of my email out of IMAP and hand it over to MH as local mail; there are a number of things that will do variants of this job. In practice my feeling is that doing this is swimming upstream against a pretty powerful current, and thus generally a mistake. Certainly I expect that I won't be energetic and annoyed enough to do it. By that point MH will have had an extremely good multi-decade run for me, and very few programs last forever. I can change.

(Also, by that point I expect that I will be really tired of how MH and exmh don't really deal all that well with HTML email, because I expect that HTML email is only going to get more and more common from now onwards.)

PS: The clever reader will have guessed that I don't currently have a smartphone or other limited secondary device that I want to read my email from, because all of those really want you to use IMAP (or something even less MH-cooperative, like GMail). That may change someday, at which point I may have to re-think all of this.

Sidebar: I don't see this happening any time soon

Locally we're still very strongly attached to filesystem accessible inboxes and procmail and all of the other artifacts of a traditional Unix mail system. There would be quite a lot of upset users if we took that away from them, so I don't expect it to happen short of something truly drastic happening with our overall mail system.

(Nor is the department likely to give up running its own email system any time soon.)

As for my personal email, well, that's tangled but certainly my attachment to MH complicates my life. There are lots of places that I could get IMAP mail, probably even IMAP mail for a custom domain, so if I was happy with IMAP alone life would be quite a bit easier. Until I de-tangle my personal email it gets a free ride on work email's MH friendliness; after I de-tangle it, I will probably still run my own servers for it and so I could run MH there if I wanted to.

(At that point I might want to switch to IMAP for various reasons.)

An email mistake I've made as a long-term university sysadmin

I've been a sysadmin here at the University of Toronto for quite a long time, which has enabled me to make a natural and probably common (university) mistake with my email. Namely, I have by now totally commingled my work email and my personal email by the simple mechanism of just using my normal university account for everything. I get random technical mailing lists I'm interested in sent to my address here, any number of people who correspond with me use my address here, and so on.

This is unfortunate for various reasons, including that it makes taking a break away from work email and work systems much harder. I can't just not log in, because if I do I'll miss personal email, and the moment I log in it starts being tempting to take a peek at various work things.

Some people would and could deal with this by moving their personal email to an outside provider (GMail is obviously a popular choice). This is completely possible, and in fact I got people to change the personal email they used for me once (I moved it between systems here). Unfortunately I'm very attached to how I handle my email today. The tools that I use intrinsically require a Unix system with a local mail spool, and running my own Unix system for this is (still) enough of a hassle that I haven't taken a deep breath and just got to it.

(The problem with running my own Unix system is not the basic work, it's all of the additional things I'd have to worry about and spend my limited free time on. There's backups and maintenance and monitoring and keeping track of how to rebuild everything and so on and so forth. All of this is simple at work because we've built an entire infrastructure to make it that way. And then there's the whole issue of anti-spam filtering, where I currently get to lean on a commercial package.)

I know it's less than ideal to keep my email commingled, but it's just easier to let this situation go on. Everything works today and I don't have to worry about any number of things and most of the time the commingling barely matters. Inertia is a powerful force, as is little incremental steps; they're how I've wound up in this situation in the first place. Big bang things like setting up a new mail system et al from scratch are hard, because there's so much work before you get anywhere.

(I should do it anyways. Someday.)

Sidebar: My past efforts at this

At one point I attempted to have at least a personal versus work email address split here, but that went down in flames many years ago because spammers got their hands on my then personal email address (partly because the address predated spammers so I did not take the extensive array of precautions that I do today). Today that old address lives on basically only as a way of getting information about spammer behavior (eg).

Almost ten years ago when I shifted to Computer Science I had the opportunity to (re)split personal email from work email (since I was changing my primary email address anyways), but at the time I was so busy with other aspects of the transition that I didn't really have either time or energy to even think about setting up a new Unix system with a new email setup and so on. Anyway, ten years ago one didn't have the modern wide variety of inexpensive hosting options, at least as I imperfectly remember it.

My concern about the potential dominance of the mobile web

Yesterday I wrote about how I don't know and can't find out how dominant the mobile web actually is. Today, let's take the (implicit) claims of the 'mobile first' design people as given and posit that the mobile web is either already dominant or going to become dominant in the not too distant future. In that case, well, I have concerns. In fact they are predictable concerns and are basically wrapped up in that nice phrase of 'mobile first' design.

Put simply, the web has seen this show before but back then it was usually called 'IE first design' or before then 'Netscape first design'. What these really translate to is that everyone else is a de facto second class citizen. Oh, sure, some amount of designers have good intentions and want to do a good job for other people (like the desktop web), but designers (and web developers) also have deadlines and priorities and only so much time. When you say 'mobile first', you implicitly accept that desktop people are allowed to have a degraded experience. That's what putting someone second means.

This is a direct concern of mine because I'm exclusively a desktop user, and more than that a Linux user of Firefox. This puts me well outside even the outside area of a broad 'mobile first' design (which is likely to target Safari and Chrome). This goes beyond worries about non-functional websites, as mobile first designs are likely going to be built for significantly smaller screen sizes than I have (well, smaller browser window sizes than I have). A functional website that wastes three quarters of the screen space I have and squeezes all of the content into a small area is probably not going to make me very happy.

(In theory responsive design can deal with this. In practice, I'm not at all convinced that people can (or will) make designs that scale across that large a screen size change, especially if mobile becomes their first priority.)

All of this concern is probably somewhat overblown, for many reasons including that much of my use of the web is fundamentally a diversion. In fact, the growth of the mobile web may have seen off the largest threat to the usable web that I've faced, namely the growth of Javascript-mandatory websites. Those defy even 'view page in no style' or Firefox's reader view.

(My understanding is that JavaScript on mobile is a lot slower and more power-hungry than it is on desktop, especially on lower end smartphones. My impression is that this has led to less client side JS and more server side content rendering in mobile focused website designs, although I may be out of touch here.)

How dominant is the mobile web (versus the desktop web)?

I was all set to write an entry about how the increasing dominance of the 'mobile web' (web usage from mobile devices) concerned me, but then I made the mistake of trying to find some reasonably current and solid facts about how web traffic is split between mobile and desktop and how it's been changing. Now I'm writing a different entry, because when I started looking I couldn't find anything that felt solid enough.

Everyone seems to agree that mobile usage as a whole is quite high (at least in first world countries), but a bunch of sources also claim that a lot of that time is spent in apps, not in browsing the web from mobile devices. Google apparently said in 2015 that more searches came from mobile devices than desktops in a number of countries (including the US and Japan). But the only actual usage numbers I could find seem to still have mobile browsing under half of browsing, and apparently not changing too fast; however, I have no idea how trustworthy those numbers are.

(Most people who put forward numbers seem to be out to sell you something, and who knows how reliable their methodologies are. I do believe Google's claim, but they did carefully hedge it.)

There is certainly a 'mobile first' movement for website design, but again there seems to be an aspect of boosterism to it. Certainly there have been plenty of past web design 'trends' that people advocated quite hard for but that never materialized in practice to the degree that their boosters thought (semantic markup, anyone?). And to the extent that this material is produced by pundits, well, pundits have to have something to hold forth about and that's often a future trend instead of a present reality.

(Some of this is a self-fulfilling prophecy, too. Depending on your perspective and how the website works, a 'mobile first' website might either unleash the pent up demand for well-done content from mobile devices or drive away some amount of the remaining desktop usage.)

Sidebar: 'Important' versus 'dominant'

If the mobile web is approaching half of all your web traffic, it's certainly big enough to be worth catering to and designing for. But if it's only (less than) half of traffic you can't focus on it to the exclusion of desktop design, the way you might if desktop usage was (say) only 20% and decreasing.

(It seems pretty clear that including the mobile web in your design is important these days. If nothing else, Google will rap your knuckles in search results if they don't consider you mobile-friendly (although some sources say that that's only for mobile searches).)

My views of Windows 10 (from the outside)

This sort of starts with my tweet:

It's funny; I have a great deal of anger for what Microsoft has done with Windows 10, even though none of it affects me directly.

Since I was asked, I'm going to change my mind and write enough here to explain myself.

Based on information casually available to an outsider to the Windows ecosystem, Microsoft has done two things with Windows 10. First, they have significantly to drastically increased the amount of privacy invasive 'telemetry' that Windows 10 installs send to Microsoft and have also added all sorts of advertising to it. The normal versions of Windows 10 will pitch you in the Start menu, on lock screens, in included Windows applications like Solitaire, and so on.

Second, as everyone has heard by now, Microsoft has been aggressively pushing the upgrade to Windows 10 on people (or more accurately Windows machines). At this point it seems to be almost impossible to escape the upgrade; certainly it requires so many contortions that many people will be upgraded even if they don't want to be. Stories abound about important PCs in various places basically being hijacked by these forced upgrades.

All by themselves, either of these things would be bad and obnoxious; no one wants ads, invasive telemetry, or forced upgrades. Together they ascend to an entirely new level of nastiness, as Microsoft is forcing you to upgrade to an intrusive, ad-laden new operating system (and they've made it clear that the amount of ads will be increasing over time). The whole thing also comes at what could politely be called a bad time for both ads and privacy intrusion; people are becoming more and more sensitized and angry about both, as we see with the popularity of adblockers and so on.

In my view, what Microsoft has done is to reveal that as long as you use a Microsoft operating system, your computer really belongs to Microsoft instead of you. By forcing this upgrade to an OS with very different behavior for advertising and privacy intrusion, Microsoft has now demonstrated that they are willing to drastically change the terms on which they let you use your computer, as they see fit. Your computer and OS does not exist to benefit you, it exists to benefit Microsoft. If it is not doing enough for them, they will change things until it does and you do not get a vote in the matter.

(Microsoft could try to sell more telemetry as better for you, but that is absolutely impossible with ads. Ads universally make your experience worse. By including and then increasing ads in Windows 10, Microsoft is clearly prioritizing themselves over you in the operating system.)

In my view, by doing this Microsoft has shown that they are not particularly different from the big OEMs who have for years been loading down Windows laptops and desktops with pre-installed crapware. Dell, HP, Lenovo, et al have all been more than willing to ruin the experience for people buying their hardware in order to make some additional money from other channels; now Microsoft has joined the crowd. As a result, Microsoft is just as un-trustworthy as the big OEMs are.

(More fundamentally, Microsoft is showing that they do not care about people's experience of using their operating systems, or at least that they don't consider it a priority. Microsoft will happily make your time using Windows 10 less pleasant in order to deliver some ads. And as you know, when you are clearly not the customer, you are the product. It is especially offensive to be the product when you are paying for the privilege, but apparently that is life in Microsoft's world.)

I very much hope that this winds up causing Microsoft massive problems down the road. There certainly should be consequences to changing your product from a premium thing that was the best solution to a downmarket option used by people who don't have the money to avoid the annoyances it inflicts on them. However, I cynically doubt that it will, and it may be that Microsoft Windows has already become the downmarket product that Windows 10 positions it as.

In the mean time the whole situation makes me angry every time I consider it, especially when I think of the various relatives and people I know who will have no choice but to use Windows 10 and be subjected to all of this. If Microsoft goes down in flames someday, this move of theirs has made sure that I will applaud the fires.

Sidebar: The danger of intrusive telemetry

The ever more intrusive (default) telemetry makes me especially angry because if there is one thing we have learned over the past three, five, or ten years, it is that collecting and retaining data is inherently dangerous. Once that data exists it becomes a magnet for people who want a look at it, whether that is with subpoenas in civil lawsuits, warrants in criminal cases, or NSLs from three letter agencies. Today, the only safe thing to do with data is not collect it at all or at the very least, totally minimize your collection. That Microsoft has chosen to do otherwise basically amounts to them shrugging their shoulders over the fundamental privacy of people using their operating system.

(Now we know how much Microsoft really cares about the privacy of people using their systems, as opposed to things that cause inconvenience or bad PR for Microsoft.)

I work in what is increasingly a pretty different sysadmin environment

I've written before about what our sysadmin environment is like but that description doesn't really convey how and why our environment is increasingly different from what the rest of the world seems to be moving to, with the resulting very different needs. Today I'm going to describe our environment from another perspective .

In our environment, we broadly do three different things as far as computing goes. First off, we provide a number of standard services to people in the department, things like Samba file service, printing, email with IMAP (and webmail), DNS, and so on. This sort of internal service provision is probably still quite common in reasonable sized organizations (trendy ones have no doubt outsourced it to GMail, Dropbox, et al). Users are not exposed to the backend details of what software stacks power these services, and although our current stacks are stable we have shuffled them around in the past and could in the future. We definitely feel no need to run the latest and greatest software stacks and versions, and generally prefer to leave these services alone for as long as possible (this too is probably common in organizations doing this).

Second, we provide general multiuser computing to our users in several forms (general login service, compute login service, and various forms of web service ranging from plain HTML pages through completely custom web servers that they run). Naturally, how people use these services and what they run varies widely; we frequently get requests to install various bits of open source software that people want, for example. Our users obviously are pretty exposed to what OS and software we're running, and we couldn't make significant changes in it without serious disruption (even mild changes like Ubuntu version upgrades can be disruptive). Our users also care a fair bit about having current or relatively current software in this environment (for a wide variety of open source software). My impression is that we are one of the few environments left that provides this sort of computing.

Finally, the large collective we run a certain amount of custom services and applications, both for people inside the department and for people outside it. Some of these services are developed by the same sysadmins who run the hosts they're on, but others are increasingly going to have separate developers and sysadmin people (these are generally the complicated applications). Users of these systems aren't exposed to the backend details, but obviously the developers are since they have to write code for some deployment environment. The developers probably care (to some extent) about working with commonly chosen environments (eg Linux and Apache) and with current or reasonably current versions of things like databases, web servers, and so on. This sort of thing is closest to ordinary 'operations' or 'devops' work in the outside world but is generally less demanding for various reasons (there is very little here that could be described as 'business critical', for example).

(I wouldn't be surprised if some day we wind up with developers who want to deliver their applications as Docker containers or the like, rather than dancing around with asking us to set up a database this way and an Apache/PHP web environment that way and so on.)

So far, nothing in our environment faces high load or high demand, neither in our standard services nor our custom services. When unusual surge demand descends, so far it is not really our problem; we have some responsibility to keep the overall system up and responding, but very little to make sure the specific service affected does not collapse under the load it's experiencing.

In theory we could run these three different sorts of environments using different operating systems and software stacks. In practice we have limited staff and so the needs of the multiuser computing wind up spilling over to affect what we run for the other environments; provided that it works reasonably well, it's simply less work to have a uniform setup across all three environments. Today most of that uniform environment is Ubuntu LTS, because Ubuntu LTS remains the best environment for providing the multiuser computing part of things.