Wandering Thoughts archives

2018-01-19

I'm one of those people who never log out from their desktop

Only crazy people log out from their desktop every time they step away from it for moderate amounts of time. Whether you're leaving to get lunch or to go to a long meeting, sensible people just lock the screen (something that I've deliberately made very easy in my X setup). But my impression is that a fair number of people log out at the end of the day, or at least the end of the week.

I'm not one of those people; not at home and especially not at work. With rare exceptions, I log in when I boot up my machine and then I stay logged in until I'm going to reboot it (and then I log right back in again). When I leave, whether for an evening, the weekend, or the university's multi-week winter break, I just lock my X session (which at least purges my SSH keys). As a sysadmin who cares about security to some degree, this can feel a bit embarrassing; it would probably be moderately more secure to actually log off my office machine every night and log in again every morning.

(At home there's less reason to worry about the security issues and I use my desktop every day.)

A large part of why I do this is simply that I'm lazy. Both locking and unlocking my screen are a lot faster than logging out (in an orderly way) and then starting up my X session all over again. While I've automated a fair amount of starting my X session, there's still a number of manual steps involved (for example, I start some programs by hand and manually place their windows). The whole thing is enough of a hassle that I don't feel inclined to do it more often than I really have to. It also takes a bit of time, for various reasons; even if everything magically started automatically, it would probably take sixty seconds or so until my desktop was all up and running.

(Logging out requires a bit of work because things like Firefox are much happier if I shut them down in an orderly way instead of just yanking the X session out from underneath them.)

A certain amount of this manual startup work is because I've added a few more 'always present' windows but haven't gotten around to adding them to my startup script. Some of them are a bit awkward to automate (because they are really 'start an xterm with a shell, then run a command in the shell'), but I could probably glue something together. Other programs have to be started by hand because they provide no way to specify things like where to place their windows or that they should start iconified (with their icon in a specific spot). Possibly I could arrange a sufficiently complicated set of supporting scripts to automate this (using things like wmctrl), but just not logging out is a lot easier.

Staying logged in all of the time has some interesting consequences. The obvious one is that I normally keep all of my regular X programs running continuously for days on end (and sometimes weeks). Unsurprisingly, programs do not always expect this or handle it perfectly. Even when a program doesn't have issues with running for a long time, it may do somewhat inconvenient things like only loading certain information at startup.

(Awkwardly, one of the programs I use with this 'only on startup' flaw is one that I wrote myself. My excuse is that it was by far the easiest way to code that particular feature, the data involved doesn't change often, and I can always restart the program if I need to. Still, I should probably fix this someday.)

StayingLoggedOn written at 19:11:58; Add Comment

2018-01-14

Our small tools for running commands on multiple machines

A while back I wrote about the personal shell scripts I had for running commands on multiple machines. At the time, they were only personal scripts that I used myself; however, over time they kept informally creeping into worklog entries that documented what we actually did and even some shell scripts we have to pre-write the commands we need for convoluted operations like migrating ZFS filesystems from server to server. Eventually we decided to adopt them as actual official scripts, put in our central location for such scripts.

My own versions were sort of slapped together, especially the machines script to print out the names of machines that fall into various categories, so making them into production-worthy tools meant cleaning that up. The oneach script needed only moderate reforms and as a result the new version is only slightly improved over my old personal version; in day to day usage, I probably couldn't notice any difference if I switched back to using my old one.

(The big difference is that the production version has more options for things like extra verbosity and a dryrun mode that just reports the ssh commands that would be run.)

The machines command got completely redone from scratch, because I realized that my hack approach just wouldn't work. For a start, I couldn't ask my co-workers to edit a script every time we added a machine; there would have been a revolt. So I wrote a new version in Python that parsed a configuration file. This new production version is a drastic improvement over my shell script hack; because I wrote it in Python, I was able to include significantly more features, in addition to making it more convenient and regular (since it's parsing a configuration file). The most important one is support for 'AND' and 'EXCEPT' operations, so you can express machine categories like 'all machines with some feature that are also Ubuntu 16.04 machines' or 'all Ubuntu 14.04 machines except ...'. This is supported both in the configuration file, where it sees a little bit of use, and on the command line, where I take advantage of it periodically.

(The configuration file format is nothing special and basically duplicates what I've seen other similar programs use. Although I didn't consciously set out to duplicate their approach, it feels like we wound up in the same spot because there's only so many good solutions for the problem.)

Using a configuration file doesn't just make things more convenient and maintainable; it also makes them more consistent, in several senses. It's now much harder for me to accidentally forget to add machines to categories they should be in (or not remove them from categories that no longer apply). A good part of the reason is that the configuration file is mostly inverted from how my script used to do it. Rather than list machines that are in categories, it mostly lists the categories that a machine is in:

apps0    apps  ubuntu1604  allnfs  users

There are a few categories that are explicitly specified, but even then they tend to be in terms of other categories:

all=ubuntu1604 ubuntu1404

This approach wouldn't have been feasible in my original simple shell script, but it's a natural one once you have a configuration file (especially if you want to make adding new machines easy and obvious; for the most part you can copy an existing line and change the initial host name).

In theory I could have done all of these improvements in my own personal versions, and writing the Python version of machines didn't take too long (even writing a Go version for my own use only added a modest amount of time). In practice it took the push of knowing that these had to now be generally usable and maintainable by my co-workers to get me to spend the time. Would it have been wrong to spend the time on this when they were just personal scripts? Probably, and even if not I doubt I could have persuaded myself of that. After all, they worked well enough as they were originally.

ToolsOneachII written at 02:15:11; Add Comment

2018-01-04

The goals and problems of our Dovecot IMAP configuration migration

We have a long standing backwards compatibility issue with our IMAP server, which is that we have it configured so that the root of the IMAP mail folder storage is people's $HOME. Originally this led to Dovecot session hangs, but now it's led to running out of inodes on the Dovecot server machine and general NFS load as people's Dovecot sessions rummage all through their home directories on our fileservers. Today I'm going to talk about our ideal IMAP configuration, the problems of trying to migrate to it, and then some thoughts on what we might settle for.

(In other words, our Dovecot configuration currently sets mail_location to 'mbox:%h:INBOX=/var/mail/%u'.)

If I could wave a magic wand, the Dovecot IMAP configuration we want is simply one where all mail folders are stored under some directory in people's home directories, say $HOME/mail, and IMAP clients wouldn't need or want an IMAP path prefix. In this world, if you had a mail folder for private stuff your client would know it as the PrivateStuff folder and it would be stored in $HOME/mail/PrivateStuff. Should your IMAP client do a 'LIST "" "*"' operation, Dovecot would only traverse through $HOME/mail and everything under it, not all of $HOME.

There are three problems with migrating from our current configuration to this setup. First, there's all of people's current mail folders that are in places outside of $HOME/mail, which must be moved into $HOME/mail in order to stay accessible via IMAP. Second, even for people who have their mail folders in $HOME/mail, their clients know it under a different IMAP path; right now an existing $HOME/mail/PrivateStuff would be known to the IMAP client as mail/PrivateStuff (even if the client hides this from you by having an IMAP path prefix), where in the new world it would be known as just PrivateStuff. Finally, some people have their clients set up with an IMAP path prefix, which would have to be removed to get our ideal setup (even if their IMAP path prefix is currently mail because they're storing everything under $HOME/mail right now).

There are some ways to improve these. First, if people are willing to accept extra directories under $HOME/mail, they can avoid needing any client changes even if they were already putting all of their mail folders in a subdirectory; all you do is preserve the subdirectory structure when you move mail folders around. If you currently have $HOME/Mail/PrivateStuff, move it to $HOME/mail/Mail/PrivateStuff instead of $HOME/mail/PrivateStuff. You wind up with the theoretically surplus $HOME/mail/Mail directory (and a 'Mail/' IMAP path component), but all your clients can continue on as they are.

Second, if a person uses IMAP subscriptions so that the server-stored subscription information knows all of their mail folders, we can reliably move all of them into the same hierarchy under $HOME/mail using only server-side information. Server side mail processing in things like .procmailrc may need to be updated, however, since the actual Unix paths will obviously change. Also, use of IMAP subscriptions (and the IMAP LSUB command) is far from universal among IMAP clients (as I've discovered). As far as I know, Dovecot doesn't provide a way to log information about what mail folders are SELECT'd, so we can't determine what actual mail folders exist through tracking client activity; it's server side IMAP subscriptions or nothing.

So far I've described the configuration that we want, not necessarily the configuration we're willing to settle for. So what are our actual minimum goals? While we haven't actively discussed this, I think what we'd settle for is an end state configuration where IMAP clients can't search through all of $HOME or store mail folders anywhere outside a small set of subdirectories under it. We could live with a configuration where mail folders could be in any of $HOME/mail, $HOME/IMAP, $HOME/Mail, and a few others. We can also live with these being visible in the IMAP mail folder names that clients use, so that instead of seeing a folder called PrivateStuff in your client, you either see mail/PrivateStuff (or Mail/PrivateStuff or so on), or you set an IMAP path prefix in your client to hide it.

Sam Hathaway's comment on my entry on IMAP paths in clients and servers brought Dovecot's namespaces to my attention, especially the backwards compatibility examples. I don't think these can be used to migrate towards our ideal configuration, but it's possible they could be used to create something like what we're willing to settle for.

(They could also be used to strip out prefixes from the IMAP paths that clients send us, but in our specific situation I don't think there's much point in doing that. The hard part is getting people's mail folders under $HOME/mail, and we don't really care if their path there winds up being eg $HOME/mail/Mail/<something>.)

Sidebar: A brief note on the mechanics of migration

For various reasons, we have no intention of operating a single Dovecot server with different configurations for different users (ie, with some migrated to the new, confined configuration and others using the old one). Instead we'd do the migration by building an entire new IMAP server under a new name with the new configuration, and then telling people what they had to do to switch over to using it. New people would be pointed to the new server (and blocked from using the old one), while existing people would be encouraged and perhaps helped to migrate. Eventually we'd be down to a few stubborn holdouts and then we'd give them no choice by turning the old server off.

Conveniently the current IMAP server is running Ubuntu 14.04, which means that it has a natural remaining lifetime of about a year and a quarter. This is perhaps enough time to actually get everyone migrated without too much pain.

(Then in a year or two more we'd quietly switch back to using the old IMAP server name, because it really is the best name for an IMAP server.)

IMAPMigrationGoalsProblems written at 00:20:56; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.