Wandering Thoughts: Recent Entries

A general point about SSH personal keys

Recently I've seen a number of articles on suggested good ways to use SSH securely and other SSH tricks (unfortunately I can't find URLs to all of them, so I'm not going to try to put any here). As it happens I have a few modest suggestions on this, but before I started I wanted to make a broad meta-point about the use of personal SSH keys, aka SSH identities.

The big thing to understand about all advice about SSH personal keys is that when you choose to use personal keys for your own logins, you are deciding to balance convenience with security. After all, if security was your primary concern you would not use personal keys at all; you would use one time passwords with two-factor authentication.

(Things are different for cron'd scripts and the like, when there is no human there to interact with the system. I'm purely talking about using SSH identities to avoid typing passwords.)

Now, everyone has different views of the amount of security that they need and the convenience that they want. People fall along a spectrum between the two poles and where you wind up is not necessarily where I do. Thus, people's security advice about personal keys is not necessarily right for you even if it's correct (in some sense). The trick is to understand your particular tradeoffs and circumstances, to figure out what irritates you and what you need, and then to pick what works for you rather than blindly following someone else's suggestions and being either frustrated or dangerously insecure (in your environment) or both.

Yes, some things will make you less secure than others but they can also be more convenient (and vice versa). Sometimes this is the right tradeoff for you and sometimes it is not (even if it's the right tradeoff for me or whoever you're reading). And yes, there are some SSH tricks that usually increase both security and convenience. These are excellent things to know when you can find them.

(Sadly, my suggestions to come are not of this nature.)

PS: as always when you consider security related issues, you want to think about not just security in the abstract but security in the concrete in your environment with your risks.

My view on what will kill 'traditional' system administration

Phil Hollenback recently wrote DevOps Is Here Whether You Like It Or Not, in which he writes that traditional system administration is dying. While I sort of agree with him about the death, I don't think it's necessarily for the reasons that Phil points to.

Fundamentally, there has always been a divide between small systems and large systems. Large systems have had to automate and when that automation involved applications, it involved the developers; small systems did not have to automate, and often do not automate because the costs of automation are larger than the costs of doing everything by hand. Moving to virtualization doesn't change that (at least for my sort of system administration, which has always had very little to do with shoving actual physical hardware around); if you have only a few virtualized servers and services, you can perfectly well keep running them by hand and it will probably be easier than learning Chef, Puppet, or CFEngine and then setting up an install.

(If you're future-proofing your career you want to learn Chef or Puppet anyways, so go ahead and use them even in a small environment.)

There are two things that I think will change that, and Phil points to one of them. Heroku is not just a virtualization provider; they are what I'll call a deployment provider, where if you write your application to their API you can simply push it to them without having to configure servers directly. We've seen deployment providers before (eg Google App Engine), but what distinguishes Heroku is how unconstrained and garden variety your API choices are. You don't need to write to special APIs to build a Heroku-ready application; in many cases, if you build an application in a sensible way it's automatically Heroku-ready. This is very enticing to developers because (among other things) it avoids lockin; if Heroku sucks for you, you can easily take your application elsewhere.

(This has historically not been true of other deployment providers, which makes writing things to, say, the Google AppEngine API a very big decision that you have to commit to very early on.)

Deployment providers like Heroku remove traditional system administration entirely. There's no systems or services to configure, and the developers are deeply involved in deployment because a non-developer can't really take a random application and deploy it for the developers. If there is an operations group, it's one that worries about higher level issues such as production environment performance and how to control the movement of code from development to production.

The other thing is general work to reduce the amount of knowledge you need to set up a Chef or Puppet-based environment with certain canned configurations. Right now my impression is that we're still at the stage where someone with experience has to write the initial recipe to configure all N of your servers correctly, and you might as well call that person a sysadmin (ie, they understand Apache config files, package installation on Ubuntu, etc). However it's quite possible that this is going to change over time to the point where we'll see programs shipped with Chef or Puppet recipes to install them in standard setups. At that point you won't need any special knowledge to go from, say, writing a Django-based application to installing it on the virtualization environment of your choice. This really will be the end of developers needing conventional sysadmins in order to get stuff done.

The general issue of the amount of hardware in a small business (and virtualizing the hardware) ties into a larger question of how much hardware the business of the future is going to need or want, but that's a different entry. I will just observe that the amount of servers that you need for a given amount of functionality has been steadily shrinking for years.

Sidebar: what virtualization does change now

I think that plain virtualization does mark a sea change today in one way: it moves sysadmins away from a model of upgrading OSes to a model of recreating their customizations on top of a new version of the OS. Possibly it moves away from upgrading software versions in general to 'build new install with new software versions from scratch, then configure'.

This is partly because the common virtualization model is 'provide base OS version X image, you customize from there' and partly because most virtualization makes it easy to build new server instances. It's much easier to start a new Ubuntu 12.04 image than it is to find a spare server to use as your 12.04 version of whatever.

(Note that virtualization may not make it any easier to replace your Ubuntu 10.04 server with a new 12.04 server; there are a host of low level practical issues that you can still run into unless you already have a sophisticated management environment built up.)

I don't think that this is a huge change for system administration, partly because this is pretty how much we've been doing things here for years. We basically never upgrade servers in place; we always build new servers from scratch. Among other things, it's much cleaner and more reproduceable that way.

Dealing with Fitts' Law on widescreen displays

One of the usual sayings derived from Fitts' Law is that four of the five easiest locations to reach with the mouse are the four corners of the screen, because they require very little precision (the edges trap the mouse and guide it into the corner). Over the years I've made some modifications to my desktop environment to make better use of this principle. The most important one is how I use the top left corner; I have my taskbar equivalent arranged so that when an iconified terminal window gets output, I can just zoom my mouse to that corner and click in order to reveal the terminal window.

Zooming to a corner is a fast operation in most setups; it works fine on a single monitor, even a single widescreen monitor, and on a normal dual-monitor setup such as my work desktop. But recently (for reasons beyond the scope of this blog) my work setup got updated to dual widescreen monitors, which revealed two problems with my application of Fitts' Law in this environment.

The first problem is that the sheer number of side to side pixels in a pair of 1920x1200 LCD panels seems to be a bit too many to easily zoom a mouse across. My mouse pointer generally winds up in the middle of the right hand display; getting it to the top left corner of the left display was no longer anything like a little flick of the wrist. The second problem is that the top left corner was sufficiently physically far off to the side that it was no longer an easy casual action to glance at it to see if there was anything with new output that I needed to deiconify; I was less glancing off a bit and more peering off into the distance.

(I had my old dual displays relatively flat against each other, but I think that I probably need to move the new displays into a much more pronounced V shape.)

My current solution to this issue exploits Fitts' Law once again. The often-overlooked fifth easy to reach location is 'where the mouse is right now', or failing that 'some large area very near where the mouse is'. So I've created a new mouse button binding for my window manager; if the mouse is over the root window, hitting the left button with Shift+Control now de-iconifies the (alphabetically) first terminal window. My mouse is frequently parked over the root window and when it's not there's generally an exposed patch of the root window close to it.

(Technically the binding toggles the window's iconified state, which means that I can flip the first window back and forth from iconified to not. This is a great way to fidget.)

To deal with the 'too far to look' issue and to make things in my terminal windows taskbar easier to reach in general, I've repositioned it so that it's at the top left corner of my second (right) display; this puts it more or less in the center of my overall workspace and makes it easier to both reach and look at. I don't think this move away from a screen corner is a loss for Fitts' Law because everything except the first window already had to be targeted carefully.

Of course, now I just have to train myself out of a many years habit of reflexively looking and going to the top left of the left display. This shouldn't take too long, right?

(What I'd really like to do is duplicate my taskbar equivalent in the top left of both displays. Unfortunately this isn't possible right now with my window manager.)

PS: I experimented briefly with increasing the mouse acceleration (which would make everything effectively closer) but didn't like the effects it had on my ability to target things with the mouse in general; I kept overshooting and missing stuff. Possibly I would have acclimatized with time and I just gave up too soon.

How I use FvwmIconMan

I've mentioned FvwmIconMan in the tour of my desktop and sort of mentioned part of how I use it, but I've never really explained the details.

As I've set it up, FvwmIconMan is essentially a compact taskbar for my various sorts of terminal windows. In a dense display, it shows the window name for each one (well, the first part of it at least), an indicator if the terminal has been iconified, and an indicator if that terminal has the keyboard focus. This is part of how I work around not having conventional titlebars on terminal windows; the window name information from the titlebar is dumped in small text in the 'taskbar', and through long experience I can pick out the label for the current window pretty easily.

(Possibly I should make the current window more distinctive than it is right now. A lot of my FvwmIconMan configuration, much like a lot of my fvwm configuration in general, dates from days with much slower machines that had much more limited graphics.)

Left-clicking on FvwmIconMan's label for a window toggles whether or not it's iconified. Like other taskbar implementations, an iconified (or 'minimized') window is only present as a label in FvwmIconMan; to deiconify it, I have to go click on the label. This means that I care a lot about finding the window labels for specific windows, and I do two things to help with this. First, the window labels are always sorted into alphabetical order; if and when a window is renamed, the order shuffles (this is very important for my use of xterm's ziconbeep feature). Second, I give my windows very consistent names based on either the host they're on or what I'm using them for (and sometimes both). This scheme usually works okay but breaks down a bit if I have a lot of iconified windows on the same host; usually I don't and this isn't an issue. Lots of non-iconified windows on a single host are generally not a problem because they're directly visible and I usually keep them straight by how they're arranged on the desktop.

(This alphabetical sorting does mean that the label for a particular window isn't in a consistent physical spot; it can jump around wildly depending on what other windows get named or renamed. This doesn't bother me, partly because a lot of my terminal windows come and go rapidly anyways. Non-alphabetical taskbars actually drive me up the wall because I never can find anything once I have more than a few things running, or at least I can only find them by scanning through the entire taskbar.)

Some taskbar implementations only show windows from the current virtual desktop or virtual screen or the like. While I use virtual screens I have FvwmIconMan configured to include all terminal windows, regardless of where they are. Among other things this lets me easily yank terminal windows between virtual screens; I move to another screen, then iconify the window and immediately deiconify it again (windows always deiconify on the current virtual screen) with two clicks on the window's label. I can also use FvwmIconMan to switch to the virtual screen that holds a particular deiconified terminal.

(Iconified terminals aren't on any particular virtual screen; they've been effectively swallowed by FvwmIconMan.)

Sidebar: terminal windows versus Firefox windows

A long time ago I would have confidently told you that I did this for terminal windows, and only for terminal windows, because they were by far my most numerous sort of window and I also often had a lot of them iconified. If I had the iconified windows represented as real icons on the root window, I would run out of space; therefor I condensed them all into a much more compact area. Then my Firefox window habit grew out of control and at this point I often have as many iconified Firefox windows as I have terminal windows.

So why do I have a taskbar for terminals and real icons for Firefox? The simple answer is that useful Firefox window names are too long, whereas I can make xterm window names short enough that I can pack them in very compactly. Because Firefox window names are long, a taskbar that showed enough of the titles to remind me what they were would be too big to be feasible. Instead it actually takes less space to have real icons and count on my spatial memory to remember what the Firefox icon over there is for.

(Well, the spatial memory plus the bit of the start of the window title that fvwm shows me below the actual Firefox icon.)

The death of system administration: I'm all for it

Recently there was a little Twitter commotion about Julian Dunn's Chef, devops, and the death of system administration (he later clarified his views). Although it may surprise people, my snap reaction to the idea of the death of system administration was 'good'.

(I have a number of other reactions to portions of this debate, but 'good' was my first one.)

Most of what many people think of today as 'system administration' is scutwork, at best boring and uncreative. Racking servers, configuring switches through interminable web or CLI interfaces, running network cables, installing OSes in any way that takes more than about one line of typing, writing an Apache or a mailer or Samba config file yet again, restoring files for people, and so on. That's what I'm talking about. At best these are interesting the first few times you do them; after that, very much not.

(System administration wasn't always this sort of work, but times have changed.)

Unless you really do like spending your time doing that or you feel that that sort of work is all that you have to contribute, you are better off without this near monkeywork. Regardless of what your job is called after 'system administration' goes away and the dust settles, you will have shifted to doing actual engaging and creative work and you'll be contributing much more to your organization's success. As I've written before in a different context, having spare time from ordinary day to day 'system administration' is what you need in order to create the big wins. The ultimate version of this spare time is not to have to do the ordinary day to day gruntwork at all.

As you may have gathered, I am not particularly fond of the scutwork currently involved in a great deal of 'system administration' (although I think there's uses for doing it every so often). As far as I'm concerned, the sooner this sort of system administration dies the better.

(At the same time, let's not fool ourselves. This death of system administration will put a significant number of people out of work, ie those people who are currently well paid to do nothing but this scutwork. Many of them do not currently have the skills to move up in the food chain; they will either move down to be less well paid operations monkeys or have to change fields entirely. This is going to be a wrenching process that will be very unpleasant for the people involved, and we should both have sympathy for them and understand the full implications of this shift we're casually discussing, advocating, and cheering for.)

(As a corollary, if you have junior people in your organization and you believe in this shift you should be working with them to make sure that they're developing the skills they'll need for the future, not just spending all of their time doing scutwork for you. And you should be honest with them about how you see their future.)

Every so often, I solve a problem with a hammer

For reasons beyond the scope of this entry, I maintain a special Firefox profile and instance for uploading pictures to my Flickr account. Back in the old days, Firefox had a very convenient behavior for this: when it asked you to choose files to upload in an upload form, the default directory was the current directory that you'd started Firefox in. This meant that I could cd to the day's photo directory, start my Flickr Firefox instance, and have the GTK file chooser dialog start in exactly the right directory. Then at some point Firefox changed this so that the default file chooser directory was something like your configured download directory.

I poked at this off and on but couldn't find a way to make Firefox get its old behavior back. So recently I decided to fix the problem with brute force. The script that I use to start my Flickr Firefox instance now looks somewhat like this:

#!/bin/sh
ln -nsf $(pwd) $HOME/CURDIR
exec firefox -P flickr "$@"

This is inelegant and not a real solution, but it makes things a lot more convenient; it's now much faster to navigate to exactly where I want to be. Sometimes that's the right way to deal with a problem, when either the real solution is too much work or the problem is too small to justify anything more than a quick hack.

(I suppose that this could be slightly improved by putting the symlink directly in the download subdirectory. I'm not sure why I didn't do that.)

What my physical desk is like

A while back I read The shrinking desk, where the author winds up asking other sysadmins how messy their desks are. As it happens, I have a slightly unusual reaction to that: depending on how you define it, either not at all messy or very messy.

You see, I don't use a desk; I never have. Right from the beginning of my sysadmin career, I've preferred to put my monitor, keyboard, and mouse on a plain table. As far as I'm concerned, the main difference between a desk and a table is that you have less free space for your legs underneath a desk. This is the setup I currently have at work, and I keep the computer table basically completely clear (apart from the stuff that has to be there; monitor, keyboard, mouse, speakers, and cables). I don't even put my computer on the table; it lives on the floor besides the table.

This leaves me needing space to put everything else (starting with the phone). All of that goes on, well, an actual desk (although I'd generally be happy with a second table, maybe with a storage box or two underneath it). In my current setup at the office the desk actually has a side extension and so I have the whole collection set up in the shape of a sideways U; the computer table is in front of me, the desk proper is behind me, and the desk's side extension is on my left. The side extension has things like the phone, my coffee mug (when it doesn't have coffee in it), and a notepad. The actual desk has a random drift of paper and sometimes various hardware bits. I try to keep the amount of accumulated paper down by throwing it out periodically, but I don't always succeed.

My current office is luxurious enough to also have a second table, positioned off to my right, which accumulates most of the random hardware bits. I think everyone around here has at least two tables; some people have three and use them. I think that this arrangement is pretty much ideal if you need that much space for things, although it's sort of better not to need that much space.

(On the accumulating paper, a short shameful confession: when I moved offices in 2006, my old desk was entirely covered in paper to a minimum depth of six inches (and deeper in some spots). After being horrified by the archaeological discoveries I unearthed, I have tried to be more disciplined with the new desk. My current rule of thumb is that if I haven't read whatever it is in six months, it gets thrown out regardless of how much I think that I should read it someday.)

Labs versus offices for sysadmins (or at least us)

On the one hand, lab areas are great because they mean that noisy machines aren't in your office. On the other hand, lab areas are bad because they aren't your office, including that they're noisy, often uncomfortable, don't have your system setup, your phone, and so on. Really what you want is machines in the lab that you can fully control from your office with at most occasional in-person visits; sadly we rarely get that.

This means that there's a constant tension between putting test machines in the lab area and putting them in your office. At least around here, what tends to happen is that relatively quiet hardware winds up in people's offices for testing rather than being dropped in one of our lab areas; the annoyance of having the hardware in your office is less than the annoyance of having them not in your office. In turn, this drives our desire for lots of drops (per earlier entries), and when we don't have lots of drops people wind up running network cables between offices because it's still more convenient than trying to rig something up in a lab area.

(Individual tolerances for noise vary; my co-workers are far more tolerant than I am and so they have a lot more stuff in their offices than I do in mine. Also possibly this means our lab areas aren't set up well, which is possible; they have no more network drops than our offices do, since the entire building was wired up a long time ago.)

Now, I've kind of given an incomplete view of what we do with hardware. We don't really have a good lab area that's isolated enough for actively loud hardware, like your garden variety really noisy 1U servers, so they wind up getting shoved into a machine room if they're going to hang around for long. What we mostly wind up using either in offices or in the lab area is things like switches or various desktop machines we use to build test servers and test networks; for example, if we want to build a duplicate of our Samba and CUPS environment we don't do it on actually 1U servers, we just grab a couple of spare desktops and start installing. They're not as powerful as the real thing and they're not quite identical to it, but we can put the same software on them and they're a lot more convenient (quieter, less demanding of power, easier to find space for, etc).

(Some people use virtualization for this. Locally, I'm the only really active user of this approach; my co-sysadmins mostly prefer using real hardware.)

More wiring for sysadmins: sysadmins and gigabit networking

In reaction to comments on his entry The Other Way, Matt Palmer wrote in part about my concerns about office switch uplink bandwidth for sysadmin drops (in WiringForSysadminsII):

[...] What I question is the need for constant, sustained gigabit over an extended period to another isolated machine such that you need a dedicated link to them.

I sort of half-agree that sysadmin machines and drops don't need constant, sustained gigabit bandwidth (although I'm not entirely sure about that). But what they do need is occasional periods of real gigabit bandwidth, and real gigabit bandwidth when you can be absolutely sure that the underlying link will deliver gigabit data rates and the only performance limits are those created by the machines, switches, and so on at either end.

If I'm testing how fast hardware and software can go or if I'm trying to investigate network performance problems that have been reported to us, I need an environment that is not artificially contaminated by other networking traffic coming in to my heavily VLAN'd office switch. I know that some amount of contamination is there (I have tcpdumps of our internal networks and some of them are remarkably noisy); it may be enough to be significant, or it may not be. I don't want to have to guess about it and make assumptions. I want a clean gigabit, one that's as close as possible to what machines and users would see in the real environment in our machine rooms or in user offices.

Why office switches plus VLANs aren't the answer for sysadmins

In more or less reply to my last entry on wiring offices for sysadmins, I got a tweet:

@thatcks 2 drops is enough. Desk switch per sysadmin + vlans and bob's your uncle

(I can't find the tweet in the person's public stream right now so I'm not going to attribute it. Maybe I should paraphrase it instead of quoting directly, but any paraphrase would probably be longer.)

I thought of this, but there are three reasons why this doesn't work: uplink bandwidth, maintenance overhead, and a network design that doesn't have everything as part of the same VLAN fabric.

The maintenance overhead is pretty straightforward. Any time you want to set up a new network or tear it down, you need to modify a bunch of switches to add or delete VLANs; each sysadmin's switch and then your master uplink switch for all of the sysadmin offices. Even if sysadmin office uplink is the only thing this switch does, it is a 'production' switch in that other sysadmins are not going to be happy if something goes wrong on it and things suddenly drop off the network. This is a pain at the best of times and can rapidly reach the point where running cables around the office is easier.

The uplink bandwidth issue is twofold. First, your total bandwidth across all VLANs is limited by the switch uplink bandwidth. In all realistic configurations this means you have a 1 GB total limit, and yes this can easy get in the way of certain sorts of testing. Second, the more VLANs you push over a single uplink, the more bandwidth you lose to background chatter on those VLANs and any ordinary traffic your regular machines may be doing (on regular production VLANs). Among other things, this complicates efforts to measure the true network performance of machines; are you getting less than a gigabit because they just can't deal with a gigabit, or is it because of other traffic on your office switch?

You can deal with some of the uplink issue by not propagating currently unnecessary VLANs to your office switch, but then you wind up needing to reconfigure at least your master switch every time your VLAN needs change. We are currently in this situation and take it from me, it is a pain in the rear that discourages testing.

The network design issue is that some of your networks may not be designed to run over your normal VLAN fabric and switches. There are three examples of this here (for background, details of our network setup):

• we have several port isolated internal networks that are kept out of our regular VLAN fabric except at touchdown points for internal firewalls.
• we have a completely isolated management network that runs over its own dedicated switch fabric. Bringing it into contact with our regular VLAN fabric in order to get it to our office is at least a violation of its design and has potentially bad effects if, for example, some of the switches involved also have management ports that are directly connected to the management network.
• we have deliberately isolated, non-VLAN'd, non-connected iSCSI networks. We may at some point want a port on an iSCSI network in our office, but we definitely do not want to pull the iSCSI networks into our VLAN fabric; we want a direct port to the iSCSI switch.

Trying to merge together otherwise isolated networks and VLANs on a subset of switches makes me nervous. There is a real possibility for accidental leaks and contamination (as well as weird side effects), and it's especially acute when you're reconfiguring the master office switch often. Of course this also holds for putting new VLANs for test networks on the master office switch, since these new VLANs are not part of your regular VLAN fabric and should never be propagated to it.

Sidebar: what I want in an office setup

The short form version is what I want is one port to carry the primary networks for my main machine, one port for a switch with all of our regular VLANs on it so I can connect to them for testing, one port with our port isolated network for user machines, one port for our isolated management network, and several other ports that I can connect to anything as the need arises. This is at least five ports.

(Thus I think the basic need is two ports for static stuff (one for your primary machine, one for the 'has everything' VLAN switch), plus some number of ports for floating things.)

Right now I have, effectively, three ports: one port with a selection of our regular VLANs that feeds through to my main machine and one port with an Ethernet splitter that gives me our port isolated network for internal machines plus our management network. The latter two networks only run at 100 Mbits each but that is not currently a problem. This is not enough ports, and I don't do as much networking work as my co-workers.