Wandering Thoughts archives

My rules of thumb for picking conference talks

The yearly on-campus technical conference has just wrapped up for another year, so it seems a good time to write down my rules of thumb for picking what conference talks to go to, in the hopes that I'll remember them for next year. (This year, I only remembered them about halfway through.)

First, I should note that this is about picking interesting talks, not relevant ones. With rare exceptions, I don't go to talks at our local technical conference to be educated, partly because an hour-long talk at a general campus IT conference is a fairly bad setting for anything more than a superficial exposure to an issue.

• discard any talk with a description written in management speak. (Around here, people write their own talk descriptions.)
• similarly, discard any talk where a vendor is presenting directly. Perhaps they have a fascinating and interesting talk, but the odds are against you.

• try to remember speakers with good track records; there are some people who consistently give interesting presentations, or consistently do interesting work, or both.

• pick the talk where people are presenting about something they have built or done. These people have done something concrete and will hopefully have come out of it with interesting war stories.

  If a time slot has more than one such talk, pick the talk that's about something that's been used in the field (as opposed to things that have been built but not yet used). If there's still more than one talk, pick the subject that's closest to teaching students.

(Thinking about this makes me very glad I did not try to do a presentation this year, because I would have presented about something that's merely theoretical at this point.)

One of the interesting things that I discovered at this year's conference is that I seem to have really lost my tolerance for the typical presentation style where the presenter basically reads their slide deck to the audience. I suspect that I've just been too exposed to the Lessing Method of presentations (which I am pretty sure I discovered sometime in the past year, as I don't remember feeling like that after last year's presentations).

And for people who remember last year, I regret to say that no vendor appears to have been reading my suggestions for trade show giveaways. In fact there was less variety than last year: pens, a combination pen and peculiar calculator, a blue LED thing, and the most useful giveaway, a mini 4-port USB 2.0 hub.

(The paucity of giveaways may partly be because the tradeshow seems to have shrunk since last year.)

Sizing your swap space (part 2)

On modern machines, the temptation is to make heroically big swap sizes; you'll hardly notice giving your swap 5G or 10G, and it means that you can just forget about the whole worry that you've made it too small. But this can be counterproductive, because too much swap can be dangerous.

The problem with a too large swap is that it lets your system thrash itself more and more into overload, instead of outright exploding and then recovering. When a system is overloaded enough, it is simply not going to recover on its own in a reasonable amount of time, because everything is too busy paging things in to make much progress; the only way out is to do something drastic, like kill programs off and start over.

Once your system gets into this state, swap space is just rope to hang it with; the more swap space, the more rope and the longer the situation is going to continue. (Generally when things get this overloaded, the situation feeds on itself and things get worse and worse.)

So what you really want is enough swap space that your system can survive a temporary emergency long enough for you to kill the runaway process or the like, but not so much swap space that it can keep going and bog down in the mud.

(In the peculiar world of software, it is generally faster to blow the vehicle up and rebuild it from scratch than to patiently dig it out of the mud.)

Sizing your swap space (part 1)

The amount of memory that a program needs in order to actually get some work done is called its Resident Set Size (RSS). The total amount of virtual memory a program uses is called various things; ps uses VSZ (Virtual Size, presumably), for example.

The premise of paging things out to your swap space is that most programs don't need to use all of their memory all of the time; their RSS will be smaller than their VSZ. The difference can be pushed out to swap space in order to free up little-used pages to make room for another program's RSS.

(This is a simplification; not all of the inactive pages have to be pushed out to swap, since some of them are backed by files instead.)

So the simple answer is that your desired swap size is more or less the difference between the total virtual memory size of all of your programs and their total RSS, plus some margin for error. This gives you enough room to push all of the unproductive pages out of RAM to free it up for more productive uses.

(This does assume that your total RSS usage fits into your RAM, but if it doesn't you have bigger problems: you're trying to do too much for your machine and it's going to be thrashing to death.)

Unfortunately, modern operating systems don't really make it very easy to compute useful numbers for the amount of system-wide virtual memory that might need to be swapped out and the total system-wide RSS. While you can mostly get figures for individual processes, summing them all together will overcount shared pages, and these days (with shared libraries and the like) there are a fair amount of them.

The other problem with network booting

Following on to NetbootProblems, there's one final problem with using netboot to install machines: everyone does network installs differently. With CD/DVD based installs, you throw the disc in the drive, boot the machine, and follow the installer's directions. With network installs, well, it depends; there is very little commonality on how you get to the 'follow the installer's directions' stage.

This is the real attraction of virtual DVD-ROMs: like the old math joke, they reduce network installs to an already solved problem.

I see at least two reasons the netboot problem persists:

• the only real way to make things common across different environments is to base network installs on putting ISO images on the network instead of burning them to CDs, but making reading ISO images off the network look like reading CDs takes a fair amount of work somewhere in the installer.

  (Linux makes it look somewhat easy, but Linux gets to reuse a bunch of general support, and your remote access protocol needs to look like a filesystem to Linux.)

• we have no common, simple network protocol that is good at (read-only) random access to large files. NFS and SMB/CIFS are not simple, and HTTP is not good at random access (I'm not sure FTP is simple, and it's no better at random access than HTTP).

Thus, unfortunately I suspect that I will be burning CDs and DVDs for quite some time to come. The only bright spot on the horizon is the possibility that service processors that can do virtual DVD drives will become common in the servers we buy.

A stupid switch configuration trick

Today we discovered that we'd accidentally done a stupid switch trick to cheat ourselves out of some link bandwidth: we'd accidentally configured some untagged ports on one of our top-level switches to also carry some tagged VLANs.

(Normally you configure switch ports in one of two ways: either they carry one untagged VLAN and nothing else, or they carry some tagged VLANs and no untagged traffic.)

This is effectively invisible, because the regular traffic is still flowing, and nothing complains about the extra stuff. These days switches generally drop tagged VLANs that they're not expecting, and hosts just ignore the strange traffic just like they ignore all the other weirdness that is probably floating around your network.

(Every so often I run tcpdump on my networks, just to see what sort of broadcast traffic and misdirected stuff turns up. It's usually good for some nervous amusement.)

This can be an annoyingly easy mistake to make, because many switches don't have a per-port summary of VLAN membership that lets you see at a glance all the VLANs that a port is part of; you have to infer a port's VLAN membership list by going through each VLAN and looking to see what ports are members. This is especially irritating when you have a lot more VLANs than ports, as we generally do.

(Most of our VLAN-aware switches have been configured with all of the VLANs we have, even if they don't currently carry traffic for that VLAN.)

We think that the mistake crept in when we converted the ports from carrying various bundles of tagged VLANs (and no untagged traffic) to being untagged touchdown points for particular VLANs. Since you can't set or clear a port's entire VLAN configuration in one go, you have to go through each VLAN and remove its tagged membership; if you miss some, you get what we wound up with.

To be honest, I suspect that we weren't losing too much link bandwidth to this, although it depends on just how noisy some of our networks are. And we'd only have lost outbound bandwidth, since the ports are only being sent extra traffic, they're never sending any back.

(This was not a security exposure because we control everything plugged into the switch and switch ports in question.)

Some good practices for web servers

Here's something important for file ownership on web servers:

The user that the web server runs as should not own anything that the web server serves.

This means that the web server's user shouldn't own anything under the document root or in CGI areas, and there shouldn't be any writeable directories under either that it can use to put things in. Strongly resist the temptation to have CGIs write data files under the CGI area or in the docroot, even if they are normally blocked from being served; if CGI programs need to write data files somewhere, that somewhere should be a completely separate directory hierarchy.

(A close cousin of this is to strongly resist having anything but actual executable CGI programs in your CGI area. Their configuration and data files should live somewhere else, somewhere where a clever attacker can never try to run them as a CGI and see if something interesting happens.)

Unfortunately, this configuration is pretty hard to arrange if you use Apache's suexec, because suexec rightfully insists that the UID it's going to run things as own the program it's about to run. About the best you can do is have two machines and (NFS) export the CGI area to the web server read-only.

An important corollary: the user that the web server runs as should not have a real home directory, and ideally not a real shell either.

A second corollary: if you keep your website (or portions of your website) in a version control system like SVN, makes sure that the user the web server runs as does not have write permissions for the repository. In fact there's no reason it needs even checkout permissions, since the checkouts should be done as another user.

(Hopefully I do not have to mention that the user that the web server runs as should never own the web server's configuration files or be able to write to the directory where they are. Especially if your web server binds to port 80, and thus is started as root.)