Wandering Thoughts archives

Being specific about where your systems are

For a long time, whenever we installed systems we didn't bother to be terribly specific about where they were located, which generally comes up when you get asked about time zones. For example, a lot of things default to 'America/New York', and since that's in the Eastern time zone a lot of the time we said 'sure, whatever' and went on.

The recent American DST rule change illustrates that this is now hazardous; it is now important to actually be specific about where your machines are located, because someday there may be a real difference between 'America/New York' and 'America/Toronto'.

(Yes, Canada adopted the American DST rules. This time. I'm not sure I can count on that always being the case and besides, who knows, some day New York might decide to have their own different set of DST rules.)

All of this is irritating but probably inevitable. And I shouldn't complain too loudly, since there are places with much worse tangles of DST rules.

But while I'm mumbling about timezone issues, I must grumble at all of the people who don't offer Toronto as a timezone location choice. I know, we're not all that big and important, but at the same time many of those people seem to offer Thunder Bay instead; this strikes me as at least an eccentric choice.

A clever way of doing IP address assignment

One of the little hassles with hardware that can be managed over the network is the bootstrapping problem: it has to somehow get an IP address so that you can manage it and give it the right IP address. Over the years, vendors have come up with any number of solutions to this, such as the ever-popular 'give it an initial static IP address and it's up to you to arrange a host that can talk to it'.

Recently we ran into a piece of hardware that has the most creative and clever solution to this that I've yet seen. If it's unconfigured, it listens for a 113-byte ICMP ping packet to its hardware Ethernet address; when it gets one, it plucks the destination IP address out of the packet and adopts it.

Of course this would normally present a chicken and egg problem, since it's not answering ARP requests until it knows what its IP address is, and your system doesn't know what Ethernet address to send the packets to until it does. But this is no problem, because what you do is just manually add an ARP entry for it on the system you're pinging from. (The unit's Ethernet address is printed on it.)

(For my future reference, this is 113 bytes total size, not payload size; on Linux, you need to use 'ping -s 85 ...' to get the right packet size, since -s sets the payload size. The documentation did not make this entirely clear.)

This particular piece of hardware also has a serial interface, so you can also fall back on that. We may wind up primarily managing it through the serial connection (partly because we have a well developed serial console infrastructure), although the shiny features of the built-in web server are reasonably attractive.

A little gotcha about binding services to interfaces

I ran into this over this about a year ago, but I got reminded of it today:

Regardless of what interface a service is bound to, its outgoing packets are going out through the interface that the default route sits on.

(Or the route to the specific destination.)

The only packets that are guaranteed to go out the same interface they came in on are those from sources on that interface's local networks.

This would have bit me hard a year ago if we'd had more load on the mirrors I set up. After the dust had settled, I noticed that while the mirrors listened on different subnets, the two machines involved actually had default routes that ran over the same 100 mbit subnet so their outgoing packets (which would have eaten all the bandwidth) would have been choked down to a combined 100 mbit/s.

The same thing is true of merely connecting to services by a specific IP. Your packets to the service will go through the right network, but the return packets may come out through a different one if the target machine is multi-homed. (This is true even if the target machine does not have IP forwarding enabled; IP forwarding only affects what the machine does with other people's packets.)

Getting around this generally requires policy based routing to route packets based on the origin IP as well as the destination. This is generally more complicated to set up than it needs to be, since what you really want is an option to make source addresses glued to interfaces; a glued source address would require that any packet coming from it must be routed out its interface.

(If there is no such route, you drop the packet with 'network unreachable'.)

Why network booting is not a good answer

It might seem that the obvious answer to my desire for remote controllable DVD drives is to netboot my machines. Unfortunately it's not a good answer.

Booting a machine is only a tiny sliver of the problem; the real problem is (re)installing and testing machines. While a lot of operating systems have some sort of a network install system, there are three core problems with netinstall setups: they are almost always second class citizens (partly because they are usually used much less), they take extra work to set up, and they make the netinstall server a crucial core part of your disaster recovery environment and an exception, since it is not installed the same way you install almost everything else.

It need not be a big disaster to cause problems; all you need is to lose both the netinstall server and another server. At that point, either you have to put the other server back together not following your usual build procedures, or you have to get the netinstall server rebuilt first. (You can keep a backup netinstall server, but then you have to make sure that the primary and the backup don't drift out of sync.)

If you have lots of machines in a uniform enough environment, sure, netinstall setups are completely worth it. But otherwise, network installs are nowhere near the level where it's easier to use them for some random new machine and operating system release than to swap a few CDs or DVD.

Wanted: remote controllable DVD drives

With remoteable KVMs and power outlets that can be remote controlled, there is only one important missing bit until I can give up visiting machine rooms almost entirely: a remote-controllable DVD drive.

The easiest way to do this would be with a USB device that pretended to be a DVD ROM drive, but was actually a big pool of disk space with ISO images. You'd give it an Ethernet connection so you could control it remotely to 'switch' the 'discs' and so on.

(Since Linux has target mode USB drivers, I believe you could actually build this today with the right hardware and some work. You'd want a machine with a lot of USB connections so you can hook it up to as many servers at once as possible.)

Our local experiences is that modern hardware and modern operating systems are generally happy to boot and install from USB DVD drives, but your mileage may vary.

(If you are spec'ing new servers, please give them DVD drives. Unless you are buying in industrial quantities, the amount of money you'll save by doing otherwise is not worth the aggravation, and probably literally not worth the cost of the wasted staff time.)

An irritation about rails

Coming from an environment where we just put everything on shelves in racks, I have to say that proper rack mounting is kind of neat, but rack rails irritate me. It's all because of one simple little problem:

Why, oh why, can't people agree on a single sort of rails?

(Or at least a single sort of rails for things of each rack unit, so you'd have 1U rails and 2U rails and 3U rails and so on.)

We have server hardware from a number of vendors, and several generations of hardware from the vendor we usually wind up buying from (their name starts with a D). This has created an amazing assortment of rails and rail mountings, where it sometimes feels like no two servers are the same. And every so often we get to play the fun game of 'hunt the rails kit for this server (if we have one)'.

Of course, the rails and the servers are generally not labeled so that you can tell which goes with which; you are apparently supposed to remember what each different sort of rails and rail mountings looks like. This becomes important when you have loose rails and servers floating around, or merely when you have to relocate an assortment of servers (hopefully complete with their rails) from rack to rack and not lose track of what fits what.

(If you have rails kits sitting around, I strongly recommend getting a marker and putting a big label on their boxes about what sort of servers they fit. Now. Don't delay; you can thank me later.)

On a side note, putting numbered marks (in our case, marking off rack units) on rack posts has to be one of those great ideas (no more exciting 'is this side really lined up with that side?' games, especially when several people are working at once). However, I must say grumpy things about the kind of quality control in rack posts that results in screw holes that are just different enough diameters that the same screws go in easily in one rack post and must be forced hard in another. In the same rack.

Sidebar: a terminology note

'Rack posts' are the vertical strips that equipment is bolted to. The Wikipedia racks entry currently says that they are sometimes also called 'rails'; ignore that, because it will only confuse you. Rails are purely the too many different sorts of horizontal things that bolt into racks and support servers, possibly letting them slide easily in and out.

A brief sad note about root passwords

As far as I can see, the inevitable result of having several different root passwords across your systems is that a smart intruder who compromises /bin/su on a single system will sooner or later collect all your root passwords.

(Or perhaps other people are better than we are at typing the right root password at the current Password: prompt.)

(Please do not suggest sudo as the solution. For a start, we'd still want actual root passwords that are distinct from our own passwords.)

Some lighttpd bits

Lighttpd is a popular lightweight web server I've been experimenting with; in the process, I've wound up stumbling over various things that I should write down this time around (I played around with lighttpd about a year ago but then forgot a bunch of this stuff).

• unlike Apache, lighttpd's error log is global; there's only one, not one per virtual host. If you accidentally set server.errorlog multiple times (perhaps in virtual host setups), only the last one gets any messages.

  (Not being able to set per vhost error logs is relatively harmless, because much less winds up in lighttpd's than in Apache's.)

• the way you set the character set for HTML pages and so on is to change the MIME type assignments to append a '; charset=<whatever>' to the end of various MIME type entries. Unfortunately, you must do this by changing the main mimetype.assign table (or repeating it wholesale); you cannot do it with something like:

  mimetype.assign += (".html" => "text/html; charset=UTF-8")

  (What happens with += is that the .html entries add to each other instead of your version replacing the old one, so you generate Content-Type headers that look like 'text/html, text/html; charset=UTF-8'. This generally ends in tears.)

• most matching of URLs is purely textual; for example, if you just match:

  alias.url += ( "/cgi-bin" => ...)

  you will block access to '/cgi-bin2-foobar' and the like. In this case you want to use '/cgi-bin/' (with the slash on the end), and in general the magic regexp to use is '^/foo($|/)', which matches /foo only when it is a complete path component.

I have to say that lighttpd's configuration stuff can be a bit scary sometimes; once you start using its conditional things, it's practically a programming language. I'm more used to configuration files that don't have to be partially interpreted on every request, and thus where you can clearly see what settings are in effect.

(Lighttpd allows you to set core server configuration variables based on matching against various bits of the HTTP header. For example, the lighttpd way of doing simple virtual hosts is to check the HTTP Host: header and set completely different document roots, log files, CGI configurations, and so on.)

It is kind of a pity that you can't take lighttpd configuration to the logical next step and write macros. I suppose you can get pretty close with include files, but you can't easily pass them arguments.

Sidebar: sending a URL hierarchy to a CGI

There's at least two ways of sending an URL hierarchy to a CGI program in lighttpd; the cgi-bin way and the direct way. The cgi-bin way is to configure yourself a traditional cgi-bin directory area, and then use URL rewriting to point stuff there (ie, the traditional Apache approach transplanted). With your /cgi-bin/ stuff already configured, it looks like this:

url.rewrite-once = ("^/foo($|/.*$)" => "/cgi-bin/foo$1")

The direct way is to send the root URL straight off to an executable without bothering to configure a generic CGI area:

$HTTP["url"] =~ "^/foo($|/)" {
  alias.url = ("/foo" => "/some/where/cgi/foo")
  cgi.assign = ("" => "")
}

It's safe to stomp all over alias.url and cgi.assign here, since we're only running one thing in the section and thus can't need any other settings they may have. I was pleased to see that setting alias.url to point to a file instead of a directory works fine.

(This sort of thing is where the configuration stuff starts turning into a programming language.)

I personally prefer the direct way, because it leaves less stuff accessible for excessively clever people. Why have an accessible set of /cgi-bin/ URLs if you're not actually using them for anything?

An advantage of virtualization for license servers

One of the more subtle advantages of full hardware virtualization is that your virtualized hardware always stays the same, no matter what actual hardware you happen to be running on. This is a real advantage for license servers, which are often nodelocked with various crazy schemes that assume you are trying to cheat on them if you change hardware.

(Sure, the company might fix the problem for us. If they're still in business, still interested in our business, still having anything to do with that ancient version of the software, and so on. All too often, companies seem to treat license problems as a great opportunity to extract more money from you.)

Virtualized hardware can even survive nodelocking schemes that require the inode numbers of crucial files to not change, because the disk is virtualized too. All you have to do is back up and restore the disk image (which is generally just some files), and everything comes back just as it was, inode numbers and all.

In fact, simplifying backup and restore in general is a nice feature of virtualized hardware. We no longer have to wrestle with various system specific programs that may or may not completely work; instead we can just snapshot, back up, and restore some files, something for which we have lots of tools. And we're guaranteed to back up and restore everything, with complete fidelity. And the whole process is much simpler, especially restoring a system (which can often be a nightmare).

The other great advantage of static hardware is static operating system stuff that depends on hardware, which in turn means that we never again have to do things like dig up a nearly exact replacement for a piece of dead hardware in order to get a system running again in a hurry without monkeying madly with its configuration. This is especially an issue for old systems, where you may not be able to reinstall the operating system on your recent hardware even if you want to because it lacks the necessary drivers.