Wandering Thoughts archives

Keeping secrets as a system administrator

There are two sorts of secrets in the world: known secrets and unknown secrets.

A known secret is something that people already know exists; for example, your root password is a known secret. Keeping a known secret is simple: you just don't tell people your root password. You have to guard known secrets against inadvertent disclosure (it's always embarrassing to type your root password into the wrong window), but that's mostly it.

(I am ignoring the possibility of deliberate attacks against the secret.)

Unknown secrets are secrets that people do not know even exist. When you are keeping these, it is not sufficient to avoid disclosing the content of the secret; you must conceal the very existence of the secret. In short, to keep an unknown secret you must behave as if the secret doesn't exist. For example, if the unknown secret is a message in a private message system it is not sufficient to not disclose the contents; you had better not mention that the message even exists.

(This is especially the case because unknown secrets are usually the most sensitive sort of secrets that system administrators have to deal with or accidentally come to know.)

If you are trying to figure out if something is a known or an unknown secret, ask yourself if an outside person would know about the existence of the secret. If the answer is even 'probably not', then it is safer to assume that it is an unknown secret and behave appropriately.

(Getting good at keeping unknown secrets has uses outside of system administration, too; quite a lot of spoilers for books and movies and TV shows qualify.)

Tradeoffs in where you store volume management metadata

One of the divisions between different logical volume systems is where and how they store the metadata about what logical volumes exist and what they're made up of. Roughly speaking there are two choices: 'in-band', where you store the metadata in a reserved area of some of the disk space you're managing, and 'out-of-band', where you store it outside. In-band storage is used by Linux software RAID and LVM; out of band storage is used by Solaris DiskSuite.

(Technically you can get Linux software RAID to use a form of out of band storage, where the 'metadata' is saving a script to assemble the array. My impression is that this is not widely used.)

The advantage of in-band metadata is that your data is self-describing. If you have access to the underlying storage, you always know enough to reassemble the object; you are never stuck with valid data that you can't touch because you don't know enough to put it together.

The advantage of out of band metadata is that you can import existing data into your volume system, because adding storage space to the volume system doesn't require writing metadata to some spot of it. (Obviously this only works for RAID 0 or RAID 1 storage.)

One of DiskSuite's cooler tricks is the ability to import an existing filesystem on an existing partition into DiskSuite without having to dump and restore anything. You unmount the filesystem, tell DiskSuite to make the partition into a metadevice, and then you can mount the metadevice and your filesystem is back, fully intact. (In the extreme this can be done without unmounting, although you have to reboot to activate things; this is handy for things like the root filesystem.)

Despite the cool tricks possible with out of band metadata, I prefer in-band metadata; it's simpler and you don't have to scramble to find space for the metadata and to back it up.

The easy way to keep a week's worth of something

As an example, say that you want to keep a week's worth of backups from your PostgreSQL database. The pg_dump command will get the dumps, but how do you handle keeping a week's worth (or only keeping a week's worth, depending on your perspective)?

Simple: use GNU date's ability to give you the day of the week to cycle through a set of seven backup files, one for each day. It would look something like this:

DAY=$(date +%a)
pg_dump >db-back.$DAY

(I admit that I tend to use %u, the day of the week as a number, instead of %a, the day of the week as a name.)

This has the great virtue that you don't need to write (or reuse) any code at all to age things; you get aging for free, at the cost of the most recent version not always having the same name.

This works equally well if you want to keep seven days worth of a collection of things; just put them in subdirectories named after the day. You can adopt it for other intervals that date gives you convenient indices for, such as day of the month (%d) to keep roughly a month's worth of something, or week of the year (%U et al) to keep a year's worth of a weekly backup.

(Attempts to take day of the year mod something are coercing this trick further than it should be pushed. And if you do hours, don't forget the effects of going into and out of daylight savings time (if you do).)

The difference between operations and system administration

Here is a thought that just crystallized for me:

In operations, crud rains down out of the sky and the sysadmins have to make it go and keep it going.

In system administration, at least you get to design and build the crud yourself.

(Corollaries about things you inherit from a previous sysadmin when you move into an existing environment are left as an exercise for the reader.)

How we make Exim discard bounces of spam

Making Exim discard things in general is simple: you create an Exim router using the redirect driver that blackholes messages, and guard it with a suitable set of conditions. I decided to be safe, so we only discard locally generated bounces of spam that are going to an outside domain.

(You can detect locally generated bounces with the condition 'and {{!eq{$interface_port}{25}} {!def:return_path}}'. The easy way to discard messages is to use 'data = :blackhole:' in the router; we actually use an Exim filter, because it lets us do somewhat better logging and makes for an easier way to check the 'is a bounce of spam' condition.)

The tricky bit is detecting that this bounce is a bounce of spam, and doing so reliably. Exim routers and filters only have convenient access to headers, and when you're dealing with a bounce this means the headers of the bounce itself, not the headers of the message that bounced. You can dig into the body a bit, but that's unstructured text with all of the problems that that implies.

However, Exim lets you customize the generated bounce message via the bounce_message_file main configuration setting; in particular, you can add headers to it, and when the bounce message is generated you do have access to the original message's headers. Our spam filter tags the Subject: header, so we have a custom bounce message that copies the original message's Subject: into the bounce as X-Original-Subject:. Our bounce-discarding router then recognizes bounces of spam by looking for this header and then checking to see if it starts with the magic tags.

(We don't have to worry about users accidentally putting this header in their messages and having them get discarded, because we only trigger for locally-generated messages with the null envelope sender. Technically a user that tried hard enough could generate such a message, but then they could just send it to /dev/null directly.)

How not to set up your DNS (part 18)

We got contacted by a user reporting that he couldn't get mail with an address in kuet.ac.bd; our inbound mail gateway was consistently rejecting the address as temporarily unresolvable. When I started looking into the DNS situation, all sorts of peculiar things started crawling out of the woodwork.

• in the root zone, the bd country domain has two servers, dns.bd and slave.bttb.net.
• if you query either of them you get a third server as well, dns.bttb.net.
• all three nameservers allow recursion.
• dns.bd returns non-authoritative answers, which is especially fun when it returns a non-authoritative SOA for the bd country domain that lists itself as the primary nameserver.

• everything except dns.bd knows that the nameserver for the ac.bd subdomain is slave.bttb.net (under a different name).
• dns.bd returns SERVFAIL when queried for the ac.bd nameservers, much like a slave nameserver without the zone available. It does this even if you make a recursive query for the information.

• if you directly query any of the three about the nameservers for kuet.ac.bd you'll get the correct answer back. Yes, including from dns.bd.

We're not done yet: once you actually find the two nameservers for kuet.ac.bd, one of them doesn't respond at all. (It's not a simple connectivity failure either, since they have adjacent IP addresses.)

(Going along with the theme so far, the kuet.ac.bd nameserver that answers will also do recursive lookups for you.)

In theory there is a lookup chain that will get you the correct information, but in practice I don't blame our nameservers for throwing up their hands and returning a temporary failure for long enough to time out some email.