Wandering Thoughts archives

Some notes on (dynamic) Bind 9 logging

Bind 9's rndc has tasty looking options to change and increase logging on the fly, which is just what you want if you're trying to track down an erratic behavior in a caching nameserver (where restarting the server might well make the whole thing go away so you can't troubleshoot what's going on). However, it turns out that you need some Bind configuration options already configured in order to make it work. Since I just went through figuring this out (and I may need to do it again), here's what's needed:

• you need one or more logging channels configured with 'severity dynamic'. Without this, you can issue all the 'rndc trace 99' commands you want to and nothing extra will get logged.

  (Note that 'severity debug' does not do what you want.)

  Doing this only to a file-based channel is good, unless you really want to flood your syslog for some reason. If your Bind is running chroot'd (as is the default on, eg, OpenBSD) the file path for this is relative to the chroot.

• you must also add whatever logging categories you're interested in and configure them to use the channel you've set up with dynamic severity.

  Note that some categories will log additional things even at 'rndc notrace' level; this may make you want to comment them out when not running in a debug setup. Sadly there seems to be no way to enable and disable logging categories through rndc.

• 'rndc reconfig' may be helpful in getting a running Bind instance to notice your changed or added logging configuration without too many changes in its behavior. This really depends on what you're trying to debug, though.

  (It was successful for me, in that it didn't suddenly make the Bind instance start properly answering the query it had previously been failing on.)

Once you have done all of this, 'rndc querylog; rndc trace 99' should turn on more or less maximum logging, and then 'rndc querylog; rndc notrace' will turn it off. You may want to verify the current running setup with 'rndc status'.

Sadly, I cannot (yet) say what categories it is necessary to enable in order to get Bind to log enough information to reconstruct where a particular query answer comes from. Maybe I'll figure that out Monday, assuming I don't decide that the server has just started to malfunction and needs be restarted.

One aspect of getting used to modern version control

I've spent a very long time using RCS as a sysadmin. In that time, certain things about how it works have just gotten ingrained in my hindbrain, to the level where I don't think about them consciously; they just are. Now that we're slowly moving away from RCS and to modern version control systems, I sometimes wind up running into bits of this that are no longer true, where I have to let how stuff works in modern version control gradually soak into my mind until old reflexive assumptions get dissolved.

One of them is the idea that committing things does not change the files that I am committing. Some of you may be going 'well of course it doesn't', but this is not how it works in RCS; when you commit a file in RCS, RCS removes and then recreates the file. For developers this may not be something that's very important, but it can really matter for sysadmins; among other things, anything that modifies live files makes us nervous.

(This also means that committing a file can change the file ownership, which is important when you absently ci a file as root or are working in a shared directory with group permissions.)

Modern version control systems don't do this; making a commit may change the repository, but it does not change the live file itself. This makes commits much less twitch inducing to sysadmins and much more like actual backups. Forgetting this difference makes me more twitchy about making commits than I should be and leads to occasional embarrassing mistakes.

(It also means that it is much less of an impact to set up an initial repository than it is to do the equivalent in RCS; with a modern VCS, it is effectively the same as making a tar or rsync or whatever backup of the files.)

A simple thing that every package management system should have

Here's something that I was reminded about recently, most directly due to doing some work on a Solaris machine (although the general issue has come up before).

Every single package management should provide a way to save the name and version information for all packages and patches (if you use them) that are installed on a system, and also a way of easily setting up another system with exactly that set of packages and patches. This should be done through straightforward command line tools; it should certainly not require installing or adopting a great big system management system, using a web frontend or management console, or whatever. Both of these tools should be installed by default.

Ideally these tools would produce and consume plain text in a straightforward format. If you insist on making your version of these tools use XML or some other private database format, you need to also provide tools that dump a readable form of this and report the differences between two package states.

(Sooner or later sysadmins are going to want both abilities.)

By the way, if your operating system or Linux distribution can't always recreate a given set of package versions for whatever reason, you cannot really aspire to being a 'production' or 'enterprise' ready operating system. One common reason is feeling that some updates are mandatory and so not giving people any option about those updates when they install new systems; another is removing old versions of packages or patches when newer versions are released.

(This means that I don't think that Fedora is production ready, since Fedora deletes the binary RPMs for obsoleted updates from mirrors. You can get around this by maintaining your own local Fedora mirror that never deletes anything. Note that I understand why Fedora does this, and being 'production ready' in this way is not part of Fedora's goals anyways. Red Hat Enterprise does keep all package versions around forever for exactly this reason.)

An observation from changing my password

I've changed my password at work, or started to change it at least (this will be an extended process). Doing this has reinforced some things that I know but rarely think about, and exposed a surprising inconvenience in how I do things.

The big thing is that you don't really remember how many machines you have accounts on until you try to work out how many different places you need to change your password. This is not really an issue for users (if us sysadmins are doing our job right, they change their password once and it magically propagates everywhere), but as a sysadmin I have access to all sorts of isolated machines that are not part of our password propagation system. Which means that I get to change my password on all of them, assuming that I can remember what they all are.

(In looking at this, I see that usermod on Linux machines actually has an option to just staple a new encrypted password into place. This reduces the problem to running a command as root on most of those machines, which is a mostly solved problem around here. In fact, I was already using 'run a command everywhere' to check /etc/shadow to see if I'd updated my password by looking at the last-changed field.)

The surprising inconvenience is that I have set up ssh identities to give me passwordless access to my account on most machines; in fact, a lot of my usual environment relies on it. This did not strike me as a problem until I changed my password and suddenly started wanting to type the new one as much as possible to reinforce it in my mind and my fingers. Suddenly all of that passwordless access was inconvenient as well as convenient, since it meant that I'm really not typing my password all that much. This has both surprised and amused me, because sometimes I am easily amused by the perversities of life.

(Turning my ssh identities off completely would likely make various parts of my environment explode in even less convenient ways, so I've resorted to modifying an ssh cover script I already had lying around to turn this off, and using the cover script periodically just to reinforce things. You might wonder why I have an ssh cover script lying around, one that I do not mind hacking up this way; the answer is that it's set up to ignore my known-hosts file, which is very convenient when you keep reinstalling virtual machines that you want to ssh in to.)