sysadmin/RsyncReplicationSetup written at 23:54:57; Add Comment
Using rsync to pull a directory tree to client machines
Suppose that you have a decent sized directory tree that you want some number of clients to mirror
from a master server (with the clients pulling updates instead of the
master pushing them), perhaps because you've just noticed undesired
NFS dependencies. Things in the directory
tree are potentially sensitive (so you want access control), it's updated
at random, and it's not in a giant VCS tree or something; this is your
typical medium-sized ball of local stuff. The straightforward brute
force approach is to use rsync with SSH; give the clients special SSH
identities, put them in the server's authorized_keys, and have them
(You also have to set the SSH access up so that the clients can't run arbitrary commands on the server.)
Rsync's solution to this is its daemon mode, which can restricted to operate in read only mode. Normally rsync wants to be run this way as an actual daemon (listening on a port and so on), but that requires us to use rsync's weaker and harder to manage authentication, access control, and other things. I would rather continue to run daemon mode rsync over plain SSH and take advantage of all of the existing, proven SSH features for various things.
(The rsync manpage suggests hacks like binding the rsync daemon to only listen on localhost on the server and then using SSH port forwarding to give clients access to it. But those are hacks and require making various assumptions.)
How to to do this is not obvious from the documentation, so here is
the setup I have come up with for doing this on both the server and
the clients. First, you need an
use chroot = no[somepath] comment = Replication module path = /some/path read only = true # if necessary: uid = 0 gid = 0
Next, you need a script on the server that will force an incoming SSH
login to run rsync in daemon mode against this configuration file and do
nothing else. We will set this as the
Note that this completely ignores any arguments that the client attempts
to supply. However, this doesn't matter; as far as I can tell, the
command line that the clients send will always be '
On the server, the login that you're using for this should have
Finally, on the client you need to run rsync with all of the necessary arguments. You probably want to put this in a script:
Potentially useful additional arguments for
If you run this from cron, remember to add some locking to prevent two copies from running at once. If the directory tree is large and you have enough clients, you may want to add some amount of randomization of the start times for the replication in order to keep load down on the master server.
(There may be a better way to do this with rsync; if you know of one, let me know in the comments. For various reasons we're probably not interested in doing this with any other tool, partly because we already have rsync and not the other tools. Another tool would have to be very much better than rsync to really be worth switching to.)
sysadmin/PowerdownLessonsLearnedII written at 00:37:34; Add Comment
Things I will do differently in the next building power shutdown (part 2)
Back at the start of last September, we had an overnight building wide power shutdown in the building with our machine room and I wrote a lessons-learned entry in the aftermath. Well, we just had another one and apparently I didn't learn all of the lessons that I needed to learn the first time around. So here's another set of things that I've now learned.
Next time around I will:
My entry from last time was very useful in several ways. I reread it when I was preparing our checklist for this time and it jogged my memory about several important issues; as a result our checklist for this time around was (I think) significantly better than for last time (and also noticeably longer and more verbose). This time I at least made new mistakes, which is progress that I can live with.
I will also probably try to put more explanation into the checklist the next time around. I'm sure it's possible to put too much of it in, but I don't think that's been our problem so far. In the heat of the moment we're going to skim anyways, so the thing to do is to break the checklist up into skimmable blocks with actions and things to check off and then chunks of additional explanation after them.
(In a sense a checklist like this serves two purposes at once. During the power down or power up it is mostly a catalog of actions and ordering, but beforehand it's a discussion and a rationale for what needs to be done and why. Without the logic behind it being written out explicitly, you can't have that discussion; once you have that logic written out, you might as well leave it in to jog people's memories on the spot.)
On a side note, a full power up is an interesting and useful way to find problematic dependencies that have quietly worked their way into your overall network, ones that are not so noticeable when your systems are in their normal steady state. For example, DHCP service for several of our networks now depends on our core fileserver, which means that it can only come up fairly late in the power up process. We're going to be fixing that.
(There is a chain of dependencies that made this make sense in a steady state environment.)
* * *
Atom feeds are available; see the bottom of most pages.