Wandering Thoughts archives

In practice, anything involving the JVM is often a heavyweight thing

Last week I asked on Twitter if anyone had a good replacement for swish-e for indexing and searching some HTML pages. Several people suggested Apache Solr; my immediate reaction was that this sounded too heavyweight for what we wanted. It was then asserted that Solr is not that heavy if you disable enough things. I had a number of reactions to that, but my instant one was 'nothing involving the JVM is lightweight'. Today I want to talk about that.

I don't call JVM-based things 'heavyweight' because Java itself can easily eat up lots of memory (although that's certainly a potential issue). What makes the JVM heavy for us is that we don't already run any JVM-based services and that all too often, Java is not like other languages. With languages like Python, Perl, Ruby, or even PHP, as a sysadmin you can generally be pretty indifferent to the language the system is written in. You install the system (ideally through a package manager), you get some binaries and maybe some crontab jobs, and you run the binaries. You're done. With Java, my impression and to some extent my experience is that you also have to administer and manage a JVM. A Java system is not run some programs and forget; it's putting .jars in the right place, it's loading certificates into JVM trust stores, it's configuring JVM parameters, and so on and so forth. There is a whole level of extra things to learn and things to do that you take on in order to maintain the JVM environment for the system you actually want to run.

(One way to put it is that a JVM seems to often be a system inside your normal system. You get to maintain your normal system and you also get to learn how to maintain the JVM system as well.)

All of this makes any JVM-based system a heavyweight one, because adopting it means not just learning the system but also learning how to manage a probably-complex JVM environment. If we were already running JVM based things it would be a different issue, of course, because we'd probably already have this expertise (and the JVM way might even work better for us), but as it stands we don't.

(Similar issues probably hold for any Node-based system, partly because of Node itself and partly because Node has its own very popular package management system that we'd probably have to learn and wrangle in order to run any Node-based thing.)

It's probably possible to design JVM-using systems that are not 'JVM-based' in this way and that encapsulate all of the complexity inside themselves. But I suspect that something labeled on its website as 'enterprise' has not been designed to operate this way.

(I've mostly talked about the JVM instead of Java specifically because I suspect most of these issues also apply to any other JVM-based language, such as Scala, Clojure, and so on.)

Perhaps it's a good idea to reboot everything periodically

Yesterday around 6pm, the department's connection to the campus backbone had its performance basically fall off a cliff. Packet loss jumped and bandwidth dropped from that appropriate to gigabit Ethernet down to the level of a good home connection (it seems to have been running around 16 Mbits/sec inbound, although somewhat more outbound). People started noticing this morning, which resulted in us running around and talking to the university's central NOC (who run the backbone and thus the router that we connect to).

Everything looked perfectly normal on our side of things, with no errors being logged, all relevant interfaces up at 1G, and so on. But in the process of looking at things, we noticed that our bridging firewall had been up for 450 days or so. Since we have a ready hot spare and pfsync makes shifting over relatively non-disruptive, we (by which I mean my co-workers) decided to switch the active and hot spare machines (after rebooting the hot spare). Lo and behold, all of our backbone performance problems went away on the spot.

We reboot our Ubuntu machines on a relatively regular basis in order to apply kernel updates, because they're exposed to users. But many of our other machines we treat as appliances and as part of that we basically don't reboot them unless there's some compelling reason to do so. That's how we wind up with firewalls with 450 day uptimes, fileservers and backends that have mostly been up since they were installed a year or so ago, and so on.

Perhaps we should rethink that. In fact, if we're going to rethink things and agree to reboot machines every so often, we should actually make a relatively concrete schedule for it in advance. We don't have to schedule down to the day or week, but even something like deciding that all of the firewalls will be rebooted in March is likely to drastically increase the odds that it will actually happen.

('We should reboot the firewalls after they've been up for a while' is sufficiently fuzzy that it is at best a low priority entry in someone's to-do list, and thus easy to forget about or never get to. Adding 'in March' pushes things closer to the point where someone will put it on their calendar and then get it done.)

Why I never tell people how I voted

Canada just had a federal election, and I got asked in passing who I voted for. As always, I gave my standard answer for this question, namely that I don't ever talk about how I voted. This is the answer I give to everyone who asks, even my kith and kin; I'll never admit to voting for anyone, no matter what.

It doesn't matter that my likely votes are probably often easy to guess (especially for my kith and kin), and it's not quite as simple as me feeling that on principle you should not interrogate people directly about this question (although asked bluntly it does make me twitch). Instead it is in large part about keeping and concealing secrets by denying people information.

Imagine that I normally told people who or what I voted for in votes, and that someday I had a vote that I wanted to conceal or not admit to for some reason. I'd have a problem; if I said 'I'm not going to tell you' that time, it'd be immediately obvious that I was hiding something (and that I probably hadn't voted for the predictable choice). I'd have to try to lie (and lie convincingly), perhaps to people who knew me.

By issuing a blanket 'no comment' about my votes (with no coy hints or anything) even when I don't deeply care, I buy myself cover for a day when I really do want to conceal my vote. That day I can give exactly the same answer I have all the times before, in exactly the same way, and there (hopefully) won't be any signs that anything is going on.

None of this is new or novel. It's always been the case that if you think you might someday have something to conceal that you should shut up about all aspects of it now; if you answer questions or give people information today and then stop, it is much more damaging than never giving information. As we all know, a sudden switch to 'no comment' might as well be an admission to the wise.

This is of course applicable to sysadmins, because we are partly keepers of secrets and sometimes, what we deal with is only sometimes secret. Or maybe it's not secret now and we don't realize that someday in the future it might be, so we share it openly now and then oops. We may not have leaked the secret when we clam up, but we've certainly admitted that there's a secret.

I don't think that this means sysadmins should never share or expose information. Instead, we should consider whether the information we're sharing and exposing and casually admitting to people now when they ask might, sometime in the future, become sensitive (and how possible or likely that is). If we can easily imagine scenarios where it becomes sensitive, well, maybe we should clam up up front and get used to saying 'I'm sorry, I can't tell you that, it's potentially sensitive information' even if it's perfectly harmless right now.

(I'm honestly not sure how likely this is to come up, though. I have the feeling that a lot of what we deal with is obviously potentially sensitive, as opposed to currently harmless but later dangerous. On the other hand, I may just be mistaking the latter sort of stuff for things that are entirely harmless.)

OS installers should be easy to open up and modify

A while back I tweeted:

Every so often I dream of genuinely sysadmin-friendly OS installers. I doubt I'll ever get them, though.

As it happens, I have a specific vision of what being sysadmin friendly here should be and that is easy modifications.

The reality of life is that OS installers are necessarily very general but they are most frequently used in much more specific situations, ones where you already know the answers to a lot of the questions they'll ask. At the same time there's often a few local questions of your own that are important and need to get answered in order to set up a machine properly, and not infrequently the set of packages that is available on the install media is not ideal.

A genuinely sysadmin friendly installer would be one that fully understood this reality. It would make it not merely possible but easy to open up the installer image, make this sort of straightforward changes to it, put the whole thing back together, and have a custom setup that fit your environment. Want to run a post-install script? It should let you do that. Want to drop some additional scripts and files into the installed system? Ditto. And so on.

(A nice implementation would allow you to ask your own questions within the installer's Q/A framework and then export those answers to your post-install scripts or write them into the system or the like. A really nice implementation would allow you to use these answers to drive some of the regular installer actions somehow, although that gets more complicated.)

Of course I know that this is a quixotic quest. The modern world has decided that the answer to this problem is doing as little as possible in the actual installer and customizing the system after install using an automation framework (Puppet, Chef, Ansible, etc). I can't even argue with the pragmatics of this answer; getting an automation framework going is a lot easier than trying to persuade Debian, Red Hat, FreeBSD, OmniOS, and so on and so forth to change their installers to be nicer for sysadmins (and a full framework gives you a much more uniform cross system experience).

(Debian and Ubuntu have a 'preseed' feature but apparently rebuilding installer images is somewhat of a pain and preseed only goes so far, eg I don't think you can add your own questions or the like.)

(This whole chain of thought was sparked by working with our standard install system and thinking about how much of it could and really should be in the actual installer.)

Bad news about how we detect and recover from NFS server problems

In a comment on this entry, Sandip Bhattacharya asked me:

Also, sometimes transient NFS server issues can cause the NFS mount to be wedged, where any access to the NFS mount hangs the process. How would do you escape or detect such conditions?

This is a good question in general and I am afraid the bad news is that there don't seem to be any good answers. Our usual method of 'detecting' such problems is that a succession of machines start falling over with absurd load averages; generally this is our central mailer, our primary IMAP server, our web server, and our most heavily used login server. This is of course not entirely satisfactory, but doing better is hard. Client kernels will generally start spitting out 'NFS server <X> not responding, still trying' messages somewhat before they keel over from excess load and delays, but you can have temporary blips of these messages even without server problems and on top of that you'd need very fast response before active machines start getting into bad situations.

(A web server is an especially bad case, since it keeps getting new requests all the time. If processes are stalling on IO, it doesn't take very much time before your server is totally overwhelmed. Many other servers at least don't spawn new activity quite so fast.)

As far as escaping the situation, well, again we haven't found any good solutions. If we're really lucky, we can catch a situation early enough that we can unmount currently unused and thus not-yet-wedged NFS filesystems from our clients. Unfortunately this is rare and doesn't help the really active machines. In theory clients offer ways to force NFS unmounts; in practice this has often not worked for us (on Linux) for actively used NFS filesystems. Generally we have to either get the NFS server to start working again (perhaps by rebooting the server) or force client reboots, after which they won't NFS mount stuff from the bad server.

(If a NFS server is experiencing ongoing or repeated problems, sometimes we can reboot it and have it return to good service long enough to unmount all of its filesystems on clients.)

In theory, you can fake a totally lost NFS server by having another NFS server take over the IP address so that at least clients will get 'permission denied, filesystem not exported' errors instead of no replies at all. In practice, this can run into serious client issues with the handling of stale NFS mounts so you probably don't want to do this unless you've already tested the result and know it isn't going to blow up in your face.

The whole situation with unresponsive NFS servers has been a real problem for as long as NFS has existed, but so far no one seems to have come up with good client-side solutions to make detecting and managing problems easier. I suspect one reason for this is that NFS servers are generally very reliable, which doesn't give people much motive to create complicated solutions for when they aren't.

(For reasons covered here, I feel that an automounter is not the answer to this problem in most cases. Anyways, we have our own NFS mount management solution.)

Our low-rent approach to verifying that NFS mounts are there

Our mail system has everyone's inboxes in an old-fashioned /var/mail style single directory; in fact it literally is /var/mail. This directory is NFS mounted from one of our fileservers, which raises a little question: how can we be sure that it's actually there? Well, there's always going to be a /var/mail directory. But what we care about is that this directory is the actual NFS mounted filesystem instead of the directory on the local root filesystem that is the mount point, because we very much do not want to ever deliver email to the latter.

(Some people may say that limited directory permissions on the mount point should make delivery attempts fail. 'Should' is not a word that I like in this situation, either in 'should fail' or 'that failure should be retried'.)

There are probably lots of clever solutions to this problem involving advanced tricks like embedded Perl bits in the mailer that look at NFS mount state and so on. We opted for a simple and low tech approach: we have a magic flag file in the NFS version of /var/mail, imaginatively called .NFS-MOUNTED. If the flag file is not present, we assume that the filesystem is not mounted and stall all email delivery to /var/mail.

This scheme is subject to various potential issues (like accidentally deleting .NFS-MOUNTED some day), but it has the great virtue that it is simple and relatively bulletproof. It helps that Exim has robust support for checking whether or not a file exists (although we use a hack for various reasons). The whole thing has worked well and basically transparently, and we haven't removed one those .NFS-MOUNTED files by accident yet.

(We actually use this trick for several NFS-mounted mail related directories that we need to verify are present before we start trying to do things involving them, not just /var/mail.)

(I mentioned this trick in passing here, but today I feel like writing it up explicitly.)

Sidebar: our alternate approach with user home directories

Since user home directories are NFS mounted, you might be wondering if we also use flag files there to verify that the NFS mounts are present before checking things like .forward files. Because of how our NFS mounts are organized, we use an alternate approach instead. In short, our NFS mounts aren't directly for user home directories; instead they're for filesystems with user home directories in them.

(A user has a home directory like /h/281/cks, where /h/281 is the actual NFS mounted filesystem.)

In this situation it suffices to just check that the user's home directory exists. If it does, the NFS filesystem it is in must be mounted (well, unless someone has done something very perverse). As a useful side bonus, this guards against various other errors (eg, 'user home directory was listed wrong in /etc/passwd').

There are two approaches to disaster recovery plans

One of the things I've come to believe in is that there are two ways to approach thinking about your disaster recovery planning (even if you're being quite abstract). The first approach is to think about how you'd restore your services, while the second approach is to think about how you'd get your users working again.

Thinking about how you'd restore your services is very sexy. It leads to planning out things like alternate server rooms, temporary cloud hosting, essential initial infrastructure, what you'd buy if you needed replacement hardware in a hurry, and so on. If you carry it all the way through you wind up with binders of multi-step plans, indexes of what backups are where, and so on.

Thinking about how you'd get your users working again can wind up in a much less comfortable and sexy place, because the first question you have to ask is what your users really need in order to get their work done (or at least to get their important work done). Asking this question can confront you with the reality that a lot of your services and facilities are not really essential for your users, and that what they really care about may not be things you consider important. The first stage disaster recovery plans that can result from this may wind up being much more modest and less sexy than the 'let's rebuild the machine room' sort of plans.

(For example, in places like us the first stage disaster recovery plan might be 'buy everyone important a laptop if they don't already have one, maybe restore some people's address books, and they all go set up GMail accounts and getting back in touch with people they email'.)

Focusing on what your users need to get working again doesn't mean not also having the first sort of disaster recovery plans, since presumably you are going to want to get all your services back eventually. But I think it puts them in the right perspective. The important thing is that necessary work gets done; your services are just a means to that end.

(This is kind of the flipside of what I wrote a while back about the purpose of computer disaster recovery.)

(Of course, if you have an actual disaster without preallocated resources you may find out that some of your services are not important enough any more to come back, or to come back in anywhere near their original form. There's nothing like starting from scratch to cause drastic reassessments of situations.)