Wandering Thoughts

The evolution of our account creation script

One of the things about system administration automation is that its evolution often follows the path of least resistance. This can leave you with interesting and peculiar historical remnants, and it can also create situations where it takes a relatively long time before a system does the obvious thing. As it happens, I have a story about this.

To go with our account request system, which handles people requesting new accounts and authorizing requested accounts, we have an actual script that we run to actually create Unix accounts. Until relatively recently that script asked you a bunch of questions, although they all had default answers that we'd accept essentially all of the time. The presence of these questions was both a historical remnant of the path that the script took and an illustration of how unquestioningly acclimatized we can all become to what we think of as 'normal'.

We have been running Unix systems and creating accounts on them for a very long time, and in particular we've been doing this since before the World Wide Web existed and was readily accessible. Back in the beginning of things, accounts were requested on printed forms; graduate students and suchlike filled out the form with the information, got their account sponsors to sign it, handed it to the system staff, and the system staff typed all of the information into a script that asked us questions like 'login?', 'name?', 'Unix group?', 'research group affiliation?', and so on.

At a certain point, the web became enough of a thing that having a CGI version of our paper account request form was an obvious thing to do. Not everyone was going to use the CGI form (or be able to), and anyway we already had the account creation script that knew all of the magic required to properly create an account around here, so we adopted the existing script to also work with the CGI. The CGI wrote out the submitted information into a file (basically as setting shell environment variables) and this file was then loaded into the account creation script as the default answers to many of the questions that had originally been fields on the printed form. If the submitted information was good, you could just hit Return through many of the questions. After you created the account, you then had to email some important information about it (especially the temporary password) off to the person it was for; you did this by hand, because you generated the random password by hand outside of the script.

(For reasons lost to history, the data file that the CGI wrote and the script loaded was a m4 file that was then processed through m4 to create shell variable assignments.)

When we wrote our account request system to replace the basic CGI (and the workflow around it, which involved manually emailing account sponsors to ask them about approving accounts), the simple and easy way for it to actually get accounts created was to carefully write the same data file that the CGI had used (m4isms and all). The account request script remained basically unchanged, and in particular it kept asking us to confirm all of the 'default' answers, ie all of the information that the account request system had already validated and generated. More than that, we added a few more bits of special handling for some accounts, with their own questions.

(Although the account request system was created in 2011, it took until a 2016 major revision for a new version of Django for us to switch from generating m4 data files to just directly generating shell variable assignments that the script directly sourced with the . command.)

That we had to actually answer these questions and then write the 'you have a new account' email made the whole process of creating an account a tedious thing. You couldn't just start the script and go away for a while; you had to periodically interact with it, hitting Return, generating a password in another window and pasting it in to the password prompt, and composing email yourself. None of these things were actually necessary for the backend of the account request system, but they stayed for historical reasons (and because we needed them occasionally, because some accounts are created outside of the account request system). And we, the people who used the script, were so acclimatized to this situation that we didn't really think about it; in fact I built my own automation around writing the 'you have a new account' form email.

At this point I've forgotten what the exact trigger event was, but last year around this time, in the middle of creating a bunch of new graduate student accounts (where the existing script's behavior was at its most tedious), we realized that this could be fixed. I'll quote my commit messages:

New 'fast create' mode for account creation that takes all the defaults and doesn't bother asking if we're really sure.

For fast mode, add the ability to randomly generate or set the initial password at the start of the process.

offer to send new-account greeting email.
make sending greeting email be the default (if you just hit return).

(In theory we could make sending the greeting email happen automatically. In practice, asking a final question gives us an opportunity to look back at all the messages printed out just in case there's some problem that the script didn't catch and we want to pause to fix things up.)

This simple change has been a real quality of life improvement for us, turning a tedious slog into a mostly fire and forget exercise that we can casually run through. That it took so long to make our account creation script behave this way is an illustration not just of the power of historical paths but also of the power of habituation. We were so used to how the existing system worked that we never really questioned if it had to be that way; we just grumbled and accepted it.

(This is, in a sense, part of the power of historical paths. The path that something took to get where it is shapes what we see as 'normal' and 'just how things are', because it's what we get used to.)

Sidebar: There were some additional steps in there

There are a few questions in the account creation script where in theory we have a genuine choice to make; for example, some accounts have several options for what filesystem they get created in. Part of what made the no-questions version of the script possible was that we realized that in practice we always made a particular choice (for filesystems, we always picked the one with the most free space), so we revised the script to make this choice the default answer.

Had we not worked out default answers for all of these questions, we couldn't have made the creation script not even ask the questions. We might have done both at the same time if it was necessary, but in practice it certainly helped that everything already had default answers so the 'fast create' mode could just be 'take all of the default answers without requiring confirmation'.

The benefits of driving automation through cron

In light of our problem with timesyncd, we needed a different (and working) solution for time synchronization on our Ubuntu 18.04 machines. The obvious solution would have been to switch over to chrony; Ubuntu even has chrony set up so that if you run it, timesyncd is automatically blocked. I like chrony so I was tempted by this idea briefly, but then I realized that using chrony would mean having yet another daemon that we have to care about. Instead, our replacement for timesyncd is running ntpdate from cron.

There are a number of quiet virtues of driving automation out of cron entries. The whole approach is simple and brute force, but this creates a great deal of reliability. Cron basically never dies and if it were ever to die it's so central to how our systems operate that we'd probably notice fairly fast. If we're ever in any doubt, cron logs when it runs things to syslog (and thus to our central syslog server), and if jobs fail or produce output, cron has a very reliable and well tested system for reporting that to us. A simple cron entry that runs ntpdate has no ongoing state that can get messed up, so if cron is running at all, the ntpdate is running at its scheduled interval and so our clocks will stay synchronized. If something goes wrong on one run, it doesn't really matter because cron will run it again later. Network down temporarily? DNS resolution broken? NTP servers unhappy? Cure the issue and we'll automatically get time synchronization back.

A cron job is simple blunt force; it repeats its activities over and over and over again, throwing itself at the system until it batters its way through and things work. Unless you program it otherwise, it's stateless and so indifferent to what happened the last time around. There's a lot to be said for this in many system tasks, including synchronizing the clock.

(Of course this can be a drawback if you have a cron job that's failing and generating email every failure, when you'd like just one email on the first failure. Life is not perfect.)

There's always a temptation in system administration to make things complicated, to run daemons and build services and so on. But sometimes the straightforward brute force way is the best answer. We could run a NTP daemon on our Ubuntu machines, and on a few of them we probably will (such as our new fileservers), but for everything else, a cron job is the right approach. Probably it's the right approach for some of our other problems, too.

(If timesyncd worked completely reliably on Ubuntu 18.04, we would likely stick with it simply because it's less work to use the system's default setup. But since it doesn't, we need to do something.)

PS: Although we don't actively monitor cron right now, there are ways to notice if it dies. Possibly we should add some explicit monitoring for cron on all of our machines, given how central it is to things like our password propagation system. Sure, we'd notice sooner or later anyway, but noticing sooner is good.

One simple general pattern for making sure things are alive

One perpetual problem in system monitoring is detecting when something goes away. Detecting the presence of something is often easy because it reports itself, but detecting absence is usually harder. For example, it generally doesn't work well to have some software system email you when it completes its once a day task, because the odds are only so-so that you'll actually notice on the day when the expected email isn't there in your mailbox.

One general pattern for dealing with this is what I'll call a staleness timer. In a staleness timer you have a timer that effectively slowly counts down; when the timer reaches 0, you get an alert. When systems report in that they're alive, this report resets their timer to its full value. You can implement this as a direct timer, or you can write a check that is 'if system last reported in more than X time ago, raise an alert' (and have this check run every so often).

(More generally, if you have an overall metrics system you can presumably write an alert for 'last metric from source <X> is more than <Y> old'.)

In a way this general pattern works because you've flipped the problem around. Instead of the default state being silence and exceptional things having to happen to generate an alert, the default state is an alert and exceptional things have to happen to temporarily suppress the alert.

There are all sorts of ways of making programs and systems report in, depending on what you have available and what you want to check. Traditional low rent approaches are touching files and sending email to special dedicated email aliases (which may write incoming email to a file, or simply run a program on incoming email that touches a relevant file). These can have the drawback that they depend on multiple different systems all working, but they often have the advantage that you have them working already (and sometimes it's a feature to verify all of the systems at once).

(If you have a real monitoring system, it hopefully already provides a full selection of ways to submit 'I am still alive' notifications to it. There probably is a very simple system that just does this based on netcat-level TCP messages or the like, too; it seems like the kind of thing sysadmins write every so often. Or perhaps we are just unusual in never having put together a modern, flexible, and readily customizable monitoring system.)

All of this is a reasonably obvious and well known thing around the general community, but for my own reasons I want to write it down explicitly.

The hidden danger of using rsync to copy files instead of cp

I have a long standing reflexive habit that most of them time when I want to copy files around, I reach for 'rsync -a' by default. I do it on the command line, and I do it in things like our local postinstall system setup scripts. It's not really necessary these days ('cp -a' now works fine on everything I commonly use), but I started doing this in an era when rsync was the clearly safest choice for a 'copy file, preserving all attributes, no matter what system I'm on' command. Today I made a mistake and was reminded that this is not necessarily the best idea, because there is a small difference in behavior between rsync and cp here.

What happened today is that in a system setup script, I wrote:

set -e
[...]
rsync -a /master/loc/etc/cron.d/cron-thing /etc/crond.d/

I ran the script, it went fine, and then afterward the system didn't actually seem to be doing what the cron.d entry was supposed to have it do. I spent some time wondering if I'd gotten some other bit of the system setup wrong, so that the script I was invoking from cron couldn't do anything, and then finally I looked in /etc/cron.d for some reason and the penny dropped.

You see, the important difference between rsync and cp here is that rsync will create a destination directory if necessary and cp won't. The drawback of this sometimes-handy behavior is that rsync's behavior hides typos. Had I written 'cp -a ... /etc/crond.d/', the cp would have errored out (and then the entire script would have aborted). With rsync, it quietly created /etc/crond.d and put my cron-thing in it, just as I'd typed but not as I'd wanted.

After this happened, I went back through this script and turned all of my reflexive 'rsync -a' usage into 'cp -a'. I've been burned once, I don't need to stub my toe a second time. I don't currently plan to revise our existing (working) scripts just for this, but certainly I'm now going to try to shift my reflexes and use 'cp -a' in the future.

(In this sort of context, even if I want the directory created too I think it's better to use 'mkdir -p' in order to be explicit about it. On the command line I might exploit rsync's side effect, but in a script there's no reason to be that compact and obscure.)

Being reminded that an obvious problem isn't necessarily obvious

The other day we had a problem with one of our NFS fileservers, where a ZFS filesystem filled up to its quota limit, people kept writing to the filesystem at high volume, and the fileserver got unhappy. This nice neat description hides the fact that it took me some time to notice that the one filesystem that our DTrace scripts were pointing to as having all of the slow NFS IO was a full filesystem. Then and only then did the penny finally start dropping (which led me to a temporary fix).

(I should note that we had Amanda backups and a ZFS pool scrub happening on the fileserver at the time, so there were a number of ways it could have been overwhelmed.)

In the immediate aftermath, I felt a bit silly for missing such an obvious issue. I'm pretty sure we've seen the 'full filesystem plus ongoing writes leads to problems' issue, and we've certainly seen similar problems with full pools. In fact four years ago I wrote an entry about remembering to check for this sort of stuff in a crisis. Then I thought about it more and kicked myself for hindsight bias.

The reality of sysadmin life is that in many situations, there are too many obvious problem causes to keep track of them all. We will remember common 'obvious' things, by which I mean things that keep happening to us. But fallible humans with limited memories simply can't keep track of infrequent things that are merely easy to spot if you remember where to look. These things are 'obvious' in a technical sense, but they are not in a practical sense.

This is one reason why having a pre-written list of things to check is so potentially useful; it effectively remembers all of these obvious problem causes for you. You could just write them all down by themselves, but generally you might as well start by describing what to check and only then say 'if this check is positive ...'. You can also turn these checks (or some of them) into a script that you run and that reports anything it finds, or create a dashboard in your monitoring and alert system. There are lots of options.

(Will we try to create such a checklist or diagnosis script? Probably not for our current fileservers, since they're getting replaced with a completely different OS in hopefully not too much time. Instead we'll just hope that we don't have more problems over their remaining lifetime, and probably I'll remember to check for full filesystems if this happens again in the near future.)

Sidebar: Why our (limited) alerting system didn't tell us anything

The simple version is that our system can't alert us only on the combination of a full filesystem, NFS problems with that fileserver, and perhaps an observed high write volume to it. Instead the best it can do is alert us on full filesystems alone, and that happens too often to be useful (especially since it's not something we can do anything about).

Word-boundary regexp searches are what I usually want

I'm a person of relatively fixed and slow to change habits as far as Unix commands go. Once I've gotten used to doing something in one way, that's generally it, and worse, many of my habits fossilized many years ago. All of this is a long-winded lead in to explaing why I have only recently gotten around to really using the '\b' regular expression escape character. This is a real pity, because now that I have my big reaction is 'what took me so long?'

Perhaps unsurprisingly, it turns out that I almost always want to search for full words, not parts of words. This is true whether I'm looking for words in text, words in my email, or for functions, variables, and the like in code. In the past I adopted various hacks to deal with this, or just dealt with the irritation of excessive matches, but now I've converted over to using word-boundary searches and the improvement in getting what I really want is really great. It removes another little invisible point of friction and, like things before it, has had an outsized impact on how I feel about things.

(In retrospect, this is part of what how we write logins in documentation was doing. Searching for '<LOGIN>' instead of 'LOGIN' vastly reduced the chance that you'd run into the login embedded in another word.)

There are a couple of ways of doing word-boundary searches (somewhat depending on the program). The advantage of '\b' is that it works pretty universally; it's supported by at least (GNU) grep, ripgrep, and less, and it's at least worth trying in almost anything that supports modern (or 'PCRE') regular expressions, which is a lot of things. Grep and ripgrep also support the -w option for doing this, which is especially useful because it works with fgrep.

(I reflexively default to fgrep, partly so I don't have to think about special characters in my search string.)

Per this SO question and its answers, in vim I'd need to use '\<' and '/>' for the beginning and end of words. I'm sure vim has a reason for having two of them. Emacs supports '\b', although I don't actually do regular expression searches in Emacs regularly enough to remember how to invoke them (since I just looked it up, the documentation tells me it's C-M-s and C-M-r, which ought to be reasonably memorable given plain searches).

PS: Before I started writing this entry, I didn't know about -w in grep and ripgrep, or how to do this in vim (and I would have only been guessing about Emacs). Once again, doing some research has proven beneficial.

PPS: I care about less because less is often my default way of scanning through pretty much anything, whether it's a big text file or code. Grep and company may tell me what files have some content and a bit of its context, but less is what let me poke around, jump back and forth, and so on. Perhaps someday I will get a better program for this purpose, but probably not soon.

We've decided to write our future Python tools in Python 3

About a year ago I wrote about our decision to restrict what languages we use to develop internal tools and mentioned that one of the languages we picked was Python. At the time, I mostly meant Python 2 (although we already had one Python 3 program even then, which I sort of had qualms about). Since I now believe in using Python 3 for new code, I decided that the right thing for us to do was explicitly consider the issue and reach a decision, rather than just tacitly winding up in some state.

Our decision is perhaps unsurprising; my co-workers are entirely willing to go along with a slow migration to Python 3. We've now actively decided that new or significantly revised tools written in Python will be written in Python 3 or ported to it (barring some important reason not to do so, for example if the new code needs to still run on our important OmniOS machines). Python 3 is the more future proof choice, and all of the machines where we're going to run Python in the future have a recent enough version of Python 3.

That this came up now is not happenstance or coincidence. We have a suite of local ZFS cover programs and our own ZFS spares handling system, which are all primarily written in Python 2. With a significantly different fileserver setup on the horizon, I've recently started work on 'porting' these programs over to our new fileserver environment (where, for example, we won't have iSCSI backends). This work involves significant revisions and an entirely new set of code to do things like derive disk mapping information under Linux on our new hardware. When I started writing this new code, I asked myself whether this new code in this new environment should still be Python 2 code or whether we should take the opportunity to move it to Python 3 while I was doing major work anyway. I now have an answer; this code is going to be Python 3 code.

(We have Python 3 code already in production, but that code is not critical in the way that our ZFS status monitoring and spares system will be.)

Existing Python 2 code that's working perfectly fine will mostly or entirely remain that way, because we have much more important things to do right now (and generally, all the time). We'll have to deal with it someday (some of it is already ten years old and will probably run for at least another ten), but it can wait.

(A chunk of this code is our password propagation system, but there's an outside chance that we'll wind up using LDAP in the future and so won't need anything like the current programs.)

As a side note, moving our spares system over to a new environment has been an interesting experience, partly because getting it running initially was a pretty easy thing. But that's another entry.

Having your SSH server on an alternate port provides no extra security today

Every so often I either hear someone say that having your SSH server on a non-standard TCP port provides extra security or get asked whether it does. On today's Internet, the simple answer is no, it doesn't provide any extra security, or at least that it shouldn't. To explain that and convince you, let's talk about the two risks that your SSH server opens you up to. Let us call these the the scattershot risk and the targeted risk.

The scattershot risk is mass SSH password guessing. Pretty much any system on the Internet with an open SSH port will see a legion of compromised zombie machines show up to repeatedly try to guess username/password combinations, because why not; if you have a compromised machine and nothing better to use it for, you might as well turn it loose to see if you can get lucky. Of course SSH is not the only service that people will try mass password guessing against; attackers are also constantly probing against at least authenticated SMTP servers and IMAP servers. Probably they're trying anything that exposes password authentication to the Internet.

However, you should never be vulnerable to the broad risk because you shouldn't have accounts with weak passwords that can be guessed. Especially you shouldn't have such accounts exposed to SSH, because there are a number of ways to insure this. First, obviously you want to enforce password quality rules (whether just on yourself, for a personal machine, or on everyone). If you're worried about random accounts getting created by software that may mis-manage them and their passwords, you can lock down the SSH configuration so that only a few accounts can log in via SSH (you probably don't need that postgres system account to be able to SSH in, for example). Finally, you can always go all the way to turning off password authentication entirely and only accepting public keys; this will shut down all password guessing attacks completely, even if attackers know (or guess) a username that's allowed to log in remotely.

(SSH is actually a quite easy daemon to lock down this way, because it has good authentication options and it's relatively easy to configure restrictions on who can use it. Things like IMAP or authenticated SMTP are generally rather more troublesome because they have much weaker support for easily deployed access restrictions and they don't have public key authentication built in in the same way.)

The targeted risk is that someone will find a vulnerability in the (Open)SSH server software that you're running and then use it to attack you (and lots of other people who are also running that version). This could be the disaster scenario of full remote code execution, or an authentication bypass, or merely that they can work out your server's private key just by connecting to it (see also). This is an actual risk (even if it's probably relatively unlikely in practice), but on today's Internet, moving SSH to an alternate port doesn't mitigate it due to modern mass scanning and things like shodan and zmap. If your SSH is answering on some port, modern mass scanning will find it and in fact it's probably already in someone's database, complete with its connection banner and any other interesting details such as its host keys. In other words, all of the things that someone unleashing an exploit needs to know in order to target you.

You could get lucky, of course; the people developing a mass exploit could be lazy and just check against and target port 22, on the grounds that this will sweep up more than enough machines and they don't need to get absolutely everyone. But don't count on it, especially if the attackers are sophisticated and intend to be stealthy. And it's also possible that the Shodan-like search engine or the Internet scanning software that the attackers are using makes it easier for them to just ask for 'all SSH servers with this banner (regardless of port)' than to specifically limit things to port 22.

PS: The one thing that moving your SSH server off port 22 is good for is reducing log noise from all of the zombie machines trying scattershot password guessing attacks against you. My personal view is that there are better alternatives, including not paying attention to your logs about that kind of thing, but opinions differ here.

How and why we sell storage to people here

As a university department with a centralized fileserver environment plus a wide variety of professors and research groups, we have a space allocation problem. Namely, we need some way to answer the question of who gets how much space, especially in the face of uneven grant funding levels. Our historical and current answer is that we allocate space by selling it to people for a fixed one-time cost (for various reasons we can't give people free space). People can have as much space as they're willing to pay for; if they want so much space that we run out of currently available but not allocated space, we'll buy more hardware to meet the demand (and we'll be very happy about it, because we've historically had plenty of unsold space).

In the very old days that were mostly before my time here, our fileserver environment used Solaris DiskSuite on fixed-size partitions carved out from 'hardware RAID' FibreChannel controllers in a SAN setup. In this environment, one partition was one filesystem, and that was the unit we sold; if you wanted more storage space, my memory is that you had to put it in another filesystem whether you liked that or not, and obviously this meant that you had to effectively pre-allocate your space among your filesystems.

Our first generation ZFS fileserver environment followed this basic pattern but with some ZFS flexibility added on top. Our iSCSI backends exported standard-sized partitions as individual LUNs, which we called chunks, and some number of mirrored pairs of chunks were put together as a ZFS pool that belonged to one professor or group (which led to us needing many ZFS pools). We had to split up disks into multiple chunks partly because not doing so would have been far too wasteful; we started out with 750 GB Seagate disks and many professors or groups had bought less total space than that. We also wanted people to be able to buy more space without facing a very large bill, which meant that the chunk size had to be relatively modest (since we only sold whole chunks). We carried this basic chunk based space allocation model forward into our second generation of ZFS fileservers, which was part of why we had to do a major storage migration for this shift.

Then, well, we changed our minds, where I actually mean that our director worked out how to do things better. Rather than forcing people to buy an entire chunk's worth of space at once, we've moved to simply selling them space in 1 GB units; professors can buy 10 GB, 100 GB, 300 GB, 1000 GB, or whatever they need or want. ZFS pools are still put together from standard-sized chunks of storage, but that's now an implementation detail that only we care about; when you buy some amount of space, we make sure your pool had enough chunks to cover that space. We use ZFS quotas (on the root of each pool) to limit how much space in the pool can actually be used, which was actually something we'd done from the very beginning (our ZFS pool chunk size was much larger than our on FC SAN standard partition size, so some people got limited in the conversion).

This shift to selling in 1 GB units is now a few years old and has proven reasonably popular; we've had a decent number of people buy both small and large amounts of space, certainly more than were buying chunks before (possibly because the decisions are easier). I suspect that it's also easier to explain to people, and certainly it's clear what a professor gets for their money. My guess is that being able to buy very small amounts of space (eg 50 GB) to meet some immediate and clear need also helps.

(Professors and research groups that have special needs and their own grant funding can buy their own hardware and have their Point of Contact run it for them in their sandbox. There have been a fairly wide variety of such fileservers over the years.)

PS: There are some obvious issues with our general approach, but there are also equal or worse issues with other alternate approaches in our environment.

The hardware and basic setup for our third generation of ZFS fileservers

As I mentioned back in December, we are slowly working on the design and build out of our next (third) generation of ZFS NFS fileservers, to replace the current generation, which dates from 2014. Things have happened a little sooner than I was expecting us to manage, but the basic reason for that is we temporarily had some money. At this point we have actually bought all the hardware and more or less planned out the design of the new environment (assuming that nothing goes wrong on the software side), so today I'm going to run down the hardware and the basic setup.

After our quite positive experience with the hardware of our second generation fileservers, we have opted to go with more SuperMicro servers. Specifically we're using SuperMicro X11SPH-nCTF motherboards with Xeon Silver 4108 CPUs and 192 GB of RAM (our first test server has 128 GB for obscure reasons). This time around we're not using any addon cards, as the motherboard has just enough disk ports and some 10G-T Ethernet ports, which is all that we need.

(The X11SPH-nCTF has an odd mix of disk ports; 8x SAS on one PCI controller, 8x SATA on another PCIE controller, and an additional 2x SATA on a third. The two 8x setups use high-density connectors; the third 2x SATA has two individual ports.)

All of this goes in a 2U SuperMicro SC 213AC-R920LPB case, which gives us 16 hot swappable 2.5" front disk bays. This isn't quite enough disk bays for us, so we've augmented the case with what SuperMicro calls the CSE-M14TQC mobile rack; this goes in an otherwise empty space on the front and gives us an additional four 2.5" disk bays (only two of which we can wire up). We're using the 'mobile rack' disk bays for the system disks and the proper 16-bay 2.5" disk bays for data disks.

(Our old 3U SC 836BA-R920B cases have two extra 2.5" system disk bays on the back, so they didn't need the mobile rack hack.)

For disks, this time around we're going all SSD for the ZFS data disks, using 2 TB Crucial SSDs in a mix of MX300s and MX500s. We don't have any strong reason to go with Crucial SSDs other than that they're about the least expensive option that we trusted; we have a mix because we didn't buy all our SSDs at once and then Crucial replaced the MX300s with the MX500s. Each fileserver will be fully loaded with 16 data SSDs (and two system SSDs).

(We're not going to be using any sort of SAN, so moving up to 16 disks in a single fileserver is still giving us the smaller fileservers we want. Our current fileservers have access to 12 mirrored pairs of 2 TB disks; these third generation fileservers will have access to only 8 mirrored pairs.)

This time around I've lost track of how many of these servers we've bought. It's not as many as we have in our current generation of fileservers, though, because this time around we don't need three machines to provide those 12 mirrored pairs of disks (a fileserver and two iSCSI backends); instead we can provide them with one and a half machines.

Sidebar: On the KVM over IP on the X11SPH-nCTF

The current IPMI firmware that we have still has a Java based KVM over IP, but at least this generation works with the open source IcedTea Java I have on Fedora 27 (the past generation didn't). I've heard rumours that SuperMicro may have a HTML5 KVM over IP either in an updated firmware for these motherboards or in more recent motherboards, but so far I haven't found any solid evidence of that. It sure would be nice, though. Java is kind of long in the tooth here.

(Maybe there is a magic setting somewhere, or maybe the IPMI's little web server doesn't think my browser is HTML5 capable enough.)