Sometimes the way to solve a problem is to rethink the problem

November 20, 2014

After a certain amount of exploration and discussion, we've come up with what we feel is a solid solution for getting our NFS mount authentication working on Linux. Our solution is to not use Linux; instead we'll use OmniOS, where we already have a perfectly working NFS mount authentication system.

To get there we had to take a step back and look at our actual objectives and constraints. The reason we wanted our NFS mount authentication on Linux is that we want to offer a service where people give us disks (plus some money for overhead) and we put them into something and make them available via NFS and Samba and so on. The people involved very definitely want their disks pace available via NFS because they want their disk space to be conveniently usable (and fast) from various existing Linux compute machines and so on. We wanted to do this on Linux (as opposed to OmniOS (or FreeBSD)) because we trust Linux's disk drivers the most and in fact we already have Linux running happily on 16-bay and 24-bay SuperMicro chassis.

(I did some reading and experimentation with OmniOS management of LSI SAS devices and was not terribly enthused by it.)

We haven't changed our minds about using Linux instead of OmniOS to talk to the disks; we've just come to the blindingly obvious realization that we've already solved this problem and all it takes to reduce our current situation to our canned solution is adding a server running OmniOS in front of the Linux machine with the actual disks. Since we don't view this bulk disk hosting as an critical service and it doesn't need 10G Ethernet (even if that worked for us right now), this second server can be one of our standard inexpensive 1U servers that we have lying around (partly because we tend to buy in bulk when we have some money).

(Our first round implementation can even take advantage of existing hardware; since we're starting to decommission our old fileserver environment we have both spare servers and more importantly spare disk enclosures. These won't last forever, but they should last long enough to find out if there's enough interest in this service for us to buy 24-bay SuperMicro systems to be the disk hosts.)

This rethinking of the problem is not as cool and interesting as, say, writing a Go daemon to do efficient bulk authentication of machines and manage Linux iptables permissions to allow them NFS access, but it solves the problem and that's the important thing. And we wouldn't have come up with our solution if we'd stayed narrowly focused on the obvious problem in front of us, the problem of NFS mount authentication on Linux. Only when one of my coworkers stepped back and started from the underlying problem did we pivot to 'is there any reason we can't throw hardware at the problem?'.

There is a valuable lesson for me here. I just hope I remember it for the next time around.

Comments on this page:

By Twirrim at 2014-11-21 01:25:18:

I have a bad habit of getting bogged down in the nitty gritty details. My biggest motivation in life has always been solving problems. Give me a problem and I'll keep on worrying away at it, digging in until I have a satisfactory solution.

Unfortunately that means I can often fail to see the woods for the trees. I don't necessarily stop, look up and realise that really I'm wasting time.

The older I get, the more conscious I've become that I do this, but I find it hard to stop. I've been particularly noticing that over the last year or so. I have at least started getting better at reaching out to my manager or co-workers to review where I am, our deadlines and our goals. Hopefully some day I'll get good enough at it that I'll finally be able to do it without reaching out. I'm not holding my breath though!

Written on 20 November 2014.
« Finding free numbers in a range, crudely, with Unix tools
Lisp and data structures: one reason it hasn't attracted me »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Nov 20 00:27:04 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.