Wandering Thoughts archives


Early impressions of CentOS 7

For reasons involving us being unimpressed with Ubuntu 14.04, we're building our second generation iSCSI backends on top of CentOS 7 (basically because it just came out in time). We have recently put the first couple of them into production so now seems a good time to report my early impressions of CentOS 7.

I'll start with the installation, which has impressed me in two different ways. The first is that it does RAID setup the right way: you define filesystems (or swap areas), tell the installer that you want them to be RAID-1, and it magically figures everything out and does it right. The second is that it is the first installer I've ever used that can reliably and cleanly reinstall itself over an already-installed system (and it's even easy to tell it how to do this). You would think that this would be trivial, but I've seen any number of installers explode; a common failure point in Linux installers is assembling existing RAID arrays on the disks then failing to completely disassemble them before it tries to repartition the disks. CentOS 7 has no problems, which is something that I really appreciate.

(Some installers are so bad that one set of build instructions I wrote recently started out with 'if these disks have been used before, completely blank them out with dd beforehand using a live CD'.)

Some people will react badly to the installer being a graphical one and also perhaps somewhat confusing. I find it okay but I don't think it's perfect. It is kind of nice to be able to do steps in basically whatever order works for you instead of being forced into a linear order, but on the other hand it's possible to overlook some things.

After installation, everything has been trouble free so far. While I think CentOS 7 still uses NetworkManager it does it far better than how Red Hat Enterprise 6 did; in other words the networking works and I don't particularly notice that it's using NetworkManager behind the scenes. We can (and do) set things up in /etc/sysconfig/network-scripts in the traditional manner. CentOS 7 defaults to 'consistent network device naming' but unlike Ubuntu 14.04 it works and the names are generally sane. On our hardware we get Ethernet device names of enp1s0f0, enp1s0f1, and enp7s0; the first two are the onboard 10G-T ports and the third is the add-on 1G card. We can live with that.

(The specific naming scheme that CentOS 7 normally uses is described in the Red Hat documentation here, which I am sad to note needs JavaScript to really see anything.)

CentOS 7 uses systemd and has mostly converted things away from /etc/init.d startup scripts. Some people may have an explosive reaction to this shift but I don't; I've been using systemd on my Fedora systems for some time and I actually like it and think it's a pretty good init system (see also the second sidebar here). Everything seems to work in the usual systemd way and I didn't have any particular problems adding, eg, a serial getty. I did quite appreciate that systemd automatically activated a serial getty based on a serial console being configured in the kernel command line.

Overall I guess the good news is that I don't have anything much to say because stuff just works and I haven't run into any unpleasant surprises. The one thing that stands out is how nice the installer is.

linux/CentOS7EarlyImpressions written at 01:00:20; Add Comment


You want to turn console blanking off on your Linux servers

Let's start with the tweets:

@thatcks: Everyone should strongly consider adding 'consoleblank=0' to the kernel command line on your Linux servers. #sysadmin
@thatcks: The Linux kernel blanking the console screen is both unnecessary and dangerous on modern servers and modern setups. You want it off.

By default if you leave a Linux machine sitting idle at a text console, the kernel will blank the display after a while (I believe it's normally ten minutes of inactivity); Linux has probably done this since the very early days. Back in the time of CRT displays this made a reasonable amount of sense, because it avoided burning in the login prompt or whatever other static text was probably on the screen. Screen burnin is not really an issue in the modern age with LCDs, and it's even less of an issue with modern servers that spend a close approximation to all of their time without a display plugged in at all.

The problem with this console blanking is that it is a kernel function and thus the kernel has to reverse it. More specifically, the kernel has to be fairly alive and responding to the keyboard in order to unblank the screen. There are plenty of ways to get a kernel so hung that it is not alive enough to do this, at which point any helpful diagnostic messages the kernel may have printed on its way down are lost, locked away behind that blank screen. We have had this happen to us more than once.

And that is why you don't want your servers to ever blank their consoles; it's not getting you anything worthwhile and it can really hurt you. The best way to disable it is, as I tweeted, to add 'consoleblank=0' to the kernel command line arguments.

(Some people fiddle around with 'setterm -blank 0' in various ways but the kernel argument is more sure and easier.)

(I found out about 'consoleblank=0' and a bunch of additional useful information from this stackexchange question and its answers, when I finally decided to see if we could disable console blanking on our new iSCSI backends. I admit that my motivation for it was rather more petty than the reason here; a blank console can sometimes make their KVM-over-IP Java program freak out in a really irritating way and I was getting tired of that happening to my sessions.)

linux/ConsoleBlankingOff written at 23:02:38; Add Comment

Some notes on bisecting a modified Firefox source base with Mercurial

Suppose, not hypothetically, that you maintain your own copy of the Firefox master source (aka 'Nightly') with private modifications on top of the Mozilla version. Of course you don't commit your modifications, because that would lead to a huge tangle of merges over time. Now suppose that Mozilla breaks something and you want to use Mercurial bisection to find it.

The first thing you need is to figure out the last good version. What I do is I don't run my modified Firefox version directly out of the build directory; instead I periodically make an install tarball and unpack it elsewhere (and then keep the last few ones when I update it, so I can revert in case of problems). Among other things this tarball copy has an application.ini file, which for my builds includes a SourceStamp= value that gives the Mercurial commit identifier that the source was built from.

So we start the procedure by setting the range bounds:

hg bisect --good 606848e8adfc
hg bisect --bad tip

Since I'm carrying local modifications this will generally report something like:

Testing changeset 192456:d2e7bd70dd95 (1663 changesets remaining, ~10 tests)
abort: uncommitted changes

So now I need to explicitly check out the named changeset. If I skip this step Mercurial won't complain (and it will keep doing future 'hg bisect' operations without any extra complaints), but what I'm actually doing all of the time is building the tip of my repo. This is, as they say, not too useful. So:

hg checkout d2e7bd70dd95

This may print messages about merging changes in my changed files, which is expected. In general Mercurial is smart enough to get merging my changes in right unless something goes terribly wrong. Afterwards I build and test and do either 'hg bisect --good' or 'hg bisect --bad' followed by another 'hg checkout <ver>'.

(If I remember I can use the '-U' argument to 'hg bisect' so it doesn't attempt the checkout and abort with an error, but enhh. I actually think that having the error is handy because it reminds me that I need extra magic and care.)

In some cases even the 'hg checkout' may fail with the uncommitted changes error message. In this case I need to drop my changes and perhaps re-establish them later. The simple way is:

hg shelve
hg checkout ...

Perhaps I should routinely shelve all of my changes at the start of the bisection process, unless I think some of them are important for the testing I'm doing. It would cut down the hassle (and shelving them at the start would make it completely easy to reapply them at the end, since they'd be taken from tip and reapplied to tip).

After the whole bisection process is done, I need to cancel it and return to the tip of the tree:

hg bisect --reset
hg checkout tip
# if required:
hg unshelve
# optional but customary:
hg pull -u

(This is the sort of notes that I write for myself because it prevents me from having to reverse engineer all of this the next time around.)

Sidebar: Some related Mercurial bits I want to remember

The command to tell me what checkout I am on is 'hg summary' aka 'hg sum'. 'hg status' doesn't report this information; it's just for file status. This correctly reports that the current checkout hasn't changed when a 'hg bisect ...' command aborts due to uncommitted changes.

I don't think there's an easy command to report whether or not a bisection is in progress. The best way to check is probably:

hg log -r 'bisect(current)'

If there's no output, there's no bisection in flight.

(I believe I've left bisections sitting around in the past by omitting the 'hg bisect --reset'. If I'm right, things like 'hg pull -u' and so on won't warn me that theoretically there is a bisection running.)

programming/FirefoxBisectNotes written at 00:38:32; Add Comment


The core security problem of SSL on the web is too much trust

Every so often I'm surprised by what people don't already know (although I really shouldn't be). Today's surprise, due to Tim Bray's Blow up the Cert Business, is that people don't understand the core security problem with SSL on the web. I generally operate in an environment where this is common shared information, something that everyone just knows, but this is not the world in general.

So let's be explicit about what the real problem is:

The core security problem with SSL on the web is that your browser trusts a lot of Certificate Authorities and gives them power over any and all domains.

Your browser or operating system has a large list of built in CAs. Almost all of the time, SSL certificates signed by any of these authorities will be trusted to establish that you are talking to any random and arbitrary domain. There is no restriction on what domain names any of these CAs can issue valid certificates for and no mechanism that allows a domain to publish something that says 'no, no, wait, only trust SSL certificates for me that are signed by CA <X>'.

(Google has added special magic to Chrome to hard-code this for some domains, especially Google's. This has been extremely useful for turning up imposter SSL certificates, but it doesn't scale at all and thus it doesn't protect your site or my site.)

This means that the (global) security of your domain is hostage to the worst CAs and the CAs that are most subject to government coercion, because any of those CAs can issue certificates that web browsers will accept as valid for your domain. This is real SSL security (as opposed to the theoretical model with perfect CAs) and fundamentally why SSL certificates are a commodity. As the core weakness of SSL it has caused almost all of the SSL certificate security failures; some random CA that should not have any power to say 'I certify that this is website X' for some important X has in fact issued a certificate that says exactly that.

(The CA involved this time around simply makes it that much more obvious than usual, because it was not a general CA.)

This core problem of SSL cannot be fixed by getting better run CAs (or by imposing liability on CAs, never mind the problems with that). Extremely well run CAs are still vulnerable to government coercion and legal orders (orders that may come complete with gag requirements that prevent the CA from speaking out; we have seen that sort of thing in any number of places). For that matter, some CAs are themselves agencies of various governments (and your browser trusts them all and doesn't restrict what they can issue certificates for).

This problem cannot be solved by putting some or many commercial CAs out of business, as Tim Bray proposes. For a start you won't have gotten rid of the CAs that are effectively arms of their government. Beyond that, a root certificate that browsers trust is an extremely useful and valuable asset (perhaps a CA's most important one), one that will be transferred from owner to owner for as long as possible. I'm sure that there are any number of entities who would be happy to operate a CA at a loss merely to have power over such a certificate.

(Also, available evidence says that you can apparently operate a profitable CA that gives away free certificates. This suggests that putting commercial CAs out of business is going to be harder than you might expect.)

As far as I can tell the only way to solve this problem for real is for people to be able to somehow restrict which CAs can issue certificates for their web sites; an extreme version of this is for people to be able to restrict which actual certificates are accepted for their websites. Until this happens we will all remain hostage to all of those CAs that our browsers trust.

(I've written about these issues before, but I don't think I've previously laid out the core problem in black and white like this.)

PS: I hope it's obvious how much of a politically explosive non-starter it would be to try to drop perfectly operable CAs (either commercial or governmental) from browsers. As always, proposing something that cannot be done is not solving the real problem.

web/SSLCoreProblem written at 01:08:59; Add Comment


What the differences are between Python bools and ints

I mentioned in the previous entry that Python's bool class is actually a subclass of int (and the bool docstring will tell you this if you bother to read it with help() before, say, diving into the CPython source code like a system programmer). Since I was just looking at this, I might as well write down the low-level differences between ints and bools. Bools have:

  • a custom __repr__ that reports True or False instead of the numeric value; this is also used as the custom __str__ for bool.

    (The code is careful to intern these strings so that no matter how many times you repr() or str() a boolean, only one copy of the literal 'True' or 'False' string will exist.)

  • a __new__ that returns either the global True object or the global False object depending on the truth value of what it's given.

  • custom functions for &, |, and ^ that implement boolean algebra instead of the standard bitwise operations if both arguments are either True or False. Note that eg 'True & 1' results in a bitwise operation and an int object, even though 1 is strongly equal to True.

That's it.

I'm not quite sure how bool blocks being subclassed and I'm not curious enough right now to work it out.
Update: see the comments for the explanation.

The global True and False objects are of course distinct from what is in effect the global 0 and 1 objects that are all but certain to exist. This means that their id() is different (at least in CPython), since the id() is the memory address of their C-level object struct.

(In modern versions of both CPython 2 and CPython 3 it turns out that global 0 and 1 objects are guaranteed to exist, because 'small integers' between -5 and 257 are actually preallocated as the interpreter is initializing itself.)

python/BoolVsInt written at 00:27:09; Add Comment


Exploring a surprise with equality in Python

Taken from @hackedy's tweet, here's an interesting Python surprise:

>>> {1: "one", True: "two"}
{1: 'two'}
>>> {0: "one", False: "two"}
{0: 'two'}

There are two things happening here to create this surprise. The starting point is this:

>>> print hash(1), hash(True)
1 1

At one level, Python has made True have the same hash value as 1. Actually that's not quite right, so let me show you the real story:

>>> isinstance(True, int)

Python has literally made bool, the type that True and False are instances of, be a subclass of int. They not merely look like numbers, they are numbers. As numbers their hash identity is their literal value of 1 or 0, and of course they also compare equal to literal 1 or 0. Since they hash to the same identity and compare equal, we run into the issue with 'custom' equalities and hashes in dictionaries where Python considers the two different objects to be the same key and everything gets confusing.

(That True and False hash to the same thing as 1 and 0 is probably not a deliberate choice. The internal bool type doesn't have a custom hash function; it just tacitly inherits the hash function of its parent, ie int. I believe that Python could change this if it wanted to, which would make the surprise here go away.)

The other thing is what happens when you create a dictionary with literal syntax, which is that Python generates bytecode that stores each initial value into the dictionary one after another in the order that you wrote them. It turns out that when you do a redundant store into a dictionary (ie you store something for a key that already exists), Python only replaces the value, not both the key and the value. This is why the result is not '{True: 'two'}'; only the value got overwritten in the second store.

(This decision is a sensible one because it may avoid object churn and the overhead associated with it. If Python replaced the key as well it would at least be changing more reference counts on the key objects. And under normal circumstances you're never going to notice the difference unless you're going out of your way to look.)

PS: It turns out that @hackedy beat me to discovering that bools are ints. Also the class documentation for bool says this explicitly (and notes that bool is one of the rare Python classes that can't be subclassed).

python/EqualityDictSurprise written at 23:57:23; Add Comment

Some thoughts on SAN long-term storage migration

In theory one of the advantages having of a SAN instead of simple disk servers is relatively painless storage migration over the long term, where given a suitable setup and suitable software you can more or less transparently migrate data from old storage backends to new ones without any particular user-visible downtime. In practice I've come around to the idea that we may never be able to do this in our fileserver environment and that in general it takes a particular sort of environment to make it work.

Put simply, the disks you get today are generally going to be significantly different than the disks of four or five years ago, in both capacity and perhaps performance (if you go to SSDs). These differences may well cause you to want to reshape how your storage is laid out to do things like consolidate to fewer spindles (or to spread out to even more to preserve IOPs while growing space). So in order to have transparent storage migration you need not just a SAN but frontend storage software that can do this sort of potentially drastic rearrangement of storage. Replacing individual disks with bigger disks is something that almost every storage system can do, but that's the simple case. It's less common to have support for transformations like moving from two smaller disks to one bigger disk or moving from a few big disks to more smaller disks (as might happen if you went from HDs to SSDs).

Put another way, you often want to design different storage setups today than four or five years ago even if you keep the same top level technology (eg ZFS). Given this, a transparent migration either requires some way to transmogrify the storage setups from five years ago into the storage setups of today or just living with a five year old setup (which is often less than desirable).

While our hand was forced by ZFS this time around, this is certainly one thing that probably would have biased us towards a non-transparent migration anyways. Moving from 750 GB and 1TB disks to 2TB disks caused us to roughly double our standard chunk size, but politics mean we can't give people double the space for free and anyways some people have space split across multiple ZFS pools (on different fileservers) that we'd like to consolidate into one. Doing the necessary reshaping transparently would take features that ZFS just doesn't have. I suspect that we'll run into similar issues in four or so years when we next migrate to another set of hardware and disks; the disks of four or five years from now are hopefully going to be significantly different from today.

Our migration from our previous generation DiskSuite based fileservers to our ZFS ones was obviously forced to be non-transparent because we were moving to ZFS, but in retrospect it also had the same issue; we moved from 35 GB 'chunks' to much larger ones and that would have again requires support for major storage reshaping even if we'd stuck with DiskSuite.

(And in general we've been lucky that ZFS has remained a viable option for us across this many years. If Sun had not released ZFS and Solaris as open source we'd probably be migrating away from ZFS now, just as we migrated away from DiskSuite last time. This ties into the arrogance of trying to plan for long term storage management in the face of all of the uncertainty in the technology world.)

sysadmin/SANStorageMigration written at 00:04:51; Add Comment


Goroutines versus other concurrency handling options in Go

Go makes using goroutines and channels very attractive; they're consciously put forward as the language's primary way of doing concurrency and thus the default solution to any concurrency related issue you may have. However I'm not sure that they're the right approach for everything I've run into, although I'm still mulling over what the balance is.

The sort of problem that channels and goroutines don't seem an entirely smooth fit for is querying shared state (or otherwise getting something from it). Suppose that you're keeping track of the set of SMTP client IPs that have tried to start TLS with you but have failed; if a client has failed TLS setup, you don't want to offer it TLS again (or at least not within a given time). Most of the channel-based solution is straightforward; you have a master goroutine that maintains the set of IPs privately and you add IPs to it by sending a message down the channel to the master. But how do you ask the master goroutine if an IP is in the set? The problem is that you can't get a reply from the master on a common shared channel because there is no way for the master to reply specifically to you.

The channel based solution for this that I've seen is to send a reply channel as part of your query to the master (which is sent over a shared query channel). The downside of this approach is the churn in channels; every request allocates, initializes, uses once, and then destroys a channel (and I think they have to be garbage collected, instead of being stack allocated and quietly cleaned up). The other option is to have a shared data structure that is explicitly protected by locks or other facilities from the sync package. This is more low level and requires more bookkeeping but you avoid bouncing channels around.

But efficiency is probably not the right concern for most Go programs I'll ever write. The real question is which is easier to write and results in clearer code. I don't have a full conclusion but I do have a tentative one, and it's not entirely the one I expected: locks are easier if I'm dealing with more than one sort of query against the same shared state.

The problem with the channel approach in the face of multiple sorts of queries is that it requires a lot of what I'll call type bureaucracy. Because channels are typed, each different sort of reply needs a type (explicit or implicit) to define what is sent down the reply channel. Then basically each different query also needs its own type, because queries must contain their (typed) reply channel. A lock based implementation doesn't make these types disappear but it makes them less of a pain because they are just function arguments and return values and thus they don't have to be formally defined out as Go types and/or structs. In practice this winds up feeling more lightweight to me, even with the need to do explicit manual locking.

(You can reduce the number of types needed in the channel case by merging them together in various ways but then you start losing type safety, especially compile time type safety. I like compile time type safety in Go because it's a reliable way of telling me if I got something obvious wrong and it helps speed up refactoring.)

In a way I think that channels and goroutines can be a form of Turing tarpit, in that they can be used to solve all of your problems if you're sufficiently clever and it's very tempting to work out how to be that clever.

(On the other hand sometimes channels are a brilliant solution to a problem that might look like it had nothing to do with them. Before I saw that presentation I would never have thought of using goroutines and channels in a lexer.)

Sidebar: the Go locking pattern I've adopted

This isn't original to me; I believe I got it from the Go blog entry on Go maps in action. Presented in illustrated form:

// actual entries in our shared data structure
type ipEnt struct {
  when  time.time
  count int

// the shared data structure and the lock
// protecting it, all wrapped up in one thing.
type ipMap struct {
  ips map[string]*ipEnt

var notls = &ipMap{ips: make(map[string]*ipEnt)}

// only method functions manipulate the shared
// data structure and they always take and release
// the lock. outside callers are oblivious to the
// actual implementation.
func (i *ipMap) Add(ip string) {
  ... manipulate i.ips ...

Using method functions feels the most natural way to manipulate the data structure, partly because how you manipulate it is very tightly bound to what it is due to locking requirements. And I just plain like the syntax for doing things with it:

if res == TLSERROR {

The last bit is a personal thing, of course. Some people will prefer standalone functions that are passed the ipMap as an explicit argument.

programming/GoGoroutinesVsLocks written at 22:51:07; Add Comment

The problem with filenames in IO exceptions and errors

These days a common pattern in many languages is to have errors or error exceptions be basically strings. They may not literally be strings but often the only thing people really do with them is print or otherwise report their string form. Python and Go are both examples of this pattern. In such languages it's relatively common for the standard library to helpfully embed the name of the file that you're operating on in the error message for operating system IO errors. For example, the literal text of the errors and exceptions you get for trying to open a file that you don't have access to in Go and Python are:

open /etc/shadow: permission denied
[Errno 13] Permission denied: '/etc/shadow'

This sounds like an attractive feature, but there is a problem with it: unless the standard library does it all the time and documents it, people can't count on it, and when they can't count on it you wind up with ugly error messages in practice unless people go quite out of their way.

This stems from one of the fundamental rules of good (Unix) error messages for programs, which is thou shalt always include the name of the file you had problems with. If you're writing a program and you need to produce an error message, it is ultimately your job to make sure that the filename is always there. If the standard library gives you errors that sometimes but not always include the filename, or that are not officially documented as including the filename, you have no real choice but to include the filename yourself. Then when the standard library's error or exception does include the filename, the whole error message emitted by your program winds up mentioning the filename twice:

sinksmtp: cannot open rules file /not/there: open /not/there: no such file or directory

It's tempting to say that the standard library should always include the filename in error messages (and explicitly guarantee this). Unfortunately this is very hard to do in general, at least on Unix and with a truly capable standard library. The problem is that you can be handed file descriptors from the outside world and required to turn them into standard file objects that you can do ordinary file operations on, and of course there is no (portable) way to find out the file name (if any) of these file descriptors.

(Many Unixes provide non-portable ways of doing this, sometimes brute force ones; on Linux, for example, one approach is to look at /proc/self/fd/<N>.)

programming/FilenamesInErrors written at 00:37:23; Add Comment


Another reason to use frameworks like Django

The traditional reason to use web app frameworks like Django is that doing saves you time and perhaps gives you a more solid and polished result, possibly with useful extra features like Django's admin interface. But it has recently struck me that in many situations there is another interesting reason for using frameworks (or a defence of doing so instead of writing your own code).

Let's start by assuming that your application really needs at least some of the functionality you're using from the framework. For example, perhaps you're using the ORM and database functionality because that's what the framework makes easiest (this is our reason) but you really need the URL routing and HTML form handling and validation. Regardless of whether or not you used a framework, your application needs some code somewhere that does all of this necessary work. With a framework, the code mostly lives in the framework and you call the framework; without a framework, you would have to write your own code for it (and you use it directly). The practical reality is that the code for the functionality your application genuinely needs has to come from somewhere, either from a framework (if you use one) or from your own work and code.

If you write your own code, what are the odds that it will be as well documented and as solid as the code in a framework? Which will likely be easier for a co-worker to pick up later, custom code that you wrote from scratch or code that calls a standard framework in a standard or relatively standard way? If you only need a little bit of functionality and thus only need to write a little bit of code, this can certainly work out. But if you need a lot of functionality, so much that you're duplicating a lot of what a framework does, well, I am not so optimistic, because in effect what you're really doing is creating a custom one-off framework.

This suggests an obvious way to balance out whether or not to use a framework (or from some perspectives, to inflict either a framework or your own collection of code on your co-workers). To maximize the benefits of using a framework you should be writing as little of your own code as possible, talking to the framework in its standard way, and the framework needs to be well documented, because all of this plays to the strengths of the framework over your own code. If the framework is hard to pick up, your code to deal with it is complex, and replacing it would only be a modest amount of custom code, well, the case for your own code is strong.

(I'm not sure this way of thinking has anything to say about the ever popular arguments over minimal frameworks versus big frameworks with 'batteries included' and good PR. A big framework might be worse because it requires you to learn more before you can start using the corner of it you need, or it might be better because you need less custom code to connect various minimal components together. It certainly feels like how much of the framework you need ought to matter, but I'm not sure this intuition is correct.)

python/FrameworkUsageReason written at 01:31:39; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.