Wandering Thoughts archives


For me Chrome clearly wins over Firefox on Javascript-heavy websites

For my sins, I periodically use a number of Javascript-heavy websites (some of them are sites where the Javascript is core to the experience). I have similar Javascript enabled environments in both Chrome (in incognito mode) and Firefox, with relatively similar sets of extensions (Firefox version and Chrome version). My consistent experience over the past while is that Chrome is clearly faster on these sites than Firefox, and in particular Firefox often feels actively laggy.

(Years ago I sort of had the same experience on Flickr, but I believe that more or less went away later for a while. It returned a couple of years ago, and I quietly switched from using Firefox on Flickr to mostly using Chrome.)

This result sort of surprises and depresses me (partly because using Chrome has its pains). My understanding is that in theory Firefox and Chrome are usually relatively neck and neck as far as performance goes, with Firefox at least competitive, and that especially on common sites Firefox should not be laggy. There are a number of things that could be causing this for me and not for other people, especially general users. For a start I'm on Linux and using Fedora's build of Firefox instead of the Mozilla build, while I think most performance comparisons are made on Windows or MacOS and use the official Mozilla builds.

(I'm also using a relatively odd Linux environment with relatively modest OpenGL and compositing support, which might hurt Firefox more than Chrome.)

Beyond that, possibly my core Firefox extensions are slowing down Firefox more than I expect. But if so, well, they're my core extensions for a good reason (and the obvious suspect of NoScript is not entirely the cause, since some of my Firefox usage is without it). What matters to me is the performance of the browsers I've created from Firefox and Chrome, not the performance of the stock versions in some platonic ideal state that I will never use them in. Given that I have one decently performing browser, that's what I'll wind up using for Javascript-heavy sites even if it's not Firefox.

(And I'm using some extensions in Chrome's incognito mode that I would expect to be sort of heavyweight, like uBlock Origin and a mouse gestures extension.)

PS: I care about this partly because I dislike some things Google does with Chrome and partly because I care about Firefox being competitive and good in general. The overall web ecology benefits when we have a real choice in browsers, and part of having a real choice is good performance.

(I also think that Mozilla fundamentally cares more about Linux for Firefox than Google does for Chrome. As a non-Windows, non-Mac user, I remember the days when I was a second class citizen on the web and I would rather like to not go too far back to them.)

web/LinuxChromeFasterJavascript written at 00:05:29; Add Comment


Shrinking the partitions of a software RAID-1 swap partition

A few days ago I optimistically talked about my plans for a disk shuffle on my office workstation, by replacing my current 1 TB pair of drives (one of which had failed) with a 1.5 TB pair. Unfortunately when I started putting things into action this morning, one of the 1.5 TB drives failed immediately. We don't have any more spare 1.5 TB drives (at least none that I trust), but we did have what I believe is a trustworthy 1 TB drive, so I pressed that into service and changed my plans around to be somewhat less ambitious and more lazy. Rather than make a whole new set of RAID arrays on the new disks (and go through the effort of adding them to /etc/mdadm.conf and so on), I opted to just move most of the existing RAID arrays over to the new drives by attaching and detaching mirrors.

This presented a little bit of a problem for my mirrored swap partition, which I wanted to shrink from 4 GB to 1 GB. Fortunately it turns out that it's actually possible to shrink a software RAID-1 array these days. After some research, my process went like this:

  • Create the new 1 GB partitions for swap on the new disks as part of partitioning them. We can't directly add these to the existing swap array, /dev/md14, because they're too small.

  • Stop using the swap partition because we're about to drop 3/4ths of it. This is just 'swapoff -a'.

  • Shrink the amount of space to use on each drive of the RAID-1 array down to an amount of space that's smaller than the new partitions:

    mdadm --grow -z 960M /dev/md14

    I first tried using -Z (aka --array-size) to shrink the array size non-persistently, but mdadm still rejected adding a too-small new array component. I suppose I can't blame it.

  • Add in the new 1 GB partitions and pull out the old 4 GB partition:

    mdadm --add /dev/md14 /dev/sdc3
    # (wait for the resync to finish)
    mdadm --add /dev/md14 /dev/sdd3
    mdadm --repace /dev/md14 /dev/sde4
    # (wait for the resync to finish)
    madam -r /dev/md14 /dev/sde4

  • Tell software RAID to use all of the space on the new partitions:

    mdadm --grow -z max /dev/md14

At this point I almost just swapon'd the newly resized swap partition. Then it occurred to me that it probably still had a swap label that claimed it was a 4 GB swap area, and the kernel would probably be a little bit unhappy with me if I didn't fix that with 'mkswap /dev/md14'. Indeed mkswap reported that it was replacing an old swap label with a new one.

My understanding is that the same broad approach can be used to shift a software RAID-1 array for a filesystem to smaller partitions as well. For a filesystem that you want to keep intact, you first need to shrink the filesystem safely below the size you'll shrink the RAID array to, then at the end grow the filesystem back up. All things considered I hope that I never have to shrink or reshape the RAID array for a live filesystem this way; there are just too many places where I could blow my foot off.

(Life is easier if the filesystem is expendable and you're going to mkfs a new one on top of it later.)

You might ask why it's worth going through all of this instead of just making a new software RAID-1 array. That's a good question, and for me it comes down to how much of a pain it often is to set up a new array. These days I prefer to change /etc/mdadm.conf, /etc/fstab and so on as little as possible, which means that I really want to preserve the name and MD UUID of existing arrays when feasible instead of starting over from scratch.

This is also where I have an awkward admission: for some reason, I thought that you couldn't use 'mdadm --detail --scan' on a single RAID array, to conveniently generate the new line you need for mdadm.conf when you create a new array. This is wrong; you definitely can, so you can just do things like 'mdadm --detail --scan /dev/mdNN >>/etc/mdadm.conf' to set it up. Of course you may then have to regenerate your initramfs in order to make life happy.

(I hope I never need to do this sort of thing again, but if I do I want to have some notes about it. Sadly someday we may need to use a smaller replacement disk in a software RAID mirror in an emergency situation and I may get to call on this experience.)

linux/ShrinkingSoftwareRAIDSwap written at 23:18:30; Add Comment

Link: Introduction to Certificate Transparency for Server Operators

Alex Gaynor's Introduction to Certificate Transparency for Server Operators (via) is what it says in the title, and taught me some things about certificate transparency in practice. Sadly, one of the things it taught me is that once again Lighttpd seems to be coming up short as far as modern TLS goes. I really should switch over my personal site to using Apache, even if it will kind of be a pain because Fedora fumbled good Apache configuration.

(I also hadn't known about Cert Spotter, which has the advantage that you don't need a Facebook login to use it and thus don't have to helpfully (for Facebook) tie one or more domains to your Facebook login. All you need is an email address and on the modern Internet, you already need a source of individualized disposable ones.)

links/CertTransForServerOps written at 13:50:48; Add Comment

What I mostly care about for speeding up our Python programs

There are any number of efforts and technologies around these days that try to speed up Python, starting with the obvious PyPy and going on to things like Cython and grumpy. Every so often I think about trying to apply one of them to the Python code I deal with, and after doing this a few times (and even making some old experiments with PyPy) I've come to a view of what's important to me in this area.

(This has come to be more and more on my thoughts because these days we run at least one Python program for every incoming email from the outside world. Sometimes we run more than that.)

What I've come around to caring about most is reducing the overall resource usage of short running programs that mostly use the Python standard library and additional pure-Python modules. By 'resource usage' I mean a combination of both CPU usage and memory usage; in our situation it's not exactly great if I make a program run twice as fast but use four times as much memory. In fact for some programs I probably care more about memory usage than CPU, because in practice our Python-based milter system probably spends most of its time waiting for our commercial anti-spam system to digest the email message and give it a verdict.

(Meanwhile, our attachment logger program is probably very close to being CPU bound. Yes, it has to read things off disk, but in most cases those files have just been written to disk so they're going to be in the OS's disk cache.)

I'm also interested in making DWiki (the code behind Wandering Thoughts) faster, but again I actually want it to be less resource-intensive on the systems it runs on, which includes its memory usage too. And while DWiki can run in a somewhat long-running mode, most of the time it runs as a short-lived CGI that just serves a single request. DWiki's long-running daemon mode also has some features that might make it play badly with PyPy, for example that it's a preforking network server and thus that PyPy is probably going to wind up doing a lot of duplicate JIT translation.

I think that all of this biases me towards up-front approaches like Cython and grumpy over on the fly ones such as PyPy. Up-front translation is probably going to work better for short running programs (partly because I pay the translation overhead only once, and in advance), and the results are at least reasonably testable; I can build a translated version and see in advance whether the result is probably worth it. I think this is a pity because PyPy is likely to be both the easiest to use and the most powerful accelerator, but it's not really aimed at my usage case.

(PyPy's choice here is perfectly sensible; bigger, long-running programs that are actively CPU intensive for significant periods of time are where there's the most payoff for speeding things up.)

PS: With all of this said, if I was serious here I would build the latest version of PyPy by hand and actually test it. My last look and the views I formed back then were enough years ago that I'm sure PyPy has changed significantly since then.

python/FasterPythonInterests written at 02:05:16; Add Comment


Migrating a separate /boot filesystem into the root filesystem

In my notes on moving a software RAID-1 root filesystem around, I mentioned that I still had a separate /boot partition on my home and office machines and I wanted to merge it into the root filesystem at some point as I no longer believe in a separate /boot under most circumstances. Having now done this both in a test Fedora 25 virtual machine and on my office workstation, it's time to write up the procedure. For a slight extra complexity, my /boot and root filesystems are both in Linux software RAID-1 arrays.

First, make sure you're actually booting off your root drives, which may require BIOS fiddling. I probably still have to do this at home (on my new SSDs), but it was already done on my office machine a while back. Then my process is:

  • Make the directory that will be the new mount point of your old separate /boot filesystem. I use /boot2.
  • change /etc/fstab to have your separate /boot filesystem mounted there.
  • unmount /boot and remount it on /boot2. With fstab updated, this is just 'umount /boot; mount /boot2'.

  • copy what is now /boot2 to /boot. I reflexively do this with dump:
    cd /boot && dump -0f - /boot2 | restore -xf -

    (We don't need to remake /boot, because unmounting the filesystem from it leaves behind an empty directory that previously was the mountpoint.)

  • You need to know the UUIDs of the root filesystem and your root software RAID device. You can get these from /etc/fstab and /etc/mdadm.conf, or you can use commands such as 'lsblk --fs /dev/mdNN' and 'mdadm -D /dev/mdNN'.

    The software RAID UUID will have :'s in it, for example '35d6ec50:bd4d1f53:7401540f:6f971527'. You're going to need a version of the UUID without them, ie just one long string of hex digits.

  • Also find the filesystem UUID of what is now /boot2 and the software RAID UUID of its array. One convenient way to do this is to look in your current /boot/grub2/grub.cfg for lines of the form:

    set root='mduuid/05a2f6e13830b3a102f7636b98d651f3'


    search --no-floppy --fs-uuid --set=root 91aed8b1-c145-4673-8ece-119e19f7038f

    Those are the MD UUID and the filesystem UUID respectively. As you may have guessed, you need a version of the MD UUID without colons, because that's what Grub2 wants.

  • Now you need to edit /boot/grub2/grub.cfg to make two sets of changes. The first one is to change the old /boot2 MD UUID and filesystem UUID to the MD UUID and filesystem UUID of your root filesystem, so that Grub2 is looking in the right filesystem for everything. This should be easy with something like_vim_ search and replace, but remember to make sure your command changes all occurrences on a line, not just the first one (eg, vim's g modifier on the s command).

    (Note that the MD UUIDs used in grub.cfg must be the version without :'s in it. Grub2 probably won't like you too much if you use the other form, although I haven't tested it. Yes, it is a bit annoying that Grub wants a different form of MD UUID than the tools normally produce.)

    However, this isn't sufficient by itself, because grub.cfg contains filesystem-relative paths for the Linux kernel and its initial ramdisks. In your old /boot2 these were in the root of the filesystem, but now they are /boot/vmlinuz-*, because /boot is now a filesystem subdirectory. So you need to edit every set of lines that are like:

    linux  /vmlinuz-[...]
    initrd /initramfs-[...]

    Change them to add a /boot at the front, like so:

    linux  /boot/vmlinuz-[...]
    initrd /boot/initramfs-[...]

    Some versions of grub.cfg may use 'linux16' and 'initrd16' as the directives instead.

  • Finally, run grub2-install to reinstall the Grub boot blocks onto your boot drives, eg 'grub2-install /dev/sda'. This updates the embedded Grub so it knows about your new location for /boot; if you skip this step, your Grub boot blocks will probably quietly keep on using what is now /boot2.

To test that this is working, reboot your machine and go into the Grub menu during boot. When you look at the text for a specific boot entry, you should see that it's using 'linux /boot/...' and so on; the presence of the /boot bit is your proof that Grub is actually reading your updated /boot/grub2/grub.cfg instead of the old, un-updated grub.cfg in /boot2.

(As a final test you can erase everything in /boot2 and then fill it up with data from /dev/zero or /dev/urandom to overwrite all of the disk blocks. If the system still boots happily, you are definitely not using anything from the now thoroughly destroyed /boot2. I feel that going this far is only appropriate in a test environment; I did it with my Fedora 25 test virtual machine, but not on my office workstation.)

At some point in the future when you decommission the /boot2 filesystem, you will want to edit the kernel command lines in /boot/grub2/grub.cfg to take out the 'rd.md.uuid=<...>' argument that is telling dracut to try to bring up that software RAID array.

(You might wonder why we didn't have to add or change a rd.md.uuid kernel command line argument for the /boot migration. The answer is that since /boot is now on the root filesystem, the kernel's command line already has a rd.md.uuid argument to bring it up on boot. Or at least it should.)

Sidebar: Why I haven't done this to my home machine yet

One version of why is that rebooting my home machine usually feels like more of a pain than rebooting my office machine, because I'm usually doing something with my home machine that I don't want to interrupt. So I drag my heels on anything and everything that can call for a reboot, such as Fedora kernel upgrades.

A more complete version is that not only do I have to reboot my machine, but I should really also open it up to take out my two old 500 GB system HDs, which won't really be in use after this shuffle (on my home machine, I put my small swap partition on my SSDs). And the reason I want to do this is that right now my pair of SSDs aren't actually mounted in drive bays, because I don't have enough. Instead they're just resting loose in a 5.25" drive bay, and the problem with that is the 5.25" drive bays get clearly less air circulation than the 3.5" drive bays.

I should really just do all of this shuffling, but I may have recently mentioned that I'm lazy. I really do want to do it soon, though. Hopefully having written this will motivate me to do it tomorrow.

linux/MigratingBootFSIntoRootFS written at 03:06:09; Add Comment


Planning out a disk shuffle for my office workstation

As I mentioned yesterday, my office workstation lost one of its remaining HDs the other day. This HD is one side of a pair of 1 TB drives that's used for various mirrored partitions, so I haven't lost anything (unless the other side also fails before Monday, so let's hope not), but now I need to replace it with one of the ready spares I have sitting around for exactly this eventuality.

The obvious way to replace the failed disk is to do literally that; put in the new disk, partition it up to basically match the existing setup, and attach appropriate partitions to software RAID devices and my ZFS pool. That would certainly work, but it's not necessarily the right answer because my office workstation's current disk setup has mutated over time and the actual current setup of much of it is not what I would build from scratch.

Specifically, right now I have five drives with the following odd split up:

  • two paired SSDs, used for / and a SSD ZFS pool where my home directory and other important things live. (That my home directory and so on is on the SSDs is one reason I am not too worried about the current disk failure.)

  • two paired 1 TB HDs that used to be my only two drives. Because of that they still have partitions for my old HD-based root filesystem (in two copies), the /boot partition I just recently migrated into the SSD /, and finally my HD-based ZFS pool, which used to hold everything but now mostly holds less important data. Oh, and it turns out they also have my swap partition.

  • One 500 GB HD that I used as overflow for unimportant virtual machines, back in the days when I thought I needed to conserve space on the mirrored 1 TB HDs (in fact it may date from the days when these were actually 750 GB HDs).

My replacements for the 1 TB drives are 1.5 TB, so this already gives me a space expansion, and there's now any number of things on those two 1 TB drives I don't need any more, and also one thing I would like to have. So my current plan is to replace both 1 TB drives (the dead and the still alive one) and set up the space on the new pair of 1.5 TB drives as follows:

  • a small 1 GB swap partition, because it still seems like a good idea to give the Linux kernel some swap space to make it happy.
  • a 200 GB or so ext4 partition, which will be used for an assortment of things that aren't important enough to go in the SSD / but that I don't want to have in ZFS, partly because they're things I may need to get back access to my ZFS pools if package upgrades go wrong.

  • a 100 GB backup / partition. As is my current habit, before major events like Fedora upgrades I'll copy the SSD / to here so that I can kludge together a reversion to a good environment if something goes badly wrong.

  • all the remaining space goes to my HD-based ZFS pool, which should expand the space a decent amount.

(All partitions and ZFS and so on will be mirrored between the two drives.)

Given the likely space expansion in my HD-based ZFS pool, I think I'll also get rid of that overflow 500 GB HD by folding its 120 GB or so of actual space usage into the main HD-based ZFS pool. It was always sort of a hack and a bit awkward (on top of having no redundancy). Plus this will simplify everything, and I can always put the drive (or a bigger replacement drive) back in and redo this if I turn out to actually need a bunch more space for virtual machines.

In a way, I'm grateful that my 1 TB HD finally died and also that it happened under the circumstances it did, where I couldn't immediately rush into replacing it in the most obvious way possible and had forced time to sit back and think about whether the obvious path was the right one. I'm probably going to wind up with a nicer, more sensibly set up system as a result of this disk failure, and I probably never would have done this rearrangement without being pushed.

linux/OfficeWorkstationDiskShuffle written at 22:53:11; Add Comment

Sometimes laziness doesn't pay off

My office workstation has been throwing out complaints about some of its disks for some time, which I have been quietly clearing up rather than replace the disks. This isn't because these are generally good disks; in fact they're some Seagate 1TB drives which we haven't had the best of luck with. I was just some combination of too lazy to tear my office machine apart to replace a drive and too parsimonious with hardware to replace a disk drive before it failed.

(Working in a place with essentially no hardware budget is a great way to pick up this reflex of hardware parsimony. Note that I do not necessarily claim that it's a good reflex, and in some ways it's a quite inefficient one.)

Recently things got a bit more extreme, when one disk went from nothing to suddenly reporting 92 new 'Offline uncorrectable sector' errors (along with 'Currently unreadable (pending) sectors', which seems to travel with offline uncorrectable sectors). I looked at this, thought that maybe this was a sign that I should replace the disk, but then decided to be lazy; rather than go through the hassle of a disk replacement, I cleared all the errors in the usual way. Sure, the disk was probably going to fail, but it's in a mirror and when it actually did fail I could swap it out right away.

(I actually have a pair of disks sitting on my desk just waiting to be swapped in in place of the current pair. I think I've had them set aside for this for about a year.)

Well, talking about that idea, let's go to Twitter:

@thatcks: I guess I really should have just replaced that drive in my office workstation when it reported 92 Offline uncorrectable sectors.

@thatcks: At 5:15pm, I'm just going to hope that the other side of the mirrored pair survives until Monday. (And insure I have pretty full backups.)

Yeah, I had kind of been assuming that the disk would fail at some convenient time, like during a workday when I wasn't doing anything important. There are probably worse times for my drive to fail than right in front of me at 5:15 pm immediately before a long weekend, especially when I have a bike ride that evening that I want to go to, but I can't think of many that are more annoying.

(The annoyance is in knowing that I could swap the drive on the spot, if I was willing to miss the bike ride. I picked the bike ride, and a long weekend is just short enough that I'm not going to come in in the middle of it to swap the drive.)

I have somewhat of a habit of being lazy about this sort of thing. Usually I get away with it, which of course only encourages me to keep on being lazy and do it more. Then some day things blow up in my face, because laziness doesn't always pay off. I need to be better about getting myself to do those annoying tasks sooner or later rather than putting them off until I have no choice about it.

(At the same time strategic laziness is important, so important that it can be called 'prioritization'. You usually can't possibly give everything complete attention, time, and resources, so you need to know when to cut some nominal corners. This usually shows up in security, because there are usually an infinite number of things that you could be doing to make your environment just a bit more secure. You have to stop somewhere.)

sysadmin/LazinessSometimesBackfires written at 00:40:00; Add Comment


On today's web, a local Certificate Authority is fairly dangerous

In a comment on my entry on generating self-signed TLS certificates today, James suggested:

My go-to tool is OpenVPN's EasyRSA. Admittedly that creates a CA which you can then sign certificates with, but for your internal hosts it would mean you could install said CA into your browser and then trust them all.

Superficially, this is certainly an appealing idea. If you have a fleet of IPMIs or other internal websites that need TLS certificates and that have names where you can't get public certificates, you can avoid everyone having to trust them one by one. Just set up a local CA, sign all the internal website certificates with them, add the local CA certificate to your browser, and you're done.

Unfortunately if you do this, you have just loaded a fairly large security-defeating gun and pointed it straight at your face. It's not just that your local CA can be attacked to sign certificate for any host, not just your internal ones; more importantly, certificates signed by a manually added CA specifically bypass all of the modern TLS protections built into browsers. This isn't just things like HTTP Public Key Pinning headers that your browser may have memorized, it's also even critically important pinned keys hard-coded into browsers themselves. A certificate signed by a manually added CA bypasses all of those checks.

(For all of this we may blame HTTPS interception middleware. Browser vendors have extremely reluctantly bowed to the demands of businesses that want to deploy them and have them intercept absolutely everything, partly because businesses basically hold the cards here if they're willing to go far enough.)

As far as I know there's no way in either Firefox or Chrome to constrain a manually added CA to only have its certificates accepted for certain (sub)domains. This means that no matter what you want, your local CA intended for intranet websites has just as much TLS interception ability as the TLS CA for a mandatory HTTPS middleware box. If an attacker can compromise it, they gain complete HTTPS interception capabilities for web browsing, both internal and external. None of the usual precautions and warnings will protect you in the least.

This means that a local CA that you have people's browsers trust is a very big deal, even (or especially) if only the sysadmins are trusting it. If you're going to have one at all, I think that it should involve some sort of hardware security module, even a simple and cheap one. If you are not willing to strongly protect a local CA, at least to the level of buying basic HSM hardware for it, then you should not even think of having one; it's simply far too dangerous in the event of a serious attacker. Even if you buy HSM hardware for it, I think that the balance of risks versus gains are often not going to be in favour of a local CA.

(To be clear, all of this is specific to local CAs that you will have your browsers trust. There are perfectly sensible and not particularly dangerous uses for a local CA outside of this. The general way to know if you're safe is that every operation that is supposed to use the local CA should have to explicitly trust the local CA's root certificate, whether that's through a command-line option or a specific configuration file setting. You should never add a local CA to your general trust roots, whether those are the browser trust roots or the system's generic trust roots.)

(Years ago I sort of wrote about this here, but I didn't take it anywhere near far enough and in particular I didn't think of what an attacker could do with access to your local or organizational CA. Not that overzealous security people aren't a serious risk in and of themselves, and it's not as if middleware HTTPS interception has a good reputation. Rather the contrary.)

web/LocalCAQuiteDangerous written at 21:35:59; Add Comment

Generating good modern self-signed TLS certificates in today's world

Once upon a time, generating decently good self-signed certificates for a host with OpenSSL was reasonably straightforward, especially if you didn't know about some relevant nominal standards. The certificate's Subject name field is a standard field with standard components, so OpenSSL would prompt you for all of them, including the Common Name (CN) that you'd put the hostname in. Then things changed and in modern TLS, you really want to put the hostname in the Subject Alternative Name field. SubjectAltName is an extension, and because it's an extension 'openssl req' will not prompt you to fill it in.

(The other thing is that you need to remember to specify -sha256 as one of the arguments; otherwise 'openssl req' will use SHA1 and various things will be unhappy with your certificate. Not all examples you can find on the Internet use '-sha256', so watch out.)

You can get 'openssl req' to create a self-signed certificate with a SAN, but since OpenSSL won't prompt for this you must use an OpenSSL configuration file to specify everything about the certificate, including the hostname(s). This is somewhat intricate, even if it turns out to be possibly to do this more or less through the command line with suitably complicated incantations. I particularly admire the use of the shell's usually obscure '<(...)' idiom.

Given how painful this is, what we really need is a better tool to create self-signed certificates and fortunately for me, it turns out that there is just what I need sitting around in the Go source code as generate_cert.go. Grab this file, copy it to a directory, then:

$ go build generate_cert.go
$ ./generate_cert --host www.example.com --duration 17520h
2017/04/11 23:51:21 written cert.pem
2017/04/11 23:51:21 written key.pem

This generates exactly the sort of modern self-signed certificate that I want; it uses SHA256, it has a 2048-bit RSA key (by default), and it's got SubjectAltName(s). You can use it to generate ECDSA based certificates if you're feeling bold.

Note that this generates a certificate without a CN. Since there are real CN-less certificates out there in the wild issued by real Certificate Authorities (including the one for this site), not having a CN should work fine with web browsers and most software, but you may run into some software that is unhappy with this. If so, it's only a small modification to add a CN value.

(You could make a rather more elaborate version of generate_cert.go with various additional useful options, and perhaps someone has already done so. I have so far resisted the temptation to start changing it myself.)

A rather more elaborate but more complete looking alternative is Cloudflare's CFSSL toolkit. CFSSL can generate self-signed certificates, good modern CSRs, and sign certificates with your own private CA certificate, which covers everything I can think of. But it has the drawback that you need to feed it JSON (even if you generate the JSON on the fly) and then turn its JSON output into regular .pem files with one of its included programs.

For basic, more or less junk self-signed certificates, generate_cert is the simple way to go. For instance my sinkhole SMTP server now uses one of these certs; SMTP senders don't care about details like good O values in your SMTP certificates, and even if they did in general spammers probably don't. If I was generating more proper self-signed certificates, one where people might see them in a browser or something, I would probably use CFSSL.

(Although if I only needed certificates with a constant Subject name, the lazy way to go would be to hardcode everything in a version of generate_cert and then just crank out a mass of self-signed certificates without having to deal with JSON.)

PS: We might someday want self-signed certificates with relatively proper O values and so on, for purely internal hosts that live in our own internal DNS zones. Updated TLS certificates for IPMI web interfaces are one potential case that comes to mind.

PPS: It's entirely possible that there's a better command line tool for this out there that I haven't stumbled over yet. Certainly this feels like a wheel that people must have reinvented several times; I almost started writing something myself before finding generate_cert.

sysadmin/MakingModernSelfSignedSSLCerts written at 00:23:59; Add Comment


How TLS certificates specify the hosts they're for

Any TLS-like system of public key cryptography plus Certificate Authorities that vouch for your identity needs a format for certificates. When Netscape came up with the very first version of SSL, they opted not to invent a certificate format to go with their new encryption protocol; instead they reused the certificate format that had already been proposed for email in 1993 in RFC 1422 (as part of PEM). This foundational work didn't define its own certificate format either; instead it opted to politely reuse an existing specification, X.509, which associated with the X.500 directory specification, which was as a broad whole intended to be part of the OSI network stack before OSI as a whole was completely run over by the Internet TCP/IP juggernaut. This reuse in PEM was apparently intended in part to help interoperability with X.500/X.509 based email systems that more or less never materialized.

(The Internet used to worry a great deal about talking to foreign email systems and how to map between SMTP email concepts and those systems.)

A standard part of X.509 certificates is their Subject name, ie what the certificate is about. X.509 subject names are not freeform text; instead they are a set of attributes making up an X.500 Distinguished Name. Since these attributes come from an OSI directory specification, they're focused on identifying people in some role, as you can see from the list of standard DN keys in RFC 1779. This presented Netscape with a little problem, namely how to use a certificate to identify an Internet host instead of a person. So back in 1995 and 1996 when SSL v2 and v3 were specified, people took the obvious way out. X.500 DNs have a field called the CN, or Common Name, which for people is your name. For host certificates, the CN field was reused as the hostname (with the other DN fields being used to identify the organization officially associated with the certificate and so on).

In the beginning this was fine. However, as SSL went on people started wanting to have more than one hostname associated with a single SSL certificate; if nothing else, people liked having both www.<domain> and plain <domain>. This is a problem because the CN only contained a single hostname. Fortunately X.509 certificates can contain extensions, and so by no later than RFC 2459 (which went with TLS 1.0) there was the Subject Alternative Name extension. A SubjectAltName was explicitly a sequence of things (including DNS hostnames), so you could put multiple hostnames in it.

Then things get complicated. As of RFC 3280 in 2002, including SANs in TLS certificates was theoretically mandatory (via), and in fact using a CN was deprecated in RFC 2818 in 2000. How SANs and the CN interacts depends on the software involved; per this Stackoverflow answer, RFC 5280 in 2008 said to look at both, while RFC 6125 in 2011 theoretically mandated checking SANs only if they were present (and they should always be present). See also this SO answer for more information about the CA/Browser Forum Baseline Requirements view on this, and there's also the discussion in this Let's Encrypt issue.

Given that there is plenty of pre-2011 software out there on the Internet, you can in practice find TLS certificates with all variants of this; with a CN only, with SANs only, with SANs and a CN that contains one of the SAN's names, and with SANs that do not contain the certificate's CN. With the widespread prevalence of multiple hostnames in SANs in certificates (sometimes a very long list of them, eg on Cloudflare's web servers), all even vaguely modern TLS software can validate certificates using only SAN names. I don't know if there's any browser or TLS software yet that ignores the CN or actively requires certificates to have a SAN, but it's probably not too far off; at this point, it's likely to be at least a little bit unsafe to use TLS certificates without a SAN.

(You probably can't get CA signed certificates without a SubjectAltName, because the CA will add them for you whether or not you like it. Self-signed certificates are a different matter entirely, but that's another entry.)

tech/TLSCertificatesNamingHosts written at 21:55:09; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.