Wandering Thoughts archives

2024-01-13

Indexed archive formats and selective restores

Recently we discovered first that the Amanda backup system has to read some tar archives all the way to the end when restoring a few files from them and then sometimes it can do quick restores from tar archives. What is going on is the general issue of indexed (archive) formats, and also the potential complexities involved in them in a full system.

To simplify, tar archives are a series of entries for files and directories. Tar archives contain no inherent index of their contents (unlike some archive formats, such as ZIP archives), but you can build an external index of where each file entry starts and what it is. Given such an index and its archive file on a storage medium that supports random access, you can jump to only the directory and file entries you care about and extract only them. Because tar archives have not much special overall formatting, you can do this either directly or you can read the data for each entry, concatenate it, and feed it to 'tar' to let tar do the extraction.

(The trick with clipping out the bits of a tar archive you cared about and feeding them to tar as a fake tar archive hadn't occurred to me until I saw what Amanda was doing.)

If tar was a more complicated format, this would take more work and more awareness of the tar format. For example, if tar archives had an internal index, either you'd need to operate directly on the raw archive or you would have to create your own version of the index when you extracted all of the pieces from the full archive. Why would you need to extract the pieces if there was an internal index? Well, one reason is if the entire archive file was itself compressed, and your external index told you where in the compressed version you needed to start reading in order to get each file chunk.

The case of compressed archives shows that indexes need to somehow be for how the archive is eventually stored. If you have an index of the uncompressed version but you're storing the archive in compressed form, the index is not necessarily of much use. Similarly, it's necessary for the archive to be stored in such a way that you can read only selected parts of it when retrieving it. These days that's not a given, although I believe many remote object stores support HTTP Range requests at least some of the time.

(Another case that may be a problem for backups specifically is encrypted backups. Generally the most secure way to encrypt your backups is to encrypt the entire archive as a single object, so that you have to read it all to decrypt it and can't skip ahead in it.)

SelectiveRestoresAndIndexes written at 23:28:30; Add Comment

2024-01-10

MFA today is both 'simple' and non-trivial work

Over on the Fediverse I said something a while back:

I have a rant bubbling around my head about 'why I'll never enable MFA for Github'. The short version is that I publish code on GH because it's an easy and low effort way to share things. So far, managing MFA and MFA recovery is neither of those; there's a lot of hassles, worries, work to do, and 'do I trust you to never abuse my phone number in the future?' questions (spoiler, no).

I'll deal with MFA for work. I won't do it for things I'm only doing for fun, because MFA makes it not-fun.

Basic MFA is ostensibly pretty simple these days. You get a trustworthy app for your smartphone (that's two strikes right there), you scan the QR code you get when you enable MFA on your account, and then afterward you use your phone to generate the MFA TOTP code that you type in when logging in along with your password. That's a little bit more annoying than the plain password, but think of the security, right?

But what if your phone is lost, damaged, or unusable because it has a bulging battery and it's taking a week or two to get your carrier to exchange it for a new one (which happened to us with our work phones)? Generally you get some one time use special codes, but now you have to store and manage them (obviously not on the phone). If you're cautious about losing access to your phone, you may want to back up the TOTP QR code and secret itself. Both the recovery codes and the TOTP secret are effectively passwords and now you need to handle them securely; if you use a password manager, it may or may not be willing to store them securely for you. Perhaps you can look into age.

(Printing out your recovery codes and storing the paper somewhere leaves you exposed to issues like a home fire, which is an occasion where you might also lose your phone.)

Broadly, no website has a good account recovery story for MFA. Figuring out how to deal with this is not trivial and is your problem, not theirs. And while TOTP is not the in MFA thing these days, the story is in many ways worse with physical hardware tokens, because you can't back them up at all (unlike TOTP secrets). Some environments will back up software Passkeys, but so far only between the same type of thing and often at the price of synchronizing things like all of your browser state.

However, all of this is basically invisible in the simple MFA story. The simple MFA story is that everything magically just works and that you can turn it on without problems or serious risks. Of course, websites have a good reason for pushing this story; they want their users to turn on MFA, for various reasons. My belief is that the gap between the simple MFA story and the actual work of doing MFA in a way that you can reliably maintain access to your account is dangerous, and sooner or later this danger is going to become painfully visible.

(Like many other versions of mathematical security, the simple MFA story invites blaming people (invariably called 'users' when doing this) when something goes wrong. They should have carefully saved their backup codes, not lost track of them; they should have sync'd their phone's TOTP stores to the cloud, or done special export and import steps when changing phones, or whatever else might have prevented the issue. This is as wrong as it always is. Security is not math, it is people.)

MFAIsBothSimpleAndWork written at 22:49:15; Add Comment

2024-01-07

TLS certificate expiry times are fundamentally a hack

Famously, TLS certificates expire, which even today can take websites offline because they didn't renew their TLS certificate in time. This doesn't just affect websites; people not infrequently create certificates that are supposed to be long lived, except sometimes they make them last (only) ten years, which isn't long enough. When people argue about this, let's be clear; TLS certificate expiry times, like most forms of key expiry, are fundamentally a hack that exists to deal with the imperfections of the world.

In a spherical and frictionless ideal world, TLS certificate keys would never be compromised, TLS certificates would never be issued to anyone other than the owner of something, TLS certificates could be effectively invalidated through revocation, and there would be no need to have TLS certificates ever expire. In this world, TLS certificates would be perpetual, and when you were done with a website or some other use of a TLS certificate, you would publish a revocation of it just to be sure.

We don't live in a spherical and frictionless world, so TLS certificates expire in order to limit the damage of key compromise and mis-issued certificates. That TLS certificates expire only imperfectly limits the damage of key compromise, not only because you have to wait for the certificate to expire (hence the move to shorter and shorter lifetimes) but also because there's generally nothing that stops you from re-using the same key for a whole series of TLS certificates. Since we don't have effective certificate revocation at scale, both mis-issued certificates and certificates where you know the key is compromised can only really be handled by letting them expire. If they didn't expire, they would be dangerous forever.

(If you're a big place, the browsers will give you a hand by shipping an update that invalidates the required certificates and keys, but this isn't available to ordinary mortals.)

This isn't particularly specific to TLS; other protocols with public keys often have the same issues and adopt the same solution of expiry times (PGP is one example). There are protocols that use keys without expiry times, such as DKIM. However, DKIM has extremely effective key revocation; to revoke a key, you remove the public part from your DNS, and then no one can validate anything signed by that key (well, unless they have their own saved copy of your old DNS). Other protocols punt and leave the whole problem up to you, for example SSH keypairs.

(Some protocols have other reasons for limiting the lifetime of keys, such as making encrypted messages 'expire' by default)

The corollary of this is that if you're dealing with TLS certificates (or keypairs in general) and these issues aren't a concern for you, there's not much reason to limit your TLS certificate lifetimes. Just don't make their lifetimes be ten years.

(My current personal view is that there are two reasonable choices with TLS certificate lifetimes. Either you have automated issuance and renewal, in which case you should have short lifetimes, or you have manual issuance and rollover, in which case they should be as long as you can get away with. TLS certificates that live for a year or three and have to be manually rolled over are the worst of both worlds; a key compromise or a mis-issuance is dangerous for a comparatively long time, and the rollover period is long enough that you'll have issues keeping track of it and doing it.)

TLSCertificateExpiryHack written at 18:10:40; Add Comment

2024-01-06

Some ballpark numbers for fun on filling filesystem cache with NFS traffic

Our current ZFS fileserver hardware is getting long in the tooth, so we're working on moving to new hardware (with the same software and operational setup, which we're happy with). This new hardware has 512 GB of RAM instead of the 192 GB of RAM in our current fileservers, which means that we're going to have a very big ZFS filesystem cache. Today, I was idly wondering how long it would take to fill the cache to a reasonable level with NFS (read) traffic, since we have metrics that include, among other things, the typical bandwidth our current fileservers see (which usually isn't all that high).

ZFS doesn't use all of the system's memory for its ARC cache, and not all of the ARC cache is file data; some of it is filesystem metadata like the contents of directories, the ZFS equivalent of inodes, and so on. As a ballpark, I'll use 256 GBytes of file data in the cache. A single server with a 1G connection can read over NFS at about 110 Mbytes a second. This is a GByte read in just under ten seconds, or about 6.4 GBytes a minute, and a bit under 46 minutes of continuous full-rate 1G NFS reads to fill a 256 GByte cache (assuming that the ZFS fileserver puts everything read in the cache and there are no re-reads, which are some big assumptions).

Based on what I've seen on our dashboards, a reasonable high NFS read rate from a fileserver is in the area of 300 to 400 Mbytes a second. This is about 23.4 GBytes a minute (at 400 Mbytes/sec), and would fill the ZFS fileserver cache from a cold start in about 11 minutes (again with the assumptions from above). 400 Mbytes/sec is well within the capabilities of SSD-based fileservers.

However, most of the time our fileservers are much less active than that. Last Thursday, the average bandwidth over the workday was in the area of 1 Mbyte/sec (yes, Mbyte not GByte). At this rate filling a 256 GByte cache of file data would take three days. A 20 Mbyte/sec sustained read rate fills the cache in only a few hours. At the low end, relatively 'small' changes in absolute value clearly have an outsized effect on the cache fill time.

In practice, this cache fill requires 256 GBytes of different data that people want to read (possibly in a hurry). This is much more likely to be the practical limit on filling our fileserver caches, as we can see by the typical 1 Mbyte/sec data rate.

(All of this is actually faster than I expected before I started writing this and ran the numbers.)

FilesystemCacheAndNFSBandwidth written at 21:31:39; Add Comment

2023-12-30

Email addresses are not good 'permanent' identifiers for accounts

Every so often someone needs to create a more or less permanent internal identifier in their system every person's account. Some of the time they look at how authentication systems like OIDC return email addresses among other data and decide that since pretty much everyone is giving them an email address, they'll use the email address as the account's permanent internal identification. As the famous saying goes, now you have two problems.

The biggest problem with email addresses as 'permanent' identifiers is that people's email addresses change even within a single organization (for example, a university). They change for the same collection of reasons that people's commonly used names and logins change. An organization that refuses to change or redo the email addresses it assigns to people is being unusually cruel in ways that are probably not legally sustainable in any number of places.

(Some of the time there will be some sort of access or forwarding from the old email address to the new one, but even then the old email address may no longer work for non-email purposes such as OIDC authentication. And beyond that, the person won't want to keep using their old and possibly uncomfortable email address with you, they want to use their new current one.)

The lesser problem is that you have no particular guarantee that an organization won't reuse email addresses, either in general or for particularly desirable ones that get reused or reassigned as an exception because someone powerful wants them. Sometimes you sort of have no choice, because account recovery has to run through the email address you have on file, but at other times (such as in theory with OIDC), you have some form of internal ID that is supposed to be unique and permanent, which you should use.

Even if you have to remember an email address for account recovery, you want your internal identifier for accounts to be meaningless. This will make your life much simpler in the long run, even if this is never exposed to people.

(There are also security issues lurking in the underbrush of reading too much into email addresses, cf (via).)

EmailAddressesBadPermanentIDs written at 23:22:46; Add Comment

2023-12-25

Standards often provide little guidance for handling 'bad' content

In a comment on my entry on what I think SMTP Smuggling enables, Leah Neukirchen noted something important, which is that SMTP messages that contain a CR or a LF by itself aren't legal:

I disagree. The first mail server is also accepting a message with a non-CRLF LF, which violates RFC 5322 section 2.3

CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body.

The capitalization in the RFC quote is original, the emphasis is mine, and the meaning of these terms is covered in RFC 2119. What it adds up to is unambiguous at one level; a SMTP message that contains a bare CR or LF isn't an RFC 2119 compliant message, much like a C program with undefined behavior isn't a valid ANSI C program.

But just like the ANSI C standard doesn't (as far as I know) put any requirements on how a C compiler handles a non-ANSI-C program, RFC 2119 provides no requirements or guidance on what you should or must do with a non-compliant message. This is quite common in standards; standards often spell out only what is within their scope and what must be done with those things. They've historically been silent about non-standard things, leaving it entirely to the implementer. When it comes to protocol elements, this generally means rejecting them (you don't try to guess what unknown SMTP commands are), but when it comes to things you don't act on like email message content, things are much fuzzier.

At this point two things often intervene, The first is Postel's Law, which suggests people accept things outside the standard. The second is that strong standards compliance is often actively inconvenient or problematic for people using the software. I've lived life behind a SMTP mailer that had strong feelings about RFC compliance (at least in some areas), and by and large we didn't like it. Strict software is often unpopular software, which pushes people writing software to appeal to Postel's Law in the absence of anything else. If you don't even have an RFC to point to that says 'you SHOULD reject this' (or 'you MUST reject this') and you have people banging on your door wanting you to be liberal, often the squeaky wheel gets the grease (or has gotten until recently; these days people are somewhat less enamored of Postel's Law, for various reasons including security issues).

(C compilers and their reaction to undefined behavior is a complex subject, but I don't know of any mainstream compiler that will actually reject code that has known undefined behavior.)

At this point there's not much we can do here. It's obviously much too late for existing RFCs and standards that don't have any requirements or guidance on what you should do about bad contents, and I'm not sure that people would agree on adding it anyway. People can attempt to be strict and hope that not much will be affected, or they can try to write rules about error recovery (which HTML eventually did in HTML5) to encourage software to all do the same, agreed-on thing. But these will probably mostly be reactive things, not proactive ones (so we're probably about to see a wave of SMTP mailers getting strict in the wake of SMTP Smuggling).

StandardsAndBadContent written at 21:44:15; Add Comment

2023-12-08

A possible path to reliable name constraints on internal TLS CAs

A while back I wrote an entry about TLS CA root certificate name constraints for internal Certificate Authorities. One of the traditional problems with your own TLS CA is that this CA can be used to sign any name, even names you don't want to sign. A name constraint would limit that, but traditionally these weren't widely supported, especially on TLS CA root certificates. Then I read Michal Jirků's Running one's own root Certificate Authority in 2023 and had a realization about a general way out so that everything would accept your TLS name constraints.

Some software will accept name constraints on root CA certificates, so you create your root certificate with them. Some software will only accept name constraints on intermediate CA certificates, so then you create an intermediate certificate with the same constraints as your root certificate; it should also have the same validity period as your root certificate (or as long a validity period as you expect to need your CA for). At this point, you throw away the CA root certificate's private key, so no one can make any more intermediate certificates. This insures an attacker can't create a new intermediate certificate without name constraints and then issue certificates from it that will be accepted by older Chrome versions and other things that ignore root CA name constraints.

(Modern Chrome versions support root CA name constraints, but I expect that older Chrome versions will linger on for quite some time in the form of old Android devices that aren't getting updates.)

I suspect that there are some uncertainties and potential problems with this approach. Most obviously, you're in trouble if TLS clients decide to start limiting the lifetime of intermediate CA certificates, since you can't make any more once you destroy the root CA key. As far as I can tell, the CA/Browser Forum Baseline Requirements don't currently limit intermediate certificate lifetimes, but this may change someday.

(One option would be to pre-mint a series of intermediates with relatively short (and overlapping) lifetimes as a precaution. But then you have to protect all of these intermediate certificates for possibly several decades.)

Since using intermediate CAs is what's done in public (web) TLS, probably pretty much everything supports it so maybe there aren't really any uncertainties beyond the long validity period of the intermediate certificate. I don't know how many levels of intermediates are commonly supported, so maybe there could be problems if you wanted two levels of intermediate (the one that's effective your 'root' CA and then a second, shorter term one below it that you used to directly issue TLS certificates).

TLSInternalCANameConstraintsII written at 23:02:40; Add Comment

2023-12-05

Doing work that scales requires being able to scale your work

Several years ago I read Tobias Bernard's Doing Things That Scale. To summarize the article badly, Tobias Bernard talked about how they had moved away from customizing things (and then having to maintain the customizations) in favour of doing generally useful changes to upstreams. I have various feelings about this as a general principle (and comments on my entry on thinking about the sensible limits of customization gave me more things to think about), but one of the thoughts that firmed up is that you can only do work that scales if you can scale your work.

As I read Bernard's article, scaling your work in this context means getting the things you want into the upstream, even in some imperfect form, or perhaps becoming an upstream yourself (by creating something of and for general use). In order to scale your work this way, you must be both willing to undertake this and able to do so. For the most common way to scale your work, this means working with the upstream and even becoming part of it, if you think there are significant things that need it. This is work, which is to say that it takes time and energy, and with some upstreams it can be a challenging and fraught endeavour (I suspect readers will have their own examples).

(In the other option, you need to have the time and energy to build and run a successful, generally useful thing that becomes reasonably popular. This is often considered an especially thankless and wearying task.)

If you won't be scaling your work for whatever reason (including lack of the time required or lack of willingness to wrestle with muddy pigs), your choices are to either live with the current state of affairs, whatever it is, or do work that doesn't scale, the sort of personal, individual customization that Tobias Bernard discusses moving away from. I have some broader views on this that I've decided don't fit into the margins of this entry, so I will confine myself to saying that I think there are plenty of people who fit into this general category.

(As noted in comments on my earlier entry, there's also the aspect of customization as something you enjoy doing for fun. There's no reason to restrict yourself from fun because it only benefits you; if anything, that's kind of the point of fun.)

RequirementToScaleYourWork written at 22:40:54; Add Comment

2023-11-29

The quietly impressive thing mail clients do when you write HTML mail

It's not news that a great deal of email in the world today is written in HTML, and has been for some time. If you insist on plain text email, you're increasingly an anachronism. Many people writing email probably don't even think about people who prefer plain text, and I think many mail clients will default to HTML even if you're replying to a plain text message, so even if you write to me in HTML and I write back in plain text, your reply is back to HTML again.

But while that description is true at the level of what people experience, it's not true at the technical level (at least, not usually). Even today, most 'HTML' email is actually a MIME multipart/alternative, with text/plain and text/html alternate parts. An awful lot of the time, the contents of the text/plain part isn't a little message saying 'read the HTML', it's in fact a faithful plain text version of the HTML that people wrote. Pretty much universally, mail clients quietly create that plain text version from the HTML version that people write, following an assortment of conventions for how to render HTML-isms in plain text. Looked at from this angle, it is quietly impressive. Here is a feature and a chunk of code that could be considered partially vestigial, yet almost everyone implements it.

One of the reasons this may be somewhat easier than it looks is that people rarely literally write HTML in their mail client. Instead they tend to work in a WYSIWYG environment, where the mail client can mark up the text with intentions, like 'bold' or 'links to <X>', and then render the intentions in both HTML and plain text. But I'm only guessing about how mail clients implement this. I don't think it's as simple as pushing the HTML through some sort of plain text rendering, because the plain text and the HTML sometimes change styles for things. For instance, in the HTML, bits quoted from the message being replied to may be indented, while in the plain text they get rendered with the customary '> ' in front of them.

It's not only mail clients used by people that (still) do this. A fair number of major sources of (HTML) email more or less automatically generate a plain text version as well, often coming at least partially from people's input. For one example I experience regularly, Github issues are natively in Markdown and are commonly seen in HTML format, but Github faithfully makes a quite usable text/plain version. It might not be much effort with Markdown, but it's at least some.

(Not all plain text plus HTML email has the same content in both forms, and sometimes the plain text content is broken in various ways (also). But this is still the exception; the vast majority of these emails that I get have functionally the same content in both the plain text and the HTML version.)

(This entry was sparked by me idly wondering if it would be possible to easily write HTML-format emails in MH-E, and then realizing that it wasn't enough to just write HTML emails; I'd need to generate the plain text version too.)

MailClientsAndCreatingHTMLMail written at 22:18:34; Add Comment

2023-11-15

WireGuard and the question of link-local IPv6 addresses

Suppose, not hypothetically, that you are setting up a WireGuard tunnel to extend IPv6 connectivity to a machine that is (still) on an IPv4-only subnet. One part of IPv6 in general is link-local addresses, which are required for the IPv6 Neighbor Discovery Protocol (NDP) and are used for other things. However, under Linux WireGuard interfaces disable automatic kernel link local generation, leaving either your higher level software or you to configure them. So the obvious question is whether you should set up IPv6 link local addresses by hand (if your software will do it for you, you might as well let it).

WireGuard interfaces are point to point links and don't do NDP, so they don't need a link-local address for that, and I don't know if you can run DHCPv6 over one even if you want to. Apparently OSPFv3 requires link-local addresses, and you might want to run that in some more complicated WireGuard IPv6 situations. A simple point to point WireGuard link to extend IPv6 to a host will work (as far as I can tell) if you only configure it with the global peer IPv6 addresses involved and don't have link local addresses, but this may be an IPv6 crime.

However, it may be that one or both ends doesn't have a fixed IPv6 address; for example, they may obtain them through IPv6 Stateless Address Autoconfiguration (SLAAC) and change them over time. In this case you can't configure a fixed global (peer) address on the WireGuard interface, because it's not fixed (if you tried, you'd have to coordinate updates with SLAAC address changes). The only fixed addresses you have are link local ones you generate yourself.

(Hopefully you at least know what IPv6 /64s (or greater) are on each end of the WireGuard link so that you can set up appropriate routing and allowed IP information.)

The other reason I see to set up link local addresses even if you don't strictly need them is that it gives you an address for the peer that's generally going to sidestep any routing configuration issues. You can use this peer IP (with a scope or interface specification) to ping or talk to the peer over the WireGuard link to test it, and be pretty sure that this is exactly and only what's happening. Now that I've realized this, I think I'm going to configure link local addresses on all future IPv6 point to point links just for this.

(I've spent enough time being puzzled by IPv4 routing issues involving clever WireGuard configurations that I don't want to repeat it with IPv6, although right now I'm not doing anything complicated.)

PS: Learn from my mistakes and remember to add your IPv6 link local address range to the WireGuard allowed IPs (on both sides, if applicable; in my case one side's allowed IPs is already all of IPv6, so I only needed to add fe80::/64 to the other).

WireGuardAndLinkLocalIPv6 written at 22:25:55; Add Comment

2023-11-10

Backup systems and how much they do or don't know about storage formats

One of the divides in large scale systems for handling backups is whether they have their own custom storage format (or formats) for backups, or whether they rely on outside tools to create what I'll call 'backup blobs' that the backup system then manages. This division is fractal, because sometimes what you're backing up is, for example, database snapshots or dumps, and even if the backup system has its own custom storage format it may well treat the database dump as an opaque blob of a file that it only deals with as a unit. (It's a lot of work to be able to peer inside all of the storage formats you might run into, or even recognize them.)

The advantage of a backup system that relies on other tools is that it doesn't have to write the tools. This has two benefits. First, standard tools for making backups of filesystems and so on are often much more thoroughly tested and hardened against weird things than a new program. Second, if you allow people to specify what tools to use and provide their own, they can flexibly back up a lot of things in a lot of special ways; for example, you could write a custom tool that took a ZFS snapshot of a filesystem and then generated a backup of the snapshot. More complex tricks are possible if people want to write the tools (imagine a database 'backup' program that treated the database as something like a filesystem, indexing it and allowing selective restores).

(Generally, backup systems insist that tools have certain features and capabilities, for example being able to report a list of contents (an index) of a just-made backup in a standard format. It's up to you to adapt existing programs to fit these requirements, perhaps with cover programs.)

The advantage of a backup system that has its own storage format for backups and its own tools for creating them, restoring them, and so on is that the backup system can often offer more features (and better implemented ones). A backup system that relies on other tools for the actual work of creating backups and performing restores is forced to treat those tools as relatively black boxes; a backup system that does this work in-house can tightly integrate things to provide various nice features, like knowing exactly where a specific file you want to restore is within a large backup, or easily performing fine grained backup scheduling and tracking across a lot of files. And the storage format itself can be specifically designed for the needs of backups (and this backup system), instead of being at the mercy of the multiple purposes and historical origins of, say, tar format archives.

(But then the backup system has to do all of the work itself, and fix all of the bugs, if it manages to find them before they damage someone's backup.)

In practice, backup systems seem to go back and forth on this over time depending on their goals (including where they're backing up to) and the state of the commonly available tools on the platforms they want to work on. For commercial backup systems, there can also be commercial reasons to use a custom format that only your own tools can deal with. Over some parts of the past, general tools have been limited and not considered suitable so even open-source people built fully custom systems. Over other parts, the tools have been considered good enough for the goals, so open-source backup systems tended to use them and focus on other areas.

(For open source backup systems it is in some sense a waste to have to write your own tools. There's only so much programming resources you have available and there are lots of things a good backup system needs to implement that can't be outsourced to other tools.)

BackupSystemsStorageFormatDivide written at 23:13:11; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.