Wandering Thoughts archives

What ZFS messages about 'permanent errors in <0x95>:<0x0>' mean

If you use ZFS long enough (or are unlucky enough), one of the things you may run into are reports in zpool status -v of permanent errors in something (we've had that happen to us despite redundancy). If you're reasonably lucky, the error message will have a path in it. If you're unlucky, the error message will say something like:

errors: Permanent errors have been detected in the following files:

        <0x95>:<0x0>

This is a mysterious and frustrating message. On the ZFS on Linux mailing list, Richard Elling recently shared some extremely useful information about what they mean in this message.

The short answer of what they mean is, to quote directly:

The first number is the dataset id (index) and the second is the object id. For filesystems, the object id can be the same as the file's "inode" as shown by "ls -i" But a few obect ids exist for all datasets. Object id 0 is the DMU dnode.

The dataset here may be a ZFS filesystem, a snapshot, or I believe a few other things. I believe that if it's still in existence, you'll normally get at least its name and perhaps the full path to the object. When it's not in existence any more (perhaps you deleted the snapshot or the whole filesystem in question since the scrub detected it), you get this hex ID and there's also no information about the path.

The reason the information is presented this way is that what the ZFS code in the kernel saves and returns to the zpool command is actually just the dataset and object ID. It's up to zpool to turn both of these into names, which it actually does by calling back into the kernel to find out what they're currently called, if the kernel knows. Inspecting the relevant ZFS code says that there are five cases:

• <metadata>:<0x...> means corruption in some object in the pool's overall metadata object set.

• <0x...>:<0x...> means that the dataset involved can't be identified (and thus ZFS has no hope of identifying the thing inside the dataset).

• /some/path/name means you have a corrupted filesystem object (a file, a directory, etc) in a currently mounted dataset and this is its full current path.

  (I think that ZFS's determination of the path name for a given ZFS object is pretty reliable; if I'm reading the code right, it appears to be able to scan upward in the filesystem hierarchy starting with the object itself.)

• dsname:/some/path means that the dataset is called dsname but it's not currently mounted, and /some/path is the path within it. I think this happens for snapshots.

• dsname:<0x...> means that it's in the given dataset dsname (which may or may not be mounted), but the ZFS object in question can't have its path identified for various reasons (including that it's already been deleted).

Only things in ZFS filesystems (and snapshots and so on) have path names, so an error in a ZVOL will always be reported without the path. I'm not sure what the reported dataset names are for ZVOLs, since I don't use ZVOLs.

The final detail is that you may see this error status in 'zpool status -v' even after you've cleaned it up. To quote Richard Elling again:

Finally, the error buffer for "zpool status" contains information for two scan passes: the current and previous scans. So it is possible to delete an object (eg file) and still see it listed in the error buffer. It takes two scans to completely update the error buffer. This is important if you go looking for a dataset+object tuple with zdb and don't find anything...

PS: There are some cases where <xattrdir> will appear in the file path. If I'm reading the code correctly, this happens when the problem is in an extended attribute instead of the filesystem object itself.

(See also this, this, and this.)

PPS: Richard Elling's message was on the ZFS on Linux mailing list and about an issue someone was having with a ZoL system, but as far as I can see the core code is basically the same in Illumos and I would expect in FreeBSD as well, so this bit of ZFS wisdom should be cross-platform.

How to run a mail sending service that will probably never send spam

I have written any number of times before that mail sending services could take steps that would make sure almost no spam would be sent through them, but they don't bother (eg on Amazon, on modern mailing list services in general and earlier). However, I have not written down my view of these steps, partly because I have considered them obvious in the community in general. For various reasons, I now feel like writing these steps down. So here is an incomplete list of obvious steps to take that would mostly gut sending spam through such a service.

In no particular order:

• Charge people a decent amount of money for your service, possibly with a deposit up front. Don't have a free or a cheap tier, because it attracts the wrong sort of customer (Patrick McKenzie has written at length on how too-low or free pricing is a bad idea in SaaS in general).

• Force people to put their address lists on your service, not just funnel their email through in bulk sending. Forced uploads allow you to scan the address list in advance to look for known warning signs, such as definitely nonexistent domains or known-bad addresses that never accept your email.

• Require all email addresses submitted to you by a particular customer to be confirmed. The gold standard would be confirming separately for every alleged mailing list the customer sets up; the silver standard is confirming once when the customer first submits the address as part of any list and then assuming that the customer has the right to use that address in other lists. As part of requiring confirmation, provide an extra link that communicates to you 'I have never heard of these people and I do not know why they have my email address'. Even a moderate level of use of this link is a warning sign.

  It should go without saying that having more than a trace level of bounces or email rejections during confirmation should be a big warning sign.

  Probably the silver version is the most realistic, since these days customers may not have distinct 'mailing lists' as such, if they're using you to deliver event-based notifications to people's email and so on.

  (Even sending an initial notification email to people saying 'your address has been added to our system by <customer>' would be a step forward. These days a mail sending service could claim it was a GDPR requirement.)

• Run all submitted mailing list messages through all of the available free open source anti-spam and anti-virus systems, and perhaps at least one of the commercial ones. If any of the systems flag the message, don't send the email and surface this in an alert both to the customer and to your abuse handling team.

  (It's not a service to the customer to let them send out email that you know will trip spam alerts for some recipients. Legitimate customers will likely thank you for such a pre-check service, and may even want a way to submit draft messages to it.)

• Make it trivial for people to report unsolicited email and spam, and to 'unsubscribe'.
• Pay attention to bounces, SMTP rejections in general, unsubscribes, and spam complaints. Mine them for addresses to add to your list of warning addresses. Rejections after SMTP DATA are probably an especially bad warning sign, because they suggest it was content filtering that caused the rejection.

  (As part of this, you should obviously recognize and parse the various SMTP 4xx and 5xx messages that major email providers use when they're dealing with questionable email messages. But this is so obvious that I suspect that any mail sending SaaS that wants to be successful is already doing it.)

I'm assuming as a baseline that you will do things like accept bounces and replies and properly implementing SMTP. These days you may want SPF, DKIM, or DMARC in order to pacify various large email providers who are getting increasingly insistent on it, but that's more in the realm of 'competently operating a commercial service'.

I'm pretty confident that any mail sending service that implemented all of these would send almost no spam, and I'm reasonably confident that it would still have a business. But of course it wouldn't have a business that's as big as you'd get by not bothering to do some of these things (especially confirming email addresses), and it would cost more to operate, and you wouldn't have as many customers because a certain number of the more shady people would stay away (as would all of the cheap people).