What ZFS messages about 'permanent errors in <0x95>:<0x0>' mean
If you use ZFS long enough (or are unlucky enough), one of the things
you may run into are reports in
zpool status -v of permanent errors
in something (we've had that happen to us despite redundancy). If you're reasonably lucky, the error message
will have a path in it. If you're unlucky, the error message will say
errors: Permanent errors have been detected in the following files:<0x95>:<0x0>
The short answer of what they mean is, to quote directly:
The first number is the dataset id (index) and the second is the object id. For filesystems, the object id can be the same as the file's "inode" as shown by "ls -i" But a few obect ids exist for all datasets. Object id 0 is the DMU dnode.
The dataset here may be a ZFS filesystem, a snapshot, or I believe a few other things. I believe that if it's still in existence, you'll normally get at least its name and perhaps the full path to the object. When it's not in existence any more (perhaps you deleted the snapshot or the whole filesystem in question since the scrub detected it), you get this hex ID and there's also no information about the path.
The reason the information is presented this way is that what the
ZFS code in the kernel saves and returns to the
zpool command is
actually just the dataset and object ID. It's up to
zpool to turn
both of these into names, which it actually does by calling back
into the kernel to find out what they're currently called, if the
kernel knows. Inspecting the relevant ZFS code
says that there are five cases:
<metadata>:<0x...>means corruption in some object in the pool's overall metadata object set.
<0x...>:<0x...>means that the dataset involved can't be identified (and thus ZFS has no hope of identifying the thing inside the dataset).
/some/path/namemeans you have a corrupted filesystem object (a file, a directory, etc) in a currently mounted dataset and this is its full current path.
(I think that ZFS's determination of the path name for a given ZFS object is pretty reliable; if I'm reading the code right, it appears to be able to scan upward in the filesystem hierarchy starting with the object itself.)
dsname:/some/pathmeans that the dataset is called
dsnamebut it's not currently mounted, and
/some/pathis the path within it. I think this happens for snapshots.
dsname:<0x...>means that it's in the given dataset
dsname(which may or may not be mounted), but the ZFS object in question can't have its path identified for various reasons (including that it's already been deleted).
Only things in ZFS filesystems (and snapshots and so on) have path names, so an error in a ZVOL will always be reported without the path. I'm not sure what the reported dataset names are for ZVOLs, since I don't use ZVOLs.
The final detail is that you may see this error status in '
-v' even after you've cleaned it up. To quote Richard Elling again:
Finally, the error buffer for "zpool status" contains information for two scan passes: the current and previous scans. So it is possible to delete an object (eg file) and still see it listed in the error buffer. It takes two scans to completely update the error buffer. This is important if you go looking for a dataset+object tuple with zdb and don't find anything...
PS: There are some cases where
<xattrdir> will appear in the file
path. If I'm reading the code correctly, this happens when the
problem is in an extended attribute instead of the filesystem object
PPS: Richard Elling's message was on the ZFS on Linux mailing list and about an issue someone was having with a ZoL system, but as far as I can see the core code is basically the same in Illumos and I would expect in FreeBSD as well, so this bit of ZFS wisdom should be cross-platform.
How to run a mail sending service that will probably never send spam
I have written any number of times before that mail sending services could take steps that would make sure almost no spam would be sent through them, but they don't bother (eg on Amazon, on modern mailing list services in general and earlier). However, I have not written down my view of these steps, partly because I have considered them obvious in the community in general. For various reasons, I now feel like writing these steps down. So here is an incomplete list of obvious steps to take that would mostly gut sending spam through such a service.
In no particular order:
- Charge people a decent amount of money for your service, possibly
with a deposit up front. Don't have a free or a cheap tier,
because it attracts the wrong sort of customer (Patrick McKenzie has written at length on how too-low
or free pricing is a bad idea in SaaS in general).
- Force people to put their address lists on your service, not just
funnel their email through in bulk sending. Forced uploads allow
you to scan the address list in advance to look for known warning
signs, such as definitely nonexistent domains or known-bad
addresses that never accept your email.
- Require all email addresses submitted to you by a particular customer
to be confirmed. The gold standard would be confirming separately
for every alleged mailing list the customer sets up; the silver
standard is confirming once when the customer first submits the
address as part of any list and then assuming that the customer has
the right to use that address in other lists. As part of requiring
confirmation, provide an extra link that communicates to you 'I have
never heard of these people and I do not know why they have my email
address'. Even a moderate level of use of this link is a warning sign.
It should go without saying that having more than a trace level of bounces or email rejections during confirmation should be a big warning sign.
Probably the silver version is the most realistic, since these days customers may not have distinct 'mailing lists' as such, if they're using you to deliver event-based notifications to people's email and so on.
(Even sending an initial notification email to people saying 'your address has been added to our system by <customer>' would be a step forward. These days a mail sending service could claim it was a GDPR requirement.)
- Run all submitted mailing list messages through all of the available
free open source anti-spam and anti-virus systems, and perhaps
at least one of the commercial ones. If any of the systems flag the
message, don't send the email and surface this in an alert both to
the customer and to your abuse handling team.
(It's not a service to the customer to let them send out email that you know will trip spam alerts for some recipients. Legitimate customers will likely thank you for such a pre-check service, and may even want a way to submit draft messages to it.)
- Make it trivial for people to report unsolicited email and spam, and to 'unsubscribe'.
- Pay attention to bounces, SMTP rejections in general, unsubscribes,
and spam complaints. Mine them for addresses to add to your list of
warning addresses. Rejections after SMTP DATA are probably an
especially bad warning sign, because they suggest it was content
filtering that caused the rejection.
(As part of this, you should obviously recognize and parse the various SMTP 4xx and 5xx messages that major email providers use when they're dealing with questionable email messages. But this is so obvious that I suspect that any mail sending SaaS that wants to be successful is already doing it.)
I'm assuming as a baseline that you will do things like accept bounces and replies and properly implementing SMTP. These days you may want SPF, DKIM, or DMARC in order to pacify various large email providers who are getting increasingly insistent on it, but that's more in the realm of 'competently operating a commercial service'.
I'm pretty confident that any mail sending service that implemented all of these would send almost no spam, and I'm reasonably confident that it would still have a business. But of course it wouldn't have a business that's as big as you'd get by not bothering to do some of these things (especially confirming email addresses), and it would cost more to operate, and you wouldn't have as many customers because a certain number of the more shady people would stay away (as would all of the cheap people).