Running your own email is increasingly an artisanal choice, not a practical one

December 23, 2021

Over on Twitter, I said something that's been on my mind for a while, and is a bitter thing for me to accept:

Gloomy sysadmin take: it's no longer possible for ordinary organizations to operate a quality email infrastructure themselves. You can run a little artisanal one and feel happy about it, but it will not at all measure up to the quality of systems run by eg Google and Microsoft.

This is not directly about the big providers making it harder and harder to send them email, although that doesn't help. It's because a quality modern email environment is big, complex, and takes a lot of work to create and keep running.

To be clear, you absolutely can still run your own email infrastructure, getting email delivered to you, filtering incoming spam, sending email (with DMARC signatures and other modern email practices), providing IMAP access, and even run your own webmail setup. You can even do this with all open source software. But the email environment you get this way is increasingly what I called an artisanal one. It's cute, decent enough, and hand-crafted, but it doesn't measure up in usability, features, and performance to the email infrastructure that is run by big providers. Your IMAP access might be as good as theirs, but things like your webmail, your spam filtering, and almost certainly your general security will not be as good as they have.

In short, if you run your own email infrastructure, it will not be up to the general quality you could get from outsourcing to big providers (they can't really be called specialists). And you cannot fix this by trying harder, nor with the magical right choice of open source software, nor with the magical right choice of commercial software. Entirely "on premise" email is now an inferior thing for almost everyone.

Once upon a time this definitely was not the case. The big provider offerings were inflexible, limited, and often below the level of service and quality that you could achieve, except in limited areas like GMail's webmail interface. Although it's a bitter pill for someone like me to swallow, this is no longer the way things are. The big providers are crushing the field through the sheer amount of resources they can throw at problems, including and especially problems that matter to users like security and spam filtering. Getting even close to what they provide increasing requires climbing on board with things much like them (for example, a third party cloud service to spam-filter your incoming email). Especially, open source can't compete on features like webmail and performance in things like spam filtering.

(And let's not even talk about additional services like organizational calendaring.)

Artisanal email systems can still have some virtues, just like other artisanal things do. You can be more responsive to the special desires of your users and implement features they want (to some extent) and have high limits on things like mailbox space, and you're independent; some people will value that enough. Also, there will probably always be some people who use run and use independent email systems, just as there are people who use IRC.

(Our email system certainly supports a variety of features that the big institutional email doesn't and probably never will. But there are also things we just can't feasibly do, like add features to the webmail system we use.)

In the old days it made sense for a lot of organizations to run their own email systems. Often they had no choice about it, because there were no real alternatives that could meet their requirements at acceptable cost. But those days have been fading for years and are probably gone for most organizations.

(For individuals and perhaps very small organizations the picture is murkier, because you have more to lose to the capricious and random actions of big providers and fewer remedies available when they abruptly close your account.)

PS: The other issue with artisanal email systems is that the organization needs people who can run them, and those people have to spend time on email; generally the better and more artisanal a system you want, the more time it takes.


Comments on this page:

Still running my own mail server, one thing I refuse to give up is access to log files, to investigate what happened to an email...

In $DAYJOB we use Google, and there are frequent cases where our users send email to microsoft destinations, remote mail server accepts the email with 250, the recepients reports they never got it, but their IT team claims they don't know where it is... I find that unacceptable!

By Arnaud Gomes at 2021-12-25 06:15:24:

This is where I disagree with Chris: in my experience, big mail hosters, particularly Microsoft, are not really better at spam filtering. Their filters are just more skewed towards false positives.

In fact I think this is a recent development, maybe the last couple of years; Microsoft (and Google to a lesser extent) do not guarantee legitimate mail delivery anymore. In this, they are much worse than most other mail hosting options.

   -- A
By Private Email Administrator at 2021-12-25 12:24:51:

I run my own, primarily to learn how it all works, plus to avoid having all my email scanned for data to sell to brokers. However, it also allows me to see something of how the big providers handle email.

For example, Yahoo! will attempt to use SSLv3 (deprecated in 2015 for lack of security). If my system doesn't offer that, it simply switches to sending in the clear instead of using any of the more secure options my server offers it. ATT periodically blocks my system, sending bounce messages. If I follow the process in the bounce message, I find I'm not on any of the upstream block lists they use, and the admin who responds to me states that my server "is not blocked". Amazon's mail infrastructure bounced a message back to me, because it was looping through their systems. After 25 hops, they just reject the email. Oh, and the "Received:" headers gave me the names and IPs of 25 internal machines. GMail is not broken, just annoying. They will bounce an email saying "this message fails RFC822 checks", with no further details. And no awareness that RFC822 has been obsoleted for some time. Even Apple will periodically block my emails. They, at least, will respond reasonably and fix it within a day or two. In this case, it appears to be an over-zealous filtering system. It's only happened twice in the past 5 years.

I don't know if this refutes or bolsters your argument. I think it does show that an email system is hard to get right, even for the big providers. I think there's an argument in favor of running a small email system, though. Being smaller, and more targeted, it's easier to get right, and easier to keep right.

Private Email Administrator

By rephlex at 2021-12-26 02:34:27:

Sorry to hear you are unable to run a scalable, working email service. But it is a quite simple one man task to set up and maintain such infrastructure.

I do hope your article is mostly trying to generate discussion rather than being fully your experience, because that would mean that the internet is a cause lost to gatekeepers. But I do think that there is one reality about - lets' call it what it is - outsourced email that does ring true in a corporate context, and that is that issues that would be considered critical with services that are run locally are shrugged off with an "oh well, what can one expect?" when something similar happens to big providers. Users' tolerance, or perhaps senior management tolerance, of problems or inability to add features is much higher with outsourced services. If this were not so, recent days-long outages on many of the big platforms would have sent corporates scurrying back to local services, but that doesn't seem to have happened. To put it another way, imagine the outcome if a local email system was unavailable for days on end. Heads would roll, meetings would be had, projects would be initiated. Outsourced problems? Oh well, what can be done.

By Nils at 2022-01-06 05:28:11:

I have to agree with Arnaud, it doesn't seem like the big providers like Google or Microsoft have better spam filter, they are just filtering far more aggressively. If you run your artisinal mail server you'll often see your legitimate mail filtered by the bigger providers (due to "reputation") while at the same time receiving heaps of Spam from them.

The main drawback for using someone else's computer to run my e-Mail infrastructure is that they then also have access to the contents of my e-Mails (provided the sender doesn't encrypt). That's serious enough for me to continue running my own infrastructure for now. As long as there is a sufficient number of third party services I'm assuming/hoping I can continue to run my small setup.

Otherwise I have to agree, I wouldn't host friends on my infrastructure, at least not those used to using proprietary Webmail.

By David Matthews at 2022-01-14 10:22:31:

You do need a VM or some machine with a non-domestic always on connection, but otherwise I couldn't agree less. Any technically experienced person can set this up and maintenance is negligible apart from an annual dist upgrade.

If you're hosting you own web site, why wouldn't you also host your own email?

https://dmatthews.org/perfect_email.html

The more we give up on hosting our own email (those of us ready and willing to do it, at least) the more we cost everyone else the ability to do so. I've been running my own email services for a few decades. I'm not willing to be flotsam and jetsam within a global monopolists' systems - I want to control my own email destiny and demonstrate that it's possible for others, too.

After years of building my own systems from cobbled-together components, I've been running something far better (and yeah, it's also component-based and entirely Free and Open Source Software, but far more comprehensive and well considered than anything I've previously deployed): (Docker-based) MailCow instances for myself and my work and organisations I'm involved in.

Been using MailCow for several years, through multiple upgrade cycles. It takes an hour or so to set up on a $20/month cloud VPS. It offers all those things you suggest are hard/impossible for individuals to provide, e.g. calendaring, really nice webmail, solid distributed anti-spam (and, for the benefit of Windows users, anti-virus), searchability, and more. The only thing it doesn't offer is diverse outgoing server identities, so it is occasionally subject to being put on overzealous blocklists (e.g. of Cloud provider IP ranges)... but it just shows how capricious and generally 'too big to care' the 'big tech' providers are, and how they run roughshod over the whole idea of email. That makes me all the more adamant that we need to ensure that individually hosted email is possible, and we do that by encouraging those who are able to host their own to do so.

Written on 23 December 2021.
« Using pipx in a PyPy installation more or less just works
Sadly, my experience is that big commercial anti-malware detection is better »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Dec 23 23:55:55 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.