> I would say deliverability is the most important feature for email.
You're not wrong, but the thing is, the deliverability problem is not completely broken.
I don't have a problem sending emails with my self-hosted stack with DKIM, DMARC, and SPF set up. That said, some people do.
I lightly monitor my domains to make sure they're not on blacklists and have figured out how to get things working over time.
The battle isn't lost yet -- some people have problems and others do, but discouraging people from self-hosting is not the way to fix it.
My main point is that while deliverability can still be hard, many of the other things that made sending email hard (configuring postfix, dovecot and DNS correctly) have become drastically easier.
> But how much time do you spend dealing with that? Is it worth it?
Oh not much now -- to be honest with you 99% of the time when it's an issue it's because I changed something on my side.
Maybe I got lucky with the IPs I have (I use dedicated servers), and I've held them a while. What was jarring about the original post by Carlos is that he did all that AND he's been doing it 23 years and it was still a problem.
It made me recall that one of the things I struggled with a long time ago was just setting up Postfix/Dovecot correctly. These days no one should be running into that.
> I self hosted my email for more than a decade but I gave up when the time spent was just too much.
Unfortunately I don't doubt this assessment. I just want people to know that if difficulty/technical work was a blocker, it is now mostly not (with the news software out there).
As other people pointed out, there are services for improving deliverability that could fix the other (bigger) issue, and hopefully people could consider doing that going forward.
At the end of the day people have to do what's right for them of course but I just want people to know that things HAVE gotten easier. It's the big companies that are making it hard now.
But Carlos did not do "all that". He said he tried VPS servers, not dedicated servers. Maybe that's his problem. VPS servers tend to have IP addresses with low reputation, whereas dedicated servers tend to have higher reputation IPs.
Deliverability really depend on solid reputation of IP address blocks, including reputation of your neighbors. Ideally you want an IP near other IPs who also send high volume of non-spam.
I used to self-host, on my home internet connection, for 20 years. I used ISPs that were email-friendly, and gave me clean IPs. I started out with Sendmail, but switched to Postfic/Dovecot after a few years.
I stopped a couple of years ago, and moved my mail to my ISPs servers (they let me use my own domain). I switched because of the hassle of keeping the thing up, backing-up the mailstore and so on; and because I was moving home, so I wanted a service that couldn't be knocked-out by the move. Not because of any deliverability issues. I haven't had deliverability issues since about 2010, and those were probably because I couldn't be bothered with DMARC.
My ISP is a niche outfit, targeted at nerds; their mailservers are Postfix/Dovecot, which suits me. They support Sieve, and they have a control panel with various knobs and buttons. As an ISP they're a bit pricey, but for email hosting they're cheap.
One thing that was really buried in that article was this bit:
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned [emphasis mine], due to arbitrary reasons, by mistake, it doesn't matter. It's not if, it's when.
In other words, at least some of the time, the one of his users' accounts had been compromised, and as a result, his servers were actually sending out spam. But rather than take responsibility for detecting and blocking outgoing spam on his own servers, he's blaming The System for doing exactly what it was meant to do.
This exact thing happened to me. A test account I created with a weak password that I should have deleted but forgot about got pwned. I have spam detection in place and within an hour I disabled that account. But that was enough to send over 30,000 spam emails, and then your IP reputation is gone for a very long time.
The point being that outgoing spam detection won't save you, by the time you are pwned it's too late.
My solution is to have a script that scans the mail server log files and aggressively block ips of failed authentication attempts that are not quickly followed by a successful authentication attempt, and get immediate notification when there is a successful authentication attempt from an unexpected country (I think it's not the same bot that scans for vulnerable accounts and sends spam so you have maybe an hour to react).
That sounds interesting, could you share the script?
I've encountered a few obvious login attempts like that, but since they come from a broad pool of IPs it's not something fail2ban can easily handle without collateral damage.
It’s parsing smartermail logs which have a kind of funky format and the script quite tied to my setup (I have a central IP ban list because I also momitor non mail related protocols on multiple machines), so not sure it would be very useful to someone else.
What I find so baffling is that these experiences do not match my own atall.
I've been running a non-profit ISP for about 23 years with a few friends. We've always been doing this from our own IP Space (/20) in RIPE.
Even when we had problems sending spam (compromised user accounts, compromised php websites) and we ended up on a blacklist, we usually could get removed pretty fast.
For a few years our mail system would sometimes generate late-bounces, that is accept a mail on the incoming MX only to then figure out that it actually cannot be delivered later on and generate a delivery failure notification mail.
Not a good situation.
That got us into some trouble here and there. But even that could easily be unblocked again.
When we finally managed to set up a new mail infrastructure (2 year project cause it's a hobby) we set up new outgoing SMTP servers which cycle through multiple IP addresses. There was exactly one ISP (Deutsche Telekom T-Online) that was not accepting mail from some of these IPs. One mail and a turnaround time of abour 12hrs later this was fixed.
Gmail or Hotmail/live.com/Outlook never had any problems with deliverability. Even with a few users forwarding all their email to their gmail accounts including the spam that slips through our filters.
That might mean that a single mail would not be deliverd, but other users never suffered as our outgoing IPs are not being blanket-banned.
There's one residential ADSL provider that has a blanket ban on one of our outgoing IPs. There's no way to get that resolved because their mail infrastructure is unmaintained and nobody is reading their mail. Common problem with that one ADSL provider, googling their name shows other people have the same problem. shrug We just use a different outgoing IP for them.
No DMARK or DKIM setup at all for outgoing mail.
So I wonder, what really makes the difference in experience? Is it just the fact that we have a decent sized IPv4 Network in our name as PI space?
> But how much time do you spend dealing with that? Is it worth it?
I don't track the time but it's within a rounding error of no time at all.
I set up SPF for my domains when it became popular many years ago, took maybe a few hours.
Some years later I set up DKIM, that took a bit more time over a weekend. Some time later I set up DMARC as well.
I haven't had to make any changes since then, it's been at least four years. No problems of any kind. It's absolutely worth it to own my own email, no question.
Same, a bit of time invested upfront figuring out all the different tech: dkim, spf, failover smtp, letsecrypt certs (I am using IIS so not trivial plus you want to use those certs for smtp/imap too), log scanning and ip banning, etc.
But not accessible at all to non technical people.
> But how much time do you spend dealing with that? Is it worth it?
Not much, maybe on hour per month at most. I've been running Postfix on a dedicated server for 20+ years and don't need much time dealing with it.
But I've made sure to keep the same ip address when upgrading servers (possible with Hetzner) and besides configuring postfix to reject various connection attempts I do run spamassassin. Resulting in about one or two spam mails per week (moved into their own box by procmail in the filter chain) which I handle once a week by training spamassassin (takes just a minute).
The “secret sauce” for deliverability is _volume_. If you don’t send enough emails from your servers to each major email provider (at least 100 emails per day to each of these) you do not even show up in their reputation scoring systems which means whatever you do, you do not accumulate positive reputation score for your actions. Not an issue to get to that volume if you’re even a small ISP but quite complicated for these guys who manage only their own email.
That's not my experience; my self-hosted setup rarely sent more than 10 messages per day, but I could deliver to gmail, hotmail, yahoo. No problem, at least not since about 2009. Before that I did have a fair bit of mail dropped on the floor by the big providers. I spent quite a bit of time stressing about it. I don't think it's anything that I did that fixed it; it just seemed to fix itself.
> I would say deliverability is the most important feature for email.
That's only because true it's the one feature the oligopoly is withholding from self-hosters.
It's like saying "the most important feature of an operating system is the ability to multitask." Well, yes... but I'm glad we don't have to worry about that anymore when comparison shopping.
It’s like saying “it’s easier than ever to build a car! But you can’t drive it on the street”.
I would say deliverability is the most important feature for email.