Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Why can't I host my own email?
350 points by warent on April 27, 2022 | hide | past | favorite | 396 comments
I can host my own Mastodon server, or all kinds of other novelty / fun things which don't seem easily decentralized.

Email feels like one of the most decentralized internet concepts, and ironically it's seemingly the one thing I can't self-host unless, from what I've heard, I enjoy being permanently marked as spam / blacklisted. What's going on? How do we fix this?




The problem is that spam was/is so bad that extreme measures were taken to curb it. There are all kinds of invisible forces that you abutt that can be difficult to figure out, such as IP blacklists and the like. And even if you set everything up properly and host your email with a responsible host, Microsoft will still mark your mail as spam.

I host my own email server with Vultr on an OpenBSD VM using OpenSMTPD and Dovecot, relaying all outbound mail through SMTP2Go (their free tier more than meets my needs). I have all of the necessary DNS entries set to mark my mail as legit, and I sign all outgoing mail using strong 2048-bit RSA keys. Thus far, I'm able to send mail and not have it marked as spam (at least to everyone that I've corresponded with thus far). It was a lot of work to get to that point, but not terrible.


Prediction: Any distributed social media (like Mastodon) that gains mainstream popularity will share the same fate. Sure, you'll be able to host your own Mastodon instance, but 99% of people will be on the top 10 hosts and they won't peer with you.

I think the only way to make distributed social media practical is to have an extremely inexpensive turnkey self-hosting solution for the average person. A Chromecast-like device that they plug into their TV that backs up all their photos, plays music, and also hosts a Mastodon instance. Some kind of very friendly backup solution where you make an "emergency contacts" list, and the device encrypts all of your data and stores it on your emergency contacts' devices as a backup, and vice-versa.


cough XMPP federation cough

Not only did Facebook and GChat refuse to peer with little players, they refused to peer among the big players too. We could have had something like IRC for the masses, peered chat servers with bring-your-own-client. Instead, we waited decades for iMessage to get Android support which only happened long after everyone moved on to IG, Messenger, WeChat, etc.

Email is probably one of the last great open[ish] distributed systems we’ll ever see. There are just too many incentives to build walled gardens instead.


I'm not sure what the competitive edge could be to not wall garden. It's always going to be more expensive to try to work with those who don't work with you.

Going the self-host route, I'd still want a service of some sort so I didn't have to maintain it myself. Almost like an evergreen program that self-hosts my data and synchronizes and backups and transfers anything and everything.

Everything would be accessible outside of the program as local human readable or viewable files where possible. That'd be the best way to be non-walled garden.


> I'm not sure what the competitive edge could be to not wall garden.

I've had several email providers die since the 1980s. Each time it was a major disruption in my life. The last time, I coped by mostly dispensing with email whenever possible. Like most people I have a "good" email address for important things, that I check weekly, and a "garbage" email address I only bother to check when I have a need to.

Hosting my own mail server, not subject to some provider's ideas of filtering, or simply vanishing in the night, would make email more attractive. But the festering mess that SMTP email evolved into isn't just something you can set up in an evening. It's not even a hobby. It looks pretty much like at least a part-time job. Weighing the options, I don't really need email that badly.


> I'm not sure what the competitive edge could be to not wall garden.

If only one entity does it, as far as I can imagine it is only a marketing statement to appeal to a niche demographic - people who care about it from an ideological standpoint.

If more than one entity does it, it could lower the bar for critical mass. Instead of having to get enough people on your platform to start benefiting from the network effect, you only have to get enough people on your platform and platforms that you have bidirectional integration with.


It's actually fascinating to see the re-emergence of "vines" but as parts of other apps. With the explosion of Tiktok, every platform decided to make their own version - YouTube Shorts, Facebook/Instagram Reels, etc.

Really, a format should be created (e.g. file.[short|vine], etc.) that could be then edited by any editor and viewed by any viewer, and all that you'd need to do to copy a YT short to a facebook reel is to copy the file itself to each platform.

It's literally the same exact concept over and over again, just wall gardened instead. So much wasted development time doing the exact same thing.


Does this give credence to cryptoeconomics?


Exactly. GChat supported federation at the start, but they removed it later due to spam and other challenges [1].

[1] https://news.ycombinator.com/item?id=11795658


matrix.org has entered the chat

XMPP doesn't even support E2E encryption out of the box. It's as outmoded as IRC.


Lots of XMPP clients support E2EE/OMEMO out of the box now. If you think XMPP is outmoded you should try it again some time. I recommend Conversations on Android.


The problem with email is that identity and authentication are an afterthought. Don't forget that (in theory) it is possible to get any email server to relay a message for you. Newer protocols do not have these kind of problems.


>Don't forget that (in theory) it is possible to get any email server to relay a message for you.

That would be an open relay. That is simply not something that mail servers do anymore. If one was to deliberately set up an open relay, one would find that their email server was blacklisted pretty much immediately.


but then they just start a new relay.


I don't think so, I believe open relays are virtually extinct. People rarely run MTAs those days, and default configurations are quite protective. And if someone still manages to mess it up, they're gonna get famous with all the RBLs in days if not hours.

I self-host my mail for over 17 years. Most of the spam I'm observing those days comes from hacked/broken websites (sometimes it's probably some stolen SMTP credentials, sometimes sent from the server directly). Legit domain name, SPF and even DKIM present, looks totally legit in this regard - only stopped by RBLs and content filtering.


indeed, my ISP only recently closed their open relay for all customers

I remember back in the day having to change your SMTP settings whenever you travelled to whatever the ISP was where you were staying. then you could finally send email from your @homeisp.example email


If it was only open to their customers, that’s a closed relay, and typical for an ISP.


Open relays were a thing in the early 90's. I remember a friend of mine relaying email through 20 different servers, bang-path style. Any open relays today would immediately be used for spam, so they just don't exist, at least not for very long.


What was the original intent of open relays? Why allow emails without authentication?


> What was the original intent of open relays? Why allow emails without authentication?

Store and forward.

Do remember that email was THE great federated protocol.

The goal of a mail server was to get your email "at least one hop" closer to your destination. And that wasn't an easy task.

Servers came online and went offline. Users logged in and out. Connections came up and went down. IP wasn't the only transit. DNS? Oh, the hosts file? Even higher things--thing DECnet and Janet.

Email was barely functional most days. Your best bet if you weren't an Internet God and weren't able to write your own super complicated sendmail.cf was to know a sysadmin at a node who had an Internet God and ask him if you could forward emails that you couldn't handle to their server.


Email would be so amazing were it not for the spam problem. In the early days you’d just send a mail to your computer and your address was yourlogin@yourdomain and mail just ended up on your machine in a folder. Relays were like p2p networks. It was actually beautiful in its simplicity and in a perfect world with everyone being good actors could have been incredible.


And it was, back when any hint of "commercial use" could get your machine booted from the mail routes and usenet. After Cantor and Siegel, it was every spammer for himself.


Open relays were offered in the spirit of cooperation that was characteristic of the early internet.

Unfortunately, greedy people soon jumped in to take advantage of this generosity, resulting in a tragedy of the commons.

John Gilmore used to run an open relay, and I used to get spam from it. He was really stubborn about promoting the freedom of the spammers over the peace and quiet of the poor recipients. He eventually got shut down, still complaining.

http://www.toad.com/gnu/verio-censorship.html


> Why allow emails without authentication?

Because the people on the 'net were generally not bad actors back in the day, so we would you need to lock things down?


Hmm, now the internet is a dark forest. Leave anything open/visible, and all the bad actors immediately swamp you.


On Mastodon, I believe it's currently somewhat backwards from this. The largest instances are filled with Japanese anime porn, and the smaller instances end up blacklisting them.


Anecdotally, this has happened every time I've set up any kind of social media instance / discussion forum / BBS (back in the day!) / whatever. It immediately gets consumed by people who use it to host porn, and then all the intended users leave.


Have you considered creating a discussion platform where people can't post images, URLs / things that would be URLs if you added a URL scheme to them, ASCII-armored baseN-encoded anything, etc? For 99% of the discussion you want on the platform, text is all you need. For spammers and people who want to host porn, text alone is useless.


Plenty of spammers rely on text, though. A good chunk of my spam folder is text only.


Likely text containing URLs, though, which your email client likely helpfully auto-formats into links. Which is why spammers would bother. They need you to go visit something to make money.

There are types of bulk unsolicited email/web-comments which involve text without links, but they aren't spam per se; they're (the initial bait emails for) scams.

(And the usual solution to scam postings on forums, is to prevent people from sharing any off-forum contact details, except maybe via forum private message. No Nigerian prince is going to run their whole scam through the forum; they want/need to convert you ASAP into having a non-intermediated conversation with them. So they won't bother with any forum where they can't stuff their email address / Telegram handle / etc into each post.)


I've been itching for an experiment; might be fun to make a modernized BBS system for people to deploy.


If you're not spontaneously flooded with hentai, how do you even know your internet is working?


Japanese anime porn is not something that would trigger a banwave. A few posts from Trump, however...


Well, one of them is rotting the moral fabric of the country, and the other is just some hand drawn people having sex.


I was reading these tweets today confirming that this is already happening in Mastodon: https://twitter.com/Gravecat/status/1518598015396818944


This thread matches my experience with Mastodon and Diaspora*. It's fine if you are happy to live on individual instances and pretend that other instances do not exist, but they are not so great if you want a global audience. In this sense, they are more like the random disjoint online forums of the early 2000s, and not so much like the large monolithic social networks that people have come to expect.


Sounds like discord without voice and with easier linking. It does seem like forum approaches are becoming more common. I've heard that groups were the only part of Facebook with a lot of activity, but I'm not on that platform.


If anything, their story is more likely than not showing that the centralization is not going to happen. If the users of the instances were the ones doing the segregation (due to some tribal/cultural divide), then you'd end up with a small number of highly polarized instances.

But if this is only a fight between admins, the intuition is that we would end up with the big instances constantly losing users to smaller ones (created by those breaking away from the bad admins) who would then federate among themselves.


Even a magic dongle isn’t going to work. People don’t want to buy things, let alone sysadmin their own television.

I’d love a world where data was truly distributed and federated, but unfortunately, the barrier of entry is too high. Because of this people will start hosting nodes for people. Which isn’t necessarily a bad thing, but network effects will take over, and we’ll be back where we started.

Look at git. It’s distributed in all the right ways, but almost everyone uses github.

The web is decentralized, but the same few websites dominate to the point that people — even people on this very site — think that you can’t post a video except to YouTube.


Disagree. If the authentication mechanism are available from the get-go, it could work.


Unless you can reliably tie a user to their real-life identity, authentication isn't useful in this case. If a spambot tries to peer with my instance, it's not super helpful to know that their accounts will always be the same spambot and not a different one.


Identities could be signed by a centralized authority, which would have the same desired effect as centralized hosting without the drawbacks.


A person is not required. A reputation is required.


In a digital world, with no financial penalties, it's easy to build reputation with 'spurious' transactions and exhaust that reputation for one "Large Evil Event" and rinse and repeat.


I noticed a lot of German sites don't peer with anyone who has the exact same rules (as in almost literally the same) as them. I was surprised to see such kind of box-thinking in a protocol that's been designed to be as open as possible.


I assume this is unavoidable. The only solution are protocols where the network is owned and stored in the data (cryptographically) rather than in the servers. Then the servers apply censorship and rules over the data, but you can still rebuild any conversation chain as long as you connect to enough servers that don't censor it instead of requiring 1 server to keep all the network relationships.

This also allows authors to seamlessly switch servers without losing audience or at least being able to recreate it very easily.


That's another problem: Moving your account to a different server in the fediverse. Which is indeed not possible currently.

Perhaps some kind of blockchain would be a solution? (No, I'm not trying to appeal to tech investors, I actually think it might offer just the solution here :P )


nostr (https://github.com/fiatjaf/nostr) seems to be a minimal possible solution. It doesn't seem to be much in use though, so I guess once that happens a few issues will come up.


I hope matrix can avoid this fate. The federation can of course be limited, but currently it seems like most users are federated. And they have plans to combat spam etc. https://element.io/blog/moderation-needs-a-radical-change/


> Any distributed social media that gains mainstream popularity will share the same fate.

The experience behind this predated peer-to-peer electronic cash and related developments. You may be right, and it may still be too soon. But problems can be solved.


> But problems can be solved.

Looking at the 30 years and millions of dollars poured into making email work, the evidence seems to be against this


I don't agree.

There is no technical solution for people being assholes.

Well OK there is - turn off computer or server :)


One idea I've had is what if the protocol were designed in a way that a server can't be scaled too much, thus forcing lots of small servers to federate instead of having single entities running a large server with tens of thousands of users.


Arguably, your prediction is even a feature, not a bug.

The right to peer implies the right to not peer.


Agreed. I suspect most users have been tricked into thinking they want massive, global social media platforms.

(1) People are turning their noses up at Mastodon because all of Twitter isn't already there and because you'll be cut off from instances that aren't federated with yours.

(2) People are worried about "all of Twitter" becoming more people than they would like. There are communities they'd rather be cut off from and words they'd rather not read.

It's not a bug, it's a feature. Unfortunately, very influential companies that have figured out how to game our attention have tricked users into thinking they want something they don't.


> The right to peer implies the right to not peer.

No? Even if you don't want to force smaller instances to peer (which generally makes sense) you can apply more strict requirements to huge instances that contain a significant portion of the population.


> I think the only way to make distributed social media practical

The only technical way maybe. You could always legislate that large enough networks MUST peer.


Just because at one time the majority of users are federated, a market/threat force can enter any time that would drive users to centralised solutions.


If it's so easy to self host, surely attackers will host thousand of those instances and spam you.

PoW has been the best solution so far.


Can you give me an example of a PoW-based social network or chat application which has been more successful than conventional alternatives like Mastodon?


I am hardware stupid, but I have thought about this exact solution for so long. I hope someone figures this problem out!


General observation: no matter how distributed you make a system or protocol, it eventually becomes centralized.


Yup. Spam is the root problem. With an enormous amount of complexity between that and the mail admin's day to day experience.

I hosted my own mail for more than 20 years. A couple years back I just got tired of trying to solve deliverability puzzles, plus the fears that deliverability issues generate. (E.g., "Did that potential employer get my email about the job?") Especially since some of the puzzles are not solvable, like why GMail does what it does. I even had friends at Google, and I still couldn't find out why GMail occasionally didn't like my server. And arguably, that's the right choice for them, as the more spammers know about how they work, the worse it is for Google staff and GMail users.

For me, switching to Fastmail hosting was a big win. It's not like I'm out of technical challenges to solve, but I get to apply that to things where the upside is greater than, "The thing everybody expects to work still works."


the spam problem advantages google, as your own story illustrates, so it's unlikely they'd really want to help solve deliverability/spam issues systemicly. making personal email hosting more difficult means they have a chance to capture your email data streams via gmail. whether you switch there or not, it creates a pressure for most to aggregate on gmail, which means they can see most email exchanges.


For sure. Good spam filtering was one big reason for people to switch to GMail. And a lot of people who gave up hosting their own email have switched to Gmail as well. I'm sure this doesn't rise to the level of conspiracy, but there's little incentive for them to fix the broader problems.


My issue with fast mail et all is storage is so unnecessarily expensive. I have many gigs of email that I don’t want to lose, but I also don’t want to pay many tens of dollars/month to host it.


I use SES just for delivering emails while hosting the emails myself. Feels like an optimal solution.


> I have many gigs of email > don’t want to pay many tens of dollars/month

How many gigs? It appears that the 100GB plan is only $9/mo.


Whoa. That’s very different from the last time I checked, perhaps they changed either their plans or their structure. I saw it previously as additional storage I’d have to add. 100G would be enough! Wow thank you!


Is it? I pay $5 for 50gb, which is many years of email (including all the spam/newsletters, which I really don’t need to keep)


If you have iCloud you have the option of using a custom domain. I think I only pay a few bucks a month.


For various reasons I’d rather have an indie provider. I’m currently on Google and trying to decentralize my data.


Ok, if the amount of data is a problem, another option is to just download your mbox from Google takeout, treat it as an archive, and use something like notmuch to search your old mails. You can then store and backup the mbox anywhere you want.

In addition to the iCloud, I have a free ProtonMail account, which I use sparingly, and anything large or important that comes in there I move to local backup and delete. 99% of the messages I get can just be deleted. If you set up pop3 you can auto delete from the server. That’s the old school solution, but it depends on your use case. The pop3 option doesn’t work well with multiple devices.


Maybe self-host an IMAP server, and periodically archive from FM to your box/VM?


> And even if you set everything up properly and host your email with a responsible host, Microsoft will still mark your mail as spam.

I did some experiments back when I ran my own mail. Sending from my mail server to my Microsoft account it not only marked everything as spam, it continued marking everything as spam after I marked a bunch of them as not spam.

After that, I tried also answering several of them and composing several new mails to send to my non-Microsoft email to see if Microsoft's spam system was smart enough to figure out that if I'm actively corresponding with someone their incoming mail should not be marked as spam. It was not smart enough.

Then I tried whitelisting. Nope, still spam.


Microsoft had marked an email from a professor from an vt.edu domain email address as spam causing me to miss an interview for a PhD funding.


Microsoft is especially notorious for flagging legit emails as spam if they are not from one of the regular providers.


Flagging if you're lucky, they outright 550 refused my mail until I joined their sender program and applied to have my domain unblocked. Then they proceeded to gaslight me claiming my mail was never blocked even after I forwarded their own error messages and IDs back to them.


You got a 550? Lucky! When I worked at a non-profit that hosted our own email server, we had many instances where Microsoft would /dev/null our newsletter e-mails. Their servers would give a 250 indicating acceptance, but the e-mail was nowhere to be found (and yes, we checked the spam folder).


yea unfortunately I have seen those as well. It is ridiculous at times.


That's nothing compared to the joy of dealing with legit emails that are flagged as high confidence phishing.


Regular provider == (Microsoft 365 || an Exchange Server)


Honest question- why can't people sue for this?


On what grounds would they sue for? Email is not the post; there is no legal right to receive one or to have one routed.

If one wants such legal protections, there is the post.

(Now, should there be such a right? That's an interesting question. But a world in which one exists would raise the bar to starting one's own email server even higher).


> On what grounds would they sue for?

Negligent interference.

https://en.m.wikipedia.org/wiki/Tortious_interference


Possibly, but it would be a hell of an uphill battle. There was no contract in place for the email provider to negligently interfere with. And the email provider's operation was perfectly regular and within the bounds of the standards of that service (which offers no delivery guarantees).


You can't sue if a product doesn't work as intended and results in harm?


It depends on the circumstances. In some cases, when guarantees are made and those guarantees are broken, you can sue civilly to be made whole (in a context like this where there was no bodily harm, merely an opportunity missed).

It's real unlikely any such guarantees were made. To do so would be extremely foolish for several reasons (the false-positive rate of spam identification is known and emails can fail to deliver because of an error at either end of the transaction).


You can't sue someone (and win) for "this person did something that I don't like". You only have a case if you have a contract with them that lays out specific duties, or if they are otherwise a fiduciary of some form. Unless you signed a contract with microsoft for them to deliver your mail, they have no obligation to do so.


A legal protection would mostly entail disabling of spam filters.


Email (SMTP) has no delivery guarantees. It's basically "best effort."

If you want guaranteed delivery with proof and tracability, send a registered letter at the post office, FedEx, etc.


unfortunately, access to the Internet is not well defined enough for this, and you basically have no right to a connection or any guaranteed privileges if you have a connection, which sucks.


No guarantees clause in the terms of service you accepted to use their product.


I was using free email, I don’t know if MS has any obligation


Former VT employee: It's not just Microsoft. Virginia Tech has a problem with their stuff getting flagged as spam.


I can't find it right now but a few years ago there was a post here on HN by a grad school admission officer who recommended to never use Gmail for applications, no matter what school you apply to. Apparently, it's anything but uncommon for emails from @*.edu to end up in spam.


Yes to the OP, you most definitely can host your own email fully.

Many of us do it. If you have any interest in the topic, either due to the fun of managing the servers and learning something along the way or due to the moral high ground of supporting decentralization above proprietary walled gardens, do it!

Ignore the naysayers, if you're interested you can do it.

Will some emails very occasionally end up in the spam folder of a recipient? I mean, yes, but that is true of everything. You can end up in spam folder sending from Microsoft Office mail to gmail or vice versa. Heck, every now and then an email from my manager will end up in my spam folder in gmail even though he's emailing me from gmail to gmail, both of us in the same corporate gsuite account! So on average, once you set everything up correctly, your deliverability will be as good as gmail to gmail, which is to say not 100% perfect but no worse than any other solution. And you'll be in control of your email infrastructure and address. No longer will google/microsoft/apple/yahoo be able to cut you off all your accounts on the whim an AI gone bad.

The parent post mentions a useful safety valve to know about if you're worried about deliverability and want to take baby steps to get there. You can always, either selectively or wholesale, use a commercial relay for outbound mail from your email server. Some have free tiers that are plenty for personal/family use.

Personally I don't use any third party relay, I deliver to everywhere from my own infrastructure. No issues.


Me too, and this is my experience as well. In the rare event that I find out that someone isn’t getting my emails, I tell them that they should complain to their provider or use a different one. I’m no longer willing to jump through hoops so that hotmail delivers my email.


> There are all kinds of invisible forces that you abutt that can be difficult to figure out

This was my main experience, and all I did was try to set up the ability to simply send emails to myself (gmail) (and no-one else). Things like: this script crashed, or btrfs scrub finished + scrub results, and similar.

The first thing I tried was just setting up a VM with postfix running on it locally with my residential ISP. I don't even remember what the error was for this scenario, but it was just totally dead in the water. Absolutely zero mail delivery. I think I eventually figured out it's because google defers to spamhaus, and spamhaus says residential IPs = hard no.

That next thing I tried, and what I ended up doing, was writing a docker container that just runs an SSH port forward to jump from my local network to a digitalocean host, which is where another docker container runs postfix. I had done this bit once before, and I tried to just set up DKIM (since DKIM was, to my reading, basically bulletproof - why bother with SPF when you have real cryptographic identity assurance?). This led to weird error messages from google about my IP having a super low reputation. This was something I'd been worried about so I spent a bit of time trying to cycle my IP. But I eventually figured out it was just a bad error message and setting up SPF suddenly made my emails start delivering.

My main ongoing issue is that I had to add all my sending addresses (things my internalhostnamehere@myrealdomain.com) to my contacts in gmail, otherwise there was like a 50% chance they'd just go to spam. I've been running this setup for about a year and it's still a coin toss whether emails will come through fine, or if they'll say "this would've gone to spam but it's in your contacts". When that happens, I check the DKIM and SPF status in "original message" in gmail, and gmail itself says they both passed.

Absurd tbh.

For my "not self-hosted but better than letting google own my digital identity" solution, since I use apple icloud+ or whatever it's called, I set up the SPF stuff to let me send+receive email from my custom domain, so while icloud could still scan my mail, at least if I get banned, I still own the actual domain and could move somewhere else.


Even if one setups everything by the book (SPF, DKIM, DNS.) etc. No one at @outlook.com will receive email, based on my experience. Thus, it does not work well if email is important for business-to-business use.

Outlook and Gmail are basically having opaque rules who can receive email and there is no process to get “whitelisted” on these receivers.


I had exactly this problem too. I elaborated below.

If you keep an eye on your logs, when your emails are being blackholed (it accepts them but it does not deliver them!) it does provide a link in one of the 550 status messages, where you can get yourself unblocked. I've elaborated here: https://news.ycombinator.com/item?id=31185297

However this only works temporarily, after a month you're back in the doghouse. Only senders which send a large volume of legit traffic are allowed. It's ridiculous but sadly true.

Edit: I found the message in my old emails:

---

550 SC-001 (BAY004-MCxxx) Unfortunately, messages from XXX.XXX.XXX.XXX weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

---

In that link the "SC-001" code also refers to that reputation thing. This was the same at outlook.com / hotmail.com and live.com . It did not, however, affect corporate customers using Office 365 / Exchange 365. Only customers of MS' consumer offerings.

My "internet service provider" was a legit colocation service and nothing funny was going on in their network by the way. Microsoft was the only party that had issues with my server. All known blocklists had no issues with it. It was just MS being difficult and making up their own rules.

Anyway going to that link there is a form somewhere to temporarily unblock it. Give it a try.. Perhaps you can create an account at live.com yourself and send a daily test email or something... I thought of doing this but eventually I got so frustrated I gave up on it.


> Only senders which send a large volume of legit traffic are allowed. It's ridiculous but sadly true.

That’s the thing I’ll never understand: why would one have to spam email to be considered not spam?

I wonder at which point it is legitimate to go full conspiracy theory, and suspect they’re just trying to block the little players so they can keep their relative monopoly. Maybe they don’t do it on purpose, but the way their anti-spam measures make life real hard for the little ones sure looks convenient.


If you get a 550 then it did NOT accept the mail. What grandparent is talking about would be the server replying with 250 OK and then neither delivering the mail nor sending a delivery failure notification.


One potential 1%-of-the-complexity answer to the problem of personal notification (which I presume was what email-to-self was solving) is to set up a Telegram bot. I recently really wanted realtime notifications on my phone (package tracking) and realized that all the top-level "send notifications to phone" type services are either mass push notification shops ($$$$) or bundled offerings ($$$) that were entirely overkill for my purposes.

There are two ways to run a bot on Telegram, either by running the bot client directly (meh, interesting but extra setup) or by using Telegram's bot hosting system that works over HTTPS. It's the second approach that takes 3 minutes (!) to get to an MVP state for notifications.

- You walk through a flow with a specific account (@Botfather) on Telegram to create a new bot account, which gives you an API key

- Find the new bot using the search function then open a conversation with it and (after sending /start) send a junk message

- Call `curl "https://api.telegram.org/bot$APIKEY/getUpdates"` and fish out the "chat"->"id" value from the JSON representation of the message you just sent to obtain your user ID

- Call something like `curl "https://api.telegram.org/bot$APIKEY/sendMessage" -X POST -H 'Content-Type: application/json' -d '{"chat_id":"1234567890","text":"boop"}'` (set chat_id to your account id) to send a new message - yup, it's literally this simple to send messages

- Go into Telegram's settings and add the bot as a notification exception (assuming you have notifications universally turned off by default)

- If you also set the full-screen popup to "when off" Telegram will (even when your device is locked) show an instant notification containing the sent text

- Because this is a conversation, the message history will be preserved unless you explicitly delete the messages (which you can do on a per-message basis)

- The Telegram bot API supports both polling and push-based I/O, where you can periodically poll /getUpdates or have Telegram call a webhook you configure. IMHO the way easier approach is just running the bot client locally at that point, *but*, for just sending out one-way notifications where replies don't matter, the default polling setting (no webhooks) is ideal as the bot server will delete un-acknowledged messages after IIRC 24 hours or so - so you don't have to worry about queue quotas or whatever, you can just ignore the whole receive side and it just works

Obviously the caveat is that this is 1% of the complexity and equally 1% of the... provenance, for want of a better way to put it. But in terms of "I need realtime notifications now" I am yet to find a better system. It worked perfectly.


This is the answer. Blocked emails happen for random reasons and fixing them is a black art that involves talking to ISPs and stuff. It's really too much for an average person to handle.

At work we've had issues with email delivery due to things like outdated IP block lists at some random ISP four hops away, only impacting deliverability when mail gets routed through that part of the web.


I have an email address with "spam" in the name (this is through gmail) and lately I've had all kinds of problems with emails to it disappearing - I've had to call several places and have them change my email because I can't log in and the reset emails don't ever show up... but changing to myfirst.mylast works fine.

I've run into this with both Sam's Club and Speedway Rewards.

Only thing I can think of is that some outbound mail service they're using is dropping them, or some relay in the middle is dropping them... I can see where the word "spam" would be a keyword you might use, but I've had this email address for 15 years now and it's only been a problem in the last few years.


I have this part:

> I host my own email server with Vultr on an OpenBSD VM using OpenSMTPD and Dovecot

But with outgoing mail being relayed internally to dkimproxy which signs it before being relayed back to OpenSMTPD for delivery to the other email server.

I had to set up SPF and DKIM DNS records, and one time I had to request that my IP be removed from the Abusix blacklist. Other than that, it's pretty rare for my emails to be marked as spam. Outlook 365 seems to do it much more often than Gmail though.


That's very interesting. I never thought to relay mail internally to dkimproxy. I'll have to give that a shot. I like the idea of hosting the entire solution myself and not relying on any 3rd party solutions, but relaying through SMTP2Go was the only thing that I tried that actually solved the problem. Perhaps this will offer a good solution! Thanks!


I also use the dkimproxy package, but there's now a third-party OpenSMTPd module that can sign messages in-line.[1] I've always found dkimproxy setup a little confusing compared to a built-in/in-line solution. I might try to switch to the module during the OpenBSD 7.1 upgrade process.

[1] I think this is the one I had I mind, though I didn't realize it was already in ports: https://cvsweb.openbsd.org/ports/mail/opensmtpd-filters/dkim...


If you run a mailing list you generally have to worry about ARC (re-signing 'chain of custody') in addition to DKIM:

* https://en.wikipedia.org/wiki/Authenticated_Received_Chain

I've found ARC to fiddle some to get going than ARC.


> relaying all outbound mail through SMTP2Go

So it's not an entirely self-hosted solution, is it?


No, but it's quite difficult to have email reliably and consistently delivered to Gmail and other major email providers without sending it via a relay. The relay provider is in the business of maintaining IP addresses with good reputations that aren't blocked by spam lists etc. If you can find and keep a reputable IP address, then you're fine, but it's usually easier to pay someone who does that for a living—you have no guarantee that the IP address assigned to you by Digital Ocean or whoever wasn't used for spamming at some point.


Digital Ocean has an extremely poor reputation over a long period to the point where their droplets are blocked on mass in many places now [1]

Even my local ISP refuses mail from them.

[1]: https://discourse.mailinabox.email/t/digital-ocean-ips-being...


Really sorry, I don't normally nit pick spelling and grammar, but it's "en masse" rather than "on mass".


This is also why I went with Vultr as my server host. They block port 25 by default and make customers file a support ticket with them to unblock that port. They also require your account be active for at least a month and be using their service in good standing during that time. Wasn't an instant process, but was simple enough to accomplish in the end.


Yeah, lots of places just straight up block entire IP ranges, such as anywhere you can get a VM for cheap/free, or residential IP ranges, etc.


> The problem is that spam was/is so bad that extreme measures were taken to curb it.

Man, and there's such an easy solution, too - just use Hashcash[1] (invented in 1997) and 90%+ of spam disappears overnight (if not more, depending on how high you set the difficulty).

Well, ok, "easy" in the sense that We Have An Algorithm For This - it'd still be hard to get email clients/servers to agree on a protocol...

[1] https://en.wikipedia.org/wiki/Hashcash


Ah yes, time to break out this old classic: https://craphound.com/spamsolutions.txt

  ( ) Requires immediate total cooperation from everybody at once
  ( ) Unpopularity of weird new taxes
  ( ) Public reluctance to accept weird new forms of money
  ( ) Huge existing software investment in SMTP
  ( ) Sending email should be free


It's fortunate that none of those are "checked" in the ASCII art, because none of them actually apply.

> Requires immediate total cooperation from everybody at once

False. As a silver lining to the Google/Microsoft email oligopoly, those providers could announce that anyone wanting to send email to those services will have to implement this protocol, and it could be done in less than a year.

> Unpopularity of weird new taxes

Irrelevant. No taxes involved - there's no money here, and users won't care if their mail takes an extra few seconds to send, because they don't expect email to be low latency anyway

> Public reluctance to accept weird new forms of money

Irrelevant - no new money involved.

> Huge existing software investment in SMTP

Irrelevant - a small number of server software are used by the majority of users. Also, see earlier point about oligopoly.

> Sending email should be free

Bad idea, and irrelevant, because it still would be.

I suggest you put thought into copypasta before putting it into a comment.


If there's a compromised machine it will be the victims paying the cost in energy bills for spammer's nefariously installed malware to send garbage.


If there's a compromised machine, the scammer can drain the victim's bank accounts and cost them far more than an electricity bill, and/or mine cryptocurrency directly. Regardless, their spam-sending rate will still be significantly decreased.

So, this argument is completely invalid.


> If there's a compromised machine, the scammer can drain the victim's bank accounts […]

Not if the machine is a server and was gotten into via (e.g.) a bug in a web app. I don't know about you, but I don't keep my bank account information on the LAMP systems I sysadmin.


That part of the post was specifically about consumer devices.

You missed the rest:

> and/or mine cryptocurrency directly. Regardless, their spam-sending rate will still be significantly decreased

Your argument remains invalid.


Is there such a service that will tell me the reputation of an email domain, i.e. whether mail originating at that domain would be likely to be treated as definitely spam or not? (I don't really care about "no reputation"; I want to know if a domain has known bad reputation.)

I feel like, if there was such a service, it would be pretty useful to use it to prevent account registrations on other services, from users whose email addresses have domains with bad reputations. After all, they'd very likely just be registering with the intent of using the service to send or post spam in some way.


multirbl.valli.org

Contains blacklists on the domain level, also on the ip block and AS level.


Possibly interesting... but these are rules about outgoing SMTP servers (MSAs), yes? How much of a relation does the outgoing SMTP server for a domain have to the canonical set of receiving SMTP servers (MTAs) for the domain held in the domain's DNS MX record? These can certainly be one and the same server; but it's not a requirement. So how often are they in practice? Especially for people actively trying to evade these sorts of RBLs?


> The problem is that spam was/is so bad that extreme measures were taken to curb it.

The problem with spam is that there's no real legal recourse for spam. If it's in your own country then maybe. But outside of your country? Well the easiest thing to do is to IP block and the next best thing to do (when IP block isn't an option) is to use some sort of "smart detection" to put spam into a special box labeled "spam". There's no deterrence and literally no criminal prosecution for spam.


Also, one should subscribe to the mailop mailing list, which serves as a Distant Early Warning line for email deliverability issues (ie. like NANOG for netops issues).

https://www.mailop.org/best-practices/


zddziura, I want to say a big thank you, thank you, thank you for pointing me to SMTP2Go. I have been trying to get my DMARC and DKIM email woes solved for months, but couldn't get it figured out. When I read your post I signed up with SMTP2Go at the free tier and I had a 100% Mailgenius score in less than an hour after I set it up. So awesome! No more big yellow warning boxes in Gmail when receiving mail from my own domain! Yea!!!


Spam got significantly worse but this is also an chance to curb the federalization of mail by large companies. Of course they would like you to use a Microsoft or Google account to send mail.


Any chance you'll provide a detailed write-up of your experience with tips and whatnot?


I've been self-hosting email for about 20 years, from a dedicated server in Europe. The server hardware has been replaced a couple of times but kept its IP.

If you set everything up right, and choose the host for your mail server carefully, and never change IP, after a fairly short time you won't have much problem with being marked as spam. No more so than with any other email host.

As is so often the case, the people that say you should never do it probably have little relevant experience, they are just repeating something they heard.


> If you set everything up right, and choose the host for your mail server carefully, and never change IP, after a fairly short time you won't have much problem with being marked as spam. No more so than with any other email host.

This is untrue. If you are the only person using your email server, your volume will be so low that the big providers (Gmail, Outlook, etc.) won't track your reputation. So, ironically, being a low-volume sender means your email will be constantly classified as spam.

I speak from experience: https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...


"This is untrue" and yet I and others in this thread have been doing this for a long time without encountering the issue that you so confidently claim exists.

My email server is used by two people. Reputation is tracked by all the big providers, as evidenced by a) my email not being classified as spam, and b) them showing reputation of my domains in their various reputation dashboards.

"Those who say it cannot be done should not interrupt those that are doing it."


> "This is untrue" and yet I and others in this thread have been doing this for a long time without encountering the issue that you so confidently claim exists.

When you make a claim that supposedly applies to all people, a single counterpoint is sufficient to disprove the claim. It's as if you had said "all rabbits are black", then I showed you a white rabbit to counter that not all rabbits are black, and you come back with "look, I have a black rabbit here". How does that make sense to you?

> them showing reputation of my domains in their various reputation dashboards.

I never got access to their dashboards because my email volume was so low. If you somehow did, good for you.

> "Those who say it cannot be done should not interrupt those that are doing it."

I'm not "interrupting you from doing it". I'm interrupting you from giving bad advice to OTHER people.


> When you make a claim that supposedly applies to all people, a single counterpoint is sufficient to disprove the claim. It's as if you had said "all rabbits are black", then I showed you a white rabbit to counter that not all rabbits are black, and you come back with "look, I have a black rabbit here". How does that make sense to you?

Well said!

Your claim is "it is not possible to self-host your own mail on a low-volume server and not get consistently marked as spam by GMail / other large operators". The existence of a single person successfully doing exactly that (and there are numerous such people in this very thread) is sufficient to disprove your claim.


> Your claim is "it is not possible to self-host your own mail on a low-volume server and not get consistently marked as spam by GMail / other large operators". The existence of a single person successfully doing exactly that (and there are numerous such people in this very thread) is sufficient to disprove your claim.

Perhaps that was their claim - but I've generally read advice as: "There's no predictable way to guarantee that any given person can today take over hosting their own mail with predictable and good delivery to Gmail and o365."

So just that a, b and c have, so far, good delivery from their setup is not a guarantee that person x can just "set things up correctly" and somewhat straightforwardly get good delivery.

Last I did it, I had to go via undocumented api/pages for both o365 and Gmail in order to improve delivery - and mail that gmail/o365 smtp servers swore they accepted without problems - still sometimes ended up as spam, or simply vanished after delivery.

This was all individual low-volume. Never found any reason for it.

That said, I'll probably go back to hosting my own mail, and just live with certain parties being bad net citizens, eating the occasional mail without error or bounce. It's not like I really expect them to do better. Although especially in the case of Gmail, it's a little like Disney eating up public domain stories and spitting out copyrighted and trademarked content. Google did a lot to force people away from proper quoting (by hiding the fact of how Gmail quoted things in the "friendly" ui) and they pretty much killed Google groups - after marginalizing alternatives. But those ships have sailed.


You're correct, I shot myself in the foot there. But can we agree that some people manage to successfully run their own email servers and some people don't?


Oh, absolutely. I wouldn't recommend it casually to everyone, but if someone says "hey, I want to learn to host my own mail, but everyone tells me I shouldn't", I'm totally going to recommend they do it. If they have the desire to learn, it's likely (not certain! but likely) that they'll succeed.

I do think that most of the effort/risk is at the beginning. Making sure you're on a reputable provider, checking the history of your IP, setting your mail server & the security features up correctly, monitoring deliverability etc.

After everything is working well, if you got that part right, the ongoing effort should basically just be keeping software up-to-date. You could always get unlucky and e.g. someone starts sending spam on a nearby IP and you have to waste some time dealing with that, but hopefully if you picked your provider well that won't happen. It's yet to happen to me, but my provider only offers dedicated servers, which are probably not so popular with spammers.


> Making sure you're on a reputable provider

Oh, that tiny little detail where it has become impossible for almost anyone to actually send email from home, either because residential IPs are all marked as spam, or because your provider is giving you an incomplete internet connection that blocks outgoing SMTP.

> You could always get unlucky and e.g. someone starts sending spam on a nearby IP

Can someone tell me why providers ever thought it was a good idea to block entire IP ranges?


> Oh, that tiny little detail where it has become impossible for almost anyone to actually send email from home, either because residential IPs are all marked as spam, or because your provider is giving you an incomplete internet connection that blocks outgoing SMTP.

Well, by "provider" I was referring to the hosting provider. I wouldn't typically recommend hosting from home [0]. Apart from the issues you identified, over the course of 20 years I've moved house multiple times so would have had email downtime and likely had to change IP address, likely leading to reputation starting again from zero (at best) or picking up an IP with bad history (much worse).

> Can someone tell me why providers ever thought it was a good idea to block entire IP ranges?

If you repeatedly receive spam from multiple IPs within a given ASN, and the abuse contact is non-functional or doesn't actually resolve the problem, it is fairly reasonable to consider that the ASN is friendly to spammers. Blocking by prefix rather than entire ASN is a bit less of a sledgehammer, but follows the same logic.

[0] If you own your house, have no plans to move any time soon, and can get a connection from a provider who is willing to give you a stable static IP that isn't categorised as 'residential/dynamic' by the major lists, hosting from home should be fine. But that's quite a few 'if's.


Their requirement for it applying to all people make it generally universally true. If you have a dedicated IP with reverse DNS pointing to an A record on the primary domain that server is responsible for, and you have SPF, TLS and DKIM running smoothly, it’s very unlikely any spam markers are due to the fact that you’re sending mail from <regular Ip> instead of <Microsoft 365>. This doesn’t mean you’ll never be in the spam folder, it means your spam reputation is based on the content and frequency of your emails.


Not an expert on this, but maybe this is because you have been hosting your server for 20 years? Maybe newer servers have a higher threshold to cross? Seems a logical hypothesis to me, which would mean you're both right.


I selfhosted my email server on the same IP address for 10 years and never got my reputation tracked. I remember once reading that they needed something in the range of a few thousand emails going to their servers. I swear it used to be listed on the google dashboard but like everything else email related they hide this information to "protect from spammers". Sounds like an excuse for anti-compeditive behaviour to me


> being a low-volume sender means your email will be constantly classified as spam

"will" is a strong word. I've read that very low volume sending server can sometimes have issues, but never experienced it. My outgoing volume is about as low as it gets since it's just me and some family that don't use it much, but don't experience any problems.


Have you measured your deliverability?


I have a low volume mail server and don't have any problem with this.


Is this a feeling or did you actually measure your deliverability?


> As is so often the case, the people that say you should never do it probably have little relevant experience, they are just repeating something they heard.

More likely, they're saying that 99% of people don't know how to self-host, and for 99% of the rest it's not worth the trouble. Also, if you have to ask, then you shouldn't self-host it.


"If you have to ask, you shouldn't be doing it," said the tired old King of Gatekeeping.

For a second, I thought I was on Stackoverflow. If you aren't starting by asking questions about the possibilities or limitations of a system you're about to work in, then you aren't starting properly.


That adage exists because those who really want to do it won't be asking to be spoonfed and will instead start doing the research themselves.


I get why it exists, but we also have to be honest with ourselves and admit that there is a massive problem in communicating without assumption of prior knowledge/experience. The tech world is particularly guilty of this, with veterans having forgotten that there was a time when they didn't know what they know, and lacking any patience to adequately help those seeking understanding. To add to the issue, the Internet proper is so noisy with partial or broken information that the task of finding the correct information is far more daunting than it was 20 years ago when I started learning.

We need to give newcomers a break and answer their questions well, and discuss to promote understanding, instead of swatting at them with our canes. The only way knowledge passes to the next generation of thinkers and tinkers is if we fuel that curiousity.


Running an email server, that is secure, is not easy. In the mean time, before you're qualified and know what you're doing, your mail server is a danger to others. From being an open relay through just plain unpatched security vulnerabilities. It's not the consequences for you, it's the consequences your fuckups have for everyone else.

There's plenty of other tech they can screw with.


For perspective, some may perceive doing research as being spoon-fed. Sometimes, you don't know where to start and need a hand. At one point, I didn't know how to turn on a computer, and now I'm a software engineer.


The area of 'i barely know how to keep a server running' is a problem for everyone else on the internet as your mail server starts relaying a deluge spam and phishing emails


Asking the question in an open forum is doing the research on it.

I've wanted to do this for a long time, and am using this very HN post as a source of information. I don't expect anyone to set it up for me, but I am hoping to identify the "gotchas" that are likely not discoverable by reading man pages.


Except Google and StackOverflow exist now. You start by searching and reading tutorials, not by asking other people.


People who have been hosting their own email for decades like GP probably built up a solid reputation for their domain and IP before spam filtering became such a kafkaesque business and IPv4 blocks became so fragmented.

If you start self-hosting now, you should be prepared to lose quite a few emails randomly for the first X months while everyone else tries to figure out whether you're legit or not. Though I would encourage anyone who can to try to self-host at least some part of their email infrastructure, even if just for the learning experience, I would also recommend that they avoid using self-hosted email for anything business-critical until they're sure they've got the hang of it.


Tip: Start with a solid SPF, DKIM, DMARC policy and register for microsoft, yahoo, etc.’s admin tools and add your domain to google’s webmaster and postmaster tools. (Yes, even if Google postmaster won’t show you anything yet)

Use mail-tester.com or similar tools to ensure everything is configured correctly.

And then just start sending. As long as your volume grows slowly over the first few months, you’ll get basically no rejects.


Yep, it takes patience and lots of trial and error to build and maintain a reliable email server, unlike an HTTP server or Minecraft server which you can fire up with a script any time you want. Probably explains why so few people do it successfully.


I know about the Google postmaster tools but I'm coming up blank finding anything about Microsoft and Yahoo. Do you (or anyone else) have links to these?


Yahoo has merged theirs with AOL (kinda) and currently they only seem to offer https://senders.yahooinc.com/contact/#complaint-feedback-loo...



What is the yahoo version of SNDS (the microsoft thing) called?


> you should be prepared to lose quite a few emails randomly for the first X months while everyone else tries to figure out whether you're legit or not

And then prepared to lose quite a few emails consistently for the next 10 years when some decide you're not legit.

Source: I self-host.


Don't forget about attacks. If you lose control of that domain you are pwnd.


I'll definitely take that (which is largely under my control) over e.g. Google deciding that I've done something wrong one day and cutting off my email.


I agree with your sentiment, but It's not under your control. It's under the registrar's control. I'd argue registrars are way more prone to social engineering attacks than google is. I also don't use Google as my email provider though.


The choice of registrar is under my control, though.

Also, little-known fact: if you register a UK company (probably more practical if you already have one, but the effort is not actually that big), you can register .uk domains directly with Nominet, the UK registry, by setting yourself up as a self-managed registrar. It doesn't cost anything (beyond the cost of the domain name) and is very easy. I'd love to know if there are any other registries that allow something similar.


Man, this is really cool. Email is central to online identity and sounds like you have more control over yours than anyone I’ve heard of.


I'm in the USA and would totally bite the bullet and do something like that, if possible


> More likely, they're saying that 99% of people don't know how to self-host, and for 99% of the rest it's not worth the trouble.

Certainly way more than 99% of the general population wouldn't know how to self host, but within a techie population like HN, easily ~50% can be capable of doing it if they wanted to. Whether it's worth the effort is a personal decision, but there's a lot of value in owning your own email so I recommend it to anyone who's curious about it and willing to do it.

> Also, if you have to ask, then you shouldn't self-host it.

We should be encouraging curiousity (a HN value) not stomping on it.

If anyone asks, I say go for it. Worst case you'll learn new things, best case now you own your email.


I would amend this to say, "If you have to ask, then you shouldn't self-host it for anything mission critical."

Otherwise, how would anyone learn anything?


I'm in a similar boat, but have a slightly different conclusion: if you started 20 years ago, you'll have a much easier time today than if you started six months ago.

I think it's also fair to say that personal mail for a small domain is much easier than even a small amount of transactional email and don't even try sending newsletters beyond your friend group.

I have run mail off three different IPs over the ~20 years I've been hosting, switching IP address didn't affect me all that much.

Another thing to note is that receiving mail is really easy. Sending it is hard, filtering out the spam (and only the spam) from your inbound email is harder.


Spam has become much easier to avoid on inbound mail at some point. I turned off all Bayesian/heuristic based spam filtering around 2015 and now just check SPF and DKIM, and have a fake first MX record. No "Junk" folder, everything that passes the checks goes to the inbox. I get maybe one spam mail every week or so, which I just delete.


Could you please elaborate on the "..., and have a fake first MX record" ?


Legitimate SMTP servers will try your domain's MX records one by one, in order of priority, until they reach one that accepts the message. Spammers' scripts usually don't bother, they just try the first one and move on to the next address on their list.

Of course, this is not 100% reliable, as it's not too difficult for spammers to adapt and improve their scripts. Of course, vast majority of spammers are either not sophisticated enough, or do not care enough to do so, so if you don't mind your incoming mail to be slightly delayed, it's kind of a low-hanging fruit, as it cuts off a huge amount of low-effort spammers.


Not GP, but I think what they meant is having a MX record with a higher priority pointing to an unroutable IP

    blackhole  IN A 240.0.0.1

    @ MX 10 blackhole.example.com
    @ MX 20 mail.example.com


Yup, exactly, for the reasons described in the sibling comment to yours.

Doing it this way doesn't even delay mail much most of the time; many legitimate MTAs connect immediately to the priority-20 MX after failing to connect to the priority-10 one.


Yes, receiving is very easy. Just doing that has a lot of value because now can't be arbitrarily cut off from your internet identity by gmail/et.al. for no reason.

So a easy way to get started is to receive everything directly and use a commercial (often with low-volume free tier service) relay for outbound until you get comfortable enough to remove the training wheels. (Or never remove them, that's a legit choice as well.)

> filtering out the spam (and only the spam) from your inbound email is harder.

I don't find that at all. Filtering spam is the easiest part. All I do is if SPF doesn't match, goes to spam folder. Beyond that, apply a bayesian filter.

I get no false positives and the spam that gets through to my inbox can be counted on one hand per quarter. Basically none.

That's yet another benefit of self hosting, since my bayesian filter is trained on my personal email specifically, it tends to become very good. Unlike generic gmail filters for example, where there'll always be some mail that ends up in spam no matter how many hundreds of times you mark it not-spam.


It's easy to block all spam, if you don't mind also blocking legitimate mail. What's harder is being confident you're not blocking any legitimate mail. I'm glad you've managed it :).

I have Bayesian filtering behind DMARC checks and a few other DNS and RBL checks. Still not perfect -- for all folk talk of the difficulties getting mail accepted by the big providers, a surprising number of companies still don't have matching forward and reverse DNS records :P.

One principle I follow is that I don't drop anything. Every message will either be rejected or delivered, with a narrow band of SpamAssassin scores that delivers to a spam folder.


> So a easy way to get started is to receive everything directly and use a commercial (often with low-volume free tier service) relay for outbound --- I used to do that. Did it for almost a decade, in fact. Then the spam filterers decided any message with mismatched send and receive routes was spam.


Use aliases?


Yup, came here to say pretty much that. I've been self-hosting email for decades, first from a box on my home network (back when that was possible) and later from a VPS. It's not nearly as hard as folks make it out to be.

At the very least, getting your server marked as spam/blacklisted is not inevitable. Just make sure you aren't an open relay and that you've got properly configured SPF and DKIM records in your DNS. Once that's set up you can pretty much forget about it. I haven't had to touch any of my configs in years.

Initial setup takes maybe a day or two if you know your way around Linux or one of the BSDs.


I agree, I've been hosting my own email for a few years now, using Mailcow on a Linode. With SPF and DKIM properly set up, mail is /usually/ accepted by the receiving end.

The only nag is that Microsoft is EXTREMELY strict for their hosted email. It's the only provider that consistently denies recieving mail from my server when the IP range it's on becomes greylisted in UCE-PROTECT -- which happens every so often...

Easily solved with getting the MX and backup MX IP's whitelisted there, but I haven't bothered cashing out for that yet...


> when the IP range it's on becomes greylisted in UCE-PROTECT -- which happens every so often

This is a common complaint with Linode specifically, but probably fairly common with low-cost virtual server providers in general. It's worth looking into the history of your IP before you start using it to host mail, and if it's feasible, shelling out for a dedicated server (ideally from a provider that doesn't also offer virtual servers, or has enough network separation between them) makes it much less likely that your neighbour is a spammer. Mine's never been on UCE-PROTECT.


It also depends on the reputation of your neighbours / AS... try to setup an e-mail server with a cheap VPS in DigitalOcean|OVH|Hetzner and even if you kept using the same IP address for 10 years I don't think you will be able to relay to gmail


I‘ve been doing exactly that for roughly 10 years and never had a problem with Gmail. You just have to set everything up to latest best practices (DKIM etc.). I‘ve even changed IP addresses a couple of times.


I self-host for 20+ years, the last 12 years on Hetzner. Did some transactional mailings as well professionally. Not spam, but forum notifications, mailing lists.

2007..2014 were probably the worst. Gmail was chainging often, Microsoft was blocking everyone.

I think self-hosting is easier now than 10 years ago.


I self-host on Linode, and they occasionally fall into UCEPROTECT level 3 (the ISP-wide blocklist), likely due to a spammer trying to set up business as a Linode client. You can't really do much about ISP-wide blocklists, and honestly mail servers should not aggressively reject E-mail simply because some other rando at the ISP sends spam. I've always had success working with the aggressive mail server admin to get unblocked. Often it's just an E-mail to provider-abuse@provider.com


> honestly mail servers should not aggressively reject E-mail simply because some other rando at the ISP sends spam

This is because the business department will try to force admins to ignore requests to remove spammers as the spammers generally pay decent money.

By blocking everyone, it forces the ISP to get off their ass and kill the spammer irrespective of how much they are willing to pay.


Technically true, but UCEPROTECT, in particular, is known to have some shady practices:

https://blog.sucuri.net/2021/02/uceprotect-when-rbls-go-bad....


No argument. But I see this as vindication.

ISPs are now so sensitive to not allowing spammers that the RBLs can't make money from it anymore.


They come across as pretty scummy: "Yea, we put your whole VPS provider in our blocklist. It's their fault for having one spammer customer: Blame them! By the way there's nothing you can do about it--except of course pay us to be whitelisted..."


I run two different private email servers, both set up pretty much identically with all the modern "best practices" as you say. One of them is on an address that has been stable for the past 10 years, and on a quality ISP that actually cares about reputation, and mail delivery is mostly not a problem. The other one is on a cheapo ISP, and mail delivery is so poor due to neighboring IP addresses causing me to get block that I gave up and resorted to using a free SendGrid account to relay my outgoing mail through.


Which provider do you use? (by the way, I didn't mean that every cheap provider will have issues, prgmr/Tornado for instance works great for me)


I have a friend who is self-hosting on Linode. All mails from her reach my inbox (Gmail).


You started self hosting before Gmail was a thing. It’s completely likely that Google has had you whitelisted since the inception of gmail when they were far more open. The “reputable IP” is the hardest part of delivering mail reliably; and your post says the equivalent of “just have your dad gift you a reputable IP from the 90s”


I've also been hosting since before Gmail was a thing but have since set up new servers with new domains in disreputable subnets, and I haven't had any deliverability issues. If you configure things right (SPF, DKIM) you won't have problems.

This idea that self-hosted email is impossible is wildly overblown.


I'd be very surprised if I was on some kind of static whitelist from the early days of Gmail. Such a list, if it exists, would likely be reserved for much higher-volume sending IPs.

I've helped others set up self hosting much more recently, and haven't had any reputation problems beyond the early period where the IP has no history. (It is important to find an IP that doesn't have recent bad reputation, but that is fairly easy to do. Unless your host is in the business of hosting spammers most IPs will be clean.)

The reality is honestly just that self-hosting mail is not as hard as all the people who don't do it say it is.


Google would ignore your data because you are too small?

Your ip and those of others who have sent mail to gmail have been recorded.

Your reputation score is high.

Try a new ip and see how hard it is.


It's not exactly _hard_ on a new IP, as long as the IP is clean (which is reasonably simple to ensure, as long as your host isn't in the business of hosting spammers). You just have no reputation to start with, so you can't expect perfect deliverability from day one. But after some time (and we're talking weeks-to-months, not years), you'll be fine.


Sure if you want a lot of them to go through but not necessarily all. If someone marks your email as spam you may have to start all over again. If you want to send more than a few emails per day you have to warm the ip by sending specific volumes over time. It's doable, easier with a service to help you.


I guess it depends what kind of mail you're sending? I send & receive a lot of mail, but I can't imagine anyone has ever marked a mail from me as spam (except perhaps by clicking the wrong button), since it's just normal conversational personal & business mail. I never did any kind of deliberate 'warming up', I just set up the server and then started sending mail. In the first few months people would occasionally see mail from me go to their spam folder, but that hasn't happened in many years.


I've setup 2 mail servers in the past 3 years and can reliably deliver to Gmail and Hotmail.. AT&T can be a chore I've found, they have been mostly unresponsive, but very little of my targets host there.

The other option is to relay through Send in Blue, or Sendgrid or something like that.


It's wild that something as pedestrian as hosting your own email comes with a pile of caveats comparable to those attached to browsing I2P or using cryptocurrency to buy stuff.


> It's wild that something as pedestrian as hosting your own email comes with a pile of caveats

No, it's not "wild".

Its just that we're in 2022, not 1997.

Long gone are the days of "fire up Sendmail and you're good to go".

To those thinking of self-hosting, I would say they should start by understanding modern anti-spam.

Understanding modern anti-spam will not only help them with their inbound email, but will also help them understand how to ensure deliverability of their outbound email too.


> If you set everything up right,

You don't even need to set everything right. Up until very recently (months), I was sending emails from a few of my servers, and I had NOTHING set right. As in, I was sending from IP addresses that were never mentioned on my DNS, no PTR no SPF no DKIM no nothing. Just good old "here's an email from this address, trust me I actually own that address and it's legit".

And it worked just fine.

Surely just a reputation thing, as I had been doing this for over a decade, and all emails were very important (password recovery, order details, etc), no newsletter or anything.

I recently replaced all that with zoho because I wanted something a bit more secure and didn't want to configure it myself.


Self hosting my emails, grade of 100/100 on mail-tester, I'm on none of the dozens of blacklists I've queried... and Microsoft marks every single one of my emails as spam.

Just because you're lucky doesn't mean other people are "unexperienced". I don't know what experience has to do with it tbh. Maybe if you have experience working for Microsoft and you can contact the right people there...


Have you contacted Sender Support at Microsoft? They're very responsive and if your mail activity is legit they'll certainly help you solve the issue. Also make sure you set up the SNDS dashboard so you can view your IP reputation: https://sendersupport.olc.protection.outlook.com/snds/index....

In my experience the difference in outcomes that people have with self-hosting mail are largely down to, from most important to least, (1) the nature of the mail they are sending, (2) their choice of hosting provider, and (3) the correctness of their mail server setup. It sounds like you've got (3) sorted, so maybe (1) or (2) are relevant here.


I have been self hosting my private domains for a couple of years now and spam is not a problem with rspamd and it needs very little resources. Some large hosts just outright blocked me but i always get it unblocked with some assistance of my hoster. So, yes it is fine. I even set up webmail with rainloop. Better than anything once you got it set up right.


I did the same for like 10 years, and got similar results…

…except when Hotmail decided my entire IP block was barred from sending emails to them. Happened several times, I always had to wait a few days for my emails to stop bouncing.

In my experience it mostly works, but it’s never reliable.


> probably have little relevant experience

Or maybe they have 10x the experience you do, but it was different experience for reasons beyond their control. Don't over-generalize from a sample of one. That's hubris.


That's what the word relevant means in relevant experience.


OK, then. They might have 10x the relevant experience that you do, but the outcomes were still different for reasons beyond their control. Better? Your scattershot insult was still full of hubris.


No, that's not better. If the outcomes were different, that's for reasons within their control. That's the whole point of this entire thread! There is no voodoo in self-hosting mail!

You either get everything right (due diligence of your provider and the IPs they're giving you, configuration of your mail server, not sending spam, monitoring your reputation) and your mail is delivered, or you don't and it isn't. That's it. Getting everything right, and knowing what you have to get right, comes with experience.


May be you've been grandfathered into a white list when things weren't so sophistcated 20 years ago?

Have you tried setting up a brand new email server with a fresh IP from say, Digital Ocean or AWS?


I've helped friends set up mail servers fairly often, and they've had the same experience. But I wouldn't recommend DO (especially) nor AWS for this.

I'd recommend a smaller, tech-savvy hosting provider that has been around for a long time, who has no tolerance of abuse on their network, who is easy to communicate with directly (no paying for 'Business Support' just to talk to a clueful person), and ideally who mostly rents dedicated servers / colo rather than cheap virtual servers.


There's a lot to say here from both sides - people running their own mail infrastructure (like I have for almost 25 years) and big mail providers dealing with brutal, unrelenting spam.

But there is one piece of this that's ridiculous, broken and almost cruel: silently dropping messages marked as "spam" with no notification given back to the sender.

Why does this practice exist ? Who believes that this is decent or acceptable behavior ?

If gmail doesn't want my inbound message - for any reason - that is just fine.

If they drop it on the floor without telling me that is totally shitty.


Mailservers drop mail on the floor because they are doing spam-filtering AFTER accepting mail for final delivery. Once it's been accepted for final delivery, it can no longer be rejected.

The delivery server doesn't generate the bounce message you were expecting; that's generated by your own mailserver, on seeing a REJECT status code from the delivery server.

Mailservers do spam-filtering after accepting for final delivery because spam filtering can be processor-intensive. Sometimes it's farmed-out to an appliance or whatever. To have the SMTP process suspended while Spamassassin goes through it's contortions multiplies the consumption of server resources on the SMTP server.

The delivery server CAN'T (and shouldn't) send you the desired bounce message, because it doesn't really know who you are. It can't rely on the From: address, because you could be sending on behalf of someone else.

In my view (and the view of the RFCs), if a server says "200 OK Accepted for final delivery", then it MUST deliver the message.

There's an awful lot of the kind of server-side spam-filtering that does actually involve delivering: the kind that filters mail into the recipient's spam folder. That mail hasn't been dropped on the floor. It's been delivered, just not to the inbox.


You've described how my mailserver should work.

I'm willing to stipulate that this is correct and would, in my case, be a difficult problem to solve.

But nobody is losing job offers or missing kids' schedules or breaking their summer plans because of my mailserver.

I am talking about gmail. I am talking about MS (whatever it is). I am talking about yahoo.com.

Their spam heuristics are, in many cases, laughably bad - they are demonstrably, clearly broken. If I email my wife twice daily for 15 years and then one of my responses to her emails gets put in the gmail spam folder ... what words to even use for that ?

They need to fix this. I don't care how sticky of a problem it is.


$employer uses MS for email hosting. MS recently started dumping every single email from every single Apache mailing list in the spam folder. These are mailing lists to which I've been subscribed for a decade, to which I regularly send emails, and which probably have thousands of subscribers. There is no option for me to whitelist the mailing list, only individual senders, of which there are hundreds.

Everything about megacorp spam filtering is broken.


> they are demonstrably, clearly broken

Well, gmail, MS and Yahoo have their own ideas about what "broken" means. Google in particular forces changes to standards by simply implementing them in their own services. Those changes never make it easier for small-fry postmasters; so I conclude that Google would like all small-fry mailservers to disappear.

Discrimation through spam-filtering isn't unthinkable, and it would be hard to prove (especially if they claimed there was "AI" involved in the filters). Google used to have really good spam filtering; I can only suppose that the reason it's got worse is that they want it worse.


if gmail wanted to, they could provide some SMTP extension. eg. they could advertise that the sender server can send a callback url where gmail will post back the results

there's already some kind of feedback loop mechanism, but mostly available for large senders.


Responding gives the sender the ability to better explore the detection rule space and find a way to get through.

Exactly what a legitimate sender wants and what a provider would not want to give an adversary. Now the adversary has to also incur a cost to determine successful deliver vs open/engagement rates make it just that much harder.


I think it's a false economy. At best it is security by obscurity.

Google tell you why things are spam and often they even return quite detailed error emails when they don't accept stuff it doesn't benefit an attacker that much. Any decent attacker already knows that they should sign stuff and make identifiers align and can do so trivially easily.

People like Yahoo are the opposite and are completely opaque as if they are doing anything that clever. All they can realistically do is check originating IPs, message content, alignment etc. just like everyone else.

Since I can still a lot of very decent SPAM in my inbox, their lack of transparency clearly doesn't work so they might as well help legitimate senders to deliver stuff properly.


"Google tell you why things are spam and often they even return quite detailed error emails when they don't accept stuff ..."

Where are you seeing these details ? I have never seen any bounce messages from messages I send into @gmail.com that end up in a spam folder ...


In case of gmail you can have an account there and see what goes through.


I would think returned spam would have to be checked for being spam by servers receiving the returned messages.

If they don’t do that and decide whether checking is needed based on presence of some mail headers, or using heuristics on the subject line, spammers will start faking returned messages.

I think that makes it expensive to return spam. https://99firms.com/blog/spam-statistics/ says about half of all e-mail is spam, so returning that would increase e-mail traffic and spam checking by 50%.


A bounce message can be very small, specifying only three fields: To, From, Date to identify the rejected message. Compared to this spam messages would be much bigger, consisting of html, css, javascript, pdf attachmens, zipped executables etc. The message could be sent to a fixed address, like abuse@same.server; how to check messages on that address is a different story as it's not user messages, but technical messages.


> If they drop it on the floor without telling me that is totally shitty.

No, it's the ONLY reasonable thing to do when something like 98% of all SMTP traffic on the Internet is spam.

If mail administrators bounce back an explanation for every "bad" message:

1) Their outbound mail volume would go through the roof.

2) The host sending all the bounces would look like a spammer to _other_ automated spam-classification systems.

3) In the unlikely event that a spammer actually reads bounches, they could use the feedback to tweak their systems to avoid the spam filters.


What a recieving mail server should do is reply immediately after the DATA command with “That mail is spam and I refuse to acknowledge recieving it”. This would be fair, as the sender would know that the mail was not passed on. What it should not do is to simply say “Yep, I got this mail now, I’m good!” and then later, silently look at the contents of the mail and decide that it’s spam and delete it.


There are sane email providers who do exactly this, for example Posteo — see question "Is there a spam folder" here: https://posteo.de/en


> If mail administrators bounce back an explanation for every "bad" message:

Nobody is talking (I don't thing anyone is) about sending back a bounce message, that would indeed make no sense.

A responsible email server should:

1) Reject the email during the SMTP conversation if it's going to do that. Then the sender knows it didn't go through because it got the error code. There's nothing to bounce back.

2) If it accepts the mail during SMTP conversation, then always deliver to the recipient.

2a) Some disagree, but I think it's totally fine to deliver it to the recipients spam folder if post-processing determines it might be spam. That's not wonderful, but it still got delivered and the recipient can go find it in the spam folder. Most people are used to looking there regularly anyway since many of the larger providers (coughgooglecough) have such terrible false positive rates. The important thing is to never lose email.

What's never ok is to accept the email during SMTP and then silently file it in /dev/null.


Does this really happen? Because I imagine that, if Gmail really did drop some super-spammy incoming messages, it would includes quite a lot of my current spam folder[0]. Maybe they only drop spam when it’s an IP [range] ban at some firewall level?

0: https://cdn.discordapp.com/attachments/468851579487780905/96...


I run my own mail personal mail server, and do my own spam filtering. Things that are likely spam get delivered into a spam folder for the occasional review, but things that are extremely likely get rejected entirely (with a "spam rejected" message).

I guess my feeling is: if I want to quarantine suspected spam without telling the sender, that's my prerogative -- why does the sender get any say in this?


I don't think he's saying you can't quarantine spam if you choose. What his complaint is, I send you an email, your mail server says it accepted it, the email is then just discarded entirely (not put in a spam folder).


I guess the volume of spam-reject messages would dwarf legitimate mail.


You can host your own email.

I have been hosting my own for at least 15 years now, and I don't have big issues - I can deliver email to MS, gmail, et al.

Pick a decent hosting provider (not the cheapest options around!), make sure you have matching reverse DNS, forward DNS and HELO name (exactly the same is best!) on both v4 and v6 (if you have v6), disable IPv6 privacy addressing for your mail server (again, if you have v6), make sure you set up SPF, DKIM and DMARC, and keep your server secured.

By following these rules, in 15 years I have had only had deliverability issues with AT&T and Deutsche Telekom - both of which were fairly easily resolved.

In terms of software, you can use one of the out-of-the-box email server packages, but I personally run postfix, dovecot and rspamd on a debian stable VM. Stick to the versions from the repos and you'll have very few problems upgrading it in future, too - my current mailserver VM started on Debian Squeeze or Lenny around 2010, and is currently on bullseye (the latest stable).


I've run the mailu.io stack for 4 years off a Hetzner dedicated server. It's an easy stack to set up and update through several major versions. About 25 different mail users, friends and family who aren't in IT.

The issues I've had have been Microsoft (hotmail/outlook/office365) dumping messages into their Spam folder, but that went away in the last year. I had put in a hack to deliver to Microsoft through a more "reputable" SMTP host, but only when people complained.

I'd say give it a go with a new email address, any software that seems manageable to you, and move your usage over gradually.


> Pick a decent hosting provider (not the cheapest options around!), make sure you have matching reverse DNS, forward DNS and HELO name (exactly the same is best!) on both v4 and v6 (if you have v6), disable IPv6 privacy addressing for your mail server (again, if you have v6), make sure you set up SPF, DKIM and DMARC, and keep your server secured. By following these rules, in 15 years I have had only had deliverability issues with AT&T and Deutsche Telekom - both of which were fairly easily resolved.

How did you measure your deliverability? If this is true, then congratz for succeeding, but it's still bad advice to give to other people, as most people will not succeed no matter how many hours they put into that.


By the fact I get answers to emails I send to others (eg people using gsuite, people using o365), and I don't get issues from the sites that I run that use the mail server for sending - it sends confirmation emails for a forum I run, for example. I know that gmail, o365, yahoo, aol, etc work, because people use those emails to sign up and manage to validate their accounts.

I don't agree that most people will not succeed, I know many other people personally who run their own mail servers. It's doable, and it's not nearly as bad as some like to make out.

I've run low volume mail servers and high volume mail servers sending (GDPR compliant!) marketing mail.

edit: where low volume = <1 outbound message a day.


> By the fact I get answers to emails I send to others (eg people using gsuite, people using o365)

Do you get answers to 100% of emails you send? I don't find this plausible. Now, if you get answers to maybe 30% of emails you send, how do you know the other 70% is just because people didn't write anything back? How are you ruling out the possibility that some of those 70% never received your email in the first place?

> I don't get issues from the sites that I run that use the mail server for sending - it sends confirmation emails for a forum I run, for example.

So far I haven't encountered a single email provider that successfully delivers 100% of mail sent. Postmark sometimes fails to deliver, SendGrid sometimes fails to deliver, etc. But you're claiming that you have found the secret sauce and you actually have better deliverability than SendGrid and Postmark - and that's for confirmation emails of all things, the type of mail that very often lands in the spam folder. I don't believe you.


> Do you get answers to 100% of emails you send? I don't find this plausible. Now, if you get answers to maybe 30% of emails you send, how do you know the other 70% is just because people didn't write anything back? How are you ruling out the possibility that some of those 70% never received your email in the first place?

Most of the personal email I send is to companies where I do expect and get responses, or to my family, or to mailing lists. I know family get my emails because they respond. I know companies do because they respond to support queries. I know mailing lists do because I see my messages in the list archives. I know there’s a good mix of receiving operators because I get DMARC reports etc.

> So far I haven't encountered a single email provider that successfully delivers 100% of mail sent. Postmark sometimes fails to deliver, SendGrid sometimes fails to deliver, etc. But you're claiming that you have found the secret sauce and you actually have better deliverability than SendGrid and Postmark - and that's for confirmation emails of all things, the type of mail that very often lands in the spam folder. I don't believe you.

I don’t really care if my email ends up in spam folders as long as it does not get dropped on the floor entirely, but I genuinely do not get complaints where people have not received/can’t find their confirmation emails. I do practise good automated email hygiene (automatic removal when things bounce permanently, etc).

I don’t know what to say that will convince you that I have not personally experienced issues except with DT and AT&T, but… I haven’t.


> Most of the personal email I send is to companies where I do expect and get responses, or to my family, or to mailing lists. I know family get my emails because they respond. I know companies do because they respond to support queries. I know mailing lists do because I see my messages in the list archives. I know there’s a good mix of receiving operators because I get DMARC reports etc.

Ok, fair enough.

> I don’t really care if my email ends up in spam folders as long as it does not get dropped on the floor entirely, but I genuinely do not get complaints where people have not received/can’t find their confirmation emails. I do practise good automated email hygiene (automatic removal when things bounce permanently, etc).

I care very much if my email ends up in spam folders. But if you're only talking about your email landing (in some folder), then sure, you convinced me.


Nobody who is anxious about a first contact with someone new will rely on email alone, anti-spam has made sure of that for 25+ years. You just can't measure whether someone _saw_ your email, which is all that matters to email users.

Users do complain about unexpected bounce messages (often it's an address typo). And I am pretty sure that people who use gmail & hotmail are used to "checking their spam folder" and fixing deliverability problems for new senders that way.

I've been pretty slapdash about this, including selling 1000s of mail servers and (apart from the adoption of SPF, DKIM, DMARC) it's all the same as it was 20 years ago. So I've no problem advising technically-inclined people to give it a go gradually.


> You just can't measure whether someone _saw_ your email, which is all that matters to email users.

You can still measure deliverability with different methods. I've used GlockApps to send test emails to a variety of different inboxes at different providers and it tells me what percent of those emails hit the inbox, what percent went to spam folder, and what percent disappeared.


Watever GlockApps is, there's no way it can tell whether my local rules moved a message to my spam folder. It also can't tell whether my mailserver moved it to spam using my Sieve filters. Therefore it also can't tell whether the email appeared in my Inbox.

I'm not sure what "disappeared" means in this context; perhaps that's the same as "dropped on the floor". The whole reason why dropping email on the floor is A Bad Thing is that doing that makes it impossible to tell whether the message has been delivered at all.

[Edit] Ah, I see. GlockApps can only tell you about the destiny of your outgoing email if the destination was one of a handful of big freemail providers; and I imagine you'd have to provide GlockApps with credentials to the recipient accounts, so that it can see if the mail ended up in Inbox or Spam. You could do that with a few lines of Bash script.


Yes! That site is great as a spot check, I've used it. But imo if gmx.de are silently binning my messages (and nobody else) it's their problem. If it's Gmail, it's my problem :)


> So far I haven't encountered a single email provider that successfully delivers 100% of mail sent.

There is no email provider that will deliver 100%. None. As I mentioned in another comment, you can buy a gsuite corporate account and send email from gmail to gmail within your own company and still end up in spam. If you expect 100% from any solution, you'll be disappointed.


Basically the same setup as me other than I use ubuntu.


If literally anyone could find your Mastodon address and message you, you'd get spammed daily there too, and it'd be the same problem.

So you think, fine, whitelists! But you still need to be able to accept messages by new authors without knowing their From: address ahead of time. You'd have to comb through your spam folder past tens of thousands of messages from new authors to find the one new genuine sender. Rings of trust don't solve it either because either you get spammed by someone in a ring of trust, or messages end up in concentric rings of spam folders.

You can host your own mail. It's just very hard to do it correctly, easy to screw up, and there's basically no gain whatsoever by doing it yourself. Some problems are just difficult and cannot be easily solved by a single person. You can't be your own CA [and have anyone trust your connections]. You can't create your own TLD [and have everyone be able to resolve it]. You can't create your own ASN. You can't create your own IP address. There are some things in life you have little to no control over, even on the Internet.


Microsoft in particular is a total pain in the ass to deal with.

I was hosting my own mail server, did not have open relays and I know 100% sure nobody on my server sent spam. It was fully configured with all the DMARC and SPF trimmings.

Yet one of my users needed to email users at live.com/outlook.com/hotmail.com and kept getting banned. Every time I was able to unblock it using an automated link.

One time it didn't work and I actually got through to someone. He was like "Yeah, your server doesn't send enough legitimate emails so it doesn't build up 'reputation'". This sounds ridiculous, not sending spam is not enough, you have to send a certain amount of legit mails to stay unblocked??

Anyway it kept happening so I eventually gave up :( It only happened with consumer MS-hosted emails addresses though. I had no issue reaching companies using M365 for business.

But email is just so incredibly broken... All the patches to kinda try and fix it are a mess. We need a whole new protocol.


nah, email is okay, the protocol is fairly extendible. storage and processing capacity is cheap for it, even naive spam filters are pretty good and quarantine can take care of the rest.

what's missing is simply the coordination to force the big ones to stop fucking over the small players.

but since paying for 100k mails/month is less than 1 hour of how much one engineer's hour costs a lot of companies just don't care. there's no real incentive to fix this :/


You certainly /can/ self host. I've been self hosting my own email since circa 1999 on my home internet link. I've been through four different ISP's in the ensuing 22 years, but email still flows.

What often happens is that virtual hosting firms (Linode, Digital Ocean, etc.) are often used by spammer's for their hosting too, and so if you try to host by renting a "cloud vm" or "cloud server" and are unlucky to have an IP address a spammer previously poisoned, or just happen to be in the same netblock as a prior spammer, you find your new IP often 'blocked' from the big services, for no good reason than you happen to be from a "bad neighborhood". And this is usually the genesis for all the scary stories about "can't self host".

But reality is, you can self host, but you do have to set things up with all the modern requirements (SPIF, DKIM, etc.) as well.


You can set up your own email server. There isn't anything to "fix" (unless you can "fix" all the spammers) but does require setting up SPF in your DNS and it helps to support DKIM/DMARC. Also, your internet provider / VPN host likely blocks port 25; if that is the case, you need to use a "smart host" email relay service.

https://www.google.com/search?client=firefox-b-1-d&q=self+ho...

https://www.google.com/search?q=self+hosted+email+server

https://www.google.com/search?q=dkim+dmarc+spf


Self hoster of 3 years here. You can do it. Don't let the big players scare you off when they send your mail to spam (GMail) or outright 550 refuse to accept it (Microsoft).

So far I've managed to avoid needing to relay my mail out using something like SMTP2Go but eventually I may have to. For now GMail seems to be learning when I email my regulars and Microsoft unbanned me after I joined their Outlook.com Smart Network Data Services (SNDS)

In better news, incoming mail works flawlessly. It's even spam free if you use a catch all address (dodgywebsite.example@yourdomain.example) and drop mail from any company that leaks your address out.


I also self host but on a VPS. Incoming mail works great. Outgoing mail not so much unless I send it to a trusted third party SMTP server. No outgoing mail fees if low traffic. Might be an option to consider while building up reputation?


Setting SPF, Dmark, DKIm and having a proper PTR record significantly increases your server acceptance.


I had all that setup from day one and still ended up in the GMail spam and banned by Microsoft/Outlook.com (which also includes anyone using Office 365 in their workplace for mail)

I assume that's because I had no reputation, although I did start on a DigitalOcean VPS before I learnt about their terrible ongoing reputation for ignore abuse reports. [1]

[1]: https://discourse.mailinabox.email/t/digital-ocean-ips-being...


Which program do you use to filter incoming spam on the VPS and is it working well?


The two typical choices are rspamd and amavisd/SpamAssassin, at least when you're using Postfix on a unity environment. I use amavisd on my primary and rspamd on the secondary, both seem to work OK.

Yes, spam will still make it through and you have to train the filters in either case.


>How do we fix this?

You start by taking every person who says "not worth it, man, just use GMail" and beating them with a rubber hose until they install and run a mail service for their vanity domains.

More seriously, it's possible we've let this problem fester for so long that it's going to take serious effort to fix. By which I mean governmental intervention. Google, Microsoft and Yahoo cannot be allowed to dictate who gets to send and receive email, as they effectively do now through their massive marketshare dominance.

Spam is a problem, but it's not an intractable one. In the 90s, sure, the technical problem was pretty hard. By the 00s, everybody just let Google handle it because Google wasn't going to Be Evil, and Google managed to solve it with a giant technical hammer.

Technical people also tend to dismiss solutions because they don't fix every problem. The old Spam Solutions Checklist exemplifies this attitude. But what we have now is worse, i.e. just letting the world's most invasive corporation control all of it.


You can, and should, host your own email. The more people do this, the more companies like Google and Microsoft will be forced to accept email from small servers.

That being said, when you set it up, make sure you set up an SPF record. Also, check the IP Address to make sure it is not already blacklisted.

Cpanel makes it almost effortless to set up an email server, if you have just a little bit of tech know-how.


I've hosted my own email since, at least 1993 (that's on the Internet: I was on UUCP at least some years prior to that).

If you have a static IPv4 in a range that is not actively hostile, and you have proper SFF/DMARC records, things should generally work out?

And otherwise, services like https://www.mailchannels.com/ should help? (Still, you will need proper SPF records.)

I've literally had a 95+% delivery rate from users in actual Lagos Nigeria using the strategy outlined above.


You start by grouping all sources of advice in to two categories:

1) those who say you can't and/or shouldn't do it. They don't know you. They might as well say you can't fix your own computer, you can't learn to write a shell script, or you can't fix your own car. They "can't" because they're afraid of failing. Ignore them completely.

2) those who say you can, and give you tips on what's difficult and how to make things better. Obviously we can self host, as many people, myself included, do self host, have done so for ages, and will continue to do so.

Some people in category 1) try to make themselves seem reasonable by bringing up these huge lists of things you have to do, but it's all completely doable. Just recognize when a particular person happens to be in category 1), and stop wasting time with them :)

I've self-hosted continuously since the late '90s, and I've even experimented with starting over, so to speak (that is, starting with a completely new domain and new IP), and it's work, but nothing beats OWNING your own data and email. Having direct access to logs means you know exactly whether delivery attempts were made, whether destination servers accepted email for delivery, and precisely when. If you have an interest, it's totally worth it.


You absolutely can host your own email server. I did for over a decade. Spam ranking is tied to IP addresses so you just need to get an IP address that doesn't already have a bad reputation, and build it a positive reputation over time. Then, as long as you don't send spam from that IP you should be good as long as you keep the same IP for your server.

That said, I abandoned running my own email server years ago. It only went down a couple times on me, but when it did it was always when I really didn't have time to fix it (which is basically always). It's not really difficult at all, but it's MUCH HARDER than just using gmail or whatever.


Of course you can. Just try it, and you'll see that the problem isn't "I can't", but rather "I don't want to deal with this shit"


> "I don't want to deal with this shit"

This is probably the most sensible comment on this thread


> "I don't want to deal with this shit"

Funny, that is my response to most 3rd-party/hosted email. We all have to pick which bag of shit we want to carry.


In my opinion, the difference between Mastodon/ActivityPub/the fediverse and email is adoption. If Mastodon was more popular:

- A handful of instances provided by large companies would probably crop up and end up hosting the majority of users

- Spammers would notice that they could reach a large number of people via Mastodon, and start spamming

- The providers of these large instances would moderate heavily to prevent their own instances being used for spam, and begin blocking / not federating with small instances

I should add that spam is probably _already_ a problem on Mastodon, but perhaps not to the extent that it is for email since the average Mastodon user is (for now!) way less likely to fall for a scam and therefore a much less valuable target.


The other problem with Mastodon / the Fediverse is how fractured it is. Here[0][1] are some examples of the sort of blocks that instance admins implement, including reasons such as "Allows controversial content" and "A lot of trolls, very [My Little Pony] themed".

I suppose it is to their credit that these instances are so transparent about their blocking policies, but I think the world would be a worse place if email or even Twitter made it impossible for people with different politics to message each other.

[0] https://toot.cafe/about/more#blocked-instances

[1] https://im-in.space/about/more#unavailable-content


I want a whitelisting system for email where:

* If you're whitelisted you get through or I can manually whitelist you.

* If you're not whitelisted, I send a bounceback response stating that I'll look at your email for $X where X is set by me ( e.g. $0.25 or $1, but I decide). No guarantee of refund, but I have the option of refunding. For me, if you wasted my time, I won't refund. However, if you're a legit human that isn't marketing to me, then I would refund.

Then, I just adjust the price until spam disappears or I'm willing to look at your spam at that price.



:-)

Not sure about the legality and such, but rate limiting by economics does work for my physical mailbox. I get spam, but it isn't 1000+ letters a day like my email inbox. If I was allowed to increase the cost of delivery to my house, I think I could eliminate most physical spam mail in my physical mailbox as well or happily find a price where I'd quit my dayjob and just read whatever physical spam is sent at an hourly rate I'd like.


I want a system where I can create destroyable email addresses, so when I sign up for something it creates a single channel, and if they resell that email address I can burn that channel.

Unfortunately `+alias`ing built into gmail is too easy to subvert as everything after the + can be removed and the email will still reach me.


There are plenty of services that allows that. One is Apple's [Hide My Email](https://support.apple.com/guide/icloud/what-you-can-do-with-...) and Firefox's [Relay](https://relay.firefox.com/).


Fastmail's "Masked Email" feature will do this. You can create unique addresses for services. And then you can deactivate them to block future emails arriving at that alias. And the aliases are random (couple of random words and some numbers e.g. busy.sock2881@fastmail.com ) so can't be externally linked back to the main address.


I like the first part. In fact, Hey email had this concept and was the only thing I liked of that horrible email service. I would like to see that somewhere else.


Apologies if I'm being dismissive, but obviously it's possible to host email. How much there is to fix is a matter of perspective.

All that's required is to properly configure the service. Beyond that you probably haven't paid the people who run the other system enough to accept mail from you; they're under no obligation.

I'd go on about socioeconomic factors, demand-side economies of scale, perverse incentives and why it's more expensive to send than receive but that's a whole thesis dissertation and belaboring the point a fair bit.


I've set up my own mail server with Mailcow (https://mailcow.github.io/mailcow-dockerized-docs/). Can't have taken more than an hour to get everything set up, including SPF/DKIM records for the domain. Set the domain, throw a `docker-compose up -d` at the repo and it Pretty Much Works (TM). I need to set up something to parse and visualize the DMARC reports from other mail servers at some point, but that's it: fully-featured mail server with ActiveSync, spam quarantines, AV, no diving into obscure config files necessary. Alternatives like Mailinabox provide a similar experience.

Some mail servers (Gmail, Outlook) discriminate against small mail providers by marking their stuff as spam. Ironic, because the spam I receive almost exclusively comes from free mailboxes. It doesn't happen consistently, and it tends not to happen anymore once the other party responds.

Truth to be told, I receive WAY more email than I actually send so I usually don't need to care about being marked as a spammer. I care more about control over my emails than I care about the occasional reminder I need to give Outlook users to check the spam folder.


Mailcow is the solution we use. Reliable since years and feature rich. A ticket to add a dmarc parser exists but no one ever contributed the plug and play solution discussed in the ticket. Works pretty straight forward.

Mailcow, can recommend


+1 for mailinabox from me. Easy to use, reasonably complete and secure, works from a data-centre or home (you'll need a sensible ISP and to manage your network yourself). Can send to gmail ok (after a receiving a couple of emails from a gmail account) and to microsoft (after going through their I'm-ok process). And like jeroenhd, I receive much more than I send.


I host my own email. I'm not permanently marked as spam/blacklisted. Although it takes a fair bit of tedious setup and configuration to get it to work right. And every now and again a blacklist site will mark my domain temporarily when it detects that some server in my IP block is acting suspicious. This mainly happens because I host on a VPS.

The only problem with email self-hosting is just how many moving parts are involved in a typical setup if you're using tools from unix-land. You need many different programs to work together in a typical setup:

- postfix

- dovecot

- spamassassin

- fail2ban

- kerberos

- ssl+tls

- etc..

And you have to know about how unix account security works because some of the older programs haven't been updated to use modern authentication mechanisms and so they need to be isolated and carefully managed, etc.

The other problem is DNS/verification. You have to set up your DNS records with arcane configuration options that are not well documented in order to play along nicely with the email community and not get blacklisted/blocked.

Some projects have popped up to try and offer containers that have everything pre-configured. ymmv.


I suggest running mailinabox. It installs all that stuff and configures appropriately. Only downside is that it wants Ubuntu 18.04 (for the moment) and doesn't cooperate well with other software - put anything else on a different machine.


There’s a lot of FUD out there about hosting your own email. I’ve been doing it for years and it’s great. Use postfix and dovecot, and research how to maximize your delivery.

I notice that people recommend against self hosting by pointing out that gmail, aol, hotmail, etc. are likely to hide your email in spam folders, refuse it, or just silently drop it on the floor. The flip side of that is that these companies are providing broken email service to their customers: it’s not a mail delivery problem for me, it’s a mail acceptance problem for you. My email setup gives me about one false positive on incoming mail per year, at most. So don’t use these providers; their service is broken.


I have run my own mail server since maybe 1995? As is the case with most things I have learned how to do with computers, there are a few guidelines to follow (SPF, PTR record, etc.) and it helps to have a static IP from your ISP, although not necessary - I have several clients still hosting email using DDNS.

You may wish to consider using something like a Synology NAS where a stripped down mail server is a free feature for 5 mailboxes or less. They also support DDNS...

And when spam levels get high, a quick analysis of source IP addresses gives me new entries for a block list at my firewall. I wrote a simplistic visual basic script to harvest the IP addresses, since I still use Outlook as my PIM.


How do you manage a dynamic IP? Do you whitelist the entire subnet? If not, do you update the DNS records every time the IP changes?


I am referring to the client having a dynamic IP connection with their ISP - The client's IP address may change for their email server. So DDNS service providers (i.e. ChangeIP, etc.) handle DNS for those - An application locally installed for the client (Synology offers one for their NAS devices, for example) polls the public IP every few minutes (or some frequency), then reaches out to the DDNS provider with any updates, that provider in turn updates DNS so the rest of the world "sees" the email server at the new address.


Currently, because MS and Google more or less have a duopoly on email. They don't trust you not to send spam, and they will make it as hard as possible for you to prove that you're not a spammer. Ostensibly this is so they can protect their users. More cynically, I suspect they're quite happy about this.

Historically, because we are - for whatever reason - unwilling or unable to deal with spammers. I mean the people sending the spam and profiting from it. There are virtually no repercussions for spamming millions of people with garbage on a daily basis. Every cent you make is profit.

Putting spammers in prison would make it a lot easier to go back to hosting our own mail servers.


I believe the magic part is to find a VPS whose IP block isn’t spam-tainted. Avoiding the largest VPS hosting providers may conceivably help. But really I have no idea, and maybe one just has to try a few. I’m one of those people who have been self-hosting their own domain and email for roughly twenty years, adding stuff like DNSBL, graylisting, DKIM, DMARC and SPF as they became established. But very low-maintenance overall. I never had any perceptible problems with mail delivery. Some spam gets through DNSBL/spamassassin, but it’s on a very manageable level. I guess what I’m saying is you should give it a try. We need more people doing that.


You can self host; you just can't (easily) self-host your outbound SMTP.

That's okay; SMTP is a system based on forwarding hosts. You still have an SMTP server; it just doesn't send directly to a destination, but is configured to talk to an other SMTP server, one which is reputable (or else itself routes through a reputable one).

Hosting the receive side of e-mail is no problem at all; you do need an ISP with port 25 open. You can also run your own SMTP server easily. I run a TLS-enabled SMTP server on port 587, to which K-9 mail on my phone connects.

Hosting you rown IMAP server, webmail and all that: all no problem.

Just you have to figure out the sending situation.


I've self-hosted my email on my own domain, own box, in my own home, on my own internet connection for two decades now.

I've not had trouble being marked as spam, I have set up dkim and spf.

The real reasons you cannot self host is a combination of:

ISPs blocking outgoing TCP traffic with destination port 25 and does not provice a smart-host / relay for you to use or does provice a smart-host, but do not document it, or configures it in such a way that it only relays if you have some authentication that you don't, or that reverse-dns is configured (at the same time, they do not provide reverse-dns for you).


"I can host my own Mastdon server ..."

Is an e-mail provider required in order to run a Mastodon server.

https://docs.joinmastodon.org/user/run-your-own

I think e-mail, i.e., the software, is "decentralised" if one is willing to break conventions and standards applying to how it is used. The way email software is used today is of course highly centralised, for various reasons. However the option of using email software to administer email in a decentralised way, i.e., going against standards and conventions, is not a topic one ever sees discussed.

With every item of software, there was a time before the standards and conventions existed. The software still works without the standards and conventions. Someone had to test it before those "rules" were established.

But an "email industry" has arisen, and a ridiculous number of individuals and organisations choose to depend on mostly unregulated third party "email providers". When people encourage others to follow conventions and standards, such as using third party email providers, e.g., because it's easy, and others follow these suggestions, a decentralised solution can effectively become centralised.


I host my own servers and have no problems with it, here some hints:

https://support.google.com/mail/answer/81126

https://www.mail-tester.com/

https://mxtoolbox.com/diagnostic.aspx

Test test test, and you will be fine.


It is entirely possible to host your own email server, I do it for example, there is no way to reliably know that other people are receiving your message.

That is: When self-hosting email you can reliably receive email, but can't reliably send it.

The fundamental problem is that email is a broken protocol and too many people are making too much money mitigating the problem of spam rather than solving it.

Companies that need to keep their email servers working have to deal with extortion from anti-spam companies to attain reliable message delivery. It is a racket.

This means that even if you get everything working 100% with all the perfect security protocols and conventions in place the chances of anybody actually receiving your email at this point is roughly 50/50. There is nothing you can do to ensure reliable message delivery without getting your servers whitelisted by most of the popular spam houses. And even then you have to deal with large public companies like Google and Microsoft that may or may not forward messages to recipient based on secret rules that change constantly.

So while it is possible, I host my own email, I can't rely on it. I use gmail for situations were reliability matters.

It is better to use something like Mastodon for correspondence if you can help it.


I host my own email and never get marked as spam.

One fantastic tool is: https://www.mail-tester.com/

And another is Gmail. When you send an email to Gmail you can see some spam info in the headers which you can use to fix problems.

I just hope I never get blacklisted. Sometimes you can fix this by sending the blacklist a message but this is not always possible.


I've successfully hosted my own private mail server [1] on a Hetzner system for the last 6-7 years. It's a steep learning curve, and a rather large time sink in the beginning to set everything up, but it's been running smoothly ever since.

[1] https://news.ycombinator.com/item?id=30428882


You can host your own email. I do.

What you can't do easily is talk to other large public email servers. Which I don't.

My home email server receives email I send to it, but won't forward email to other domains. This is very useful - basically anything on the iPhone is shareable via email, so now I can capture a copy of things outside of the phone easily and privately from anywhere without having to go through an untrusted third party such as Google.

In addition to having SMTP setup, I have IMAP setup, so I can retrieve email from my phone or other email client that various other self-hosted services on my local network generate, or my desktop PC running Thunderbird.

I do have to monitor logs as I get regular password-guessing attempts on the services. I have a script that checks for repeated failed logins or other bad behaviors and adds malicious IPs to a blocklist ipset.

It would be interesting to federate with others and participate in a private SMTP network. I'm sure these already exist.


I run an AWS instance that hosts my own domain mail server. I have hundreds of email addresses, and each of them is forwarded to a gmail account. (I use the myaddress+tag@gmail.com to make each one unique. I do this to identify and squash spam - if one email address becomes contaminated, I delete it and change my email on the compromised company's server.

I also run a mailing list server.

So my email is usually sent from a gmail.com address, and I usually receive email on my own domain.

Some lessons - sending email from your own domain is difficult as you have to not only make it accepting to spammer-averse sites. You also have to protect it from sites that would LOVE to relay email through your server.

As for receiving and reading email on your own domain - you have to provide your own spam filters - and this is VERY DIFFICULT. 320 billion spam emails are sent every day, and 94% of malware is delivered in those emails. That's one reason I use gmail as the way I read email.


if you are running on aws, then using ses "solves" outgoing mail?


I was a little discouraged after reading previous threads here, but decided to give it a try anyway.

I tried mailinabox and mailcow which didn’t work out, but mailu did.

Final setup is a pfSense vm on vultr, VPN back to my local pfSense box (with snort filtering) and mailu in an LXD container. Mailu guides you through setting up various dns records which helped me a lot.


I've been using https://maddy.email/ for self-hosting from my own home server. It's amazing! It's the first time email setup has been easy for me. I don't have any problems with SPAM so far, and gmail accepts my messages without issue.


I will second this. Using for some time and it works like a charm.


You can host your own email but there are a few things you need to do:

1. Setup SPF, DKIM, and DMARC - you need to do all three.

2. You want to use an IP range that is not otherwise used by spammers or marketing people. This may be hard with a virtual server in just about any datacenter.

3. Consider a 3rd party spam filter, unless you like spending a lot of time on spam filtering.


I hosted my own email for years and had no real issues. The only reason I stopped was that my main motivation for self-hosting was security, and there are now providers (ProtonMail, FastMail, et al) that are reasonable options to solve for my needs and require no maintenance effort on my part.

My experience has been that all the major providers will deliver mail if it's appropriately signed with DKIM and you have proper SPF records, as long as you aren't originating from an IP that has a low reputation score. The biggest challenge is getting a clean IP, since there is limited IPv4 address space and most IPs have been recycled so much at this point that they all have low reputation scores. The best way to get an IP with a good reputation score is to host on physical hardware, not on VMs, with a smaller provider that has minimal customer churn.


Hosting email isn't hard. Hosting email that blocks out spam, and prevents your server from being hacked is more work.

There are some nice middle grounds for self hosting like zimbra, or for the less adventurous, a tool like MDaemon which lets you self host but provides support in the right areas. Incoming email is pretty trivial, and outbound email can be greatly improved by using an external outbound email service. Whatever works for the level of time you want to invest and maintain.

Try it out with a domain and you'll see if it's ok for you. You have to remember a lot of developers today don't have devops or hosting skills. There are developers who did have to manage complex web hosting themselves so it's not completely out of the realm of possibility. I self hosted for a long time, switched to cloud, and am thinking of switching back.


First, you need to set up the technical side: rDNS, SPF, DKIM, DMARC, no IPv6, TLS, etc. Lots of guides on the internet for that.

After setting up the technical side (which you can test with a site like https://www.mail-tester.com), you need an IP (v4) without a bad reputation. This is the hardest part, because it's less easy to control.

If your IP has been used to send spam, or sometimes even the neighbouring IP's, you won't get through to a lot of providers.

These are the best places (in my experience) to check IP reputation:

https://talosintelligence.com/reputation_center/lookup https://senderscore.org https://www.barracudacentral.org/lookups/lookup-reputation

And also check for blacklists, for example with https://mxtoolbox.com/blacklists.aspx.

If your IP is on a blacklist, you can remove is most of the time by requesting it via their (90's looking) websites.

Also, if you send a lot of mail, Microsoft and Google have programs for senders to monitor reputation:

https://sendersupport.olc.protection.outlook.com/snds/ https://www.gmail.com/postmaster/

But... even with an IP with a neutral reputation, your mails may be sent to the spambox. You need some volume of legitimate email over time to build trust (this is called warming up an IP).


Write a letter to your Senator and House representatives. This is an abuse of monopoly position by Google.


Its true that self-hosted email servers usually get marked as spam but you can fix a lot of things and make it less likely to be blocked as spam using tools like: https://www.mail-tester.com/


I run my business' email on a cloud VPS. It's a two step solution, with a FreeBSD host running a tightly configured Postfix as my mail gateway, and a Windows server running hMailServer (This is a great piece of Windows-based open source software. Highly recommend.).

This solutions captures the majority of spam and phishing. Occasionally, a well-crafted piece of spam gets through, and I check the Postfix config to see if I can close that hole.

What I do monitor closely are the valid emails that Postfix rejects. This happens a couple times per month, and is mainly due to the sender using GMail, and Google's mail servers being marked as sending spam.

Overall, I'm pleased with this solution. It's minimal configuration, minimal maintenance, maximum usefulness.


You can self-host, it works fine. If you want to run the server from a residential IP though, you will have a bad time. Otherwise, you may have to manually delist your IP from a few blacklists. It's not nearly as odious as everyone claims, it just isn't very fun.


no matter how impeccable your rdns, spf, dkim and dmarc configuration is, even if you're an absolute master of configuring postfix and opendkim, etc, your outbound smtp deliverability success rate is going to be very much dependent on the reputation of your IP space.

the best possible IP space will be somewhere that the entire /24 and parent /22 or larger block does not belong to anybody else's low cost VPS, VM, dedicated server or shared hosting. Which is hard to find these days unless you personally know somebody at a mid sized regional ISP that can sell you a custom package of colocation and some small sized piece of public IP space (like a /28 or /29 for your server) in known clean IP ranges.


Networks are two things: technological constructs and societal constructs.

Technologically: email still works the way it always has.

Societally: because email as a construct wasn't built to account for spammers, and the after-the-fact patches to address this issue aren't universally rolled out or universally adopted, and there's little incentive for the big incumbents to provide opportunities for new players by shaking up this status quo, email has devolved into a (human)-trust-based network where your mail only gets relayed if the relaying source trusts you as a person, not a pseudonymous node online.

So you can stand up an email server, but the real legwork is in getting a trusted authority to peer with you and route your traffic.


I "sort of" hosted my own email using Sendgrid API free tier to send and webhook to receive, and a custom client in a Python MPA. I was chuffed I wasn't paying 5 a month for GSuite.

I looked later at, and set up, https://mailinabox.email/ and that worked fine too.

I'm not sure about all the cant, it's definitely possible and I never had an issue with deliverability. I had no idea what I was doing but I made sure I got all the right dkim secret and signing keys or whatever that was required set up "extra special like" for both solutions.

I was not sending mail merges though so maybe that would have thrown things off I don't know.


The biggest difference between the 2 protocols (Mastodon and email) are age and volume.

Email is old and used by everyone, Mastodon is new and used by nobody.

Email is targeted for attacks because it's used by a lot of people and there has been enough time to develop mass messaging tools.


A simple way to get most of the benefits of self-hosted email (run from your own domain, keep full control of your data, manage mailboxes yourself, etc.) without the problem of being marked as spam is to use a service like Amazon SES as an SMTP relay.


It is very doable, we host our own email servers at SmartSurvey and send a relatively high amount of traffic.

Depending on your skill/software it might take time for you to test your configuration is setup correctly although there are sites like mail-tester.com that will tell you whether you have SPF/DKIM etc. setup correctly.

The problem you might have is with cloud IP addresses. Since these are reused heavily, it is possible some attacker previously used your IP to send spam and got it blacklisted. If not (there are services to check IPs) then you should be fine but note that some lists block /24 ranges of IPs instead of specific IPs so some providers are fairly unusable.


What I don't get that we now have all the DMARC infrastructure, that should in theory enable to use the reputation of domains instead of IP addresses, but apparently nobody cares, and your IP has to be reputable, no matter what.


I was looking for this recently but as a Web3 service (so emailing other Ethereum addresses) and found one that let you host an email account but you had to stake X amount of money to use it to avoid spam (spam gets too expensive /awkward to do when you have to hold $100 for a week before you can use it). Didn't sign up but seemed promising that there could be some cool self-hosting options in the future that somewhat mitigate spam and avoids the whole current email issue. Obvious giant caveat that you can only email other people who have Ethereum wallets but still, thought it was pretty cool to avoid the blacklist problem.


I think this could actually be a viable system for stopping spam if it was retrofitted onto the existing email system. Rather than each account putting up the bond, though, it should be done at the level of domain names, and (to encourage adoption) all existing mail-sending domains should be grandfathered in (allowing them to use a $0 bond instead).

The only question is how do you stop spammers walking away with their stake and creating another domain? There needs to be a way to slash someone's bond, which requires some sort of consensus. With DKIM signatures, it's fortunately possible to cryptographically prove that a given email was sent by a given domain, but ultimately you need to give potential censorship powers to some entity.

My suggestion for doing that is getting the ITU to vote on a set of maybe 7 organisations (e.g. mail providers, universities, non-profits) who share spam reports with each other and can slash a bond if at least 5 of them agree. Of course individual mail providers would be able to override these decisions and continue to accept emails from the blacklisted domains (and they could obviously continue to use other forms of spam filtering), but ideally the bond slashing mechanism would only end up being used once as spammers tested it and then gave up.


> The only question is how do you stop spammers walking away with their stake and creating another domain?

This is why I think it makes sense for an email-by-email basis if you also enforce a deposit and withdrawal delay. It's the simplest solution that I think could work. Not perfectly, but who's willing to spend $100 per email account with a 2-week delay between addresses? Add a public blacklist where you can vote for people as spammers and implement on your email services if you so choose, and I think you have a pretty good system. You may well be right that there's an elegant domain-based system with slashing and consensus but I haven't thought about it long enough to think of one myself!

And yes you could do this off of Web3 but then you'd need an escrow account and a centralised party to hold the funds which isn't as decentralised as I was hoping for (at least with Web3 the stake remains in my wallet), but definitely possible!

Edit: Maybe you could do it at an account level - then if you get blacklisted you'd have to open a new wallet meaning not only the 2-week delay and $100 cost but an additional network fee. That means it's unsustainable in the longterm.


> And yes you could do this off of Web3 but then you'd need an escrow account and a centralised party to hold the funds which isn't as decentralised as I was hoping for

Right. In fact I was already thinking that this was a (rare) situation where a smart contract on a blockchain would make sense, because you want transparency and consensus and low-throughput financial transactions between countries that don't necessarily trust each other (or even have banking connections to each other).

I wouldn't necessarily call that Web3, especially as there wouldn't need to be any web servers involved, but I suppose the system would affect people's webmail, so I can't really object to the label.


I run several small business email systems in the UK. You do need to get at least SPF and the usual MX, A and PTR records sorted at a minimum. Also A and PTR and (E)HELO must agree.

You can relay your mail via another service if you need to gather some karma for your domain but ensure you get your DNS records right. That way you can run your own full mail system from a "dodgy" IP address.

It's not for everyone but neither is IT in general. If you can fathom a Mastodon server then you can manage an email system - technically speaking. However, you must get the basics sorted out and don't send anything that can be construed as spam!


You can do it, but spam filters are so extreme that you may find your mail goes undelivered or auto-filtered into spam.

Small hosting companies struggle with email deliverability, so you have to ask yourself if it's something you want to deal with. Sometimes if you end up on a blocklist there's no easy way to get off of it, and you may find yourself unable to get an email into someone's inbox until you have the time to redo a lot of work (moving to different servers, IPs, etc).

From my perspective, you're better off using your time to do literally anything else unless email is something you want a very deep understanding of.


There are two sides of email: sending and receiving. They use different protocols and often different servers. How about self hosting an IMAP/POP3 server and sending mail through the SMTP server of the registrar for your domain? I didn't investigate the feasibility but I own two domains and I'm sending email for both from the SMTP server of one of them (can't even remember why, so long time passed since it happened.) I receive on their separate POP3 accounts, check with K9 on Android (the original UI version) and download and filter to folders with Thunderbird when I have to.


Being a bit of a pedant here, but you aren't "receiving" mail via POP3, you are "retrieving".

The sending and receiving (i.e. transfer) are both part of SMTP.

If you want to self host your own mailboxes (e.g. IMAP/POP3) you will need some way to get the mail into them (mail delivery) which is also (usually) SMTP.

But your point definitely stands - You can handle the inbound side and outsource the outbound mail delivery issues.


You can definitely self host. In terms of spam there's a bunch of config you need to do to not be put into the spam folder automatically.

I've done it for my side project, not saying it was easy and require a bit of linux knowledge but nothing you can't google.

Off the top of my head you have to setup your DNS and enable DNSSEC SPF and DKIM records Make sure your reverse dns is setup correctly DMARC as well

After that it's pretty much the same as your big email providers you build up your reputation by not sending out spammy shit emails then that's it.

Note: Be sure to secure your email server otherwise someone will try to hack it


Google still marks some of my mail as spam even though I have jumped through all the spf/dkim/dmarc hoops. Yet I still get tons of actual, obvious, previously-marked spam in my gmail inbox. F google.


You can just fine. The nay-sayers are, frankly, just repeating old-wives tales for the most part.

I've been running my own mail server from a VPS (vpsdime) for somewhere around 6-7 years now and the only issues I've had were related to general sysadmining headaches.

Postfix+dovecot(+mysql) setup, SPF/DKIM/DMARC and reverse-pointer record set. Both microsoft and gmail happily accept my emails and give them all the little checkmarks. Checked just a few weeks ago as I added an new domain.

It's a bit of set-up, but email is ancient tech: once set-up it doesn't need much tending.


> I can host my own Mastodon server, or all kinds of other novelty / fun things

Curious, how easy it is to set up things like a self-hosted Mastodon? Is it a few minutes, a few hours, or a few days worth of work?


It depends on skill level. I have a lot of experience spinning up services and own a home server, so probably it would take me a few minutes. That being said I've not tried it yet but will be doing this weekend


I self-host my own e-mail.

There's a webpage out there (https://dnschecker.org/ip-blacklist-checker.php) that lets you look to see if you've been blacklist and there's only one massively aggressive DNS rbl that blacklists me out of many dozens of them.

I'd suggest actually trying to setup your own e-mail/dns and see if it works or not. If you wind up on a static IP that is in a ton of RBLs, move to another block or another cloud service.



I was kind of hoping someone else would followup with more complete info if I posted that...

(Corollary to Cunningham's Law)

One of those multichecks found 2 lists that I'm on out of ~250 checked lists (and one of the ones banning me is a .ru address)


What annoys me most about email is that it's painfully hard to deploy. Other things are pretty straight forward, you just run them as service, probably some database with them, some http proxy and that's it, you're good to go.

That's not the case with email. You have a lot of different components to it, like postfix, dovecot, opendkim, and wtf else with as many confusing configuration options and DNS fiddling. It is so overly complicated for what it does that I'm starting to think that email was a mistake....


Email just like all of the first generation internet protocols (like FTP) suffer from the same problem. It was born in an era where security and issues stemming from mass usage were non existent. As time went by some US companies made email almost a monopoly (Gmail) while other parties added layers of after though security (DMARC, SPF, etc) while at the same time advertisment (spam) also made the situation worse.

Hosting your own email is a problem because of these things, otherwise a fun exercise (that I gave up long time ago).


At a high level, email is two-way (email hosts have to accept your email servers' communication if you choose to send to them), but navigating to a webpage is one way (client initiates an HTTP download from the server). Therefore email needs some kind of 'web of trust', since email servers have to trust that other valid email servers regulate their spam and aren't malicious. Not saying the current situation is ideal necessarily.


I think the standard answer is to use a smarthost.

The blacklisting shouldn't be permanent, though; my self-hosted mailserver took about a year to earn the trust of GMail. I hzave no idea how Hotmail's spam filtering works. I think they just shitcan randomly, unless it's sent from Hotmail.

I'm no longer running a self-hosted server. It's hosted by my (obscure) ISP, but it's still a custom domain. I have no deliverability problems, not even to Hotmail.


You can. I did that for decades until I got tired of maintaining that server.

Use a reputable VPS provider (one that's not likely to tolerate their customers being spammers), once you get the server check the IP against various blacklists (get a new server / IP if that happens). Make sure you set up your SSL certs and DKIM, SPF and DMARC properly.

Across the years, I had very few instances where my outgoing mail ended up in someone's spam box.


I've been self hosting email for 10 years now, don't have an issue with spam/blacklisting, I would say claims of self hosting emails death have been greatly exaggerated.

The main thing is to have a regular TLD, not something like .biz or .xyz etc, use DKIM/SPF, >> ensure your PTR record is setup << (critical) to match the domain name you're hosting your mail server on and you should be good to go.


You need to have a domain name with the DMARC information in the TXT DNS record. Also most large email providers have some sort of separate "registration" process that internally helps them perform heuristics. For example, Google calls this "Google Postmaster Tools" [1].

[1] https://www.gmail.com/postmaster/


you should be able to host your own e-mail system.

the first BUT ... :)

* an "e-mail system" consists of many components which have to be (tightly) integrated -- MTA, MDA, MUA just to name the most obvious ones

* e-mail touches a lot of different fundamental areas of (inter)networking -- especially for example DNS ... if you don't get those things right, your outgoing mail will be considered SPAM by most of the larger mailsystems out there

* last but not least: the annoying incoming SPAM ... you have to setup some kind of filtering and maintain it

* additionally: you are targeted by several kind of attacks, because mailboxes are an essential bottleneck in lots of security-related workflows ... read: password reset / access-tokens etc.etc.

to integrate all of this is not trivial, but also not rocket-science, it just demands experience and knowhow in a lot of areas.

which leads to the second BUT ... ;)

those areas are sooo uncool right now - cloud/services/mobile/crypto/web3 ... these are the hip places to be today ...

why bother with something sooo 20th century like operating a mailsystem or a server-system at all!?

only "weirdos with alu-hats" and "old farts" are doing this ... (i'm one of them ;))

br v


After all websites started requiring email for interactivity, it became a proxy for the identity, propped up by some trust in Google and whoever else. No one is interested in letting you ‘confirm’ your identity. This is more evident now that email hosts require the phone number in turn.

If you're reluctant to use the centralized identity, you obviously have some malicious intent.


You can, I do. It's really no problem.

You do have to make a bit of effort to setup DKIM, SPF etc. But really it's not too difficult.


I've been runnning mailcow[1] on a small Hetzner vserver for a bit more than 4 years with minimal downtime and maintenance.

I followed their guides for the most optimal setup, and so far i feel deliverability has been comparable to commercial offerings.

Can certainly recommend.

[1] https://mailcow.email/


Yes, SMTP is amazing. It's a decentralized, fault tolerant messaging protocol. Civ2 used to have multiplayer over SMTP.


Oh man, mention civ 2 and hit me in the heart strings.


Is the impact of DKIM and SPF on email delivery measurable? Have any studies on this subject been published? Such an important part of our lives should be better understood. Reading this thread feels like witnessing clergy swinging thuribles towards the congregation while mumbling the gospel to build a connection with God.


DKIM and SFP by themselves haven't done a ton, but they both feed into DMARC which absolutely has had an effect. It's probably just not what you think.

The point of DMARC is to protect senders from spoofing. I.e. if I'm PayPal I don't want anyone to be able to spoof email from my domain, so I set a DMARC policy to reject if SFP and DKIM both fail. If I own a domain that I never want to be used for email I can set that same DMARC policy and just neglect to set up SFP and DKIM, therefore ensuring they fail.


You can host your own email. I do it. I don't really have many problems. Every now and again someone's misconfigured mail server decides that my email is spam but really this doesn't matter to me so much. At least the major email providers that everyone else seems to use happily accept my emails.


You can host it yourself. For example with a Freedombox: https://wiki.debian.org/FreedomBox/Manual/Email?action=show&...


E-Mail isn't email.

Provide as MX to your own is impossible if you only have dynamic ip: Most ranges are on spam lists, for good reasons.

Hosting everything else is possible.

I work for a company that makes use of three components: a so called smarthost, that does AV and Spam detection, a newsletter host and groupware server.

Many more combinations are possible.


Have you actually tried to? I've been running my own server for like 3 years now, works fine. You need to actually set up your SPF/DKIM/DMARC/PTR records/etc correctly, but I've never had an issue with my mail getting delivered or being sent to spam.


Nothing can be done. Everything will eventually succumb to spam. Spam is omnipotent. In fact, Google and FB are just giant very successful spammers. The only difference between Mastodon and email is that email is 40 years older. As Mastodon grows, spammers will take over.


You absolutely can host your own email. I do and have since before Google existed. Antispam hurdles do exist but clearing them is just a process. For most people it isn't worth the effort. Sometimes I wonder why I still bother, but it is definitely not impossible.


You can do whatever you want. For a pull-based service, I have no problem with you posting things, but if you're sending to me, I make the rules and I'm not interested in prioritizing your freedom to send things over my freedom to not receive things.


What do you mean by hosting? It is basically necessary to relay outgoing mail via a reputable ISP in order to not be spam-sorted. They will then make sure their outgoing IPs are respected.

For incoming mail you need a spam-filter, but that is in your hands.


Without scarcity, there will always be spam.

https://messari.io/article/look-to-the-stars-navigating-the-...


Mastodon is too obscure and small for anyone to notice. If Mastodon ever achieved the scale of email, the necessary anti-abuse measures would also result in centralization of Mastodon services, just like we see with email.


I don't get it: I'm selfhosting my own mailserver and I have no issues with gmail / outlook marking me as spam, what is everyone talking about? You just need to set up DKIM, SPF and wait a couple of weeks.


...and make sure your IP range is never in any blacklists, or pay to whitelist your IPs


my ip is in the joke blacklist UCEP because "my provider doesn't care about spam". Still no issues sending to gmail.

I'm running a tor node too (guard / middle obviously).


Yeah Gmail doesn't care about UCE-PROTECT level 3, but Outlook.com extremely does


Why can't you? I've done it for the last decade or so with absolutely no issues.

I even run my own webmail suite.

I even have about half a dozen friends who also use it on the regular.

Not sure what's stopping you or anyone else?


That's strange. I have a virtual server running vor about 15 years with several upgrades. Debian stable. Email works just fine with the occasional spam mail coming through.


I hosted my own email server from 1995 to 2010 (Solaris on a Sun IPC, sendmail and IMAP compiled from source). What finally drove me to move my domain to gmail was the spam.


I have been hosting my own mails for years

Although not directly. I got a webhoster, with my own domain, and the hoster also provides mail servers

I never noticed any problems.

Although often people do not respond to my mails.


The problem is that decentralized services only work when everyone is on decent behavior. When the degenerates onboard, you need the harsh hand of a centralized authority.


> you need the harsh hand of a centralized authority.

What you actually need is reputation information, which you can make your own decisions based on. Unfortunately the cheapest way to gather reputation is with something like a panopticon, where one party sees all interactions and everyone trusts their judgement.

It's at least theoretically possible, though, for individual nodes to securely gossip reputation information to each other, which is the model that the Matrix.org team are working towards:

https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix...


Nothing to fix, it's already solved . I host my mail using cloudron on digitalocean. Have you tried selfhosting or are you guessing it will be hard?


this looks like prime territory for another federal oversight.

essentialy email has been taken over by the minority for financial benefit, and creates a noncompetetive situation.

it has been obvious for years that small email servers are being squelched for thier properties, and independence rather than for something that has actually occured.

this is direct interference with a computerized messaging system, and it is causing harm.


I have been self-hosting my email since 20 years. It can be done. I haven't had problem with mails being marked as spam for a long time.



Recently switch to Hetzner and have had 0 problems. They maintain clean IPs and if they catch you spamming they block outbound mail


Why you no host? (https://yunohost.org/) ;)


You can host your own mastodon server but if it fills with spam others will refuse to federate with you.


You definitely can but trust me - it is a huge pain in the ass

And I generally recommend doing a lot of hosting yourself


i can. so you can too. the hardest part is finding a clean ip in a clean block. the rest is quite simple and straightforward. dkim+spf set up was enough for my opensmtpd to deliver my mail to all the big players without any going to spam


You need to maintain your ip’s reputation, which, for ourselves, is barely possible:(.


It is not you. It is the monopolies like Gmail, Outlook.com etc that make it hard.


You can. Ive set up mail records for jobs before. Its just a question of maintaining the dns records and paying whatever fee to your registrar like everything else.

Not sure what the story is with self hosting spam filtering. I think most people call an external service


To me it seems you can, provided you own a good IP address.


I set up mail-in-a-box a few years ago and it works great.


That's what I use, but because I'm running on a DigitalOcean vps Microsoft still blocks me. I've not had any issues other than Microsoft though, Gmail was easy to get whitelisted.


Email is as close to ‘fixed’ as you’re ever going to get.


you can host your own email : https://mailinabox.email/


Because all the big players will mark you as spam.


It's doable, and it's quite easy


because it's secretly a social network and those are very hard


Works for me. Shrug.


The flagging and blocking of legitimate messages from domains with good standing is not a general problem. But it is a malpractice primarily of Microsoft and Google.

The reason is that they have clear financial incentives to sabotage communications in order to press people into using their services.

Given the centrality of email to daily life and business, big-tech service providers do cause a clear harm by their actions, and what is needed is are successful legal prosecutions against them for that practice.

It's an interesting problem. In days of yore, the Royal Mail and US Postal Service had special powers of state that made letters sent effectively certain to be delivered. Interfering with the mail was a very serious criminal/federal offence. At one time in British law, the mere act of posting a letter was considered tantamount it having been received, and to this day many legal processes derive from that, even though the delivery has gone electronic.

The principle agent problem is that as service providers they are not impartial in their actions. While they may superficially seem to have every right to block communications to users within their domains "to prevent spam", or on "security" grounds, they are intermediaries whose good faith is not ascertainable. Even if they were acting in good faith it is a classic example of iatrogenic over-reach of "care", by which nefarious motives may be veiled by an appeal to seemingly reasonable "security" policies.

While their Terms of Service clearly make no warranty for delivery of important messages, this is not an acceptable level of service. We have hobbled along in this legal no-mans-land for 30 years at least.

Therefore I see the problem as more akin to a net-neutrality issue and is more subtle as a rights issue than it first appears. How can big service providers with an obvious conflict of interest and incentive to sabotage legitimate communications be made to play ball?

Technically we can prove, from mail relay messages, that communications have been delivered to the boundary of their digital estate, so the technical basis for a legal intervention is there. Part of the problem is that recipients who are harmed need to coordinate with senders that experience harm. And both parties need the basic technical competence to know and prove that they are being harmed.

Your mileage and anecdata may vary, but I maintain multiple mail domains for projects and businesses in Europe I've set up for people which send dozens of messages per day from self-hosted servers. Almost none of these have any problems. The only problems people report come from recipients on Microsoft and Google systems.

I sometimes politely ask people explicitly to white-list addresses, or carefully explain why a non-Google/Microsoft alternative address is a more professional and reliable contact point.

Nontheless, the 'crimes' of these big companies causes me tangible cost and I do with there was a legal remedy for it.


A lot of people have asked us why they can't host their own email. It's a great question!

Hosting your own email is about as easy as hosting your own website, or server. Basically, you would have to install an email server on your own computer, and then configure it so that it can send and receive email.

If you've ever tried to set up your own web server, you know what kind of a pain it can be. You'd have to install the right software, configure it, make sure it's secure, and do all kinds of other stuff that you probably don't want to do (unless you're one of those rare people who just loves doing stuff like that).

If you think about how much time and effort it takes to manage a single computer—let alone a network or an entire data center full of them—you'll see why we're in the business of helping people with hosting, instead of telling them to host their own stuff.


if i’m not mistaken this happen because e-mail server don’t have an identity.

it’s too easy for scammer to create a domain name.

or is it only because smtp doesn’t have any kind of authentication and rely only on IP adress?


out of the box this is the case, but improvements like DKIM/DMARC/SPF etc have allowed for a bit more authentication


I wonder if a crypto-based email system would solve these problems. With crypto, you have cryptographically secure identities, and if you had to pay for sending emails, this would deter unlimited indiscriminate spam.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: