I wouldn't be surprised if that's intentional. There's an explicit hesitance on the part of mail providers to accept v6 mail, since they use IP addresses as a reputation mechanism. IPs that originate spam mail get summarily executed, and getting new IPs that have a high antispam reputation is actually quite expensive.
In other words, it's a Sybil-resistance mechanism, called Proof-of-IPv4. It works specifically because v4 addresses are scarce. v6 addresses are not nearly as such. Everything that makes IPv6 great for the Internet at large makes it terrible for mail providers. For example, because the original v6 design wanted to eat lower link layers, it reserves half the v6 address for an embedded MAC64. This never really panned out, but it's terrible for security, so every v6-capable OS nowadays will rotate addresses every few hours. The average machine will have hundreds of addresses. How do you assign a usable notion of per-IP reputation to that?
You could use v6 subnets for reputation, but there's still 64 subnet bits - enough to stick an entire IPv4 subnetwork inside of each IPv4 address. Some ISPs actually will assign a /64 per customer (because Comcast needs something to sell to Business customers), while others assign /56s or /48s. So there isn't even one granularity of subnetting that you can use for reputation tracking on v6.
Meanwhile, v4 pricing is getting worse and worse, which is great for mail providers. They don't necessarily need to turn a profit on incoming mail, but they do need to make it expensive for people who want to send lots of spam.
This could likely be the reason for poor IPv6 support but highlights the importance of shifting (much more) to domain based reputation. If a domains reputation is at risk, you can bet domain holders will be extremely careful not to allow outgoing spam.
Spammers and scammers already use domains as a disposable commodity creating them or using hacked ones for single campaigns and moving on. Part of filtering based on IPv4 is not only scarcity but accountability. When the owner of the netblock reassigns the ip and its already blacklisted it can create a problem for them and incentivize them to police their own network. Domains are also worse in that its easier to use fake information and be untraceable. its also understandably easier to get a response legal or otherwise from a co-location or isp than a domain registrar. Maybe ipv4 will always be preferred for email just because its more difficult/expensive and therefore less appealing for temporary malicious use.
Domains are less scarce than addresses. By design you can create as many subdomains as you like. (e.g. `abc.spam.com` is too low-rep? Now let's try `def.spam.com`...) You might imagine negative reputation to travel up to parent domains, but that causes problems with public suffixes and TLDs. (e.g. is `microsoft.com` a bad domain because it's got the same TLD as `spam.com`?)
The whole point of using a scarce identifier is to allow for a "neutral" reputation for new identifiers. If identifiers are less scarce, then known-bad actors can get free reputation (from bad to neutral) by just starting over with a new one. Which means that you have to distrust neutral reputation more. Without some level of scarcity of identification, introductions don't really work, because I have no idea if the new host I'm being talked to from today is just the one I banned yesterday wearing a different mask. This ultimately implies e-mail moving to some kind of federated whitelist system rather than the current system of federated blacklists.
Subdomains are not as much a problem as you would think. There is a resource ( https://publicsuffix.org/ ) that lists all public suffix domains. All direct subdomains of these are in full control of all their own subdomains, and thus can share the same spam reputation.
e.g. when .com is on the list, and .somesite.com is not on the list, mail@somesite.com is from the same entity as mail@subdomain.somesite.com
Didn't publicsuffix effectively get DoSed by one of Apple's new requirements, causing a ton of people to apply to have their suffixes added to the list?
From what I gathered from that, publicsuffix is a poorly-funded semi-volunteer org that shouldn't be relied upon for anything critical.
They unfortunately use both domain and ip based reputation scores. The problem is there are effectively an infinite number of usable domains. Even after eliminating the sub-domain problem the fact is there are simply too many possible domain names that can be created and discarded on the fly for less than $5 a pop. Given the fact that bad reputation decays, they can simply rinse a repeat that process practically forever so long as they manage to make more than $5 per hour from thier spam. IPv4 addresses however, are far more scarce which is why most spam email opperations try to take over existing legitimate small email servers (commonly small businesses with thier own domain get targeted) in order to send out thier spam. Every time they succeed they use it not only to send spam emails, but Trojan viruses to all users contacts in the hope of infecting other businesses. They can even achieve this without infecting the server itself, but simply getting recipients to unknowingly run a script that tells Outlook to send the emails from whatever addresses the users has access to.
Filtering based on domain reputation has the same problem as trying to filter based IPv6 address reputation. They can easily change thier domain name at any time, and most spam operations do it every 15 minutes or so.
This also has a secondary problem for legitimate domain buyers. If the domain name they buy was previously used for spam that reputation will affect thier business for quite a while. There's actually a market where people buy domains with bad reputations, setup small legitimate businesses and get the reputation cleaned up, then sell the site domain and business for a substantial profit because a site with a good reputation history and established line of business will show up higher on internet searches.
Or more strict enforcement by the world on SPF, DMARC and DKIM policies
The problem of spam is actually solved, the problem is no one setups any of these security parameters correct, large and small companies alike all have bad SPF Records, bad or no DMARC, etc etc etc
Go to any internet-related forum and search history for those keywords. You will find countless stories of seemingly technically people who in the end give up on self hosting and switch to managed mail provider. Because even if you solve those policies perfectly, a personal mail server will have such a low rate of outgoing mail that all the big players will effectively treat it as history-less server and will occasionally route the mail into the black hole. There is no recourse for that.
If 99% of contacts you want to send mail to are on google/yahoo/microsoft you have to play by their rules. And those rules are effectively "send mail internally or gtfo".
I have self hosted personal mail for over a decade. There are occasional hiccups with deliverability to new gmail addresses, but that is it. In those cases, once a recipient marks me as not spam once, there aren’t any more problems.
I think maybe once in the last 3 years I ended up in someone’s spam box, total. In fact I just sent to a new gmail address and to a university I have never contacted before this week and both were delivered without issue.
Setting up DKIM/SPF/etc isn’t that hard and it’s fairly easy to verify with existing tools FYI.
It would be amazing if those of you who have successfully self-hosted would get together and make a comprehensive write-up of how to self-host without getting blacklisted. I frequently see comments on both sides of this ("I can't send anything!" vs. "You just have to do it right") and it seems like if there could be some resource (that cuts down the complexity as far as is reasonable) on how to do this from the ground up, there could be much more wide-spread adoption of self-hosting.
Personally, I'm hesitant because I don't know if the end of all my effort will be constant blacklisting. If I could be confident that if I do it right I won't get blacklisted, I probably would.
I think the parent is perhaps just lucky. A reason given for mail not getting to Microsoft servers was that a server IP controlled by the same hosting company had in the past been on a blacklist.
I was happy to move hosts to one that was considered trusted, but there was no way for me to know the IPv4 addresses the company had in the past, never mind if they'd been on a pertinent blacklist at some time.
Based on that I think it could work, but there are no guarantees that outside, historical characteristics won't screw things up for you.
I think I agree with you (RE: make a comprehensive write-up), but I also seem to remember thinking the same thing when I was setting up the latest iteration of the server and then googling it and finding that such a guide already exists :) Part of the problem is that it's an evolving system and things change over time.
Spam is an interesting problem. Assuming one self-hosts and makes their email address publicly available, then one can get a metric for how much spam is flying around. Eventually one will try to stop spam from coming to their inbox, and on doing so one might build a mental model for how the big mail providers combat spam and realize why one's emails are not being delivered. Then one might realize that one is sending mail that one would not willingly receive! And then take action to resolve.
In general though, there is some base effort to establish trust, and as long as you don't ruin it by sending spam, then you shouldn't end up on a blacklist. If you find that your IP was on a blacklist before it became yours, then work with the people that are blacklisting - but at that point it does become a bit of a job. I actually ran into an issue in my professional life where an AWS WAF rule started alerting on one of our own servers hosted in AWS because someone had previously used the IP for malware C&C.
Anyway - I will think on this and see if I can write something up. It's a good idea. My main concern is that there is a gap between the way I did things (sysadmin style) and the "new" way of doing things (containerized).
I use shared hosting and my trick is to only send emails that are more important to the recipient than to myself. In this way, if they don't receive it, they complain and then I resend it from gmail instead.
The biggest thing people do wrong with self hosting email is using residential IPs which are almost universally black/grey listed. Using a provider like linode, and checking the IP reputation ahead of time gave me better results when I self hosted.
That's not them doing something wrong though. There's nothing wrong in hosting a server at home. The problem is clearly the black/grey lists if they black/grey list residential IPs just because.
Might want to check the Terms of Use / AUP on your Residential service, because every one I have ever read says you can not host email, and 80% of them say you can host any server of any kind.
Now most of them do not actually enforce it unless you become a problem, but most of them do put active measures on the network to stop SMTP Servers
For example here is a Exerpt from Comcast AUP prohibiting email and web hosting [1]
>>>use or run dedicated, stand-alone equipment or servers from the Premises that provide network content or any other services to anyone outside of your Premises local area network (“Premises LAN”), also commonly referred to as public services or servers. Examples of prohibited equipment and servers include, but are not limited to, email, web hosting, file sharing, and proxy services and servers;
Google Fiber AUP[0] allows for hosting of anything for non-commercial purposes. Email is allowed as long as the email sent is not "unsolicited bulk commercial email". Time to update that worldview :) Every time we repeat something like "no one allows self hosting", the more people that will hear it and repeat it and make it a self fulfilling prophecy.
Why? Google Fiber is Available to less than 1% of US Households. Comcast is the largest Residential ISP last I looked in about 60-70% of US Markets...
ATT has the same policy, and I believe most of the other Cable Providers do as well. My guess would be over 90% of Residential Internet Plans today have a policy inline with Comcast not Google Fiber
Pointing to an outlier to the norm does not mean i need to update my worldview at all.
i don't know about that. i've always used leased gigabit connections for my email. this article is about Hetzner specifically, which I've leased tons of servers and additional IPs from. AFAIK you don't know the IP you'll be getting until you pay for it unless things have changed recently.
With Tools like Mail-In-a-Box it is not really the tooling
The people that say "I Cant send anything" are likely trying to setup it up on a Residential or "Business" (which is really just a Residential with a slightly better SLA and less overselling) Internet Circuit... Not an Enterprise Internet Circuit
Hint: If they are trying to bundle TV Services with your Internet you are not on an Enterprise Line.
Even if you buy a dedicated IP from these services they still make is hard to impossible to send email on the circuit
you are confusing spam filtering with blocked emails.
The issue people have trying to send mail is with the latter, where the email won't even show up in the spam filters, it will either be blocked by the mail system or silently ignored.
When mail has gone undelivered due to non-spam reasons, I typically receive a bounce.
On my server, when I block a message due to trust, I reject the connection. When I block due to spam, the message is received but goes in the spam folder.
I get reports from Google/Microsoft/etc when other people try to send using my domains but their messages fail due to DKIM/SPF failures.
I just want to push back against the “it’s impossible to self host email” meme that seems comes around occasionally. Every time I’ve run into an issue there has been a solution.
> On my server, when I block a message due to trust, I reject the connection.
Is that how Google/Microsoft/etc. do it too? If not, then your practice really doesn't matter. Most of my friends have an @gmail.com address, so if Google would pretend to accept mail from my hypothetical mail server, but instead just drop it on the floor, that's a non-starter for me hosting my own mail.
I have never had a problem sending to gmail or Microsoft addresses that didn’t involve at least making it to the spam box. I can’t speak with absolute certainty how every email admin handles things, but I have run into issues sending mail to some domains where I got a bounce and was able to reach out to the admins and get it corrected. Accepting mail for delivery but failing to deliver must be a violation of some RFC or email standard as it makes it impossible to send mail as a non-megacorp.
What I would like to get across is that self hosting is not impossible or unacceptably unreliable. If Google and Microsoft have policies that make it difficult to send messages to gmail or hotmail users, that isn’t a reason in and of itself that we should not self host. It’s a reason that we should work with Google/Microsoft to have better policies - but accepting things as they are and writing off self hosting as impossible is eventually accepting control of email by a limited handful of corporations, which I don’t think is a good thing.
I've also had luck with self-hosted solutions. Gawd, email is a really backwards technology these days isn't it?
But...I've also been in the unfortunate position of leasing IPv4 addresses which were already blacklisted by various sources. It's not a terribly easy problem to solve if you need to contact customers NOW without using a 3rd party solution.
I'd imagine personal emails trigger different flags than a bulk newsletter from a small business. One of the key suggestions on setting up a mail server (or even something like Amazon SES) is to 'warm up' by progressively sending more and more emails. Volume and uniqueness are a big part of filters.
As the saying goes, the best time to plant a tree was 10 years ago, the next best time is today. Maybe it is tough to get trusted by the major providers in a short time period today, but if it’s something you’re interested in, don’t let that stop you!
Running a mailserver with correct DKIM, DMARC etc is not that difficult.
I recently made a presentation that has a full explanation of the techniques, why they exist and how they work, on Hetzner Cloud (from the original post):
Self-Hosted Mail servers have the problem that most ISP's will automatically add all of their Service IP;s to BlackLists specifically to prevent people from running mail services on their Internet connections.
Unless you have a Commercial Line that has be specifically designated for hosting content then it is likely any IP you are issued is added to Google/Microsoft/etc Blacklist by the ISP. Most of them clearly spell out in their terms that running a Mail Server on the circuit is forbidden.
I don't see why this is downvoted. Domains assign reputation to mail instead of the source IP, and it's fairly obvious that just buying a new domain and spamming from it would tank that domain's reputation for quite a while, even with proper spf/dkim and dmarc p=reject. If all of these are set up, you won't have issues sending from bad shared IPs like the default SES ones.
You often have to build a domain reputation first. Certainly for Microsoft hosted email. I for instance show users with a Microsoft email a plain mailto:support@domain.tld link on my contact/support form. This way the first email is from them to me which helps building reputation and minimizes the chances of my response going straight into the spam box or worse, silently dropped. Regular users can fill in a proper form and submit it from the support page.
I get more spam from Microsoft and other tech giants than anyone else.
It's the companies whom you rely on for email that are the worst abuser e.g. airlines need to inform you about delays and abuse this trust with holiday adverts incessantly.
Any company that claims to require your email for two factor auth should be given automatically generated fines for every email they ever send that is not auth related.
I like it. I've always wanted to promote mailto: links over silly contact us forms (and all the hoops you have to jump through to keep them functional and not abused) but never had a really good argument for non-techy folks and lots of pushback that mailto: is "not standard" and "does not work for some people" with very little evidence. This is a nice story for the 'pros' column.
Classifying IP sets is a fantastic idea, I’ve seen mail bounce for the ASN. That parameter is unchanged between IPv4 and IPv6. Certainly, you can do it only when the provider is a classic spam heaven.
>Some ISPs actually will assign a /64 per customer (because Comcast needs something to sell to Business customers), while others assign /56s or /48s. So there isn't even one granularity of subnetting that you can use for reputation tracking on v6.
This is what is hindering adoption everywhere to be honest.
All of my forums, wikis, and game servers reject ipv6 purely for the same reason.
given abuse coming from a given IPv6 address: which subnet do I need to block to stop the user behind that address
(for fraud detection it switches from block to identify)
for IPv4 this is generally the /32 (the single IPv4 address)
for IPv6 it's probably a /64, but may be a /56 or even a /48, and on some crappy providers even a /128
if the subnet is smaller than you think it is you risk banning an entire ISP (or country), whereas if if it's too large the abuse continues
it's quite a complicated problem as by design you can have subletting (subnetting!) within a block, e.g. a VPS provider gets a /48 from its ISP, and then they sublets out /64s to their customers (while not necessarily giving them all their own RIPE/ARIN records)
can i ask a question? is it possible for people to "own" ipv4 addresses? like we can own domain names? something like /29 Subnet or /28?
if i spent like a hundred bucks or something, i dont know... just asking. how would that work, does that "bring your own ip" that vps providers talk about mean this?
> do need to make it expensive for people who want to send lots of spam.
You can use cloud providers, sure small ones do get blacklisted (which happens to also benefit Microsoft as they also are a cloud provider) but they can't really blacklist Googles or Amazons Cloud.
In other words, it's a Sybil-resistance mechanism, called Proof-of-IPv4. It works specifically because v4 addresses are scarce. v6 addresses are not nearly as such. Everything that makes IPv6 great for the Internet at large makes it terrible for mail providers. For example, because the original v6 design wanted to eat lower link layers, it reserves half the v6 address for an embedded MAC64. This never really panned out, but it's terrible for security, so every v6-capable OS nowadays will rotate addresses every few hours. The average machine will have hundreds of addresses. How do you assign a usable notion of per-IP reputation to that?
You could use v6 subnets for reputation, but there's still 64 subnet bits - enough to stick an entire IPv4 subnetwork inside of each IPv4 address. Some ISPs actually will assign a /64 per customer (because Comcast needs something to sell to Business customers), while others assign /56s or /48s. So there isn't even one granularity of subnetting that you can use for reputation tracking on v6.
Meanwhile, v4 pricing is getting worse and worse, which is great for mail providers. They don't necessarily need to turn a profit on incoming mail, but they do need to make it expensive for people who want to send lots of spam.