I think part of it might be that I use Linode, and there are other spammers in their data centre, so I could just be on a subnet bad list. But I think a lot of it has to do with Google/Microsoft's spam filters just being crazy over aggressive.
A huge part of the problem is that while your personal smtpd might be set up impeccably, you're in the same /24 as a bunch of other low cost bulk hosting customers that have in the past several years set up VPS with much less clue than you. As a result the entire ARIN netblock that your server resides in has a "poor" IP space reputation as seen by the opaque incoming anti-spam measures put in place by Microsoft and Google.
There's very little that can be easily done about this other than moving your smtpd to an ipv4 address with an ISP that has never had an outgoing spam problem (such as for example a /24 that's been held by the same company for 8+ years, in ARIN/RIPE/APNIC/whatever space very tightly controlled by the network engineering team of a clueful local ISP where you know the staff).
There's a pretty direct inverse correlation between the cost of a hosting service ($5/mo VPS vs. minimum $200/mo colocation of a 1U server) and how much outgoing abuse traffic has been sent from the particular netblock assigned to the enduser customers. Cheap hosting company = poor IP space reputation.
>There's a pretty direct inverse correlation between the cost of a hosting service ($5/mo VPS vs. minimum $200/mo colocation of a 1U server) and how much outgoing abuse traffic has been sent from the particular netblock assigned to the enduser customers. Cheap hosting company = poor IP space reputation.
Eh...this is pretty much bullshit. Unless you're running your own, good-reputation AS (or happen to know someone running a clean AS who can either lend you an IP or announce a clean block for you), the /25|/29 assigned to you by your colo provider has just as good a chance as being dirty as that VPS providers.
Your colo provider is also more than likely happily delegating blocks to one of those cheap VPS providers out of the same larger netblock your IP. Chances are even good that you'll be given a /26 or a /29 that used to be used by one of the VPS providers.
If you see a provider offering "up to 256 clean IPs!!!", those aren't clean IPs. Those are, at best, greylisted IPs bought on the cheap and at worst /22's rented from poor rep /16's and rebadged in the hope no one will notice
Not really bullshit if you find the right hosting company. There's shitty $200/mo single server colo options, and less shitty ones. In my experience it comes down to personally knowing the people who run the ISP and how serious they are about network abuse issues (and general network engineering best practices). If you find a colo/cabinet space hosting company in a major city that's one of the founding members of a major IX there's somewhat more chance that they'll have network engineering staff who take such issues seriously.
On the customer side, one of the major bars to entry for clueless/spamming customers is whether it's possible to directly purchase hosting services online with a credit card and have them immediately provisioned and available. If you can get a VPS in 5 minutes by paypal it's easy to be clueless. If you need to set up and ship a server with its rails to a colo it's likely but not guaranteed that you have more clue than usual.
Yes, it's likely you might get an IP in a /26 that's part of a hosting company's much larger /22 or /20 which also contains shitty VPS. The key part there is to find a hosting company/ISP that doesn't do low budget hosting and never has.
Most people aren't in your or my position, where we might know (sometimes personally) the management at an ISP. Most people aren't in a position to build their own server. Heck, the vast majority of people who want to host any thing aren't in that position. And who wants to spend $200+/m to on their personal email?
The number of colo providers who refuse to allow customers like VPS hosting companies (who buy rooms not just rent a piddly little 2U from a shared cabinet), is pretty small.
All in, your advice isn't tenable for the majority of technophiles, let alone your slightly-more-technical-than-average user who wants to setup a mailserver. The barrier to entry, your way, is insurmountable.
Maybe it's time for a startup in the "DIY mailserver" space, that offers clean IPs for personal use mailservers and hand-holds through the process?
I agree that the vast majority of people who use email in general, or even the vast majority of people who own a domain name and want to have their own email server are not in our position...
The venn diagram of people who are capable of operating a secure Linux or *BSD based email server implementing, for example, SPF, DKIM and DMARC with postfix+opendkim+spamassassin+dovecot overlaps a great deal with the sort of people who want to colocate a $100-200/month server. A lot even have "free" colocation through their work at an ISP or with friends that have extra rack space and power for a small 1U box.
It certainly doesn't make sense to spend $200/mo on colocating a server just for email - but if you're colocating a whole physical server in this era, it's not hard to make it a box with 64GB or more of RAM and two good quality 512GB SSDs in RAID-1: Make it a hypervisor platform and put twenty of your own VPS on it doing many different things. Balanced with the need to not centralized too much stuff on one piece of hardware as a single point of failure. Public IP space availability depending, of course.
In my experience the best colo is with ISPs that are not actually colo companies, but will only do it as a side thing for people they know and trust. The absolute best colo I've ever had is with a company that has a core business doing X.509/SSL stuff for healthcare enterprise customers. Gear in racks two hops network-topology away from their core routers at a major IX point.
The vast majority of people who do not want to maintain and secure a world-facing Linux/BSD based smtpd are probably better off going with a google apps or hosted email solution where all of the smtp and imap/TLS1.2 infrastructure is handled for them.
I'd disagree, there are several packages that make it relatively beginner friendly (Mailinabox, sovereign) -- everyone has to learn some how. There are also lots of good tutorials, and one of the great pleasures of setting up a mailserver is once you've done it you can reuse the configs. It is a great learning experience for relative novices and "greybeards" alike.
You've just cut the number of people who can do it your way down to <1000 people (or may as well). No normal person, who wants to learn how to setup a mailserver and be successful at it, is going to drop ~$5k on a server or know people who run T1/T2 datacenters and are willing to pop you in a cage for free.
Your last sentences dismisses everyone who doesn't do things to your impossible standards as a waste.
You're basically saying no one should run a mailserver, or learn how, or be given the opportunity to learn how. Which is sad.
Yes, sort of, but not really a new problem. Low budget hosting companies have had poor IP space reputations and outgoing spam problems since such a thing as a hosting company began to exist, 20 years ago, way before there was such a thing as an x86-64 bare metal hypervisor or a VPS. The lower the cost the less clueful the customers. The problem is more on the google and MS side.
Have you also set up reverse DNS and done it all for IPv6 too? The second part was what I was missing a while back. I do agree that Google and Microsoft are extremely strict with what they accept and it's not always easy to tell why you get thrown to the spam pile.
Yep, I think there was a post about setting up IPv6 correctly on HN a while back. Alternatively, you can make your host not respond to IPv6 at all.
If I remember correctly, the problem isn't that IPv6 needs to be set up, but that if your host listens on IPv6, Google will default to that and then your SPF/DMARC/DKIM setup needs to work with IPv6. If you don't have an external IPv6 address, then you don't need to configure SPF etc... for IPv6 and you should be fine.
It depends, really. When you're just starting out with sending email from a fresh domain and IP address, you will most likely be flagged no matter what. Do not underestimate signals that come from user interaction (e.g. open and click rates).
I'm running on Linode and have been doing OK. Initially I was getting dropped in spam inboxes, but after getting things sorted out using mail tester, it seems to be better.
http://penguindreams.org/blog/how-google-and-microsoft-made-...
I think part of it might be that I use Linode, and there are other spammers in their data centre, so I could just be on a subnet bad list. But I think a lot of it has to do with Google/Microsoft's spam filters just being crazy over aggressive.