Yeah :/. I have tons of operational knowledge of email and have been running my own email setup since 1998. My email servers were ever only used for email I typed manually by hand (due to being dumb I never even used shortcuts to auto type repetitive responses to customer messages). Then, one day a couple years ago, I realize no one @yahoo.com is getting my email anymore. Talking to Yahoo, I get barely any response and certainly no explanation. I can easily believe that my entire IP block was cut for being a hosting provider.
I now pay some company to deliver my personal mail, which is just dumb, and clearly isn't caused by them being better about spam (as I sent no spam): it is caused by a militarized anti-spam-crazed group of people who have managed to build a system that essentially requires an oligopoly for email to function. A lot of these people don't even consider it a problem to use "report as spam" as a ways to punish companies for things that aren't even sending spam (which is something I only bring up as it demonstrates the "militant" comment).
The reality is that an open internet where people can talk to each other without going through third parties requires a world in which you, heaven forbid, receive some spam. It is actually a similar construction as the one for why we should have network neutrality: large internet carriers in the world of email should, if you really believe in open access, not be allowed to say "these people can't send mail because they are too small and not paying someone". We all need to accept at least a little spam to guarantee everyone can send mail.
If this means that you have to get better at dealing with spam, as some of the heavy-handed techniques people have put in place for dealing with spam are no longer possible, well: so be it; and if that means you need to turn off the notifications for new email on your phone to keep from getting angry when you receive some spam (which tends to come up as a reason to hate spam), I hope people realize that they caused their own reason to be upset, and that getting notifications for anything is probably unhealthy, at least for you, anyway.
Sadly, the large companies who have slowly come to "own email" have no incentive to break this cycle, whether in action or in words, as the current state of affairs is what is giving them their power: whether they build solutions that can solve spam in a way that is both fair and federated, or they just try to educate users about how to better deal with spam and decrease the stress in their lives (including telling people to stop letting email interrupt them), it will just harm them. This doesn't sound like something the private sector can solve.
> A lot of these people don't even consider it a problem to use "report as spam" as a ways to punish companies for things that aren't even sending spam
I get a fair many spams that have "unsubscribe" links, or even that look like newsletters (for companies or organizations of varying legitimacy). I'm always pleased when they come through one of the major mailing list providers that provides a separate "report abuse" link, with which I can report that no, I don't just want to unsubscribe, I want to report that I never subscribed in the first place so that the list itself gets terminated.
If it's possible for someone to be subscribed to your "newsletter" without having explicitly consented to doing so (and in the process proving ownership of the subscribing email address), you are sending spam.
> If it's possible for someone to be subscribed to your "newsletter" without having explicitly consented to doing so (and in the process proving ownership of the subscribing email address), you are sending spam.
A problem with this attitude is that most people have no idea whether or not they consented. Perhaps they consented in 2003 but don't remember. Perhaps they consented yesterday but want to revoke consent because they're too busy right now to read your newsletter. There's a reason why Gmail silently converts your spam report into an unsubscribe if the correct headers are set. Too many PEBCAK false positives.
No, the problem is figuring out whether or not to punish the sender.
If it's obviously spam, the sender's IP needs to be blacklisted, his hosting account terminated, etc. in order to protect other people.
If the recipient just changed his mind about whether he wants to receive a newsletter that he explicitly signed up for less than a week ago, there is no need to punish the sender. The sender just needs to be notified that the recipient unsubscribed.
I don't know about you, but if all the social networks, instant messengers, and "we're gonna replace email" startups in the world went dark for 24 hours, I probably wouldn't even notice. They're all but useless to me, after all. But if my email went down for 24 hours, I'd definitely make a big fuss about it. Ditto if someone makes the wrong decision about which senders to block.
If the sender is sending mail that causes significant numbers of people to flag as spam, even if those people signed up and confirmed just a week ago, they need some reminder that their behaviour is not acceptable.
Very many senders stretch the boundaries of what they send.
I disagree with your description of what is / isn't spam. Any automated message-generation system should be exceptionally open in sorting out how to detect lack of interest in continued communications. Recipients simply cannot and will not sort through anything more complex than "delete" or "this is spam".
Many major email providers do provide some form of full-loop feedback to bulk senders, and aren't unreasonable in their policies. The expectation that all email senders be perfect isn't reasonable (and generally isn't applied), but reasonable diligent effort really must be followed, and far too often isn't.
The model I've strongly endorsed for a long time is to simply apply a mail acceptance criteria that's scaled to the level of non-problematic mail originating from a sender. If most of the mail meets acceptability tests, most of it is accepted. If most of the mail doesn't meet acceptability tests, then most of it's rejected. If the problem's modest, the rejections are temporary (e.g., re-try in a bit). If it's severe, the rejections might be made permanent.
Poorly-behaved senders (those trying to re-transmit the same message before a reasonable retry interval transpires) are penalized harshly (all connection attempts are refused for some period).
RFC 2821 suggests an initial retry delay of 30 minutes, and up to 4-5 days for delivery attempts.
A typical exim4 retry configuration "specifies retries every 15 minutes for 2 hours, then increasing retry intervals, starting at 1 hour and increasing each time by a factor of 1.5, up to 16 hours, then retries every 6 hours until 4 days have passed since the first failed delivery."
Arranging with specific high-interest peers for expedited delivery, especially from individually trusted senders, might also be a useful thing.
Note that email is not guaranteed to be reliable or instantaneous. And as I've commented elsewhere on this thread, the problem now is that both you as an individual server admin have to deal with everyone else on the Net. And they've got to deal with you....
It's the sheer drudgery, more than anything else, of that, which doesn't scale. What's forgotten of the "golden age" of the Internet (1980 - 1992 or so) is that there were, comparatively, very few hosts. Dozens initially, a few tens of thousands toward the end of the period. Individual accountability largely worked. And while some individuals had direct connections, it was largely universities, a few employers, and government agencies who played the role of ISP / mail service provider. The system wasn't quite so chaotic as is commonly thought.
I recently saw a talk where someone held up a physical copy of the directory to the Internet circa 1985 or so. And it actually had everyone listed not once but 2-3 times -- by name, by organization, and by email address, or something like that. A not-very-substantial document, really.
Today's equivalent would have many millions of entries, I suspect.
Maybe it'd be useful to have a standard for mailing list confirmations that would let the mail client keep track of what you actually sign up for and/or confirm as opt-in.
As a user I also appreciate the ability to report abuse to someone I trust to deal with it.
But as someone operating various mailing lists for customers, the vast majority of the time when we get abuse reports from customers that "never subscribed", we have detailed history records showing that they did actually specifically take action to subscribe and confirm. This is consistent with pretty much every other thing - for every action in the systems, there are always customers that insists they have done no such thing.
Until we show them the audit trail.
Users are notoriously bad at remembering what they did. Which is not so odd given that most of the time the action is an inconsequential spur of the moment thing that they do or don't do as a matter of reflex and never think about again. But it makes "Report as spam" horribly abused.
Maybe the world you describe could have been possible, but so long as it is possible to send email by a bot we need bits to filter it.
And yes I Mark things as spam that may not meet the legal definition of spam, because companies tend to start sending you all sorts of crap once you sign up for their service.
In my ideal world you wouldn't have to pay a third party to send the email you would have to pay me for me to receive it.
It's never "some" spam. On my own mail domain, I get slightly more spam than wanted mail after spam filtering. If I didn't filter it would be tens or hundreds of spam items to actual mail.
The notifications question: well, in some ways people have moved to IM because it offers lower latency than email, generally better antispam, and usable notifications. Of course this is because all the IM solutions are centralized and can ban abusers more effectively.
I've only ever encountered one truly "militant" person against spam, and he refuses to accept mail from gmail because of NSA surveillance. Everyone else wants to avoid rejecting legitimate mail as much as possible while keeping email usable at all.
Lack of feasible antispam mechanisms is one of the things responsible for the decline of USENET.
I've thought about this. All the spammer has to do is 'buffer overflow' the capacity of the heuristics algorithm. If google's anti spam can only worry about X number of unique spam message signatures write a script the generates x + 1 unique human readable spam messages. Not easy but probably possible, what's a good guess for X?
I've also had similar issues in which it appears that email from me was being blocked purely because I was within a particular IP range, not because my system was actually sending any spam.
I now pay some company to deliver my personal mail, which is just dumb, and clearly isn't caused by them being better about spam (as I sent no spam): it is caused by a militarized anti-spam-crazed group of people who have managed to build a system that essentially requires an oligopoly for email to function. A lot of these people don't even consider it a problem to use "report as spam" as a ways to punish companies for things that aren't even sending spam (which is something I only bring up as it demonstrates the "militant" comment).
The reality is that an open internet where people can talk to each other without going through third parties requires a world in which you, heaven forbid, receive some spam. It is actually a similar construction as the one for why we should have network neutrality: large internet carriers in the world of email should, if you really believe in open access, not be allowed to say "these people can't send mail because they are too small and not paying someone". We all need to accept at least a little spam to guarantee everyone can send mail.
If this means that you have to get better at dealing with spam, as some of the heavy-handed techniques people have put in place for dealing with spam are no longer possible, well: so be it; and if that means you need to turn off the notifications for new email on your phone to keep from getting angry when you receive some spam (which tends to come up as a reason to hate spam), I hope people realize that they caused their own reason to be upset, and that getting notifications for anything is probably unhealthy, at least for you, anyway.
Sadly, the large companies who have slowly come to "own email" have no incentive to break this cycle, whether in action or in words, as the current state of affairs is what is giving them their power: whether they build solutions that can solve spam in a way that is both fair and federated, or they just try to educate users about how to better deal with spam and decrease the stress in their lives (including telling people to stop letting email interrupt them), it will just harm them. This doesn't sound like something the private sector can solve.