No, the problem is figuring out whether or not to punish the sender.
If it's obviously spam, the sender's IP needs to be blacklisted, his hosting account terminated, etc. in order to protect other people.
If the recipient just changed his mind about whether he wants to receive a newsletter that he explicitly signed up for less than a week ago, there is no need to punish the sender. The sender just needs to be notified that the recipient unsubscribed.
I don't know about you, but if all the social networks, instant messengers, and "we're gonna replace email" startups in the world went dark for 24 hours, I probably wouldn't even notice. They're all but useless to me, after all. But if my email went down for 24 hours, I'd definitely make a big fuss about it. Ditto if someone makes the wrong decision about which senders to block.
If the sender is sending mail that causes significant numbers of people to flag as spam, even if those people signed up and confirmed just a week ago, they need some reminder that their behaviour is not acceptable.
Very many senders stretch the boundaries of what they send.
I disagree with your description of what is / isn't spam. Any automated message-generation system should be exceptionally open in sorting out how to detect lack of interest in continued communications. Recipients simply cannot and will not sort through anything more complex than "delete" or "this is spam".
Many major email providers do provide some form of full-loop feedback to bulk senders, and aren't unreasonable in their policies. The expectation that all email senders be perfect isn't reasonable (and generally isn't applied), but reasonable diligent effort really must be followed, and far too often isn't.
The model I've strongly endorsed for a long time is to simply apply a mail acceptance criteria that's scaled to the level of non-problematic mail originating from a sender. If most of the mail meets acceptability tests, most of it is accepted. If most of the mail doesn't meet acceptability tests, then most of it's rejected. If the problem's modest, the rejections are temporary (e.g., re-try in a bit). If it's severe, the rejections might be made permanent.
Poorly-behaved senders (those trying to re-transmit the same message before a reasonable retry interval transpires) are penalized harshly (all connection attempts are refused for some period).
RFC 2821 suggests an initial retry delay of 30 minutes, and up to 4-5 days for delivery attempts.
A typical exim4 retry configuration "specifies retries every 15 minutes for 2 hours, then increasing retry intervals, starting at 1 hour and increasing each time by a factor of 1.5, up to 16 hours, then retries every 6 hours until 4 days have passed since the first failed delivery."
Arranging with specific high-interest peers for expedited delivery, especially from individually trusted senders, might also be a useful thing.
Note that email is not guaranteed to be reliable or instantaneous. And as I've commented elsewhere on this thread, the problem now is that both you as an individual server admin have to deal with everyone else on the Net. And they've got to deal with you....
It's the sheer drudgery, more than anything else, of that, which doesn't scale. What's forgotten of the "golden age" of the Internet (1980 - 1992 or so) is that there were, comparatively, very few hosts. Dozens initially, a few tens of thousands toward the end of the period. Individual accountability largely worked. And while some individuals had direct connections, it was largely universities, a few employers, and government agencies who played the role of ISP / mail service provider. The system wasn't quite so chaotic as is commonly thought.
I recently saw a talk where someone held up a physical copy of the directory to the Internet circa 1985 or so. And it actually had everyone listed not once but 2-3 times -- by name, by organization, and by email address, or something like that. A not-very-substantial document, really.
Today's equivalent would have many millions of entries, I suspect.
If anyone indicates in any way, shape, or form, whatsoever, that they don't want your crap, stop sending it to them.
Email is all but useless these days. Postal mail's hardly any better.