Question for HN: how do you handle DMARC reports? We have an email where those are just pulling up in a S3 bucket. I was hoping to find an easy solution that unzips them, aggregates, visualizes, and alerts us (if necessary).
In the meantime, I'm hoping to avoid yet another subscription service. Any suggestions?
Postmark have a free service: https://dmarc.postmarkapp.com It's simple, but works well for what it is. They added a separate commercial service fairly recently.
It definitely fails the "avoid another subscription service" criteria but I've found dmarcian works pretty well for me https://dmarcian.com/ I was considering setting something up myself but it just seemed like a lot of hassle.
Try https://www.mailhardener.com which includes (amongst many other features) DMARC and SMTP TLS report aggregation.
You can use it for free for personal (non-commercial) domains. Paid tiers (starting at 19 EUR/month) also offer additional services such as MTA-STS policy hosting and BIMI asset hosting.
Common technique is to specify address for reports that bounces. Bonus points when bounce comes from the same domain and generates another report, engaging in eternal ping-pong every 24h. /s
we use
https://easydmarc.com/pricing
and fit in the free tier. I've found that not that many MTAs actually report to the DMARC URLs when they decide to discard the email so it's not as useful as it could be.
We have started using OnDMARC recently https://redsift.com/products/ondmarc which is unfortunately a subscription service, but does add value to us through their "Dynamic DMARC" and other email health goodies, not only reporting.
I've modified SPF and set up DKIM and DMARC for my organization and 2 other organizations we merged with in the past year. They're fairly small-sized, but it was definitely a fascinating learning experience if not a bit of a pain. I ended up using Postmark for RUA reporting.
Oh and interestingly I had to set a separate transport rule in M365 in order for M365 to follow our DMARC policy for internal emails. A bit weird, but there you go.
Question for the experts, as I haven't yet seen a good explanation of this: Why are both SPF and DKIM required to prevent fraudulent emails? My semi-naive intuition is that maybe SPF alone is not sufficient, because of IP address spoofing, but DKIM alone should be sufficient if mail servers adequately protect their private keys. Does SPF just help in the cases where private keys are stolen? Maybe something about DNS cache poisoning, since that's where recipients get the DKIM public keys?
Most of confusion with email authentication historically came from two things: SMTP envelope address might not be equal to content From: address, and there was no clear policy whether to expect DKIM signature, or to which address apply SPF check.
SPF is enough to prevent spoofed source of SMTP envelope, but not for From: spoofing. (applying it to From: address would reject any forwarded mail, which probably isn't desired)
DKIM ensures legitimacy of From: address, and works correctly with forwarding. But DKIM itself doesn't specify whether you should require DKIM signature. (ADSP record allows to specify that, but it wasn't deployed nearly at all)
DMARC record is used to specify unambiguous policy, and it passes when either SPF is aligned with From: address, or there's valid DKIM signature. Thus DKIM+DMARC is enough to prevent spoofed mail, but SPF is usually kept in case some servers still don't support newer methods. SPF+DMARC would also prevent spoofing, but will also reject forwarded mail.
Note that SPF+DKIM without DMARC doesn't really protect against spoofing, because SPF checks only envelope address, and you cannot tell whether DKIM signature is required.
spf : the receiving server can check (in a public dns record) which server is allowed to send for this domain
dkim : the email is signed (using a private key) with a unique signature, the receiving server can validate the DKIM signature by running a DNS query to search for the public key for that domain .
dmarc : publish a policy on how emails that does not satisfy the above condition above (spf & dkim) should be handled (quarantine, reject) and optionally sends you a report on who send/tried to send emails on the behalf of your domain
Edit : I misread your question... it is odd indeed that only satisfying one condition is enough for a pass...
I would add that SPF alone may not be enough, and that's why some of the other mechanisms are needed. AFAIK if you have a softfail SPF (a record ending in `~`), you would need DMARC or DKIM. But the SPF is needed always in order to specify the domains/ip that are allowed. The other two mechanisms are used for making it more strong and reducing the potential phishing attacks. In particular, a softfail SPF may be fixed by using a "rejected" or "quarantine" DMARC policy.
But this is from the RFC document, so it may be that in practical cases things are more nuanced.
Then E-Mails which fail the SPF and DKIM checks should be rejected by the receiving server. Many mailservers still let them through though because they just don't check dmarc or put them in a spam folder or quarantine.
A message satisfies the DMARC checks if at least one of the supported authentication mechanisms:
1. produces a "pass" result
Personally I did not test yet what a DMARC result 1 fully failing check produces.
I can say however that only 1 successful identifier-alignment criteria (and the other credential being valid but not aligned) is enough for an overall pass.
Of course the receiving mailserver is free to enforce a strict DKIM&SPF requirement aside the DMARC check/spec. If your question is why that would be necessary, I'd love to hear the reason aswell
Based on the RFC, for the rest of possible SPF configurations (more strict), SPF record would be enough. But I am guessing based on the RFC document and my experience.
I think part of it is that the presence of either is sufficient to validate a sender, but the absence of either doesn't tell you much, because there are multiple valid reasons either could be missing from a valid sender.
Is there any progress about the displayed from address being incorrect for phishing attacks than the actual sender? For example I get a (phishing) mail from "Someone <someone@facebook.com>" whereas the message isn't from Facebook at all. And my mail client is perfectly fine with delivering it to my inbox and showing Facebook as the sender.
If I'm not tech savvy I'd easily fall for it. If I go down the rabbit hole and view all message headers I see it's fake, but why does the mail client show it as if it's from Facebook in the first place? It's a huge attack vector still present for many non-tech savvy users.
(For clarity I'm not talking about the sender name which can be anything that is free text. I'm talking about the "apparent" from field which does not reflect the actual sending address and looks perfectly legit as if from @facebook.com)
I don't understand why clients do that either, the information's there as you say, they just assume the normal/'happy path' case always occurs for some reason.
In my own (rather WIP, wouldn't really suggest anyone else use it yet) client I display both if different - which should make it clear even if the content isn't stereotypically crap and obvious spam/phishing (to a more technical user).
If you read the RFCs, it was intended for use cases like 'from Mr Blah, but sent by his secretary, reply to his secretary'. Primarily legitimately used today almost like that I suppose - with mailing list / support case management type software being the 'secretary'.
So maybe I'm half answering my own point - Gmail et al. show `From` because it's usually friendlier to a non-technical user, it's who it's 'really' from, not confusing them with 'who is this Intercom.io' or whatever. But.. surely it's worth showing both, to help in the phishing cases as you say.
Facebook is an interesting example here. If you give them your PGP indentity they will encrypt and sign the emails they send you. So you will know if the email is actually from them.
So technically this is a very solved problem. Facebook unfortunately is very much the exception. It is pretty much impossible to convince entities like your bank to sign their emails.
This is fantastic! I've always wanted to get a more in-depth understanding of how these protocols are used in the real world, and this was a great format to explore that.
Well, a real-world understanding of these protocols would include the knowledge that of the three, DMARC is the least important. Most people do not really need to know about it. If you send valuable transactional emails and you need to prevent or at least reduce the ability of third parties to impersonate you, then you might want DMARC.
Thanks. Given that it's just another TXT record, I would assume it's just a part of setting up a email domain, especially since one has to create DNS records for SPF and DKIM anyway?
That doesn't make a ton of sense, since the point of DMARC is to stop messages from being delivered.
Certainly if you have a DMARC policy of p=reject and then you screw up your outbound, then your DMARC policy becomes relevant, but not in a good way. I don't see the harm in not having one.
Email providers such as outlook and gmail think otherwise. Your emails will not be reliably delivered to them because you are missing a restrictive DMARC policy.
Emails send without a DMARC policy are less trusted by default and more likely tagged as spam.
people cannot send spoofed emails with my domain. I'm just an enthusiastic running his mail server so no big gain. But I can see why $BIG_BANK would like to prevent people to be able send fishing email from accounts@$BIG_BANK.com from random computers on the internet...
I don't have any of these on a mail server I've run for 20+ years. Mail still works reliably (both send and receive), but all the naysayers are beginning to make me think about converting to GSuite or similar.
It's not just about your own server sending and receiving mail.
If you've ever gotten loads of mailer-daemon bounces for emails that you didn't send, then a spammer has been using your address(es) in the "From" header.
One of the reasons for your success is the fact that you've been doing it for 20+ years.
Big providers like Gmail, Outlook, etc all have a scoring system for sender IP addresses and hostnames. The longer your IP and hostname behaves properly, the more credit it will get.
I imagine starting a new mail server with new ip addresses/hostnames may be more difficult.
If you're happy with the way it's working and it's not too much overhead for you to administer it, please don't take down your server. The fewer independent email servers out there, the easier it will be for the big boys to justify no putting in effort to interoperate with the ones that are left.
IP reputation is a different thing. I assume that this is why ESPs tend to drop mail from self-hosters. Low/no reputation or domestic IP often signal malicious activity, so the big ESPs act accordingly. The later reasoning comes from the fact that a vanishingly small amount of email comes from domestic IP addresses, and the stuff that does is often malicious.
NB - I work for a large messaging security company.
I'm not exactly sure what argument you are making. Yes, ESPs often drop most self-hosted email because that's easy and economical (if, perhaps, dickish). No, if you're an international ESP (like mine) there is no real 'domestic IP' you can block against.
Oh sure...no argument there. But it's a scale issue. In this space, I have, say, 500M 'questionable' IP addresses who potentially send/receive mail (known bad, residential networks, uncategorized, etc.). So let's be wildly generous and say 0.1% of them want to host an SMTP server. Does the ISP commit resources to this tiny community, or do they commit those resources to keep Google/M$/Yahoo/etc happy, who constitute 9-something% of the business.
Email is in a funny space. Spf showed up. And although it meant some political fights when people expected to send email from anywhere, it was free to run in the dollar sense. Same with dkim.
Dmarc is increasingly required to get past spam filters. And doing dmarc properly requires a reporting service. Either you run servers or use a hosted service like the one that created this page. That's fine for some business, but putting a cost on dmarc reporting and then making dmarc a heavy signal in spam filters is a huge pain for some scenarios.
It's not highly expensive, but on non monetized side projects it can frustrating how how much the internet expects you to have dmarc on every domain now.
> And doing dmarc properly requires a reporting service.
This is incorrect.
DMARC requires nothing but an email address to send reporting to. It costs nothing to implement and there are open source [0] solutions out there if you want to monitor the reporting being sent.
And to clarify - the reporting you're getting back from a DMARC is either an aggregate or a forensic report. There aren't many email providers that actually send forensic reporting anymore, the 99% of reports you'll get are aggregate in nature.
And finally, remember that if you don't have a policy of "REJECT" set on your domains, DMARC isn't doing you a whole lot of good.
Being able to firmly reject all unauthenticated messages is still the target end state. The risk of a threat actor sending an email that looks completely legit, and simply asking the user to "check their spam folder", is very real.
> This is also incorrect. Reporting (rua/ruf) are optional.
I never stated RUA/RUF were required.
My intent was to state that the parent I was replying to implied "services" or "servers" were required to collect and/or process reporting. One can simply collect reporting and process offline or simply consume manually. I do this, personally, for a few low email volume domains I own. You can't get RUA/RUF any other way.
You stated that an email address is required. It's optional. So the requirements are effectively non-existent (although who doesn't have an email address).
> You can't get RUA/RUF any other way.
There are free services that provide analysis, such as https://dmarc.postmarkapp.com/, although arguably you still need an email address to sign up. But it is another, actually useful, way to receive the reports.
Once you get DMARC setup correctly you can easily run it without any sort of reporting service.
Using DMARC is more about having a process for properly adding new senders once everything is setup than it is a long term cost sink. You can keep a reporting service around to monitor your email delivery, but you don’t have to at all.
Do you have any personal experience regarding the DMARC requirement claim? AFAIK just having SPF and DKIM without dmarc rua or ruf is still enough to not get hit by Gmail and Office 365 (as in, they’re enough to have your email reputation reliably tied to your domain instead of the email providers having to base a spam decision on the sending IP’s spam reputation).
Unfortunately I've had many related experiences. Mail works at first or on a small scale, but eventually a customer with Google workspace mail tells you email goes to junk, and Google support sends you this article in response which describes a range of requirements including dmarc. Same story for office 365.
Just a note that SPF/DKIM and DMARC are not "get out of jail free card".
Google support is just brushing people off with the link and they are somewhat right, because a lot don't have anything set. There is also a bit of black magic with spam filters that Google support will never tell you about because it will be just like SEO, someone will use it to pass filters.
That said, there are multiple other reasons that your email can be qualified as spam by the spam filter. Like sending out email from one domain and having links to another domain in the body as an example. Not having "unsubscribe" links on mass mail. There are legitimate reasons to throw email to spam because some companies get their infra or "contact us" forms hacked.
In the end users can be annoying because they might not understand what "this is spam" button means for the sender - they just think your mail was annoying and they did not expect it and press that and you are qualified as a spam.
In my experience sadly few people really check its presence. Sure, there might be an aggressive spam filter in place that collects these mails. But this protection will almost always be removed once a customer or supplier couldn't contact you.
DKIM isn't hard to manage, SPF can be a pita. Although IP blocks are a very sensible network management tool (cloud provider and customers hate it), it is more difficult to administrate than setting up some domain keys. It will have to be managed whenever anything that is sending mail changes its public IP address, which very often happens if you host it in the "cloud".
Do you have a reporting service you like to use for yourself/clients?
I've never tried, but but wondered if it would it be enough to simply use a inbox@whatever inbox you have setup, most likely gmail on your domain (don't know what google calls that now workplace idk)?
In cases where it's super unlikely someone would try to send as the domain so it's just setting it up to check a box for deliverability.
You don't, but I've tried what you suggested and it fills with thousands of xml files attachments you have no hope of reading. You could just ignore them, but that gets down to what I mean by "doing it properly".
yikes. for me/clients it's just marketing so deliverability is more important than actually reading all of the reports. i guess some could potentially have an 'attack' where someone tries to spoof so I'm going to do some more research see if I can find a better solution.
best tester I found so far... very detailed, explains interactively what it is doing and gives you the RFC number... was in the process of migrating my mail server, and it reminded me about my spf record :)
This is content marketing done right. You taught me something first in exchange for visiting your service, should I ever need it. I'm fine with such form of advertising.
Great idea, there are lots of others that are showing you if SPF/DMARC/DKIM work correctly but this one is also great for people trying to set up mail server and are confused what is actually going on.
Nice work.
It would be very nice if more header information would be exposed so you exactly see what is taken into account.
Has anybody else been getting spam from "security researchers" and "ethical hackers" who tell you that you are missing a DMARC record and then ask for a bounty?
Yes. I was the lead security engineer at my previous job, and these messages were always marked as “Informative” and, eventually, “Duplicate”.
The major bug bounty websites like Bugcrowd and HackerOne have rules about duplicate and non-actionable submissions, which usually hurt the researcher, for example, by decreasing their reputation points. After the first couple of submissions, I got curious, so I decided to open a separate category for these submissions where I asked every researcher to provide a proof of concept of the attack. Unsurprisingly, none of them were able to. Some of them closed their own submissions, and the others lost some reputation points due to Bugcrowd or HackerOne’s intervention.
Somehow, we did not receive more submissions about DMARC, SPF, or DKIM after that, which was interesting, as if all the researchers were working together to spam as many bug bounty programs as possible with the same “vulnerability” to get easy money from unsuspected program managers.
No specific action is taken by DMARC regarding the delivery of the message. This usually means the message will be delivered successfully. Keep in mind that other mechanisms such as a spam filter can still reject or quarantine a message."
make note that it took me no more than an hour at most in my first attempt to set up the email server including learning about dmarc, spf and where to put that into my email server as well as in the domain registrar.
i have had no problems for the past almost a year and i am confident there shouldn't be many problems in future.
Fastmail tells you very explicitly what to do and how to do that, I recommend doing that, at one day you may be hit with a very important email going undelivered.
I'm using the recommended Fastmail setup with my own name server, which doesn't support DMARC out of the box. Is this something I should look into to improve deliverability?
I haven't bothered with anything beyond the recommended SPF and DKIM records, and am not aware of any issues. IMO it will be years before not having DMARC is a serious issue for delivery, and I expect Fastmail to tell me when that time is getting close and/or take care of it for me.
It was previously possible for an attacker, especially a selective one to impersonate the domain of a major provider like Paypal or Dropbox in the display address. I think the fact this is no longer possible is a good thing. Phishing has got a lot more aggressive, and obviously if these measures were not being deployed the situation would simply be a lot worse.
If you don't mind using jQuery Based Terminal[1] library. You can do almost everything (except I don't know how to create floating domain animation). You can think about jQuery Terminal as a framework. You don't need ReactJS or Angular for this, unless you need this as part of bigger application.
The library supports typing animation and changing colors. You can easily create the same interactive console.
If you're interested I can create a demo for this.
In the meantime, I'm hoping to avoid yet another subscription service. Any suggestions?