Negotiating is difficult if you show your hand. It is arguably beneficial to both the state and Elon that the e-mails stay redacted. I agree it is unfortunate though.
Sensitive negotiations like those should be handled by low-level bureaucrats with supervision, not by the damned Governor. Again, negotiating directly with a state governor is obviously corrupt no matter what the contents. This would have been instantly clear to any American back when we were an advanced nation.
Could you elaborate why an executive having sensitive conversations with a governor is corrupt? I'm having difficulty understanding this outlook.
My industry has historically been extremely corrupt. This is why rules were defined for reportable expenses. Could you boil down your viewpoint into rules that can be applied in an organization, rather than a vibe test? I think I could better understand you, then.
If the result of the negotiations are in good faith and beneficial for his constituents, then it is not corruption. Of course, that's open to interpretation.
Corruption is usually when there is personal benefit to the politician themselves.
Use-after-free bugs (such as the vulnerability in question, https://issuetracker.google.com/issues/440183164) usually can be exploited to result in remote code execution, but not always. It wouldn't be prudent to bet that this case is one of the exceptions, of course.
That behaviour is indeed totally unacceptable. At your job. Where they're paying you, and especially if they're paying you at FAANG type pay scales.
If you're an unpaid volunteer? Yeah - nah. They can tell you "Sorry, I'm playing with my cat for the next 3 months, maybe I'll get to it after that?", or just "Fuck off, I don't care."
(I'm now playing out a revenge fantasy in my head where the ffmpeg team does nothing, and Facebook or Palantir or someone similar get _deeply_ hacked via the exploit Google published and theat starts the planets biggest ever pointless lawyers-chasing-the-deepest-pockets fight.)
Or perhaps you’re a FAANG security researcher and your time will be better spent serving the OSS community as a whole by submitting as many useful bug reports as possible, instead of slightly fewer reports with patches included.
In this particular case it’s hardly obvious which patch you should submit. You could fix this particular bug (and leave in place the horrible clunky codec that nobody ever uses) OR you could just submit a patch that puts it behind a compile flag. This is really a decision for the maintainers, and submitting the latter (much better!) patch would not save the maintainers any meaningful amount of time anyway.
I don’t understand how it helps the community to publicly release instructions for attacking people, unless you’re trying to incentivize a company to fix their crap. In this case, there is no company to incentivize, so just report it privately.
You can say publicly that “there is an ABC class vulnerability in XYZ component” so that users are aware of the risk.
It’s OSS so somebody who cares will fix it, and if nobody cares then it doesn’t really matter.
This also informs users that it’s not safe to use ffmpeg or software derived from it to open untrusted files, and perhaps most importantly releasing this tells the distro package maintainers to disable the particular codec when packaging.
This bug might lead to vulnerability and that's enough. It makes no sense to waste lot of time and research whether it is possible or not - it is faster to remove the buggy codec nobody needs or make a fix.
1) dedicating compute resources to continuously fuzzing the entire project
2) dedicating engineering resources to validating the results and creating accurate and well-informed bug reports (in this case, a seriously underestimated security issue)
3) additionally for codecs that Google likely does not even internally use or compile, purely for the greater good of FFMPEG's user base
Needless to say, while I agree Google has a penny to spare to fund FFMPEG, and should (although they already contribute), I do not agree with funding this maintainer.
> - benefiting from the bugs getting fixed, but not contributing to them.
I would be very surprised if Google builds this codec when they build ffmpeg. If you run a/v codecs (like ffmpeg) in bulk, the first thing to do is sandbox the hell out of it. The second thing you do is strictly limit the containers and codecs you'll decode. Not very many people need to decode movies from old LucasArts games, for video codecs, you probably only want mpeg 1-4, h.26x, vp8, vp9, av1. And you'll want to have fuzzed those decoders as best you can too.
Nobody should be surprised that there's a security problem in this ancient decoder. Many of the eclectic codecs were written to mimic how the decoders that shipped with content were written, and most of those codecs were written assuming they were decoding a known good file, because why wouldn't they be. There's no shame, that's just how it is... there's too much to proactively investigate, so someone doing fuzzing and writing excellent reports that include diagnosis, specific location of the errors, and a way to reproduce are providing a valuable contribution.
Could they contribute more? Sure. But even if they don't, they've contributed something of value. If the maintainers can't or don't want to address it, that'd be reasonable too.
"While reading about the 4xm demuxer vulnerability, we thought that we could help FFmpeg eliminate many potential low-hanging problems from the code by making use of the Google fleet and fuzzing infrastructure we already had in place"
"At no cost to Google" seems difficult to substantiate, given that multiple sources indicate that Google is sponsoring FFmpeg both with engineering resources (for codec development) and cold hard cash (delivered to the FFmpeg core team via their consulting outfit[1]).
This is excellent, to be clear. But it's not compatible with the yarn currently being spun of a purely extractive relationship.
Yes, according to the license selected by ffmpeg. And google, according to this license selected by ffmpeg, paid them nothing. And then do some additional work, beneficial to ffmpeg.
They could, but there is really no requirement on them to do so. The security flaw was discovered by Google, but it was not created by them.
Equally there is no requirement on ffmpeg to fix these CVEs nor any other.
And, of course, there is no requirement on end-users to run software from projects which do not consider untrusted-input-validation bugs to be high priority.
> They could, but there is really no requirement on them to do so.
I see this sort of sentiment daily. The sentiment that only what is strictly legal or required is what matters.
Sometimes, you know, you have to recognise that there are social norms and being a good person matters and has intrinsic value. A society only governed by what the written law of the land explicitly states is a dystopia worse than hell.
What's "strictly legal or required" of Google here is absolutely nothing. They didn't have to do any auditing or bug hunting. They certainly didn't have to validate or create a proper bug report, and there's no requirement whatsoever that they tell anyone about it at all. They could have found the bug, found it was being actively exploited, made their own internal patch and sat quietly by while other people remained vulnerable. All of that is well within what is "strictly legal or required".
Google did more than what is "strictly legal or required", and what they did was submit a good and valid bug report. But for some reason we're mad because they didn't do even more. Why? The world is objectively a better place for having this bug report, at least now people know there's something to address.
Google did more than what is "strictly legal or required", and what they did was submit a good and valid bug report. But for some reason we're mad because they didn't do even more. Why?
"I noticed your window was broken, so I took the liberty of helping you, working for free, by posting a sign that says UNLOCKED WINDOW HERE with exact details on how it was broken. I did lots of gratis work for you which you do not need to do yourself now. The world is safer now. Why are you not grateful?"
I mean if we’re going to do sloppy analogies, a bug report for open source software as widely used as ffmpeg is more like “I noticed the trees in the back corner of your free apple orchard are producing apples with trace amounts of heavy metals. I did some literal digging and sent some soil off to the labs and it looks like your back corner field may be contaminated. Here’s a heads up about that, and also just FYI in 90 days, if you haven’t been able to get your soil remediated, I’m going to put up a sign so that people can know to avoid those apples and not get poisoned by your free orchard while it’s getting fixed.”
Yes, this is a good illustration why The Copenhagen Interpretation of Ethics makes sense when Ffmpeg is allowed to criticise the manner of actions of Google.
I think it’s a good example of why it makes no sense. People are saying they would rather a world where people unknowingly get heavy metal poisoning from apples at the free orchard rather than a world where people are informed that some of the apples in one part of the orchard may have heavy metals in it because the person letting people know about it didn’t also remediate the soil, even though they don’t own the orchard.
That is plainly ridiculous. An orchard without heavy metals is obviously an ideal world in this case, but a world where people are at least informed of the places where the heavy metals are is orders of magnitude better than one where they’re unknowing getting heavy metal poisoning.
You're correct, but it's the social norms -- or at least, the norms as I perceive them -- that I am talking about here.
If you find yourself with potentially serious security bugs in your repo, then the social norm should be for you to take ownership of that because, well, it's your repo.
The socially unacceptable activity here should be treating security issues as an irritation, or a problem outside your control. If you're a maintainer, and you find yourself overwhelmed by genuine CVE reports, then it might be worth reflecting on the root cause of that. What ffmpeg did here was to shoot the messenger, which is non-normative.
It seems to me that they are not treating the security issue as an irritation, but instead the manner at which it was presented to them that is the problem.
> And, of course, there is no requirement on end-users to run software from projects which do not consider untrusted-input-validation bugs to be high priority.
What's this even saying?
Then they're free to fork it and never use the upstream again.
It is my understanding that the commenters in FFMPEG's favor believe that Google is doing a disservice by finding these security vulnerabilities, as they require volunteer burden to patch, and that they should either:
1) allow the vulnerabilities to remain undiscovered & unpatched zero-days (stop submitting "slop" CVEs.)
2) supply the patches (which i'm sure the goalpost will move to the maintainers being upset that they have to merge them.)
3) fund the project (including the maintainers who clearly misunderstand the severity of the vulnerabilities and describe them as "slop") (no thank you.)
Yep, that's clearly what I was saying. I want to just keep moving the goalposts (which I didn't even know I had set or moved in the first place) again and again.
Or I just want the $3.5 trillion company to also provide the patches to OSS libraries/programs/etc that their projects with hundreds of millions or billions in funding happen to find.
What is the mission of Project Zero? Is it to build a vulnerability database, or is it to fix vulnerabilities?
If it's to fix vulnerabilities, it seems within reason to expect a patch. If the reason Google isn't sending a patch is because they truly think the maintainers can fix it better, then that seems fair. But if Google isn't sending a patch because fixing vulns "doesn't scale" then that's some pretty weak sauce.
Maybe part of the solution is creating a separate low priority queue for bug reports from groups that could fix it but chose not to.
> After finding a number of flaws in software used by many end-users while researching other problems, such as the critical "Heartbleed" vulnerability, Google decided to form a full-time team dedicated to finding such vulnerabilities, not only in Google software but any software used by its users.
It did that but it did not decide to form a team dedicated to fixing issues in software that it uses? That's the misallocation of funds that's at play here.
The ideal outcome is that Project Zero sends its discoveries off to a team who triage and develop patches for the significant vulnerabilities, and then the communication with the project is a much more helpful one.
The security and privacy org is much large than just GPZ, but the security and privacy org does not have a general remit to fix all vulns everywhere. GPZ is also not the only part of the org that finds bugs in open source software but is not generally obligated to fix them. Projects like ossfuzz are similar.
Google could staff a team that is responsible for remediating vulns in open source software that doesn't actually affect any of Google's products. Lord knows they've got enough money. I'd prefer it if they did that. But I don't really see the reasoning behind why they must do this or scrap all vuln research on open source software.
> Our mission is to make the discovery and exploitation of security vulnerabilities more difficult, and to significantly improve the safety and security of the Internet for everyone.
> We perform vulnerability research on popular software like mobile operating systems, web browsers, and open source libraries. We use the results from this research to patch serious security vulnerabilities, to improve our understanding of how exploit-based attacks work, and to drive long-term structural improvements to security.
If you are deliberately shipping insecure software, you should stop doing that. In ffmpeg's case, that means either patching the bug, or disabling the codec. They refused to do the latter because they were proud of being able to support an obscure codec. That puts the onus on them to fix the bug in it.
I can tell you with 100% certainty that there are undiscovered vulnerabilities in the Linux kernel right now. Does that mean they should stop shipping?
I do think that contributing fuzzing and quality bug reports can be beneficial to a project, but it's just human nature that when someone says "you go ahead and do the work, I'll stand here and criticize", people get angry.
Rather than going off and digging up ten time bombs which all start counting down together, how about digging up one and defusing it? Or even just contributing a bit of funding towards the team of people working for free to defuse them?
If Google really wants to improve the software quality of the open source ecosystem, the best thing they could do is solve the funding problem. Not a lot of people set out to intentionally write insecure code. The only case that immediately comes to mind is the xz backdoor attempt, which again had a root cause of too few maintainers. I think figuring out a way to get constructive resources to these projects would be a much more impressive way to contribute.
This is a company that takes a lot of pride in being the absolute best of the best. Maybe what they're doing can be justified in some way, but I see why maintainers are feeling bullied. Is Google really being excellent here?
You will note the Linux kernel is not crying on Twitter when Google submits bugs to them. They did long ago, then realized that the bugs that Google reported often showed up exploited in the wild when they didn’t fix them, and mostly decided that the continuous fuzzing was actually a good thing. This is despite not all the bugs being fixed on time (there are always new OSSFuzz bugs in the queue for fixing).
There are other CVE numbering authorities you can report a vulnerability to and apply for a CVE, or appeal, but this does possibly have a chilling effect if the vendor's CNA refuses valid vulns. (Like with MS in https://news.ycombinator.com/item?id=44957454 )
> this does possibly have a chilling effect if the vendor's CNA refuses valid vulns
The Linux kernel went in the opposite direction: Every bugfix that looks like it could be relevant to security gets a CVE[1]. The number of CVEs has increased significantly since it became a CNA.
>If Google really wants to improve the software quality of the open source ecosystem, the best thing they could do is solve the funding problem.
Google is not a monolith. If you asked the board, or the shareholders of google what they thought of open source software quality they would say they don't give a rat's ass about it. Someone within google who does care has been given very limited resources to deal with the problem, and are approaching it in the most efficient way they can.
>it's just human nature that when someone says "you go ahead and do the work, I'll stand here and criticize", people get angry
Bug reports are not criticism, they are in fact contributions, and the human thing to do when someone contributes to your project is to thank them.
>This is a company that takes a lot of pride in being the absolute best of the best.
There was an era when people actually believed that google was the best of the best, rather than saying it as a rhetorical trick, and during that era they never would have dreamed of making such self centered demands of google. This project zero business comes across as the last vestige of a dying culture within google. Why do people feel the need to be so antagonitic towards it?
>I can tell you with 100% certainty that there are undiscovered vulnerabilities in the Linux kernel right now. Does that mean they should stop shipping?
The ffmpeg authors aren't "shipping" anything; they're giving away something they make as a hobby with an explicit disclaimer of any kind of fitness for purpose. If someone needs something else, they can pay an engineer to make it for them.
This has nothing to do with payment. Not deliberately infecting your users with vulnerabilities is simply the right thing to do. Giving something away for free doesn't absolve you of certain basic ethical responsibilities.
They're not deliberately infecting users with anything. There effectively saying "here's example code showing how to deal with these video formats. NOTE THAT THESE ARE EXAMPLES THAT I WROTE FOR FUN. THEY ARE NOT MEANT FOR SERIOUS USE AND MAY NOT HANDLE ALL CORNER CASES SAFELY. THIS SHOULD BE OBVIOUS SINCE WE HAVE NO COMMERCIAL RELATIONSHIP AND YOU'RE DOWNLOADING RANDOM CODE FROM SOMEONE YOU DON'T KNOW ON THE INTERNET".
If someone goes on to use that code for serious purposes, that's on them. They were explicitly warned that this is not production commercial code. It's weekend hobby work. There's no ethical obligation to make your hobby code suitable for production use before you share it. People are allowed to write and share programs for fun.
Deliberate malware would be something like an inbuilt trojan that exfiltrates data (e.g. many commercial applications). Completely different.
They are not effectively saying that. The way they talk about the library everywhere else makes it clear that they do expect serious use. Disclaimers in the license don't override that, especially when 99% of software has a disclaimer like that. Those words are there for legal reasons only.
If they wanted to market ffmpeg as a toy project only, not to be trusted, they could do that, but they are not doing that.
Except the very idea that they owe you anything is so absurd that even if they had a contract document stating that they'd do work for you, they still wouldn't have an obligation to do so because society has decided that contracts without some consideration from both sides are not valid. Similarly, even if something you buy comes with a piece of paper saying they don't owe you anything if it breaks, the law generally says that's not true. Because you paid for it.
But they don't say they warrant their work. They have a notice reminding you that you are receiving something for free, and that thing comes with no support, and is not meant to be fit for any particular use you might be thinking of, and that if you want support/help fulfilling some purpose, you can pay someone (maybe even them if you'd like) for that service. Because the way the world works is that as a general principle, other people don't owe you something for nothing. This is not just some legal mumbo jumbo. This is how life works for everyone. It's clear that they're not being malicious (they're not distributing a virus or something), and that's the most you can expect from them.
Computer security is always contextual, but as a general rule, if you're going to be accepting random input from unknown parties, you should have an expert that knows how to do that safely. And as mentioned elsewhere in these comments, such an expert would already be compiling out codecs they don't need and running the processing in a sandboxed environment to mitigate any issues. These days even software written in-house is run in sandboxed environments with minimal permissions when competent professionals are making things. That's just standard practice.
So they should be proud that they support obscure codecs, and by default the onus is on no one to ensure it's free from bugs. If an engineer needs to make a processing pipeline, the onus is always on them to do that correctly. If they want to use a free, unsupported hobby tool as part of their serious engineering project, it's on them to know how to manage any risks involved with that decision. Making good decisions here is literally their job.
All I'm asking for right here is consistency about whether the library is mostly secure. The ethical requirement is to follow through on your claims and implications, while making claims and implications is completely optional.
> Computer security is always contextual, but as a general rule, if you're going to be accepting random input from unknown parties, you should have an expert that knows how to do that safely. And as mentioned elsewhere in these comments, such an expert would already be compiling out codecs they don't need and running the processing in a sandboxed environment to mitigate any issues.
Sandboxing is great defense in depth but most software should not require sandboxing. And expecting everyone to have an expert tweaking compilation is not realistic. Defaults matter, and security expectations need to be established between the site, the documentation, and the defaults, not left as a footgun for only experts to avoid.
The library probably is mostly secure, and it might even be the best library out there for what it does. That still leaves them with no ethical requirement at all.
People are allowed to make secure, robust software for fun. They can take pride in how good of a job they do at that. They can correctly point out that their software is the best. That still leaves them with no obligations at all for having shared their project for free.
If you are not an expert in hardening computers, don't run random untrusted inputs through it, or pay someone to deliver a turnkey hardened system to you. That someone might be Adobe selling their codecs/processing tools, or it might be an individual or a company like Redhat that just customizes ffmpeg for you. In any case, if you're not paying someone, you should be grateful for whatever goodwill you get, and if you don't like it, you can immediately get a full refund. You don't even have to ask.
The person doing serious things in a professional context is always the one with the obligation to do them correctly. When I was at IBM, we used exactly 1 external library (for very early processor initialization) and 1 firmware blob in the product I worked on, and they were paid deliverables from hardware vendors. We also paid for our compiler. Everything else (kernel, drivers, firmware, tools) was in-house. If companies want to use random free code they found on the Internet without any kind of contract in place, that's up to them.
It is if they fix bugs like this. Status quo everything is fine with their actions, they don't need to do anything they aren't already doing.
If they decide they don't want to fix bugs like this, I would say they have the ethical obligation to make it clear that the software is no longer mostly secure. This is quite easy to accomplish. It's not a significant burden in any way.
Basically, if they want to go the less-secure route, I want it to be true that they're "effectively saying" that all caps text you wrote earlier. That's all. A two minute edit to their front page would be enough. They could edit the text that currently says "A complete, cross-platform solution to record, convert and stream audio and video." I'll even personally commit $10 to pay for those two minutes of labor, if they decide to go that route.
> Providing a real CVE is a contribution, not a burden. The ffmpeg folks can ignore it, since by all indications it's pretty minor.
Re-read the article. There's CVEs and then there's CVEs. This is the former, and they're shoving tons of those down the throats of unpaid volunteers while contributing nothing back.
What Google's effectively doing is like a food safety inspection company going to the local food bank to get the food that they operate their corporate cafeteria on just to save a buck, then calling the health department on a monthly basis to report any and all health violations they think they might have seen, while contributing nothing of help back to the food bank.
I have read the article. The expectation for a tool like ffmpeg is that regardless of what kind of file you put into it, it safely handles it.
This is an actual bug in submitted code. It doesn't matter that it's for some obscure codec, it's technically maintained by the ffmpeg project and is fair game for vulnerability reports.
Given that Google is also a major contributor to open-source video, this is more like a food manufacturer making sure that grocery stores are following health code when they stock their food.
Mind you, the grocery store has no obligation to listen to them in this metaphor and is free to just let the report/CVE sit for a while.
It’s a reproducible use-after-free in a codec that ships by default with most desktop and server distributions. It can be leveraged in an exploit chain to compromise a system.
I'm not a Google fan, but if the maintainers are unable to understand that, I welcome a fork.
It’s a reproducible use-after-free in a codec that ships by default with most desktop and server distributions.
The recent iOS zero-day (CVE-2025-43300) targeted the rarely used DNG image format. How long before this FFMPEG vulnerability is exploited to compromise legacy devices in the wild, I wonder?
I’m not a fan of this grandstanding for arguably questionable funding. (I surely would not fund those who believe these issues are slop.) I’d like to think most contributors already understand the severity and genuinely care about keeping FFMPEG secure.
Bugs in little-used corners of the project are a massive red flag, that's how some of the most serious OpenSSL bugs have emerged. If the code is in there, and someone can trigger it with a crafted input, then it is as bad as any other bug.
1. Yes, they are technically supported but sourcing material is hard.
2. It does not currently, but I’ll consider adding support for that! Although I feel that it might not work well on macOS Tahoe, which ships with a new ”faded blur” menu bar background.
I would say it is more a testament to the unfortunate state of cybersecurity. These "theoretical" attacks happen. Everyone just thinks it won't be them.
My rebuttal is that the DNSSEC root keys could hit Pastebin tonight and in almost every organization in the world nobody would need to be paged. That's not hyperbole.
You are mostly right, but I would hope that certain core security companies and organizations would get paged. Root CAs and domain registrars and such should have DNSSEC validation.
Unfortunately, DNSSEC is a bit expensive in terms of support burden, additional bugs, reduced performance, etc. It will take someone like Apple turning DNSSEC validation on by default to shake out all the problems. Or it will take an exploitable vulnerability akin to SIM-swapping to maybe convince Let's Encrypt! and similar services reliant on proof-by-dns that they must require DNSSEC signing.
SIM-swapping is a much more important attack vector than on-path/off-path traffic interception, and are closer to how DNS hijacking happens in practice (by account takeover at registrars).
It does in fact make sense to address the most important attacks before the exotic ones, especially when addressing the exotic attacks involves drastically more work than the common ones. I think you're making my case for me.
If that happened, we'd revert to pre-DNSSEC security levels: an attack would still be hard to pull off (unless you own a root DNS server or are reliably in the path to one). It's like knowing the private key for news.ycombinator.com - it still doesn't do anything unless I can impersonate the Hacker News server. But that was still enough of a risk to justify TLS on the web. Mostly because ISPs were doing it to inject ads.
It is important. This is unfortunate rhetoric that is harming the safety of the internet.
"For instance, in April 2018, a Russian provider announced a number of IP prefixes (groups of IP addresses) that actually belong to Route53 Amazon DNS servers."
By BGP hijacking Route53, attackers were not only able to redirect a website to different IPs, globally, but also generate SSL certificates for that website. They used this to steal $152,000 in cryptocurrency. (I know I know, "crypto", but this can happen to any site: banking, medical, infrastructure)
Also, before you say, RPKI doesn't solve this either, although a step in the right direction. DNSSEC is a step in the right direction as well.
The idea is important. What it aims to protect is important. The current implementation is horrible, far too complex and fraught with so many landminds that no one wants to touch it.
A common reason, if not the vast majority of cases, is that people mix up which key they publish and which key they are actually using. I don't doubt there are a lot of things they could do to improve the protocol, but this very common problem is fairly difficult to solve on a protocol level.
I remember back in the days when people discouraged people from using encrypted disks because of the situation that could happen if the user lost their passwords. No disk encryption algorithm can solve the issue if the user does not have the correct password, and so the recommendation was to not use it. Nowadays people usually have TPMs or key management software to manage keys, so people can forget the password and still access their encrypted disks.
DNSSEC software is still not really that developed that they automatically include basic tests and verification tools to make sure people don't simply mix up keys. They assume that people write those themselves. Too many times this happens after incidents rather than before (heard this in so many war stories). It also doesn't help that dns is full of caching and caching invalidation. A lot of the insane step-by-step plans comes from working around TTL's, lack of verification, basic tooling, and that much of the work is done manually.
> No disk encryption algorithm can solve the issue if the user does not have the correct password, and so the recommendation was to not use it.
This problem is accurate, but it's the framing that makes it wrong.
No disk encryption algorithm can simultaneously protect and not protect something encrypted, what you're missing is the protocol/practices around that, and those are far less limited.
There is heaps of encryption around these days, there are people losing access to their regular keys, and yet procedures that recover access to their data while not entirely removing the utility of having encrypted data/disks.
A TPM is absolutely not a reliable way to store your key. Think about how often you get asked for a BitLocker recovery code, and imagine if every time that happened, you lost all your data.
What parts do you agree about? Someone making an argument that we should return to the drawing board and come up with a new protocol, one that doesn't make the "offline signers and authenticated denial" tradeoffs DNSSEC makes, would probably be saying something everybody here agrees with --- though I still don't think it would be one of the 5 most important security things to work on.
But the person you're replying to believes we should hasten deployment of DNSSEC, the protocol we have now.
I would love to go to back to the drawing board and solve the security pitfalls in BGP & DNS. I wish the organizations and committees involved did a better job back then.
Sadly, we live in this reality for now, so we do what we can with what we have. We have DNSSEC.
You understand that it is a little difficult for people to take seriously a claim that you're interested in going back to the drawing board while at the same time very stridently arguing that hundreds of millions of dollars of work should go in to getting a 1994 protocol design from 4% deployment to 40% deployment. The time to return to the drawing board is now.
I don't read that reply as them saying we should hasten deployment of DNSSEC. If that was the intention of the comment then no, I don't agree with that aspect of it.
I saying say I agree with the statement "I am saying it is dishonest to discount the real security threat of not having DNSSEC."
I believe we do need some way to secure/harden DNS against attacks, we can't pretend that DNS as it stands is OK. DNSSEC is trying to solve a real problem - I do think we need to go back to the drawing board on how we solve it though.
They definitely believe we should hasten deployment of DNSSEC --- read across the thread. For instance: Slack was taken down for a half a day owing to a deployment of DNSSEC that a government contract obligated them to undertake, and that commenter celebrated the contract.
It's fine that we all agree on some things and disagree on others! I don't think DNS security is a priority issue, but I'm fine with it conceptually. My opposition is to the DNSSEC protocol itself, which is a dangerous relic of premodern cryptography designed at a government-funded lab in the 1990s. The other commenter on this thread disagrees with that assessment.
slightly later
(My point here is just clarity about what we do and don't agree about. "Resolving" this conflict is pointless --- we're not making the calls, the market is. But from an intellectual perspective, understanding our distinctive positions on Internet security, even if that means recognizing intractable disputes, is more useful than just pretending we agree.)
The eventual plan is to limit certs to 48 hours (AFAIR), right now they're already allowing 6-day certs: https://letsencrypt.org/2025/02/20/first-short-lived-cert-is... In this scenario, if Let's Encrypt goes down for just a couple of days, a lot of certs will expire.
There are also operational risks, as Let's Encrypt has to have their secret key material in close proximity to web-facing services. Of course, they use HSMs, but it might not be enough of a barrier for nation-state level attackers.
The offline signing feature of DNSSEC allows the root zone and, possibly, the TLDs to be signed fully offline.
That's why in my ideal world I want to keep DNSSEC as-is for the root zone and the TLD delegation records, but use something like DoH/DoT for the second-level domains. The privacy impact of TLD resolution is pretty much none, and everything else can be protected fully.
That is not why DNSSEC has offline signers. DNSSEC has offline signers because when the protocol was designed, its authors didn't believe computers would be able to keep up with the signing work. Starting sometime in the middle of the oughts, people started to retcon security rationales onto it, but that's not the purpose of the design.
Do you have links for that? I don't really doubt that, since the work was done mid 90-s. But I'm genuinely curious about the early history of failed protocols (like IPv6 and DNSSEC), and I read most of the early archived discussions about IPv6.
Yes, somewhere I do; I wrote a complete history of the protocol, including archives I found of 90s-vintage mailing lists. I'll have to dig it up, though.
The proposal is to make LE certs 9 days long or something. Which means if LE is down for even a short time thousands and millions of certs will expire.
You don't wait till the last second to renew. 9 day certificates would mean 7 day renewals for example. And at that point you could have 2 or 3 ACME-compatible services configured as a backup.
That's ok, this scales with requirements / experience / money on the line. Most people won't care about a day of downtime that much. And those who really don't know anything about SSL will be using a platform solving this for them.
There are two things mixed up. "We need secure DNS" != "we need DNSSEC".
There is a huge demand for securing DNS-related things, but DNSSEC seems to be a poor answer. DoH is a somehow better answer, with any shortcomings it may have, and it's widely deployed.
I suspect that a contraption that would wrap the existing DNS protocol into TLS in a way that would be trivial to put in front of an existing DNS server and an existing DNS client (like TLS was trivial to put in front of an HTTP server), might be a runaway success. A solution that wins is a solution which is damn easy to deploy, and not easy to screw up. DNSSEC is not it, alas.
Yes. But DoH was built in a way which is reasonably easy to adopt, and offers obvious benefits, hence it was adopted. DNSSEC lacks this quality, and I think this quality is essential.
TLS internally does not depend on a domain in the DNS sense, it basically certifies a chain of signatures bound to a name. That chain can be verified, starting from the root servers.
The problem is more in the fact that TLS assumes creation of a long-living connection with an ephemeral key pair, while DNS is usually a one-shot interaction.
Encrypting DNS would require caching of such key pairs for some time, and refreshing them regularly but not too often. Same for querying and verifying certificates.
I'm sorry, this is just such an incredibly fine-tuned threat model for me to take it seriously.
You start with a BGP hijack, which lets you impersonate anybody, but assume that the hijacker is only so powerful as being able to impersonate a specific DNS server and not the server that the DNS server tells you about. You then use that specific control to get a CA to forge a certificate for you (and if the CA is capable of using any information to detect that this might be a forgery, the attack breaks).
And of course, the proposed solution doesn't do anything to protect against other kinds of DNS hijacking--impersonating somebody to the nameserver and getting the account switched over to them.
> I'm sorry, this is just such an incredibly fine-tuned threat model for me to take it seriously.
You claim it is fine-tuned, but it has happened in the real world. It is actually even better for attackers that it is "obscure", because that means it is harder to detect.
> but assume that the hijacker is only so powerful as being able to impersonate a specific DNS server and not the server that the DNS server tells you about.
Yes, all layers of the stack need to be secure. I am not making assumptions about the other layers - this thread is about DNS.
> if the CA is capable of using any information to detect that this might be a forgery
They are not. The only mitigation is "multi-perspective validation", which only addresses a subset of this attack.
> And of course, the proposed solution doesn't do anything to protect against other kinds of DNS hijacking
Yes, because other kinds of DNS hijacking are solved by HTTPS TLS. If TLS and CAs are broken, nothing is secure.
> You claim it is fine-tuned, but it has happened in the real world.
Sure, but it seems like his comment is still responsive; if DNSSEC is deployed, they perform a BGP hijack & can impersonate everyone, and they just impersonate the server after the DNS step?
If that's the threat model you want to mitigate, it seems like DNSSEC won't address it.
> and they just impersonate the server after the DNS step?
Yes, there are different mitigations to prevent BGP hijacking the webserver itself. Preventing a rogue TLS certificate from being issued is the most important factor. CAA DNS records can help a bit with this. DNS itself however is easiest solved by DNSSEC.
There are a lot of mitigations to prevent BGP hijacks that I won't get too much into. None are 100%, but they are good enough to ensure multi-perspective validation refuses to issue a TLS certificate. The problem is that if those same mitigations are not deployed on your DNS servers (or you outsource DNS and they have not deployed these mitigations) it is a weak link.
I don't see you responding to the question. You're fixating on protections for DNS servers, because that is the only circumstance in which DNSSEC could matter for these threat actors, not because they can't target the address space of the TLS servers themselves (they can), but because if you concede that they can do this, DNSSEC doesn't do anything anymore; attackers will just leave DNS records intact, and intercept the "authentic" server IPs.
So far your response to this has been "attackers can't do this to Cloudflare". I mean, stipulated? Good note? Now, can you draw the rest of the owl?
I am focusing on DNS because this thread is about DNSSEC. The topic of doing it in to the TLS servers themselves is a tangent not relevant to this thread.
No, I'm sorry, that's not the case. You're focusing on DNS servers as the target for BGP4 attacks because if you didn't, you wouldn't have a rebuttal for the very obvious question of "why wouldn't BGP4 attackers just use BGP4 to intercept legitimate ALPN challenges".
> You start with a BGP hijack, which lets you impersonate anybody, but assume that the hijacker is only so powerful as being able to impersonate a specific DNS server and not the server that the DNS server tells you about.
An attacker impersonating a DNS server still won't be able to forge the DNSSEC signatures.
An attack against BGP where the attacker takes over traffic for an IP address isn't at all prevented by DNSSEC.
The sequence there is:
1. I hijack traffic destined for an IP address
2. Anything whose DNS resolves to that IP, regardless of whether or not they use DNSSEC, starts coming to me
In this model, I don't bother trying to hijack the IP of a DNS server: that's a pain because with multi-perspective validation, I plausibly have to hijack a bunch of different IPs in a bunch of different spots. So instead I just hijack the IP of the service I want to get a malicious cert for, and serve up responses to let me pass the ALPN ACME challenge.
Sure. But you won't have a TLS certificate for that address, if the host uses a DNS-based ACME challenge and prohibits the plain HTTP challenge: https://letsencrypt.org/docs/caa/
Ok, so deploying DNSSEC would specifically solve the threat model of an attacker who can perform a BGP hijack of IP addresses, but doesn’t want to hijack multiple DNS server IPs because that’s more work, for a domain that has CAA records and disallows validation by ALPN.
That feels like a pretty narrow gain to justify strapping all this to all my zones and eating the operational cost and risk that if I mess it up, my site stops existing for a while
> but doesn’t want to hijack multiple DNS server IPs because that’s more work
No. I'm saying that you can _not_ hijack a DNSSEC-enabled DNS name, even if you have a full control over the network.
The DNSSEC public keys for the domain are stored in the top-level domain zone. Which in turn is protected by a signature with the key from the root zone.
I don’t think you’re grokking what a BGP hijack looks like. The attacker steals traffic destined to an IP address at the routing layer. They aren’t hijacking a name, they’re hijacking traffic to the IP that name resolves to.
In the case of attacking the ALPN ACME validation, they hijack the IP address of the site they want a TLS certificate for: example.org resolves to 1.2.3.4, I hijack traffic to 1.2.3.4, the DNS flow is unchanged, the verification traffic comes to me, and I get a certificate for example.org
The DNS server hijack works the same way: I don’t try to change what ns1.example.org resolves to. I hijack traffic to the real IP that it resolves to and serve up responses for the site I want to hijack saying “yea, these are the records you want and don’t worry, the DS bit is set to true”.
Though it’s worth remembering that both DNS and BGP attacks are basically a rounding error compared to the instances of ATO-based attacks
I know exactly how BGP works, I actually implemented a BGP reflector long time ago. My home has two DIA circuits and my home network is announced via BGP.
> In the case of attacking the ALPN ACME validation, they hijack the IP address of the site they want a TLS certificate for: example.org resolves to 1.2.3.4, I hijack traffic to 1.2.3.4, the DNS flow is unchanged, the verification traffic comes to me, and I get a certificate for example.org
As I said, a CAA record in DNS will prohibit this, instructing the ACME CA to use the DNS challenge.
> I hijack traffic to the real IP that it resolves to and serve up responses for the site I want to hijack saying “yea, these are the records you want and don’t worry, the DS bit is set to true”.
And then your faked DNS replies will have a wrong signature because you don't have the private key for the DNS zone.
And DNSSEC-validating clients will detect this because the top-level domain will have a DNSKEY entry for the hijacked domain. You can't fake the replies from the top-level domain DNS because it in turn will have a DNSKEY entry in the root zone.
> It is important. This is unfortunate rhetoric that is harming the safety of the internet.
DNSSEC was built for exactly one use case: we have to put root/TLD authoritative servers in non-Western countries. It is simply a method for attesting that a mirror of a DNS server is serving what the zone author intended.
What people actually want and need is transport security. DNSCrypt solved this problem, but people were bamboozled by DNSSEC. Later people realized what they wanted was transport security and DoH and friends came into fashion.
DNSSEC is about authentication & integrity. DNSCRYPT/DOH is about privacy. They solve completely different problems and have nothing to do with one another.
If you have secure channels from recursers all the way back to authority servers (you don't, but you could) then in fact DoH-like protocols do address most of the problems --- which I contend are pretty marginal, but whatever --- that DNSSEC solves.
What's more, it's a software-only infrastructure upgrade: it wouldn't, in the simplest base case, require zone owners to reconfigure their zones, the way DNSSEC does. It doesn't require policy decisionmaking. DNS infrastructure operators could just enable it, and it would work --- unlike DNSSEC.
(Actually getting it to work reliably without downgrade attacks would be more work, but notably, that's work DNSSEC would have had to do too --- precisely the work that caused DANE-stapling to founder in tls-wg.)
I'd love to see DoH/DoT that uses a stapled DNSSEC-authenticated reply containing the DANE entry.
There's still a chicken-and-egg problem with getting a valid TLS certificate for the DNS server, and limiting DNSSEC just for that role might be a valid approach. Just forget that it exists for all other entry types.
Stapling is dead: nobody could agree on a threat model, and they ultimately ended up at an HPKP-style cached "this endpoint must staple DANE" model that TLS people rejected (reasonably).
But if you have DoH chaining all the way from the recurser to the authority, it's tricky to say what stapled DANE signatures are even buying you. The first consumers of that system would be the CAs themselves.
BGP attacks change the semantic meaning of IP addresses themselves. DNSSEC operates at a level above that. The one place this matters in a post-HTTPS-everywhere world is at the CAs, which are now all moving to multi-perspective validation.
As you should be aware, multi-perspective validation does not solve anything if your BGP hijack is accepted to be global. You will receive 100% of the traffic.
DNSSEC does greatly assist with this issue: It would have prevented the cited incident.
1. Hijack the HTTP/HTTPS server. For some IP ranges, this is completely infeasible. For example, hijacking a CloudFlare HTTP/HTTPS range would be almost impossible theoretically based on technical details that I won't go through listing.
2. Hijack the DNS server. Because there's a complete apathy towards DNS server security (as you are showing) this attack is very frequently overlooked. Which is exactly why in the cited incident attackers were capable of hijacking Amazon Route53 with ease. *DNSSEC solves this.*
If either 1 or 2 work, you have yourself a successful hijack of the site. Both need to be secure for you to prevent this.
In summation, you propose a forklift upgrade of the DNS requiring hundreds of millions of dollars of effort from operators around the world, introducing a system that routinely takes some of the most sophisticated platforms off the Internet entirely when its brittle configuration breaks, to address the problem of someone pulling off a global hijack of all the Route53 addresses.
At this point, you might as well just have the CABForum come up with a new blessed verification method based on RDAP. That might actually happen, unlike DNSSEC, which will not. DNSSEC has lost signed zones in North America over some recent intervals.
I do like that the threat model you propose is coherent only for sites behind Cloudflare, though.
"I do like that the threat model you propose is coherent only for sites behind Cloudflare, though."
The threat model I proposed is coherent for Cloudflare because they have done a lot of engineering to make it almost impossible to globally BGP hijack their IPs. This makes the multi-perspective validation actually help. Yes, other ISPs are much more vulnerable than Cloudflare, is there a point?
You are not saying DNSSEC doesn't serve a real purpose. You are saying it is annoying to implement and not widely deployed as such. That alone makes me believe your argument is a bit dishonest and I will abstain from additional discussion.
No, I'm saying it doesn't serve a real purpose. I've spent 30 years doing security work professionally and one of the basic things I've come to understand is that security is at bottom an economic problem. The job of the defender is to asymmetrically raise costs for attackers. Look at how DNS zones and certificates are hijacked today. You are proposing to drastically raise defender costs in a way that doesn't significantly alter attacker costs, because they aren't in the main using the exotic attack you're fixated on.
If we really wanted to address this particular attack vector in a decisive way, we'd move away, at the CA level, from relying on the DNS protocol browsers use to look up hostnames altogether, and replace it with direct attestation from registrars, which could be made _arbitrarily_ secure without the weird gesticulations DNSSEC makes to simultaneously serve mass lookups from browsers and this CA use case.
But this isn't about real threat models. It's about a tiny minority of technologists having a parasocial relationship with an obsolete protocol.
yeah, the same for the rest.
your fanboys are happy and the rest is just tired, because everyone who does not share your point of view has a invalid opinion.
>It’s full of rhetoric and bluster, appeals to authority and dismissal of arguments not from what he considers an authority, and when he runs out of arguments entirely, he stops responding.
Or his broken record commentary on how Signal absolutely needs to ask people for their mobile phone numbers in order to at all be able to provide a functional service, and how doing so does not at all provide Signal with a highly valuable social network map. Exact same story as soon as the arguments are dismantled.
Counterpoint: no it isn't, which is why virtually nobody uses it. Even the attack this thread centers on --- BGP hijacking of targeted DNSSEC servers to spoof CA signatures --- is a rounding error sidenote compared to the way DNS zones actually get hijacked in practice (ATO attacks against DNS providers).
If people were serious about this, they'd start by demanding that every DNS provider accept U2F and/or Passkeys, rather than the halfhearted TOTP many of them do right now. But it's not serious; it's just motivated reasoning in defense of DNSSEC, which some people have a weird stake in keeping alive.
> Counterpoint: no it isn't, which is why virtually nobody uses it. Even the attack this thread centers on --- BGP hijacking of targeted DNSSEC servers to spoof CA signatures
Wait, wait, wait. How can you hijack a DNSSEC server? Its keys are enrolled in the TLD, and you can't spoof the TLD server, because its keys in turn are enrolled in the root zone. And the root zone trust anchor is statically configured on all machines.
DNS is an industry that has a race to bottom in terms of quality and price, and will generally avoid any costs in order to keep as much of the small margin they get between the costs from the TLD and the end user price. For many registrars and domains the margin sits around $0-1 per domain and year, and the real profit comes from uppsells and dark patterns that hikes up the price until the customer manage to leave.
High-end registrars that focus on portfolios and brand management generally do have any kind of authentication that the customers want, and the price will reflect that. Costumers are not just buying the service of a middle man that acts as a go-between the TLD and the domain owner.
You are again ignoring the fact that DNSSEC would have prevented a $152,000 hack. Yes, we are aware organizations are not always serious about security. For those that are though, DNSSEC is a helpful tool.
No, it isn't. It attempts and mostly fails to address one ultra-exotic attack, at absolutely enormous expense, principally because the Internet standards community is so path-dependent they can't take a bad cryptosystem designed in the mid-1990s back to the drawing board. You can't just name call your way to getting this protocol adopted; people have been trying to do that for years, and the net result is that North American adoption fell.
The companies you're deriding as unserious about security in general spend drastically more on security than the companies that have adopted it. No part of your argument holds up.
Citation? A BGP hijack can be done for less than $100.
"You can't just name call your way to getting this protocol adopted"
I do not care if you adopt this protocol. I care that you accurately inform others of the documented risks of not adopting DNSSEC. There are organizations that can tolerate the risk. There are also organizations that are unaware because they are not accurately informed (due to individuals like yourself), and it is not covered by their security audits. That is unfortunate.
Slack's house literally did burn down for 24 hours because of DNSSEC back into 2021.
When you frame the risk as "marginal benefit against one specific threat" Vs "removes us from the internet for 24 hours", the big players pass and move on. This is the sort of event the phrase "sev 1" gets applied to.
Some fun companies have a reg requirement to provide service on a minimum SLA, otherwise their license to operate is withdrawn. Those guys run the other way screaming when they hear things like "DNSSEC" (ask me how I know).
What percentage of the fortune 500 is served over DNSSEC?
Oh. I thought it burned down because of their engineers not having fully acquainted themselves with the tool before applying it. It's misguided to hold DNSSEC culpable for Slack's engineers' goof-up. Like advising people against ever going near scissors because they might run with one in their hands.
That is an extremely uncharitable take for an outage that involved two of the most poorly defined and difficult to (correctly) implement DNS features (wildcards and NSEC records) that the three major DNS operators mentioned in the Slack post-mortem (AWS, Cloudflare, and Google) all implemented differently.
IIRC, Slack backed out of their DNSSEC deployment because of a bug with wildcards and NSEC records (in Route53, not Slack), but the problem Slack subsequently experienced was not caused by that bug, but was instead caused by the boneheaded way in which Slack tried to back out of DNSSEC. I.e. Slack’s problem was entirely their own doing, and completely avoidable if they had had any idea of what they were doing.
Having read the post mortem, I disagree. Slack engineers did something dumb, under pressure during an outage. Even if they hadn't, they still would have been in a degraded state until they could properly remove their DNSSEC records and/or get the Route53 bug they hit fixed. In other words, they still would have had a 24+ hour outage, albeit with a smaller blast radius.
The design of DNSSEC is simply not fit for purpose for zone operators. It is far too easy to screw your zone up for far too marginal a benefit, to say nothing of the huge increase in CPU resource required to authenticate DNSSEC record chains.
The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options.
To give just one example, consider the NSEC3 iteration limit field. NSEC3 itself was designed mostly[0] to prevent zone enumeration when validating negative responses (which is trivial to perform with NSEC.) The iteration count was designed to give zone operators the ability to increase the cost to an attacker of generating a dictionary of nsec3 names[1]. Of course, a high iteration count also raises the cost to a non-malicious resolver of validating a negative response for a nsec3-enabled zone.
In good old DNSSEC fashion, the iterations field is a single number that is subject to a... wide variety of potential limits:
* 16 bits by the wire protocol
* 150 for 1,024 bit keys (RFC 5155 10.3[2])
* 500 for 2,048 bit keys (RFC 5155 10.3[2])
* 2,500 for 4,096 bit keys (RFC 5155 10.3[2])
* 0 (RFC 9276 3.2)
Why 0? It was noted -- after publishing the NSEC3 spec -- that high iterations just don't provide that much benefit, and come with a high cost to throughput. Appendix B of RFC 9276 shows a roughly 50% performance degradation with an iteration count of 100. So, RFC 9276 3.2 says:
Validating resolvers MAY also return a SERVFAIL response when processing NSEC3 records with iterations larger than 0.
Of course, their guidance to implementers is to set the limits a bit higher, returning insecure responses at 100 iterations and SERVFAIL at 500. That said, if you want to be maximally interoperable, as a zone operator, you should pretend like the iteration count field doesn't exist: it is standards compliant for a validating resolver to refuse an nsec3 response with more than a single hash round.
As I said, this is one example, but I'm not cherry picking here. The whole of the DNSSEC spec corpus is filled with incomprehensible verbiage and opportunities for conflicting interpretations, far beyond what you see in most protocol specs.
0 - also to reduce the size of signed top-level zones
1 - all NSEC and NSEC3 records, while responsive to queries about names that don't exist, consist of obfuscated names that do exist.
2 - According to the letter of the standard, the limits applied to the iterations field should be 149, 499, and 2,499. Implementations are inconsistent about this.
IIUC, if Slack had done the correct thing, only wildcard DNS records (if any) would have been affected. They would certainly not have had a complete DNS blackout. I would classify that as significant.
> The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options.
I don’t care. So is almost every other standard, but until something better comes along, DNSSEC is what we have. Arguing that a working and implemented solution should not be used since it is worse than a non-existing theoretical perfect solution is both:
1. True
2. Completely and utterly useless, except as a way to waste everyone’s time and drain their energy.
> IIUC, if Slack had done the correct thing, only wildcard DNS records (if any) would have been affected
There's the problem - you DON'T understand. Straight from the post-portem that you clearly have not read:
One microsecond later, app.slack.com fails to resolve with a ‘ERR_NAME_NOT_RESOLVED’ error:
[screenshot of error ]
This indicated there was likely a problem with the ‘*.slack.com’ wildcard record since we didn’t have a wildcard record in any of the other domains where we had rolled out DNSSEC on. Yes, it was an oversight that we did not test a domain with a wildcard record before attempting slack.com — learn from our mistakes!
> I don’t care. So is almost every other standard,
Cool story. I do care. I'd like to see greater protection of the DNS infrastructure. DNSSEC adoption is hovering around 4%. TLS for HTTP is around 90%. At least part of that discrepancy is due to how broken DNSSEC is.
They could have done a quick fix by adding an explicit app.slack.com record. But instead they removed the DNSSEC signing from the whole domain, thereby invalidating all records, not just the wildcard ones.
> I do care.
I will care once something else comes around with any promise of being implemented and rolled out. Until then, I see no need to discourage the adoption of DNSSEC, or disparage its design, except when designing its newer version or replacement.
> I'd like to see greater protection of the DNS infrastructure. DNSSEC adoption is hovering around 4%.
I work at a registrar and DNS hosting provider for more than 10.000 domains. More than 70% of them have DNSSEC.
> They could have done a quick fix by adding an explicit app.slack.com record. But instead they removed the DNSSEC signing from the whole domain, thereby invalidating all records, not just the wildcard ones.
1) That would do nothing to fix resolvers that had already cached NSEC responses lacking type maps.
2) That presumes the wildcard record was superfluous and could have been replaced with a simple A record for a single or small number of records. Would love to see a citation supporting that.
3) That presumes the Slack team could have quickly identified that the problem they were having was caused by the fact that app.slack.com (and whatever other hosts resolve from that wildcard) was caused by the fact the record was configured as a wildcard and would have been resolved by eliminating the wildcard record. If you read the postmortem, it is clear they zeroed in on the wildcard record as being suspect, but had to work with AWS to figure out the exact cause. I doubt that was an instantaneous process.
Any way you slice it, there was no quick way to fully recover from this bug once they hit it, and my argument is that the design of DNSSEC makes these issues a) likely to happen and b) difficult to model ahead of time, while providing fairly marginal security benefit.
At this point, I really don't care if you agree or disagree.
> I will care once something else comes around with any promise of being implemented and rolled out.
Yeah. DNSSEC is going to be widely deployed any day now. The year after the year of Linux on the desktop.
> I work at a registrar and DNS hosting provider for more than 10.000 domains. More than 70% of them have DNSSEC.
Cool. There are, what, 750 million domains registered worldwide? We are at nowhere near 10% adoption worldwide, let alone 70%. Of the top 100 domains -- the operators you would assume would be the most concerned about DNS response poisoning -- *six* have turned DNSSEC on.
> 1) That would do nothing to fix resolvers that had already cached NSEC responses lacking type maps.
The TTL for NSEC records are presumably way lower than the TTL for the DS records.
> 2) That presumes the wildcard record was superfluous and could have been replaced with a simple A record for a single or small number of records. Would love to see a citation supporting that.
It’s theoretically possible that it would not have worked for all cases, but that is, in my experience, very unlikely.
> Any way you slice it, there was no quick way to fully recover from this bug once they hit it
The bug seems to me to have been reasonably easy to mitigate, but their problem was that Slack did not know what they were doing. Thu bug itself was minor, but Slack tried to fix it by stopping to serve DNSSEC-signed DNS data, while long-TTL DS records were still being unexpired in the world. This is the worst possible thing you could do.
> Of the top 100 domains -- the operators you would assume would be the most concerned about DNS response poisoning -- *six* have turned DNSSEC on.
1. That number used to be zero, as tptacek liked to point out.
2. The huge operators often have fundamentally different security priorities than regular companies and users.
3. People said the same about IPv6 and SSL, which were also very slow to adopt. But they are all climbing.
Internally at slack the general consensus was that dnssec was a giant waste of time and money from a security perspective. We did it for compliance to sell into the Federal govt and federal contractors.
I'm not sure what this has to do with anything I've said on this thread, but we don't have to keep pressing these arguments; I'm pretty satisfied with the case I've made so far.
And at least Let's encrypt actually verifies DNSSEC before issuing certificates. IIRC it will become mandatory for all CA's soon. DNSSEC for a domain plus restrictive CAA rules should ensure that no reputable CA would issue a rogue cert.
"Most domains". Yes, it is possible that nobody bothers to DNS hijack your domains. Sadly I've worked for organizations where it did happen, and now they have DNSSEC.
I invite anybody who thinks this is a mic drop to pull down the Tranco research list of most popular/important domains on the Internet --- it's just a text file of zones, one per line --- and write the trivial bash `for` loop to `dig +short ds` each of those zones and count how many have DNSSEC.
For starters you could try `dig +short ds google.com`. It'll give you a flavor of what to expect.
And you still can't seem to make your mind up on whether this is because DNSSEC is still in its infancy or if it's because they all somehow already studied DNSSEC and ended up with the exact same opinion as you. I'm gonna go out on a limb and say that it's not the latter.
What do I have to make my mind up about? I worked on the same floor as the TIS Labs people at Network Associates back in the 1990s. They designed DNSSEC and set the service model: offline signers, authenticated denial. We then went through DNSSEC-bis (with the typecode roll that allowed for scalable signing, something that hadn't been worked out as late as the mid-1990s) and DNSSEC-ter (NSEC3, whitelies). From 1994 through 2025 the protocol has never seen double-digit percentage adoption in North America or in the top 1000 zones, and its adoption has declined in recent years.
You're not going to take my word for it, but you could take Geoff Huston's, who recently recorded a whole podcast about this.
That quote is interesting because all of the period reporting I’ve seen says that the attackers did NOT successfully get an HTTPS certificate and the only people affected were those who ignored their browsers’ warnings.
How about another incident in 2022? Attackers BGP hijacked a dependency hosting a JS file, generated a rogue TLS certificate, and stole $2 million. Keep in mind: these are incidents we know about, not including incidents that went undetected.
Noteworthy: "Additionally, some BGP attacks can still fool all of a CA’s vantage points. To reduce the impact of BGP attacks, we need security improvements in the routing infrastructure as well. In the short term, deployed routing technologies like the Resource Public Key Infrastructure (RPKI) could significantly limit the spread of BGP attacks and make them much less likely to be successful. ... In the long run, we need a much more secure underlying routing layer for the Internet."
You know why I'm not coming back at you with links about registrar ATOs? Because they're so common that nobody writes research reports about them. I remember after Laurent Joncheray wrote his paper about off-path TCP hijacking back in 1995; for awhile, you'd have thought the whole Internet was going to fall to off-path TCP hijacking. (It did not.)
The argument counter dnssec is that if you are trying to find some random A record for a server, to know if it is the right one, TLS does that fine provided you reasonably trust domain control validation works i.e. CAs see authentic DNS.
An argument for DNSSEC is any service configured by SRV records. It might be totally legitimate for the srv record of some thing or other to point to an A record in a totally different zone. From a TLS perspective you can't tell, because the delegation happened by SRV records and you only know if that is authentic if you either have a signed record, or a direct encrypted connection to the authoritative server (the TLS connection to evil.service.example would be valid).
Yes but it is still possible to execute BGP hijacks that capture 100% of traffic, rendering multi-perspective validation useless. RPKI sadly only solves naive "accidental" BGP hijacks, not malicious BGP hijacks. That's a different discussion though.
I agree and apparently so does the CA/B forum: SC085: Require DNSSEC for CAA and DCV Lookups is currently in intellectual property review.
DCV is CA/B speak for domain-control validation; CAA = these are my approved CAs.
This seems to be optional in the sense that: if a DNS zone has DNSSEC, then validation must succeed. But if DNSSEC is not configured it is not required.
Not saying they are malicious actors, but easy answer would be any Public WiFi anywhere. They all intercept DNS, less than 1% intercept SNI.
It is also public knowledge that certain ISPs (including Xfinity) sniff and log all DNS queries, even to other DNS servers. TLS SNI is less common, although it may be more widespread now, I haven't kept up with the times.
Popular web browsers send SNI by default regardless of whether it is actually needed. For example, HTTPS-enabled websites not hosted at a CDNs may have no need for SNI. But popular web browsers will send it anyway.
every single ISP in the world. it was a well documented abused channel.
they not only intercepted your traffic for profiling but also injected redirects to their branded search. honestly curious if you're just too young or was one of the maybe 10 people who never experienced this.
sending traffic to a third party like quad9 is much safer than to a company who have your name/address/credit card.
If the threat model is BGP hijacking, is DNSSEC actually the answer? If you can hijack a leaf server, can't you hijack a root server? As far as I can tell, the root of DNSSEC trust starts with DNSKEY records for the "." zone that are rotated quarterly. This means every DNSSEC-validating resolver has to fetch updates to those records periodically, and if I can hijack even one route to one of the fixed anycast IPs of [a-m].root-servers.net then I can start poisoning the entire DNSSEC trust hierarchy for some clients, no?
Now, this kind of attack would likely be more visible and thus detected sooner than one targeted at a specific site, and just because there is a threat like this doesn't mean other threats should be ignored or downplayed. But it seems to me that BGP security needs to be tackled in its own right, and DNSSEC would just be a leaky band-aid for that much bigger issue.
The much-more-important problem is that the most important zones on the Internet are, in the DNSSEC PKI, irrevocably coupled to zones owned by governments that actively manipulate the DNS to achieve policy ends.
This is a problem with TLS too, of course, because of domain validation. But the WebPKI responded to that problem with Certificate Transparency, so you can detect and revoke misissued certificates. Not only does nothing like that exist for the DNS, nothing like it will exist for the DNS. How I know that is, Google and Mozilla had to stretch the bounds of antitrust law to force CAs to publish on CT logs. No such market force exists to force DNS TLD operators to do anything.
My (ahem) "advocacy" against DNSSEC is understandably annoying, but it really is the case that every corner of this system you look around, there's some new goblin. It's just bad.
I agree that the problem lies with BGP and we definitely need a solution. You can also say the problem is with TLS CA verification not being built on a solid foundation. Even with that said, solving those problems will take time, and DNSSEC is a valid precaution for today.
RPKI plus ASPA does solve the hijack problem by securing both the origin of a prefix and the AS path of a route.
Yes ASPA is new. Reference implementations in open source routing daemons and RPKI tools are being developed and rolled out. If you want to be a pioneer you can run a bird routing daemon and secure the routes with ASPA. Only experimenters have created ASPA records at this point, however once upon a time we were in the same position with RPKI.
reply