The thing that I said was "table stakes" for implementing cryptography was passing the algorithm test vectors, which this author's previous post claimed as a security feature.
If you're unfamiliar with the concept, a test vector is a series of strings and intermediate values used to ensure that your (say) OCB3 is the same as everyone else's OCB3.
Had he asked, I'd further claim that not having dumb C bugs is also table stakes for cryptography, since serious crypto vulnerabilities happen at a higher level of abstraction. The author grazes past (somewhat disconcertingly) one such class of bugs when he discusses carry propagation and limb scheduling in the context of Poly1305. Improper carry propagation is an example of the kind of security vulnerability for which there are no test vectors and no memory safety validation tools.
You just have to know what a carry propagation bug is, where to look for them (not Poly1305), and what the impact of one is.
> You just have to know what a carry propagation bug is
This is one of the things that drives me nuts about the crypto community. The natural response upon reading, "You just have to know X" if you don't already know is to go search for X. Well, if you go search for "carry propagation bug" you will find lots of examples of carry propagation bugs being found and fixed, but no explanation of what one is. The thing that "you just have to know" is not so easy to find. The courteous thing to do when mentioning something that "you just have to know" but which is not easy to find is to provide a pointer or a short in-line explanation. When you don't do that you leave your reader with no choice but to publicly expose their ignorance of this purportedly crucial knowledge by asking, which I will now proceed to do as a public service: What the hell is a carry propagation bug? Where do you look for them? And what is their impact?
Carry propagation bugs are something that can go wrong in a bignum implementation.
Bignum is what you need when you have one number that you're splitting between multiple variables (because it's a 256-bit number, and you want your library to work on 32-bit PCs, for example).
Carry propagation is if part of your result is too large for one part of your number and needs to overflow into the next part.
Sean, Marcin, and I spent the better part of a year individually replying to tens of thousands of emails from people doing the Cryptopals challenges that we wrote and published for free. Do you know how much work that was? Here's a hint: it was a lot of work.
There are a lot of charges you can level at me to try to win a dumb message board argument, but the one where I want cryptography to be a secret priesthood known only to the anointed select is not one of them --- nor is the argument that I haven't taken the time to help make these kinds of cryptographic attacks easier for people to learn about.
Please immediately find one of those other arguments to substitute for this one, and, if you have it in you, take a moment to retract.
I do sincerely apologize for taking my frustrations out on you, because you have indeed gone above and beyond the call to help make crypto accessible.
Nonetheless, I still stand by the substance of my comment: when you (not you specifically, but anyone) say, "You have to know X" you should check that doing a Google search for X yields some reasonable results, and if it doesn't, provide a hint on how to proceed.
> Nonetheless, I still stand by the substance of my comment: when you (not you specifically, but anyone) say, "You have to know X" you should check that doing a Google search for X yields some reasonable results, and if it doesn't, provide a hint on how to proceed.
Do you actually believe you are personally entitled to demand special research tips from domain experts?
I don't think it's too much to ask for links to resources.
If it's a common enough remedy that needs linking, you could just write a dedicated blog post describing the recommended resources or whatever and just link to that.
Sometimes the hard part of researching things is knowing what to look for or where to look for it.
This is more an indictment of the whole community that there aren’t clear glossaries or other introductory resources with good google juice, and doesn’t necessarily reflect much on your specific comment or intent.
He could have phrased it much better, but I think you’re also overreacting.
I thought so too, but then I read his comment downthread about how my wording "rubbed his nose" into the fact that he doesn't know what a carry propagation bug is. Sorry, I stand by what I wrote.
Carry propagation is one pixel in the 4K movie that is safe crypto implementation. One reason not to link to trivia and factoids about safe crypto implementation is that it builds a dangerous false sense of competence.
There are programs that teach this, but they take years. Nobody who's passed through that is interested in creating an easy scaffolding whose levels are unclear.
> Carry propagation is one pixel in the 4K movie that is safe crypto implementation
This is exactly the sort of thing I'm talking about. This is semantically equivalent to, "If you don't already know this, then you are too stupid, and my time is too valuable, for me to do anything beyond pointing out that you are stupid and my time is valuable." That may be true, but it's not helpful.
> There are programs that teach this
But you aren't going to tell me what they are or how to go about finding them? Again, this is not helpful.
Funny you should choose flying an airliner as your example because I happen to be a pilot. I don't fly airliners. I'm just a private pilot who flies small planes. But despite the fact that I'm not a "trained professional", I can fly a plane safely. And the reason I can do this is because the flying community does not have this attitude that flying should only be done by trained professionals. To the contrary, the flying community has the attitude that anyone can learn to fly, and if you want to to it, all you have to do is find your nearest general aviation airport where you will almost certainly find a flight school, which will put you behind the wheel (so to speak) of a plane the very same day (though, obviously, with supervision).
The crypto community's attitude seems very different, far less welcoming of newcomers. The aviation community's attitude towards newcomers is, "Come on in, we are happy to help you take your first step on this exciting journey." The crypto community's attitude feels more like, "Go away, you ignorant dweeb, we are far too busy doing Hard Stuff to bother with the likes of you."
And, BTW, I say this as someone who runs a company that sells a crypto product (https://sc4.us/hsm), so I'm not exactly a beginner any more. But I was once and I still remember what it was like, so I have sympathy for people who want to climb the learning curve and find it difficult. I think the world would be a better place if more people learned more about crypto, and so I do what I can to try to remove obstacles.
The "flying community" does not have the attitude that any random person should take the stick on a 737. Like the cryptography community, the flying community encourages newcomers to learn and get certified.
This is a genuinely weird argument you're trying to make.
As a member of both communities I can tell you that I perceive a genuine difference in attitudes. The crypto community says, "Don't write your own crypto." The aviation community says something more analogous to, "Absolutely, write your own crypto! Just don't use the results of your first effort for anything mission-critical until it has been checked out by someone who knows what they are doing. And oh by the way, here's a list of people you can contact who will be more than happy to help you." (In fact, the aviation community literally encourages people to build their own airplanes!)
It doesn't encourage people to try and build commercial jet airliners in their back sheds. You can follow a tutorial and build a KitFox or whatever, which is the equivalent of coding, I dunno, a virtual Enigma machine or something. You can't follow a tutorial to learn how to design an Airbus A380.
I missed the last sentence. But I think this is also misleading. General aviation is obviously far more dangerous than commercial aviation, and home-built planes are, by a factor of 2-3x, more dangerous than the rest of general aviation.
If you're telling me that there's a mainstream part of general aviation advocating that random people build planes to take strangers for rides in (or, to complete the analogy to crypto, to give to random strangers to themselves fly), I'm going to push back on that.
You're telling me that the general aviation community is, in the mainstream, of the mind that "anyone who wants to" build airplanes to sell to other people should be doing so? With no additional qualifiers?
Not to sell. But for their own use, yes, absolutely. Anybody who wants to. It turns out not very many people want to, particularly when they find out how much work it is. But almost certainly the same is true of crypto.
Not only is that not true, it's demonstrably not true: when people write code, they actively want other people to use it. By way of example, the author of the library we're discussing here, "Monocypher", has declared it "ready for production" and has a web page selling its virtues versus libsodium and NaCl.
People who write crypto code as a rule are not doing it for their own edification, which is why so many more people spend time writing libraries and encryption tools and so few people spend time writing the code to exploit crypto vulnerabilities.
You said that it's almost certainly the case that people building their own crypto are doing it solely for their own purposes and that people won't even bother to complete their work. Obviously: no.
I wrote "almost certainly the same is true of crypto" meaning that almost certainly very few people actually want to implement crypto (just as very few people actually want to build airplanes) for any reason, and so the desire to do so serves as a pretty effective filter to eliminate "random people" doing it.
Of those few who want to do it, some will want to do it for personal reasons and some will want to do it for commercial reasons, just as in aviation.
Yes, but home-built airplanes come with all kinds of limitations that are, to continue the analogy, more like "can't use commercially ever, and any use for non-commercial purposes requires a big disclaimer on every message sent.
Yes, but 1) those are government regulations, not a community policy. As a community aviation is much more welcoming of newcomers than crypto, and in the case of crypto this attitude is entirely community-driven, since crypto is not regulated the way aviation is. And 2) homebrew is often a stepping stone towards professionalism in many fields. But the crypto community actively discourages homebrew with its "Don't build your own crypto, period, end of story" message. The message should be, "Sure, build your own, just don't rely on your first efforts for anything mission critical."
You can't view community policy and government regulation as two unrelated things. The aviation community has a more welcoming attitude because of government regulation.
They know that someone's backyard airplane isn't ever going to be used commercially without certification, and even if it were the entire world's aviation fleet isn't going to start using that model overnight, as might happen with some homebrew crypto and a popular new app. So the situation is entirely unanalogous.
No, that's not true. The aviation community's welcoming attitude goes way back before the government started to regulate it. I'd say it's because aviation began as a homebrew activity whereas cryptography was born in the military and academia.
I wish the meme that crypto is hard would just die.
The reason is that the statements is wrong and is probably encourage exactly what it wants to discourage. Crypto is hard is not factually incorrect. The problem is that it is trivially correct, because programming is always hard. Every algorithm and design is hard to get right.
The correct statement, and the one that should be repeated instead is that crypto's stakes are high. Getting it wrong carries bigger consequences than a normal program.
So stop propagating that crypto is hard. (Because people do hard stuff everyday and might thus think they've got the right stuff to write crypto.)
Say that crypto has too high stakes to be done off the cuff.
No, crypto is actually hard. One difference from regular programming is that there really are no 'intermediate' errors. Experience shows that getting one small detail wrong is as bad as getting the whole thing wrong.
I can't think of another programming domain where that holds.
But there are lots of domains where details matter and small errors have big consequences. Vehicles, medicine, and manufacturing are all places where errors have killed people. Obviously severity depends on the situation, but generally speaking I'd rather my encryption get broken than die.
Software defects in the Therac-25 radiation therapy machine killed several people [1].
Software defects in Toyota's acceleration killed 37 people [2]. "We've demonstrated how as little as a single bit flip can cause the driver to lose control of the engine speed in real cars due to software malfunction that is not reliably detected by any fail-safe," Michael Barr, CTO and co-founder of Barr Group, told us in an exclusive interview. Barr served as an expert witness in this case." [3]
Self-driving autopilot has got to be pretty hard. [4]
Crypto software also has the property that "internal implementation details matter". Nearly all other software does not have this property - you don't care how your hardware/software multiplies two numbers; only that it does it correctly in a reasonable amount of time. That property does not hold for crypto software - the nature of your multiplication may actually leak information, depending on how it is implemented. So in a way crypto software goes against the entire edifice of modern software engineering, which says that you can build software against an abstract lower-level "stack" and generally not care how the lower parts work.
Yeah, that's a great point, security requires that you understand the insides of any dependencies, and treating any part as a black box is a bad idea.
I think performance critical code sometimes shares this particular side of engineering and it's inability to fit into the easy patterns and abstractions we usually prefer.
The point I'm trying to make is that even in critical software, where people's lives are in the balance, there can be minor errors that don't bring down the entire edifice. Whereas in crypto code, there seems to be only one kind of error possible.
I see a few kinds of intermediate crypto errors possible. You can weaken the crypto without breaking it, you can leak metadata, you can screw up protocols without compromising the algorithm, etc..
But yes in some sense you're completely right, perhaps just because the aim of a crypto algorithm is so narrow? The number of things you can do right with a pair of encrypt/decrypt functions is so limited that the number of things you can do wrong is also pretty small.
There's certainly a lot more than can go wrong with "drive a car" than with "encrypt this message".
In the same sense that modern vulnerability research is in large part the study of leveraging the smallest possible memory corruption bug to obtain remote code execution, modern practical cryptanalysis is the study of leveraging the smallest possible unexpected behavior of any sort to full breaks in both confidentiality and integrity.
By way of example: you could "weaken" an RNG so that where a construction expects a 256 bit nonce, you generate only 255 secure bits. If that construction is ECDSA, there's a good chance you've managed to disclose to attackers your signing private key. You could "weaken" a protocol so that attackers can replace an original plaintext with 16 uniform random bits. If the protocol is using CBC mode, you've allowed attackers to recover whole plaintexts.
There aren't a lot of errors that get routinely made in crypto code that don't have devastating consequences.
> You could "weaken" a protocol so that attackers can replace an original plaintext with 16 uniform random bits. If the protocol is using CBC mode, you've allowed attackers to recover whole plaintexts.
I'm surprised this is the thing you want the link for, and not "1 biased bit destroys the security of a 256 bit nonce where the other 255 bits come from secure random".
Ah! Wouldn't that be "attackers can replace an original ciphertext with two chosen blocks"?
> I'm surprised this is the thing you want the link for, and not "1 biased bit destroys the security of a 256 bit nonce where the other 255 bits come from secure random".
Ariane 5 rocket crashed in 1996 because of one wrong int conversion, costing Euro Space Agency something like $500 millions... how about that for "one small detail"? :)
(business) minimum entry requirement for a market or business arrangement OR (poker) limit on the amount a player can win or lose in the play of a single hand [1]
"table stakes" is an idiom that comes from poker. It's the amount you can win or lose in a single hand, that is, the smallest amount of money you need to have to be able to sit at the table and play the game.
Is it possible that you are expecting the cryptography community to be one organized around being newbie-friendly and encouraging to beginners?
The crypto community has repeatedly been burned by overeager beginners with false senses of competence making rather nasty mistakes. Consequences can be dire, from destroying companies to getting people killed. On the lighter end, you have things like Decrytocat.
In some arenas, it's appropriate and a good practice to assume that everyone reading is a beginner who needs 101-level info on everything. Webdev is a good example - CSS, or cascading style sheets, are good for handling the display logic of semantic markup. The stakes are low and the costs for screwing up trivial. With cryptography, it needs to be driven home to the idle reader that this is serious stuff.
> Is it possible that you are expecting the cryptography community to be one organized around being newbie-friendly and encouraging to beginners?
Not at all. I'm not actually expecting anything. All I'm saying is that as a matter of common courtesy (in any field) if you're going to say, "You just have to know X" then you should do at least one of the following:
1. Check that if someone doesn't know X that they can get at least a lead on how to learn about it by searching for X.
2. Provide a pointer for someone who wants to learn about X to follow as a starting point
3. Provide a brief in-line primer on X
If none of those conditions hold then it's not constructive to say, "You just have to know X." What you're really doing in that case is not providing useful information, but rubbing your audience's nose in the fact that they are ignorant and you are not, and worse, that you acquired your knowledge through some privileged channel to which they do not have ready access.
Honestly, what possible reaction could you reasonably expect other than frustration?
If someone posted a message on HN about doing DIY appendectomies with Arduino-controlled robots and I pointed out that you need to go to med school to do surgeries, you would not in fact be harping on me for not having explained why.
I think you are arguing yourself into an absurd corner here, because I know for a fact that you can name off the top of your head five university crypto programs, and I'm sure you realize that for each of the ones you can name, there are 10 more that others can name.
And: for what it's worth, I don't think going to school is the only way to learn this stuff. I just don't think putting end-users at risk, especially since they can't possibly be expected to tell the difference between Dan Boneh and some guy who assembled a broken ECDH out of random npm libraries.
I re-read your GP comment, and I think I misread it the first time. Let me try this again.
> If someone posted a message on HN about doing DIY appendectomies with Arduino-controlled robots and I pointed out that you need to go to med school to do surgeries, you would not in fact be harping on me for not having explained why.
Actually, I might well harp on you for saying that, but that's a different topic.
My complaint is not that you're pointing out that crypto is hard. It absolutely is. My complaint is much narrower than that. It is specifically a complaint about making statements of the form, "You just have to know what X is" when X is not something that is easy to look up. "Carry-propagation bug" is such an X. "Appendectomy" is not.
And all I'm asking for is that you (not just you, but anyone who writes about things like this) add something like, "If you don't know what a carry propagation bug is, here's a pointer..." or "... it's when you're doing bignum math, you have a carry from one digit to the next, and you mess that up. These bugs are particularly nasty because they only manifest themselves very rarely and so are easy to miss in testing." Or something like that.
I submit that "carry-propagation bug" is easily looked up.
You're asking for people to provide footnotes and extra information so that you can learn what they're talking about when they delve into jargon. I get it. It makes it so much easier to follow along. Then you can learn!
Two points come to mind. First, the bit of jargon you've picked on is in fact easily investigated with tools you are already familiar with. Second, you're asking people writing for an audience of their choice to redefine it to include arbitrarily lay people. Which may not be maximally reasonable - there's a reason Watson and Crick's paper didn't include a primer on what X-rays are.
I clicked your link, and the results weren't useful. They were mostly links to reports about fixes for carry-propagation bugs in various pieces of software (which did not explain what a carry-propagation bug is).
Ironically, the fourth result was a link to a different HN discussion about someone trying to figure out what a carry-propagation bug is, and failing to find any good resources through Google. (Although, to be fair, if you dig into that link a bit more, someone does attempt to explain.)
So I wouldn't say it's impossible to find on one's own, but saying it's "easy" is a bit of a stretch.
It's not especially easy! That's what worries me about laypeople writing crypto libraries. If you know to look, you stand a reasonable chance of learning enough about error oracle side channels to avoid writing something that recapitulates Bleichenbacher's RSA attack. But even if you remember someone telling you to look out for carry propagation bugs, I'm not sure you have a good chance of figuring how how to test for (or exploit) a carry propagation bug. And there are other bugs like this.
> I submit that "carry-propagation bug" is easily looked up.
Well, obviously anything is easily looked up. The problem is that in this case the results aren't very helpful if you don't already know what it is. I even said this in my OP: " if you go search for "carry propagation bug" you will find lots of examples of carry propagation bugs being found and fixed, but no explanation of what one is."
You're right! That's perfectly normal and extremely common.
People are skittish about this in the realm of cryptography because the costs of getting it wrong can be very high. Further, programmers have a culture where experimenting by writing something yourself is perfectly reasonable and extremely common.
When these two meet in the context of crypto, I know just enough to get worried that someone's going to get overconfident from a primer and do something they believe to be very clever. History has a nasty tendency to validate fear this as well-intentioned clever people do things that they believe are good enough.
Decryptocat isn't the lighter end of things. Snowden used it to communicate with Glenn Greenwald's associates during the same time period Decryptocat was found (there are worse flaws in that software than Decryptocat, by the way).
"Table stakes" in competetive Poker means that all you can bet is what's on the table, in front of you.
So this old movie cliché where the protagonist has lost almost all his chips and needs a huge win, so he casually throws in his car keys and the title to his house in the middle of play is not allowed. Neither is grabbing your wallet and throwing cash in.
The idea is that (a) rich people cannot bully others around at will with their huge wealth "in the background" and also (b) players are protected from losing possessions extraneous to the game.
You’re getting confused between two different things with the same name. This use of “table stakes” means something like the minimum buy-in to join the table or the table’s minimum bet to play a hand.
When used as an analogy, the table stakes are just features expected of anyone, not something special to advertise. For example, you wouldn’t say “I’m the best driver because I have a driver’s license and car insurance” – those things are just “table stakes” for drivers.
Yes, and this is kind of the point I'm trying to make. "Don't write your own crypto" is the wrong message IMHO. The message should be something more like, "Absolutely, write your own crypto, because that's the best way to learn, and here's a pointer to how to get started. Just don't use the results of your early efforts for anything mission critical, and if you want to know why, read this..."
And, taking my own advice, here's the link everyone should be pointing to:
I have the strongest instinct that "Don't write your own crypto" is one of the few successful barrier-to-entry phrases in the same spirit as something like "Surfing sucks, don't try it".
> Improper carry propagation is an example of the kind of security vulnerability for which there are no test vectors
I beg your pardon?
Some tests vectors from the RFC are designed specifically to handle some overflow that if handled improperly will give you the wrong results. I have checked, this hits the relevant paths. Or were you talking about something else?
By the way, I haven't attempted to implement curve25519 field arithmetic myself, that would have been suicide (compared to curve25519, Poly1305 is a piece of cake). It all comes from ref10, minus the left shift UB it had.
> which this author's previous post claimed as a security feature.
Here's my current view on security: correct software is secure, period. Vulnerable software is incorrect, period.
Which means that anything that helps assert correctness is a security feature. Test vectors, sanitisers, static analysis, Valgrind… are all security tools, and passing any one of them increases confidence in the product, thus making it more secure. Add enough of these, and confidence gets close enough to 100% that we can deem it ready for production.
So, yeah, test vectors are a security feature. Laughably insufficient by themselves, but still necessary to assert the security of a crypto suite.
> Had he asked, I'd further claim that not having dumb C bugs is also table stakes for cryptography
Yeah, sorry about that. I assumed the same, but I thought Valgrind had me covered at the time —oops. Using sanitisers on Monocypher has been quite the eye opener. I didn't realise how dangerous I used to be.
I still have my doubts, by the way. While I would gladly trust Monocypher with my data by now, I don't dare openly recommending it over Libsodium just yet. That may have to wait for a serious external audit.
I feel like I'm reading you again claiming that the test vectors for an algorithm in the standards documents constitutes a battery of security tests for the algorithm. That's not the case; I think 'lisper will tell you the same thing, despite being much more on your side of this issue than I am. Test vectors are an implementation aid and mostly intended for interop.
And yes, I believe I'm talking about a more complicated class of bugs than you are. For instance: OpenSSL's algorithm implementations pass test vectors (OpenSSL is FIPS certified, too). But OpenSSL has had recent carry propagation bugs.
I am not concerned about your use of C, or about the presence of memory corruption bugs in your library. If I was going to harp on about that, I'd also have to go after everyone who's written a Markdown library. My concern is that the kinds of things you say you've done to ensure the security of your cryptography don't address real cryptographic vulnerabilities.
You may have done other things and I just haven't read about them! I don't really care about your library one way or the other; the only thing I care about is the idea that if you pass test vectors and Valgrind is OK with your binary, you're probably OK. No!
I'll look up this carry propagation business, but again, this looks like a mundane correctness issue (with possibly tragic consequences), though it is hard to assess without looking at intermediate results (I have).
Also, didn't the carry propagation bug reports come with failing tests vectors? Or are they hard to find even if one knows of the bug?
> the only thing I care about is the idea that if you pass test vectors and Valgrind is OK with your binary, you're probably OK. No!
Definitely agree. Even test vectors + safe Rust aren't enough. Which is why I added random comparison tests with libsodium (for every possible length input between 0 and a couple times the size of the block), and property based tests.
I'm still a bit uneasy about carry propagation in Poly1305, but I really checked the hell out of my code.
I originally didn't. But when I noticed it was designed to facilitate 32-bit limbs, I couldn't resist. The result is pretty simple.
I also figured it would perform well, though it currently seems to be 20% slower than Donna (32-bit version). I'll try to find why (I can think of 2 causes: sub-optimal loading code, and crappy data flow in the main carry operation. If it's the latter, I'll use the Donna implementation.)
EDIT: I tweaked the loading code again, it's even faster now. I'm now 7% than poly-donna-32 (and 5% faster than libsodium). My implementation is also simpler. I'm keeping it.
> go do the research about how these bugs are found and exploited,
I'd have to know how.
My first searches turned up no methodology —just instances of such bugs being found. And again, this is just a correctness issue. One just have to prove the whole thing works as intended (even informally). I wouldn't trust myself to do it for curve25519, but poly1305 was doable.
I am terrified that I do not consider myself competent enough to write a crypto library, and yet there isn't a single mention - in this article, nor at the time of writing the comments here on Hacker News - of many of the pitfalls I know to avoid when undertaking such an endeavour. There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items.
I know "don't roll your own crypto" comes across as dismissive or patronising. I'm not a huge fan of the phrasing. But as a first-order approximation, it is correct. If you want to roll your own crypto, that needs to be your _thing_. It's very unlikely you're going to be a fantastic full-stack web developer _and_ be able to do that. If it's really what you want to do then awesome! Go study it, learn it, practice it. But if you're doing it as a side-hobby, never put it into production.
Hey, like you I decided to dive into cryptography coming from a different background. Although this quote is not directly related to your problem it can be safely applied to it:
"Almost certainly you will get the urge to invent new cryptographic algorithms, and will believe that they are unbreakable. Don't resist the urge; this is one of the fun parts. But resist the belief; almost certainly your creations will be breakable, and almost certainly no one will spend the time breaking them for you. You can break them yourself as you get better."[1]
The problem that Schneier is referring to is that doing a careful analysis (that is required if you want to use your crypto in production) is tedious, time consuming, and requires expertise in the area to know most blank spots of the algorithms (heck a single one is hard enough). That's why he recommends you to be have enough experience breaking many algorithms before you make any serious claim about your the safety of your crypto.
So all in all it's great that you got the interest in the area, there is a lot of work needed in OSS. But be very careful about claiming that your lib is ready for production. What you got is a bunch of volunteers to glance at your code. Few cryptographers (if any) will seriously try to break it.
Note that the author is not inventing crypto algorithms, rather implementing them (although the part about XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line). Still a risky business, and tricky to get right, but several orders of magnitude safer than "hey, what if we just XORed everything with a random number? Unbreakable, eh?"
> the part about XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line
It was a few months ago. Then Libsodium danced on the same line, and I was able to compare the results (they behave the same, phew).
It's not inventing crypto either: it's a straightforward application of what was done to Salsa20. The security reduction that worked for XSalsa20 (guaranteed secure if Salsa20 is secure) applies to XChacha20 as well.
> Then Libsodium danced on the same line, and I was able to compare the results (they behave the same, phew).
Do they really? What about timing attacks? Does all your code run in constant time? If not, there's a whole class of vulnerability that basically will only be found when experienced attackers have a chance to poke at it. Timing attacks are a great example of why "don't roll your own crypto" is sound advice. No amount of testing your library against a reference library will uncover them.
Can't say for sure, but it look like they didn't have an independent implementation to compare to. Just like me a few months a go.
---
> What about timing attacks?
Monocypher is immune.
First, the primitive were chosen precisely because they are easily immunised against timing attacks.
Second, this is easily checked simply by looking at the source code: no secret-dependent branches, no secret-dependent indices, and that's about it. One doesn't need to be an expert to check that, a junior programmer could do it after being told what to look for.
What's wrong with xoring with a stream of random numbers? Isn't it how stream ciphers work? Get a good cryptographic RNG, initialize it properly with a long enough key and you should be fine.
You're right about the theory, but you're also wrong about how to implement - which is the point of telling people not to roll their own crypto.
What about reseeding or prediction resistance? How do you handle biased entropy input from that hairdryer someone is blowing on your chips? Oops, the quantum random number generator card doesn't work anymore after adding that extra GPGPU to the system.
You wouldn't want true random numbers anyways. If you're encrypting it, then you'll want to decrypt it later. That means you'll need the random data you used again. If you had a secure channel over which to send or store the random data, then you could just use that channel to send or store the plaintext itself (note that the random data will be the same length as the plaintext). Such a system could only be useful if the cryptographic key was shorter than the plaintext. So it would have to be a deterministic RNG, and you would only need the seed and the ciphertext in order to decrypt.
Also, if someone has acccess to the hardware, then you're screwed. No one would bother wasting time with a hairdryer. If the plaintext was on your harddrive, they would just grab that and run.
I think you missed the point of the comment, which wasn't about physical possession of a computer.
You also make assumptions, such as chips being inside the computer, when they can in fact be in an outside environment-based RNG. The hairdryer example is the canonical and prototypical weakness in that particular type of "rng". I'll try to be explicit about that in the future.
I didn't write anything about environment-based RNG. RNG must be deterministic and seeded by preshared encryption key. One point of using a cryptographically secure RNG is that it should not be possible to learn the seed by observing the output of the RNG without bruteforcing the key.
If you had a RNG that sampled some hardware noise but then you altered the environment by introducing extreme temperatures, maybe the RNG would end up with biased output. A simple example: you can use a camera as a good source of randomness, even taking a picture of a wall indoors. But if an attacker was able to flood that room with light and overwhelm your sensor, he could predict the image would be 100% white pixels or near to it and guess the random output.
But the comment is exaggerating. Encryption algorithms do not handle the case of your entropy source being compromised. Good RNG algorithms like Yarrow try to mitigate this somewhat.
Even if you need a real entropy-based RNG, e.g. for key generation, you need to gather entropy once at the beginning to seed the RNG. Then running RNG does not need any more entropy, so your hairdryer trick won't work. A common myth I hear is that "entropy" gets "depleted" during RNG operation, but in fact it does not. You only need enough bits of entropy once at the beginning, equal to the internal RNG state length.
Strong RNG designs re-seed to deal with compromise or poor seeding. That way if an attacker has temporary read access they do not permanently compromise the system going forward. Or if on boot you are in a compromised or low entropy environment the system will improve over time. Perhaps the utility of this is arguable. If an attacker is able to see your private RNG state then it is most likely game over. But it is a cheap operation with little downside and again, designs like Yarrow put considerable time into thinking it through.
Well, to be fair, a good implementation should throw some simple statistical sanity checks at RNG input during self-test and throw an error when it doesn't pass. Passing such tests doesn't indicate that the RNG is cryptographically secure, but failing them indicates that it is insecure.
Nothing wrong with what you wrote - but note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption).
> note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption)
No, that's not a difference. A "stream of numbers" is just one very large number. What you're worrying about is how large the numbers are, not whether there's one or more than one.
Original point was certainly to point out the most common mistake of typical homegrown cryptosystems. Even programmers who know that XORing stuff with constant byte is not viable encryption can produce systems that do essentially the same thing (usually by XORing everything with RC4 output with long lived or even constant key)
And instead of saying nothing those who harp on "C's pitfalls" could politely enumerate those relevant to the issue at hand, that way we actually have something discrete to discuss.
I apologize if that sounds dismissive (it's not intended to) but your post is not more than the usual vague FUD. Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle.
Surely people who invent and implement cryptographic algorithms for a living can be expected to be better than your run-off-the-mill programmer, but as I've said elsewhere implementing crypto is also not a black art. It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system. That rules out many programmers but not all. That concerns implementations. As for algorithms, their development requires extensive experience in cryptanalysis (not just applied cryptography!). Some pros have that, others don't.
Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise...
As I said I'm not a fan of the phrase "don't roll your own crypto". I don't disagree that that (obviously!) prevents new crypto libraries being written. But if you're going to do it, you damn well do it right, and these days that means an amount of work that would likely total millions of dollars. If you aren't going to do it right, don't cut a 1.0 and use it in production.
Sidenote: how we started these things historically is not relevant anymore. We have learned a lot since then. Failing to "stand on the shoulders of giants" and learn from history is one the biggest factors that makes home-rolled crypto so easy to break - they are doomed to repeat the mistakes of history. Mistakes that are nowadays well documented and well known.
Having implemented crypto algorithms for educational and enjoyment purposes, I've always looked at "don't roll your own crypto" as less having to do with the actual writing of code which, as you state, isn't really that hard. Instead, what that statement is saying is that when you write your own code, you'll almost never get enough attention from the white-hat community to have any confidence that the code can be used in production.
You're right that the code written by professionals has lots of bugs. Bug is a term that describes a problem in code that has been found. With crypto software, the danger isn't the problems that get found (and then addressed), it's the problems that aren't found. "Roll your own" solutions have problems that never get found. That's the danger and it has nothing to do with the skill of the programmer.
The thing that normally nobody says, yet most the people telling you to not roll your own crypto assume you will understand is that you should rely only on well vetted libraries.
Rolling a usable crypto library is not a single person endeavor. Your own crypto won't be peer reviewed, thus you should not trust it.
> Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle.
This doesn't diminish the claim that the modal individual should not attempt to "roll their own crypto." You're pointing out that professionals make mistakes, but of course they do - that indicates nothing about the propensity for amateurs to make mistakes.
> It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system.
I have never implemented a low level network protocol or a file system (although at least the former sounds like a fun project) - so I cannot claim you're wrong here. However, I will point out that a network protocol is not designed to be error-proof in an intentionally antagonistic environment, which is a difficulty unique to cryptographic software. There are incentives involved in breaking crypto code that do not present themselves in other areas of software development, even if the programming difficulty itself is not entirely dissimilar.
> Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise...
They can start as side hobbies, but they are not going to remain that way for any meaningful amount of time if they are to be widely deployed and safe. You can only choose two out of those three.
More importantly, there is no cabal of cryptographers spreading FUD to line their own pockets with consulting fees. The cryptography consulting industry is very small compared to the actual security consulting industry. In the application security consulting industry I do believe there are firms that try to secure business this way, but that industry is much larger (and has firms which try to misrepresent cryptographic competency). I have had significant interactions with engineers at Riscure and NCC Crypto, which is a sizable portion of all the "real" cryptanalytic consulting work that occurs (at least in the United States) - they never struck me as being the sort to suggest what you're saying. That leaves Rambus/CR and a select few other firms doing real crypto consulting, which leads me to believe the industry is, on the whole, very legitimate.
> a network protocol is not designed to be error-proof in an intentionally antagonistic environment
Err, really? That may be the case sometimes. Also these days many things can be antagonistic: anything online is exposed to adversaries. So is anything that routinely reads untrusted input on everyone's computer (like a video player, or a pdf reader).
Thinking of the sheer amount of potentially vulnerable code out there makes me a little scared.
But is the code you're speaking of the security-specific code, or the network-specific code? If you attack a networking protocol, what are you attacking if not the associated cryptography? A networking protocol without any cryptography doesn't really need to be attacked because it's already wide open.
You introduce the confidentiality, authentication and integrity (which is really just going to be part of the authentication) through cryptography and security-specific code.
I'm not saying people don't attack networking protocols, or that there is no incentive for doing so. I'm saying that the portion that plays the defensive role is in the security software, not the overlying networking protocol.
Anything that accepts data from potentially-malicious sources is security-critical if it's itself privileged or produces data/events that go to something else privileged. Such analysis is usually used to determine the "Trusted Computing Base" (TCB) of a system that encompasses everything that might be used to break the security policy. All people who do real, security engineering keep it in mind, minimize it, strengthen it, and document what's left.
Here's an example from high-assurance field of what that looks like where they prototype an architecture for VPN's that minimizes TCB of key components:
Also shows their TCB is a lot bigger than their tiny kernel. Fortunately, there's been high-assurance developments to solve most of those things. I've seen robust, networking stacks but don't think they're high-assurance yet. If they're complex and done in C, here be dragons.
I was thinking of attacking the implementation of whatever processes attacker controlled input. If sending 100 pings in a row triggers a stack overflow, this will interest the attacker.
I would say that security specific code would be any code that directly processes the untrusted inputs. Parsers and decoders, mostly. You may want to isolate them from the rest of the program (ideally a separate address space), and make sure the interface between them and the rest of the program is much simpler than the input they are processing.
At least, you want to ensure such code either produces sound output for the rest of the program or fails cleanly. The attacker will have a harder time exploiting a bug deeper in to the program, I think —hope.
While I actually mostly agree with you, I feel I need to point this out... You write:
> I will point out that a network protocol is not designed to be error-proof in an intentionally antagonistic environment, which is a difficulty unique to cryptographic software.
But that's actually not true, at least not anymore. People attack protocols on the internet all the time, and it's a difficult task to harden them against that.
That said, it doesn't really take aware from your point, I think; but rather confirms it: Think twice before rolling your own network protocols (at least if they are meant to be used in large scale on the open internet), be aware of all the complicated pitfalls.
> This doesn't diminish the claim that the modal individual should not attempt to "roll their own crypto."
Just wanted to point out that "individual-should-not-attempt-to-roll-their-own-crypto" is ambiguous and unnecessarily antagonistic while the paragraph quoted above from Schneier is clear and difficult to argue with.
"As simple as possible but no simpler" applies quite well to summaries, esp. when they regard cryptography.
Okay, what exactly is the claim you'd like me to argue with? That Bruce Schneier was a hobbyist when he developed any of his cryptographic algorithms? He wasn't, and more importantly that misses the point since those algorithms were widely reviewed in competition settings.
I'm not sure what you're getting at here. Yes, someone has to develop cryptography, just like someone has to work on designing rockets. Theoretically everyone begins an aspiration as a hobby, but that's pedantic. A cryptographic design or implementation is not going to survive as a side hobby regardless of how it begins.
There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items.
Our "field" of programming is so weak in its preservation of institutional knowledge, that we have to occasionally fight the recurrence of bad ideas about very basic things, like using string filtering to avoid SQL injection. Yes, this happens, even in "hip" new parts of the field, like the communities around newer languages. (I have a specific example in mind, but I don't want to deal with the resulting war.)
I would have to read their source code to answer that question (I know almost nothing about 1Password). Some password managers just call a PGP executable. Others are assembling crypto primitives into larger pieces and making choices like "let's use CBC mode" themselves.
Anyone who can write code can write a crypto library.
The trick is to have a second person helping you, who has all your mathematics notes, a copy of your source code, physical access to your hardware, and the ability to run your program inside a debugger. You don't even get to know whether they put cream in their coffee.
If you can keep a secret from them, you're doing okay. So then you rent a smarter person to do the same thing to make sure all the tricks and shortcuts have been attempted.
If someone manages to break it in the wild after that, you're still doing about as well as some of the highest-paid professionals in the industry.
I generally also think that it's a bad idea to do, but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything.
It means that normal citizens won't be completely helpless against tyrannic governments and it helps to educate that writing crypto from scratch is most definitely something that criminals can do. The government making it illegal or backdooring it will only help against lowest-effort criminals, not against organized terrorist groups. Sure, those probably won't write unbreakable encryption, but it'll be enough to bypass a government that's expecting everything to be in plain text.
> but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything.
On the contrary, this is -exactly- why you don't want to roll your own crypto. Because they can and will break it. Good crypto takes serious thought and effort from people who put a lot of thought and effort into it.
You can do crypto as a side-hobby and safely put it into production. Of course, you can also do it wrong, but it is with little work possible to do it correctly.
Normally I wouldn't speak against what you said because the critique is only very minor, but I heard this too often. Your warning is too strong. People should try their own crypto and with little care it is not unsafer. Maybe giving a list of do's for that would be a good start.
> You can do crypto as a side-hobby and safely put it into production.
No, you really can't, unless you stretch the definition of "side-hobby" to the point of breaking at the seams. I'd be shocked if a single person who professionally works in security or cryptography is going to agree with you in this thread. You need far more than a "little care" to assure that a new cryptographic library is safe. It's an endeavor suited to a collaborative academic or corporate environment, with robust human capital and strong oversight.
I'm willing to accept that someone can safely implement a primitive or a construction in a new library with significant time, effort and feedback from others. But that level of effort doesn't really qualify as a side-hobby, and until the implementation was formally checked by professional cryptographers I'd consider it suspect.
This is to say nothing of designing novel primitives or constructions, which I would consider as far removed from a "side-hobby" as a local 5K is to the Olympics.
However, I absolutely think people should attempt to implement their own cryptography if they're curious about it and want to learn. Just don't use it in production and assume it's unsafe. People who work in the industry don't make these claims because we think it's fun, we do it because we've all seen the consequences.
We are talking in vague terms, but assume you want to encrypt some offline data.
- You can just use any recommended (by whom?) tool.
- Or you encrypt it with recommended tool with it's own dedicated key. Then you encrypt it again by whatever way you made up in five minutes, and you need only to ensure you didn't use the key material of the recommended tool. Then there you go, you just rolled your own not-unsafer crypto.
That is not rolling your own crypto. Rolling your own would be relying on your custom construction, not double encrypting and relying on the known construction for safety.
...Furthermore I'm confused as to why someone would do that. If you already have a safe way to encrypt your plaintext and you're the only one with both keys, why would you bother double encrypting? You gain absolutely nothing because it's redundant, and you lose performance.
What you're doing is tantamount to a weird sort of encoding that doesn't offer any benefit.
If a vulnerability is found for the outer encryption method, additional encryption of the inner message by a different method may provide some defence.
If that is the goal, though, it would be better to use two well studied encryption methods rather than something homemade.
Right, that observation is correct on the surface. But the reason why that's almost never done is because the goal of a cryptographic algorithm is to contribute enough safety margin on its own. Instead of encrypting twice, it's better to encrypt with a greater number of rounds, or to come up with a superior algorithm. In practice you sacrifice an unreasonable amount of performance double encrypting in a production environment for a threat model that is fantastically unrealistic.
Depending on the algorithm: yes. I'm not saying all algorihms that are widely used and well known are extremely unlikely to be broken.
I would happily bet $10,000 that AES will not be broken in the next ten years. I'd make the same bet for several hash functions.
And as a corollary to this point: I think it's incredibly foolish to try and combat the threat of an encryption algorithm being broken by double encrypting a plaintext, including with that algorithm.
I wouldn't take that bet either as I agree that is a very unlikely thing to happen.
Note, though, that you are using the specific word 'algorithm', where as I am talking about 'methods'. Most cryptographic failures are in how the algorithms are implemented or applied, not a problem in the underlying maths.
I would not be nearly so confident about a similar bet that applied to the actual code being widely used for encryption.
You'd lose the bet so hard. Don't take anything for granted. Even the most supposedly secure and widely used primitives should be scrutinized and are subjects to constant attacks.
Read that paper again. Cache timing attacks (and more generally timing attacks) are a subset of side channel attacks, which are not a break in the fundamental design of a cryptographic algorithm. When Bernstein mentions that he considers it a design flaw, that's a misnomer in the mathematical sense: he means that AES is antagonistic to a software implementation that can resist a timing attack, not that the design of the algorithm is fundamentally unsound. This is essentially true for all encryption schemes that utilize lookup tables and S-boxes, which he even mentions in the paper.
This is similar to using acoustic analysis or fault injection for key retrieval - yes, you've done it, but it's not really fair to say you've broken AES whatsoever. That remains a pipe dream. You've merely broken an AES implementation on a particular platform that was not hardened for this threat. Just a few months ago a team came up with a way to retrieve an AES key through acoustic analysis in a paper that was published here on HN. That's not actually the same as a cryptanalysis, the best of which is this: https://eprint.iacr.org/2009/317
There are effective CPU mitigations that allow you to safely implement algorithms in order to avoid this issue - for example, AES-NI theoretically mitigates cache timing attacks. See: http://cseweb.ucsd.edu/~hovav/dist/aes_cache.pdf
I know this is a side channel attack. I know people have used acoustic analysis to retrieve 4096 bit RSA keys, but the attack outlined in the paper is more feasible in real life situations, and systems do get compromised due to implementation details. AES might not be broken as a whole but that doesn't mean you can't attack specific usages.
AES isn't proven to be secure. I don't know about $10k, but I would not bet my life on any hash function not being broken.
That's not an (algorithmic) break, but a (side-channel) attack on some implementation. "Break" is precise jargon here; you're right that a particular real-world implementation of a strong algorithm may be weak, but that doesn't contradict anything dsacco said.
Doing your own encryption does clearly not exclude also using other encryption. Or is implementing AES already not anymore doing your own encryption, because AES wasn't invented by you? You _always_ use work of others.
Doing your own crypto can be used as way to safeguard against unknown vulnerabilities in the tools you would use otherwise. If the recommended library is safe, then fine, you are good to go. How do we know it is safe?
My idea above is a way to increase the safety margin in a way that is orthogonal to the increase of the key space of the recommended tool. That sounds like a very good thing to me.
I'm sorry, I don't understand what you're trying to say. Can you clarify?
What I am trying to say is that you gain nothing by, say, implementing your own encryption algorithm, using it on a plaintext, then encrypting the resulting ciphertext with a known good implementation of a known good algorithm like AES. That is how I interpreted your previous comment.
So - you don't gain anything from that, if that's what you meant. If you mean just implementing a known good algorithm on your own (again, like AES) - why? You are overwhelmingly more likely to shoot yourself in the foot, and there are plenty of open source implementations that you can inspect if you don't trust the code.
Finally, if you distrust the design of a known good algorithm, you're out of luck, because (again) you're almost certainly not going to come up with something better. If you do, by all means use it, but it's simply not likely whatsoever.
And again: you're not increasing the safety margin by double encrypting, unless you know both algorithms and implementations are solid. And even if they both are, the safety margin improvement will be negligible compared to the decrease in performance.
> "If you mean just implementing a known good algorithm on your own (again, like AES) - why? You are overwhelmingly more likely to shoot yourself in the foot"
I've never understood this argument. Encryption algorithms are deterministic. If you implement a known good algorithm on your own, and it gets the same outputs given the same inputs, then you are highly unlikely to shoot yourself in the foot. This is not what people are talking about when they warn not to roll your own crypto.
In cryptography, the output of an algorithm is far less important than how that output was generated (with very precise specificity). Crypto code requires far more heavy lifting in the "how" of the output than other software does. This is not intuitive, because ordinarily there are several ways to do things and you can compare on output, and it's probably fine to develop something in a functional but unorthodox way (minus performance consequences, maybe).
As the parent comment here mentions, there are extremely bad consequences to subtle mistakes in how the algorithm and its implementation arrives at the output, even if the output is correct.
If Alice sends Bob an encrypted message and the custom crypto provides an incorrect output, Bob can't decrypt the message. But if Alice sends Bob an encrypted message with the correct output generated in an unsafe way, there are many ways by which the secret key could theoretically be recovered, which is far worse than Bob not being able to read the message in the first place.
> "Just because an AES implementation matches the test vectors does not make it correct or safe."
Actually, this does make it correct. Whether or not it is safe depends on the application. For example, I sent my friend Bob some cyphertext that I calculated by hand with a pencil and paper using the AES algorithm. I sent it via my trusted courier Eve. It took four hours for me to do the calculation. At the last minute I started to second guess my math so I double checked it against a respected crypto library. It was fine, so I handed it off to Eve. I am pretty sure that in this application my choice to calculate the answer by hand was exactly as safe as if I had just used the library. In fact, I am so sure (given that the answers were identical), that just as Eve was walking out the door, enroute to Bob, I pulled her close and whispered in her ear: "Eve, be very careful with this cyphertext, it took me four hours to create it."
...what? Yes, if you compute by hand and compare with a known implementation, than it's likely you computation is correct. But this has nothing to do with test vectors. You could match all test vectors while still giving incorrect results for values not in the test vectors.
But there's more to cryptography than just inputs and outputs.
Like, let's say we have two functions which validate a password.
Function A takes a user-defined parameter, compares it to a stored value byte by byte, and as soon as the strings differ, it exits.
Function B takes a user-defined parameter, compares it to a stored value byte by byte, sets a flag if the strings differ, and exits when all bytes have been compared.
Both of those functions take the same input, and both of them return the same output. Are you prepared to tell me that the two functions are equally secure?
> Encryption algorithms are deterministic. If you implement a known good algorithm on your own, and it gets the same outputs given the same inputs, then you are highly unlikely to shoot yourself in the foot. This is not what people are talking about when they warn not to roll your own crypto.
That is exactly what people mean by "don't roll your own crypto." If your belief is that you only need to match the same output per input, then your implementation might be an oracle that leaks information.
Crypto people don't say this for job security or to feel important. They say it, because they, themselves, have made mistakes, and they constantly see examples of programs that trivialize both security and cryptography while simultaneously introducing foreseeable bugs and obviously weak crypto.
They are deterministic, but timing attacks are a big deal. There were lots of practical key extraction attacks on RSA with nothing but timing, even on a network.
You can also do things like forget input validation in ways that reveal key material, or reuse a nonce or IV in a way that breaks the whole system.
Sorry, but this is not good advice. Widely-used crypto implementations have had the benefit of years of analysis by dozens of high-expertise stakeholders who have a lot to lose should the crypto fail. Even that isn't always enough to catch all weaknesses and vulnerabilities.
This cowboy-programming attitude being extended to security is no small part of why we are so vulnerable as a society to attacks on our computer systems that can compromise our core infrastructure[1], our secrets[2], and our economic security[3].
[1] Think weeks-long country-wide power outages
[2] Think juicy blackmail material on anyone with their hands on levers of power. Look at the damage that North Korea's attacks did to Sony Pictures, for example.
[3] Think small-scale attacks on services that even a single major company, or many small companies, rely on.
> Of course, you can also do it wrong, but it is with little work possible to do it correctly.
GCHQ loves people who think this.
Have a look at some of the game console cryptography. These are companies with strong financial incentive to get it right. They're large well funded multinational organisations. They still get it wrong.
Game companies are also a great example of a situation where a large group of people with time on their hands (teenagers still in high school) are motivated to look for vulnerabilities in the method used.
Have any studies been done on relative security of crypto algorithms that are either:
a) Well known, well studied but also attractive targets for attackers to study
b) Unknown (aside from the developer) until an attacker encounters a specific piece of encrypted data
It is a common assumption that well-known methods are better (and it is the assumption I work under) but does empirical data on security breaches back that up? There are plenty of examples of security breaches where 'standard' methods were being used. Are there similar examples where people using previously unknown methods have been compromised?
GCHQ and others invest a huge amount of resources in finding vulnerabilities in well known encryption methods. When they find one, everyone who used that method is vulnerable.
I have no doubt that if they really wanted a piece of data that I had encrypted with a homemade method, they would be able to break it.
However, are they going to invest the resources to do that if I am not being specifically targeted? Are they going to invest the resources to crack hundreds of different people's home-made encryption methods? Thousands? Hundreds of thousands?
If am being specifically targeted by something like GCHQ, they will get what they want one way or another.
I'm not aware of academic studies on the subject (I think that would be hard to do, because it's virtually impossible to know how many proprietary algorithms exist, how many are trivially breakable and how many have been broken).
However, this point is featured as 101 material in basically every cryptographic textbook. To put it very succinctly: there are conditions in which it can be beneficial to use proprietary cryptography, especially when you require very unique interoperability constraints. However it is almost never a benefit for the safety of the algorithm.
I've come across a proprietary algorithm and successfully broken it, in a black box setting, with differential cryptanalysis. This algorithm was deployed to disguise the sequential order numbers for a very large delivery company. It took me about a month, but it was done. The challenge in proprietary algorithms is shifted to figuring out what's going on because it's unrecognizable. That is a significantly easier challenge that identifying a vulnerability in an algorithm like AES, which has never had a meaningful vulnerability in a decade and a half of cryptanalysis.
If you use a proprietary algorithm it might be safer than a known unsafe open algorithm, but it's virtually guaranteed to be worse than widely studied algorithms, and most likely in a trivially breakable way. They can be safe, but that still means you're going to be working with professional cryptographers at a company like Riscure to assure it's safe.
I am aware of the usual (and strong in my opinion) argument.
Every time this discussion does the rounds, though, I do wonder whether the hypothesis could be tested.
Most vulnerabilities do not come from breaking the core algorithm but rather from a flaw in how they are implemented or applied. Standardisation can lead to monocultures that become tempting targets for those with plenty of resources to throw at them.
That's what leads me to being reasonably sure the hypothesis is valid.
As a scientist, though, I'm always going to wonder whether there is a way to subject it to a proper test rather than just relying on opinion (no matter how much I respect those opinions)
It's not thought policing. Would you consider us adamantly telling you not to develop and deploy your own rockets or medical software without significant expertise and third party review to be "thought policing"?
Furthermore, your argument is fallacious. That expert professionals make mistakes does not tell you anything about the likelihood of an amateur to make a mistake. You can't draw any logical conclusion from that statement on its own.
That's great, but that's not what I'm telling you. I'm telling you not to develop and deploy them with human passengers, in a production environment, on a trip to orbit. If you get collaboration from professionals with significant expertise and audit the design and development extensively, then sure, go for it.
Even were that as accessible as developing and deploying your own cryptography, I'd still advise against it.
If you roll your own crypto and say, "that was a fun learning experience" and write a blog post about what you learned implementing AES in some novel way, nobody is going to show up and tell you not to roll your own crypto. If you roll your own crypto and then deploy it in a production system, then you are putting actual users at risk.
> There is cult around "Don't roll your own crypto" that is thought-policing
You know who else gets "thought policed"? The people who actually need to rely on cryptography. In their case it isn't grumpy people on Internet message boards telling them not to do something, it's agents of the state killing them or imprisoning them for many years.
It was a fun read, except when all the C bugs were listed. It's getting tiresome to hear about folks writing nominally
Important software in a language where it is extraordinarily difficult to get things absolutely correct, with paltry excuses such as "it needs to be fast" or "it needs to be portable [0] to a VAX-11/750." It doesn't give me confidence that these completely usual bugs popped up early on, and it will not surprise me when more show themselves.
The author's lack of practical familiarity with the memory hierarchy, caching, and pipelining—evidenced by his surprise by the speed gained by fetching more than a byte at a time—also makes me suspicious of his understanding of the semantics of his program. (I understand that this is a form of ad hominem as it relates to arguing the correctness of software, but it's a consideration worth noting in a broader context of trust.)
It would be great if this was written in another language.
[0] Most "portable" code written in C is not portable. Look at any mature C project and see the layers and layers of macros and hacks to make things portable.
> The author's lack of practical familiarity with the memory hierarchy, caching, and pipelining
Maybe I didn't express myself clearly. I knew loading stuff word by word would be much faster, thanks to my knowledge of the memory hierarchy, caching, and pipelining.
What I didn't expect was the magnitude of the overall impact. The core round functions are much, much faster that I had anticipated. I didn't think the loading code, even in its slow version, would account for a significant percentage of the total running time.
> It would be great if this was written in another language.
Easier said than done. I aim for easy integration in projects written in basically anything: C, C++, OCaml, Rust, Python… What would you suggest? Rust? That's a possibility, but (i) it's LLVM only at the moment, and (ii) I believe there's an impedance mismatch between Rust and a C compatible ABI.
I would totally recommend to rewrite the whole thing in Rust for Rust projects, though.
One of the major Rust design goals is simple C ABI compatibility, and it delivers there. That it's LLVM only appears to me irrelevant, as the generated libraries can be linked anywhere, and the specifics of how the Rust compiler are built are out of scope. So long as you get rustc up and running, what does it matter it's architecture?
> I would totally recommend to rewrite the whole thing in Rust for Rust projects, though.
Any time you rewrite anything, you run the risk of introducing new and unexpected bugs, and you're just doing unnecessary, complex work fraught with pitfalls (doubly so here) -- generally a Rust project would create a wrapper around the C code, invoke the methods and expose idiomatic APIs on top to avoid just that.
> One of the major Rust design goals is simple C ABI compatibility, and it delivers there.
I was just telling one has to write a wrapper either way. Idiomatic Rust is not idiomatic C, so you would need to have a "C ABI" interface in addition to the clean Rust interface.
> That it's LLVM only appears to me irrelevant […]
Not if it targets less platforms than GCC. Though that will likely matter less and less in the future.
> So long as you get rustc up and running, what does it matter it's architecture?
You have to get it up and running. You have another build step. You have to use a binary library (can't just include the source code). It's not much, but I want to maximise ease of deployment.
It's even more tiresome hearing a new generation of programmers constantly lamenting the fact that C
requires a great deal of skill to develop in and that
X or Y new language completely avoids C pitfalls.
Not only is it dishonest but it seems to be marketing 70%
of the time.
Portability does not equal 'without blemishes'. It simply
means that when compiled across platforms it will perform
as expected.
I like to think that I'm rather neutral, since I'm neither pro-C or against-C.
But from a rather philosophical point of view, C is basically Portability 1.0, it was one of the first truly portable "fast" languages. I'd like to think that we could do better than C with 40 years or CS research and industry experience.
Of course, there's the whole second-system syndrome so it's a wicked problem :)
> that X or Y new language completely avoids C pitfalls
Yeah, you don't even need a new, hip language to avoid C's pitfalls.
> Portability does not equal 'without blemishes'. It simply means that when compiled across platforms it will perform as expected.
By that measure, C is not portable, since it will behave differently across multiple platforms – or multiple compilers, multiple compiler versions, even multiple optimization settings.
The quotes I selected do not change the context, nor do they feign a different context. Unfortunately, since you replied with nothing but a snarky comment, I do not know in what way you think I misrepresented your arguments.
We've had to wait till C99 to get stdint.h, clearly portability is kind of an afterthought in C.
C is meant to map easily to a wide range of hardware without overhead, conceptually it's almost the opposite of portability since it precludes creating standard abstractions that would hold true across architectures as those could not map to native functionality across the board.
It's easy to compile C programs on a different architecture however, which is probably where this argument stems from. Remember when we switched from 32 to 64bits? How many codebases managed to make the jump painlessly?
I can't edit this comment anymore but I'm replying to myself to point out this little bit of C "trivia": the printf function itself, probably the most iconic standard function of all, is not portable.
The problem is that it returns an "int" which contains a negative value on errors and the number of character written on success.
Of course that means that it's possible to write a string whose size doesn't fit in an int.
For instance if you have a 64bit computer (and enough RAM) you can try the following code:
Don't forget to redirect stdout to /dev/null. You can pipe through `wc -c` if you want to make sure the billion 'a's are printed.
On my computer the "printf" manages to print the whole string successfully but it returns... -1, signaling an error.
The size shouldn't be returned that way, it should be a "size_t" as an output parameter (since you also need a special value for errors). In Rust you'd use "Result<usize>" and you'd have a more elegant solution. Of course you can't really do that in C (except by returning a struct, which is a bit cumbersome) so I guess they said "meh, who prints that much text anyway?" and assumed an int would be good enough.
> We've had to wait till C99 to get stdint.h, clearly portability is kind of an afterthought in C.
C provided "at least N bits" guarantees for its integer types since its first standard (for the curious: char at least 8-bit, short at least 16-bit, long at least 32-bit; in C99, long long as least 64-bit). This is the most that you can get if you want absolute portability, because there are architectures out there where you simply can't address memory in, say, 8-bit chunks (see SHARC for an example).
Consequently, while stdint.h defines types like int32_t, they are "conditionally supported", and so a C program that has a hard dependency on them is not universally portable. Types like int32_least_t and int32_fast_t are unconditionally supported, but they don't give you anything that you didn't have before.
Sure but as as said in an other comment in this thread those "at least N bits" guarantees are not very practical.
Suppose you want to give a unique id to users of your application. Clearly 16bit is a bit small, you might get more than 65535 users. You'll probably want 32bits to be comfortable (unless you're google or facebook I guess).
So what type do you use? On any modern 32 or 64bit computer "unsigned int" would do the trick but if you want to be portable then you can't assume that "unsigned int" is more than 16 bits. So you have to use long instead.
But then that makes your type 64bits on modern computers. Completely overkill. If this id is used all over the place it will significantly increase the memory footprint and could hurt performance as well.
In truth when you write an application you never really care about the size of the CPU registers outside of device drivers. What you want to say is "I have this value that I estimate will max out at N, find me the fastest type where that fits".
So in the end, pre-stdint everybody would use machine-specific typedefs to do just that, portability be damned. In particular any big C library ended up doing that (glib has "gint8", "gint16", Qt has "qint8", "qint16, etc, libpng has "png_uint_16", "png_uint_32", etc...). Clearly people were not fine just using the minimum requirements of the standard.
>Types like int32_least_t and int32_fast_t are unconditionally supported, but they don't give you anything that you didn't have before.
But they do! Pre-C99 there's no way to say "I want a type that's at least 32bits and as fast as possible" or "I want a type that's at least 32bits but as small as possible". "int" could potentially be 16bits and "long" could be unnecessarily large, wasting RAM and cache. The only way to have this functionality was to use architecture-specific definitions.
> But they do! Pre-C99 there's no way to say "I want a type that's at least 32bits and as fast as possible" or "I want a type that's at least 32bits but as small as possible".
I stand corrected on that point.
> Clearly people were not fine just using the minimum requirements of the standard.
Most of these typedefs are used to interop with some other standard or file format that requires that many bits exactly.
But even then I would argue that there was an overuse of such typedefs in C. I can think of several reasons for that:
First, there's a general lack of awareness that C does in fact provide "at least N bits" guarantee for most of its types. This comes up pretty often in questions on StackOverflow, and apparently there are still enough people surprised by the fact that e.g. long can never be 16-bit.
Partly, I think, it's because of the historical mess with int. Obviously, int is what people use by default, and the fact that it was 16-bit on many popular architectures in the past, and then became 32-bit on their successors, resulted in much broken code. I think that the lesson many took away from this is that all integer types in C are similarly broken, and switched to int32_t (or equivalent typedefs) just to avoid having to think about it.
Partly it's because of 64-bit ints. Standard C didn't have a 64-bit type for a long time. Popular implementations provided it as an extension, but in different ways - long long in most Unix compilers, __int64 in 32-bit DOS and Windows compilers. So if you ever needed 64 bits, you needed a typedef for that. Since most devs work on systems where word sizes are 8/16/32/64, they did the simplest and most obvious thing, and defined something like int64_t (as opposed to int64_least_t) - it just didn't occur to them that something beyond that was needed. And that set a precedent.
Then there was the part where long was made 64-bit in LP64 model, which meant that portable C had no "at least 32 bits, but no longer than it needs to be" type until C99. With the precedent established by int64_t, the logical step was to add int32_t, and int16_t as well for good measure.
And that's how we got where we are today. I would still argue that most of those libraries that use int32_t and similar types don't strictly need it. It's just a combination of historical factors, and also the fact that platforms where such types do exist comprise the vast majority of platforms, and all platforms that people care about. So there's no particular incentive to even think about how you'd write code for something else - why deal with the extra complexity, if you can just wave it off? Can't blame anyone for that.
And to me this shows clearly that we shouldn't be using C then.
If an arch can't address an 8 bit sized element there are two choices for what happens if you write:
char *a = "test";
char x = a[1];
Either char takes 32bytes or the a[1] read will shuffle the bytes to get a 32bit value whose least significant digits are a[1]
And saying that int32_t are "conditionally supported" is really proof we're making C worry about 8-bit microcontrolers when we're doing big application with it, it's trying too hard
On SHARC, char is 32-bit (so is short, int and long).
I don't see why it's a problem, though? As noted earlier, C accommodates that. Sure, there's a lot of C code that's written with the assumption that chars are always 8-bit, because they usually are. But it's definitely possible to write portable C code that doesn't make such assumptions.
int32_t is required to be supported on any architecture on which one of the standard types is exactly 32-bit. Since long is required to be at least 32-bit, and any 8-bit microcontroller would have to support long, it would be 32-bit as well, and so it'd have int32_t (typedef'd to long). The only case where you wouldn't have it is either if the architecture has a word larger than 32 bits as its smallest addressable unit, or if it has non-8-bit bytes.
In practice, this is really not an issue, because you don't have to target the absolute most portable subset of C. The reason for making those features conditionally supported is so that when they are there, they behave the same. So you can say "my code runs on any platform that supports ISO C, and has int32_t in stdint.h". Not only that, but you can enforce it at compile-time with clear error messages, e.g.:
Your code is still portable. It's less portable than the mandatory subset of C, but it's a strictly defined part of the spec that behaves the same on all platforms that support that part.
Arguing that C shouldn't support microcontrollers seems a bit unfair. What should embedded software engineers use, if not "high level assembler"? There are plenty of modern languages that sacrifice performance in order to be fully hardware agnostic.
I agree that the jab at microcontrollers is unwarranted (after all, I do want to be able to program microcontrollers in Rust!)
But you could still have any-bit integers even on an 8 bit processor, you'd just have to emulate them.
Clearly that's not really in the spirit of C which is probably why the standard doesn't force the compiler to implement those, instead relying on the coder to make a decision.
But that just proves my original point: C is not about portability. It's about writing fast code on any architecture. If portability was the concern the standard would make fixed-width integers mandatory and would define things like signed overflow, forcing the compilers to emulate the behavior when the hardware doesn't behave in the same way.
That's what portable languages do, that's what Rust does.
> Pretty sure 'char' in this case would be 32 bits.
On ARM it is 8 bit + manipulation of value
> Arguing that C shouldn't support microcontrollers seems a bit unfair
No, I'm saying the opposite
What I want to say is that because it supports a big variety of processors it tries to do too much AND that our focus should be in more expressive languages in more powerful machines
C is great for a microcontroller with limited resources and inputs
Indeed; the language is portable. Programs written in the language require great effort to be themselves portable.
Also, portability isn't a binary attribute. For every way two platforms vary, there are often a subset of states and transitions that are identical, and two disjoint subsets that are different.
As long as your inputs start out in the identical subset and don't wander too far, you're good to go. For example, if you never put a value above 32767 into an int, you don't really care whether your int is 16 or 32 bits in size. If that int is e.g. line number in a file, it may mean that your program works fine on most files, but only the 32-bit version works correctly on large files. And of course the cleverer you are the closer you're likely to skate to the edge of the common intersection.
Relying on a macro processor to patch and bandage your program at compile time depending on a collection of symbolic features isn't really representative of portability of a language and your code. To me, that is brute forcing portability, and is a web of conditional inclusions/exclusions that needs to be maintained and added to per-architecture and per-OS.
Something that is, in some sense, truly portable is one that has a single specification that runs on all platforms the base implementation supports. Virtual machines and certain styles of language standard are often able to accomplish portability for a vast number of applications.
I would argue the other way. Something which requires a base implementation to be installed on a system which generates different machine code based on a single specification is less portable than a macro processor generating different versions of a program. Sure, people will have had something like python installed since a long time ago, but a new setup requires downloading 2 objects (the base implementation and the program) compared to a single one (a platform-specific copy of a program).
You don't need a C compiler to run an already compiled program. You may need libraries, just like almost every language, but only the developer needs to download the compiler and not every single user.
It gives options to the developer to target multiple platforms without extra effort from the users of those platforms.
> Relying on a macro processor [...] isn't really representative of portability of a language and your code.
I would argue that the C preprocessor is, for all intents and purposes, part of the C language. C is not quite practical without its preprocessor, and the preprocessor is not quite practical with any language other than C (or a derivative like C++).
The preprocessor does not add portability. It adds a path for you, the programmer, to succinctly specify additions and deletions from your program so as to make it portable.
This is markedly different from a language that, for example, guarantees the existence of integers of unbounded size. In this case, when working with integers, we make no mention of the platform to execute on. We are provided with an abstract guarantee that one can rely on regardless of the concrete platform.
Portability is not black or white. Macros are essentially selecting different code depending on the platform. A good high-level language would not need to do this.
Yes but this is a separation of concerns. This is precisely the point of an abstraction, especially at the level of a language and its implementations.
Writing code in Common Lisp means you're writing something with semantics promised by the Common Lisp specification. Vendors implement the spec (perhaps using nasty #ifdefs), and developers write code against the spec. The spec is a contract, and from the point of view of the programmer, it's indifferent to platform change and so on (up to what the spec says is platform-dependent, like machine integer size or maximum array length).
Just because a particular implementation may not be considered portable, doesn't mean my program (which I may choose to run on that implementation) isn't portable.
Sticking to the spec is one way to be portable. Building a portability layer is another one; this is how Common Lisp grows, by taking advantage of the work of different implementations and using standardized tools (FEATURES, reader conditionals) to build a common interface. See e.g. CFFI which has become the (defacto) standard for loading C libraries (https://common-lisp.net/project/cffi/spec/cffi-sys-spec.html...).
Unfortunately the architectural details of the machine will leak through to the program you are writing, and you will need to twist your code in order to get correct behaviour and fast execution.
Case in point is the recent article posted on HN about the platform specific details of getTimeMilliseconds() in Java.
The issue here is handling portability yourself or let the language and its toolchain do it. The odds that you are doing a better job than the community to handle portability are pretty low right from the start and they get lower and lower if the laguage is well-supported.
My "ugly macros" are tested against my own implementation only. Compiler's are tested against thousands.
> It was a fun read, except when all the C bugs were listed. It's getting tiresome to hear about folks writing nominally Important software in a language where it is extraordinarily difficult to get things absolutely correct, with paltry excuses such as "it needs to be fast" or "it needs to be portable [0] to a VAX-11/750."
Are there any practical alternatives when that kind of portability is needed?
What about transpiling to C? I know Haskell and OCaml can be made to compile their code into C code that you can then pass on to a C compiler.
1. Manage resources with RAII (though, in C you could use non-portable attribute cleanup in similar manner).
2. Use type safe wrappers around builtin types like in [0] and [1].
3. Use containers with more extensive bounds checking and iteration validity checks [1].
This could help detect some of those bugs at runtime, or even possibly prevent them from being written in the first place.
Use of uninitialized value could be prevented with types that require explicit initialization or have default one (Bug 1).
Left shifting a negative value could have been caught at runtime (Bug 2).
Bounds check could prevent uninitialized memory read (Bug 3)
Type safe wrappers could prevent an accidental promotion from being written in the first place (Bug 5).
Though, you need to go out of your way to actually do all those things, not to mention that your code would integrate poorly with existing library ecosystem.
IMHO choosing C++ alone doesn't improve safety of your programs compared to ones written in C all that much.
It's easier to write safer code in C++ but there's still many pitfalls and it's still a lot of effort. There's really nothing even remotely close to being practical?
C adds a layer between your code and the system. It abstracts away memory layout, locations, as well as function call semantics, and a many other things. For example, given a C program, you will not be able to tell me with certainty that a variable will occupy memory on the stack, any of your particular processor registers, etc. Formally, C is a language to control the C Abstract Machine, as described in the standard.
The sort of direct control one might look for is, as far as I know, only practically found in your processor's assembly language, and your operating system's collection of syscalls.
I think Rust, Common Lisp, OCaml, D, Haskell, and other languages give you a lot of benefits that are simply not worth giving up in a new crypto library, like safety.
I think writing timing-attack resistant code in Haskell would be very hard to impossible, at least without writing very unidiomatic code (basically "C in Haskell"). I'm happy to be proven wrong, though.
Honest question: What makes you think you can't mitigate? A̶ ̶t̶y̶p̶e̶s̶a̶f̶e̶ ̶c̶r̶y̶p̶t̶o̶ ̶t̶h̶a̶t̶ ̶m̶e̶a̶s̶u̶r̶e̶s̶ ̶i̶t̶s̶ ̶e̶x̶e̶c̶u̶t̶i̶o̶n̶ ̶t̶i̶m̶e̶ ̶a̶p̶p̶e̶n̶d̶e̶d̶ ̶w̶i̶t̶h̶ ̶a̶ ̶f̶i̶n̶a̶l̶ ̶d̶e̶l̶a̶y̶ ̶t̶i̶m̶e̶ ̶r̶e̶a̶d̶ ̶f̶r̶o̶m̶ ̶/̶d̶e̶v̶/̶u̶r̶a̶n̶d̶o̶m̶ ̶c̶a̶n̶ ̶s̶t̶i̶l̶l̶ ̶b̶e̶ ̶i̶d̶i̶o̶m̶a̶t̶i̶c̶.̶ It has to be monadic, that is for sure but abstracting a crypto algorithm as IO is actually treating it as a device which sounds like a safe metaphor.
I believe padding the timing always has been considered a bad idea. If you want to guarantee you won't exceed the timing, you probably have to take so much margin that your primitive becomes prohibitively slow. And I'm not even sure it would protect you against cache timing attacks.
Simply put, there are easier and cheaper ways to guarantee timings. Perhaps Haskell is even capable of harnessing them. (I'm thinking of the usual "no secret dependent branches", and "no secret dependent indices".)
Random delays do not protect against timing attacks. Best thing it can do, is to force the attacker to collect a few more timing samples. (To the point where the the random noise averages itself out.)
Rust doesn't even exist on a plethora of platforms which C targets. I have high hopes with D and give or take ~2 years, it'll be able to progress forward in lots of domains. Not sure about other languages.
I think it might be a bit more than that. As far as I can see, all the primitive data types in Rust have sizes which are multiples of eight. However, there are still architectures out there with 9 bit bytes, or 36-bit words. Maybe it's good that all the primitive data types have an explicit size (encoded in their name), but this comes at a cost (in case a byte is 9 bits long, one of the bits will effectively not be used). On the other hand, C and C++ are less explicit about the sizes of their data types so they can accommodate these more weird architectures more easily (and, if you need an explicit size, you can explicitly ask for it).
You have a point but in general you wouldn't write "generic" C code on these exotic (by modern standards) architectures. So I guess if you wanted to port Rust to these systems you could introduce an i9 and an i36 and use that instead. I doubt you could get firefox and its dependencies to work on a CHAR_BIT == 9 system without some heavy modifications. I could be wrong though, it's not like I tried.
In my experience most modern codebases tend to target POSIX systems (explicitly or implicitly). That gives you a subset of C to work with and makes your life easier. The exceptions are things like DSPs which can have rather non-standard layouts but you generally write very specific and unportable code for these anyway.
Of course if you want to be completely portable you can't assume that CHAR_BIT == 8 or that you can convert back and forth between function and data pointers but in practice a lot of software relies on this.
Even before stdint became part of the standard it was very common for programs and libraries to typedef their own "sized" integer types. It's the only reasonable way to deal with ints in C IMO, otherwise you can't assume that your ints are greater than 16bits (which is probably too tiny for many uses) so you're tempted to put "longs" everywhere. But then longs will be 64bits on modern architectures which could be overkill and reduce performance.
So in theory those variable-size integer types are interesting but in practice they're basically useless because you end up making assumptions, one way or an other. I think it was a good idea for Rust to get rid of them.
What? You think other languages don't have "fan-people"? People who like Rust tend to like it very much, and it does provide a real solution to problems that C and C++ create, while providing a no-compromise performance story. So why wouldn't people talk about it?
Perl had evangelists, Ruby (on Rails) had evangelists, C++ had evangelists, Java had evangelists (some would say Java even had apostles and Messiahs...), almost every language has evangelists.
Most programming languages with compilers to native code (JIT/AOT), the myth of high level Assembler for C only applies if your computer is a PDP-11 or a basic 8-bit CPU like a 6502 or Z80.
The ANSI C and C++ standards define the concept of abstract machine for the language semantics, just like in most languages.
Additionally you have the concepts of sequence points, the new memory model semantics for multi-threaded code and the beloved UB.
UB which doesn't exist in most languages, because their rather leave it implementation defined or lose the opportunity to target some strange CPU not able to support the language semantics.
Also Assembly doesn't has UB, making it ironic that it is safer to write straight Assembly at the expense of portability than C or C derived languages.
> Also Assembly doesn't has UB, making it ironic that it is safer to write straight Assembly at the expense of portability than C or C derived languages.
You just need to know your compiler flags/configuration to produce exact assembly output (if you want that). And about assembly not having UB, BSF and BSR are pretty good examples for that.
That just is quite ironic, because it is even version dependent across releases of the same compiler, which means anything fine tuned for version X can break on version X + 1, due to changes on the optimizer.
And I doubt anyone is regularly reviewing the produced Assembly every single time they change compilers.
Assembly doesn't have more than 200 documented cases that hardly any human is capable of remembering.
You could argue that each processor's errata is UB in assembly. I remember that programming for the XScale we had a whole manual of errata that we had to program around. Though because it's documented it's not "Undefined", it's just "Incorrect but we know how incorrect".
You can't really accidentally or unintentionally arrive at undefined behaviour in assembly language though. You would have to explicitly use an undocumented instruction or opcode and the result would probably be a hardware interrupt or exception every time.
I'd say that Ada would be well-suited for this. Particularly for implementing crypto. It has a steep learning curve, though.
Edit: Sorry, wrote this too hastily and want to put it in perspective. While Ada is well-suited for cryptography and has some good implementations, it is probably not as secure as C crypto libraries in practice. The reason is that the implementations I know don't do OS-specific things like locking memory or preventing core dumps, and doing this in Ada is a bit harder than in C.
TLS != crypto. TLS is actually very well-suited for a higher-level language like Rust or OCaml, especially due to how state machine patterns can be expressed safely in those languages. Crypto, on the other hand, often relies on low-level intrinsics to get good performance and cache timing attack resistance.
TLS depends on crypto and the mirage team implemented it using OCaml. It supports low-level intrinsics for timing attack resistance. See: https://github.com/mirleft/ocaml-nocrypto
found an error in the Argon2 reference implementation.
This is exactly why I like as many people writing crypto as possible[0]. Bugs like the one the writer found show up in reference implementations, and are found by absolutely no one unless someone writes an alternate implementation that behaves differently.
Edit: A further example, a lot of people write crypto as part of various CTFs. This is a case of "writing crypto" that poses no threat to the community.
[0] That doesn't mean they should use it in production.
I've always seen 'rolling your own crypto' as not being recommendation against writing your own library, but creating your own primitive.
Sure, writing your own library is very difficult, but you have a simpler set of problems, which proper testing, another set of eyes and enough tools will take care of the big problems.
Now, implementing your own primitive and recommending to use it is bad. For a primitive to be deemed secure, it has to undergo years of review and testing by seasoned cryptographers.
This isn't a recommendation to write your own library, but my opinion that it is okay to do so if you want something that none of the major libraries offer.
Every line of code written can be a severe vulnerability, not just code dealing with crypto. In case of crypto, being well-informed is good enough to not mess up those kind of implementations (on a theoretical level). The problems mentioned can be pointed out and verified on paper. But you can't say the same thing for code.
No, you can write crypto code, even when knowing stuff about the crypto, and still mess up. Crypto is problematic because you often cannot easily check whether you're wrong (it could be a tiny edge case that isn't tested because your input space is huge).
I didn't deny messing up crypto code is a thing. I'm saying messing up code is a separate problem from messing up how one architects projects using crypto.
What you say as messing up with edge cases, etc. is a programming problem. The errors talked about in the link he posted is about using faulty crypto. Today it is common knowledge to never use ECB.
What cipher mode you use is a problem you think about before you even start writing a single line of code. But you are talking about programming errors, which one can make in all areas, not only ones involving crypto. Those errors can break security similar to messing up crypto implementation.
I strongly disagree with this sort of attitude. It scares people away from learning how cryptography works. We should be encouraging everyone to write their own crypto: it's a fantastic learning experience. Just don't use your first go in production.
Hey, fantastic work! Really happy to see some amateur crypto development out there. I started as an amateur also, now I work in crypto product dev.
Definitely impressed if this is your first crypto work. I really like the simplicity, reminds me of OpenSSH.
I gotta agree with some of the naysayers though - this needs more time to brew. Even if it's perfect, there still more proof to show.
My suggestions:
* A long-running interoperability test between your lib and another. Test vectors don't catch everything. Run it for a week, check results and publish the results.
* Fuzzing. You need to to catch false results, random crashes, stack overflows, etc. Fuzz every input, do it for random lengths (both underflow and overflow).
* Maybe port it and test on Win, Mac, ARM, MIPS. I find a lot of bugs shake out during ports.
* Timing. You can't prevent all side-channel attacks, but bits going out are easy attack vectors. Use a high-precision timing mechanism to catch any unusual spikes. Capture a few hours of points for each functional path, and publish the results. (crypto_aead_unlock has two return paths, it might be better to maintain timing).
* Personal preferences: Rather than using int returns, use bool. Zero your ctx vars, e.g. a_ctx ctx = {0,};
I spend more time in testing products than development, honestly. Publish as much relevant data as you can.
> Definitely impressed if this is your first crypto work.
It is. :-)
> A long-running interoperability test between your lib and another.
Already have one for all primitives. Just tweak the parameters to make it run for a long time. The test code is in test/sodium.c and test/donna.c
> Fuzzing. You need to to catch false results, random crashes, stack overflows, etc. Fuzz every input, do it for random lengths (both underflow and overflow).
I'm not sure how to go about it. I already check every input size, from zero to several times the size of the internal block, in my comparison tests.
Of course, all inputs are correct. Incorrect inputs result in undefined behaviour, checking for that is out of scope. (That's why most functions return void.)
> Maybe port it and test on Win, Mac, ARM, MIPS. I find a lot of bugs shake out during ports.
Yeah, I only tried GCC and Clang on my Intel Ubuntu machine. But I did use Valgrind, ASan, MSan, UBSan, and the TIS-Interpreter. I also compiled under various standards: C99, C11, C++98, C++11, C++14, C++17.
> Timing. […] Use a high-precision timing mechanism to catch any unusual spikes
Isn't that a bit overkill? I'd rather manually review the source code for the absence of secret dependant branches or secret dependent array indices. I'm not implementing AES or RSA, the primitive I have chosen were specifically designed to easily avoid timing attacks.
> crypto_aead_unlock has two return paths, it might be better to maintain timing.
No. There are 2 cases to consider: failure, and success. Each path runs in constant time, so timing can only distinguish success from failure… which is revealed anyway by the return code.
> Rather than using int returns, use bool.
I'm afraid bool is a tiny bit less portable than int. I also went with the flow about error codes, which are traditionally int.
> Zero your ctx vars, e.g. a_ctx ctx = {0,};
I'm not sure how to do it portably, with zero dependency.
Whenever I feel the need for a tin foil hat I start to wonder if there is a FUD campaign powered by the "establishment" to encourage people not to investigate this area of computer science so that security holes will remain unnoticed.
But, yes I wouldn't start out on writing a crypto library, then again I wouldn't attempt to build an OS or a 3D stack or even an web server either. All cases where a security breach could have devastating effects as well.
Daniel Bernstein, Daniel Bleichenbacher, Dan Boneh and Thomas Pornin are professional cryptographers and world-renowned experts. Even I would feel comfortable writing crypto if Bleichenbacher was watching behind my back.
Frank "wrote" libsodium, but libsodium is effectively a port of NaCl. Frank had Daniel Bernstein watching behind his back.
Eric Young wrote OpenSSL. Look how that turned out! Has there been a class of cryptographic vulnerability that OpenSSL hasn't had?
And yet consider: for all the effort that people like Bernstein and Boneh put into writing strong, safe crypto, the entire world has blundered into vulnerability after vulnerability because the amateurish crypto in OpenSSL is the one every else ended up using.
It's as if you set out to prove my point.
Finally: you obviously know that "writing your own crypto" isn't the only way to become good at it. In fact, it's a terrible way to get good at it. The right way to get good at crypto is to learn how to break it. But that takes effort, and you can have a blog post crowing about your new crypto library just a few weeks after deciding to write one.
> are professional cryptographers and world-renowned experts
They were not in the beginning. And implementing crypto helped them get there. That's my point.
> Finally: you obviously know that "writing your own crypto" isn't the only way to become good at it. In fact, it's a terrible way to get good at it. The right way to get good at crypto is to learn how to break it.
I strongly disagree with you on this, but I'll add a twist: I'm mostly saying to become good in "applied" crypto you must roll your own. There are too many problems you can't understand until you actually write code and try to make it secure. If you want to become good in theoretical crypto, then don't roll your own obviously.
This works in any field btw. The best web pentesters are the people who have implemented websites on their own as well.
I would suggest that for each of them, a 10+ year career in academic cryptography, in each case accompanied by significant new research results, is what helped them get there. Implementing a library, not so much.
As for your second point: literally everyone can implement cryptography that appears secure to them. It's tautological.
> I would suggest that for each of them, a 10+ year career in academic cryptography, in each case accompanied by significant new research results, is what helped them get there. Implementing a library, not so much.
I believe that what made them exceptional (for most of them) is the combination of theoretical learning and hands-on programming. That's my entire point. I think we can agree to disagree. But that's an opinion I'll probably hold for the next years as this is the direction I want to take as well.
> As for your second point: literally everyone can implement cryptography that appears secure to them. It's tautological.
And the process of putting it out there and looking for ways to make it more secure is how you learn. Proof: he already learned a bunch about Frama-C and other C oddities and documented these via his blogpost. Plus he found a bug in the Argon2 implementation. That's a win.
Why don't you ask them? Most of these people aren't hard to get ahold of. One of them you even share a Slack with. I would be surprised to learn that any of them believed doing a library implementation of pre-existing crypto constructions was an important part of their education, but I like to be surprised.
I indeed learned a lot, and still learn a lot, by doing implementations. Doing a proper implementation forces me to consider all aspects; when the code runs properly, I know that I have, by definition, been exposed to all the parts. You cannot get that kind of exhaustiveness from simply reading an article.
However, doing implementations is not at all the same thing as publishing implementations! The first one or two attempts are always flawed in some way; only the third one can hope to be reasonably good. I took care to properly kill and dispose of the corpses of all my learning code.
The trick (and it's a difficult one) is to decide in advance that the code you write to learn will have to be deleted -- and stick to it. Developers have trouble letting go of their creations, in general. If you can maintain that discipline, then there is no problem in "writing your own crypto". But that is a big "if".
I've found that a good motivation for writing learning-only "throwaway" crypto code is as models for writing attack code; you don't even have to throw the code away, just publish it with the exploit.
But then, I'm a believer that everyone should learn crypto by breaking it, and clearly not everyone agrees with me.
Thanks for the input! This reminds me of what Amelie Nothomb says: "unlike a lot of writers, I have the decency to toss most of what I write". I don't think it's necessarily bad to publish crypto code. Marketing it as secure is another thing. I've marked most of my implementations as "readable" implementations meaning that are only there for educational purposes.
"libsodium" is just NaCl with autoconf cruft added.
NaCl does not use autoconf nor even make.
It is not necessary but if someone really feels compelled to use it, why stop them?
I have never seen any author or end user comment that they dislike using autoconf, or write alternatives to it. Autoconf is loved by all, isn't it?
I think what you could say is that using NaCl in your own software is the only way to get good at using it. Maybe the existence of libsodium means that more developers will have a go at using NaCl? Is it easier to use than an SSL library? The only way to find out is to try both and compare.
There is no need to use autoconf to build libnacl.a. The original build system for NaCl is beautifully simple. But if someone misses autoconf and make, then libsodium has added in those dependencies.
As far as I can tell, the authors listed on the above site are not "rolling their own crypto" for their projects, they are using cryptography written by the author of NaCl.
Personal bias disclosure: I find NaCl easier to use than all the SSL libraries I have tried. If I am not mistaken, I believe this was a design goal by the author.
Another angle is that a compromised security protocol used by a 100,000 developers is far more fruitful than having to assign a human specialist to crack & reverse engineer 100,000 uniquely thought-out implementations.
This is the point people miss. Same as wordpress vs creating your own blog. The more popular the software the larger the interest and the bigger exploit if successful.
It is less likely, though, and more likely to get fixed everywhere fast.
100,000 easy to crack implementations with obvious errors isn't really more secure than 1 super difficult, super well optimized/secured implementation, is it?
It is more difficult to crack 100,000 different interfaces because the time required to learn each one is a fixed cost. Having one central implementation makes it much easier to expliot because if an issue is discovered it can be immediately used everywhere. Don't put all your eggs in one basket.
exactly. a very shitty unique implementation is much less likely to be hacked than a near bulletproof implementation used by millions with a single flaw.
First let me say that I've followed your work a bit and I'm really impressed with the library. I'm currently planning on rolling my own crypto as well and your library is on my list of the things to look at.
One question:
> I was shifting a uint8_t, 24 bits to the left. I failed to realise that integer promotion means this unsigned byte would be converted to a signed integer, and overflow if the byte exceeded 127.
This sounds weird to me, how is a shift (which only apply itself on the bit value, not the number value) triggering a sign extension? I don't think you're talking about integer promotion here unless you had a uint16_t = uint8_t << 24 or something like this.
The problem happens if you do uint64_t = uint8_t << 24, for example. The shift will promote the uint8_t to int (not unsigned int), and then the conversion to uint64_t will sign extend, so
It prints "00000000ff000000" as most people expect. (I think the second example with (uint32_t) will only work if "int" is at most 32 bits, but I might be wrong).
If you say "foo + 3", and foo is a uint8_t, then 3 is technically an int, so the addition will do an integer promotion.
If you say "foo + bar" and both are uint8_t, then both will be promoted to integer. That's how C works. Assigning the result to another uint8_t will truncate it.
Please do not write your own crypto library until you really understand how C works, especially including this kind of detail.
Your example is different, this is because you're doing an addition. I can see how this could be a problem if you want to do a rotation (a << 4) + (a >> 60) and hence why you should use a ^ instead of a + here (example: https://github.com/gvanas/KeccakCodePackage/blob/master/SnP/... )
There's no sign-extension involved. Let's say the input is (uint8_t)128. This gets promoted to (int)128, which is then shifted 24 bits to the left. If the input was (int8_t)128, it would get sign-extended into (int)-128, but that is not the case here.
With 32-bit ints, this operation will result in 0x80000000, shifting a bit into the sign bit, which constitutes an overflow (and undefined behavior). Think about it in terms of multiplication: you're multiplying a positive integer by 2^24, and it turns out negative!
right, but he only used unsigned types. I think there is another problem here:
u64 = u8 << 24
Here the integer promotion might transform u8 into a 16-bit or 32-bit int (depending on the arch). Which will then be converted to a u64 AFTER the shift. Which might have you lose half or more of the bits.
His code is not that though, it's
u32 = u8 << 24
which should work only if ints are 32-bit on the system you compile on.
> The integer promotions are performed on each of the operands. The type of the result is that of the promoted left operand.
6.3.1.1 § 2
> If an int can represent all values of the original type (as restricted by the width, for a bit-field), the value is converted to an int; otherwise, it is converted to an unsigned int. These are called the integer promotions.
So if you shift a uint8_t, it is first converted to a (machine-dependent) signed int, then shifted, then converted to whatever the expression type ends up being.
It's probably the latter part (conversion to "whatever the expression's type ends up being) doing the sign-extension, see moefh's sibling comment with a uint64_t final destination, the int -> uint64_t is sign-extending.
I should have quoted the proper part I was replying to ("I don't think you're talking about integer promotion here")
I just checked that myself, it is not the casting to uint64_t since the sign-extension happens even before that. The problem is that the result from the shift is stored in a register, which is probably 64-bit on your machine, and to do that it is sign extended from a signed int to a signed 64-bit value. Wrote more about that here: https://www.cryptologie.net/article/418/integer-promotion-in...
The size of the cpu registers don't matter here. What matters is that 'int' is a bigger type than uint8_t and smaller than uint64_t: the resulting unexpected behaviour is mandated by the C standard regardless of the register size of the cpu (for instance both 32-bit i386 and 64-bit x86_64 will behave the same, because both typically have 32 bit 'int').
More generally, reasoning about this kind of thing by thinking about the CPU leads you down the wrong path -- modern C as implemented by compilers today is an implementation of a standardized language, and they behave as that standard says, which is not always as you might expect from the hardware behaviour. (Simple example: left shifting of negative numbers does a simple straightforward thing on practically all hardware, but the C spec says it's UB and if you write code that assumes the h/w behaviour you'll get bitten.)
I believe this[1] is the patch that fixes this bug. I tried to reproduce the behavior but couldn't succeed. Maybe I was doing something wrong. Would really appreciate it if someone here could show a test case where this matters.
Your failure to reproduce the behaviour is normal: on the compilers I have tested, this doesn't affect the generated binary.
To see the difference, you need to first modify the makefile to use UBSan instead of just GCC (just comment the active CC line, and uncomment the right one). You may want to tone down the optimisations for faster compilation as well. Then you can run `make clean` and `./test.sh`. You should have a warning on commits prior to this patch.
I'm amazed that the reference implementation of Argon2 had a bug. So that means anyone who deployed Argon2 today didn't really use Argon2 (I'm being pedantic), but something else?
libsodium also got it wrong, so that brought down my opinion of it being a trusted and well-reviewed library.
Now the question is, will they continue to use the same implementation or move to the fixed code?
Maybe the authors will fix the code in subsequent versions of Argon2. Right now, backward compatibility is deemed more important. The effects of this bug are practically negligible anyway. I bet a single bit of additional entropy in a password would compensate that a hundred fold.
I'm a bit disappointed however at their not updating the specs. I signalled the bug in January, and the latest version of the appear to have been published in March. I guess they had other priorities.
Don't be too hard on Libsodium: they only got it wrong because they reused the reference implementation. (Also, I don't see them breaking backward compatibility either.)
I'm also referring to the orgs who used Argon2 in production. If they move to the fixed version of Argon2 in libsodium, they'll have to make a plan to securely move to the correct implementation by hashing the passwords again.
It can be done, but I'm wondering if they'll even bother.
Those who deployed Argon2 must have deployed the reference implementation, therefore the code became the de facto standard. Khovratovich confirmed in https://github.com/P-H-C/phc-winner-argon2/issues/183#issuec... that the code is the de facto standard and the spec will be edited to reflect this.
libsodium implemented the same behavior simply because it reused the Argon2 reference implementation.
I really liked that read, good writing style and the author was coming from the same place that I'm usually coming from (can we redo this ourselves and shave off some bloat in the process).
However, as it stands now, it looks more like a cautionary tale on why he should have listened when people told him not to do it :-)
I suspect a worthier goal would have been to look for ways to improve the linking footprint of libsodium instead of rolling your own library. Maybe turn the "I want the smaller version without the huge lookup table please" in a compile time option or so. That way more people could have used it without having to commit to "I'll use this little known experimental library instead of libsodium". Commit in the sense of "people will ask why and I will have to defend my decision", not in the sense of "oh no we are locked in!!"
Reducing the linking footprint wasn't my only goal. Reducing the source code is also important, if only to facilitate tests and audits —thus increasing confidence. That huge table for instance would need to be tested. Every single value would need to be checked for correctness, or I would have to demonstrate they came from a sound generator.
---
I've been careful not to lock people in. If people want to upgrade to faster primitives, they can. I don't think they will ever need to, though.
The v1.0.0 was published the day a bug was fixed and yet the first sentence of this post is "Monocypher is ready for production". Sure, OpenSSL has bugfixes on regular basis as well, but I believe that further fragmenting the crypto ecosystem with a new library and little commercial support to properly back it and pay for security audits is risky.
It's risky but sometimes it's the only way. To be fair the code size is relatively small so it should be easier to audit than other products. My guess is that his blog post is also some motivation to get new pair of eyes on the library. If there's any fund for that, I'd be happy to look at it ;)
> The v1.0.0 was published the day a bug was fixed and yet the first sentence of this post is "Monocypher is ready for production".
You have to consider the nature of the bug, as well as the fix. The bug was an Undefined Behaviour that had no effect on the binaries I could generate. The fix was 10 characters. Running the automated test suite took longer than fixing the bug.
If there were reasons to delay 1.0, this bug was not it.
> I believe that further fragmenting the crypto ecosystem with a new library and little commercial support to properly back it and pay for security audits is risky.
I agree fragmentation is a problem. I just couldn't satisfy myself with existing alternatives. As for support, I designed Monocypher specifically to need very little of it. Ideally, the only notable changes from now on will be about the manual and the test suite.
Write a crypto library and blog about how great it is.
Apply the tools that other people used to find bugs in it and repeat blogging about how great it is.
...
In the end you get a library that is slower and smaller (because you left out the optimized code, duh). It's still not as small as the smallest.
Is this easier to audit? Dubious but it doesn't matter because if people hadn't wanted the extra speed they wouldn't have added the optimized code anyway.
Plus, that code is already audited and rolled out and works! Who cares about auditing another library that provides nothing new?
The only non obviously-garbage argument here is usability which I am too lazy to look at because it is too fuzzy to refute anyway.
By all means, write your own crypto but DON'T USE IT! And of course don't tell other people to use it either.
Let's list the pros and cons of each libraries (let's assume we trust all 3 libraries):
Monocypher vs TweetNaCl:
Good:
Monocypher uses more modern primitives (Blake2b, Chacha20)
Monocypher provides password derivation
Monocypher is a bit easier to use
Monocypher is much faster
Bad:
Monocypher is twice as big
Conclusion:
Monocypher utterly outclasses TweetNaCl.
Monocypher vs Libsodium:
Good:
Monocypher is much smaller
Monocypher is easier to deploy
Monocypher is a bit easier to use
Bad:
Monocypher is a bit slower (except Argon2i, which is faster)
Conclusion:
If performance matters, use NaCl. Otherwise, use Monocypher.
Nothing new, you say?
> By all means, write your own crypto but DON'T USE IT! And of course don't tell other people to use it either.
Get back to me in a couple years, we'll count the CEVs since version 1.0.
Main lesson not learned: instead of testing, prove correctness in high assurance code like this. Rigorously and formally.
Preferably even refine the proof to executable code.
(Yes, it would take somewhere on the order of 10k LOC to prove correctness of this 1k.)
In this case 3 of the bugs would be caught when trying to prove the absense of undefined behaviour/use of uninitialized value.
Can existing formal proof tools be used for that?
Proving correctness isn't enough. You have to test it too. It's perfectly possible for the proof to have errors (bugs). It's also possible for the proof to be incomplete: you can prove the algorithm is correct and implemented correctly to spec and still have vulnerabilities.
I'm sure you encounter "high assurance code" like this every day -- your operating system, the secure protocol used to post your comment, the controls for your Anti-Lock Breaking System in your car.
TLS has been subject to lots of formal verification. The other things, probably not so much.
Verifying some (but not all) properties of crypto is relatively (but not absolutely) easy, because it's essentially just math with pure functions. Operating systems and physical controllers with real-time constraints have different requirements and more interacting state, which means that the desired properties are not easily expressed as a mathematical expression.
Interestingly it looks like there was a bug in every Argon2 implementation specifically because everyone just used the reference implementation.
Personally, I think the intention of the phrase "don't roll your own crypto" was lost over time. It was advice to companies to use standard cryptographic algorithms rather than everyone coming up with their own thing for no good reason. It doesn't mean that only some blessed individuals should be allowed to write cryptographic software. I think OpenSSL has taught us that not rolling your own crypto results in a monoculture of self-professed experts that didn't follow that advice.
Those are two separate arguments. Of course there should be competition in crypto libraries. However the "don't roll your own" advice is to prevent people who don't have sufficient experience, education nor resources to write their own library that can compete - effectively leaving their systems vulnerable.
The thing about the "don't roll your own" advice is it's aimed at people who don't competently know what they're doing rather than those who do.
How do you expect people to become competent if you constantly tell them to not do something? All that will happen is that people who don't listen to warnings will write crypto libraries and everyone will be forced to use them because nobody else wrote one.
The fact that there was no independent implementation of Argon2 that found there was a bug in the reference implementation shows that the "don't roll your own" advice is discouraging competition in crypto libraries to the point where the reference implementation wasn't sufficiently verified to produce correct output.
Again I feel the need to reiterate that the rule only needs to apply to those it needs explaining to. Ie nobody who is smart enough to roll their library would not do so for a production system without it having undergone rigorous testing first. Sure we might write hobby projects (I've written my own encryption ciphers for fun) but that's very different from actually using said cipher on live and/or business critical systems.
Simply put, the rule is there to remind people that rolling your own crypto libraries is dangerous. It's not meant to be taken literally to the daft extremes that yourself and a few others on here have. It's just meant to stop those lacking common sense from leaving themselves vulnerable.
I agree with what you're saying in general, that people should not write a crypto library unless they have a strong need to and are going to have it thoroughly audited.
> Simply put, the rule is there to remind people that rolling your own crypto libraries is dangerous. It's not meant to be taken literally to the daft extremes that yourself and a few others on here have.
My problem is that this nuance is not present in the statement "don't roll your own crypto". And that lack of nuance leads to no competing Argon2 implementation, and having OpenSSL control the future of TLS.
It's not just a few people who have taken it "literally to the daft extremes". Nobody in the entire crypto community re-implemented Argon2, not even as a hobby project. The message "don't roll your own crypto" is too strong because it results in nobody implementing their own crypto for any reason (which should not be surprising given the mantra).
I think it's a bit of a stretch to say nobody implemented Argon2 because of the aforementioned rule when it's also been pointed out in this thread the vast number of times that people have rolled their own crypto libraries. That would suggest to me that there are other factors at play with your example. Possibly due to the relatively young age of that specific KDF, or perhaps it's lesser known compared to many of the others? I don't really know enough about Argon2 to make any specific comments there but in the more general sense it has been proven on here already that many people do make hobby projects of rolling their own libraries.
I used Argon2 as an example because it was mentioned in TFA, as the author discovered that the test vectors were wrong because the reference implementation was wrong. That author noted that if someone else had implemented it from scratch they would've noticed this immediately when testing the test vectors.
I got why you used that as a reference. I was just saying it's a bit of a stretch to say your arguments about the aforementioned rule are the reason nobody else attempted to implement Argon2 when we already have a long list of examples of people who do roll their own ciphers.
The point of the rule is to discourage those who don't know what they're doing from compromising live systems. The kinds of people interested in Argon2 are the kind who are already security minded so will understand the point behind the rule and will understand it's fine to ignore that rule if you understand the caveats that rule implies.
The fact Argon2 hadn't been reimplemented can easily be explained for a number of other reasons too.
Most people with even a casual interest in security do understand that point behind the "don't roll your own" rule so I think it's odd to suggest that we are all too literally minded. As evidenced by the fact that a great many people do write their own hobby libraries.
If you know enough cryptography and are a good programmer, then you can do it; otherwise better not.
The advice to not roll your own crypto used to mean that you shouldn't invent your own cryptographic algorithm. That's true, unless you're an experienced cryptanalist you're likely going to come up with something bad. (Or slow, as e.g. a moderately secure Feistel cipher is easy to invent if you don't care about speed.)
As for your own implementation of standard algorithms, there are some things that can go wrong, mostly with key derivation padding and chaining modes, but it's not a secret art that only few chosen cryptographers could master. You have test vectors for the algorithms itself, and have to be extra careful about the rest. You have to get someone to audit the code. That's about it.
The fact that people no longer roll their own crypto has two major disadvantages. First, modern ciphers like AES have been designed for ridiculously low security margins. You could claim that security was subordinated to speed in NIST competitions. You can see that in the decision to choose Rijndael over Serpent and Twofish. AES ahs the lowest security margins. Second, it presumably makes life much easier for intelligence agencies all over the world. Instead of having to reverse engineer and analyze thousands of different implementations, they can focus on attacking a dozen popular crypto libraries. This saves them a lot of time and money and also makes it much easier to attack individual developers or binary distribution. (Most attacks nowadays are probably side-channel attacks anyway, but who knows...)
well if you have enough people especially working with crypto, than why not?
It's not like that the existing solutions came out of nowhere. some people actually wrote them.
I mean libsodium is actually really simple to use. however before libsodium it was probably easier and mostly more correct to roll your own. there are so many things you can do wrong with other libraries that the chance to do it wrong is as high as in your own implementation.
I would probably have the same perspective if our roles were reversed. I assume I'm a random dude on the internet to you, and trusting crypto code from a random dude is… unwise to say the least.
Nevertheless, Monocypher is pretty thoroughly tested by now. I wouldn't be surprised if Monocypher starts getting serious vetting a few months from now.
Through sweat and blood. Don't expect crypto libraries to be secure the first few years after their release. That's why people favor the mature, battle-tested libraries.
Yes, but are hackers going after common used libraries to get more vulnerable systems to attack or are they going to spend time on some unknown homebuilt crypto? In some cases, security through obscurity works well in practice.
> In some cases, security through obscurity works well in practice.
This is not one of those cases. Absolutely not. I'm moderately competent at finding security bugs in things, but I doubt I could find any in OpenSSL. I am confident I could find some in your average hand-rolled code.
The thing is people make the same mistakes. There's a set of well-known mistakes that are very easy to make, especially if you're not versed with the entire history of implementing crypto - which is the case for the majority of people rolling their own. This makes it very, very easy to guess what mistakes they will make, and if you know what you're looking for it's easy to find it.
My "personal best" for finding a crypto bug in a project is 50 seconds.
I doubt anyone has found a bug in OpenSSL (or any established crypto project) anywhere near that fast.
You got to consider your most likely risk for attack. Targeted or at random by a botnet? For example, you are most likely more secure in practice by writing your own website than using Wordpress, simply because you are more likely to get hit by a botnet targeting every Wordpress site than someone going directly for you.
Both are vectors. On one hand, you have automated systems trying to hack low-hanging fruit (unpatched, well-known-to-be-insecure libraries), while simultaneously, you could become a specific target - where security by obscurity doesn't get you much. Saying "one of these is not worth considering, as it's less probable" is security through handwaving: one day, someone might discover a bug in your obscure library, and suddenly you're right back in range of automated attacks.
tbh, they do develop like this. Some guy come up with a library, a lot of critics arise, but the guy pushes through and many years later it becomes somehow a reference when someone big on twitter mentions it as his/her favorite crypto library.
There is a difference between rolling your own crypto and rolling your own crypto. There is Monocypher, then there is this company that is trying to do some weird protocol using CRC, DES in CBC mode and MD5. The sentence was targeting the latter case.
No, it is targeting both. I see no mention of how monocypher implementation does anything 5o prevent side channel timing attacks, I haven't looked to see if there is any sensitive memory scrubbing. Of the bugs listed most are performance bugs by someone who doesn't know C very well yet have they done the things actually!my required in a crypto library like attempting to make sure all branches are the same instruction length for all implemented algorithms?
> I see no mention of how monocypher implementation does anything 5o prevent side channel timing attacks,
Oh come on, the chosen primitives are all designed for easy immunity against timing attacks. I haven't verified this formally, but I basically ripped off safe designs, and I tried to be careful about avoiding secret dependant branches and indices.
> I haven't looked to see if there is any sensitive memory scrubbing.
There's a whole test suite for that.
Look at the makefile, then select whatever sanitiser it lists (comment/uncomment the relevant CC line at the begining). Then run `./test.sh`. You can also run the relevant executables under Valgrind. Finally, there's a way to run it under the TIS-Interpreter, though that is veeery slow.
> by someone who doesn't know C very well
Could you tell me how you inferred that? That could help me improve.
This doesn't happen naturally, except for Poly105 and curve25519 (for those one indeed has to be careful). A naive implementation of Chacha20 for instance is pretty much guaranteed to be immune to timing attacks.
On a first read I though "yeah, sure, looks quite dangerous indeed", then I read:
Sylvester H. Roper, inventor of the eponymous steam-powered bicycle, died of a heart attack or subsequent crash during a public speed trial in 1896. It is unknown whether the crash caused the heart attack or vice versa.
This seems to mean "killed in an interesting way". People killed by their inventions in petrol-based transportation seem to be too numerous even for a Wikipedia list: "yeah, people die in cars en masse, that's not really newsworthy."
Actually you do roll your own parachute once you have done enough jumps. But your reserve chute is always rolled by a certified professional specifically certified to roll reserve chutes, and not by yourself. (if I remember correctly, my memory is a bit fuzzy from when I jumped a couple of years back)
I'm not sure about "don't roll your own parachute" as it should be common sense, but well, in 1912 a parachute inventor died while testing his creation.
You don't have to be an expert to make/jump the actual parachute/canopy. As long as the harness, container and reserve are TSO'd you're good to go. It's a lot of work though, at least for a ram-air.
I know a few guys who have manufactured their own canopies and harness/container systems in their garage, and I've jumped them - even though I consider these guys fallible :)
Have the library functions been tested (or proved) to be constant time ? This is a critical property of libsodium.
Another question is what make the difference in number of line of codes with libsodium ? I mean what do these lines of of libsodium that your lib doesn't have contain ?
Also, is this a fair metric ? You know we can have 1000 char long lines in C. ;)
> Have the library functions been tested (or proved) to be constant time ?
Just check the absence of secret dependent branches, and secret dependent array indices, it's pretty simple.
> what make the difference in number of line of codes with libsodium ?
Less lines means less need for eyeballs, tests, and audits. Libraries the size of TweetNaCl (and, to a lesser extent, Monocypher), can be subjected to very rigorous audits, and increase confidence accordingly.
> what do these lines of of libsodium that your lib doesn't have contain ?
I can see 3 sources: redundant primitives (Monocypher is not redundant), alternate implementation (Monocypher sticks to portable C), and verbose implementations (Monocypher keeps it simple).
> Also, is this a fair metric ?
Not quite: I did not count the size of the header in Monocypher and TweetNaCl. Still, I don't think Libsodium has less than 22K lines of actual code. (Note: I counted everything with sloccount)
On the other hand, my lines never exceed 80 characters.
I'm not sure. If there is such a test, you should find it on `ge_frombytes_neg`. I bet it returns -1 in line 1315 if that happens. (It triggers a rejection when that happens.)
If I'm right about this, it would explain why this code path is never hit: it would only trigger with invalid public keys. I'll test that as soon as I can.
It's interesting that Frama-C was mentioned. I have been playing with it the last few months and would love to hear exactly how it was used. I've read through a lot of the documentation but haven't found many examples of it being used
Note that this is the initial stage of using Frama-C. The next step would probably include writing some specifications for functional behavior (in ACSL), and then using the WP plug-in (based on weakest precondition calculus) to prove those specifications, providing much stronger guarantees.
A few Frama-C case studies are on Github (https://github.com/Frama-C/open-source-case-studies), including an old release of Monocypher, but they constitute mostly a initial parametrization of the EVA plug-in. Most advanced users of Frama-C apply it on industrial code that is not available, but academic users regularly publish papers with case studies (searching "frama-c" on Google Scholar gives good results).
I really liked your previous articles/tutorials about Poly1305 and Chacha20. Is there any plain to make similar articles about the other constructions that you implemented on this library?
Is there also any plain to implement scrypt (based on the salsa), SPHINCS (uses BLAKE(1) and Chacha), Ed448-Goldilocks, Keccak or any of the CAESAR candidates? (The ones that seemed the most interesting to me were NORX, Keyak (based on keccak) and HS1-SIV (based on chacha)).
I don't think I'll write for the other primitives, I don't know them well enough to comment on them properly.
I won't add other primitives to Monocypher, that would defeat the purpose of a "one true cipher suite". One stream cipher, one authenticator, one hash, one password key derivation, one key exchange, one signature. That is enough.
When some of those primitives becomes obsolete, I'll consider writing a new, incompatible version of Monocypher. For instance, we can switch to curve448 if curve25519 isn't secure enough. I think at least a decade will pass before we come to that, though.
Having all of that would be great, but most of the time you need to weigh one against another. Smaller code means smaller attack surface and supposedly fewer ways to shoot yourself in the foot ("I know, I'll use CBC"); of course, checking whether the author doesn't shoot you in the foot by choice of algorithms, or their implementation, is advised.
The thing that I said was "table stakes" for implementing cryptography was passing the algorithm test vectors, which this author's previous post claimed as a security feature.
If you're unfamiliar with the concept, a test vector is a series of strings and intermediate values used to ensure that your (say) OCB3 is the same as everyone else's OCB3.
Had he asked, I'd further claim that not having dumb C bugs is also table stakes for cryptography, since serious crypto vulnerabilities happen at a higher level of abstraction. The author grazes past (somewhat disconcertingly) one such class of bugs when he discusses carry propagation and limb scheduling in the context of Poly1305. Improper carry propagation is an example of the kind of security vulnerability for which there are no test vectors and no memory safety validation tools.
You just have to know what a carry propagation bug is, where to look for them (not Poly1305), and what the impact of one is.
Hopefully, the author of this library does.