As for the library problem, I'm teaming up with Alex Gaynor, Hynek Schlawack, Donald Stufft and Christian Heimes to write a better cryptography library for Python: https://github.com/alex/cryptography
New contributors very welcome!
On the other side of the coin, I agree that cryptography education is very lacking. That's why I presented Crypto 101 last year at PyCon, which was an attempt at teaching just enough crypto in 45 minutes. Obviously, that's pretty near trying to square the circle: it's a best-effort attempt, but that's about it. (If you're still interested, the recording is available for free: http://pyvideo.org/video/1778/crypto-101)
That's why I'm also writing the book version of Crypto 101. Early draft stages right now, but getting there. After listening to tptacek say it so many times, I've come to agree: if you want to teach crypto well, you probably want a bunch of exercises that teach you how to break it. As a result, both the exercises and the text-adventure game I'm adding to the book rely on breaking real-world crypto: everything from bad password storage to MITMing SSL handshakes to stream compression :)
Additionally, a significant portion of the book will be devoted to when (i.e. rarely ;)) and how to write crypto. I would hope that your journey would've been a lot less painful if you had had access to the book. Perhaps for future programmers? :)
If you'd like to review, there's a reviewers mailing list, feel free to shoot me an e-mail explaining why you think you'd be a good reviewer, and I'll probably sign you up :)
I think it's pretty ridiculous that whenever a "non crypto" person tries to use crypto the security folks go lambast them, yet there is simply not a truly usable crypto api out there! Please go out and make this the right way; I'm looking forward to it :)
There isn't one kind of "security" person. I work in security. But I don't install antivirus, I don't write hardened runtimes, I don't configure firewalls, I don't analyze malware samples, I don't design ciphers, I don't write policies, and I don't do forensic investigations.
All I do is break into software; I do vulnerability research, and that's pretty much it right now.
As a vulnerability researcher, it's not on me to provide you with a good crypto library. People like us write software in order to find vulnerabilities. We're specialized so that we can be very good at doing that by dedicating all our time and attention to that one problem.
But much more importantly: like vuln researchers, don't like vuln researchers, it doesn't matter. When vuln researchers complain about crypto libraries, they are informing you of facts. You can ignore those facts and probably be vulnerable, or accept them and act on them and not be vulnerable.
Having said all that, there are in fact good crypto APIs. You can use Keyczar, which was a Google project. You can use NaCL, which is Daniel Bernstein's project; it even has a Ruby port now. You can use Peter Guttman's cryptlib.
Ok, so maybe I came off a bit harsh there. I've worked with a number of excellent security professionals who have found weaknesses and provided great prioritizations and solutions to the problems they've found.
However that's not the case whenever some project on HN claims to use cryptography. The answer usually is: "you don't know what you're doing, so don't use cryptography." While that is "safe", for many it's not actionable: sometimes we actually do need to use it and not everyone can afford a security consultant. I wish instead there was a recommendation for some library out of the box that would handle everything from key management to supporting my specific use cases.
Said a different way: I want an API that is simply giveMeANewKey() and encryptSecretMessage() and decryptSecretMessage() and the library takes care of both secrecy and integrity with strong defaults.
You can't expect it to make the whole system secure of course, but I think a better API that makes it harder to shoot yourself in the foot (both by having sane defaults and being very opinionated to the point where it affects your system design) would go a long way, no?
EDIT: looks like the stuff you linked to is more what I am looking for. I wish I had heard of them sooner!
That's our goal at least. Personally I feel if you use our higher level API and get something wrong, then that's a bug in our code that allowed you to do that.
The lower level APIs we are exposing will hopefully be more easy to use as well, but they will not offer the same protections against mistakes the same as the higher level API would.
KeyCzar is pretty great! It's the kind of library most people that write some crypto should be using: abstracting away all of the gnarly details that they're more likely to get wrong than right.
NaCl gets a lot more attention these days, but I'm convinced both are fine. On the one hand, NaCl is pretty hardcore DJB-ware (poly1305-aes, salsa20...), and normally I'd err on the side of conservatism. OTOH, it's also really really good DJB-ware, so I can't blame anyone for using it :)
I personally prefer sodium's Python versions. Whether to deliver libraries in the Python packages or to use shared libs remains an open problem: I am somewhat convinced the former's better, but I can certainly see the cases where it'd go wrong.
I've watched your pycon talk twice. It makes this stuff pretty accessible. The only thing I had a hard time following was your description of a hash length extension attack. I had to go and look it up afterward. Otherwise, a very good talk.
It’s very early now, so I think one of the most helpful things would be to look at the API discussions and tell us if something doesn’t make any sense to you. That would mean that a) the API is bogus or b) the docs/rationale are insufficient and something needs to get fixed.
"This was because the "iv"
parameter in the pycrypto Cipher API is ignored in CTR mode. Instead, you
need to provide the data to the Counter object.
I don't know if I am being muddle-headed in thinking of the initial counter value as an IV, but I was a little annoyed with pycrypto. Couldn't it throw
an error if it's given an IV in CTR mode,"
THIS
API creators have to stop this bullshit of putting death traps in their code.
This is not about go RTFM, or "you're incompetent".
If you allow it to be used in a wrong manner, it's your liability as well.
If your IV sometimes goes into one field, but in another occasion goes in another field, THROW AN ERROR IF IT'S NOT USED CORRECTLY
It drives me a little crazy that implementations (and even some academics) have taken to calling the CTR nonce its "IV". There are particular requirements for IVs and particular requirements for nonces; they aren't the same thing.
That seems like a case of determining what the API should check for. An explicit check for e=1 seems stupid, the reasonable check would be to make sure that e was a valid exponent, which I believe means confirming that it is primes, which can have a non-trivial overhead (although in the case of RSA, the exponents are gennerally small enough that it doesn't matter.) The real question in this case is why the API let you select the exponent in the first place.
Hence, like I said: they should not have been near an API that had the letters R-S-A in it. They should've been near a very nice API with no configurable knobs to get wrong. The entire thing was part of an authenticated encryption scheme: they could've just use NaCl's `box` API and the entire thing would be two lines of code, give or take. Oh, and not utterly pointless ;-)
Crypto library authors seem to have taken the philosophy that "it was hard for me to learn all this stuff, so it should be hard to use". Since I started working on the cryptography library that lvh mentioned I've seen some absolutely amazing APIs from crypto libraries:
* Undocumented error conditions (actually in general the documentation is atrocious for crypto libraries, "crypto is easy, why would you need docs"?)
* ECB as the default mode for block ciphers
* Ignoring invalid parameters
* Allowing you to provide parameters which basically your cryptography a no-op (e=1)
* Modes whose purpose is undocumented, merely "This mode should not be used"
And finally, absolutely NO python crypto libraries expose AES GCM. (This is all to say nothing of the myriad of other issues I have)
He doesn't even get to the real problem with writing crypto - unless you can get some attention from experts, there's no way you can know you didn't leave a big gaping hole in your security. It's something you literally can't test yourself.
Kudos to OP for trying a hard problem and be frank about the hurdles. That's the way to learn. Try it, make mistakes, so what. We learned best from mistakes and failures. Mistakes burn into our memory. We'll be better the next time.
I'm not sure I'd called M2Crypto a much better API. It more or less exposes the OpenSSL API which is also notoriously hard to get right. Further more it hasn't seen an update since 2011, the SSL certificate for the bug tracker expired in 2011, and it does not compile without patches against any modern OpenSSL (1.0+).
The state of crypto in Python is horrendous, hence the new library.
Another problem is: I know I shouldn't roll my own, but I can't or don't know how to identify implementations that meet my needs and are written by someone sufficiently competent.
The time-insensitive method is to XOR every bit of HMAC_0 with HMAC_1 (keep going even if they are not of equal lengths use 0s) and then (not before) check if the lengths are equal.
either approach avoids leaking useful information, doesn't it? one does so by guaranteeing identical times. the other by passing the information through a one-way function.
Also the source code looks like encrypted text. I can't figure out what much of it does, or why it does it by reading it.
Also, I think the API is not the best, because it asks for a password. Where are people storing the password? This promotes easy mistakes... like for example, saving it to a file, or reading it from a web input form.
i don't know a good way to avoid the in-memory password from within python - if that is possible, please show me how and i'll fix the code. the larger point about passwords is, i think, part of the reason for keyczar's approach. but that is not yet available in python 3 (certainly not when i wrote the code; i did check since and last time i looked there was a patch being considered).
As for the library problem, I'm teaming up with Alex Gaynor, Hynek Schlawack, Donald Stufft and Christian Heimes to write a better cryptography library for Python: https://github.com/alex/cryptography
New contributors very welcome!
On the other side of the coin, I agree that cryptography education is very lacking. That's why I presented Crypto 101 last year at PyCon, which was an attempt at teaching just enough crypto in 45 minutes. Obviously, that's pretty near trying to square the circle: it's a best-effort attempt, but that's about it. (If you're still interested, the recording is available for free: http://pyvideo.org/video/1778/crypto-101)
That's why I'm also writing the book version of Crypto 101. Early draft stages right now, but getting there. After listening to tptacek say it so many times, I've come to agree: if you want to teach crypto well, you probably want a bunch of exercises that teach you how to break it. As a result, both the exercises and the text-adventure game I'm adding to the book rely on breaking real-world crypto: everything from bad password storage to MITMing SSL handshakes to stream compression :)
Additionally, a significant portion of the book will be devoted to when (i.e. rarely ;)) and how to write crypto. I would hope that your journey would've been a lot less painful if you had had access to the book. Perhaps for future programmers? :)
If you'd like to review, there's a reviewers mailing list, feel free to shoot me an e-mail explaining why you think you'd be a good reviewer, and I'll probably sign you up :)