As for the library problem, I'm teaming up with Alex Gaynor, Hynek Schlawack, Donald Stufft and Christian Heimes to write a better cryptography library for Python: https://github.com/alex/cryptography
New contributors very welcome!
On the other side of the coin, I agree that cryptography education is very lacking. That's why I presented Crypto 101 last year at PyCon, which was an attempt at teaching just enough crypto in 45 minutes. Obviously, that's pretty near trying to square the circle: it's a best-effort attempt, but that's about it. (If you're still interested, the recording is available for free: http://pyvideo.org/video/1778/crypto-101)
That's why I'm also writing the book version of Crypto 101. Early draft stages right now, but getting there. After listening to tptacek say it so many times, I've come to agree: if you want to teach crypto well, you probably want a bunch of exercises that teach you how to break it. As a result, both the exercises and the text-adventure game I'm adding to the book rely on breaking real-world crypto: everything from bad password storage to MITMing SSL handshakes to stream compression :)
Additionally, a significant portion of the book will be devoted to when (i.e. rarely ;)) and how to write crypto. I would hope that your journey would've been a lot less painful if you had had access to the book. Perhaps for future programmers? :)
If you'd like to review, there's a reviewers mailing list, feel free to shoot me an e-mail explaining why you think you'd be a good reviewer, and I'll probably sign you up :)
I think it's pretty ridiculous that whenever a "non crypto" person tries to use crypto the security folks go lambast them, yet there is simply not a truly usable crypto api out there! Please go out and make this the right way; I'm looking forward to it :)
There isn't one kind of "security" person. I work in security. But I don't install antivirus, I don't write hardened runtimes, I don't configure firewalls, I don't analyze malware samples, I don't design ciphers, I don't write policies, and I don't do forensic investigations.
All I do is break into software; I do vulnerability research, and that's pretty much it right now.
As a vulnerability researcher, it's not on me to provide you with a good crypto library. People like us write software in order to find vulnerabilities. We're specialized so that we can be very good at doing that by dedicating all our time and attention to that one problem.
But much more importantly: like vuln researchers, don't like vuln researchers, it doesn't matter. When vuln researchers complain about crypto libraries, they are informing you of facts. You can ignore those facts and probably be vulnerable, or accept them and act on them and not be vulnerable.
Having said all that, there are in fact good crypto APIs. You can use Keyczar, which was a Google project. You can use NaCL, which is Daniel Bernstein's project; it even has a Ruby port now. You can use Peter Guttman's cryptlib.
Ok, so maybe I came off a bit harsh there. I've worked with a number of excellent security professionals who have found weaknesses and provided great prioritizations and solutions to the problems they've found.
However that's not the case whenever some project on HN claims to use cryptography. The answer usually is: "you don't know what you're doing, so don't use cryptography." While that is "safe", for many it's not actionable: sometimes we actually do need to use it and not everyone can afford a security consultant. I wish instead there was a recommendation for some library out of the box that would handle everything from key management to supporting my specific use cases.
Said a different way: I want an API that is simply giveMeANewKey() and encryptSecretMessage() and decryptSecretMessage() and the library takes care of both secrecy and integrity with strong defaults.
You can't expect it to make the whole system secure of course, but I think a better API that makes it harder to shoot yourself in the foot (both by having sane defaults and being very opinionated to the point where it affects your system design) would go a long way, no?
EDIT: looks like the stuff you linked to is more what I am looking for. I wish I had heard of them sooner!
That's our goal at least. Personally I feel if you use our higher level API and get something wrong, then that's a bug in our code that allowed you to do that.
The lower level APIs we are exposing will hopefully be more easy to use as well, but they will not offer the same protections against mistakes the same as the higher level API would.
KeyCzar is pretty great! It's the kind of library most people that write some crypto should be using: abstracting away all of the gnarly details that they're more likely to get wrong than right.
NaCl gets a lot more attention these days, but I'm convinced both are fine. On the one hand, NaCl is pretty hardcore DJB-ware (poly1305-aes, salsa20...), and normally I'd err on the side of conservatism. OTOH, it's also really really good DJB-ware, so I can't blame anyone for using it :)
I personally prefer sodium's Python versions. Whether to deliver libraries in the Python packages or to use shared libs remains an open problem: I am somewhat convinced the former's better, but I can certainly see the cases where it'd go wrong.
I've watched your pycon talk twice. It makes this stuff pretty accessible. The only thing I had a hard time following was your description of a hash length extension attack. I had to go and look it up afterward. Otherwise, a very good talk.
It’s very early now, so I think one of the most helpful things would be to look at the API discussions and tell us if something doesn’t make any sense to you. That would mean that a) the API is bogus or b) the docs/rationale are insufficient and something needs to get fixed.
As for the library problem, I'm teaming up with Alex Gaynor, Hynek Schlawack, Donald Stufft and Christian Heimes to write a better cryptography library for Python: https://github.com/alex/cryptography
New contributors very welcome!
On the other side of the coin, I agree that cryptography education is very lacking. That's why I presented Crypto 101 last year at PyCon, which was an attempt at teaching just enough crypto in 45 minutes. Obviously, that's pretty near trying to square the circle: it's a best-effort attempt, but that's about it. (If you're still interested, the recording is available for free: http://pyvideo.org/video/1778/crypto-101)
That's why I'm also writing the book version of Crypto 101. Early draft stages right now, but getting there. After listening to tptacek say it so many times, I've come to agree: if you want to teach crypto well, you probably want a bunch of exercises that teach you how to break it. As a result, both the exercises and the text-adventure game I'm adding to the book rely on breaking real-world crypto: everything from bad password storage to MITMing SSL handshakes to stream compression :)
Additionally, a significant portion of the book will be devoted to when (i.e. rarely ;)) and how to write crypto. I would hope that your journey would've been a lot less painful if you had had access to the book. Perhaps for future programmers? :)
If you'd like to review, there's a reviewers mailing list, feel free to shoot me an e-mail explaining why you think you'd be a good reviewer, and I'll probably sign you up :)