Keccak's (pronounced: ketchup) team includes Joan Daemen, who was also on the Rijndael team, giving him a hand in both AES and SHA-3.
Keccak's structure is simple and markedly different from that of MD4, MD5, SHA1, and the SHA-2 family, all of which share a common design pattern called Merkle-Damgard (MD). MD-style hashes chunk their inputs into blocks and tag the last block with a length; they then run in a manner similar to a block cipher, where each block is combined with the previous block after the hash function core is applied. The resulting output of these MD-style hashes is literally the internal state of the hash at the last block; you can take most SHA-2 hashes, for instance, and "feed them back" to the SHA function with more data to generate a continued hash. This gives rise to a common crypto protocol flaw called a "length extension attack".
Unlike MD, Keccak uses a design called a "sponge function". It's called a "Sponge" because it splits hash into two distinct stages: one in which the hash function "absorbs" data, applying the hash core function to permute an internal state, and another in which the hash is "squeezed" to produce output (which further permutes the state). Somewhat like a cryptographic PRNG, the security of the hash function is delegated to the confidentiality of the internal state, which isn't disclosed by producing the hash.
Furthermore, Keccak's Sponge design derives security by only allowing inputs to directly influence a subset of the internal state bits. Like MD hashes, the inputs are chunked into blocks and fed to the hash algorithm. But each block fed to the hash affects only that block's worth of internal state bits; the remainder of the hash's state (called the "capacity bits") are mixed in with the "outer" bits during the application of the hash core. Here's a picture that may tell the story better:
Notice how the "M" bits of the input hit only the "r" bits of "outer" state in the hash, while the "c" bits of "capacity" are used only during the "f" function.
If all it does is prevent length extension attacks, then there are much simpler and less risky ways to do that (i.e., MD variants).
Also, your explanation of the sponge structure omits the real difference between it and MD: It is a transform that turns a non-compressing function (f in that diagram) into a compressing function. MD, on the other hand, starts with a fixed-input-size compressing function and extends its domain.
By the way, what do you mean by "Furthermore, Keccak's Sponge design derives security by only allowing inputs to directly influence a subset of the internal state bits."? That's as true for an MD-type construction as it is for a sponge construction. In fact, it's a crucial fact that allows us to build a reduction from, say, the collision-resistance of MD[f] to the collision-resistance of f.
Regarding the bits exposed to inputs in Keccak, I read the claim in the same manner as the claim that CTR is more side-channel resistant because attacker ciphertext bits never hit the AES core; here further margin is given by the additional capacity bits. That's my attempt at exposition from the Sponge paper. You would know far better than I would, though; I'm a tester, not a cryptographer.
Regarding length extension, strong disagree; we see the SHA functions routinely abused this way.
BLAKE and Skein inject a message block by first running it through the block cipher (as key in BLAKE, as plaintext in Skein), and then XORing the result with the message block to get a new state. JH XORs message block into one half of the state before permutation, and into the other half after it. Keccak XORs message block into a part of state before permutation, and that's it.
Oh great, for the next year every tech meeting I go to is going to sound like five people with terminal bronchitis coughing themselves to a standstill as everyne name drops the encryption scheme they want and cannot agree how to pronounce it. I wish Schneier had won .
If Keccak isn't vulnerable to a length extension attack, is using it to hash a secret key concatenated with a message a good MAC algorithm? If not, why not?
At some points throughout the competition, the determination criteria used by NIST were somewhat unusual, with the decision committee stating that
"We preferred to be conservative about security, and
in some cases did not select algorithms with exceptional performance, largely because something about them made us
“nervous,” even though we knew of no clear attack against the full algorithm."
My impression is that people, in the absence of obvious (to them) differences in security, will choose a cryptographic function based on speed, which is simple to understand and quantify.
Going by the eBASH benchmarks, software implementations of Keccak are mostly the same speed as SHA-2, and for some CPUs it's significantly slower.
It does have better throughput/area ratio in hardware, but chip manufacturers are not early adapters.
With that in mind I can't see any major push for general SHA-3 adoption unless the SHA-2 family is broken.
Still, I'm looking forward to the reading the report.
"Keccak, created by Guido Bertoni, Joan Daemen and Gilles Van Assche of STMicroelectronics and Michaël Peeters of NXP Semiconductors"
I don't think there will be any issue with regards to some early adopters. Maybe even a concideration with regards to common hardware usage like smartcards at some level.
But whilst your right about SHA-2 not being broken and making the rush to SHA-3 moot, it does not stop marketing types getting sales over SHA-2 only complient. In that it will be marketing and drive for sales and standing out that will drive it into market with all things being as they stand. Besides, everybody likes a plan B to fall back upon.
Surely, as tptacek's professional life (you know the one here on HN) is dedicated to proving that the security of the encryption algorithm is mostly incidental to the security of the system as a whole, then I would like the next competition to be about producing reference implementations or similar. Sort of like keyczar but down to the point where all browsers simply incorporate the same approved crypto libraries and Apis, securely use areas of memory without file handles anywhere in the background and all the other good stuff that takes us into a 80% done by default world.
I am not saying this isn't good stuff, but as someone struggling to implement a user login properly it would be much nicer to have a template to follow. (and he waits for someone to point to the template ...)
A user login, you say? First, find bcrypt or scrypt bindings for your programming language of choice. Here's one for Python, with some very concise example code:
Store the hashed password that bcrypt gives you in a database somewhere. Make sure your login goes over HTTPS. I think that should cover the basics; do you have any specific questions?
What about handling single sign on for multiple applications - what do I put in the cookie if anything?
If I am encrypting a cookie what scheme? What key rotation? How many failures of a cookie to decrypt is suspicious
What about password reclamation - do I keep the answers to their grandmothers maiden name in the same database? What about adding a new email address to the list of ones I will mail a password reset token to - do I force them to prove ownership of the first email address before adding the second ? Or will a SMS to their phone do?
I am sure I am missing a hundred items and none of them are to do with bcrypt vs scrypt. Thus is hard stuff and I am avoiding most of it with the magic words "Mozilla persona" but I still don't know how to do sso well. I can do not badly but not well.
Edit: I am being a bit snipy and I apologise but I know I am in deep waters and would be happier if NIST were producing reference implementations. Perhaps actually they have - there is kerberos. If you want a key visit room 303 with your passport
I don't think you're going to see a reference implementation of resetting a password via SMS from NIST any time soon. They make standards for everybody. As soon as you start using words like database and email, you are ruling out lots of applications that don't have those features.
Sounds like I misinterpreted what you were asking. Yes, those are some tricky problems. It looks like our best choice for a longer-term answer is Persona, but it'll be a while before it's as slick as it should be. :-(
Unless Persona drastically changed in the last couple months, it seems to fundamentally misunderstand the problem of user authentication by assuming email addresses are both canonical and authoritative.
Most people keep an email address for years and that's a lot more than own their own domain.
I simply don't know what would work better than an email address as a virtual identifier - if u have a suggestion please (seriously I want to know) say
edit: wow am I in a bad mood. sorry! I would still like to know if there is a better answer than email addresses - but more politely. Cheers
When people decide to change their email address, they don't want to lose their accounts, and when someone else gets their old email address, they shouldn't have their accounts compromised. Seriously: even just "username" is better than email address. All of the advantages of using an email address as the primary key for an account are either not true or actively dangerous once you realize how little they mean to the vast vast majority of "normal people".
Facebook is a reasonable example: they allow you to use email addresses to log in, but your account is not tied to your email address. Ever since nearly forever, they have let you add new email addresses to your account, and you can remove old ones. They also seem to have some schemes in place for mitigating "I lost access to my University email address, and someone else got it" (which one would imagine to be nigh unto endemic for their use case).
Changing addresses can be dealt with just as it is today:
Let logged in users add additional email addresses to their accounts.
Other people getting your old email address is a bit worse. The easy solution is for providers to not reuse names. You could extend the protocol with a version token, so a provider could say that new bob is different from old bob, and shouldn't be able to log in to old bob's account.
Face books use case is a pathological case from universities back when a 10MB hard disk cost more than a professor. Universities do reuse email addresses but that's policy not need - and even universities can change policy.
Username is horrific as a global identifier - I hate easily.co.uk every time my browser cache gets cleared.
Adding new addresses to an account works and really ISPs ought never reuse emails. A new RFC perhaps?
I don't think I understand Schneier's point here. This is only the second time (AFAIK) that NIST has held a contest like this to standardize a crypto primitive. In the first contest, there was a clear and urgent need to retire DES. In the case of SHA-2, it was clear going in that there wasn't a "speeds and feeds" problem with SHA so much as there was a gathering unease about the basic design of the family of hashes from which SHA-2 is derived: MD4 and MD5 are broken, and SHA-1 isn't looking long for this world. Meanwhile, hash functions are not as well studied as block ciphers and number theoretic crypto, and the pace of development in hash research is increasing.
There is value simply in diversifying the gene pool of hash functions.
I think it was just a case of Schneier being a bit of his usual contrarian self. But is he correct in his assertion that the security of SHA-3 is not going to be much greater than that of SHA-2? (I don't mean that as a leading question.)
What does "greater" mean? I buy into the idea that the attack that breaks SHA-2 is unlikely to break SHA-3, and vice versa, and that's apparently a key design goal for the standard.
"greater" here means that we would think that SHA-2 is easier to break than SHA-3. Technically speaking, it's a guess, but I think that the general consensus is that it is more likely that SHA-2 will be broken before SHA-3.
Note that Schneier hedges in a very similar matter about AES, suggesting in _Practical Cryptography_ that XSL attacks made him dubious of AES's long-term security --- or at least, that if you're "paranoid about your data", that you should use Serpent instead of AES/Rijndael.
Keccak's structure is simple and markedly different from that of MD4, MD5, SHA1, and the SHA-2 family, all of which share a common design pattern called Merkle-Damgard (MD). MD-style hashes chunk their inputs into blocks and tag the last block with a length; they then run in a manner similar to a block cipher, where each block is combined with the previous block after the hash function core is applied. The resulting output of these MD-style hashes is literally the internal state of the hash at the last block; you can take most SHA-2 hashes, for instance, and "feed them back" to the SHA function with more data to generate a continued hash. This gives rise to a common crypto protocol flaw called a "length extension attack".
Unlike MD, Keccak uses a design called a "sponge function". It's called a "Sponge" because it splits hash into two distinct stages: one in which the hash function "absorbs" data, applying the hash core function to permute an internal state, and another in which the hash is "squeezed" to produce output (which further permutes the state). Somewhat like a cryptographic PRNG, the security of the hash function is delegated to the confidentiality of the internal state, which isn't disclosed by producing the hash.
Furthermore, Keccak's Sponge design derives security by only allowing inputs to directly influence a subset of the internal state bits. Like MD hashes, the inputs are chunked into blocks and fed to the hash algorithm. But each block fed to the hash affects only that block's worth of internal state bits; the remainder of the hash's state (called the "capacity bits") are mixed in with the "outer" bits during the application of the hash core. Here's a picture that may tell the story better:
http://tinyurl.com/cryptosponge
Notice how the "M" bits of the input hit only the "r" bits of "outer" state in the hash, while the "c" bits of "capacity" are used only during the "f" function.