> Intel did add a global state bit. It just isn't useful because you can't modify it.
You mean the CPUID bit? That's free. Toggling denormals isn't.
> Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.
Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do.
I mean the bit to toggle denormals, not the one to identify support for the opcodes.
Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero.
If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.
> I mean the bit to toggle denormals, not the one to identify support for the opcodes.
bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)?
> If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.
No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets don't give a damn about mathematical purity[1]. In contrast compatibility matters because ML doesn't give you any guarantee that it's not depending on the behaviour at small values, and minor differences in rounding does cause issues. For example, Leela Chess Zero had difficulties with reproducibility because different GPUs round floats differently.
It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to zero")
Bfloat16 obviously can handle denormals. The encoding is possible. There would be no need to handle the issue if the encoding did not exist.
As hex, these would be denormal: 0x0001 to 0x0080, and 0x8001 to 0x8080. It's the same as plain old 32-bit IEEE, with half the bits lopped off.
ML is all about difficulties with reproducibility. I don't see a reason to get upset about denormals when a 3D-printed turtle can be confused with a rifle.
> It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to zero")
You mean the standard ones for normal floats? You certainly wouldn't want to reuse that for bfloat16s.
> ML is all about difficulties with reproducibility. I don't see a reason to get upset about denormals when a 3D-printed turtle can be confused with a rifle.
These are different things, despite the similarity in terminology.
Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.