The big problem in this post seems to be compiling via C (and the awkward GPU extensions thereof), which has lousy Float16 support. LLVM has adequate support for at least representing Float16s. Because of this (and a lot of work by a lot of people), Julia has pretty good support for Float16. If you're running on hardware with native Float16 support like a GPU, it works and is fast; if you're running on hardware without native Float16 support, operations are implemented by converting to Float32 and back, which is slow but gives the same results. So you can run the program either way and get the same results and it's fast if your hardware has native support. Same deal with BFloat16 [1], which is the native 16-bit floating point type on Google's TPUs.
For what it's worth, the Arm "alternative half-precision" format is not very different from IEEE binary16. The difference is that instead of using the maximum exponent value (0x1f) to represent infinities and NaNs, the AHP format uses it the same way as any other non-zero exponent, to represent normalized fp values. The tradeoff is that you lose NaNs and infinities, but you double the range of numbers you can represent (the max value goes from 65504 to 131008).
You have to select AHP by setting an FP config register bit, so (unlike bfloat16 vs binary16) it's a "for this whole chunk of code I am going to use this format" choice.
Alternative-halfprec format only applies for conversions (ie to/from single and double prec); data-processing operations on fp16 ignore the AHP bit and always assume IEEE format. If you convert a NaN from single/double into AHP you get a zero with the same sign as the input; if you convert an infinity you get the max/min representable number; in both cases the InvalidOp fp exception bit is set.
The upshot is that it's basically an in-memory storage format, and all the actual data-processing gets done at either single or double precision.
The article makes a lot of good points.
But there are some cases where f16 is very useful. In the context of deep learning it's frequently useful to move from f32 -> f16. This can allow you to double the size of your models in memory (system or GPU/TPU).
Since network size is often determinant of performance, doubling the number of parameters/activations in your model can make a big difference.
Some years ago I used them to transfer real-time waveforms from a medical device via USB and BLE links since they provided more than enough precision for the clinical application. The 2x bandwidth increase without resorting to compression (and its computational overhead) allowed us to meet the project specifications just by changing the type of the data array and recompiling.
Depends on what you want to do and how you interpret your data. IIRC we only needed to normalize one parameter in the host using percentages instead in the UI. For the rest everything translated without modification since the F16 covered the max/min parameter ranges.
Mind this was for a regulated medical device and therefore any change in the software was burdensome. The change didn't affect the internal calculations since it was only for the communication protocol and UI, and the compiler support allowed the minimal changes on both embedded and host source codes.
"just don't use f16" seems like the course of wisdom here.
I had a 3x 5 bit value, packed structure in something where memory pressure was severe, and it was such a bitch on a SPARC to deal with 16 bit quantities that actually running the data as two passes using more memory wound up being an immensely better approach. The Alphas would diddle yer bits any way you liked at speed, but that was clearly an aberrant ability.
Only after BWX landed - before that, you had to ensure all accesses were 4 or 8 bytes, and aligned on 4 byte boundary, then do appropriate bitmasking and shifting.
16-bit floating point (as a weight storage format, not used for math) is essential for fast Winograd and Fourier convolution on AVX-512 CPUs. See https://NN-512.com
Many of these algorithms are bandwidth- or cache-limited on modern machines, so you can get significant speedup by storing your data in fewer bytes, even if you expand it in registers before actually doing computation on it.
We're reaching a point where it's often faster to store pages in RAM compressed with a fast algo like LZ4 and to decompress them, than to simply copy from RAM uncompressed to L1 cache.
Wow, that's illuminating. I naively thought the overhead of converting between datatypes would not make this worth that much (in favor of saving cache misses). Though does this also have anything to do with the AVX512 instructions?
The result is that Winograd convolutions can achieve an effective FMA rate of twice the peak rate of the CPU.
The Winograd transform reduces the required number of FMAs by a factor of 5x, but you can only do FMAs at half peak rate (because you are bandwidth limited), so you come out ahead by a factor of 2.5x in theory (2x in practice).
If the CPU only has 32-bit ALUs (afaik all CPUs today have 32+ bit ALUs) there is no reason to "support" half-floats (other than converting floats to half-floats for the GPU which doesn't need to be fast or pretty since you do it beforehand and send it directly to the GPU from the model file format).
On the GPU on the other hand 16-bit floats are becoming the standard (the M1 GPU for instance has more 16-bit ALUs than 32-bit). With enough precision for possible resolutions/worlds and you save 2x the memory which makes it a no-brainer really.
If your CPU has vector/SIMD instructions then you might want fp16 format support for data processing so that you can operate on (say) a 128-bit vector of 8 fp16 values at once, rather than having to work with 4 fp32 values at a time. But I agree that there's a lot you can do with just load/store/conversion support.
The assertion that C doesn't support fp16 is just plain wrong. _Float16 is defined in standards committee work. There's also the __fp16 storage-only type widely supported by compilers.
The main issue is many compilers have issues on x86 platforms due to Intel's bizarre slowness in defining how to pass fp16 parameters in their official ABI.
My main experience is on clang primarily for arm64 systems, where it works just fine (give or take some terrible codegen at times if using llvm's extended vector support).
Intel support is a bit more dubious, especially if you're stuck on older LLVM versions, but __fp16 works for most purposes if all you want is storage. You just have to pass any fp16 function parameters as pointers/references to work around the aforementioned ABI issue. Architecturally intel has supported fp16 conversions necessary for __fp16 for quite a while now, so no reason for it not to work (whereas full fp16 arithmetic is restricted to some less common variants of avx512 iirc, so the distinction between __fp16 and _Float16 isn't much on most intel systems).
It should be fairly straightforward to write a quick test program and see what your local compilers can cope with.
Sure, why not. I don't think it would be that difficult, and can be emulated with single-precision just as well as fp16 for the systems that don't support it in hardware.
What are the reasons for changing the allocation of bits in bf16 vs. f16? Why are there no (few?) similar alternative allocation schemes for f32 and f64? Was IEEE's choice perfect for f32 / f64? How did they know? Why not for f16?
Does any hardware offer "configurable" bit allocation like f16[e=4,m=11]?
I think it's not so much that the IEEE f32/f64 choice was necessarily perfect so much as that it was "good enough", and so it's not worth the hardware costs of handling multiple formats or the headaches of picking a single choice that's something else. With f16 because you only have 16 bits the tradeoffs are suddenly much more sharp, because you don't have enough to both have a reasonable representable range (large exponent field) and a reasonable precision (large mantissa field). So you must trade one against the other, and it can be worth the extra hardware to support two points in the tradeoff range.
Huh? Half precision floats speeds up machine learning training and inference enormously. It's weird to me to argue that we should sacrifice hours of sliced off waiting time in favor of preserving the ability to write a new language over a weekend.
The entire blog post is literally an explanation of why implementing support for fp16 in Futhark was annoying. Let me repeat the key bit: support for fp16 in Futhark. As in, it exists, and you can use it. Right now! Nobody is stopping you from using it, or encouraging you not to use it. In fact the author literally gave you the ability to use it in his language, because clearly people want to use it. The literal second paragraph acknowledges not only that fp16 is popular in machine learning, but that there is actually two popular formats now, including bf16!
The post is actually quite relatable if you have experience in silly toolchain issues, i.e. worked on a compiler or something like that. It's all fairly straightforward. Please try to read the article next time, you might surprise yourself.
computation will be slower but writing new languages will be faster. Think of how many new languages you can implement while you wait for your computation to finish!!
The article didn't talk about f16 or anything like that. It just talks about some complications when implementing f16 support. I'm not sure what exactly you want the author not to bitch about.
But still, Java does not have unsigned integer types for 32 and 64 wide integers, just a way to treat them as such if you need it so much that you want to go out of your way. For example when you need to parse a binary with unsigned integers.
There is no way to declare unsigned 32 or 64 bit integer.
The author literally implemented fp16 support in his language. You clearly did not even read the post, which was just about the minor technical hurdles to doing so. Please learn to read next time instead of wasting everyone's time with your own useless bitching.
[1] https://github.com/JuliaMath/BFloat16s.jl