Interesting, I've never heard of it before. I was aware that there were better standards than zip, however adoption was always an issue with licenses. If this is as open, is it purely an issue of wide adoption?
We're stuck in the C++ paradigm (I'm sure there is a better name), where everyone agree this isn't "ideal", that there is better, but not widely adopted enough?
Unless you have to use deflate/zlib/gzip/zip for compatibility or you're very constrained in amount of memory, you should always prefer zstd over deflate - it's always both faster and compresses better, so there is no tradeoff and using deflate is just wasteful.
Google's brotli is a close competitor and got supported in Chrome first, so HTTP uses that more than it uses zstd.
zstd has a very wide range of compression levels to tradeoff compression ratio for CPU time, much wider than the difference between gzip -1, gzip -6 (default) and gzip -9. zstd -3 is default. zstd -1 is very fast, zstd -19 compresses really well.
zstd also has a long range mode, for archives that contain the same file a gigabyte away. gzip's window is only 32KB in size, so it can't notice any repetitions more than that far away.
LZ4 by the same author as zstd aims at even faster compression, at the cost of worse compression ratio. It makes sense to use over fast network. Google snappy and LZO are the usual competitors.
When you want better compression than zstd (at the cost of CPU time, both during compression and during decompression), use PPMD for natural language text or source code and use LZMA (xz) for other things.
This is more of a replacement for deflate / gzip. You can use zstandard inside a zip file using compression type 20, as of zip v6.3.7 (2020), but I suspect not much software will be able to decompress it.
Seen a number of gzip replacements go by, like .bz2 (smaller but requires lots of CPU), .lzma and .xz (strictly an improvement over .bz2), and some others. Zstandard is a fairly solid improvement over these for similar use cases so I think it will get adopted. There's also the "extreme" compression algorithms like .lz4 (extremely fast, don't care about compression ratio) or PPMD (extremely slow, care deeply about compression ratio). My sense is that a large number of projects migrated to .bz2 and then to .xz over the years, and maybe we'll see more .zstd cropping up.
> [going from xz/lzma to zstd] yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.
I think personally, the biggest factor for using zstd is that compression settings don't matter for the decompression speed.
Doesn't matter if I used zstd-fast or zstd-19 the read speed is the same. Hence a lot of linux distros adopt it for packaging; they spent a few days of CPU time compressing their packages on zstd-19 (archlinux does IIRC), the user couldn't tell that it's been hard compressed like that.
Your questions are answered in the first sentence of the article.
> Zstd or Zstandard (RFC 8478, first released in 2015) is a popular modern compression algorithm. It’s smaller (better compression ratio) and faster than the ubiquitous Zlib/Deflate (RFC 1950 and RFC 1951, first released in 1995).
We're stuck in the C++ paradigm (I'm sure there is a better name), where everyone agree this isn't "ideal", that there is better, but not widely adopted enough?