pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later.
It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.
yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.
Aye, pick the right tool for the target audience. If you are the target or you know everyone else who needs to read the output will have the ability to read zstd, go with that. If not consider pigz. If writing a script that others may run, have it default to gzip but use pigz if available (unless you really don't want that small % drop on compression).
I'm a little confused by this. My copy of zstd has the option --format=gzip; does choosing this option end up using a different, slower compression algorithm?
zstandard can indeed handle standard format gzip files to create and decompress them. From the zstandard compilation options:
HAVE_ZLIB : zstd can compress and decompress files in .gz format. This is ordered through command --format=gzip. Alternatively, symlinks named gzip or gunzip will mimic intended behavior. .gz support is automatically enabled when zlib library is detected at build time. It's possible to disable .gz support, by setting HAVE_ZLIB=0. Example : make zstd HAVE_ZLIB=0 It's also possible to force compilation with zlib support, using HAVE_ZLIB=1. In which case, linking stage will fail if zlib library cannot be found. This is useful to prevent silent feature disabling.
Yes, indeed I had already read that in the man page, but I did not feel it answered my question, because in my mind, and state of naivety about compression, logically at least, a compression algorithm and final file format needn't stand in a 1-1 relation. But I suppose that ZLIB is just the DEFLATE algorithm, then?
It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.