Tar is a standalone archive format, you can create tar archives with the tar command and not use any compression utility at all. Here is an example that creates an uncompressed tar archive directly with no compression:
tar -cf $directory.tar $directory
You can also pipe the created archive to stdout instead of to a file if you want to:
tar -c $directory | wc -c
# send to a remote system
tar -c $directory | ssh dd of=~/$directory.tar
You can actually compress the archive by just piping it to a compression command:
tar -c $directory | zstd --stdout > $directory.tar.zst
The above pipeline is probably very similar to what tar is doing internally when you pass "--use-compression-program".
So in your case using "pigz -0" is totally useless since tar creates an uncompressed archive by default. You can totally omit the "--use-compression-command" flag to do what you want.
I know about all that. My mistake was thinking that I could take advantage of the multicore nature of pigz even when no compression is being used (I'm not sure if pigz only uses multiple cores for the bzip algorithm, or if it somehow could speed up the packing as well). I'm pretty sure that disk is the bottleneck in most cases, but I wonder if that's not the case always, and a single core could become the bottleneck in very fast IO devices. I want to verify this and test it.
Oh yeah in that case tar itself is still creating the archive with a single thread, the compression gets applied "after" tar creates the archive almost exactly like in the pipeline in my comment.
By "after", I don't mean the whole archive is created before the compression part runs, but rather that tar can stream the the archive to stdout as it's being created. Tar needs to be able to do this since it was intended for tape drives, and seeking around on a tape drive is extremely slow.
It's actually pretty interesting that we still use tar so widely even though tape drives are not in common use! A new archive format that supports random access during reads (and writes maybe?) would be pretty interesting, but tar works pretty well so there isn't that much of a reason to create alternatives.
Tar is a standalone archive format, you can create tar archives with the tar command and not use any compression utility at all. Here is an example that creates an uncompressed tar archive directly with no compression:
You can also pipe the created archive to stdout instead of to a file if you want to: You can actually compress the archive by just piping it to a compression command: The above pipeline is probably very similar to what tar is doing internally when you pass "--use-compression-program".So in your case using "pigz -0" is totally useless since tar creates an uncompressed archive by default. You can totally omit the "--use-compression-command" flag to do what you want.