That's good to know because, for important things, I test the archive before throwing the original data away.
Not to single out zstd but it's a good opportunity to be reminded: if you make backups, test your backups! A bug like this can also be introduced, not only fixed. I'm not saying it's likely, or that you're likely to be affected by it, but for things you care about, just opening the backup once a year and spot checking some files is not a lot of work for a lot lower risk compared to never checking it. Most backup tools will also let you do checksum validation in an automated fashion, but I prefer to also manually open it and check that it truly works (not that a partial archive has no errors, for example).
> This [affects] the block splitter. Only levels using the optimal parser (levels 13 and above depending on the source size), or users who explicitly enable the block splitter, are affected.
So if you use, for example, zstd -9 (I didn't know it went higher, at least not at somewhat reasonable speeds) or below, then you should always have been fine unless you called the block splitter yourself explicitly (or, perhaps, if your backup software does that for you? It sounds like something relevant for deduplication but I'm not sure what this feature is exactly).
> The block splitter confuses sequences with literal length == 65536 that use a repeat offset code. It interprets this as literal length == 0 when deciding the meaning of the repeat offset, and corrupts the repeat offset history. This is benign, merely causing suboptimal compression performance, if the confused history is flushed before the end of the block, e.g. if there are 3 consecutive non-repeat code sequences after the mistake. It also is only triggered if the block splitter decided to split the block.
If I understand it correctly, the bug triggers under circumstances where the data causes the splitter to split at exactly 2^16 and then doesn't flush the block, and one example where it doesn't do that is if any part of the next 2^17 bytes (128K) is compressible? Not sure what a "repeat code sequence" is, my lz77-geared brain thinks of a reference that points to an earlier repetition, aka a compressed part.
To begin with, it requires the distance between 2 consecutive matches to be exactly 65536. This is extremely rare. Mountains of files never generate such a situation.
Then it needs to employ a repcode match as the match following the 65536 literals. Repcode matches are more common after short literal lengths (1-3 bytes). Needless to say, 65536 is far from this territory, so it's uncommon to say the least.
Finally, the block splitter must be active (hence only high compression modes),
and it must decide to split the block exactly at the boundary between the literals and the repcode match.
So this is not 0, since Google found a sample, but all these conditions _together_ have an astronomically low chance to happen, as in competitive with winning the Powerball jackpot. I wouldn't worry so much for my own dozens of archives.
> That's good to know because, for important things, I test the archive before throwing the original data away.
Compression corruptions are worse then regular corruption due to the cascading impact. A corrupt sector can be replaced inline but a corrupt compressed file will generally destroy everything downstream from the error.
Big +1 to actually verifying the round trip. Backups that aren’t tested through an actual restore are like theoretical war plans vs reality. No plan survives first contact with the enemy.
That would be a nice extra benefit, besides the speedup from being multithreaded. (I assume zstd also does multithreading but for those stuck with gzip, this is a drop-in replacement.)
Edit: bzip2 apparently does the same, "bzip2 compresses files in blocks, usually 900 kbytes long. Each block is handled independently. If a media or transmission error causes a multi-block .bz2 file to become damaged, it may be possible to recover data from the undamaged blocks in the file." (--man bzip2)
No, as far as I know, the pigz blocks, not to be confused with deflate blocks, are still compressed by referring to the preceding uncompressed data even if it belongs to another pigz block. Therefore errors would still propagate indefinitely in the worst case.
Other gzip variant formats like bgzip also make the chunks compressed in parallel completely independent. This results in ~3% worse compression ratio depending on the use case.
Note that another problem with bit flips and other errors in compression formats is that most decompression tool will simply quit on the first error even if the rest of the data could still be recovered.
Yes, bz2 is also more robust against errors because of the independent blocks.
One thing to note is that DEFLATE (underlying algorithm that Gzip uses) doesn't indicate the length of blocks, or make it easy to figure out where they start/end without decoding everything proceeding it.
This is likely why pigz can only parallelize compression, but not decompression.
But even if you could identify the block boundaries up front, DEFLATE doesn't reset the LZ77 window on a new block, so corruption could still seep through to the end.
Attempt to access the file using mmap(2), rather than I/O system calls (read(2), write(2)). Currently, use of mmap(2) is attempted only for a file opened for reading.
I'm very skeptical that this actually makes things faster. For one, zstd is definitely going to be CPU bound rather than I/O bound. But even if it wasn't, there shouldn't be a performance benefit to reading a file sequentially using mmap compared to reading it sequentially using read, which you can easily verify yourself by writing a small program that just reads files both ways.
Not sure but Naïve Bayes? It might be pointing out that their inputs used in the fuzzing is statistically independent and random that it wouldn't catch some edge cases in real world?
I don't know if it's related, but one time I had corruption too. I was playing with large sparse (ntfs) files and used zstd for temporarily store those. It was probably low compression settings and I couldn't reproduce, so I'd thought I messed up something.
oh, I remembered, it was not all zstd fault. Somehow because zstad read not only the allocated space but the whole fole, the zeros too, windows tried to allocate the whole file in memory, which was not enough, and zstd did not give (outofmemory) eror message. (It was not ntfs compressed, only sparse!) Later (maybe windowsupdate happend) tried to reproduce, but worked with larger file with memory.
I don't know zstd how could handle the outofmemory, but the sparse reading might not be perfect.
OpenZFS uses 1.4.5 currently; the bug was introduced in 1.5.0. So no.
OpenZFS does support up to level 19, though I'd generally advise not using it for many reasons, so if it were running the newer version, it could be affected.
That's good to know because, for important things, I test the archive before throwing the original data away.
Not to single out zstd but it's a good opportunity to be reminded: if you make backups, test your backups! A bug like this can also be introduced, not only fixed. I'm not saying it's likely, or that you're likely to be affected by it, but for things you care about, just opening the backup once a year and spot checking some files is not a lot of work for a lot lower risk compared to never checking it. Most backup tools will also let you do checksum validation in an automated fashion, but I prefer to also manually open it and check that it truly works (not that a partial archive has no errors, for example).
Anyway, details of the bug are here: https://github.com/facebook/zstd/pull/3517
> This [affects] the block splitter. Only levels using the optimal parser (levels 13 and above depending on the source size), or users who explicitly enable the block splitter, are affected.
So if you use, for example, zstd -9 (I didn't know it went higher, at least not at somewhat reasonable speeds) or below, then you should always have been fine unless you called the block splitter yourself explicitly (or, perhaps, if your backup software does that for you? It sounds like something relevant for deduplication but I'm not sure what this feature is exactly).
> The block splitter confuses sequences with literal length == 65536 that use a repeat offset code. It interprets this as literal length == 0 when deciding the meaning of the repeat offset, and corrupts the repeat offset history. This is benign, merely causing suboptimal compression performance, if the confused history is flushed before the end of the block, e.g. if there are 3 consecutive non-repeat code sequences after the mistake. It also is only triggered if the block splitter decided to split the block.
If I understand it correctly, the bug triggers under circumstances where the data causes the splitter to split at exactly 2^16 and then doesn't flush the block, and one example where it doesn't do that is if any part of the next 2^17 bytes (128K) is compressible? Not sure what a "repeat code sequence" is, my lz77-geared brain thinks of a reference that points to an earlier repetition, aka a compressed part.