Hacker News new | past | comments | ask | show | jobs | submit login

If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2

http://compression.ca/pbzip2/

*edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.

    megs
    3781    test.tar
    3041    test.tar.zstd (default compression 3, 3.5s)
    3170    test.tar.bz2 (default compression, 8 threads, 40s)



bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available.

For everything else, there's zstd (also natively multithread)



Decompression is multithreaded by default. Compression with an argument. However it is built-in.


on the other hand, bzip2 is pretty much obsoleted now by xzip


Also see: http://enwp.org/zstd

parallel-friendly, trades off compression level for speed


What is xzip? are you talking about xz?



The author of lzip has pointed criticism for the design choices of xz.

I generally use lzip for data that is important to me.

https://www.nongnu.org/lzip/xz_inadequate.html




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: