Chris's Wiki :: blog/sysadmin/GzipNotFast Commentshttps://utcc.utoronto.ca/~cks/space/blog/sysadmin/GzipNotFast?atomcommentsDWiki2018-04-04T19:30:47ZRecent comments in Chris's Wiki :: blog/sysadmin/GzipNotFast.By Twirrim on /blog/sysadmin/GzipNotFasttag:CSpace:blog/sysadmin/GzipNotFast:bdb089b064898271b776e3b150237acc96c76a1dTwirrim<div class="wikitext"><p>I'd agree with ScottG. I've found --fast with easily compressible content like emails, gets you pretty close in size, with a fraction of the effort.</p>
<p>I do leverage pigz for some stuff, but where I do, I tend to shove a "nice" in front of it and let the scheduler figure out priorities.</p>
<p>I've been really impressed with zstd's performance, and Facebook has moved it to dual BSD / GPLv2 licensing which means it's starting to crop up all over the place (including in the linux kernel): <a href="https://github.com/facebook/zstd">https://github.com/facebook/zstd</a></p>
</div>2018-04-04T19:30:47ZBy ScottG on /blog/sysadmin/GzipNotFasttag:CSpace:blog/sysadmin/GzipNotFast:dc81c372f9210bcf6ae2a8e647ad387c2871727fScottG<div class="wikitext"><p>In my experience, performing a "gzip --best" or even a default "gzip" is often CPU limited, but performing a "gzip --fast" is fast enough to be limited only by disk or network throughput.</p>
<p>I suggest that you simply change the word "best" to "fast" in your amanda.conf. I expect that your backups will be a little larger, but much faster.</p>
</div>2018-04-04T13:07:49ZBy Andreas on /blog/sysadmin/GzipNotFasttag:CSpace:blog/sysadmin/GzipNotFast:38034983455c5d1a6d497ba962506a7d90ccf87fAndreas<div class="wikitext"><p>Hi,</p>
<p>I just analyzed this myself last week (comparison non-parallel, parallel versions of lzop, gzip, bzip2, xz) and I achieved the best throughput with the parallel gzip compression (pigz) limited to subset at most to 75% of all cores with entry level compression (-1).</p>
<p>I compressed Oracle Database dumps and the pigz -1 on 16 threads yielded over 100 MB/sec, whereas gzip only yielded 10 MB/sec, lzop reached 25 MB/sec (yet there is no parallel version).</p>
<p>This is test was mainly focused on throughput, not best compressibility, but only for the sake of completeness: pxz -9 yielded a compressed size of 120 MB (only 0.3 MB/sec, 242s), whereas the fastest method (pigz -1) yielded 267 MB (116 MB/s, 2,25s)</p>
<p>Best,
Andreas Steinel</p>
</div>2018-04-04T12:40:26Z