Comments on: Multi-threaded gzip

By: Babak Farrokhi

Babak Farrokhi — Sat, 06 Aug 2011 04:59:17 +0000

I am afraid it does not.

By: Babak Farrokhi

Babak Farrokhi — Fri, 10 Sep 2010 10:34:45 +0000

For CentOS 5.x x64 I believe you may use the RPM from here: http://rpmfind.net//linux/RPM/epel/5/x86_64/pigz-2.1.6-1.el5.x86_64.html

As for tar, I am not sure if any parallel implementation exists.

By: Babak Farrokhi

Babak Farrokhi — Fri, 10 Sep 2010 05:33:52 +0000

Well, pigz will do that.

By: John M

John M — Thu, 09 Sep 2010 23:22:02 +0000

Speaking of tar are you aware of any parallel implementation of tar ?

By: John M

John M — Thu, 09 Sep 2010 23:18:29 +0000

excuse me… I meant gzip…not tar…

By: John M

John M — Thu, 09 Sep 2010 23:17:52 +0000

Ok great, just want to be sure. I found another archive tool that was multithreaded but due to the nature of the way the archives were made it would only extract archives made via single thread in a single thread mode.

Here it is:
http://www.linux.com/archive/feature/126412

“One caveat with pbunzip2 is that it will only use multiple cores if the bzip2 compressed file was created with pbzip2″

So pigz is a drop in replacement for tar? I’m an above average novice with Linux. Any info anywhere to help me get it installed on Centos 5 x64 ?

By: Babak Farrokhi

Babak Farrokhi — Thu, 09 Sep 2010 08:48:16 +0000

Its what exactly “pigz -d” or “unpigz” does.

By: John M

John M — Thu, 09 Sep 2010 05:37:12 +0000

I’m curious if pigz will utilize mutliple cores when decompressing archives that were compressed using gzip(single core)…

By: MM

MM — Fri, 06 Aug 2010 12:11:34 +0000

Thanks, really good tool – I often compress 3-5GB files on 8-core machine and this tool speeds it up a lot!

By: IT_Architect

IT_Architect — Wed, 08 Jul 2009 18:42:14 +0000

I had plenty of time on my hands while I was compressing a bunch of large files. It seemed like my Nehalem was doing about as good as Core 2 Duo. My curiosity got me checking the monitor wondering if gzip was smart enough to thread. Apparently not. The monitor showed I was using one core. I Googled and happened on your post. I appreciate you posting this. I will have to check it out.