Geek Style » Performance

Multi-threaded gzip

Babak Farrokhi — Tue, 14 Apr 2009 08:22:45 +0000

The traditional (yet very popular) gzip is a single-threaded application from the single-processor/single-core hardware era. Its just fine if you are compressing a few files occasionally, but it become a great pain when you are compressing 32,000 files on an 8-processor server and you suddenly figure out that you are using only 1/8 of your total processor power. Which means you should wait 8 times longer than if you could use all processing power on your machine. I encountered such case in which I should wait about 40 minutes to compress hundreds of gigabytes of a few thousand files, using traditional gzip, while I had one processor doing the whole job and 7 other processors were sitting idle.

So I thought there should be a way to speed-up the process. The most simple method I could use was to open up multiple terminal windows and run parallel copies of gzip, each of them to compress a specific set of files. While this method worked for me, but I was wondering why the gzip itself doesn’t support multi-threading.

The solution: pigz

I came across pigz after searching the internet for a multi-threaded gzip replacement. pigz is a drop-in replacement for gzip that supports parallel compression/decompression when multiple files are involved.

Figure 1: Running “systat -iostat 1” on a FreeBSD 7.2 machine running pigz

Using pigz, I could exploit more than 70% of my processing power. pigz also maintains compatibility with standard gzip command line parameter and supports all switches while adding “-p” command to specify maximum number of compression threads.

OpenOffice.org Template problem

Babak Farrokhi — Wed, 13 Dec 2006 04:56:08 +0000

Now that I am using OpenOffice.org 2 more seriously for a while (on FreeBSD you could guess), I am badly missing Microsoft Office. And to me, its not about the functionality. Its about pre-installed templates.

This shows up when you want to create pretty presentations or professional looking text documents. Now it seems like that OpenOffice people has figured out this problem and included some new templates in their recent release.

I wish they would also make it less memory hungry. Opening a very simple spreadsheet on my workstation takes up to 130MB of memory.

Compression utilities comparison

Babak Farrokhi — Mon, 26 Apr 2004 13:17:25 +0000

There is many compression software and algorithms available today. Many people are still using zip (the oldie but goodie) while many others are cult of RAR. In UNIX, the gzip is the classic and widely used compression program, but bzip2 seems to have many fans either. Mac users are still using StuffIt since I guess they have no more choice! (Correct me if I am wrong please)

You are also using your favorite compression program with your very own custom settings perhaps, which fit the best for your needs.
Everything that we know about our favorite compression program is that it “feels” faster or “probably” makes smaller archives. But have you ever tested different programs yourself to compare them? I did.

How?

The test bench is my personal computer at home, running FreeBSD 5.2-CURRENT on Intel 1.7 GHz CPU with 512K L2 Cache and 512MB DDR memory. My OS is completely rebuilt using gcc with

Mozilla 1.7b

Babak Farrokhi — Mon, 22 Mar 2004 14:42:42 +0000

Mozilla 1.7 beta has really impressed me by its features and performance.

Though its a beta, but its very stable and I haven’t discovered any bug yet. I like the new icon set and finally the “Smooth Scrolling” feature. It also feels much faster in terms of launching and also rendering web pages.

If you are interested, there is a long changelog available.

If you are still using IE, I strongly recommend you using Mozilla or FireFox. The reason is that IE is getting too old and Microsoft is not interested in updating its browser to support cutting edge technologies. My weblog, for example, looks much cleaner in non-IE browsers, despite I am using standard XHTML and CSS2 features. Its time to switch.