The traditional (yet very popular) gzip is a single-threaded application from the single-processor/single-core hardware era. Its just fine if you are compressing a few files occasionally, but it become a great pain when you are compressing 32,000 files on an 8-processor server and you suddenly figure out that you are using only 1/8 of your total processor power. Which means you should wait 8 times longer than if you could use all processing power on your machine. I encountered such case in which I should wait about 40 minutes to compress hundreds of gigabytes of a few thousand files, using traditional gzip, while I had one processor doing the whole job and 7 other processors were sitting idle.
So I thought there should be a way to speed-up the process. The most simple method I could use was to open up multiple terminal windows and run parallel copies of gzip, each of them to compress a specific set of files. While this method worked for me, but I was wondering why the gzip itself doesn’t support multi-threading.
The solution: pigz
I came across pigz after searching the internet for a multi-threaded gzip replacement. pigz is a drop-in replacement for gzip that supports parallel compression/decompression when multiple files are involved.
Figure 1: Running “systat -iostat 1” on a FreeBSD 7.2 machine running pigz
Using pigz, I could exploit more than 70% of my processing power. pigz also maintains compatibility with standard gzip command line parameter and supports all switches while adding “-p” command to specify maximum number of compression threads.
I am type of person who likes to build everything from source code in FreeBSD to get better performance and other customizations. It has become a habit to play with ports tree and system source code and now I believe I have my own template for various server platforms.
One of the most important parts of each configuration template is the /etc/make.conf file. This is were you can change general behavior of the build system. This file is where you actually say which compiler optimizations should be used or what options has to be considered as defaults. Good news is that ports collection as well as the operating system itself honor these configuration.
Here is how a typical make.conf on one of my boxes look like:
CPUTYPE?=nocona
CFLAGS= -O2 -pipe -fno-strict-aliasing
COPTFLAGS= -O2 -pipe -funroll-loops -ffast-math -fno-strict-aliasing
KERNCONF= SERVER GENERIC
OPTIMIZED_CFLAGS= YES
WITHOUT_X11= YES
BUILD_OPTIMIZED= YES
WITH_CPUFLAGS= YES
WITHOUT_DEBUG= YES
WITH_OPTIMIZED_CFLAGS= YES
NO_PROFILE= YES
BUILD_STATIC= YES
The CPUTYPE variable tells gcc to optimize generated binary code for specified processor. In this case I am using 64bit Xeon processor architecture and “nocona” is the correct CPUTYPE to use. You may want to use “pentium4” on a typical Intel P4 CPU. A list of possible CPUTYPE values can be found in the sample make.conf file located at /usr/share/examples/etc/make.conf.
Continue reading 'make.conf in FreeBSD'»
There are many buzz out there about Yahoo Music Engine, ml_ipod and many other softwares that work with ipod.
Unlike Yahoo! Employees, YME could not survive a full day on my computer. It was one of most idiot piece of software I have ever seen.
Beside being super slow and its ugly interface, it has detected the iPod as soon as I hooked it up to USB port. Then I started browsing my music library and selected some albums and synced up with the iPod. So far so good. But when I selected the albums on my iPod, it turned out the iPod is not able to play them. Guess why?
YME does not know iPod can not play WMA. That simple. The solution:
1- Uninstall YME from your computer, remove remaining file from its directory and clean up your registry.
2- Re-format your iPod completely. Because it stinks.
3- Use your iTunes to update your iPod. iTunes is the logical choice. It is supposed to be the best software for this reason (however I don’t like it, but at least it works fine)
Alexandre Dumont has pointed out some hints about “rc” suffix.
IMHO, rc has different meanings. It could be “run control“, “runlevel control”, “resource configuration” or “resource control”. But AFAIK there is no clearly defined meaning for rc anywhere.
If you are sick and tired of Real Player spyware, there is a special edition for BBC with much less spy activity.
You can download it from here and enjoy watching real videos or listening to real radios online.
Still don’t trust real? Get Real Alternative.
Update: The software link was missing. Fixed.
I am happily running BIND 9.2.3rc4 on four DNS servers around the world, blocking DNS wildcards on .net and .com zones. delegation-only feature worked fine out of the box. The only tricky part of the process was installing BIND9 on FreeBSD which has bind8 installed as part of operating system.
Installation from ports tree only installs binaries and documents. So you should perform all configurations manually.
Continue reading 'Blocking verisign DNS wildcards'»
Just a tip for sysadmins which are tired of ^M characters in their text files:
$ col -bx < input_file > output_file
Easy, huh?
Comments (7) 
Themocracy