Imagemagick

Imagemagick is a set of command-line utilities for manipulating and viewing images, the most useful of which is probably convert. convert will convert images between different formats and will apply any of a number of image transformations upon request.

Why would you use convert instead of the GIMP or Photoshop? If you're doing something as simple as changing the format of an image, you can avoid the trouble of firing up an image editor. If you're applying the same transformation to many different images (e.g. generating thumbnails), you can script Imagemagick instead of opening up all the files in an image editor. And, of course, if you need to do image manipulation on-the-fly (e.g. for an interactive web site), you'll need something scriptable.

Here's how to convert an image to a different format, say, from a JPEG to a PNG:

convert boston.jpg boston.png

To do image transformations, specify some number of transformations between the input and the output file names. Here are some of my favorites:

convert boston.jpg -resize 640x480 boston-small.jpg

Resizes the image to fit in a 640x480 box while preserving the aspect ratio.

convert boston.jpg -quality 80 boston-lowres.jpg

Changes the quality of an image (for JPEG, you can specify any number from 0 [lowest image quality, least space] to 100 [highest image quality, most space]).

convert logo.png -rotate 10 logo2.png

Rotates an image by 10 degrees.

convert logo.png -negate logo2.png

Negates the colors in an image.

convert boston.jpg -fx 'luminosity' boston-grayscale.jpg

Desaturates an image (converts it to grayscale).

You can compound operations by stringing them together:

convert in.jpg -emboss 5 -fx 'luminosity' -rotate 10 out.png

convert has more options than you can shake a stick at: everything from color transformations to special effects and commands to let you draw arbitrary shapes or text on an image. And Imagemagick also provides bindings so you can use it as a library from any of a number of languages.

How to rescue a dying hard drive

If you suspect your hard drive is dying (unusually large amounts of disk thrashing, long access times, frequent file corruption, refuses to boot), and you have a spare disk, it's not hard to clone the disk. It will be easier to recover data from the clone, since continued use of the bad disk can make things worse.

1. If the disk is your boot disk, move it to another computer. Otherwise, be sure to unmount it first.
2. Determine what device and partition the disk appears on, perhaps using mount or fdisk.
3. Use ddrescue to make an image of the disk:

   $ ddrescue /dev/sdb1 IMAGEFILE LOGFILE

   where sdb1 is the node for your source disk. ddrescue is like dd (which writes a copy of all the disk's raw data to a file), but works better for damaged disks: it fills in zeros for parts of the disk it can't read; you can run ddrescue as many times as you want, and if you provide the same LOGFILE it will attempt to fill in the gaps in the image that it didn't get before.
4. Write the image to a new disk:

   $ dd if=IMAGEFILE of=/dev/sdc1

   where sdc1 is the node for your target disk.
5. Mount the filesystem and run fsck or an equivalent. For NTFS volumes, moving the drive to a Windows computer and running chkdsk /F works wonders.

Makefiles

You may know make as a tool for building software, but it's actually a general utility for performing tasks based on a dependency graph, even if those tasks have nothing to do with compiling software. In many situations, files are processed in some way to create other files or perform certain tasks. For example:

• A .c file is compiled to make a .o file, which is then linked to other object files to make an executable.
• A .tex file is processed by LaTeX to make a PDF file.
• A file containing data is processed by a script to generate graphs.
• A web page needs to be uploaded to a remote server after it is modified.
• A piece of software needs to be installed after it is compiled.

In each case, the modification of a source file demands that some task be performed to ensure that the corresponding resource is up-to-date (i.e., it reflects the changes made to the source file).

make automates the process of keeping resources up-to-date: you tell it about what resources depend on what source files, and what needs to be done to those source files to make the resources. It then runs only those tasks that actually need to be run. No more obsessive-compulsive internal dialogues that go like this: "Did I remember to compile that program again after I was done changing it?" "Well, I don't remember, so I might as well do it again." Running a properly-configured make will compile your program if it's necessary, and do nothing if your program is up-to-date. This can save you seconds, minutes, or hours, depending on what you're doing.

make determines what to do by comparing the modification times of the resources and the source files. For example, if thesis.pdf (a resource) is created from thesis.tex (a source file), then pdflatex needs to be run if thesis.pdf doesn't exist or if it's older than thesis.tex, but nothing needs to be done if thesis.pdf is newer than thesis.tex.

To use make to manage some files in a directory, create a file called Makefile in that directory. The syntax for the Makefile is refreshingly simple. For each resource to generate (these are called targets), type:

TARGET-FILE: DEPENDENCY-FILE-1 DEPENDENCY-FILE-2 ...
       COMMAND-TO-RUN
       ANOTHER-COMMAND

Each command to be run should be indented with a single tab. Here's an example Makefile, for a system where two files are to be compiled, then linked to make an executable:

helloworld: hello1.o hello2.o
       gcc -o helloworld hello1.o hello2.o
hello1.o: hello1.c
       gcc -c hello1.c
hello2.o: hello2.c
       gcc -c hello2.c

Running $ make helloworld tells make to resolve the dependencies necessary to generate the file helloworld: make first compiles the two source files, then links them. If I were to subsequently change hello1.c and run $ make helloworld again, make would recompile that file and relink the program, but it would not recompile hello2.c.

By default, $ make (without a target name) does whatever is necessary to make the first target that appears in your Makefile (in this case, helloworld).

In fact, your make targets do not have to refer to files. You can use make to define groups of commands to be run together under an (arbitrary) easy-to-remember target name. For example, I could define a publish target which would upload my web pages:

.PHONY: publish
publish: index.html stuff.html
       scp index.html stuff.html phil@remoteserver:~/www/

(The .PHONY directive tells make to not worry that a file named publish is not being generated.) Now publishing my web page is as easy as running $ make publish.

Here are some common applications of phony targets:

• After you compile a piece of software, you can frequently install it with $ make install.
• $ make clean usually has the job of removing compiled files from a directory (so you are left with the "clean" source files). It can often be defined similarly to this:

  .PHONY: clean
  clean:
         rm -f helloworld *.o

• If a source directory contains multiple independent targets, an all target is usually created to force building the individual parts, and placed first so that $ make invokes $ make all:

  .PHONY: all
  all: part1 part2 part3

As you can see, you can use make to automate any well-defined process you're going to want to do repeatedly.

Gnuplot plots in LaTeX

I was pleasantly surprised to learn that gnuplot and LaTeX play very well with each other. Gnuplot makes it quite easy to insert professional-looking (LaTeX-worthy) graphs into your documents. To do this, set your gnuplot terminal to epslatex:

set term epslatex
set output "graph1.eps"
plot ...

When gnuplot creates your graph, it will generate not only a graph (in EPS), but also a snippet of TeX (named graph1.tex in this case) that you can include in your document to insert the graph. Then add this to your LaTeX file:

\input{graph1.tex}

The text (axes, labels, title) on the graph are all rendered in TeX, not on the EPS, so it looks very sharp. Typically you will want to wrap the above code in a LaTeX figure as well:

\begin{figure}[tbp]
  \begin{center}
    \input{graph1.tex}
    \caption{Graph caption}
    \label{graph:graph1}
  \end{center}
\end{figure}

You can control the size of the graph with:

set size 1.0, 1.0

in the Gnuplot code.

Update: you can also generate this style plot in GNU Octave. Octave is preferable in some cases because it has a few more plot options/styles than Gnuplot does, and, of course, is a full-fledged programming language.

"Being Productive With Emacs"

I taught a class during IAP 2007 titled Being Productive With Emacs. The course web site contains the slides I used, which you may use under the terms of the GFDL.

I realize that presentation slides are not the greatest resource for learning. Eventually I'll get around to revising my material from the actual class and I'll post it somewhere in a easier-to-read format.

Getting more screen real estate with x2x

x2x is a program to transfer keyboard and mouse input from one X display to another. But in X, displays can be controlled by remote mice and keyboards (using X-forwarding), so what x2x really does is let you control a desktop remotely.

One particularly neat feature of x2x is the directional mode, which can essentially make two monitors behave like a single large screen: when you move your mouse off one edge of the first display, it appears on the second, even if the displays are running on two separate computers. This is really handy if you have two computers which are near each other and you need to use both of them at the same time, or if you just want some more screen area for certain tasks (for example, writing on one display while reading on another). It's actually easy enough that you can set up this kind of sharing ad-hoc, whenever you need it.

If you can SSH from one computer to the other, then x2x is very easy to configure. Suppose you have two computers named LAPTOP and DESKTOP. Then on LAPTOP do the following:

laptop$ ssh -X desktop
desktop$ x2x -east -from $DISPLAY -to :0

Now LAPTOP's mouse and keyboard can control DESKTOP when you move the mouse off the right-hand edge of the screen. To make DESKTOP's mouse and keyboard control LAPTOP instead, do the following on LAPTOP:

laptop$ ssh -X desktop
desktop$ x2x -west -from :0 -to $DISPLAY

If neither of the computers you're using can SSH to the other, you'll need a third computer that you have access to. Suppose you want to connect the displays of LAPTOP1 and LAPTOP2. In this case, the key is to connect both to SERVER and determine which remote displays they are connected on, then invoke x2x on one of them. On LAPTOP1, do this:

laptop1$ ssh -X server
server$ echo $DISPLAY
:10

Then on LAPTOP2, running

laptop2$ ssh -X server
server$ x2x -east -from :10 -to $DISPLAY

allows LAPTOP1 to control LAPTOP2 (provided you have replaced :10 with what LAPTOP1's $DISPLAY variable actually is). To make LAPTOP2 control LAPTOP1:

laptop2$ ssh -X server
server$ x2x -west -from $DISPLAY -to :10

Of course, you can substitute the direction arguments with -north, -south, or -east, depending on how the monitors are actually arranged relative to each other.

Everything is a text file

One of the things which makes UNIX systems so powerful is the ease with which one can move data around. What makes this possible is the fact that, with few exceptions, everything is a text stream:

• Configuration files are plain text.
• Data are usually stored as flat text files.
• Even executables, which most Windows users consider to be synonymous with "binaries", are frequently text files: shell scripts, Perl scripts, Python scripts, and the like.
• Most of the UNIX core programs produce output as text streams or text files.

(On Windows, all of the above, except configurations, are usually represented as binary data, and configuration data, as stored in the registry, is not really amenable to editing by hand.)

What this means is that any text tool you learn, from less to emacs, can be put to use in almost any situation; you don't have to learn specialized tools for every new task you want to perform. Moreover, it means that applications which understand text automatically get an interface they can use to talk to each other. Text is the universal language of computing.

This model is so useful that Linux even creates text interfaces for many system internals which are not not naturally represented as text files or streams. The /proc filesystem, inspired by Plan 9, is one such "virtual filesystem" which exposes certain system vital signs. For example, /proc/cpuinfo provides information about the CPU(s):

prompt$ head /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 9
model name      : Intel(R) Pentium(R) M processor 1400MHz
stepping        : 5
cpu MHz         : 1400.000
cache size      : 1024 KB
fdiv_bug        : no
hlt_bug         : no

The /proc filesystem contains a wealth of system information (current processes, memory usage, hardware and software configuration) all in the guise of text files that you can read. You can write to some of these files to change configurations as well. For example, when executed as root,

echo www.example.com > /proc/sys/kernel/hostname

changes the host name, and

echo 1 > /proc/sys/vm/drop_caches

drops the system page cache.

So what? It means that when you want to write a program to read or manipulate some aspect of the system, you don't have to rely on bindings which are fragile, or require special headers, or are unavailable in your language of choice. All your program needs to do is read from or write to a file, which is (usually) a piece of cake.

Sometimes flat text isn't enough. What if you need structured or hierarchical data? As it turns out, a filesystem provides a fine hierarchical storage mechanism in the form of directories. For example, the /proc filesystem stores information about the process with PID N inside files in /proc/N/. But when structure is stored in directories instead of, say, nested XML elements, or in the keys of a Windows registry file, you can bring to bear all the tools you already know that operate on files. If you're deploying an application, it's trivial to copy or extract a configuration directory to ~/.appname/. It's not quite as easy to unzip an XML configuration fragment into a larger XML configuration file.

The idea of universal interfaces has found traction even as flat text files have lost ground. In the OpenDocument format, each document (extensions .odt, .odp, etc.) is really a ZIP file containing an XML description of the file and any extra media. (On the other hand, prior to Microsoft Office 2007, essentially all knowledge of the Office file format was obtained through reverse-engineering.)

The other week, I found myself in a situation where I had to save all the (numerous) images in a Word document to individual files. The fastest way was in fact to use OpenOffice to convert to an OpenDocument; when I unzipped that, one of the directories inside contained all the pictures which were in the original document. Common interfaces and tools help you to break free from the limitations of specialized tools when necessary.

Further reading: Unix philosophy, proc filesystem. The /sys filesystem is worth a look too.