wdired and renaming into nonexistent directories

(For the uninitiated, wdired is an indispensable emacs mode that lets you rename files in a directory just by editing their filenames in a buffer.)

Joakim Verona wrote a nifty patch against wdired that allows you to rename even to paths in directories that don't exist yet (emacs creates them for you when you finalize the operation). As an example of where this might be useful, suppose you have a directory like this:

  total used in directory 60K available 413675080
  -rw-r--r--  1 phil phil 318K 2009-12-31 19:00 20091231.jpg
  -rw-r--r--  1 phil phil 320K 2010-01-01 19:00 20100101.jpg
  -rw-r--r--  1 phil phil 305K 2010-01-02 19:00 20100102.jpg

Say you change that to the following (for instance, using M-x string-rectangle, among other alternatives):

  total used in directory 60K available 413675080
  -rw-r--r--  1 phil phil 318K 2009-12-31 19:00 2009/12/31.jpg
  -rw-r--r--  1 phil phil 320K 2010-01-01 19:00 2010/01/01.jpg
  -rw-r--r--  1 phil phil 305K 2010-01-02 19:00 2010/01/02.jpg

When you save, your files are now neatly organized in a year/month/day directory structure. This is the closest thing to magic I've done on my computer this month.

I've made some minor updates to Joakim's patch to fix bit-rot and conditionalize this behavior on a variable.

wdired-rename-to-nonexistent-directories.patch (1.6kb)

As an aside, I think learning about wdired turned on a lightbulb in my head that changed my understanding of Emacs. Yes, emacs is a text editor. But its power comes, in part, from the scores of major modes that let you interact with many different kinds of resources (e.g. your filesystem, as with wdired) under the guise of editing text (which you can think of as supplying a common abstraction over all those resources). Once you learn how to use, say, regexp search and replace, you can take advantage of that power when editing text files, but also when renaming/reorganizing files or composing mail or whatnot. Emacs is a text editor, but text doesn't just mean files.

GNU Parallel

I figured this was worth sharing because I myself had written two (fairly lame) clones of this program before I discovered it.

Sometimes I find myself composing and running huge shell scripts, like the following:

$ cat process-files.sh
sox input/foo.ogg output/foo.ogg channels 1
sox input/bar.ogg output/bar.ogg channels 1
sox input/baz.ogg output/baz.ogg channels 1
sox input/quux.ogg output/quux.ogg channels 1
# more of the same, for perhaps hundreds of lines...

(Aside: why not xargs? For complicated tasks, it can be error-prone or just plain insufficient. Moreover, there's a lot of value in being able to just look at the script and see exactly what is going to be executed on your behalf, especially for one-off tasks. If you know emacs macros, scripts like this are not onerous at all to generate anyway.)

If you have a sequence of tasks like this that can run independently (and they are CPU-bound), then it pays to distribute the tasks over all your CPU cores. Here's where GNU Parallel comes in handy. Just pipe into it the commands you want to execute:

$ parallel -j4 < process-files.sh

Now parallel runs up to 4 tasks concurrently, starting up a new one when each one finishes (just as if you had a queue and a pool of 4 workers). What an elegant interface.

GNU Parallel has a bunch of more advanced features that are worth checking out, for example, preserving the proper ordering of standard output across tasks (to maintain the illusion of sequential-ness), or showing an ETA.

GNU Parallel is not in the official Debian/Ubuntu repos (as far as I can tell) but it is a snap to build from source, and it's the sort of thing I'd want floating around in my ~/bin everywhere I work.