Assorted Notes

Many random notes (some for my own reference), each of which is too short to warrant a full blog post:

PSA: remote tab completion

If you have set up passwordless SSH login to some host, then bash-completion in Debian/Ubuntu (which is enabled by default, I think) automatically completes remote paths as arguments to scp. For example, pressing TAB after the following does exactly what it ought to, namely, display files within /path/to/whatever on the host example.com:

scp example.com:/path/to/whatever/

Emacs: dired-jump

Ever find yourself wanting to rename or move a file while you're editing it? M-x dired-jump brings up a dired view of the directory containing the current file (you may need to (require 'dired-x) first). Press R ("Rename") and type the new path.

The best part is that the original editing buffer gets its path updated automatically (and its buffer name too, if you changed the basename), so you can get straight back to work— no need to close and reopen the file.

Suspending from the command line

On Ubuntu 10.04 Lucid, you can suspend the machine programmatically by installing acpitool and then running the following:

gnome-screensaver-command --lock # Optional- locks the screen
sudo /usr/bin/acpitool -s

To avoid having to enter my password every time for acpitool, I whitelisted that command in sudo for use without a password. You can do this by running sudo visudo and adding the following line at the end of the file:

%admin ALL = NOPASSWD: /usr/bin/acpitool -s

Sources: Linux Living. Also see: instructions for previous Ubuntu versions; pm-suspend(8).

VirtualBox

I've been using VirtualBox (aptitude install virtualbox-ose) to run virtual machines of various GNU/Linux flavors (for me, it's primarily useful for making sure Zeya works on different platforms and configurations), and I am generally pleased with it. A couple of tips for setting up VMs for maximum effectiveness:

• The virtualbox-ose-guest-x11 package, when installed on the guest, provides a driver for the virtual display device. This seems to be necessary (and sufficient) to get not-crappy graphics performance.
• Configuring your VM to use bridged networking makes the VM appear as a device on the host's network, for example, it will pick up an IP address from your local DHCP server.

wdired and renaming into nonexistent directories

(For the uninitiated, wdired is an indispensable emacs mode that lets you rename files in a directory just by editing their filenames in a buffer.)

Joakim Verona wrote a nifty patch against wdired that allows you to rename even to paths in directories that don't exist yet (emacs creates them for you when you finalize the operation). As an example of where this might be useful, suppose you have a directory like this:

  /home/phil/tempfiles:
  total used in directory 60K available 413675080
  -rw-r--r--  1 phil phil 318K 2009-12-31 19:00 20091231.jpg
  -rw-r--r--  1 phil phil 320K 2010-01-01 19:00 20100101.jpg
  -rw-r--r--  1 phil phil 305K 2010-01-02 19:00 20100102.jpg
  [...]

Say you change that to the following (for instance, using M-x string-rectangle, among other alternatives):

  /home/phil/tempfiles:
  total used in directory 60K available 413675080
  -rw-r--r--  1 phil phil 318K 2009-12-31 19:00 2009/12/31.jpg
  -rw-r--r--  1 phil phil 320K 2010-01-01 19:00 2010/01/01.jpg
  -rw-r--r--  1 phil phil 305K 2010-01-02 19:00 2010/01/02.jpg
  [...]

When you save, your files are now neatly organized in a year/month/day directory structure. This is the closest thing to magic I've done on my computer this month.

I've made some minor updates to Joakim's patch to fix bit-rot and conditionalize this behavior on a variable.

wdired-rename-to-nonexistent-directories.patch (1.6kb)

As an aside, I think learning about wdired turned on a lightbulb in my head that changed my understanding of Emacs. Yes, emacs is a text editor. But its power comes, in part, from the scores of major modes that let you interact with many different kinds of resources (e.g. your filesystem, as with wdired) under the guise of editing text (which you can think of as supplying a common abstraction over all those resources). Once you learn how to use, say, regexp search and replace, you can take advantage of that power when editing text files, but also when renaming/reorganizing files or composing mail or whatnot. Emacs is a text editor, but text doesn't just mean files.

GNU Parallel

I figured this was worth sharing because I myself had written two (fairly lame) clones of this program before I discovered it.

Sometimes I find myself composing and running huge shell scripts, like the following:

$ cat process-files.sh
sox input/foo.ogg output/foo.ogg channels 1
sox input/bar.ogg output/bar.ogg channels 1
sox input/baz.ogg output/baz.ogg channels 1
sox input/quux.ogg output/quux.ogg channels 1
# more of the same, for perhaps hundreds of lines...

(Aside: why not xargs? For complicated tasks, it can be error-prone or just plain insufficient. Moreover, there's a lot of value in being able to just look at the script and see exactly what is going to be executed on your behalf, especially for one-off tasks. If you know emacs macros, scripts like this are not onerous at all to generate anyway.)

If you have a sequence of tasks like this that can run independently (and they are CPU-bound), then it pays to distribute the tasks over all your CPU cores. Here's where GNU Parallel comes in handy. Just pipe into it the commands you want to execute:

$ parallel -j4 < process-files.sh

Now parallel runs up to 4 tasks concurrently, starting up a new one when each one finishes (just as if you had a queue and a pool of 4 workers). What an elegant interface.

GNU Parallel has a bunch of more advanced features that are worth checking out, for example, preserving the proper ordering of standard output across tasks (to maintain the illusion of sequential-ness), or showing an ETA.

GNU Parallel is not in the official Debian/Ubuntu repos (as far as I can tell) but it is a snap to build from source, and it's the sort of thing I'd want floating around in my ~/bin everywhere I work.

Extracting the audio track from videos, and downmixing

On Ubuntu/Debian, you can use ffmpeg (in the package of the same name) to extract the audio only from video files:

$ ffmpeg -i input.ogm -ab 128k -vn output.ogg

ffmpeg detects the input and output formats automatically, so you can also convert to an MP3 (for example) just by specifying an output filename of the appropriate extension.

When converting audio for listening while working out or while traveling, it's also useful to downmix stereo to mono so you can listen with just one earbud without missing anything:

$ ffmpeg -i input.ogm -ab 128k -ac 1 -vn output.ogg

(Thanks to Rene Moser for the pointer. Also check out sox if you are looking to apply more sophisticated audio transformations.)

Further reading: ffmpeg command-line tool documentation

The first step is admitting you have a problem

A few months ago I glanced at my Google Reader Trends page:

From your 188 subscriptions, over the last 30 days you read 10,075 items...

Seeing a five-digit number there frightened me a bit, especially because I am not Robert Scoble¹ and staying on top of tech news is not my day job. Since then I've made three changes:

• I'm checking Google Reader only once a week.
• I've culled my list of subscriptions, reducing the number of items per day by about 70%.
• For the high-volume feeds that remain, I read them in sort by magic order and don't read all of the items if I don't have time.

Just the first item, dealing with the news in batch mode, is sufficient to free up a big chunk of time (and mitigate the sensation of drowning). I still get news from other sources on a more-than-once-weekly basis— mostly, Google Buzz and email lists— and I'd like to improve the way I deal with those, too, but at this time they have far better S/N ratio and lower volume.

In my ideal world, for each feed I could turn a dial to say "show me the no-more-than-N most popular items per day for this feed". But sort by magic comes pretty close to what I want.

¹ Scoble famously follows over 18,000(!) people on Twitter.

Zeya in Linux Journal

Amit Saha wrote a nice article about Zeya in Linux Journal. It's pretty much Zeya's missing tutorial and design doc (not to mention, a review and a rundown of the alternatives). Thanks, Amit!

rdiff-snapshot-fs

I use rdiff-backup for backups. It creates a mirror of your data, just like plain old rsync, but also stores a series of reverse diffs that lets you restore historical versions of each file. For the most part it works well, but the interface for browsing and restoring from backups is somewhat clunky:

$ # List all snapshots
$ rdiff-backup -l /backup
Found 3 increments:
    increments.2010-06-01T23:00:00-07:00.dir   Tue Jun  1 23:00:00 2010
    increments.2010-06-02T23:00:00-07:00.dir   Wed Jun  2 23:00:00 2010
    increments.2010-06-03T23:00:00-07:00.dir   Thu Jun  3 23:00:00 2010
Current mirror: Fri Jun  4 23:00:00 2010
$ # List contents of a snapshot
$ # (note, lists specified dir recursively)
$ rdiff-backup --list-at-time 2010-06-01T23:00:00 /backup/foo/bar
foo/bar/one
foo/bar/two
foo/bar/more/three
foo/bar/more/four
$ # Restore a file from backup
$ rdiff-backup -r 2010-06-01T23:00:00 /backup/foo/bar/one ~/one.restore

I wrote rdiff-snapshot-fs, which is an alternative to rdiff-backup's restore interface. It's a virtual filesystem (powered by FUSE) that lets you browse all your snapshots and their contents. Now you can examine each snapshot as if it were backed by a real mirror:

$ ./rdiff-snapshot-fs.py /backup /view
$ ls /view
2010-06-01T23:00:00-07:00
2010-06-02T23:00:00-07:00
2010-06-03T23:00:00-07:00
$ ls /view/2010-06-01T23:00:00-07:00/foo/bar
one
two
more
$ cp /view/2010-06-01T23:00:00-07:00/foo/bar/one ~/one.restore

Some background: another tool, called archfs, provides similar functionality, but it has performance issues, and I never got it to successfully load my backup repository (granted, my homedir backup repo has 400 snapshots and a current mirror size of 450GB). I concur with Jon Dowland's assessment that the problem stems from archfs trying to reconstruct historical snapshot data up front before it's needed. (Many problems in software, as well as in life, are really problems caused by being insufficiently lazy!)

More info about rdiff-snapshot-fs, including how to obtain the source, can be found at http://web.psung.name/rdiff-snapshot-fs.

(By the way, FUSE is really interesting to play with and FUSE-Python makes it a snap to get started. More remarks about this soon, possibly.)

Disclaimer: This is something I cooked up in my spare time. rdiff-snapshot-fs is liable to eat your data and scare your pets. The file sizes and modes that it reports are known to be unreliable under certain circumstances (though the file data itself is generated correctly, as far as I can tell). But, if you know no fear, you may find it somewhat more useful than the standard rdiff-backup interface.