Zeya 0.6

I'm pleased to announce the (long overdue…) release of Zeya 0.6.

This release contains only bug fixes and minor changes that help Zeya to "just work" under a wider variety of circumstances.

Major changes since Zeya 0.5:

• The 'dir' backend works with Python 2.5 again. (Thanks to Greg Grossmeier)
• Broken symlinks are detected and ignored. (Thanks to Etienne Millon)
• Unreadable playlist files are detected and ignored.
• The 'dir' backend sorts files case-insensitively. (Thanks to Greg Grossmeier)
• Zeya no longer leaks file handles under certain circumstances. (Thanks to Pior Bastida)
• The frontend uses relative paths to resources, so Zeya can be run behind a reverse proxy out of the box, e.g. at http://yourhostname/path/to/zeya (Thanks to Jérôme Charaoui)

See http://web.psung.name/zeya/ to learn more about Zeya, installation, getting started, reporting bugs, and development.

Why Zeya?

I noticed that Zeya is now more than two years old. (Yow! Version 0.1 was released in August 2009! Thanks to you all for all the patches, by the way. Zeya has become much more popular than I would ever have guessed at the time.)

This got me thinking about what has, and hasn't, gotten better in that time with respect to music software, and why I thought Zeya was so useful in the first place.

Conceptually, you can think of any computer that is involved in playing your music as providing one or more of the following functions:

1. Storage: storing your music in nonvolatile storage and retrieving it
2. Control: letting you push buttons and see what is playing or available to play
3. Playback: driving a set of speakers

Traditionally all three functions have been performed by the same computer. But there's no reason they can't all run on different computers, not when you have a moderate- or high-speed network connecting them all. You might ask why you would actually want to separate these functions. The answer is that each function is best suited to be performed by a computer with a specific set of attributes, and the requirements for all of the functions are at odds with each other, and so require some compromise if they are to be colocated.

1. Storage wants to be done on a computer that has an enormous disk and is possibly continuously backed up.
2. Control wants to be done on a computer that is easy to reach (physically), possibly even one that fits in your pocket or can otherwise be carried around.
3. Playback wants to be done on a computer that permanently sits somewhere you want to hang out and is attached to a set of sweet speakers.

The original impetus for writing Zeya was that it became clear to me that storage constraints were becoming increasingly annoying. People switched their portable devices from spinning rust (classic iPod and laptop HDDs) to flash memory (smartphones, smartphone-like devices like the iPod touch, and laptop SSDs), and all of a sudden, with the accompanying drop in capacity (most smartphones, for example, top out at just 16GB or 32GB), many people could no longer carry their entire music collection with them.

At a higher level, Zeya completes the trio of tools that lets you decouple the components in varying ways:

1. Zeya and similar tools decouple storage from control and playback. Conceptually Amazon Cloud Player and Google Music are doing the same thing, too, though there the storage happens in the cloud.
2. X11 decouples control from storage and playback.
3. PulseAudio decouples playback from storage and control.

(If you use any two of those, you can completely decouple all three functions, modulo the fact that X11 and PulseAudio don't work very well over high-latency links.)

There has long been a patchwork of pieces that provide flexibility along one or more of the above dimensions (iTunes, Sonos, etc.) and have various tradeoffs; though, only in the past year have we seen the launches of high-profile products like Amazon Cloud Player and Google Music, which are more directly analogous to Zeya. All the attention in this area means that people are finally starting to understand the possibilities of the technology. Namely, that when you have network access everywhere, physical distance (and the need for the colocation of certain things) becomes much less important.

Network audio with PulseAudio made (somewhat) easy

I figured it was long past time to buckle down and learn how to make PulseAudio do my bidding and redirect audio across a network link. And I was surprised to learn that it's actually not hard to set up. In fact you don't need to touch any config files in /etc. Which you would never know, from reading most of the documentation that is out there.

While network audio is still kind of flaky at times, you only "pay" for it if you use it (people complain about PulseAudio a lot, but in my experience it works very reliably when used locally), and it can come in very handy.

Background

Throughout, I'll refer to the two computer roles as audio source and audio sink (where audio data is generated/decoded and where the speakers are attached, respectively). Note that these may differ from the PulseAudio concepts of the same names.

Ubuntu ships with PulseAudio, and many apps (among the common graphical apps and music players) understand how to talk to PulseAudio now (possibly through a compatibility layer). Except maybe Flash, but everyone who uses Flash is already used to it not being a proper citizen. The default setup is to have a per-user instance of PulseAudio running alongside the X session. This means that someone has to be logged in to X on both ends. If your audio sink is headless then you might have a single system-wide PulseAudio instance instead. Configuring that is not covered here.

PulseAudio runs as a service on port 4713. We need to first make that service discoverable on the machine with the audio sink, and then provide a way for the machine with the audio source to authenticate to the sink (so that you're not just letting anyone on your network play their crappy music).

Initial steps

On the machine with the audio sink:

• Install and run paprefs &
• Go to the Network server tab.
• Check Enable network access to local sound devices and Allow other machines on LAN to discover local sound devices. Autodiscovery uses Avahi, which you'll need to re-enable, if you ever disabled it.

(You might have to restart PulseAudio or X for the settings to take effect.)

I will describe two ways you can get the machine with the audio source to supply credentials to the sink so it can play music. The first one is likely to be more generally useful.

Using .pulse-cookie

The ~/.pulse-cookie file is a shared secret that can be used for authentication. Just copy that file from either machine (sink or source) to the other so that they are the same on both ends.

At the audio source, install padevchooser and run it (padevchooser &). A menu will appear in your notification area. Under Default server you should see an entry for the sink, assuming it is on the same local network (it should be named username@hostname). Select it.

Now run the application of your choice and play some audio!

Piggybacking on X11 forwarding

This method has the advantage of not requiring working autodiscovery. So you can use it on a LAN without Avahi running, or over a non-local network. All you need is to be able to SSH from the sink to the source. We will forward the audio from the source to the sink on a TCP port.

For this method, we'll transfer the credentials that the source will need by writing them to the X11 root window. PulseAudio supplies a program that does exactly this. On the machine with the audio sink, run:

sink$ start-pulseaudio-x11

(You can double-check that this is working by running xprop -root | grep PULSE and checking that there is an entry for PULSE_COOKIE, among others.)

Now initiate an SSH connection to the source, tunneling a port of your choice on that end over to your local PulseAudio server:

sink$ ssh -X -R 9998:localhost:4713 source

Run the application of your choice and play some audio! As you are doing so, set the PULSE_SERVER variable, which now enables your remote application to talk to your local audio hardware via the tunneled port.

source$ PULSE_SERVER=localhost:9998 rhythmbox &

Conclusion and caveats

Et voila, now you've decoupled your music player from your speakers. So you can play music from your laptop using the speakers at your desktop. Or you can play music from your desktop using the speakers in your living room. Some things to watch out for.

• You may need to kill and restart pulseaudio (or X) to get it to pick up some of those configuration changes.
• PulseAudio will not move a stream while it is running, so if you use the menu to change the destination, the change will not take effect until the next song begins, unless you pause and restart playback in your audio application.
• PulseAudio is kind of finicky and some of it still feels like black magic at times. You may need to restart it if you suddenly get errors about things not working, especially if they were just working a minute ago.

Sources and further reading: pulseaudio-discuss, ArchLinux wiki, Ubuntu forums

QR codes in LaTeX

If you are adding QR codes to print media, in order to make them look really sharp, you want the QR codes to be generated in a vector format rather than a bitmap format. It turns out that the pst-barcode package allows you to easily add vectorized QR codes to your LaTeX documents.

Here are some minimal steps to generate a PDF with a QR code in it:

1. Install dependencies:

$ aptitude install texlive-latex-{base,extra}

(This works on Ubuntu 11.04, at least.)

2. Add the following to a .tex file:

\documentclass{article}
\usepackage{pst-barcode}
\usepackage{auto-pst-pdf}
\begin{document}
\begin{pspicture}(1in,1in)
  \psbarcode{PAYLOAD}{eclevel=M width=1.0 height=1.0}{qrcode}
\end{pspicture}
\end{document}

where PAYLOAD gives the data to be encoded. For a business card you might have something like:

MECARD:N:Sung,Phil;TEL:+14085551234;EMAIL:philbert@gmail.com;URL:http://web.psung.name;;

See this page for more MECARD options and for descriptions of the other protocols (URLs, email addresses, etc.) that barcode readers understand.

3. Compile your file as follows:

$ pdflatex --shell-escape yourfile.tex

Some notes:

• eclevel specifies the level of error correction, and is one of L, M, Q, H (low to high)
• width and height specify the dimensions of the barcode.
• pst-barcode knows how to generate barcodes in many other formats; see the documentation for details.

You can also change the color of the barcode by adding something like the following:

\usepackage{color}
[...]
  \psbarcode[linecolor=blue]{PAYLOAD}[...]

Sources: StackExchange, Thomas Widmann, Andrew Brampton (who has a nice template for a business card)

"X11 connection rejected because of wrong authentication."

A brief note for posterity: if you are getting an error like the following,

X11 connection rejected because of wrong authentication.
xterm Xt error: Can't open display: localhost:10.0

one thing you might try, after you have eliminated the usual suspects (no, try those first, really), is unsetting the XAUTHORITY environment variable:

$ unset XAUTHORITY

* * *

The remainder of this post is a followup to my previous post, Stupid Screen Tricks. (The setup I described therein is still alive and well.)

I received the above error when I started a screen session on display :0.0 and (after disconnecting and reconnecting) subsequently tried to launch, from within it, a new X program on a different display (even after setting DISPLAY). The session inherits the old XAUTHORITY value, which, for whatever reason, for as long as it is present, foils attempts to run programs on other displays.

To work around this, I changed my here utility alias to the following,

alias here='unset XAUTHORITY; DISPLAY=`cat ~/.last-display`'

that is, unsetting XAUTHORITY as well as setting DISPLAY before trying to run anything.

Assorted notes

• Hugin, the panorama-stitching tool, is quite slick these days (ever since it got some UI improvements in release 2010.4.0 or so, I believe). It handled perhaps 80-90% of the panoramas from my recent trip in about 3 clicks and without manual intervention. Hats off to the Hugin developers.
• I've been learning how to twiddle the "Projection" knob in Hugin, which lets you change the geometry of the resulting panorama. For example, for photos of murals (or tapestries, or other long, flat, surfaces), the Rectlinear projection corrects for the distortion to reconstitute the "flat" image, as if you were standing far away.

Default (cylindrical)

Corrected (rectilinear)

• Ubuntu has these new scrollbars with invisible scroll handles. They are quite annoying (being invisible and all). You can revert to the old ones like so:

  sudo apt-get remove overlay-scrollbar liboverlay-scrollbar-0.1-0

  Source

Spring cleaning, dealing with duplicate files, and vacuumpack

I was doing some spring cleaning of my computer, part of which was removing duplicate files. I'm a packrat, and I subscribe to the copy-first-and-ask-questions-later school of thought, so I have a few duplicates floating around, just taking up disk space. (I think I have somewhere a filesystem copy of my previous computer, which, in turn, contains a filesystem copy of my computer before that.)

There are plenty of tools that will help you remove duplicate files (this page lists no fewer than 16), but, disappointingly, none of them that I could find seem to give you a high-level understanding of where and how duplicate files appear in your filesystem. So I wrote vacuumpack, a short Python tool to help me really see what was going on in my filesystem. Nothing revolutionary, but it helped me through my spring cleaning.

(Aside 1: I noticed the parallel with coding theory: you want to remove undesirable redundancy, by removing duplicate files on the same disk, and add desirable redundancy, by using backups or RAID.)

(Aside 2: This is simple enough that undoubtedly someone will send a pointer to a tool written circa 1988 that does what I am describing. C'est la vie; unfortunately it is usually easier, and more fun, to write code than to find it.)

vacuumpack is probably best explained by example. First, vacuumpack can scan a directory and identify duplicated files:

$ ./vacuumpack.py --target=/home/phil --cache=/home/phil/.contentindex

The output shows clusters of identical files. For example, oh hey, it's the GPLv3 (the cluster is prefaced by the sha256 of the shared content):

[...]
8ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903: 140588 bytes wasted
  /home/phil/projects/raytracer/COPYING (35147 bytes)
  /home/phil/projects/rdiff-snapshot-fs/source/COPYING (35147 bytes)
  /home/phil/projects/syzygy/COPYING (35147 bytes)
  /home/phil/projects/vacuumpack/COPYING (35147 bytes)
  /home/phil/source/emacs/COPYING (35147 bytes)
[...]

The clusters are sorted in decreasing order of the amount of space wasted, so you can fry the big fish first. Having multiple copies of the GPL floating around isn't really a cause for concern; let's have a look at another cluster:

6940e64dd91f1ac77c43f1f326f2416f4b54728ff47a529b190b7dadde78ea23: 714727 bytes wasted
  /home/phil/photos/20060508/may-031.jpg (714727 bytes)
  /home/phil/album1/c.jpg (714727 bytes)

Thus far duff and many other tools do essentially the same thing. But most of the tools out there are focused on semi-mechanically helping you delete files, even though cleaning up often requires moving or copying them as well.

Duplicated photos, like the ones above, are probably something I want to resolve. Since I recall that /home/phil/photos is the canonical location for most of my photos, the other directory looks somewhat suspect. So I'll ask vacuumpack to tell me more about it:

$ ./vacuumpack.py --target=/home/phil --cache=/home/phil/.contentindex /home/phil/album1

Now, for each file in that directory, vacuumpack tells me whether it knows about any duplicates:

/home/phil/album1/a.jpg         == /home/phil/photos/20060508/may-010.jpg
/home/phil/album1/b.jpg         == /home/phil/photos/20060508/may-019.jpg
/home/phil/album1/c.jpg         == /home/phil/photos/20060508/may-031.jpg
/home/phil/album1/d.jpg         == /home/phil/photos/20060508/may-033.jpg
/home/phil/album1/e.jpg         == /home/phil/photos/20060508/may-048.jpg
/home/phil/album1/f.jpg         -- NO DUPLICATES
/home/phil/album1/g.jpg         == /home/phil/photos/20060508/may-077.jpg
/home/phil/album1/h.jpg         == /home/phil/photos/20060508/may-096.jpg

It looks like the files in /home/phil/album1 are a subset of the photos in /home/phil/photos... except that /home/phil/photos is missing a file! I need to copy that file back; once I do, the directory album1 is safe to delete.

In this mode vacuumpack is behaving like a directory diff tool, except that it uses content rather than filenames to match up files.

A majority of the code in vacuumpack is actually devoted to identifying duplicates efficiently. vacuumpack stores the file hashes and metadata in a cache (specified by --cache=...) and automatically rereads files when (and only when) they have been modified. So after the initial run, vacuumpack runs very quickly (e.g. in just seconds on my entire homedir) while always producing up-to-date reports. It's fast enough that you can run it semi-interactively, using it to check your work continuously while you're reorganizing and cleaning your files. You can drill down and ask lots of questions about different directories without having to wait twenty minutes for each answer.

I've posted a git repo containing the vacuumpack source:

git clone http://web.psung.name/git/vacuumpack.git

This has been tested on Ubuntu 11.04 under Python 2.7.1. You may use the code under the terms of the GNU GPL v3 or (at your option) any later version.

Assorted notes

• Public service announcement: earlier this year Google announced optional 2-factor authentication for Google accounts. Please use it: it's one of the least painful ways to make your data safer (most people are toast if their email gets compromised). And the implementation seems fairly well thought out:
• You download an app to your smartphone (or smartphone-like device) that generates one-time passwords (OTPs), to be used in conjunction with your regular password when needed. A single OTP can authenticate one computer for up to 30 days. Yes, the app is open source. It runs on any Android, Blackberry, or iOS device.
• The app works offline, without a data connection, because the method for generating OTPs is specified by RFC 4226 (yes, it's standardized and everything) and is either sequence-based or time-based.
• Failing that, if you don't have a smartphone, or it's busted, you can also receive an OTP via SMS to a designated number (though, obviously, then you need phone reception).
• Failing that, if you don't have a cell phone, or it's busted, you can also receive an OTP via a voice call to a designated landline.
• Failing that... if you know you'll be somewhere where you have no phone at all, you can print a list of OTPs to carry with you that will enable you to log in.
• Apps that authenticate via just a password (e.g. the phone itself, or most desktop apps, like Picasa) get a dedicated automatically generated password. You don't get the benefit of 2-factor auth here, but these passwords are less likely to be phished because you're not typing them in all the time, and you can revoke them individually.
• Good lord, Ubuntu 11.04 (Natty) is fast. My laptop (Thinkpad X201 with Intel SSD) boots from disk unlock screen (LUKS full-disk encryption) to a working Openbox desktop in about four seconds.
• I've been playing with Blender (the Free 3D modeling tool) for a personal project to be 3D printed, and it's a lot of fun, and quite rewarding. I'm still a noob at this stuff, but already I get some of these "in the zone" moments that are so rarely attained in software (Emacs being the other exception) where I feel like I'm manipulating a thing directly rather than using a software program. The Blender UI looks like an airplane cockpit, but there is a method to its madness! The other neat thing is that most of the time when you do creative work on the computer you are not rewarded with anything nearly so tangible as a 3D printed piece.
• A clever thing I noticed on Android the other week: when you use voice dictation in a text entry field, and you move the cursor back to previous words, above the keyboard it shows not the nearest alternatives based on the keyboard layout (as it would if you were typing), but the nearest alternatives based on sound— e.g. "wreck" ... "a nice beach" as suggested replacements for "recognize speech".

Reducing merge headaches: git meets diff3

git has an option to display merge conflicts in diff3 format (by default it only displays the two files to be merged). You can enable it like so:

$ git config --global merge.conflictstyle diff3

Now, when you have to resolve merge conflicts, git shows your side, the side being merged, and (here's what's new) the common ancestor in between them. Here's an example of the diff3-formatted output:

cauliflower
<<<<<<< HEAD
peas
potatoes
||||||| merged common ancestors
peas
=======
>>>>>>> topic
tomatoes

Having the merge ancestor readily available helps you to quickly determine what the correct merge is, since you can infer from it the changes that were made on both sides. Here you can see that the original state was peas. On your branch potatoes was added (compare the middle section to the top) and on the other branch peas was removed (compare the middle section to the bottom). Therefore the correct change is to both add potatoes and remove peas, leaving you with just potatoes in the conflicted section.

There's really no reason you shouldn't enable the diff3 style, because you frequently need the ancestor to determine what the correct merge is.

To see that this is true, even in the simple example above, look at what the conflict looks like under the standard style:

cauliflower
<<<<<<< HEAD
peas
potatoes
=======
>>>>>>> topic
tomatoes

There's an asymmetry between peas and potatoes: one was added and one was deleted, but this merge conflict doesn't tell you anything at all about which was which! You can't determine the correct merge unless you remember the sequence of changes that led up to this point. And why should you have to rack your brain to do that? That's exactly the sort of thing that your computer can, and should, help you with.

Bonus tip: rerere (reuse recorded resolution)

If your workflow finds you redoing the same merges over and over again you might also find git's rerere (reuse recorded resolution) feature to be useful.

One of the things that is wonderful about rerere is that it provides hardly any UI surface at all. Just set it...

$ git config --global rerere.enabled 1

...and forget it. Although there is a git rerere command, you can get a lot done without using it at all.

After enabling rerere, whenever you resolve a merge conflict, git automatically squirrels away the resolution in its database. You'll see a message like this one:

$ git commit
Recorded resolution for 'soup'
[...]

And the next time you encounter the same conflict, where you would have expected git to spit out a file with conflict markers, you will instead find that it has automatically resolved the merge for you, and printed the following message:

$ git merge topic
Auto-merging soup
CONFLICT (content): Merge conflict in soup
Resolved 'soup' using previous resolution.

Just double-check to make sure nothing has gone awry, add, and commit. Save your blood, sweat, and tears for other, more interesting problems than redoing merges.

Further reading: Pro Git on rerere

Making your own page-a-day calendar, revisited

Some time ago I posted instructions for designing and producing a page-a-day calendar, which is a moderately neat project at the intersection of metaprogramming and handicraft.

Matt Johnson made one for the year 2011; his write-up fills a lot of the gaps in my rather skeletal instructions and provides a number of suggestions for extending the original design in very good ways (including adding page content not based on photographs, and making a wooden stand for the calendar). Read it if you are thinking of making one of these things.

Thanks, Matt!

CC-BY Matt Johnson

Read more.