git grab bag

Time to flush my buffers again before the new year. Happy new year, everyone!

In no particular order here are some useful git things I've run across recently.

git stash --keep-index

git's index (or staging area) is useful, but it comes with a liability: you run into the danger of committing a state that doesn't work (compile, pass tests, etc.) because you never actually "saw", on your disk, the stuff that was going to be committed, in isolation.

That's what git stash --keep-index is for: it stashes the changes you haven't staged for commit, so you are left with only what will get committed. Do your builds/testing/verification, then git stash pop to continue working.

git clone --reference

If you're trying to clone a large remote repo, it's a waste of time to re-download objects that are already present in another repository on your local disk or local network. You can use the --reference option to obtain objects from this local repository when possible (any objects not found are pulled as usual from the real source):

git clone --reference LOCAL.git REMOTE.git my_new_clone

(This is sort of like cloning LOCAL.git, changing its remote to point to REMOTE.git, and fetching again, but much easier, and doesn't pollute your clone with branches and commits that are only present in LOCAL.git but not in REMOTE.git. From a content perspective it is exactly a clone of REMOTE.git.)

Note: if LOCAL.git is on the same filesystem, git sets up the alternates file so that object IDs can be resolved just using the object files in LOCAL.git. This may be undesirable because you won't actually have the objects in your new repo(!) and references in your new repo alone will not(!) be sufficient to keep the objects from being GC'd if LOCAL.git changes. To break this dependency, forcing the required objects to actually be copied into your new clone, do the following:

git repack -a; rm .git/objects/info/alternates

gfind

To make git print out all the files it is tracking:

git ls-tree -r --name-only HEAD

This is a useful base for searching your codebase, since it automatically lets you search all of your actual code and avoid looking at compiled code, generated code, downloaded libraries and documentation, etc.

I have aliased this to gfind and do things like this to search within a repo:

gfind | xargs grep FOO

git-filter-branch

git-filter-branch is your general purpose tool for rewriting repositories wholesale, for example, to extract a single subdirectory retaining all its history, or to excise a subdirectory, file, secret key, password, etc. that you've put in the repository.

Matthew McCullough has put together a good starting point with examples of how to use this tool. (Unfortunately, some of the links are broken; I will post alternate links once I find track down copies.)

git diff --color-words

Word based diffs in git! Great for LaTeX, Markdown, etc. More info here.

Zeya 0.6

I'm pleased to announce the (long overdue…) release of Zeya 0.6.

This release contains only bug fixes and minor changes that help Zeya to "just work" under a wider variety of circumstances.

Major changes since Zeya 0.5:

• The 'dir' backend works with Python 2.5 again. (Thanks to Greg Grossmeier)
• Broken symlinks are detected and ignored. (Thanks to Etienne Millon)
• Unreadable playlist files are detected and ignored.
• The 'dir' backend sorts files case-insensitively. (Thanks to Greg Grossmeier)
• Zeya no longer leaks file handles under certain circumstances. (Thanks to Pior Bastida)
• The frontend uses relative paths to resources, so Zeya can be run behind a reverse proxy out of the box, e.g. at http://yourhostname/path/to/zeya (Thanks to Jérôme Charaoui)

See http://web.psung.name/zeya/ to learn more about Zeya, installation, getting started, reporting bugs, and development.

Why Zeya?

I noticed that Zeya is now more than two years old. (Yow! Version 0.1 was released in August 2009! Thanks to you all for all the patches, by the way. Zeya has become much more popular than I would ever have guessed at the time.)

This got me thinking about what has, and hasn't, gotten better in that time with respect to music software, and why I thought Zeya was so useful in the first place.

Conceptually, you can think of any computer that is involved in playing your music as providing one or more of the following functions:

1. Storage: storing your music in nonvolatile storage and retrieving it
2. Control: letting you push buttons and see what is playing or available to play
3. Playback: driving a set of speakers

Traditionally all three functions have been performed by the same computer. But there's no reason they can't all run on different computers, not when you have a moderate- or high-speed network connecting them all. You might ask why you would actually want to separate these functions. The answer is that each function is best suited to be performed by a computer with a specific set of attributes, and the requirements for all of the functions are at odds with each other, and so require some compromise if they are to be colocated.

1. Storage wants to be done on a computer that has an enormous disk and is possibly continuously backed up.
2. Control wants to be done on a computer that is easy to reach (physically), possibly even one that fits in your pocket or can otherwise be carried around.
3. Playback wants to be done on a computer that permanently sits somewhere you want to hang out and is attached to a set of sweet speakers.

The original impetus for writing Zeya was that it became clear to me that storage constraints were becoming increasingly annoying. People switched their portable devices from spinning rust (classic iPod and laptop HDDs) to flash memory (smartphones, smartphone-like devices like the iPod touch, and laptop SSDs), and all of a sudden, with the accompanying drop in capacity (most smartphones, for example, top out at just 16GB or 32GB), many people could no longer carry their entire music collection with them.

At a higher level, Zeya completes the trio of tools that lets you decouple the components in varying ways:

1. Zeya and similar tools decouple storage from control and playback. Conceptually Amazon Cloud Player and Google Music are doing the same thing, too, though there the storage happens in the cloud.
2. X11 decouples control from storage and playback.
3. PulseAudio decouples playback from storage and control.

(If you use any two of those, you can completely decouple all three functions, modulo the fact that X11 and PulseAudio don't work very well over high-latency links.)

There has long been a patchwork of pieces that provide flexibility along one or more of the above dimensions (iTunes, Sonos, etc.) and have various tradeoffs; though, only in the past year have we seen the launches of high-profile products like Amazon Cloud Player and Google Music, which are more directly analogous to Zeya. All the attention in this area means that people are finally starting to understand the possibilities of the technology. Namely, that when you have network access everywhere, physical distance (and the need for the colocation of certain things) becomes much less important.

Network audio with PulseAudio made (somewhat) easy

I figured it was long past time to buckle down and learn how to make PulseAudio do my bidding and redirect audio across a network link. And I was surprised to learn that it's actually not hard to set up. In fact you don't need to touch any config files in /etc. Which you would never know, from reading most of the documentation that is out there.

While network audio is still kind of flaky at times, you only "pay" for it if you use it (people complain about PulseAudio a lot, but in my experience it works very reliably when used locally), and it can come in very handy.

Background

Throughout, I'll refer to the two computer roles as audio source and audio sink (where audio data is generated/decoded and where the speakers are attached, respectively). Note that these may differ from the PulseAudio concepts of the same names.

Ubuntu ships with PulseAudio, and many apps (among the common graphical apps and music players) understand how to talk to PulseAudio now (possibly through a compatibility layer). Except maybe Flash, but everyone who uses Flash is already used to it not being a proper citizen. The default setup is to have a per-user instance of PulseAudio running alongside the X session. This means that someone has to be logged in to X on both ends. If your audio sink is headless then you might have a single system-wide PulseAudio instance instead. Configuring that is not covered here.

PulseAudio runs as a service on port 4713. We need to first make that service discoverable on the machine with the audio sink, and then provide a way for the machine with the audio source to authenticate to the sink (so that you're not just letting anyone on your network play their crappy music).

Initial steps

On the machine with the audio sink:

• Install and run paprefs &
• Go to the Network server tab.
• Check Enable network access to local sound devices and Allow other machines on LAN to discover local sound devices. Autodiscovery uses Avahi, which you'll need to re-enable, if you ever disabled it.

(You might have to restart PulseAudio or X for the settings to take effect.)

I will describe two ways you can get the machine with the audio source to supply credentials to the sink so it can play music. The first one is likely to be more generally useful.

Using .pulse-cookie

The ~/.pulse-cookie file is a shared secret that can be used for authentication. Just copy that file from either machine (sink or source) to the other so that they are the same on both ends.

At the audio source, install padevchooser and run it (padevchooser &). A menu will appear in your notification area. Under Default server you should see an entry for the sink, assuming it is on the same local network (it should be named username@hostname). Select it.

Now run the application of your choice and play some audio!

Piggybacking on X11 forwarding

This method has the advantage of not requiring working autodiscovery. So you can use it on a LAN without Avahi running, or over a non-local network. All you need is to be able to SSH from the sink to the source. We will forward the audio from the source to the sink on a TCP port.

For this method, we'll transfer the credentials that the source will need by writing them to the X11 root window. PulseAudio supplies a program that does exactly this. On the machine with the audio sink, run:

sink$ start-pulseaudio-x11

(You can double-check that this is working by running xprop -root | grep PULSE and checking that there is an entry for PULSE_COOKIE, among others.)

Now initiate an SSH connection to the source, tunneling a port of your choice on that end over to your local PulseAudio server:

sink$ ssh -X -R 9998:localhost:4713 source

Run the application of your choice and play some audio! As you are doing so, set the PULSE_SERVER variable, which now enables your remote application to talk to your local audio hardware via the tunneled port.

source$ PULSE_SERVER=localhost:9998 rhythmbox &

Conclusion and caveats

Et voila, now you've decoupled your music player from your speakers. So you can play music from your laptop using the speakers at your desktop. Or you can play music from your desktop using the speakers in your living room. Some things to watch out for.

• You may need to kill and restart pulseaudio (or X) to get it to pick up some of those configuration changes.
• PulseAudio will not move a stream while it is running, so if you use the menu to change the destination, the change will not take effect until the next song begins, unless you pause and restart playback in your audio application.
• PulseAudio is kind of finicky and some of it still feels like black magic at times. You may need to restart it if you suddenly get errors about things not working, especially if they were just working a minute ago.

Sources and further reading: pulseaudio-discuss, ArchLinux wiki, Ubuntu forums

QR codes in LaTeX

If you are adding QR codes to print media, in order to make them look really sharp, you want the QR codes to be generated in a vector format rather than a bitmap format. It turns out that the pst-barcode package allows you to easily add vectorized QR codes to your LaTeX documents.

Here are some minimal steps to generate a PDF with a QR code in it:

1. Install dependencies:

$ aptitude install texlive-latex-{base,extra}

(This works on Ubuntu 11.04, at least.)

2. Add the following to a .tex file:

\documentclass{article}
\usepackage{pst-barcode}
\usepackage{auto-pst-pdf}
\begin{document}
\begin{pspicture}(1in,1in)
  \psbarcode{PAYLOAD}{eclevel=M width=1.0 height=1.0}{qrcode}
\end{pspicture}
\end{document}

where PAYLOAD gives the data to be encoded. For a business card you might have something like:

MECARD:N:Sung,Phil;TEL:+14085551234;EMAIL:philbert@gmail.com;URL:http://web.psung.name;;

See this page for more MECARD options and for descriptions of the other protocols (URLs, email addresses, etc.) that barcode readers understand.

3. Compile your file as follows:

$ pdflatex --shell-escape yourfile.tex

Some notes:

• eclevel specifies the level of error correction, and is one of L, M, Q, H (low to high)
• width and height specify the dimensions of the barcode.
• pst-barcode knows how to generate barcodes in many other formats; see the documentation for details.

You can also change the color of the barcode by adding something like the following:

\usepackage{color}
[...]
  \psbarcode[linecolor=blue]{PAYLOAD}[...]

Sources: StackExchange, Thomas Widmann, Andrew Brampton (who has a nice template for a business card)

"X11 connection rejected because of wrong authentication."

A brief note for posterity: if you are getting an error like the following,

X11 connection rejected because of wrong authentication.
xterm Xt error: Can't open display: localhost:10.0

one thing you might try, after you have eliminated the usual suspects (no, try those first, really), is unsetting the XAUTHORITY environment variable:

$ unset XAUTHORITY

* * *

The remainder of this post is a followup to my previous post, Stupid Screen Tricks. (The setup I described therein is still alive and well.)

I received the above error when I started a screen session on display :0.0 and (after disconnecting and reconnecting) subsequently tried to launch, from within it, a new X program on a different display (even after setting DISPLAY). The session inherits the old XAUTHORITY value, which, for whatever reason, for as long as it is present, foils attempts to run programs on other displays.

To work around this, I changed my here utility alias to the following,

alias here='unset XAUTHORITY; DISPLAY=`cat ~/.last-display`'

that is, unsetting XAUTHORITY as well as setting DISPLAY before trying to run anything.

Assorted notes

• Hugin, the panorama-stitching tool, is quite slick these days (ever since it got some UI improvements in release 2010.4.0 or so, I believe). It handled perhaps 80-90% of the panoramas from my recent trip in about 3 clicks and without manual intervention. Hats off to the Hugin developers.
• I've been learning how to twiddle the "Projection" knob in Hugin, which lets you change the geometry of the resulting panorama. For example, for photos of murals (or tapestries, or other long, flat, surfaces), the Rectlinear projection corrects for the distortion to reconstitute the "flat" image, as if you were standing far away.

Default (cylindrical)

Corrected (rectilinear)

• Ubuntu has these new scrollbars with invisible scroll handles. They are quite annoying (being invisible and all). You can revert to the old ones like so:

  sudo apt-get remove overlay-scrollbar liboverlay-scrollbar-0.1-0

  Source