tag:blogger.com,1999:blog-9700553290015930382024-03-13T03:03:37.625-07:00I Still Know What You Learned Last SummerPhilhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.comBlogger159125tag:blogger.com,1999:blog-970055329001593038.post-55690381930654698462012-12-31T22:22:00.001-08:002012-12-31T22:22:54.623-08:00git grab bag<p>Time to flush my buffers again before the new year. Happy new year, everyone!</p>
<p>In no particular order here are some useful git things I've run across recently.</p>
<p><b>git stash --keep-index</b></p>
<p>git's <em>index</em> (or staging area) is useful, but it comes with a
liability: you run into the danger of committing a state that doesn't work
(compile, pass tests, etc.) because you never actually "saw", on your disk,
the stuff that was going to be committed, in isolation.</p>
<p>That's what <tt>git stash --keep-index</tt> is for: it stashes the changes you
<em>haven't</em> staged for commit, so you are left with only what will get
committed. Do your builds/testing/verification, then <tt>git stash pop</tt>
to continue working.</p>
<p><b>git clone --reference</b></p>
<p>If you're trying to clone a large remote repo, it's a waste of time to
re-download objects that are already present in another repository on your
local disk or local network. You can use the <tt>--reference</tt> option to
obtain objects from this local repository when possible (any objects not
found are pulled as usual from the real source):</p>
<blockquote><tt>git clone --reference LOCAL.git REMOTE.git my_new_clone</tt></blockquote>
<p>(This is sort of like cloning <code>LOCAL.git</code>, changing its remote to
point to <code>REMOTE.git</code>, and fetching again, but much easier, and
doesn't pollute your clone with branches and commits that are only present
in <code>LOCAL.git</code> but not in <code>REMOTE.git</code>. From a content
perspective it is exactly a clone of <code>REMOTE.git</code>.)</p>
<p>Note: if <code>LOCAL.git</code> is on the same filesystem, git sets up
the <code>alternates</code> file so that object IDs can be resolved just
using the object files in <code>LOCAL.git</code>. This may be undesirable
because you won't actually have the objects in your new repo(!) and references in your new repo alone will not(!) be sufficient to keep the
objects from being GC'd if <code>LOCAL.git</code> changes. To break this
dependency, forcing the required objects to actually be copied into your new
clone, do the following:</p>
<blockquote><tt>git repack -a; rm .git/objects/info/alternates</tt></blockquote>
<p><b>gfind</b></p>
<p>To make git print out all the files it is tracking:</p>
<blockquote><tt>git ls-tree -r --name-only HEAD</tt></blockquote>
<p>This is a useful base for searching your codebase, since it automatically
lets you search all of your actual code and avoid looking at compiled code,
generated code, downloaded libraries and documentation, etc.</p>
<p>I have aliased this to <code>gfind</code> and do things like this to search
within a repo:</p>
<blockquote><tt>gfind | xargs grep FOO</tt></blockquote>
<p><b>git-filter-branch</b></p>
<p><code>git-filter-branch</code> is your general purpose tool for rewriting
repositories wholesale, for example, to extract a single subdirectory
retaining all its history, or to excise a subdirectory, file, secret key,
password, etc. that you've put in the repository.</p>
<p>Matthew McCullough has put
together <a href="http://www.nofluffjuststuff.com/blog/matthew_mccullough/2010/12/git_filter_branch_examples">a
good starting point</a> with examples of how to use this tool.
(Unfortunately, some of the links are broken; I will post alternate links
once I find track down copies.)</p>
<p><b>git diff --color-words</b></p>
<p>Word based diffs in git! Great for LaTeX, Markdown, etc. More
info <a href="http://idnotfound.wordpress.com/2009/05/09/word-by-word-diffs-in-git/">here</a>.</p>
Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com1tag:blogger.com,1999:blog-970055329001593038.post-2909287095384922992011-09-17T02:42:00.000-07:002011-09-17T02:43:28.789-07:00Zeya 0.6<p>I'm pleased to announce the (long overdue…) release
of <a href="http://web.psung.name/zeya/">Zeya</a> 0.6.</p>
<p>This release contains only bug fixes and minor changes that help Zeya to "just work" under a wider variety of circumstances.</p>
<p>Major changes since Zeya 0.5:</p>
<ul>
<li>The 'dir' backend works with Python 2.5 again. (Thanks to Greg Grossmeier)</li>
<li>Broken symlinks are detected and ignored. (Thanks to Etienne Millon)</li>
<li>Unreadable playlist files are detected and ignored.</li>
<li>The 'dir' backend sorts files case-insensitively. (Thanks to Greg Grossmeier)</li>
<li>Zeya no longer leaks file handles under certain circumstances. (Thanks to Pior Bastida)</li>
<li>The frontend uses relative paths to resources, so Zeya can be run behind a reverse proxy out of the box, e.g. at <tt>http://yourhostname/path/to/zeya</tt> (Thanks to Jérôme Charaoui)</li>
</ul>
<p>See <a href="http://web.psung.name/zeya/"><tt>http://web.psung.name/zeya/</tt></a> to learn more about Zeya, installation, getting started, reporting bugs, and development.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com4tag:blogger.com,1999:blog-970055329001593038.post-45061205679438708082011-09-07T22:18:00.000-07:002011-09-07T22:18:11.650-07:00Why Zeya?<p>I noticed that <a href="http://web.psung.name/zeya/">Zeya</a> is now more
than two years old. (Yow! Version 0.1 was released in August 2009! Thanks to
you all for all the patches, by the way. Zeya has become much more popular
than I would ever have guessed at the time.)</p>
<p>This got me thinking about what has, and hasn't, gotten better in that time
with respect to music software, and why I thought Zeya was so useful in the
first place.</p>
<p>Conceptually, you can think of any computer that is involved in playing your
music as providing one or more of the following functions:</p>
<ol>
<li>Storage: storing your music in nonvolatile storage and retrieving it</li>
<li>Control: letting you push buttons and see what is playing or available to play</li>
<li>Playback: driving a set of speakers</li>
</ol>
<p>Traditionally all three functions have been performed by the same computer.
But there's no reason they can't all run on different computers, not when you
have a moderate- or high-speed network connecting them all. You might ask why
you would actually want to separate these functions. The answer is that each
function is best suited to be performed by a computer with a specific set of
attributes, and the requirements for all of the functions are at odds with
each other, and so require some compromise if they are to be colocated.</p>
<ol>
<li>Storage wants to be done on a computer that has an enormous disk and is
possibly continuously backed up.</li>
<li>Control wants to be done on a computer that is easy to reach
(physically), possibly even one that fits in your pocket or can otherwise
be carried around.</li>
<li>Playback wants to be done on a computer that permanently sits somewhere
you want to hang out and is attached to a set of sweet speakers.</li>
</ol>
<p>The original impetus for writing Zeya was that it became clear to me that
storage constraints were becoming increasingly annoying. People switched
their portable devices from spinning rust (classic iPod and laptop HDDs) to
flash memory (smartphones, smartphone-like devices like the iPod touch, and
laptop SSDs), and all of a sudden, with the accompanying drop in capacity
(most smartphones, for example, top out at just 16GB or 32GB), many people
could no longer carry their entire music collection with them.</p>
<p>At a higher level, Zeya completes the trio of tools that lets you decouple
the components in varying ways:</p>
<ol>
<li>Zeya and similar tools decouple storage from control and playback.
Conceptually Amazon Cloud Player and Google Music are doing the same thing,
too, though there the storage happens in the cloud.</li>
<li>X11 decouples control from storage and playback.</li>
<li><a href="http://psung.blogspot.com/2011/09/network-audio-with-pulseaudio-made.html">PulseAudio</a>
decouples playback from storage and control.</li>
</ol>
<p>(If you use any two of those, you can completely decouple all three
functions, modulo the fact that X11 and PulseAudio don't work very well over
high-latency links.)</p>
<p>There has long been a patchwork of pieces that provide flexibility along one
or more of the above dimensions (iTunes, Sonos, etc.) and have various
tradeoffs; though, only in the past year have we seen the launches of
high-profile products like Amazon Cloud Player and Google Music, which are
more directly analogous to Zeya. All the attention in this area means that
people are finally starting to understand the possibilities of the
technology. Namely, that when you have network access everywhere, physical
distance (and the need for the colocation of certain things) becomes much
less important.</p>
Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com3tag:blogger.com,1999:blog-970055329001593038.post-25666312351530926832011-09-07T20:52:00.000-07:002011-09-07T20:52:03.498-07:00Network audio with PulseAudio made (somewhat) easy<p>I figured it was long past time to buckle down and learn how to make
PulseAudio do my bidding and redirect audio across a network link. And I was
surprised to learn that it's actually not hard to set up. In fact <b>you don't need to
touch any config files in <tt>/etc</tt></b>. Which you would never know, from
reading most of the documentation that is out there.</p>
<p>While network audio is still kind of flaky at times, you only "pay" for it
if you use it (people complain about PulseAudio a lot, but in my experience
it works very reliably when used locally), and it can come in very handy.</p>
<p><b>Background</b></p>
<p>Throughout, I'll refer to the two computer roles as audio <em>source</em>
and audio <em>sink</em> (where audio data is generated/decoded and where the
speakers are attached, respectively). Note that these may differ from the
PulseAudio concepts of the same names.</p>
<p>Ubuntu ships with PulseAudio, and many apps (among the common graphical apps
and music players) understand how to talk to PulseAudio now (possibly through
a compatibility layer). Except maybe Flash, but everyone who uses Flash is
already used to it not being a proper citizen. The default setup is to have a
per-user instance of PulseAudio running alongside the X session. This means
that someone has to be logged in to X on both ends. If your audio sink is
headless then you might have a single system-wide PulseAudio instance
instead. Configuring that is not covered here.</p>
<p>PulseAudio runs as a service on port 4713. We need to first make that
service discoverable on the machine with the audio sink, and then provide a
way for the machine with the audio source to authenticate to the sink (so
that you're not just letting <em>anyone</em> on your network play their
crappy music).</p>
<p><b>Initial steps</b></p>
<p>On the machine with the audio sink:</p>
<ul>
<li>Install and run <tt>paprefs &</tt></li>
<li>Go to the <b>Network server</b> tab.</li>
<li>Check <b>Enable network access to local sound devices</b> and <b>Allow
other machines on LAN to discover local sound devices</b>. Autodiscovery
uses Avahi, which you'll need to re-enable, if you ever disabled it.</li>
</ul>
<p>(You might have to restart PulseAudio or X for the settings to take
effect.)</p>
<p>I will describe two ways you can get the machine with the audio source to
supply credentials to the sink so it can play music. The first one is likely
to be more generally useful.</p>
<p><b>Using .pulse-cookie</b></p>
<p>The <tt>~/.pulse-cookie</tt> file is a shared secret that can be used for
authentication. Just copy that file from either machine (sink or source) to
the other so that they are the same on both ends.</p>
<p>At the audio source, install <tt>padevchooser</tt> and run it
(<tt>padevchooser &</tt>). A menu will appear in your notification area.
Under <b>Default server</b> you should see an entry for the sink, assuming it
is on the same local network (it should be named <tt>username@hostname</tt>).
Select it.</p>
<p>Now run the application of your choice and play some audio!</p>
<p><b>Piggybacking on X11 forwarding</b></p>
<p>This method has the advantage of not requiring working autodiscovery. So you
can use it on a LAN without Avahi running, or over a non-local network. All
you need is to be able to SSH from the sink to the source. We will forward
the audio from the source to the sink on a TCP port.</p>
<p>For this method, we'll transfer the credentials that the source will need by
writing them to the X11 <em>root window</em>. PulseAudio supplies a program
that does exactly this. On the machine with the audio sink, run:</p>
<blockquote><tt>sink$ start-pulseaudio-x11</tt></blockquote>
<p>(You can double-check that this is working by running <tt>xprop -root | grep
PULSE</tt> and checking that there is an entry for <tt>PULSE_COOKIE</tt>,
among others.)</p>
<p>Now initiate an SSH connection to the source, tunneling a port of your
choice on that end over to your local PulseAudio server:</p>
<blockquote><tt>sink$ ssh -X -R <b>9998</b>:localhost:4713 source</tt></blockquote>
<p>Run the application of your choice and play some audio! As you are doing so,
set the <tt>PULSE_SERVER</tt> variable, which now enables your remote
application to talk to your local audio hardware via the tunneled port.</p>
<blockquote><tt>source$ PULSE_SERVER=localhost:<b>9998</b> rhythmbox &</tt></blockquote>
<p><b>Conclusion and caveats</b></p>
<p><em>Et voila</em>, now you've decoupled your music player from your
speakers. So you can play music from your laptop using the speakers at your
desktop. Or you can play music from your desktop using the speakers in your
living room. Some things to watch out for.</p>
<ul>
<li>You may need to kill and restart <tt>pulseaudio</tt> (or X) to get it to
pick up some of those configuration changes.</li>
<li>PulseAudio will not move a stream while it is running, so if you use the
menu to change the destination, the change will not take effect until the
next song begins, unless you pause and restart playback in your audio
application.</li>
<li>PulseAudio is kind of finicky and some of it still feels like black magic
at times. You may need to restart it if you suddenly get errors about
things not working, especially if they were just working a minute ago.</li>
</ul>
<p>Sources and further reading: <a href="http://www.mail-archive.com/pulseaudio-discuss@mail.0pointer.de/msg09866.html">pulseaudio-discuss</a>, <a href="https://wiki.archlinux.org/index.php/PulseAudio">ArchLinux wiki</a>, <a href="http://ubuntuforums.org/showthread.php?p=11052460">Ubuntu forums</a></p>
Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com4tag:blogger.com,1999:blog-970055329001593038.post-37524751976829803182011-09-07T19:36:00.000-07:002011-09-07T19:40:58.821-07:00QR codes in LaTeX<p>If you are adding QR codes to print media, in order to make them look really
sharp, you want the QR codes to be generated in a vector format rather than a
bitmap format. It turns out that
the <tt><a href="http://ctan.org/tex-archive/graphics/pstricks/contrib/pst-barcode">pst-barcode</a></tt>
package allows you to easily add vectorized QR codes to your LaTeX documents.</p>
<p>Here are some minimal steps to generate a PDF with a QR code in it:</p>
<p>1. Install dependencies:</p>
<blockquote><pre>$ aptitude install texlive-latex-{base,extra}</pre></blockquote>
<p>(This works on Ubuntu 11.04, at least.)</p>
<p>2. Add the following to a .tex file:</p>
<blockquote><pre>\documentclass{article}
\usepackage{pst-barcode}
\usepackage{auto-pst-pdf}
\begin{document}
\begin{pspicture}(1in,1in)
\psbarcode{<b>PAYLOAD</b>}{eclevel=M width=1.0 height=1.0}{qrcode}
\end{pspicture}
\end{document}</pre></blockquote>
<p>where <tt>PAYLOAD</tt> gives the data to be encoded. For a business card you
might have something like:</p>
<blockquote><pre>MECARD:N:Sung,Phil;TEL:+14085551234;EMAIL:philbert@gmail.com;URL:http://web.psung.name;;</pre></blockquote>
<p>See <a href="http://code.google.com/p/zxing/wiki/BarcodeContents">this
page</a> for more <tt>MECARD</tt> options and for descriptions of the other protocols (URLs, email addresses,
etc.) that barcode readers understand.</p>
<p>3. Compile your file as follows:</p>
<blockquote><pre>$ pdflatex <b>--shell-escape</b> yourfile.tex</pre></blockquote>
<p>Some notes:</p>
<ul>
<li><tt>eclevel</tt> specifies the level of error correction, and is one of
<tt>L</tt>, <tt>M</tt>, <tt>Q</tt>, <tt>H</tt> (low to high)</li>
<li><tt>width</tt> and <tt>height</tt> specify the dimensions of the
barcode.</li>
<li><tt>pst-barcode</tt> knows how to generate barcodes in many other
formats;
see <a href="http://mirrors.ctan.org/graphics/pstricks/contrib/pst-barcode/pst-barcode-doc.pdf">the
documentation</a> for details.</li>
</ul>
<p>You can also change the color of the barcode by adding something like the
following:</p>
<blockquote><pre><b>\usepackage{color}</b>
[...]
\psbarcode<b>[linecolor=blue]</b>{PAYLOAD}[...]</pre></blockquote>
<p>Sources: <a href="http://tex.stackexchange.com/questions/10584/generating-2d-barcodes">StackExchange</a>, <a href="http://blog.widmann.org.uk/2009/05/27/1297/">Thomas Widmann</a>, <a href="http://bramp.net/blog/latex-qr-based-business-card">Andrew Brampton</a> (who has a nice template for a business card)</p>
Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-59951680506530666412011-08-08T01:07:00.000-07:002011-08-08T01:07:38.931-07:00"X11 connection rejected because of wrong authentication."<p>A brief note for posterity: if you are getting an error like the following,</p>
<blockquote><p><tt>X11 connection rejected because of wrong authentication.<br>xterm Xt error: Can't open display: localhost:10.0</tt></p></blockquote>
<p>one thing you might try, after you have eliminated <a href="http://www.cyberciti.biz/faq/x11-connection-rejected-because-of-wrong-authentication/">the</a> <a href="http://jianmingli.com/wp/?p=724">usual</a> <a href="http://www.google.com/search?ie=UTF-8&q=X11+connection+rejected+because+of+wrong+authentication.">suspects</a> (no, try those first, really), is unsetting the <tt>XAUTHORITY</tt> environment variable:</p>
<blockquote><p><tt>$ unset XAUTHORITY</tt></p></blockquote>
<p style="text-align: center">* * *</p>
<p>The remainder of this post is a followup to my previous
post, <a href="http://psung.blogspot.com/2008/10/stupid-screen-tricks.html">Stupid
Screen Tricks</a>. (The setup I described therein is still alive and well.)</p>
<p>I received the above error when I started a screen session on
display <tt>:0.0</tt> and (after disconnecting and reconnecting) subsequently tried to launch, from within it, a new
X program on a different display (even after setting <tt>DISPLAY</tt>). The
session inherits the old <tt>XAUTHORITY</tt> value,
which, for whatever reason, for as long as it is present, foils attempts to
run programs on other displays.</p>
<p>To work around this, I changed my <tt>here</tt> utility alias to the
following,</p>
<blockquote><p><tt>alias here='unset XAUTHORITY; DISPLAY=`cat ~/.last-display`'</tt></p></blockquote>
<p>that is, unsetting <tt>XAUTHORITY</tt> as well as setting <tt>DISPLAY</tt>
before trying to run anything.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com1tag:blogger.com,1999:blog-970055329001593038.post-52499597205065835952011-08-08T00:35:00.000-07:002011-08-08T00:35:45.010-07:00Assorted notes<ul><li>Hugin, the panorama-stitching tool, is quite slick these days (ever since it got some UI improvements in release 2010.4.0 or so, I believe). It handled perhaps 80-90% of the panoramas from <a href="https://picasaweb.google.com/philbert/TourOfTheAlps2011">my recent trip</a> in about 3 clicks and without manual intervention. Hats off to the Hugin developers.</li>
<li>I've been learning how to twiddle the "Projection" knob in Hugin, which lets you change the geometry of the resulting panorama. For example, for photos of murals (or tapestries, or other long, flat, surfaces), the Rectlinear projection corrects for the distortion to reconstitute the "flat" image, as if you were standing far away.</li>
</ul><p style="text-align: center"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKGDM46j4gNfvKPeGGbe7y_VlT7Ru6TO3Ww3A13ThGUEo1Y4ShK4zmnve-qVnIJ403yY-7JcXTYa6FJIScu3Zs3VM2_T0KOUdG6qV78JQlmdUW-CNULXptWn_iR8FnAP1SvdR5ZZ6aMN8/s640/4263b.jpg"><br><em>Default (cylindrical)</em></p><p style="text-align: center"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXbYjYvydqM1l0CCD4NZBWoH0aWxCg_LyZYlfdODN18bLOUQDZ4yFGYCatExZFLCO0syU1j7BuN635c9l7-PqYEjNFyfe-8aU0IANZSLk3Q3PJsFomWqJaLHLHCi2uyhHtGY8ZN1Gz9-4/s640/4263.jpg"><br><em>Corrected (rectilinear)</em></p><ul><li><p>Ubuntu has these new scrollbars with invisible scroll handles. They are quite annoying (being invisible and all). You can revert to the old ones like so:</p><blockquote><tt>sudo apt-get remove overlay-scrollbar liboverlay-scrollbar-0.1-0</tt></blockquote><p><a href="http://www.webupd8.org/2011/04/how-to-disable-overlay-scrollbars-in.html">Source</a></p></li></ul>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-51273172126115518452011-06-17T16:24:00.000-07:002011-06-17T16:24:08.678-07:00Spring cleaning, dealing with duplicate files, and vacuumpack<p>I was doing some spring cleaning of my computer, part of which was removing
duplicate files. I'm a packrat, and I subscribe to the
copy-first-and-ask-questions-later school of thought, so I have a few
duplicates floating around, just taking up disk space. (I think I have
somewhere a filesystem copy of my previous computer, which, in turn, contains
a filesystem copy of my computer before that.)</p>
<p>There are plenty of tools that will help you remove duplicate files
(<a href="http://en.wikipedia.org/wiki/Fdupes">this page lists no fewer than
16</a>), but, disappointingly, none of them that I could find seem to give
you a high-level understanding of where and how duplicate files appear in
your filesystem. So I wrote <tt><b>vacuumpack</b></tt>, a short Python tool
to help me really <em>see</em> what was going on in my filesystem. Nothing
revolutionary, but it helped me through my spring cleaning.</p>
<p>(Aside 1: I noticed the parallel with coding theory: you want
to <a href="http://en.wikipedia.org/wiki/Data_compression">remove
undesirable redundancy</a>, by removing duplicate files on the same disk,
and <a href="http://en.wikipedia.org/wiki/Error-correction_code">add
desirable redundancy</a>, by using backups or RAID.)</p>
<p>(Aside 2: This is simple enough that undoubtedly someone will send a pointer to a tool written circa 1988 that does what I am describing. C'est la vie; unfortunately it is usually easier, and more fun, to write code than to find it.)</p>
<p><tt>vacuumpack</tt> is probably best explained by example.
First, <tt>vacuumpack</tt> can scan a directory and identify duplicated
files:</p>
<p><code><b>$ ./vacuumpack.py --target=/home/phil --cache=/home/phil/.contentindex</b></code></p>
<p>The output shows clusters of identical files. For example, <em>oh hey, it's
the GPLv3</em> (the cluster is prefaced by the <tt>sha256</tt> of the
shared content):</p>
<blockquote style="font-size: x-small"><pre>[...]
8ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903: 140588 bytes wasted
/home/phil/projects/raytracer/COPYING (35147 bytes)
/home/phil/projects/rdiff-snapshot-fs/source/COPYING (35147 bytes)
/home/phil/projects/syzygy/COPYING (35147 bytes)
/home/phil/projects/vacuumpack/COPYING (35147 bytes)
/home/phil/source/emacs/COPYING (35147 bytes)
[...]</pre></blockquote>
<p>The clusters are sorted in decreasing order of the amount of space wasted,
so you can fry the big fish first. Having multiple copies of the GPL floating
around isn't really a cause for concern; let's have a look at another
cluster:</p>
<blockquote style="font-size: x-small"><pre>6940e64dd91f1ac77c43f1f326f2416f4b54728ff47a529b190b7dadde78ea23: 714727 bytes wasted
/home/phil/photos/20060508/may-031.jpg (714727 bytes)
/home/phil/album1/c.jpg (714727 bytes)</pre></blockquote>
<p>Thus far <a href="http://duff.sourceforge.net/"><tt>duff</tt></a> and many
other tools do essentially the same thing. But most of the tools out there
are focused on semi-mechanically helping you <em>delete</em> files, even
though cleaning up often requires <em>moving</em> or <em>copying</em> them as
well.</p>
<p>Duplicated photos, like the ones above, are probably something I want to
resolve. Since I recall that <tt>/home/phil/photos</tt> is the canonical
location for most of my photos, the other directory looks somewhat suspect.
So I'll ask <tt>vacuumpack</tt> to tell me more about it:</p>
<p><code>$ ./vacuumpack.py --target=/home/phil --cache=/home/phil/.contentindex <b>/home/phil/album1</b></code></p>
<p>Now, for each file in that directory, <tt>vacuumpack</tt> tells me whether
it knows about any duplicates:</p>
<blockquote style="font-size: x-small"><pre>/home/phil/album1/a.jpg == /home/phil/photos/20060508/may-010.jpg
/home/phil/album1/b.jpg == /home/phil/photos/20060508/may-019.jpg
/home/phil/album1/c.jpg == /home/phil/photos/20060508/may-031.jpg
/home/phil/album1/d.jpg == /home/phil/photos/20060508/may-033.jpg
/home/phil/album1/e.jpg == /home/phil/photos/20060508/may-048.jpg
/home/phil/album1/f.jpg -- NO DUPLICATES
/home/phil/album1/g.jpg == /home/phil/photos/20060508/may-077.jpg
/home/phil/album1/h.jpg == /home/phil/photos/20060508/may-096.jpg</pre></blockquote>
<p>It looks like the files in <tt>/home/phil/album1</tt> are a subset of the
photos in <tt>/home/phil/photos</tt>... except
that <tt>/home/phil/photos</tt> is missing a file! I need to copy that file
back; once I do, the directory <tt>album1</tt> is safe to delete.</p>
<p>In this mode <tt>vacuumpack</tt> is behaving like a directory diff tool,
except that it uses content rather than filenames to match up files.</p>
<p>A majority of the code in <tt>vacuumpack</tt> is actually devoted to
identifying duplicates efficiently. <tt>vacuumpack</tt> stores the file
hashes and metadata in a cache (specified by <tt>--cache=...</tt>) and
automatically rereads files when (and only when) they have been modified. So
after the initial run, <tt>vacuumpack</tt> runs very quickly (e.g. in just
seconds on my entire homedir) while always producing up-to-date reports. It's
fast enough that you can run it semi-interactively, using it to check your
work continuously while you're reorganizing and cleaning your files. You can drill down and ask lots of questions about different directories without having to wait twenty minutes for each answer.</p>
<p>I've posted a git repo containing the <tt>vacuumpack</tt> source:</p>
<blockquote><code>git clone http://web.psung.name/git/vacuumpack.git</code></blockquote>
<p>This has been tested on Ubuntu 11.04 under Python 2.7.1. You may use the
code under the terms of the GNU GPL v3 or (at your option) any later
version.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com2tag:blogger.com,1999:blog-970055329001593038.post-7398421123702076482011-06-17T14:38:00.000-07:002011-06-17T14:38:14.702-07:00Assorted notes<ul>
<li>Public service announcement: earlier this year Google announced optional
<a href="http://www.google.com/support/accounts/bin/static.py?page=guide.cs&guide=1056283"><b>2-factor
authentication for Google accounts</b></a>. <em>Please use it</em>: it's
one of the least painful ways to make your data safer (most people are
toast if their email gets compromised). And the implementation seems fairly
well thought out:</li>
<ul>
<li>You download
an <a href="https://market.android.com/details?id=com.google.android.apps.authenticator">app</a>
to your smartphone (or smartphone-like device) that generates one-time
passwords (OTPs), to be used in conjunction with your regular password
when needed. A single OTP can authenticate one computer for up to 30
days. Yes, the app is open source. It runs on any Android, Blackberry, or
iOS device.</li>
<li>The app works offline, without a data connection, because the method
for generating OTPs is specified
by <a href="http://tools.ietf.org/html/rfc4226">RFC 4226</a> (yes, it's
standardized and everything) and is either sequence-based or
time-based.</li>
<li>Failing that, if you don't have a smartphone, or it's busted, you can
also receive an OTP via SMS to a designated number (though, obviously,
then you need phone reception).</li>
<li>Failing that, if you don't have a cell phone, or it's busted, you can
also receive an OTP via a voice call to a designated landline.</li>
<li>Failing that... if you know you'll be somewhere where you have no phone
at all, you can <em>print</em> a list of OTPs to carry with you that will
enable you to log in.</li>
<li>Apps that authenticate via just a password (e.g. the phone itself, or
most desktop apps, like Picasa) get a dedicated automatically generated
password. You don't get the benefit of 2-factor auth here, but these
passwords are less likely to be phished because you're not typing them in
all the time, and you can revoke them individually.</li>
</ul>
<li>Good lord, Ubuntu 11.04 (Natty) is fast. My laptop (Thinkpad X201 with
Intel SSD) boots from disk unlock screen (LUKS full-disk encryption) to a
working Openbox desktop in about <em>four seconds</em>.</li>
<li>I've been playing
with <a href="http://www.blender.org/"><b>Blender</b></a> (the Free 3D
modeling tool) for a personal project to be 3D printed, and it's a lot of
fun, and quite rewarding. I'm still a noob at this stuff, but already I get
some of these "in the zone" moments that are so rarely attained in software
(Emacs being the other exception) where I feel like I'm manipulating
a <em>thing</em> directly rather than <em>using a software program</em>.
The Blender UI looks like an airplane cockpit, but there is a method to
its madness! The other neat thing is that most of the time when you do
creative work on the computer you are not rewarded with anything nearly so
tangible as a 3D printed piece.</li>
<li>A clever thing I noticed on Android the other week: when you use voice
dictation in a text entry field, and you move the cursor back to previous
words, above the keyboard it shows not the nearest alternatives based on the
keyboard layout (as it would if you were typing), but the nearest alternatives
based on sound— e.g. "wreck" ... "a nice beach" as suggested
replacements for "recognize speech".</li>
</ul>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-64645462652196291562011-02-27T23:11:00.000-08:002011-02-27T23:11:00.618-08:00Reducing merge headaches: git meets diff3<p>git has an option to display merge conflicts in <tt>diff3</tt> format (by default it only displays the two files to be merged). You can
enable it like so:</p>
<blockquote><tt>$ git config --global merge.conflictstyle diff3</tt></blockquote>
<p>Now, when you have to resolve merge conflicts, git shows your side, the side
being merged, and (here's what's new) the common ancestor in between them.
Here's an example of the <tt>diff3</tt>-formatted output:</p>
<blockquote><tt>cauliflower<br>
<<<<<<< HEAD<br>
peas<br>
potatoes<br>
||||||| merged common ancestors<br>
peas<br>
=======<br>
>>>>>>> topic<br>
tomatoes</tt></blockquote>
<p>Having the merge ancestor readily available helps you to quickly determine
what the correct merge is, since you can infer from it the changes that were made on both sides. Here you can see that the original state
was <tt>peas</tt>. On your branch <tt>potatoes</tt> was added (compare the
middle section to the top) and on the other branch <tt>peas</tt> was removed
(compare the middle section to the bottom). Therefore the correct change is to
both add <tt>potatoes</tt> and remove <tt>peas</tt>, leaving you with
just <tt>potatoes</tt> in the conflicted section.</p>
<p>There's really no reason you shouldn't enable the <tt>diff3</tt>
style, because <b>you frequently need the ancestor to determine what the correct
merge is.</b></p>
<p>To see that this is true, even in the simple example above, look at what the conflict looks like under the standard
style:</p>
<blockquote><tt>cauliflower<br>
<<<<<<< HEAD<br>
peas<br>
potatoes<br>
=======<br>
>>>>>>> topic<br>
tomatoes</tt></blockquote>
<p>There's an asymmetry between <tt>peas</tt> and <tt>potatoes</tt>: one was
added and one was deleted, but this merge conflict doesn't tell you anything at
all about which was which! You can't determine the correct merge unless you
remember the sequence of changes that led up to this point. And why should you
have to rack your brain to do that? That's exactly the sort of thing that your
computer can, and should, help you with.</p>
<p><b>Bonus tip: rerere (reuse recorded resolution)</b></p>
<p>If your workflow finds you redoing the same merges over and over again you
might also find git's <tt>rerere</tt> (reuse recorded resolution) feature to be
useful.</p>
<p>One of the things that is wonderful about <tt>rerere</tt> is that it
provides hardly any UI surface at all. Just set it...</p>
<blockquote><tt>$ git config --global rerere.enabled 1</tt></blockquote>
<p>...and forget it. Although there is a <tt>git rerere</tt> command, you can get a lot done without using it at all.</p>
<p>After enabling <tt>rerere</tt>, whenever you resolve a merge conflict, git
automatically squirrels away the resolution in its database. You'll see a message like this
one:</p>
<blockquote><tt>$ git commit<br>
Recorded resolution for 'soup'<br>
[...]</tt></blockquote>
<p>And the next time you encounter the same conflict, where you would have
expected git to spit out a file with conflict markers, you will instead find that it has automatically resolved the merge for you, and printed the
following message:</p>
<blockquote><tt>$ git merge topic<br>
Auto-merging soup<br>
CONFLICT (content): Merge conflict in soup<br>
Resolved 'soup' using previous resolution.</tt></blockquote>
<p>Just double-check to make sure nothing has gone awry, add, and commit. Save
your blood, sweat, and tears for other, more interesting problems than redoing
merges.</p>
<p>Further reading: <a href="http://progit.org/2010/03/08/rerere.html">Pro Git on rerere</a></p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com24tag:blogger.com,1999:blog-970055329001593038.post-74729275470802726232011-01-25T20:11:00.000-08:002011-01-25T20:11:19.337-08:00Making your own page-a-day calendar, revisited<p>Some time ago I posted <a href="http://psung.blogspot.com/2009/06/making-your-own-page-day-calendar.html">instructions for designing and producing a page-a-day calendar</a>, which is a moderately neat project at the intersection of metaprogramming and handicraft.</p>
<p><a href="http://blog.mattj.me/">Matt Johnson</a> made one for the year 2011; <a href="http://blog.mattj.me/latex-page-a-day-calendar" style="font-weight: bold">his write-up</a> fills a lot of the gaps in my rather skeletal instructions and provides a number of suggestions for extending the original design in very good ways (including adding page content not based on photographs, and making a wooden stand for the calendar). Read it if you are thinking of making one of these things.</p>
<p>Thanks, Matt!</p>
<p style="text-align: center"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXjFX8VWZg04C4nICWcXkk8Im2y8T35RfO9l5IUR7C-MPoeuIE5lhP8ZshR-LZHhFAL9_t19UNq5gI89h-qoQFYdtnSegvJ_WVgR5gnZu9-buHQp9wc5_e-FgFeExbVX0nBsYrWIbIM_c//"> <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCYT7vXYSgbqyYXxcsSqMRSoSkDsV-nEdFEmvWAikR4h_tc8nu-qJsql-pLKYJ-tFQcPlGVS3wx0MmYa_Gkux6t2a30fM3ieDzXbrcH8BUHXl60vq_OPmZWmrpPvMjtWk3IZIxDEufER4//"><br><small>CC-BY <a href="http://blog.mattj.me/">Matt Johnson</a></small></p>
<p><a href="http://blog.mattj.me/latex-page-a-day-calendar" style="font-weight: bold">Read more.</a></p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-13467436318535134962010-12-31T21:39:00.000-08:002010-12-31T21:39:51.853-08:002010 in Review<p>These were the most linked-to posts on my blogs in the last year:</p>
<ul>
<li><a href="http://philipsung.blogspot.com/2010/12/people-who-are-not-in-our-league-v.html">People Who Are Not In Our League V</a></li>
<li><a href="http://psung.blogspot.com/2010/10/thank-you-ubuntu.html">Thank You, Ubuntu</a></li>
<li><a href="http://psung.blogspot.com/2007/12/for-else-in-python.html">for ... else in Python</a></li>
<li><a href="http://psung.blogspot.com/2010/04/thinkpad-trackpoint-scrolling-in-ubuntu.html">
Thinkpad TrackPoint Scrolling in Ubuntu Lucid/10.04 (2010)</a></li>
<li><a href="http://psung.blogspot.com/2008/09/scrolling-with-thinkpads-trackpoint-in.html">
Scrolling with the Thinkpad's TrackPoint in Ubuntu (2008)</a></li>
<li><a href="http://psung.blogspot.com/2009/12/subtleties-of-search-and-replace-in.html">Subtleties of search-and-replace in Emacs</a></li>
<li><a href="http://psung.blogspot.com/2009/09/getting-more-flow-from-your-window.html">Getting more "flow" from your window manager</a></li>
</ul>
<p>Happy new year, peace on earth, and may all your builds be green!</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-65376253129463725132010-12-31T13:06:00.000-08:002010-12-31T13:06:47.566-08:00Making the leap to SSD...<p>The Intel SSDs were on sale around Thanksgiving, so I picked up one for my Thinkpad.</p>
<p>There is little that was remarkable in getting it set up. I swapped out the
old spinning rust device. (The X201, unlike any other laptop I've opened up,
has rubber shock absorbers that fit between the drive and the chassis. Not
that the thing they're guarding is particularly susceptible to mechanical
damage any longer.) Everything just worked.</p>
<p>Boot times and application load times are dramatically improved. Though, the
way I work, my laptop is essentially just a device for running Emacs, Chrome,
and gnome-terminal, all of which I just open and keep open. All of those apps
load fairly quickly, so the speed improvement from the SSD, while nice, is
not by any means life-changing. (And, how often do you reboot a computer,
anyway? But, for the record, my Thinkpad goes from LUKS password prompt to
login screen in 10 seconds, and to a ready-to-use desktop in about 3
more.)</p>
<p>What's really nice about the SSD is that my laptop is now nearly silent, or
at least quieter than ambient noise. Perhaps I just have unusually low noise
tolerance, but I find the steady-noise of a HDD to be somewhat annoying, and
the sound of a HDD spinning up to be even more annoying. Unfortunately, I
actually didn't correctly attribute those noises to the HDD until recently.
If this sounds like you, it's time to pay a visit to the computer store.</p>
<p>One caveat is that if you use LUKS full disk encryption (and if you're using a laptop, you really
ought to; no, I mean, you <em>really</em> ought to), then Ubuntu (10.10)
doesn't issue TRIM commands to the device. Which will eventually lead to
decreased throughput, but I believe that may just be the price of security. (IANA security expert...) LUKS goes
to great lengths by default to foil cryptanalysis, including initializing
encrypted partitions with random data to obscure which sectors of the device
are actually being filled with interesting data. This would all be for naught
if the OS was periodically telling the device exactly which sectors were no
longer being used.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-69881773645951448242010-12-31T12:16:00.000-08:002010-12-31T12:16:32.108-08:00Content-aware image resizing by seam carving<p>This is a classic, or, as much of a classic as a 3-year-old graphics paper
can be.</p>
<p><a href="http://yuwing.kaist.ac.kr/courses/CS770/reading/seamcarving.pdf">
Seam Carving for Content-Aware Image Resizing</a> (Avidan and Shamir, 2007)
describes a technique for resizing photographs along one dimension in a way
that allows the aspect ratio to be changed while reducing the amount of
distortion in perceptually important parts of the image. Watch the video, if
you haven't seen this before:</p>
<p style="text-align: center"><object width="480" height="385"><param name="movie" value="http://www.youtube.com/v/qadw0BRKeMk?fs=1&hl=en_US"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/qadw0BRKeMk?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="385"></embed></object><br><a href="http://www.youtube.com/watch?v=qadw0BRKeMk"><em>Youtube link</em></a></p>
<p>The algorithm translates into identifying the shortest path in a DAG and it
can be easily implemented by a dynamic programming algorithm. As Dmitry
pointed out to me some years ago, this makes an ideal teaching example for
dynamic programming, for a number of reasons. The algorithm is easily
visualized because the space of values to be computed maps straightforwardly onto the
pixels of the image. Both the naive exponential-time algorithm (testing all
3<sup><em>#rows</em></sup> paths) and the dynamic programming algorithm are
fairly easily expressed without too much additional machinery. And, unlike most classroom dynamic programming
questions I have encountered, which seemed either banal or contrived, the
applications of this one are visually appealing and fairly interesting. For example, as someone
pointed out to me, it makes the fallout of breaking up much easier to deal
with.</p>
<p style="text-align: center"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2Cf4sAmUAvWHEiRNu_WNvXG5lg5GoY1q_FpvKA7oaOlYkv74BMhHaMIz-EmJEG1nis4nRokKLuOHGKkfUgIjvwhJU9nQiFZyyAp7Gd3Z9i6BjlCARkl5TaDaihFj3xz-AwpCf4OkyNRc//"><br>Before... and after <em>(Source: video, above)</em></p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com2tag:blogger.com,1999:blog-970055329001593038.post-77307137130130109892010-12-31T11:47:00.000-08:002010-12-31T11:47:59.343-08:00Some Android tips<p>I've been using a Nexus One as my primary phone for about a year now. Here
are some tips and suggestions I've accumulated. Most of these tips should
also apply to the Nexus S and the G2 (which have stock Android builds), as
well as to other Android phones (Froyo+) modulo any vendor
customizations.</p>
<ul>
<li>Press and hold the Search button to activate voice search / voice
actions. This is indispensable! I use it a least a couple of times every
day. It makes the phone feel like <em>future tech</em>, not least because
it's dramatically better than anything I regularly deal with in phone
systems or on PCs. The phone recognizes special instructions, including
things like the following (for anything it doesn't recognize as an action,
it asks you to disambiguate or it falls back to a Google search):</li>
<ul>
<li>"Call Phil Sung, mobile"</li>
<li>"Navigate to Fry's Electronics"</li>
<li>"Map of gas stations"</li>
<li>"Note to self, buy more envelopes" (which sends you an email from yourself)</li>
</ul>
<li>Press and hold the Home button to easily get a list of recently used
apps.</li>
<li>Press and hold the Menu button to toggle the display of the soft
keyboard. Not usually needed, but occasionally useful.</li>
<li>When using the soft keyboard, you can drag your finger past the top edge
of the keyboard for quick access to digits and punctuation.</li>
<li>If you add your email address to the user dictionary, you can get it in
the list of autocompletions when filling out forms in the web browser.</li>
<li>It's important to control the quantity of notifications so that they're
at a level that is actually useful. Depending on how important some event
is, you can configure the phone to make noise or merely to show you a
notification the next time you turn on the phone. As an example, here's how
I'm set up:</li>
<ul>
<li>My phone only rings for phone calls, SMS, and IMs.</li>
<li>GMail generates a notification in the notification area but no noise
(configurable in GMail settings). Even so, I implemented an elaborate
system of GMail filters to keep all but the most important emails from
getting to my inbox and thus generating notifications.</li>
<li>Calendar events generate neither noise nor notifications (configurable
in Calendar settings). Instead, I put the Calendar widget on my home
screen, and I just look at it whenever I need to. My calendar time is
pretty sparsely scheduled, though. YMMV.</li>
</ul>
</ul>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com1tag:blogger.com,1999:blog-970055329001593038.post-78272183262815195532010-12-26T13:56:00.000-08:002010-12-26T13:56:00.183-08:00Easy wifi autoconfiguration with barcodes<p>ZXing's <b>Barcode Scanner</b> Android app (the most popular barcode app for
Android; it's available on the Market) can now configure a wifi network based
on information encoded in a QR code.</p>
<p>So you can say goodbye to having guests/patrons/visitors ask you how to log
in to your wifi network every time someone new visits. Instead of having to
select the correct network and type in the password (I thought computers were
supposed to relieve us of this kind of drudgery), all they have to do is scan a
barcode, and boom, they're online.</p>
<p><a href="http://zxing.appspot.com/generator/"><b>ZXing's QR Code
Generator</b></a> will help you make such a barcode (select "Wifi network" in
the dropdown), after which you can just print it out and leave it in the living
room.</p>
<p>If you wish, you can also create such a barcode yourself just by encoding
the SSID and password in the barcode payload. Here are a couple of examples
that illustrate the encoding method:</p>
<table style="font-size: small; width: 100%">
<tr>
<td style="text-align: center">
<p><b>WEP</b>: <tt>WIFI:S:mynetwork;T:WEP;P:00deadbeef;;</tt></p>
<p><a href="http://chart.apis.google.com/chart?cht=qr&chs=200x200&chl=WIFI:S:mynetwork;T:WEP;P:00deadbeef;;">
<img src="http://chart.apis.google.com/chart?cht=qr&chs=200x200&chl=WIFI:S:mynetwork;T:WEP;P:00deadbeef;;">
<br>(Example using Google Chart API)</a></p>
</td>
<td style="text-align: center">
<p><b>WPA</b>: <tt>WIFI:S:mynetwork;T:WPA;P:mypassword;;</tt></p>
<p><a href="http://chart.apis.google.com/chart?cht=qr&chs=200x200&chl=WIFI:S:mynetwork;T:WPA;P:mypassword;;">
<img src="http://chart.apis.google.com/chart?cht=qr&chs=200x200&chl=WIFI:S:mynetwork;T:WPA;P:mypassword;;">
<br>(Example using Google Chart API)</a></p>
</td>
</tr>
</table>
<p>Hat tip to <a href="http://vikram.eggwall.com/">Vikram Aggarwal</a> who
implemented this concept, originally as a standalone app, <a
href="http://vikram.eggwall.com/computers/wyscan.html">WyScan</a>, and then
later implemented it in the Barcode Scanner app.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com1tag:blogger.com,1999:blog-970055329001593038.post-14273750120071599982010-12-22T21:55:00.000-08:002010-12-22T21:55:25.593-08:00Inline expansion in bash<p><tt>bash</tt> has a lot of shortcuts to save you typing and thinking
time— globs, shell aliases, and history expansion, for starters; and a
bunch more.</p>
<p>When you use <tt>bash</tt> shortcuts extensively, though, you often end up
with inputs that are rather, er, opaque, e.g.</p>
<blockquote><tt>$ !-3 -r [mnop]* !-3$</tt></blockquote>
<p>You have to be fairly brave to press ENTER without looking over that a few
times. There is, though, a way out. Instead of trying to analyze what the
hell it is you just wrote, you could just <em>ask the computer to tell you
exactly what it's going to do</em>. <tt>bash</tt> has a number of features
that facilitate this by expanding shortcuts for you to inspect before you
execute them. I refer to these features collectively as <strong>inline
expansion</strong>.</p>
<p>There are three really useful inline expansion commands you can use at
a <tt>bash</tt> prompt that are essentially complementary to each other:</p>
<p><b>History expansion</b></p>
<p>If you add the following to your <tt>.inputrc</tt>,</p>
<blockquote><tt>$if Bash<br>
Space: magic-space<br>
$endif</tt></blockquote>
<p>then whenever you press space, any history expansions
(e.g. <tt>!!</tt>, <tt>!grep</tt>, <tt>!-2</tt>, <tt>!-3:1</tt>, etc.) are
expanded.</p>
<p>You can also, at any time, press <tt>M-^</tt> to perform history expansion
(no configuration required). I prefer to set up magic space because you can
just set it up and forget it about it, and subsequently you have one less
thing to think about.</p>
<p><b>Glob expansion</b></p>
<p><tt><b>C-x *</b></tt> expands the previous word into any filenames that match the
glob. For example, <tt>/*</tt> gets expanded into <tt>/bin /boot /cdrom /dev
...</tt>.</p>
<p><b>Shell expansion</b></p>
<p><tt><b>C-M-e</b></tt> expands shell aliases, history, variables, commands, and
arithmetic. For example:</p>
<ul>
<li><tt>ls</tt> into <tt>ls --color=auto</tt> (based on any <tt>alias</tt> directives you have)</li>
<li><tt>!grep</tt> into <tt>grep 2010 mylogfiles</tt> (based on your history)</li>
<li><tt>$DISPLAY</tt> into <tt>:0.0</tt></li>
<li><tt>`whoami`</tt> into <tt>phil</tt></li>
<li><tt>$((3+5))</tt> into <tt>8</tt></li>
</ul>
<p style="text-align: center">* * * * *</p>
<p>I find these features useful for the same reason I
use <a href="http://psung.blogspot.com/2010/08/gnu-parallel.html">GNU
Parallel</a> the way I do: they really improve the <em>interactivity</em> of
my work— in the sense that I get much faster empirical feedback on whether
I'm doing a complex task correctly (the computer tells <em>me</em> what it
thinks I'm trying to say), rather than me having to reason (unreliably) about
what I wrote and recall a bunch of history and read the man page for the
umpteenth time because I'm second-guessing myself.</p>
<p>Inline expansion is useful in a different scenario as well, namely, once
you expand a glob or a shell alias you're free to (interactively) make
changes to it. For example, if you want to delete all the files but one in a
directory, start with <tt>rm *</tt>, press <tt>C-x *</tt> (which expands the glob into a list of all the files in the current directory), and edit out the
name of the file you want to keep. Or if you want to run one of your
shell-aliased commands, but without one of the flags, expand it
with <tt>C-M-e</tt> and then edit it.</p>
<p>Further reading:
<a href="http://www.ukuug.org/events/linux2003/papers/bash_tips/">A nice <tt>bash</tt> tips slide deck by Simon Myers</a>;
<a href="http://www.gnu.org/software/bash/manual/html_node/Miscellaneous-Commands.html">"Miscellaneous [Readline] Commands"
in the <tt>bash</tt> manual</a></p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com1tag:blogger.com,1999:blog-970055329001593038.post-75171014968396050772010-10-21T08:15:00.000-07:002012-09-24T23:20:50.861-07:00Usability gems in GNU/Linux<p>My post
<a href="http://psung.blogspot.com/2010/10/thank-you-ubuntu.html">Thank you,
Ubuntu</a> attracted some attention (hello, Hacker News!). A couple of
commenters were skeptical of my assertion that GNU/Linux is more usable
than the major proprietary operating systems— and rightly so, since I
didn't really elaborate on it at the time. For example, commenter Dubc wrote:</p>
<blockquote><p><em>I think Ubuntu is a fine choice indeed, but you are going to either have to use another word than usability, or redefine the term to meet your presumptions.</em></p></blockquote>
<p>So I'd like to expand on a few
of the things that make Ubuntu such a pleasure to use, for those who aren't
familiar with its ins and outs, especially because I so rarely see people
take the time to articulate some of these nifty constructs at a high
level.</p>
<p>I will happily grant that at the application level, Windows and Mac OS have
a much more consistent base of polished applications. And Apple has an
attention to detail and psychology that others would do well to learn
from.</p>
<p>It's often said that Apple takes the time to get the little things right.
Unfortunately, at times I think they should have instead spent a bit more
time thinking through the <em>big</em> things in Mac OS. There are some
fairly glaring architectural/design/HCI problems on the Mac (and on Windows,
too) that hamper users needlessly. And because these are issues with how one
interacts with the system at its most basic levels, no amount of polish on
the "little things" can really satisfactorily outweigh them.</p>
<p>What that boils down to is this: yes, you will wrestle with GNU/Linux for a
few hours or days or weeks when you get it set up. But if you use Windows or
Mac OS, you will wrestle with it every minute of every day you're using it.
Forever. And that's just not what "usability" means to me.</p>
<p>Here are some illustrative examples:</p>
<p><b>apt</b></p>
<p>An anonymous commenter wrote:</p>
<blockquote><em>in Mac installing software is a single drag-and-drop
function</em></blockquote>
<p>Oh, if only that were the case. The flow for installing software on
Windows/Mac looks something like this (some steps omitted for brevity):</p>
<ol>
<li>Open a web browser and search for the software you want.</li>
<li>Navigate to the vendor's web page, fish around for the download link, and
start downloading the software. Wait.</li>
<li>Find the file you downloaded and open the installer/archive.</li>
<li>Accept the license agreement.</li>
<li>Drag the item to the dock (Mac only).</li>
<li>Answer some setup questions. Wait.</li>
<li>Delete the installer/archive.</li>
</ol>
<p>These steps are error-prone, and unsophisticated users will quickly find
themselves with malware on their systems. We should help users to do the
right thing by default when installing software, and subjecting them to the
vagaries of the wide-open web isn't the right thing.</p>
<p>On Ubuntu:</p>
<ol>
<li>Open Synaptic and search for the software you want.</li>
<li>Click install. Wait.</li>
</ol>
<p>Whatever the heck you want, you can find it in Synaptic. And there's no
distinction between software that was preinstalled with your system and
software that you choose to install later. It all works the same way and gets
security updates in the same way. And it looks like Mac OS is finally
starting to head in this direction with the App Store for Mac software.</p>
<p>But, what's more, each of the thousands of pieces of software available in
the Ubuntu (or whatever distro you use) repository has an audit trail, and— unlike the apps you get in the Mac App Store, or any "App Store," really—
Ubuntu has the means, motive, and opportunity to do whatever it takes to make
them "just
work," no matter where and when and for whom it was originally written. There
is a vast body of software that is just considered to be <em>part of your
operating system</em> and is maintained alongside it, and Ubuntu will vouch
for that software just as it will vouch for the software at the core of your system.</p>
<p>It's not clear whether you will ever see that sort of large-scale
integration work in a proprietary OS. Package management is really one of the
highlights of the free software ecosystem.</p>
<p><b>Remote access</b></p>
<p>X11 has pretty much decoupled <em>where a program runs</em> and <em>where
the program can be used</em>. You have a few basic composable tools—
SSH, X forwarding, and port forwarding— and all of a sudden the vast
majority of applications can be invoked remotely. So you can, very easily,
interact with a program that's using the resources (CPU, filesystem, network
bandwidth) of a computer sitting somewhere else. There are a few gotchas with
respect to X11 (watch out for high-latency network links, for example), but
by and large it works.</p>
<p>"But wait," you say, "Mac OS has SSH and X11 too." Except that most of the
apps you want to use can't be invoked remotely, because they aren't X11 apps.
So you have a litany of tricks. VNC and NX are appropriate sometimes, but they're
clunky. Mac OS has "Back to My Mac" which is a slightly more general tool
(and costs $99/year). But typically, whenever you work remotely you have to
work differently too. It's
a far cry from the situation on GNU/Linux, where you have a critical mass of
applications you can use remotely and they just work, more or less
transparently.</p>
<p>The result is that Mac and Windows users are typically <em>highly dependent
on a single piece of hardware</em>. If you have a nice light laptop, more
power to you. They say the best computer is the one you always have with
you, after all. But even better than that is not having to carry a computer
around. It's incredibly liberating, not to mention convenient. You can
actually go places without having to pack a bag.</p>
<p>On GNU/Linux, one barely even perceives the <em>concept</em> of "other
computers." Every computer you use, no matter how far away it is, is so
accessible to you, so immediate, that it might as well <em>be</em> the one
sitting on your lap right now.</p>
<p>Of course, Microsoft and Apple don't really have any incentive to make
remote access terribly effective. Since they generally sell you a license to
use a particular instance of their software on a particular computer, it
would cut into their bottom line to make remote access actually useful. So
they won't.</p>
<p><b>Window managers</b></p>
<p>The Windows/Mac window managers (the part of your OS that lets you
manipulate windows and switch between them) are pretty bad. They weren't
always this way— they have become noticeably clunkier on the large
displays that are so common today.</p>
<p>On large displays, maximizing (zooming) windows is not useful much of the
time. Maximized windows are just too big. Instead you have to move and resize
windows to obtain a useful multi-window arrangement on your screen. But
Windows and Mac OS give you very little help in this regard. To produce these
arrangements you usually have to click on some tiny button/target and
manually move windows around.</p>
<p>For example, moving or resizing windows on Windows or Mac OS typically
requires you to point at the titlebar or resize handle or window
border— both of which are just a few pixels wide.</p>
<p><a href="http://en.wikipedia.org/wiki/Fitts's_law">Fitts's Law</a>?
Bueller? <em>Bueller</em>? Having enabled alt-dragging to move or resize on
any capable X window manager (I
use <a href="http://openbox.org/wiki/Main_Page">Openbox</a>, but even the Ubuntu default window manager supports this), the entire
window becomes your drag target. The <em>entire window</em>. (Alternatively,
if you use a tiling window manager, even the whole concept of manually moving
windows around becomes basically moot.)</p>
<p>In fairness to Apple and Microsoft, they have taken some steps in the right
direction recently. Windows 7 "docks" windows to one side of the screen when
you ask it to. You can configure some recent Macs to let you drag windows
with three fingers (or so I've heard). And Mac recently gained Spaces (i.e.
virtual desktops) too.</p>
<p>Unfortunately, I think these are the exceptions that prove the rule. Mac OS
and Windows users have waited for years only to get a fraction of the window
management facilities that you could have set up in your X11 window manager.
Supposing you spent an hour setting up your WM to your liking, I figure you would earn
back that time in improved productivity in just a few weeks, rather than having to wait years for Microsoft or Apple to implement a sensible workflow. But more
importantly, your blood pressure would go down, immediately.</p>
<p>People really are more effective when they can set up their computers to work they
way they want. But on Windows and Mac OS they aren't given the tools they need to do
so. I wouldn't disparage these operating systems so much if they came with
window managers that could at least be configured to be minimally adequate.
But they don't.</p>
<p>The window manager is the one program you're using all the time, even when
you're using other programs. It's your primary vehicle for multitasking and
your command center for managing your work on a second-by-second basis. So it's really important to get this right.</p>
<p><b>Conclusion</b><p>
<p>Having read this post, you might decide that you would brand these issues
not as "usability" issues but as something else. Which is completely fine by
me. A rose by any other name, after all…</p>
<p>Many of the cumbersome rituals that Windows and Mac OS have whittled away at
over the years are completely unnecessary in Ubuntu. And this is why, while
I've had occasion to use a Mac (at various times since 2004 or so) and
Windows (only occasionally), sitting down at one always feels like death by a
thousand cuts. They're superficially simpler, but really quite tiresome once
you get beneath the surface. It seems that hey are optimized
for learnability at the expense of usability. That is is exactly the wrong
optimization to make, if you use a computer for a living.</p>
<p>Life is just too short for that.</p>
<p>Now, it's not like Ubuntu is a paragon of usability out of the box, either, though some of the things I mention above do just work by default.
The difference— the key difference— is that you can make it
into one without much hassle.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.comtag:blogger.com,1999:blog-970055329001593038.post-62074283128034410602010-10-20T19:10:00.000-07:002010-10-20T19:10:00.227-07:00Instant messaging on Android<p>I've been using a Nexus One as my primary phone for the last few months now,
and I quite like it! Upon reflection, the thing I most like about it,
however, is not something I had anticipated at all when I got the phone.</p>
<p>Being able to search the web, read email, run apps, look at maps, and listen
to podcasts are all very convenient. Though, there is not much to say
about those things: I use those tools in more or less the same way I would use them
if I had a laptop with me.</p>
<p>However, being able to send and receive IMs from my phone has been one of a
small number of applications that has led to a qualitative change in the way I
do things. I'm using IM a lot more now, at the expense of pretty much every other form of communication. And that's because <b>instant messaging is the only mode of
communication that actually feels convenient</b>.</p>
<p>You can write and reply whenever you want, not just when it's convenient for
the other person (unlike phone calls). You can have rapid back-and-forth
conversations (unlike e-mail and voice mail). You can write messages of
whatever length you want (unlike SMS). You can communicate unobtrusively,
e.g. in a library (unlike phone calls and voice mail). You can read messages
without going through an absurd interface (unlike voice mail). Your messages
reach you on whatever device you're using— desktop, laptop, or phone
(unlike phone calls and SMS). It doesn't cost an exorbitant amount of money
(unlike SMS). (As for video calling, it has pretty much all the disadvantages and restrictions of voice calls, plus some more. It's a cool tech demo, but not something I expect to use on a day-to-day basis.)</p>
<p>If you think about where and when and how you can use each mode of communication, IM matches or surpasses pretty much everything else on most axes. Sometimes you need the phone or email for a high-bandwidth conversation or to deliver a large payload. But those occasions are getting to be few and far between.</p>
<p>Part of it is just the medium— <em>IM is as synchronous or as asynchronous
as you want it to be</em>— but a good part of the goodness here is thanks to
Android's implementation. You can dictate messages with your
voice; for short common messages (e.g. "call me when you get home") it
works quite well, so you can dash off quick messages using whatever modality is more convenient, keyboard or voice. And the Android notification system notifies you of new
messages and lets you bring up conversations easily— but discreetly, without interrupting whatever else you are doing on the
phone.</p>
<p>For me, IM is the killer app for carrying a smartphone in my pocket, ranking significantly above "browsing the web" and a fair amount above "making phone calls".</p>
<p>(Incidentally, the words "phone" and "smartphone" seem so inadequate after you have come to fathom this super-communications capability you have on your hands.)</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-39247895284858435152010-10-05T23:40:00.000-07:002010-10-05T23:40:22.469-07:00Zeya 0.5<p>I'm pleased to announce the release of <a href="http://web.psung.name/zeya/">Zeya</a> 0.5 (also— Hello, <a href="http://www.reddit.com/r/linux/comments/djcdk/zeya_music_server_with_html5_interface/">Reddit</a>!).</p>
<p style="text-align: center"><img src="http://web.psung.name/zeya/images/zeya5-small.png"></p>
<p>Major changes since Zeya 0.4:</p>
<ul>
<li>Playlist support. The 'rhythmbox' and 'dir' backends detect playlists
you've saved (in Rhythmbox or as M3U/PLS files, respectively) and let you choose from among them (using the dropdown in the upper left corner).</li>
<li>PLS format playlists are now supported by the 'playlist' backend. The format of the playlist (M3U or PLS) is guessed automatically based on the extension. (Amit Saha)</li>
<li>Support for decoding m4a files. (Rainer Hahnekamp)</li>
<li>Bind Zeya to a single interface only with the new <tt>--bind_address</tt>
flag. See the <a href="http://web.psung.name/zeya/cookbook/">cookbook</a>
for more info. (Björn Lohrmann)</li>
<li>More UI.</li>
<li>Bug fixes. Zeya now "just works" under a wider variety of circumstances. (Samson Yeung and others)</li>
</ul>
<p>Known issues:</p>
<ul>
<li>The 'dir' backend doesn't work with Python 2.5. A fix has been checked in at git HEAD.</li>
</ul>
<p>See <a href="http://web.psung.name/zeya/"><tt>http://web.psung.name/zeya/</tt></a> for more information about Zeya, installation, getting started, and reporting bugs.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com9tag:blogger.com,1999:blog-970055329001593038.post-4087337680377306502010-10-01T22:02:00.000-07:002010-10-01T22:02:00.367-07:00Thank you, Ubuntu<p>Ubuntu 10.10, the Maverick Meerkat, will be released in just a couple of
weeks. That got me reflecting on the fact that I have been a happy user of
Ubuntu for what must be over 5 years now. That's a long time!</p>
<p>The GNU/Linux variants are the only OSes I've used where I really have the
flexibility to define my own workflow (<a
href="http://psung.blogspot.com/2009/09/getting-more-flow-from-your-window.html">example</a>).
So they are a pleasure to use (ok, <em>most</em> of the time). I use a computer
for many, many hours a day nearly every day. And the time spent customizing
software and learning it is a drop in the bucket when it's amortized over the
months and years I'm going to spend using it. Sure, Windows and Mac OS are a
bit more learnable and easier to get started with— but they are much
less <em>usable</em>. And for me, and most other people who sit at a computer
for a living, that is precisely the wrong optimization to make.</p>
<p>There's plenty more to love about Ubuntu: for starters, that it runs on
every piece of hardware you throw at it; how with <a
href="http://psung.blogspot.com/2008/06/managing-dotfiles-with-git-continued.html">a
modest amount of effort</a>, you can make all the computers you use behave
exactly the same; and how great <tt>apt</tt> is (really, it takes the fear
and hassle out of installing software, and it's an experience that no
proprietary desktop OS comes close to).</p>
<p>Ubuntu is far from perfect, but it is pretty marvelous, and all the
GNU/Linux operating systems have come a long way in the last 5 years. When I
step back, I'm a bit astonished that Ubuntu or anything
like it even exists at all. It works, it's powerful, it's free of charge,
and, with small carve-outs, all of it is free for <em>anyone</em> to do
<em>anything</em> they wish with it.</p>
<p>One thing I rarely stop thinking about is how technology can be made to be
an instrument of empowerment. And I believe that one necessary step in that
direction is ensuring that you are the master of all these amazing devices
you carry around with you all the time: that they serve you and carry out
your will, and not the other way around. Ubuntu has this vast collection of
software you can use as the substrate for doing <em>anything</em>, and the
question isn't "<em>Will the creators of this software give you permission to
do this?</em>" but rather "<em>Who the hell is going to stop you?</em>".</p>
<p>I find this an incredibly heartening idea, almost a cousin of the concept of
Turing's <em>universal machine</em>— the possibility, realizable in
software, that you are limited by nothing other than your imagination.
Unfettered computation is really a magical thing. And Ubuntu is a wonderful
demonstration of that assertion, though by no means the only one.</p>
<p>So, to everyone that helped to make this possible (Canonical; the Ubuntu
community; Debian developers; kernel developers; upstream maintainers and
contributors of all stripes; and yes, even the folks working on other
downstreams, like RH/Fedora— your code makes its way into Ubuntu
too):</p>
<p>You have truly helped to make something wonderful, and it's a real gift to
humanity. Thank you.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com18tag:blogger.com,1999:blog-970055329001593038.post-10270261984523307532010-09-29T22:49:00.000-07:002010-10-25T11:09:31.443-07:00Guava<p>A public service announcement for Java programmers.</p>
<p>Google has released as Free software a bunch of convenience libraries used
internally in Google Java code. The project is (very cleverly) called
<a href="http://code.google.com/p/guava-libraries/">Guava</a>.</p>
<p>The libraries span a huge range of functionality, but I think that in
general they help to promote a more functional programming style and they
paper over some of Java's warts.</p>
<p>Here are three of my favorite things from Guava, but if you <a
href="http://guava-libraries.googlecode.com/svn/trunk/javadoc/index.html">read
the Javadocs</a> you will undoubtedly find more cool stuff. Looking at all of
this, you might decide that it's all simple stuff that you could implement
yourself in about five minutes. Of course you could (in <em>some cases</em>,
anyway), but why would you?</p>
<p><b>Immutable collections</b></p>
<p>Create an immutable list (as you might recall, every Java array is mutable,
which can be a huge source of pain):</p>
<blockquote><tt>List<String> answers = ImmutableList.of("yes", "no", "maybe");</tt></blockquote>
<p>There's also a <tt>Builder</tt> pattern for more complex constructions, and
analogous classes <tt>Immutable{Map,Multimap,Set,SortedMap,SortedSet}</tt>.
<p><b>Collection factory methods</b></p>
<p>Here's some standard Java code to instantiate a collection:</p>
<blockquote><tt>List<Foo> foos = new ArrayList<Foo>()</tt></blockquote>
<p>Using the <tt>Lists</tt> class, you can rewrite that as the following:</p>
<blockquote><tt>List<Foo> foos = Lists.newArrayList()</tt></blockquote>
<p>Usually factory methods are a way for you to give the callee the flexibility
of returning different classes at runtime. That's not the case here—
<tt>Lists.newArrayList</tt> will always return an <tt>ArrayList</tt>
(unsurprisingly). But Java does type inference for a call of this type, so
you save the hassle of having to repeat yourself by writing the generic type
<tt>Foo</tt> on both sides of the assignment, every time you create a new
collection.</p>
<p>There are analogous classes <tt>Maps</tt> and <tt>Sets</tt>, which also
contain other useful utilities in addition to these factory methods.</p>
<p><b>Splitter</b></p>
<p>Some simple illustrative examples for splitting a string using Guava's <tt>Splitter</tt> class:</p>
<blockquote><tt>Splitter.on(",").split("foo,bar,baz");<br>
// ==> an iterable containing "foo", "bar", "baz"<br>
<br>
Splitter.on(",")<br>
.trimResults()<br>
.omitEmptyStrings()<br>
.split("foo,bar , baz,,,");<br>
// ==> an iterable containing "foo", "bar", "baz"</tt></blockquote>
<p>But wait, doesn't Java already do string splitting? Yes! Yes, it does.
And this is how (emphasis mine):</p>
<blockquote>
<p><b>String.split</b></p>
<p><tt>public String[] split(String regex, int limit)</tt></p>
<p>Splits this string around matches of the given regular expression.</p>
<p>The array returned by this method contains each substring of this string that is terminated by another substring that matches the given expression or is terminated by the end of the string. The substrings in the array are in the order in which they occur in this string. If the expression does not match any part of the input then the resulting array has just one element, namely this string.</p>
<p>The limit parameter controls the number of times the pattern is applied and therefore affects the length of the resulting array. If the limit n is greater than zero then the pattern will be applied at most n - 1 times, the array's length will be no greater than n, and the array's last entry will contain all input beyond the last matched delimiter. If n is non-positive then the pattern will be applied as many times as possible and the array can have any length. If n is zero then the pattern will be applied as many times as possible, the array can have any length, and <b>trailing empty strings will be discarded</b>.</p>
</blockquote>
<p>This is a bizarre and unmemorable edge case in the API. I guarantee that upon finishing this paragraph, you will promptly forget about it until about two hours into debugging a failure caused by it. You can save yourself some grief if you just use <tt>Splitter</tt>. <b>Stay away from <tt>String.split</tt></b>.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-66752189871489750302010-09-16T21:21:00.000-07:002011-04-29T16:49:06.601-07:00Thinkpad TrackPoint Scrolling in Ubuntu Maverick/10.10<p><b>Update:</b> these steps seem to work on Ubuntu Natty/11.04 for me, as well.</p>
<p>My fellow citizens, our long national nightmare of having to use bad pointing devices is over.</p>
<p>Starting with Ubuntu Maverick/10.10, configuring scrolling with the middle button and the TrackPoint is now really easy:</p>
<ul>
<li>Install <tt>gpointing-device-settings</tt> (<tt>sudo aptitude install gpointing-device-settings</tt>)</li>
<li>Run <tt>gpointing-device-settings &</tt>, enable the <b>Use wheel emulation</b> using button <b>2</b>, and enable both <b>vertical</b> and <b>horizontal</b> scrolling.</li>
</ul>
<p style="text-align: center"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNvvwMjnrFWuDb92wd89DuGDstZRZFKaHmIWlYcP9hpMRg0gyiQXGXmuYCpPVFfSVSrQbUIHhtQXz8qJ3-uKOi9cBqwzsrfwhhsRmMZVVXezjI7Nl5D-GLldOGvFovXXwUrz-Le7FlHYA//"></p>
<p>(<tt>gpointing-device-settings</tt> is not new but Maverick is the first time it has really worked well for me.)</p>
<p><b>Update:</b> some people are reporting that g-d-s settings are lost after suspend. If you run into this, you might try <a href="http://psung.blogspot.com/2010/04/thinkpad-trackpoint-scrolling-in-ubuntu.html">these alternative instructions</a>.</p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com46tag:blogger.com,1999:blog-970055329001593038.post-83081575743383592562010-09-14T23:28:00.000-07:002010-09-14T23:28:07.294-07:00Designing for colorblindness<p>Public service announcement: some 8% of men have some form of colorblindness.</p>
<p>A couple of people have pointed me to <a href="http://jfly.iam.u-tokyo.ac.jp/color/index.html">this nice article on colorblind-friendly design</a>, by Masataka Okabe and Kei Ito. Useful reading for anyone who designs for print, presentations, or UIs.</p>
<p>It's a good overview of the underlying theory/physiology of colorblindness and gives many tips for conveying information through channels other than color.</p>
<p>The authors also provide this handy colorblind-friendly color palette:</p>
<p style="text-align: center"><a href="http://jfly.iam.u-tokyo.ac.jp/color/index.html"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTwwGDrcSWFWXWcOcBW3ia99BGjFzy2-FiWYKFwPjQPjToA9i67u48FJkMElP0BPSm9we4GWParYHF4rie3gshoQ8K5mi76MTHf7w6T9H_2urHzqBz0bNVxUrbyYPUeuuEA02XkFjFGdg//" style="border: 0"></a></p>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0tag:blogger.com,1999:blog-970055329001593038.post-55713318462954563852010-09-10T01:28:00.000-07:002010-09-29T22:05:25.949-07:00Backups and rdiff-backup<p>I always thought nearlyfreespeech.net's advice on backups was good advice:</p>
<blockquote><p>You should adopt a backup policy that assumes we are storing crates of sweaty dynamite on top of the servers that hold your important data. (Even though we aren't.) <em>[<a href="http://faq.nearlyfreespeech.net/full/backups#backups">NFSN FAQs</a>]</em></p></blockquote>
<p>If you only have one copy of something, stop what you are doing, obtain a disk, and replicate it.</p>
<p>Here are some brief notes on my backup setup, including some things I've learned since <a href="http://psung.blogspot.com/2008/03/making-backups-instructions-for.html">I last wrote about backups</a>. (I had a disk failure last December and restored everything from backup. No tears, no sweat. In fact the exact same thing seems to happen about once every year, which I suppose is a good testimonial. I'm probably due for a disk failure real soon now.)</p>
<ul>
<li>My first line of defense is backing up to a secondary HDD in my machine. I mostly use <tt><a href="http://rdiff-backup.nongnu.org/">rdiff-backup</a></tt> now (and only <tt>rsync</tt> for huge files, like disk images). This system seems to work well. <tt>rdiff-backup</tt> creates reverse diffs on each backup so you can retrieve old versions. All the diffs go in the <tt>rdiff-backup-data</tt> subdir; if you remove that you just get a plain mirror, like what <tt>rsync</tt> would do.</li>
<li>I wrote a FUSE filesystem, <a href="http://psung.blogspot.com/2010/06/rdiff-snapshot-fs.html">rdiff-snapshot-fs</a>, that displays rdiff's repository format as a series of mirrors in order to make it easier to browse historical snapshots. Doing a restore of individual files from time to time is key to ensuring your system is working when you <em>really</em> need it.</li>
<li>Rather than scheduling backups with <tt>cron</tt> and having to leave my computer on at night or, alternatively, having backups happen while I'm working, I bound a hotkey to a script that backs up and then <a href="http://psung.blogspot.com/2010/09/assorted-notes.html">puts the computer into suspend</a>. I run it when I leave the computer for the day, every day.</li>
<li>I also <tt>rsync</tt> to other backup backup locations, including a portable HDD that stays in a safe place when I'm not using it.</li>
<li>When <em>restoring</em> from a mirror, the <tt><b>-c</b></tt> flag to <tt>rsync</tt> is useful. It makes <tt>rsync</tt> compare the checksum of the data being copied back with the checksum of the original. Then if you have multiple backups of the same stuff you can easily identify and reconcile any differences between them.</li>
<li>I did try <tt><a href="http://rsnapshot.org/">rsnapshot</a></tt>. Unfortunately it caused my system load average to shoot through the roof, making the system unresponsive while backups were being made. I have no idea why this is but a few other people have reported the same thing.</li>
</ul>Philhttp://www.blogger.com/profile/12760478278391942483noreply@blogger.com0