I posted an article on reading your mail with VM in Emacs, which contains a minimal VM orientation, information about how I read mail in VM, and the elisp configuration I used to set that up.
I attempted the switch a couple of months back after realizing that the volume of email I was dealing with was seriously impairing my work ability. Gmail does have some tools for managing lots of email (labels), but with VM I can be much more fine-grained about what messages I look at at any particular time. Scanning and reading mail is also much faster, if only because I don't have to wait for the RTT to a Google server and the browser render time every time I want to do something.
And, of course, the best part of VM is that it comes with a great e-mail editor.
Why VM? The big choices are Gnus, VM, and MH-E. I had tried Gnus and couldn't really wrap my head around its model of doing things. MH-E sounded acceptable but I assumed that VM, which had more Lisp implementation, would probably be more flexible. I am pretty happy with VM, but I suspect that changing mail clients is really just a matter of performing a conversion process on the mail folder files. Some operations (such as writing an entire mailbox to disk) are probably slower than they could be, but in general it is blazing fast.
I also wrote about setting up BBDB (the Insidious Big Brother Database), an address book program which is strangely satisfying to use.
Read about my VM configuration and workflow.