Actually, I was playing around with this more today and it’s more sporadic than I thought. Sometimes it works, but mostly it doesn’t. I noticed some USB errors in /var/log/messages and did some googling. It looks like this might be a kernel bug, fixed in 3.3 (see https://lkml.org/lkml/2012/5/27/86). Lot’s of people with the same issue.
I have no idea when Raspbian or other debian-based distros will be moving to 3.3, but I guess my only real option is to wait and see. I don’t particularly want to attempt compiling my own kernel.