As far as I understood it I thought pifba didn’t support more than 2 players (I tried to get it to work the same way you did and had no luck) supposedly this dude added 4 player support in his version of pifba:

http://www.raspberrypi.org/forums/viewtopic.php?f=78&t=91731