Could be a power issue: do you have a power supply that is at least 1 amp? I use a 5v 2amp power supply and it seems to work fine, if it is a power issue you could look into getting a powered USB hub.
It also could be a faulty controller.
Or it could be wonky config files.
Does it do it for all games and emulators as well as in emulation station or is it just GBA?
