Can you putty into your Pi or choose the option (when pressing start) to quit emulation station.
From there on you will be in your home directory (pi)
do “cd RetroPie” and then sudo ./retropie_setup
A menu should pop up. Update your script (always a good idea!) and then go to (configuration to be used post install, sorry I do not have my Pi booted at the moment) and from there on you can choose an option to pair the PS3 controller.
Hope this helps!