make sure that you have configured hotkeys in your retroarch.cfg file
for example, lets say you are running a snes emulator, go to your config file for your xbox controller. probably located in:
/opt/retropie/emulators/retroarch/configs/<Some_Xbox_Controller_Config.cfg>
edit it and make sure you include:
input_enable_hotkey_btn = "6"
input_exit_emulator_btn = "7"those are my button binds, substitute them for the buttons on your xbox controller, which you can find out using jstest. then pressing select and start should exit the emulator.