The Genesis emulator used in retropie right now is not part of retroarch. So changing stuff in the retroarch.cfg won’t change anything in that specific emulator. In your config/all/ folder lies a second config file next to your retroarch.cfg. It’s called dgen.* (don’t remeber the exact name). That’s the file you’re looking for. When you open the file you will see that it already has configured bindings for your gamepad. To change your select & start button on the gamepad just switch the values of the two entries:
joy_pad1_mode = joystick0-button9
“joy_pad1_start = joystick0-button8
I don’t now if it is even possible within DGEN (the Genesis emulator) to emulate the behavior of the hotkey button thingy in retroarch, but you can add a custom exit button to that config.
Hope I could help you & good luck changing the config.