You can use the hotkey button only as input_player_1. There is a difference between your files. Only Joypad 1 (PSX) or Joypad 3 can open RGUI.
Your retroarchpsx.txt
input_player1_joypad_index = 1
input_player2_joypad_index = 0
input_player3_joypad_index = 3
input_player4_joypad_index = 2
retroarch1.txt
input_player1_joypad_index = 3
input_player2_joypad_index = 2
input_player3_joypad_index = 1
input_player4_joypad_index = 0