the only n64 emulator that works on linux and rpi is mupen64plus. This is not a retroarch core, thus it uses its own config files.
To configure your controler search for the InputAutoCfg.ini file in the /home/pi/RetroPie/emulatores/mupen64-plusrpi folder
Sorry cant help you any further
