It’s a bit confusing as there are multiple places to configure N64 controls for RetroPie, however I discovered that the correct place is in the N64 directory under /opt/retropie/configs/n64

The configuration for both N64 emulators is here. I use mupen64plus, not the RetroArch version, and the configuration for that is mupen64plus.cfg. There is another file in the same directory, which allows you to map controls based on what pad is recognised by the Pi which needs playing with too. I forgot what it’s called, but I think it’s something like “autoconf.ini”.