You can try to play around with frameskip settings. You can find all config files under /opt/retropie/configs/n64. mupen64plus.cfg has rice settings. gles2n64.conf has gles2n64 settings. If you have pal games use a display resolution with 50Hz (press x or m at game startup).
