This might come from insufficient amount of RAM when compiling the N64 emulator mupen64. Unfortunately, this is not checked before compiling mupen64 in the RetroPie-Setup Script. I have added this issue to the issue tracking list of the RetroPie-Setup Script.
You might try to set the GPU memory to 16 MB via sudo raspi-config before running the complete source-based installation. Remember to set back this setting to 256 MB before running EmulationStation!