I am also playing around with the N64 emulators, basically it seems like a pain.
You should check out the RetroPie 3 Beta 2 image, as there is supposedly some improvement for N64.
Also – Check out this video: https://www.youtube.com/watch?v=lh0n5PWN2lI (towards the end)
The guy posts his config, and it seems to work well for him.
One more thing – there is a forum member on here ‘Floob’ he is very good with configuring the RetroPie image. Maybe see if you can reach out to him and have him visit this page and help us out.