I found that Mupen64 will do that if you try to force its resolution too high. For example, if I hit ‘x’ when launching a game and try to set it to 1920×1080, I get the small window in the corner, but if I set it to 640×480, it launches full screen.