I just wanted to update this post for anyone else interested in vector resolution. Thanks to help from Buzz on this post I was eventually able to specify my display size for running vector games at full resolution. Buzz actually updated the advancemame code for both 1.4 and 0.94 to look for a new setting in the .rc config file instead of forcing a 640×480 default.
To enable high-res vector, first, backup your old config files. Then, you need to have the latest RetroPie script, so update that. At the time of this writing, the latest RetroPie is 3.6, but this change came AFTER the 3.6 image, so you still need to update the script. Then, use the retropie menus to download AVANCEMAME from binary source. This will install/overwrite the version of 1.4 and 0.94 with one that has the change. Finally, add these lines to the .rc file if not already there:
display_width 1280 display_height 1024
Obviously, you should use your own display’s native resolution. This was mine. These config entries only affect the resolution used for vector games. In my case, the games kept their proper aspect ratio since my monitor is 5:4 not 4:3.
With this setup, I now get crisp vector lines at the highest resolution. It looks fantastic! I usually bump the display_intensity to 1.75 or so (3 is the highest), add a flicker of 15 and enable translucency (default) all through the .rc file. These settings only affect vector games and the result is a very nice approximation of a vector display.