The issue is that the Atari800 emulator is looking for roms having the extension of XEX. I have been able to run 5200 games using the Atari800 fine.
Edit the file /etc/emulationstation/es_system.cfg
Find Atari 800 and add the BIN extension as shown below.
<extension>.xex .XEX .bin .BIN</extension>
While in here, I added addition switches to the command line:
atari800 -5200 -bpp 8 -fs-width 800 -fs-height 600 -ntsc -fullscreen -cart-type 4 -cart %ROM%
On my Raspberry B+, I found 800×600 to give the best performance. I also start the emulator in 5200 mode.
You should now see the Atari800 emulator in Emulation Station.
Run a game. It will not work but it will create the file .atari800.cfg in /home/pi
Edit this file and tell it where you put the system roms. (example: ROM_5200=/usr/local/share/atari800/atari5200.bin)