Okay, the .xex files are 8-bit Atari binary load files. That’s the kind of file RetroPie and EmulationStation THINK the Atari800 emulator program needs; put at least one .xex file into the appropriate file folder and “Atari 800” will appear as an option. Fortunately, once you launch the emulator, you can press F1 and navigate through the options and configuration screens and set things up however you like.
I’m running the RetroPie 3.0 Beta but the file paths are probably the same. Put at least one .xex file into /home/pi/RetroPie/roms/atari800 and you should be good to go.
.atr files are Atari disk images. If the .atr file is a self-booting disk (such as most disk-based software) you’ll be able to boot straight into it when you launch the emulator, assuming you’ve got your config files setup to allow it. If not, you can navigate through the Atari800 configuration menu (F1) to insert the virtual disk image into a drive and then reboot the emulated Atari to run whatever is on the disk.
.BAS files are BASIC program files. Again, if you’ve got your Atari800 emulator configured right in the options menu, you can boot the system to a BASIC prompt and load those files, but there are complications. First, you’ll need an Atari BASIC cartridge ROM (ataribas.rom) – once you point your emulator configuration (that F1 menu) to the right BASIC ROM image, you can set your Atari system to an XL or XE machine and it can boot directly to BASIC. If you have the system set to a 400/800 type machine (atariosa.rom or atariosb.rom), you will need to tell the emulator to insert a virtual BASIC cartridge. Most Atari carts out there loose in the wild are in .bin format but you might also find .car files or something else. The Atari800 version in RetroPie works fine with .bin files. So you’ll need a verson of Atari BASIC (Rev A, Rev B or Rev C) in .bin format. Stick that wherever convenient in your file structure. You can point the emulator to the appropriate file directory through the configuration menus.
So insert your virtual BASIC cart through the Atari800 menu screens and reboot your virtual Atari. You’ll get a READY screen and you can type and run BASIC programs. To save files you’ll probably need a virtual disk image, an .atr file. This gets into a HOWTO for Atari DOS, but to run an Atari from a floppy disk (real or virtual) you’ll need an Atari DOS (or something comparable like SpartaDOS, MyDOS, DOS XL …) They’ve all got their pluses and minuses but they’re all available out on the net in .atr format. For our purposes Atari DOS 2.5 is probably your best bet – it’s ubiquitous, compatible and works with most stuff most of the time.
So find a DOS 2.5 .atr file and “insert” it through the menus into your system and then boot – once you get to the READY prompt, you should have DOS loaded and be able to access files on an .atr disk image. If you’ve got an .atr virtual disk that contains the BASIC program you want, use the Atari800 emulator menu screens to “insert” that disk into Drive 2. Then at the READY prompt, you can type LOAD D2:xxxxx.BAS, where xxxx.BAS is the name of the BASIC program you want to load. Atari BASIC actually doesn’t care what you name your files, and neither does Atari DOS, with just a couple of exceptions (DOS.SYS, DUP.SYS, and one or two others). When your BASIC program is loaded, you can type LIST to list it out or RUN to run it.
.cas files are cassette images. I haven’t actually used them with Atari800 on the Pi. On other Windows or OS X emulators you can boot them or save files to them from BASIC and I presume you can do the same here but I just haven’t tried.
Anyway, I hope the above helps a bit. If you have more specific questions once you get into the emulator config screens, let me know.