My Tankstick trackball works great in AdvMAME so long as no other mouse is plugged into the Pi. I haven’t played Golden Tee but I use it in the Atari classics Centipede, Millipede and Missile Command without any problems, as well as with Tempest (I don’t have a true spinner). Check to make sure you’ve got the analog control sensitivity in MAME set high enough. You will probably have to experiment to get the feel right.

And if that doesn’t do it, check the Tankstick with a regular PC or Mac. The trackball will show up as a USB mouse and should work just like one. If it doesn’t work with a regular computer, something may be up with your trackball mechanically or electrically.