I just did a install as you describe and am using the same controller. I just got this all working yesterday but started on Tuesday so I understand your woes. I did not run into the problem like you are experiencing but I’d like to try to help. I think your problem may lie with the USB bus activity itself.
A question that needs to be asked is please explain any and all USB peripherals that you have connected to the PI device. Hub (Powered or not), controller, keyboard, etc.
and run this command and report back the results.
Maybe there is a conflict with another USB device on your bus or for some reason the bus is having power issues. If you are using a powered USB hub, check to make sure it is one that is certified for use with Raspberry Pi. Some feed power back into the bus and can cause some strange things to occur.