Hi,
Kodi isn’t really designed to accept controller input. That’s what’s causing your issue.
There’s a few different “remote controls” out there that you can set up via tablet/smartphone and the people who make Kodi actually make one of their own.
Kore
It’s pretty easy to set up and it works well. You’ll need a keyboard, or at the very least, a mouse for the initial set up inside Kodi.