Yes, it is a sport.

Most people think riding is boring, needs little skill and they think it's not a sport. I don't think they can make that argument until they've climbed onto a horses back and tried to jump solid obsicles, raced through an open field trying to keep a horse under control or simple succesfully walk/trot/cantered in an arena. That's alot easier said then done. You need to be balanced in order to keep up with the horses movement and after only a few lessons, I've heard begginers complain about sore muscles. It also takes  I'd say riding is a sport that takes alot of time and practice to develop the skills, muscle and balance it requires.