Apparently, locking your framerate to 30fps solves it. It's a well known issue and they haven't released an official fix for it.
My guess would be it's an issue overlooked when porting it from the consoles.
To clarify, it's only when you're walking on the horse right? I have the same issue though I always gallop or walk fast (I don't know horse terminology, k?) to get rid of it.