Bill has covered this very well.
One more thing to consider, is that when you ping it uses the "ICMP" protocol. Games will use UDP, and maybe some TCP. ICMP type traffic may have traffic priority (Quality of Service) by your ISP, but the TCP/UDP ports (what your computer uses to talk to the games servers) may not have Quality of Service priority.
In a nutshell, your pings (ICMP) may show low latency when communicating with the games servers, even under congestion conditions, but your Games TCP/UDP ports may have high latency when communicating with said Games servers.
It's complicated, and hard top explain, but I hope you get the jist of what I'm trying to say! Confirming this is the case, and implementing a workaround isn't so easy. If your router supports QoS, then I recommend you set it up for gaming. However, QoS all the way through to the Games servers is only something your ISP can assist with, and likely not be able to help - unless there is a package available from your ISP suited for games players? I would call your ISP in any case, explaining your dilemma.
I would definitely investigate what Bill mentioned, as well as Google info on the games servers (which may be having issues) and consider that your opponents may be having the issues highlighted in this thread, and not you...