The game servers should have no be the cause of the issue if you are seeing the problem with the ping command and you are ping your local router IP. This is why you use real ping commands and control what you ping. You have eliminated the internet and the game code as the problem.
The so called ping times in games many times are not actual ping commands. You can see similar number for other traffic in the network activity tab. You will see latency numbers for open sessions. These numbers better represent the delay but they also can be greatly affected by the server and web application code so they are not good for troubleshooting.