Different servers are located in different areas, sometimes different contintents....; naturally, connecting to a Japanese server from New York might have higher ping times (300-400 ms) than connecting to a server in the same city (20-30 ms)...
Additionally, even if a server is in the same area, you don't know the server's connection method or effective throughput... I could run a server using my cell phone as the connection method, but, that would hardly give great ping times. Naturally, going through a VPN can/will also add 200-300 ms)