Run a traceroute to the server from a cmd prompt. You need to see if you can determine where the largest increase is occurring. Your numbers are fairly small so you may have to run it many times to get past minor variations dues to testing conditions.
The bad news is you can really only do something about the first couple of hops in the trace. The first one should be well less than 5m most times its 1ms. This is to your router in the house. Obviously if you use wireless this will be higher. The next hop represent the connection between your house and the ISP. This should also be fairly low but the actual values are affected by a number of things that only the ISP can control.
Anything further in almost always represents distance. You can't do much about this, the signals only travel so fast so the farther it is away the longer it will take. Problem is with the internet data does not always take the shortest path. Lets say you are on ISP A and the server is on ISP B but you just happen to know that the server is in the same city as you. You would think it would be very fast since it should just cross between at some common location in your city. Problem is ISP A and ISP B may have agreed that they will not use the cross point in your city as their primary connection, they may have agree to use one in a city many 100s of miles away. So now your traffic has to go all the way to the other city and back to in effect get across the street.
You can do little to affect the latency in the internet. The only solution generally is to use the same ISP as the server provider is using (they use more than one at the same time actually). You would then always stay on a single ISP network which generally give optimum pathing.