Speedtest is actually a file download and a file upload they just don't call it that. Mostly it is the server but it can also be where the server is located.
On speedtest try picking other cities maybe one in the city the server is located if you have anyway to find that out. You will see you get huge difference in the speedtest results. These speedtest sites are actually different servers in different data centers many times on different ISP.
There is really little you can do about this. The contract you have with your ISP related to speed is between your house and their network. They make no promise about speeds off their network.......they really make no real promise on their network with the "UP TO" wording. They will...