Many classes of server as well, truly depends on the hardware in question.
Most servers are designed to be efficient and process/handle many jobs at once. A game is basically one application running as fast as possible. There are certainly purpose built servers out there designed for speed and not bandwidth, but they are usually the more expensive variety.
Most servers generally don't offer much in the way of display outputs, not necessary. So adding a GPU to some servers is not possible. Either they are too small, don't have enough power, or don't have the appropriate connectors for standard PCIe cards. Power supplies is a whole other issue, many are proprietary.
Tower servers are basically just PCs, so they tend to work okay as gaming boxes.
These days though, unless you can get a really cheap server, low end brand new hardware is better. A cheap i3, a motherboard, some ram, a cheap chassis, a decent power supply, a storage drive. Just add GPU.
Saw this the other day, still considering it myself (no idea what I would do with it)
$240
i7-6700, 8GB, 512GB SSD
https://www.newegg.com/p/1VK-0001-5NER9
Proprietary power supply, but a standard sized supply will fit in the case. And there are adapters you can buy on ebay/amazon or from moddiy.com.