more processors means more threads. Not more clock speed and not more IPC, both of which are typically more important to games. Also, server boards don't have the same overclocking capabilities as gaming boards even cheap gaming boards. You might also need to use nonstandard ram types(FB, ECC, etc) and servers tend to be pickier over ram, doesn't like and mixing/matching quite as much. So you'd need to check compatibility list for certain before buying ram for it, and also for CPU's because they are just pickier in general.
But otherwise they work fine, can be a little slower in some cases. A gaming board is basically a drop safeguards and stability and let me push my hardware to the performance max, and server boards more reliability...