I would definitely not invest in 32 GB of RAM. A GTX 1080 is too powerful for these parts; you'll be hard-pressed to get any benefit past a 1060 and in truth, that's already a bit of a bottleneck.
If you're not upgrading the platform, the best you can wring out of these parts is probably an RX 580 and a new power supply; this can be done for about $250. You can get a little bit back selling the GTX 1050 used. I wouldn't spend any more than that and if you have more money, it should be saved up towards getting onto a more modern platform. Also, do not try overclocking; the motherboard is marginal as it is for a 125W CPU and if you start throwing in more voltage, there's a good chance at destroying the motherboard.
This will get you into being able to play most games at 1080p at some setting. You're likely to run into issues in AAA games of an open world variety, but you'll get them at least playable, generally.