I'm in a similar boat. I have a FX-6350 and a RX 570 so we're not that far off and I also want to upgrade soon. I recently overclocked my CPU from 3.9Ghz to 4.4Ghz, with thermals being the limiting factor. I've noticed quite a boost in performance from it. The thermals can get pretty hot at high speeds which would damage the CPU in the long run, but if you're upgrading soon it shouldn't matter.
In short, I'd suggest doing it. The FX processors aren't the best so if you're willing to trade of that extra bit of CPU lifespan for those precious frames it's a good idea.
As I say, I'm new to this too but I would recommend it completely.