I understand that you may have only purchased that PSU recently, but it's an old platform, from around 2011. If it is "new" then it is likely "new old stock" meaning it sat on a shelf for the last 8-9 years and was then sold to you. That means the capacitors inside it have aged while it was sitting. It does not need to be run in order for that to happen. So, the fact that you only recently bought it does have a lot to do with it being "old" or not. Also, as I said, it's not a very good unit to begin with.
As you say though, all of the hardware is rather old, including the motherboard and graphics card, and if the PSU is not the problem then I'd suspect the motherboard or graphics card are. Likely, the motherboard. Age is the worst enemy of motherboards, and of all components aside from power supplies that fail due to age, motherboards seem to be the most affected by aging.
Memory and CPUs rarely fail, unless there is a good reason for it such as abuse or long term overclocking, so those are unlikely.
It certainly COULD be the graphics card though, but the only way you're going to really determine that is by trying a different graphics card in the system to see if it still happens. Or, remove the graphics card completely from the motherboard, then plug the display cable into the motherboard output to use the integrated graphics and see if it still happens.