it could be the gpu is calling for too much power, check the stability of the card in another pc, and then another gpu in there. then check the cpu and the seating of the heatsink itself. make sure your fans are all running, and it's not next to a warm area in the room (add a fan 10" etc near it). Then do this just for a quick test, look in the bios and check all your gpu and pci-e settings and study what they mean. update your bios. actually update it. then if it still acts weird and the power supply is good (testers are about $15 online) and the cables aren't damaged for the gpu or ps, and your bios is updated and the settings are good (do not overclock your pc in this situation)...
try a lower pci-e slot, it will still go quite fast, and see also if the gpu has a <Mod Edit> ton of dust and hair in the fan/heatsink as well.