The PSU is probably fine. Remember the motherboard has voltage regulators that stabilize the 12V source anyway.
CPU's are unlikely to "fry" as they have a thermal shut-down diode for worst-case but should auto-throttle down before that anyway if they overheat.
*Plenty of guides, but between 4.5GHz and 4.8GHz seems doable depending on the silicon lottery (your CPU) and your CPU cooler.
You'll want at least a Cryorig H7 Ultimate for overclocking (4.6GHz?), or liquid cooler (i.e. 2x140mm).
**75degC is what Intel recommends for optimal life of the PC (you can go over, but preferably not very often).
You can use CPUID's Hardware monitor for the core temps then adjust overclock based on that (I'd use HANDBRAKE to convert a video as a real-world stress test, not Prime95... make sure all threads are at 100% usage or close for at least ten minutes and aim for 75degC, though if you don't stress the CPU like that much can use 80degC as gaming would be lower).
https://www.cpuid.com/softwares/hwmonitor.html