What do you mean by overclocking "extremely"? Using LN2? Is it just for demonstration purposes to see if it's possible? or for 24/7 use?
First thing to realize is it's not a good idea to overclock a 5800X on any board in a conventional sense as it's effectively pushed to the silicon's limits by the bin-sort AMD uses and the boosting algorithm. People with "extreme" setups might get very slight all-core improvements but it usually results in light threaded performance loss which is most important for gaming. Better all-round performance is usually found by using PBO curve optimizer and a slight VCore negative offset.
2nd...yes, it's technically possible to graft on better VRM circuit but it will not be easy. You can't just change FET's, you have to also change the controller's operating parameters and it's supporting circuitry; that means also a modded BIOS that initializes the VRM controller with the correct parameters. It's far, far, far easier to just buy a better motherboard. B550's have universally better VRM's, but of course some are better than others.
If the FET's are overheating with what you have then locate a fan blowing on them. I've mounted a fan (50-80mm) to a bracket I made from a removed I/O plate before and attached it to the rear case fan then pointed it to blow across the VRM FET's. It worked very well.
But if you're really serious about this you might be interested in some of BuildZoid's youtube vid's. He does this sort of thing with GPU's mainly but also mods some motherboard VRM's.
View: https://www.youtube.com/watch?v=wlm_lMQCRqY