There is no "fix" because there is nothing wrong. It is by DESIGN that it does this. Ryzen is very boosty on light loads. There are some things you can do to mitigate it so it is less of a problem though and the primary thing you can do is get a better cooler so you don't have to hear it doing it. The stock Ryzen coolers, all of them, suck.
A good 30-50 dollar cooler will make that go away, entirely. I highly recommend the Thermalright True Spirit Direct 140 for all the Ryzen models from 3600-3700x. Above that, 3800x, 3900x, 3950x, it's still a good recommendation, but a beefier heatsink would be preferred.
A few other things you can do are:
1. Go to the AMD website and make sure you have the MOST current chipset drivers for your motherboard chipset family (ie, B450, X570, etc.) installed. Those will be more current than what you find on the motherboard product page in just about every case.
2. Make sure you have the MOST recent stable BIOS version installed. I would not bother with beta versions unless it specifically addresses a problem you know you have. That goes for any board model.
3. Do more than just "mess around with power plans". CHOOSE the AMD balanced power plan and THEN go into the advanced settings for that plan and make sure the minimum processor power management setting is set to somewhere between 5-8% and maximum is 100%. Go into the BIOS and make sure any relevant C states are enabled and that Cool N Quiet (IF present) is enabled.
Also, in the BIOS, enable Core CPPC. Enable CPPC preferred cores. Enable either Advanced C-states or Global C-states, whichever is present. It's the same thing, but named differently depending on board manufacturer.
Also, until you get a better cooler, I'd make sure PBO (Power boost overdrive) is disabled but leave the regular Precision boost (XFR2) profile enabled. You will not lose enough to matter and it will absolutely minimize the tendon stretching boost behavior that's likely due to PBO/PBO2 being set to Auto rather than disabled. Once you have a good, capable cooler, feel free to enable it again if you wish.