I'd say chances are VERY good that you are using the stock cooler and THAT is why you are hearing and experiencing this. My advice to you, and to ANYBODY using a stock Ryzen cooler, would be to replace it with a more capable aftermarket cooler of some kind. Ignore reviews and opinions saying the stock Ryzen coolers are "good enough" because they are not, and they cause this exact quick up down ramping of the fans and a lack of some performance that would be present with a better cooler due to the behavioral boost profiles. The closer you get to the thermal spec, the more obvious the problems are, but they exist even at overall seemingly "decent" temperatures and it's more than just core temps that affect it. VRM and chipset temperatures have a direct relationship with boost behavior on Ryzen as well.
I would highly recommend at least a single finstack 140mm cooler like the Thermalright True spirit 140 direct, Noctua NH-U14S, Cryorig H5, or another similar or even higher tiered option.
Also, if you don't have a case cooling configuration that consists of at least a rear and top rear exhaust fan and at least one, preferably two front intake fans, that would be a really wise move as well. Cooling has a tremendous impact on performance when it comes to all flavors of Ryzen product stacks.