That's the way Ryzen boosting works, I've heard it called a 'rush to idle'.
First, as
@mdd1963 suggests, you may not have the same idea of what 'idle' is. In a modern multi-tasking OS like Windows 10 there many different processes and threads running ( I count over 60 in my TaskManager) with many of them constantly asking the CPU to 'DO SOMETHING'. It may be small, but the sooner the CPU can get it done it can throw the core into a deep-sleep state, essentially turning it off, and conserve energy. Zen2 chips do this as often as once every milllisecond.
Second: the processor scales clock on temperature and will always use the highest clock speed it can muster for the temperature. So when one of those little tasks asks for attention Windows' scheduler will throw the thread on an idle core. If cool enough the Zen boosting algorithm immediately boosts to max rated clock, then pulls back as the core gets warm and usually drops away because most likely it's finished in 40 or 50 milliseconds.
For more sustained workloads lasting way longer than 100 or 200 mS, especially hitting all cores, a 3800X clock should settle back to something like 4.35-4.45Ghz. Maybe more like 4.2-4.35Ghz if it's heavy, with some AVX instructions. Even longer and continuous AVX instructions, or anything that heats all the cores substantially, it will drop as far as the base clock speed, 3.9Ghz. Thats assuming you're on something like stock cooling, better and it might be 3.95-3.975. Worse than stock and bad airflow in your case can result in going below base, so less than 3.9Ghz.