According to various information found in a Google search, there's a "Turbo Core" option for the AMD FX 6300, which enables it to boost to 4.1 GHz - this is
perfectly normal.
So from what you have posted above, if Turbo Core is enabled in BIOS, your post actually describes exactly the way an AMD FX6300 is supposed to run, at least according to the information I'm aware of.
Some mentioned, that depending on what Windows Power Plan you use, the minimum boost clock can be different (depending on minimum processor state and maximum processor state settings apllied in the specific power plan)
But all the clocks you mentioned, from 3500MHz to 3800MHz to 4100Mhz, is actually what you should expect to see when your PC is in use, with the CPU you have.
Balanced Windows Power Plan should be fine for an AMD FX 6300, as far as I can understand from other posts online.
If your CPU would ever reach a critically high temperature, at first it would down-clock (thermal throttle) to lower the temperature, if that is not enough, the computer would completely shut down to protect the CPU.
So your CPU won't just suddenly die due to overheating - a process with various problems would come ahead of a possible final catastrophic failure (which is not that common - it would likely become unstable more than a complete failure).
Your computer's performance would clearly indicate if your CPU is generally running too hot. The computer would become unstable, and likely crash randomly or become unusually slow due to thermal throttling.
You haven't mentioned anything that makes me suspect your CPU isn't sufficiently cooled, so to be honest, the way you describe your computer in your posts above, you should relax and be satisfied that your CPU actually behaves and performs as it is intended to do by default design
I personally can't identify any abnormal issues or critical problems by the information you have provided so far - you should just continue to enjoy your computer
EDIT :
Link to a review describing the 4.1 GHz Turbo Core max frequency