Sorry if this has been suggested.
Lock your GPU core my friend. If you say the GPU core was dipping it may not be boosting correctly which happens on newer cards. Google your GPU specs, it should give you the boost ghz, example my 3070 boosts between 1700-1850ghz. download MSI after burner...