i have the same problem.
I have a nvidia gtx 1060 and a rx 580 nitro +, i'm using afterburner 4.3.0 and 4.4.0 beta 12 with catalyst 17.5.2 and 17.7.2, and last nvidia driver.
in afterburner i can only oc the gtx 1060 but can modifi mem and core clock on rx 580.
if i remove gtx 1060 i can oc rx...