Does your riser have external 12V power? If not, it could be that your RX580 draws too much power from the PCIe slot to use with a slot-powered riser.
You could also go into BIOS by plugging the GPU directly into the motherboard to drop PCIe x16 to 2.0 speed. If that works, then your cable might be too borderline for 3.0 at least for your hardware combination..