Ok, so with only the Radeon VII installed in PCIEX16_1 (the GTX 1080Ti is NOT in the system at all), and both displays are connected to it, you get dual-display working.
With only the GTX 1080Ti installed in PCIEX16_1 (the Radeon VII is NOT in the system at all) and both displays are connected to it, you get dual-display working.
With Both cards installed; the Radeon VII installed in PCIEX16_1 and the GTX 1080Ti installed in PCIEX16_5 and one display connected to each card, you get dual-display working.
It's only with both cards installed; the Radeon VII installed in PCIEX16_1 and the GTX 1080Ti installed in PCIEX16_5 and both displays connected to the Radeon VII that you're having issues with dual-displays.
Is the above assessment correct?
-Wolf sends
Edit:
Not sure why slot 3 is the recommended slot when it's also shared with the m.2 port.
It's a trade off. You can use one feature or the other. You cannot use both.