Question Why the system changes the order of GPU when booting in UEFI mode

[dustar1986]

Hi,
I'm not sure this is due to MB UEFI BIOS or Windows 10 UEFI regulation, I have an weird/interesting finding:
I have a ASUS Prime X299 Deluxe II with ASUS GT1030 in slot1 and Nvidia Titan RTX in slot2. My only monitor is on GT1030.
When I first installed Windows 10, I didn't realize the system booted with CSM enabled and the PCIE boot option is legacy only. In this way, GPU0 is the GT1030 and GPU1 is the RTX as expected, according to nvidia-smi.
Then I disabled CSM to boot in UEFI, and then GPU0 became the RTX and GPU1 was the GT1030.
I have no clue why this happened but want to understand the mechanism. Any idea?

 

[Colif]

The real test would be if they swapped again if you formatted drive as MBR again. I wonder if the order is just assigned at setup, and there is no logic to the order.

Its likely controlled by bios as it is where Windows gets the info from. but i also read that the card order is forced by Nvidia drivers, it sets card 1 as primary... which doesn't make any sense? as that isn't what you are seeing

This might help, it is asking how to change Primary GPU, it shows where in bios it is likely to be: https://rog.asus.com/forum/showthre...imary-gpu-display-in-sli-to-second-card/page2

This is your manual: https://dlcdnets.asus.com/pub/ASUS/...DELUXE/E13088_PRIME_X299-DELUXE_UM_V2_WEB.pdf the feature one guy mentions (PEG) is on page 3-16

 

[dustar1986]

This might help, it is asking how to change Primary GPU, it shows where in bios it is likely to be: https://rog.asus.com/forum/showthre...imary-gpu-display-in-sli-to-second-card/page2

Thanks Colif. The problem is not about primary display (I also read that link when I searched for my problem online and seems he didn't solve his at the end). What I'm curious about is how GPU is indexed. It's important for me because in my script I assign workload to the desired GPU.

My problem is partially solved by reading Siyuan Liu's blog: https://shawnliu.me/post/nvidia-gpu-id-enumeration-in-linux/. In short, CUDA's default is to indexed GPU by GPU speed. That's why my RTX card is index as 0 even when it sits in slot2, with GT1030 in slot1.

However, I'm still not sure why this speed-based index seems not work when I boot in legacy mode. After legacy boot, the cards seems being indexed by PCI_BUS_ID.

 

[Colif]

I think this would be a question for the Nvidia Forums. I expect someone there would know the answer you are after, sorry I don't really know

https://forums.geforce.com/

 

[dustar1986]

I think this would be a question for the Nvidia Forums. I expect someone there would know the answer you are after, sorry I don't really know

https://forums.geforce.com/

Yes, the answer from Nvidia is: "That's a function of your motherboard, not anything to do with NVIDIA GPUs. enumeration order is determined by your system BIOS and/or the OS, not anything do with NVIDIA"

After digging a bit into GPU enumeration, my interpretation is:

1. Legacy mode: GT1030 has VBIOS compatible to BIOS boot but RTX doesn't. I found this as the only monitor won't light up if it's connected to RTX during legacy BIOS boot. So in this situation, only GT is initiate/enumerate to OS. It takes the index position 0. After OS is loaded, OS or Nvidia driver detected RTX and added it to index 1.
2. UEFI mode: Both cards are compatible for UEFI. There might be a pre-defined order for Driver Execution Environment (DXE) dispatcher, which prefers RTX above GT card. Thus RTX is indexed first as 0 followed by GT as 1. Then OS is loaded afterwards.

I can be wrong.