[SOLVED] Motherboard for 2x RTX 3090

[fre-deric]

Hi,
I have two RTX 3090 (PALiT GeForce RTX3090 GamingProOC) for computer vision tasks. It is quite frustrating for me to choose the remaining components, because I am not hardware geek. Could you recommend me suitable motherboard and CPU for both RTX 3090?

I could read that rtx 3090 is a 3-slot GPU (PALiT version: 294 x 112 x 60 mm) and they need some free space between them for right cooling. I do not need NVLINK. All I need is a motherboard with PCI-E 16x slots that have gap between them.

Thank you!

 

[Eximo]

I suppose it depends on how much CPU power you'll need.

AMD threadripper will get you all the PCIe lanes and CPU cores you could need. Intel X299 platform basically gets you less CPU cores, but if you wanted to get high single CPU core performance that would be the way to go.

You are going to want a massive chassis with great airflow to handle two 3090s, also the largest power supply available. Think 1600W.

 

[Eximo]

Intel, increase CPU size as needed.

PCPartPicker Part List

CPU: Intel Core i9-9900X 3.5 GHz 10-Core Processor ($520.97 @ Amazon)
CPU Cooler: Noctua NH-D15 82.5 CFM CPU Cooler ($89.95 @ Amazon)
Motherboard: Gigabyte X299 UD4 Pro ATX LGA2066 Motherboard ($350.99 @ Amazon)
Memory: Corsair Vengeance LPX 64 GB (4 x 16 GB) DDR4-3200 CL16 Memory ($362.99 @ Newegg)
Video Card: Palit GeForce RTX 3090 24 GB GamingPro OC Video Card (2-Way SLI)
Video Card: Palit GeForce RTX 3090 24 GB GamingPro OC Video Card (2-Way SLI)
Case: Phanteks Enthoo Pro 2 ATX Full Tower Case ($123.99 @ Newegg)
Power Supply: Corsair HXi 1200 W 80+ Platinum Certified Fully Modular ATX Power Supply ($309.99 @ Corsair)
Total: $1758.88
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2021-03-08 09:24 EST-0500

AMD, cheaper motherboard, but options are limited, so the decent prosumer boards are quite pricey.

PCPartPicker Part List

CPU: AMD Threadripper 2950X 3.5 GHz 16-Core Processor ($639.99 @ B&H)
CPU Cooler: be quiet! Dark Rock Pro TR4 59.5 CFM CPU Cooler ($83.90 @ Amazon)
Motherboard: ASRock X399 Phantom Gaming 6 ATX sTR4 Motherboard ($249.99 @ Newegg)
Memory: Corsair Vengeance LPX 64 GB (4 x 16 GB) DDR4-3200 CL16 Memory ($362.99 @ Newegg)
Video Card: Palit GeForce RTX 3090 24 GB GamingPro OC Video Card (2-Way SLI)
Video Card: Palit GeForce RTX 3090 24 GB GamingPro OC Video Card (2-Way SLI)
Case: Phanteks Enthoo Pro 2 ATX Full Tower Case ($123.99 @ Newegg)
Power Supply: Corsair HXi 1200 W 80+ Platinum Certified Fully Modular ATX Power Supply ($309.99 @ Corsair)
Total: $1770.85
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2021-03-08 09:27 EST-0500

 

[fre-deric]

I think I do not need such powerful CPU. In blog by Tim Dettmers about hardware for deep learning, he writes:
“I recommend a minimum of 4 threads per GPU — that is usually two cores per GPU”.
So it would mean that for example AMD RYZEN 5 2600, 6 cores, 12 threads, 3,4GHz could be enough for both rtx 3090. Or do I really need such powerful CPU such as AMD Threadripper 2950X? Is there anything cheaper?

As far as PSU, I thought 1200-1250W could be enough. Should I consider to buy more like 1600W?

Thank you!

 

[Eximo]

I should point out that the AMD system supports PCIe 4.0, the Intel system does not. Probably won't make a huge difference to what you are doing.

Both systems above are limited to PCIe 3.0.

1200W is borderline if the cards and CPU get pushed at the same time.

Igor's lab shows peaks as high as 573W from a single 3090, however these would be brief. Of more concern are the 400W peaks that happen long enough to potentially trigger a PSU shutdown, that would 800W out of 1200W, and the CPU can easily draw 200W, that leaves only 200W for everything else, fans, drives, peripherals.

NVIDIA GeForce RTX 3090 Founders Edition Review: Between Value and Decadence - When Price is Not Everything | Page 16 | igor´sLAB

With the GeForce RTX 3090, NVIDIA is rounding out its graphics card portfolio at the top end today, for now. Much more is not possible with the GA102-300 anyway and so one may see the current…

www.igorslab.de

I recommend shopping around for a larger PSU with dual 3090.

 

[Eximo]

I think I do not need such powerful CPU. In blog by Tim Dettmers about hardware for deep learning, he writes:
“I recommend a minimum of 4 threads per GPU — that is usually two cores per GPU”.
So it would mean that for example AMD RYZEN 5 2600, 6 cores, 12 threads, 3,4GHz could be enough for both rtx 3090. Or do I really need such powerful CPU such as AMD Threadripper 2950X? Is there anything cheaper?

As far as PSU, I thought 1200-1250W could be enough. Should I consider to buy more like 1600W?

Thank you!

So those chips will be in consumer boards, though you would still need a high end one, but they will split x16 lanes so that each only ends up with 8x. The older 2600 would only be PCIe 3.0, cut in half.

If you want to go down that route, I would suggest a late model AMD chip that supports PCIe 4.0. I will put together another build. Though for what you are doing, I think you will be more pleased with the workstation boards and CPUs.

Those larger boards also offer you the option of putting the GPUs quite far apart (in a full tower). Most consumer boards will have three slot spacing only, so the GPUs will be pretty close to each other. The other slots will be through the chipset, whereas the larger CPUs still have lanes going to many slots.

 

[Eximo]

PCPartPicker Part List

CPU: AMD Ryzen 7 3700X 3.6 GHz 8-Core Processor ($329.99 @ B&H)
CPU Cooler: Noctua NH-U12S 55 CFM CPU Cooler ($59.95 @ Amazon)
Motherboard: Asus PRIME X570-PRO ATX AM4 Motherboard ($268.02 @ Amazon)
Memory: Kingston HyperX Fury 32 GB (2 x 16 GB) DDR4-3600 CL18 Memory ($158.99 @ Walmart)
Video Card: Palit GeForce RTX 3090 24 GB GamingPro OC Video Card (2-Way SLI)
Video Card: Palit GeForce RTX 3090 24 GB GamingPro OC Video Card (2-Way SLI)
Case: Phanteks Enthoo Pro 2 ATX Full Tower Case ($123.99 @ Newegg)
Power Supply: Corsair HXi 1200 W 80+ Platinum Certified Fully Modular ATX Power Supply ($309.99 @ Corsair)
Total: $1250.93
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2021-03-08 09:49 EST-0500

 

[fre-deric]

Thank you!

As far as the motherboard:

Motherboard: Asus PRIME X570-PRO ATX AM4 Motherboard ($268.02 @ Amazon)

it means, that the first GPU will go to the first slot and the second GPU will go to the third slot and the second slot will be empty. Is it ok?
Stupid question: Why the third slot is not white as the first one a and second one? (good image of the motherboard is here )

 

[g-unit1111]

Thank you!

As far as the motherboard:

it means, that the first GPU will go to the first slot and the second GPU will go to the third slot and the second slot will be empty. Is it ok?
Stupid question: Why the third slot is not white as the first one a and second one? (good image of the motherboard is here )

I don't have an answer for that. But personally I wouldn't move the GPU to the third slot - keep in mind that these cards are very large and moving the second one to the bottom, unless you're building in an XL-ATX case would restrict airflow. I think this is one of the few times where an XL-ATX build would probably be better suited for this job - especially getting a larger motherboard like the MSI Meg Ace. I didn't realize how large the 3XXX cards were until I unboxed mine and installed it in my case.

 

[Eximo]

Correction, the threadripper above only supports PCIe 3.0, you would have to bump up to the 3000 series to get 4.0, so that is basically like another $1000...

On the X570 motherboards:

The two reinforced slots are intended for graphics cards. They are connected directly to the CPU.

In this case you are still going to be running at 8x mode, but at least at PCIe 4.0. which is twice as fast as PCIe 3.0. So there will be minimal impact to GPU performance.

If you use the third slot, it will be connected to the chipset, and to the CPU through it. This is still PCIe 4.0 on an X570 board, but it is still slower as that total connection is PCIe 4.0 4x lanes. And it shares bandwidth with everything else but the first M.2 slot there. So your additional storage, sound, network, USB controllers, etc all communicate on those four lanes, adding a GPU to that, not ideal.

Consumer boards just aren't typically intended to have two large GPUs in them

 

[Eximo]

Could also suggest Riser cables and mounting the GPU basically wherever, but there are issues with those. You need really nice ones to avoid issues. Many were built with PCIe 3.0 in mind.

Other option is putting in water cooling. Considering the expense of the cards, shouldn't be a huge issue.

 

[fre-deric]

What about these motherboards? They have Socket AMD AM4 so I can have AMD Ryzen 7 3700X and also they have more than 2 PCIe 4.0 slots x16 so there could enough space for both RTX 3090:

• MSI MEG X570 UNIFY
• Gigabyte X570 AORUS MASTER ATX AM4
• Gigabyte X570 AORUS XTREME EATX AM4

If I put first GPU to the first slot and the second GPU to the third slot (second slot would be empty), there could be enough space for both RTX 3090 with a gap between them. Is it right?

 

[Eximo]

Again, same limits as before. That 3rd slot is only wired for 4x on all of these boards.

It will function, but it isn't ideal.

 

[fre-deric]

Again, same limits as before. That 3rd slot is only wired for 4x on all of these boards.

It will function, but it isn't ideal.

I am sorry, but could you explain me please, why the third slot has only 4 lanes?
In documentation I can read that MSI MEG X570 UNIFY has 3x PCI-E x16 Slots. What do I not understand?

EDIT:
Maybe I got it.
I found better docs for GIGABYTE B550 VISION D motherboard that also has 3x PCI-E x16 Slots :

• 1 x PCI Express x16 slot (PCIEX16), integrated in the CPU: 3rd Generation AMD Ryzen™ processors support PCIe 4.0 x16 mode
• 1 x PCI Express x16 slot (PCIEX8), integrated in the CPU: 3rd Generation AMD Ryzen™ processors support PCIe 4.0 x8 mode
• 1 x PCI Express x16 slot (PCIEX4), integrated in the Chipset: Supporting PCIe 3.0 x4 mode

 

[Eximo]

Yes, that is a common configuration.

The CPU provides 20 PCIe 4.0 lanes. 4x to the first M.2 slot. 16x to the two graphics slots. The second slot is only wired for 8x lanes. So when you plug ANYTHING into the second slot, the first slot drops to 8x. This is called bifurcation.

The third x16 slot is connected to the chipset and is only physically wired for 4x lanes. Difference between B550 and X570 is that the PCIe lanes on the X570 chipset are also 4.0 and not 3.0 and the chipset has 4x PCIe 4.0 lanes of bandwidth to share amongst all devices connected to the chipset.

Technically you can plug any PCIe card into any length of slot and they will function, but you will run into performance limitations because due to the lower bandwidth.

In an ideal world you would have x16 and x16 to your massive GPUs. However, that is only achievable on Threadripper 3000 series at the moment, which is rather pricey. You can get away with it using PCIe 4.0 at only 8x because that is still PCIe 3.0 16x bandwidth, which is still decent. Only these late model cards are really effected by the difference, and it is only a few percent.

There are older boards that came with what used to be called PLX switching chips which allowed multiple x16 lanes to connect up, but that wouldn't help if all devices were used simultaneously anyway. (Also the chip found in dual chip graphics cards that they don't make anymore)