Build advice for a 4 gpu system

jblackmd

Honorable
Mar 20, 2018
88
3
10,545
I would like to build a system that would accommodate 4 Nvidia GPU's, primarily for Iray graphics. I need advice on the motherboard and chipset. Currently, my rig has 3 cards and an I7 4790, but beyond finding a motherboard with 4 slots in it, what are the chipset considrations? I would plan on running 4 1080 Ti's

Thanks!
 

Aeacus

Titan
Ambassador
Here's full list of quad-SLI MoBos you can consider,
pcpp: https://pcpartpicker.com/products/motherboard/#l=4&sort=price&page=1

Asus X99-E WS is the best but it also costs accordingly,
specs: https://www.asus.com/Motherboards/X99E_WS/
in-depth review: https://www.tweaktown.com/reviews/6806/asus-x99-e-ws-intel-x99-workstation-motherboard-review/index.html

Chipset depends on which CPU you're planning to use. E.g X399 = AMD Threadripper; X299 = Intel Skylake-SP; X99 = Intel Haswell-E, Broadwell-E.

But if you plan to re-use your i7-4790 (Haswell) then you'll need LGA1150 socket and either Z87 or Z97 chipset MoBo.
List of compatible MoBos: https://pcpartpicker.com/products/motherboard/#l=4&sort=name&s=24&page=1

Here's also Intel chipset comparison which includes: X299, X99, Z270, Z170, Z97 and Z87,
link: https://ark.intel.com/compare/122941,81761,98089,90591,82012,75013
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


Definitely helpful! I'm not necessarily settled on the chip I have unless it would work. I've read some things I don't understand regarding the use of multiple GPU's, namely about lanes. I'm led to believe that I would need 40 lanes for the cards to run at full capacity, and that I would need a 9th generation processor to do that. Now, if my processor would run 4 cards I'm fine with that. I would then just upgrade the mobo. I have a Gigabyte Z97 board now, but it only has 3 slots. Fitting the 3 cards in it is a bit tight, and the cards are so close to each other that it runs a little hotter than I like.
 
the GPUs are not capable of saturating even 8 PCI-e lanes, much less 16...

Pretty sure most boards using 4 GPU slots would revert to 8 lanes each, at best....

(Might be some TR4 boards that offer more)
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


I know that it can handle 3, as it's doing so as we speak. What kind of chipset would I need to do 4?
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


Would I notice a difference?
 

Aeacus

Titan
Ambassador
If you want best performance then C422 chipset (LGA2066) and any Skylake-SP CPU would do since all Skylake-SP CPUs have 48 PCI-E lanes,
wiki: https://en.wikipedia.org/wiki/LGA_2066#Workstation_(Skylake-SP)

With 48 PCI-E lanes in CPU, the breakdown for quad-SLI would be as follows:
GPU 1 - 16 lanes
GPU 2 - 16 lanes
GPU 3 - 8 lanes
GPU 4 - 8 lanes

Do note that Skylake-SP CPUs don't work on X299 chipset MoBos despite they using the same LGA2066 socket. For Skylake-SP CPUs, you'll need to use the server C422 chipset MoBo. E.g Gigabyte MW51-HP0,
specs: http://b2b.gigabyte.com/Server-Motherboard/MW51-HP0-rev-10#ov

But if you were to use Skylake-X/Kaby Lake-X CPU then you can use the consumer X299 chipset MoBo just fine. Most lanes Skylake-X/Kaby Lake-X CPU has is 44 and breakdown would be as follows:
GPU 1 - 16 lanes
GPU 2 - 12 lanes or 8 lanes
GPU 3 - 8 lanes
GPU 4 - 8 lanes

Here's a list of quad-SLI X299 chipset MoBos,
pcpp: https://pcpartpicker.com/products/motherboard/#l=4&sort=price&s=35
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


Wow ...that answers a lot of questions! The Xeon processors that fit a 2066 socket seem ridiculously expensive. Now, I guess a very basic question is: for what I want to use the computer for, i.e. Iray rendering, would it work at all if I used a lesser setup, say my current 4790 with a 4 slot board? Would it tend to freeze or lock up? If so, I think a good compromise for my purposes would be a consumer X299 with the Skylake-X/Kaby Lake-X CPU. Does that sound reasonable?
 

Aeacus

Titan
Ambassador
Since your i7-4790 is consumer grade chip and not a server chip, you can't run 4x GPUs with it. There is just no support for quad GPU setup. Possible GPU combinations are:
1x GPU at 16 lanes or
2x GPUs at 8 lanes each or
1x GPU at 8 lanes, 2nd and 3rd GPU at 4 lanes each

proof: https://ark.intel.com/products/80806/Intel-Core-i7-4790-Processor-8M-Cache-up-to-4_00-GHz
Read under "Expansion Options".

So, if you want to use quad GPU setup, you'll need a server CPU that has support for 4x GPUs. Do note that even the latest consumer grade CPUs, e.g i7-8700K, doesn't support more than 3x GPUs.


In my previous reply, i only covered Intel CPUs due to the time restraint i had. But there are also AMD Threadripper CPUs that offer even more PCI-E 3.0 lanes, up to 64.
Threadripper 1900X review: http://www.tomshardware.co.uk/amd-ryzen-threadripper-1900x-cpu,review-34070.html

With Threadripper, the quad-SLI breakdown would be as follows:
GPU 1 - 16 lanes
GPU 2 - 16 lanes
GPU 3 - 16 lanes
GPU 4 - 16 lanes

Since Threadripper works only in X399 chipset TR4 socket MoBo and you need quad-SLI, here are MoBos that match those criteria,
pcpp: https://pcpartpicker.com/products/motherboard/#l=4&s=36&sort=price&page=1

You'll also need Threadripper CPU. Luckily, there's only 3 to choose from,
pcpp: https://pcpartpicker.com/products/compare/CF7CmG,cRDzK8,BvgPxr/
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


That's good to know. The AMD prices are certainly more reasonable. I'll keep that in mind for my next build! I have a question for you, a little off the topic but you seem very well versed in this stuff.

My current build is a Gigabyte Z97X Gaming 5, 4790 , GTX 980 TiX2 and a GTX 780 ti, 1300 W, Coolermaster Haf X
It's all been working fine, but I'm in the midst of upgrading my 780 Ti to a 1080 Ti, mainly because some of the scene I render are too large for the 780. Here's my question: when all 3 cards are running, the middle card (980) gets a bit hot, so I wanted to see if it ran cooler by putting it in in the lowest slot. When I did that, the card wouldn't run. The 780 ran just fine in that slot, and it ran fine in the middle slot. Looks like the top slot is PCIexX16, the middle is X8 and the lowest is X4. Is there any reason that you can think of that the 980 won't run in the lowest slot? When I put it back in the middle slot it ran. My plan was to run the 1080 and both 980's, but I'm concerned that the lowest slot isn't running the 980 when I tried it there. I read somewhere that I can set the bios so that all slots run at X4. Would that fix the problem? Is there another fix for the problem? My fallback would be to just run 2 cards, the 1080 and one of the 980's. I never use SLI so that isn't an issue.
 

Aeacus

Titan
Ambassador

According your MoBo specs, you should set the 3rd PCI-E x16 slot into x4 lane mode within your BIOS when you install x16 lane GPU in it. This is also stated so in your MoBo manual, page 56, under "PCIE Slot Configuration (PCH)".

By default, it should be set to "Auto". So go and set it to "x4" and look if you can get your GTX 980 Ti running in the 3rd slot.

If not, then you're better off making a separate topic in the Graphics Cards subforum to get more GPU experts in to help you out.
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


You, Sir, do know your stuff! I set the BIOS as you suggested, and everything works perfectly! Currently running the new 1080 Ti and 2 980 Ti's. Question about your server Mobo suggestion: I have very little experience practically with server setups. Do I have to run a server OS if I get a server mobo, or will Windows 10 suffice? I know, that may be a stupid question but you've been indulging me so far so I thought I would ask!
 

Aeacus

Titan
Ambassador
No need to run server OS with server MoBo. Both consumer and server hardware work just fine with consumer OS (e.g Win 10). What makes server OS different from consumer OS is the features it has.

Though, keep an eye out for RAM support on X399 chipset MoBos since some of them only support server RAM.
In a nutshell: non-ECC, non-registered (un-buffered) RAM is consumer RAM while: ECC, registered RAM is server RAM.
further reading: https://www.pugetsystems.com/labs/articles/Advantages-of-ECC-Memory-520/
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545

Good to know! Thanks!
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


Ok. Did a bit of window shopping, and I came up with this. I think It would do what I want it to do. I already bought 2 of the 1080 Ti's, that are in my current build:

ASRock X299 OC FORMULA LGA 2066 Intel X299 SATA 6Gb/s USB 3.1 ATX Intel Motherboard

HyperX Fury 16GB (1 x 16GB) DDR4 2400MHz DRAM (Desktop Memory) CL15 1.2V White DIMM

Intel Core i7-7800X Skylake-X 6-Core 3.5 GHz LGA 2066 140W BX80673I77800X Desktop Processor

Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan

EVGA SuperNOVA 1300 G2 120-G2-1300-XR 80+ GOLD 1300W Fully Modular Includes FREE Power On Self Tester Power Supply

Cooler Master HAF X - High Air Flow Full Tower Computer Case with Windowed Side Panel and USB 3.0 Ports

LITE-ON Black 18X DVD-ROM 48X CD-ROM SATA DVD-ROM Drive Model iHDS118-104 8U

EVGA GTX 1080 Ti

SanDisk Ultra 3D 2.5" 1TB SATA III 3D NAND Internal Solid State Drive (SSD) SDSSDH3-1T00-G25

Thoughts?
 

Aeacus

Titan
Ambassador
Hyper 212 Evo is very poor choice when it comes to the CPU air coolers. Only Intel's stock CPU cooler is worse than Hyper 212 Evo. What makes Hyper 212 Evo so popular is it's very low price.

Since i7-7800X is high wattage CPU (140W at stock clocks), it can consume near 350W under OC and even the 240mm rad AIO won't be enough for it. With 240mm rad AIO, e.g Corsair H100i v2, at 100% load, you'll hit the 80°C quite easily. And it doesn't stop there. Temps could climb even further, up to 100°C. Note: you'll get CPU thermal throttle at 90°C.

For your Skylake-X CPU, and especially when you're planning to OC it, you're looking towards 360mm rad AIO. Since there isn't many 360mm rad AIOs around, you'll get the best temps and most eyecandy out of the Thermaltake Floe Riing RGB 360 TT Premium Edition,
specs: http://www.thermaltake.com/Cooler/Liquid_Cooler_/All_in_One/C_00003122/Floe_Riing_RGB_360_TT_Premium_Edition/design.htm
review 1: https://www.tweaktown.com/reviews/8301/thermaltake-floe-riing-360-tt-premium-cpu-cooler-review/index.html
review 2: http://www.tomshardware.com/reviews/thermaltake-floe-riing-rgb-360-tt-premium-edition,5315.html
pcpp: https://pcpartpicker.com/product/CVJkcf/thermaltake-floe-riing-rgb-360-tt-premium-edition-423-cfm-liquid-cpu-cooler-cl-w158-pl12sw-a

You can mount the 360mm rad as top exhaust in your HAF X case.

Oh, best performance to price ratio 1TB SSD is Crucial MX500,
Sandisk Ultra 3D (1TB) vs Crucial MX500 (1TB) comparison: http://ssd.userbenchmark.com/Compare/SanDisk-Ultra-3D-1TB-vs-Crucial-MX500-1TB/m316514vsm406099
Crucial MX500 review: https://www.anandtech.com/show/12165/the-crucial-mx500-1tb-ssd-review
pcpp: https://pcpartpicker.com/products/compare/FLDJ7P,h3tQzy/

I'm also planning to go with Crucial MX500 (1TB) so i can replace my WD Blue (1TB) HDD inside my Skylake build.
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


I'm not planning on overclocking it at all. I chose the board based on the PCIex slots. I have a coolermaster fan in my current rig, and generally the CPU runs at under 60 c under load. I understand that the Skylake will run hotter, and I get the attraction with water cooling, but if I'm not going to overclock it is water cooling really necessary? How about the Noctua NH-U9DXi4 90mm SSO2? Looks like that got pretty good reviews.

As for the Sandisk, I basically chose that because I have experience with it. I already have five of them, 2 in my current rig, 1 on my laptop, 1 in my son's laptop, 1 in my wife's desktop. They really work great, but I don't have a issue with trying the Crucial MX500
 

Karadjgne

Titan
Ambassador
The TDP for the cpu is a number set by Intel after running a standardized set of nominal apps, basically websurfing, social media, light office work etc. So you are looking at a 140w TDP cpu. And 140w coolers and 4x 1080ti. Seems a little biased running a $6000 pc on a $30 cpu cooler.
Once you open up All the cores under any serious workload, expect that 140w TDP to dissappear, you'll be getting far closer to Peak values, which with those cpus, even at stock, is @2x TDP. That's 280w worth of heat. The hyper212 or any other budget cooler will not work. Period.
For any 2066, minimum of NH-D15s, scythe Fuma, cryorig R1. Better would be decent 280/360mm aio as air coolers only handle upto @250-300w at best, aios can handle closer to 350-400w.

Its your choice, but you can never Over Cool a cpu on consumer grade cooling, and that includes full custom loops. Under Cooling is far too easy, especially with the bigger cpus, like the x99 - TR4 enthusiast class stuff.
 

Aeacus

Titan
Ambassador

Valid point.
Though, i forgot that 4x GTX 1080 Ti GPUs put a quite good load on CPU and Noctua NH-U9DXi4 90mm SSO2 wouldn't be enough.
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545


I'm convinced. Sounds like the cooler to go with is the Thermaltake Floe Riing
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545
Well, it's been a while and now I finally have the money to build my rendering server. I'm thinking that I'll use the 3 1080Ti cards that I have, but want expand ability to 4 in the future. So I want opinions: AMD Threadripper or Intel? I was thinking either the Intel Core i7-7800X Skylake-X 6-Core 3.5 GHz LGA 2066 or the AMD - Threadripper 1900X 3.8 GHz 8-Core. Which of these 2 will perform better? I get the sense that the AMD would but I question it's reliability and I read that it can run hot. Of course I've read that about the intel too. Also, when I read the reviews for a lot of the AMD boards, they seem kind of poor. It's a bit disheartening that none of the boards seem to be rated 5/5, and there are issues apparently with all of them.

Just so you get a sense of what I'm going to expect from the build: I do a lot of 3-d rendering, and not uncommonly it might need to render for the whole day, and night, into the next day. I want a fast, solid build. You guys know a lot more about this than I do, so I would appreciate any and all advice.
 

Aeacus

Titan
Ambassador
i7-7800X has more-or-less equal performance to TR 1900X,
comparison: https://cpu.userbenchmark.com/Compare/AMD-Ryzen-TR-1900X-vs-Intel-Core-i7-7800X/m340638vsm304816

Though, at current pricing, you can get TR 1920X with a bit cheaper price than i7-7800X,
pcpp: https://pcpartpicker.com/products/compare/ymtWGX,cRDzK8,BvgPxr/

And with TR 1920X, you'll get double the cores/therads (12/24) compared to the i7-7800X (6/12). Due to that, TR 1920X multi-core performance is also considerably better,
comparison: https://cpu.userbenchmark.com/Compare/AMD-Ryzen-TR-1920X-vs-Intel-Core-i7-7800X/3934vsm304816

CPU PCI-E lane wise, you're looking at:
28 lanes for i7-7800X
60 lanes for TR 1920X

Making the Threadripper 1920X better CPU to use for your needs rather than Intel i7-7800X.

As far as MoBo for TR 1920X goes, time has moved onward since we discussed your build (about a year has passed) and there are more X399 chipset MoBos to choose from. Of course, if you look cheap MoBos then yes, you'll see quite a bit of issues with MoBos. Even the most expensive MoBos may have quite a bit of issues. The key here is to find the MoBo with least amount of issues while offering the features you need.

The best MoBo (not the most expensive) that i found for TR 1920X would be: ASRock Fatal1ty X399 Professional Gaming,
specs: https://www.asrock.com/mb/AMD/Fatal1ty X399 Professional Gaming/index.asp
review: https://www.anandtech.com/show/13036/the-asrock-x399-professional-gaming-motherboard-review
pcpp: https://pcpartpicker.com/product/x2...-atx-tr4-motherboard-x399-professional-gaming
 

jblackmd

Honorable
Mar 20, 2018
88
3
10,545
i7-7800X has more-or-less equal performance to TR 1900X,
comparison: https://cpu.userbenchmark.com/Compare/AMD-Ryzen-TR-1900X-vs-Intel-Core-i7-7800X/m340638vsm304816

Though, at current pricing, you can get TR 1920X with a bit cheaper price than i7-7800X,
pcpp: https://pcpartpicker.com/products/compare/ymtWGX,cRDzK8,BvgPxr/

And with TR 1920X, you'll get double the cores/therads (12/24) compared to the i7-7800X (6/12). Due to that, TR 1920X multi-core performance is also considerably better,
comparison: https://cpu.userbenchmark.com/Compare/AMD-Ryzen-TR-1920X-vs-Intel-Core-i7-7800X/3934vsm304816

CPU PCI-E lane wise, you're looking at:
28 lanes for i7-7800X
60 lanes for TR 1920X

Making the Threadripper 1920X better CPU to use for your needs rather than Intel i7-7800X.

As far as MoBo for TR 1920X goes, time has moved onward since we discussed your build (about a year has passed) and there are more X399 chipset MoBos to choose from. Of course, if you look cheap MoBos then yes, you'll see quite a bit of issues with MoBos. Even the most expensive MoBos may have quite a bit of issues. The key here is to find the MoBo with least amount of issues while offering the features you need.

The best MoBo (not the most expensive) that i found for TR 1920X would be: ASRock Fatal1ty X399 Professional Gaming,
specs: https://www.asrock.com/mb/AMD/Fatal1ty X399 Professional Gaming/index.asp
review: https://www.anandtech.com/show/13036/the-asrock-x399-professional-gaming-motherboard-review
pcpp: https://pcpartpicker.com/product/x2...-atx-tr4-motherboard-x399-professional-gaming
I know the performance should be better with AMD, but when I go on Newegg and look at some of the reviews for that ASRock board, they're kind of scary. Reports of DOA boards, quality issues etc. Is there a comparable board for the AMD chip that has a better track record?