How To: Building Your Own Render Farm

[Guest]

GPU rendering wont change things that much, every time I get a faster PC i increase the detail in my scenes. Effects like diffuse reflections and Bounced light will become much more easier to preveiw but in the end people will just find another way of slowing things down. if this wasnt the case CG would still look like TRON and render at thousands of frame per second

 

[Draven35]

Exactly, Roger.

 

[Guest]

Hate to break it to you guys but check out respower in Huntsville Alabama...they are an automated independent remote access render farm for everyone

 

[Draven35]

I've been using Respower for years. Unfotunately, they dont support rendering for compositing software, and I wouldn't want to upload my average comp to them anyway...

 

[mi1ez]

Windows PE is perfect for capturing and applying images.

 

[Guest]

Another alternative to a render farm project may be using playstation III's i have read somwhere that the processing power of these machines can be parallelized(is that a word) to create computing power which rivals that of some of the worlds supercomputers at an unbeatable price/teraflop of processing power.

 

[Draven35]

I saw an interesting demo at SIGGRAPH a few years ago at the IBM booth- multiple networked Cell processors (it was 8 of them iirc) doing ray tracing in near real time.

 

[ruffrider]

question for all ... could a render farm be built using Nvidia Tesla cards and of course the appropriate mobo/vid card/power supply. Daughter and I are interested in the vid game industry and would like to build/buy a "all-in-one" system for smaller jobs. Thanks for your answers.

 

[Draven35]

If your renderer supports rendering using the Tesla cards and you can get the render features you want from the cards then yes. Most video game stuff is rendered in real time on the graphics card, not pre-rendered on a farm, though.

 

[ruffrider]

Thanks Draven ... after a bunch more looking around I started to realize that.. (she is the designer/programmer) lol .. I just play at building them and tinkering

 

[spiralsun1]

I render like crazy on my AMD -- I do multiple-figure scenes in Poser all the time. Would be nice to have a render farm for that.

 

[Draven35]

or even a few nodes, if they have started to support network rendering

 

[osiris813]

Draven, if I wanted to set up some rendernodes using i7 920 processors, and 1U cases, what motherboard would be my best bet? I like the i7 920 because it has 2 threads per core and supposedly it renders almost as fast as a dual XEON 5520 setup, for a lot less $$. And could that safely be overclocked to about 3.0 ghZ in a 1U case using the same setup described in the article?

Intel Core i7 920
______ Motherboard
______ RAM
supermicro cse-512l-260
Dynatron P199
Seagate Barracuda 7200.10 ST380215A 80 GB
Windows XP Pro 64-bit

Or does it seem a complete waste to go with the i7 because of the need for expensive DDR3 RAM (or can you use DDR2 RAM in a 1366 socket Mobo?)

Thanks!

 

[Draven35]

I haven't done much testing yet with using hyperthreading on i7- series machines. If its anything like it was in the P4 days, then the benefit of using hyperthreading will vary by scene content... some scenes will get a significant speed increase from hyperthreading while other scenes will show only a miniscule improvement because other things like the memory access get overloaded by running so many processor threads.

Any i7 system will require DDR3. i5 and i7 processors have an onboard memory controller that is set up for DDR3.

That said, if you're looking into building heavy-duty render nodes I'd look into building LGA1156 systems using the i7 or i5 for that socket. They are a little more budget minded and you won't be giving up that much render power over the socket 1366 version- most of the limitations of the 1156 socket affect systems with a lot of add-on cards which a render node wouldn't have. When reading this article, remember it was written before the release of any socket 1156 equipment.

 

[osiris813]

Thanks Draven! That's what I suspected re: DDR3 and the i7.

The reading I've done suggests that the i7 works well with vray, which is my primary renderer. I also use Vue 8xStream heavily, so I'm hoping the i7 hyperthreading will speed things up there.

Thanks for the suggestion about the 1156 socket. A quick glance indicates that they are indeed less expensive than the 1366 socket mobos.

Do you have a suggestion for a particular model of motherboard?

Also, when you say that the article was written before the release of any 1156 equipment, do you mean that the rest of the equipment mentioned in the article would not work with the i7 socket 1156 motherboard?

Can I use a 1U server case with this processor/mobo? How about overclocking to 3.0 ghZ?

I'm ready to start building these bad boys and I want to make sure to buy the proper equipment. Of course, I'll post my results on here once it's up and running!!

Thanks again Draven!

 

[Draven35]

A 1U case has a very small area available for heat sinks, so overclocking a i7 to 3 Ghz in a 1U case may be difficult or unstable. You might have to move up to a 2U or 3U cause to get enough space for an adequate heat sink.

LGA 1156-wise, I'd probably look at the Gigabyte P55-UD3r. I haven't seen much if anything in micro ATX for these yet, so you're probably looking at a full size mobo for it (and a case that can take it) I would be very cautious about overclocking a render node too far because when the render node has a load- i.e. you are rendering a scene- the processor is likely to spend extended amounts of time at or near 100% usage. This is why you need to make sure that it gets adequate cooling and why a 1U case may be unacceptable. I'd have to actually test this kind of setup to see how well it works.

 

[osiris813]

Thanks, I'll look into the Gigabyte you mentioned. The EVGA P55 variant 1156 motherboards also look interesting (there is a P55 Micro in that lineup).

Check out this lineup:
http://www.evga.com/articles/00502/P55_spec.pdf

I like the onboard CPU temp monitor, the passive chipset heatsink - the EVGA seem to have a good rap.

If I were to go with the EVGA P55 LE or standard P55, and the i7 920, do you think the 2U case would be of sufficient size? Do you have a recommendation for heatsink/fan, or a particular model case?

 

[Draven35]

Supermicro has been in the server biz a looong time... so i think they have case manufacturing down by now.

The P55 micro would fit the bill even though you're paying for SLI you'll never use...

 

[osiris813]

Draven,

Here's what I settled on after lots of research and price comparisons:

SUPERMICRO CSE-822T-400LPB 2U Rackmount Server Case
Intel Core i7 860
EVGA P55 LE ATX Motherboard
CORSAIR XMS3 8GB (4 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800)
Seagate Barracuda 7200.10 ST380215A 80GB 7200 RPM
Dynatron K785 77mm 2 Ball CPU Cooler
Microsoft Windows XP Pro 64 bit

For the NAS I went with:
Synology DS109 Diskless System 1-bay SATA NAS Server
Western Digital Caviar Black WD1001FALS 1TB 7200 RPM SATA 3.0Gb/s 3.5"

Should be a good start...we'll see how it goes once I get it set up. I plan to do a dual Xeon E5520 node, and a couple of Q9400 nodes just to compare performance. At least until Mental Ray 3.8 with iRay comes out and blows away all of this CPU nonsense. And Progeniq RenderBoost.

What do you think of the setup.

 

[Draven35]

The setup sounds okay, tho I'm not sure the performance increase of the i7 is enough for multiple nodes to balance the price. These estimates are *months* old and I haven't run estimates for 1156 systems. Also, I'll believe in GPU-based rendering when we get there....

 

[osiris813]

I considered that -- whether the performance increase of the i7 was worth the extra cost. I had to test the "hyperthreading" thing for myself. The question is, whether 1 node of i7 860 is as good as 2 nodes of Q8200, which is listed in the article. That means rendering speed and electricity consumption - there's bound to be a tradeoff.

My costs for the node I listed above were ~1100, so just less than 2 of the Q8200 nodes listed in the article.

Overall, the case is 2x as expensive, the ram is 2x as expensive, and the mobo is 2x as expensive just to run an i7. Is it 2x as good as a Q8200? Cinebench would say no, but we will find out soon enough. I'll keep posting on here as I get results

 

[Draven35]

Ok, try it with hyperthreading disabled too. Hyperthreading sometimes has a tendency to overwhelm the cache and memory acces.

 

[osiris813]

Is that a bottlenecking issue? If so, is there an advantage to going with more ram? Without hyperthreading, it would seem that the i7 is just a 2.8 gHz quadcore comparable to the yorkfield processors (i.e, not worth the price increase). The reading I had done with 3ds Max, Vray + i7 seemed to conclude that the i7's "8 logical cores" put it at or about the same specs as a non-hyperthreaded dual quad setup of comparable speed (2.5-2.8 gHz).

One last thought on going with a more powerful node vs. 2 less powerful nodes. I use Vue xStream a lot, and they unfortunately charge an extra $150 per rendernode license. I believe some other software publishers do the same. So that adds $150 to the cost of your second node. Also, You're running extra wattage 24 hrs a day on longer rendering jobs, so whatever that comes out to (variable costs vs. fixed costs).

I'll definitely try it with hyperthreading disabled and give a heads up. The goal of this experiment is to try to find a perfect balance point between performance vs. costs (fixed AND variable). I have a feeling I know and you know what the results will show, and it is going to be contrary to what the 3rd party vendors are pushing these days.

 

[osiris813]

And thanks again for all of your feedback Draven!

 

[Draven35]

Yes, its a bottlenecking issue and no, more RAM won't fix it. IT also depends on the content of your scene, so some benchmarks may not show it because many benchmark scenes I've seen around are pretty lightweight when compared to, you know, actual working scenes.