Need Help: Building Render Farm ($26K) - Maya / Mental Ray

Hussain36

Commendable
May 1, 2016
3
0
1,510
Hello,
I'm planning to building a render farm in the coming a few months, somewhere within 2016.
That will be for my 3D studio rendering.
but also I may use it commercially proving render service to other small studio. (That is just better than being there doing nothing.)

As a starting stage, I'll start with hardware budget of $26,000. But will expand it over the time.

Mainly that will be for animation in Maya 2016/2017 with mental ray. It will a dedicated standalone render farm.

I don't have that much knowledge on hardware, but it least basic enough to understand.

So, I'm thinking of Intel Xeon E5-2699 v4.
I wonder is dual CPU gives double the performance (200%)?
Or is it better to have two computers with single CPU than a single computer with two CPUs?

What I'm thinking of now is to have two computers, one with dual CPU, and the other with single CPU but a motherboard for dual CPU for future upgrade.

I need you help and support to choose the other parts of the computer

CPU: Intel Xeon E5-2699 v4
(suggested)

CPU Cooler: ???
(I'd prefer liquid cooling for the luxury of silence) Also, as far as I know, cooler CPU better performance. The cooler, the better.

Motherboard: ???

Memory: ???
(I'm thinking of 64GB for each CPU. Speed: 2400) (If 128GB is worth having, I can go for that) & ECC for sure...

Memory Cooler: ???
(If worth having)

GPU: ???
(Note that mental ray is CPU based renderer with GPU accelerated)

Storage: ???
I'm thinking of PCI-e - NVM

Power Supply: ???

OS: Windows 10 Pro
(I wonder if it is better to have Server version than consumer 10 Pro version )

If there is any other part that I've missed, please mention it.
Any other hints and suggests (for example: UPS, NAS Storage Farm, & etc...), please do.

All suggestions and opinions are welcome.

Thanks in advance...
 

Hussain36

Commendable
May 1, 2016
3
0
1,510
By the way,
Anyone have an idea about Intel Xeon Phi?
Does Maya and Mental ray support it?

Is it worth having?
 



Hussain36,

In my view, there should be separate systems for 1. Modeling /Animation and 2. Rendering /Processing

SYSTEM 1 > Modeling /Animation > Focuses on single threaded performance for modeling, and fast memory and disk systems, based on the Xeon E5-1680 v3 8-core @ 3.2 / 3.8GHz which has in Passmark baselines, the highest single-threaded performance of Xeon E5 1600 series of 2153. There is not a benefit to providing dual CPU's for this system. The ASUS X99-E WS motherboard, using the X99 chipset, produces very high CPU scores and has an M.2 Drive socket supporting extremely fast NVMe memory drive, and the GPU/ SLI. The GPUs are a pair of Quadro M4000's in SLI that together provide 2X GPU's, 16GB of video memory, and 3,328 CUDA cores. 3D performance would have extraordinary bandwidth, plus very high anti-aliasing (x64 instead of x16 of a GTX, and very high single and double precision 10-bit colour, and OpenGL support.

SYSTEM 2 > Rendering /Processing Focuses on calculation intense processing power. Mental Ray is fully thread-scalar- the more cores the better. This concept uses a pair of E5-2640 v4 (10-core @2.40 / 3.4 GHz with 128GB of DD4 2133 on a Supermicro X10 DAi motherboard that supports three GPU's. The first GPU is a Quadro K5200 (8GB) which has superb bandiwth and 3D performance.

For processor-based rendering, there is a better balance obtained with the use of GPU CUDA coprocessing, that is a Maximus Quadro /Tesla combination. There are CPU with slightly fewer cores but higher turbo speeds that power the coprocessors and have good enough single-threaded performance that they are useful for 3D modeling, animation and computation processing /simulation.

This is arranged in an NVIDIA Maximus configuration with a pair of (used) Tesla K20 coprocessors. The K5200 plus the two K20's provide 2304 + (2X 2496) = 7,296 CUDA cores and 18GB of graphahics memory. The prcomputing power for Mental Ray should be enormous. Tthis system can be built using Maxwell series Quadro M5000 and a pair of Tesla M40. for an approximate additional $9,000, but there are Pascal series Quadros and already a Pascal P100 ($12,400). In my view, the suggested configuration would provide excellent rendering/processing but at a far better cost /performance ratio.

The disk systems of both systems are not at all refined, but it's a start.

BambiBoom PixelCannon Modelanimetricgrapharific iWork TurboBlast ExtremeSignaure SuperModel 9600 ®©$$™®£™©™_ 5.2.16

SYSTEM 1 > Modeling and Animation

1. CPU: Intel Xeon E5-1680 v3 8-core @ 3.2 / 3.8GHz, 20M Cache, 140W > $2057
____ http://ark.intel.com/products/82767 > (Passmark No, 10 CPU average CPU score =17166)

2. CPU Cooler: Cooler Master Hyper 212 EVO CPU Fan > $32.

3. Motherboard: ASUS X99-E WS LGA2011-v3/ Intel X99/ DDR4/ 4-Way CrossFireX & 4-Way SLI/ SATA3&USB3.0/ M.2&SATA Express/ A&2GbE/ CEB Workstation Motherboard > $488

____ http://www.superbiiz.com/detail.php?name=MB-X99E-WS

4. RAM: Samsung DDR4-2133 16GB/2Gx72 ECC CL15 Samsung Chip Server Memory > $880 ($110 each)

____ http://www.superbiiz.com/detail.php?name=D416GE21S

5. GPU 1, 2: PNY NVIDIA Quadro M4000 8GB GDDR5 4DisplayPorts PCI-Express Video Card > $1,574 ( $787 each)

___ http://www.superbiiz.com/detail.php?name=PNY-M4000

6. Drive 1: Samsung SM951 512GB (NVMe) MZVPV512HDGL-00000 MZ-VPV5120 Gen3 M.2 80mm PCIe 3.0 x4 512G SSD with a SSD protective case - OEM> $370

____http://www.newegg.com/Product/Product.aspx?Item=9SIA12K3MH3372&cm_re=samsung_nvme-_-9SIA12K3MH3372-_-Product

7. Drive 2: Intel 750 Series AIC 400GB PCI-Express 3.0 x4 MLC Internal Solid State Drive (SSD) SSDPEDMW400G4X1 > $350 (Active Projects)

____ http://www.newegg.com/Product/Product.aspx?Item=N82E16820167359

8. Drive 3,4: 2X Seagate Constellation ES.3 ST4000NM0033 4TB 7200RPM SATA3/SATA 6.0 GB/s 128MB Enterprise Hard Drive (3.5 inch) > $406 ($203 each) (RAID 1)(Files, Backup, System Image)

____ http://www.superbiiz.com/detail.php?name=HD-ST40NM3

8. PSU: CORSAIR RMx RM1000X 1000W ATX12V / EPS12V 80 PLUS GOLD Certified Full Modular Nvidia Sli ready and crossfire support Power Supply> $160

____ http://www.newegg.com/Product/Product.aspx?Item=N82E16817139140

9. Optical Drive: Pioneer Black 16X BD-R 2X BD-RE 16X DVD+R 12X BD-ROM 4MB Cache SATA Blu-ray Burner BDR-209DBK > $64

10. Case: LIAN LI PC-A75X No Power Supply ATX Full Tower Case (Black) CA-A75 $179.99

11. Operating System: Microsoft Windows 7 Professional 64-bit w/ SP1 (1-Pack, DVD), OEM MSFQC04649 $138.99
________________________________________________________

TOTAL = $6,700


SYSTEM 2 > Rendering

BambiBoom PixelCannon <Rendermodeagraphalicious iWork TurboSignature ExtremeRay 9900 ®©$$™®£™©™ _ 5.1.16

CPU: 2X Intel Xeon Processor E5-2640 v4 (10-core @2.40 / 3.4 GHz, 25M Cache) > $1,840 ($920 each) ($920 each)(Superbiiz)

CPU Coolers: 2X Supermicro SNK-P0048AP4 CPU Heatsink For LGA2011 >$64 ($32 each)

Motherboard: Supermicro X10DAI-O Dual LGA2011/ Intel C612/ DDR4/ SATA3&USB3.0/ A&2GbE/ EATX Server Motherboard > $365

Memory:128GB (8 X 16GB) Crucial DDR4-2133 16GB/2Gx72 ECC/REG CL15 Server Memory > $800 (Superbiiz)

GPU 1: PNY NVIDIA Quadro K5200 8GB GDDR5 2DVI/2DisplayPorts PCI-Express Video Card > $2,000

GPU 2,3: 2X TESLA K20 GPU ACCELERATOR > $1,800 (used about $900 each)

Drive 1: Intel 750 Series AIC 400GB PCI-Express 3.0 x4 MLC PCIe Internal Solid State Drive (SSD) SSDPEDMW400G4X1 > $350

Drive 2, 3: 2X Seagate Constellation ES.3 ST3000NM0023 3TB 7200RPM SAS3/SAS 6.0 GB/s 128MB Enterprise Hard Drive (3.5 inch) $370 > ($185 each)

Power Supply: CORSAIR HXi HX1200i CP-9020070-NA 1200W ATX12V / EPS12V 80 PLUS PLATINUM Certified Full Modular Power Supply Nvidia Sli ready and crossfire support with C-link > $239

Case: LIAN LI PC-P80NB No Power Supply ATX Full Tower (Black) > $360

Operating System: Microsoft Windows 7 Professional SP1 64-bit English (1-Pack), OEM > $139.

____________________________________

TOTAL = about $8,327

Notes:

This would be a highly capable system and for your budget, there would be far more advantage in having two of these systems for $17,000 than a single $26,000 system.In fact, unless there is a gigantic project of astounding time limitations, starting with one dedicated system of this level. In a single-user situation may be already more than is necessary. I make 3180 X 2140 renderings (single source lighting) in 7-8 minutes on the 12-core Dell Precision T5500 and in about the the same time on the HP z420 listed below. The Telsa K20 GPU coprocessors are used but these are extremely durable- a MTBF rating of 72,000 means they can be expected to run continuously for more than 9 years. I was at a research facility yesterday that had 10 dual Xeon computing nodes, each containing 4X Tesla K20 and these had run since 2012 continuously with no failures.

1. The Xeon E5-2640 v4 has not been benchmarked extensively, but performance should be very good. There are two systems using the E5-2640 v4 on Passmark and the CPU score for a single CPU is 15776 and for a dual configuration- 25080. That would place it at No. 8 in the Passmark dual CPU list. With 1 total of 20-cores /40 threads, provide a lot calculation power. The 3.4Ghz turbo speed has a Passmark of single-threaded rating of 1861 which is sufficient capability in combination with the Quadro K4200 for quite demanding visualizations, so besides rendering, animation, Arc/GIS, 3D structural design /analysis, particle and thermal simulation, and visualizations of these should be very good.

2. The Quadro K5200 was chosen as the GPU's in a NVIDIA Maximus configuration (Quadro + Tesla) have to have the same series processor- in this case all have to be Kepler series.


4. Benchmarks of GPU's:

_________NVIDIA Tesla K20[/color][/b]

OS OS X
API OpenCL
OS Windows
API OpenCL

Face Detection

_________________32.104 mPixels/s

TV-L1 Optical Flow

_________________ 11.228 mPixels/s

Ocean Surface Simulation

______________ 1427.912 Frames/s

Particle Simulation - 64k

___________ 377.433 mInteraction/s

T-Rex

__________________4.129 Frames/s

Video Composition

_________________ 64.479 Frames/s

Bitcoin Mining

________________ 179.058 mHash/s

5. The disk system is unrefined and could benefit from a RAID controller and some form of 1+0 RAID.

6. I'm not yet convinced that Windows 10 is ready for professional use.

______________________________________________________________


Cheers,

BambiBoom

Modeling:

1. HP z420 (2015) > Xeon E5-1660 v2 (6-core @ 3.7 / 4.0GHz) > 32GB DDR3 1866 ECC RAM > Quadro K4200 (4GB) > Intel 730 480GB (9SSDSC2BP480G4R5) > Western Digital Black WD1003FZEX 1TB> M-Audio 192 sound card > 600W PSU> > Windows 7 Professional 64-bit > Logitech z2300 speakers > 2X Dell Ultrasharp U2715H (2560 X 1440)>
[ Passmark Rating = 5064 > CPU= 13989 / 2D= 819 / 3D= 4596 / Mem= 2772 / Disk= 4555]
[Passmark V9.0 Beta Rating = 5019.1 > CPU= 14206 / 2D= 779 / 3D= 5032 / Mem= 2707 / Disk= 4760] 3.31.16
[Cinebench R15 > CPU = 1014 OpenGL= 126.59 FPS] 7.8.15

Rendering:

2. Dell Precision T5500 (2011) (Revised) > 2X Xeon X5680 (6-core @ 3.33 / 3.6GHz), 48GB DDR3 1333 ECC Reg. > Quadro K2200 (4GB ) > PERC H310 / Samsung 840 250GB / WD RE4 Enterprise 1TB > M-Audio 192 sound card > Logitech z313 > 875W PSU > Windows 7 Professional 64> HP 2711x (27", 1920 X 1080)
[ Passmark system rating = 3844 / CPU = 15047 / 2D= 662 / 3D= 3550 / Mem= 1785 / Disk= 2649] (12.30.15)
 

Hussain36

Commendable
May 1, 2016
3
0
1,510
Thanks BambiBoom for the detailed reply. That helped to clear a few points.

But, what I know is that mental ray is CPU based render. so, GPU wouldn't have that much improvement in the results, from time duration point of view.

Shouldn't the CPU render farm has powerful CPU with just a basic GPU?
That is what I've notice from the pre-built render nodes; like BOXX ones.

If I misunderstation things, please clearify it to me.

Thanks,
Hussain
 


Hussain,

Metal ray as used in Maya and 3ds can run on CPU cores, and the CUDA cores in Quadro and Tesla cards. I'm not a Maya user but as afar as I know, there are settings for the core utilization. That said, the Tesla GPU acceleration is only used in Mental Ray rendering and in certain effects processing- blur and gaussian. This is another reason to suggest two systems and that the Tesla suggested are used as their value is specific to rendering.

Of course, more specific information is advisable before committing as the GPU acceleration and setup is particular to knowledge of the programme- so I can't verify it excatly .

This article is a preview of CUDA acclerated features for Mental Ray for Maya and 3ds 2015. It seems to be a good explanation of the idea:

Mental Ray’s GPU Accelerated GI Engine Prototype
July 7, 2014 nvjuliamental ray, mental ray for 3ds Max, mental ray for Maya

https://blog.mentalray.com/2014/07/07/mental-rays-gpu-accelerated-gi-engine-prototype/

Autodesk 3ds Max 2015 and Autodesk Maya 2015 ship with mental ray 3.12 which includes a prototype of our new global illumination engine accelerated by the GPU. We encourage our 3ds Max and Maya users to try it out. Your feedback will help us in making this a big step forward into the future of rendering with mental ray. While the current version is in prototype status and not yet feature complete, we are constantly improving the algorithms and adding new features. Your input is most welcome in this process.

The key idea of the new GI engine is full and exact simulation of the lighting interactions in a scene. This way, we overcome drawbacks from caching techniques and interpolation, and make mental ray more interactive and predictable. The brute-force raytracing approach is accelerated on CUDA capable NVIDIA GPUs making it particularly attractive in this set-up. Its result gets combined seamlessly and automatically with the primary rendering done on the CPU. This ensures full compatibility with existing custom shaders, which do not need to be touched in order to take benefit of the new GI engine.

The following Maya scene is rendered using the new GPU GI prototype in 11 min 34 sec (2 x Quad-core Xeon E5620 @ 2.4 GHz in hyper-threading, 8 GB RAM, Quadro K5000)

Mosque - mental ray 3.12 GI GPU diffuse
Autodesk Maya scene courtesy Lee Anderson, environment from openfootage

For comparison, this image is rendered with the classical finalgather automatic mode in 20 min 52 sec (2 x Quad-core Xeon E5620 @ 2.4 GHz in hyper-threading, 8 GB RAM, Quadro K5000)

mosque_fg_auto

In the current version, the GI GPU mode considers diffuse-diffuse bounces only, similar to what final gathering typically computes. In fact, if this mode is enabled without setting further parameters then finalgather settings are used to derive reasonable default parameters to render towards the same quality. If certain prominent ray tracing effects like mirror reflections or transparent windows are not used in a scene then the fastest diffuse mode is best suited. For current limitations, see below.

The following image is rendered with the GI prototype in diffuse mode in 37 minutes (Core i7-3930K (6 cores), 16 GB RAM, Quadro K5000)

Medieval - mental ray 3.12 GI GPU diffuse
Autodesk 3dsMax scene courtesy David Ferreira

The following 3dsMax scene is rendered on the CPU with finalgather force in 13 hours (Core i7-3930K (6 cores), 16 GB RAM, Quadro K5000)

Medieval - mental ray 3.12 CPU

The GI GPU mode can be enabled and controlled with scene options or on the command line of the standalone mental ray. We also provide scripts for Maya and 3dsMax that provide a simple GUI for enabling GI GPU (see screenshot). Please, note, that this is by no means how we envision it to be integrated in the applications. It’s rather to provide easy access to users that would like to test the prototype.

max-screenshot
Script Download

The scripts to easily enable and access the GI GPU prototype can be downloaded directly from us here:

Autodesk Maya MEL Script
Autodesk 3dsMax Script

Current limitations

GI GPU transfers the scene geometry, presampled shader data, and some constant amount of buffer memory onto the GPU. Textures are not needed on the GPU. In the case that the GPU memory is not sufficient, there is an equivalent CPU mode. The new GI engine can still be used but the GPU acceleration must be disabled (uncheck the ‘Use GPU’ checkbox). There is also an absolute limit of 25 million triangles.

For GI GPU to be effective, finalgather must be turned on. Some features are not yet supported: distorting lens shaders, motion blur, particles, volume shaders, camera clipping planes, progressive rendering. There is only limited support for scattering shaders, emissive materials, and hair rendering.

Before testing GI GPU, we recommend to install a recent version of the NVIDIA graphic card driver.

Feedback and discussion

We would like to hear your feedback and see your renderings using this prototype. Please join our NVIDIA Advanced Rendering forum, if you have not already, and send us your comments and discuss the mental ray GI GPU prototype. The dedicated mental ray GI GPU Prototype forum topic here

______________________________________________________________

I hope this helps to clarify

Cheers,

BamibBoom