CFD simulations workstation

[Marco_44]

Hello everyone,
We need a workstation with about 64 xeon processors (around 3.5GHz) and 128GB of RAM, for CFD simulations. What would be a good deal?

 

[kanewolf]

You can't get 64 Xeon cores at 3.5Ghz. There is no such combination of chips. As the number of cores in a chip goes up (max of 22 now) the clock speed goes down. Even a 4 socket motherboard (which would be expensive) could give you 64 cores (E5-4660 V4 CPU) but you only get 2.2Ghz base clock speed.

 

[Marco_44]

You can't get 64 Xeon cores at 3.5Ghz. There is no such combination of chips. As the number of cores in a chip goes up (max of 22 now) the clock speed goes down. Even a 4 socket motherboard (which would be expensive) could give you 64 cores (E5-4660 V4 CPU) but you only get 2.2Ghz base clock speed.

First of all, thank you for the answer.
Secondly, I am not an expert, but the only solution then would be buying 4 workstations and run in parallel, am I right?
what would be an aproximate price?

 

[kanewolf]

Start by verifying that your code will run on 64 thread efficiently. Buy some time on Amazon EC2 hosts. You can run 64 vCPUs on an x1 instance for $7 / hour. You can try 128 vCPUs for $14 / hour.

You need to look at using coprocessors. GPUs or Intel PHI. What language is your simulation written in?

 

[bambiboom]

Hello everyone,
We need a workstation with about 64 xeon processors (around 3.5GHz) and 128GB of RAM, for CFD simulations. What would be a good deal?

Marco_44,

CFD simulation is quite compute intensive, but unless your group has created a proprietary, massively threaded application, a dual 8-core Xeon at a reasonable clock speed with a a lot of RAM and a GPU coprocessor should produce reasonable simulation run times. For example, this is the Autodesk recommended system for Autodesk CFD:

" Recommended Desktop

Model: HP Z620 with Intel Xeon E5-2680 @2.7Ghz (8 cores)*
RAM: +16GB
Video: Nvidia Quadro 4000
Hard Drive: 15K RPM Hard Drive
*If more speed is needed add a 2nd E5-2680 for a total of 16 cores"

By coincidence, we use a similar system for analysis, simulation, and CPU rendering of industrial designs:

Analysis / Simulation / Rendering:

HP z620 (2012) (Rev 3) 2X Xeon E5-2690 (8-core @ 2.9 / 3.8GHz) / 64GB DDR3-1600 ECC reg) / Quadro K2200 (4GB) + Tesla M2090 (6GB) / HP Z Turbo Drive (256GB) + Samsung 850 Evo 250GB + Seagate Constellation ES.3 (1TB) / Creative Sound Blaster X-Fi Titanium PCIe sound card + Logitech z313 2.1 speakers / 800W / Windows 7 Professional 64-bit > > HP 2711x (27" 1980 X 1080)
[ Passmark System Rating= 5675 / CPU= 22625 / 2D= 815 / 3D = 3580 / Mem = 2522 / Disk = 12640 ] 9.25.16
[ Cinebench R15: CPU = 2209 cb / Single core 130 cb / OpenGL= 119.23 fps / MP Ratio 16.84x] 10.31.16

For your use, at least 128GB RAM would be recommended.

I might mention that the above system was assembled for a total cost of under $1,500. If you are modelling an extremely large dynamic fluid systems, a series of this kind of system run in parallel would require 4 systems to accomplish the 64-core configuration. The Xeon E5-2690 runs on all cores at between 3.3-3.5Ghz.

The key to this system for this kind of simulation is the Tesla M2090 6GB GPU coprocessor. Here is a list of the CFD applications that are GPU-accelerated.

For a new, single system, perhaps:

Compute / Analysis / CFD Simulation Workstation / Server_ 3.13.17

Case / Motherboard / PSU / CPU Coolers: Supermicro SuperServer SYS-7048GR-TR Dual LGA2011-3 2000W 4U Rackmount/Tower Server Barebone System (Dark Gray) (Black) >$1,867.65 (Amazon)
https://www.amazon.com/Supermicro-SYS-7048GR-TR-SuperWorkstation-Desktop-AST2400/dp/B00QGKBYO4/ref=sr_1_6?ie=UTF8&qid=1485478554&sr=8-6&keywords=supermicro+sys+-7048

___[System] https://www.supermicro.com/products/system/4u/7048/SYS-7048GR-TR.cfm] Supermicro SYS-7048GR-TR

___[Motherboard] Supermicro X10DRG-Q https://www.supermicro.com/products/motherboard/Xeon/C600/X10DRG-Q.cfm

CPU: 2X Intel Xeon E5-2673 v4 CPU 2.3 / 3.6GHz 20-Core 135W, 50MB cache > $6,398. ($3,199. each) (Amazon)

https://www.amazon.com/Intel-E5-2673-20-Core-Similar-E5-2698v4/dp/B01MXH45XC/ref=sr_1_1?ie=UTF8&qid=1485479397&sr=8-1&keywords=E5-2673+v4]

RAM: 512GB (16X 32GB) SAMSUNG 32GB 288-Pin DDR4 SDRAM Registered DDR4 2400 (PC4 19200) Memory (Server Memory) Model M393A4K40BB1-CRC[/url][/b] > $4,192 ($262 ea) (Newegg)

https://www.newegg.com/Product/Product.aspx?Item=N82E16820147574

GPU 1 Quadro P6000 24GB 384-bit GDDR5X PCI Video Card - Workstation > $5,399 (Newegg)

https://www.newegg.com/Product/Product.aspx?Item=N82E16814133636&cm_re=quadro_p6000-_-14-133-636-_-Product]

GPU 2,3,4: 3X NVIDIA Tesla K40 Graphic Card - 1 GPUs - 745 MHz Core - 12 GB GDDR5 SDRAM 900-22081-2250-000 > $13,276 ($3,316 each)

https://www.amazon.com/NVIDIA-Tesla-Graphic-Card-900-22081-2250-000/dp/B00KDRRTB8/ref=sr_1_1?s=pc&ie=UTF8&qid=1489405117&sr=1-1&keywords=nvidia+tesla+k40

Drive 1: 960 PRO Series - 2TB PCIe NVMe - M.2 Internal SSD (MZ-V6P2T0BW)[/url] > $1,300 (Amazon)
Samsung]https://www.amazon.com/Samsung-960-PRO-Internal-MZ-V6P2T0BW/dp/B01LY3Y9PH/ref=sr_1_1?ie=UTF8&qid=1485482595&sr=8-1&keywords=960+pro+2tb]Samsung

Drive 1 PCIe Adapter /Cooler: Angelbird Wings PX1 PCIe x4 M.2 Adapter > $75 (Amazon)

https://www.amazon.com/Angelbird-Wings-PX1-PCIe-Adapter/dp/B018U79YQK/ref=sr_1_1?ie=UTF8&qid=1485482517&sr=8-1&keywords=Wings+PX1]

RAID Controller: LSI MegaRAID LSI00332 (9286-8e) PCI-Express 3.0 x8 Low Profile SATA / SAS RAID Controller - Single--Avago Technologies > $725 (Newegg)
https://www.newegg.com/Product/Product.aspx?Item=N82E16816118189&cm_re=LSI_MegaRAID_LSI00332-_-16-118-189-_-Product

Drive 2,3,4,5,6,7,9: 7X Seagate Constellation ES.3 ST4000NM0023 4TB 7200 RPM 128MB Cache SAS 6Gb/s 3.5" Enterprise Internal Hard Drive Bare Drive[/url] > $1,435 ($205 each) (Newegg)
https://www.newegg.com/Product/Product.aspx?Item=9SIA2W021H6292&cm_re=Constellation_es.3-_-22-178-306-_-Product]

Optical Drive:

LG]https://www.newegg.com/Product/Product.aspx?Item=N82E16827136264]LG Black 16X Blu-Ray BDXL SATA Internal rewriter with 3D Playback, Model BH16NS40[/url] > $100 (Newegg)

_________________________________

TOTAL = $33,467.65

Notes:

1. The Xeon E5-2673 v4 20C @ 2.3 /3.3 (3.6GHz max.) has an average Passmark single CPU rating of 24201. The pair providing a total of 40 cores /80 threads for thread- distributed compute-intensive processing, analysis, simulation animation, and CPU rendering, and the 3.6Ghz maximum speed produces a single-thread rating of 2003 ,sufficient for complex/large scale 3D modeling / animation. Alternately, the Xeon E5-2679 v4 is 20C @ 2.5 /3.3GHz with a higher compute rating of 25236 but single-thread rating of 1837.

2. The Supermicro X10DRG-Q is unique as it provides four PCie x16 GPU slots that all accommodate double-height GPU's without covering another PCIe slot.

3. The Quadro P6000 + 3X Tesla GPU's ( Pascal GP102) provide 60GB of 384-bit GDDR5X memory at a maximum 432GB/s memory bandwidth and a total of 15,360 CUDA cores, providing a compute performance of 12 TFLOPS single precision. Each P6000 can support four 5K monitors. There is only one Passmark test of a Quadro P6000, resulting in a 3D mark of 11926 which is the 3rd most powerful GPU in Passmark results. For comparison, the No, 2 result is GTX1080= 11971 and No. 1 Titan X Pascal = 12929. Power rating = 225W

4. The OS/ Programs plus: active projects, libraries, reference are accommodated on the Samsung 960 Pro 2TB.

And there is 40-cores /80 threads plus all those CUDA cores to work on the simulations. I think that would run a project up to the scale of The Black Sea. As a proprietary system, this would cost nearer $50,000.

OR:

High Performance: Computing / Scientific / Visualization / Analysis / Animation / Video Processing / Simulation Workstation_ 2.9.17

CPUs: 2X Intel Xeon E5-2687w V4 3.0 / 3.5GHz 12-Core Processor $4405.98 ($2202.98 each @ Newegg)

___https://ark.intel.com/products/91750/Intel-Xeon-Processor-E5-2687W-v4-30M-Cache-3_00-GHz

CPU Coolers: 2X Corsair H100i v2 70.7 CFM Liquid CPU Cooler $198.98 ($99.99 each @ Newegg)

Motherboard: Supermicro MBD-X10DAX EATX Dual-CPU LGA2011-3 Motherboard ($439.64 @ Amazon)

Memory: 256GB 16X Samsung 16GB (1 x 16GB) Registered DDR4-2400 Memory > $2223.84 ($138.99 each @ Newegg)

Video Cards: 2X PNY Quadro P6000 24GB Video Card (2-Way SLI) > $10813.74 ($5406.87 each @ Newegg)

RAID Controller: LSI Logic I/O Controller MegaRAID 9361-8i 8Port SATA/SAS PCI-Express 3 1GB DDR3 ($428.99)

Drive 1: Intel 750 Series 1.2TB PCI-E Solid State Drive ($908.40 @ Amazon)

Drives 2,3,4,5,6: 5X Western Digital Gold 8TB 3.5" 7200RPM Internal Hard Drive > $2174.95 ($434.99 each @ Newegg)

Optical Drive: LG BH10LS30 Blu-Ray/DVD/CD Writer

Power Supply: Corsair AX1500i 1500W 80+ Titanium Certified Fully-Modular ATX Power Supply ($409.99 @ Newegg)

Case: Caselabs Mercury S8 ($439.95)

Case Fan: 4X Noctua NF-A14 FLX 68.0 CFM 140mm Fan > $85.96 ($21.49 each @ Newegg)

Fan Controller: NZXT Sentry LX Fan Controller ($51.46 @ Amazon)
________________________________________________________________
Total: $22,531.07

OR:

Workstation_Compute / Analysis / Simulation / CPU Rendering_2.12.17

Case /Motherboard /Power supply/ CPU coolers: Supermicro SuperWorkstation SYS-7038A-I Dual LGA2011 / Suprmicro X10DA / 900W Mid-Tower Workstation Barebone System (Black) > $720

___http://www.supermicro.com/products/system/tower/7038/SYS-7038A-i.cfm
___http://www.superbiiz.com/detail.php?name=SY-7038AI

CPU: (2) Intel Xeon E5-2640 V4 10-core @ 2.4 / 3.4 GHz 25MB L3 Cache LGA 2011 90W BX80660E52640V4 Server Processor > $1,980 ($990 each)

___http://ark.intel.com/products/92984/Intel-Xeon-Processor-E5-2640-v4-25M-Cache-2_40-GHz
___http://www.newegg.com/Product/Product.aspx?Item=9SIA2F84075666&cm_re=e5-2640_v4-_-19-117-631-_-Product
___Passmark CPU average CPU score =17166 _ 21842 (dual) Single Thread Mark = 1779

Memory: 128GB (8X 16GB) SAMSUNG 288-Pin DDR4 SDRAM ECC Registered DDR4 2133 (PC4 17000) Server Memory Model M393A2G40DB0-CPB > $1,432 ($179 each)

http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM3KB7224&cm_re=Samsung_DDR4-2133_unbuffered-_-9SIA5EM3KB7224-_-Product

GPU: 2X NVIDIA Quadro P4000 8GB GDDR5 4DisplayPorts PCI-Express Video Card > $1640 (about $820 each)

To be released on 3.21.2017

Drive 1: SAMSUNG 960 EVO M.2 500GB NVMe PCI-Express 3.0 x4 Internal Solid State Drive (SSD) MZ-V6E500BW> $249.99

___https://www.newegg.com/Product/Product.aspx?Item=N82E16820147594&cm_re=960_evo_5-_-20-147-594-_-Product

RAID Controller: LSI MegaRAID Internal Low-Power SATA/SAS 9240-8i 6Gb/s PCI-Express 2.0 RAID Controller Card, Single--Avago > $197

http://www.newegg.com/Product/Product.aspx?Item=N82E16816118107&cm_re=LSI_megaraid-_-16-118-107-_-Product

Disks 2, 3, and 4: Seagate Constellation ES.3 ST4000NM0033 4TB 7200 RPM 128MB Cache SATA 6.0Gb/s 3.5" Enterprise Internal Hard Drive Bare Drive > $630 ($210 ea.) (RAID 5)

http://www.newegg.com/Product/Product.aspx?Item=9SIA2W021H6289&cm_re=Seagate_Constellation_ES.3-_-22-178-307-_-Product

Optical Disk: ASUS Black 16X BD-R 2X BD-RE 16X DVD+R 5X DVD-RAM 12X BD-ROM SATA Blu-ray Burner BW-16D1HT> $84

https://www.newegg.com/Product/Product.aspx?Item=N82E16827135306

Operating System: Microsoft Windows 7 Professional SP1 64-bit English (1-Pack), OEM > $140.

http://www.superbiiz.com/detail.php?name=MSFQC08289
_______________________________________

TOTAL = about $7,043.00

The Supermicro Superworkstation is very convenient for assembly /configuration as the user need only plug in the CPU's, RAM, GPU, and drives, saving a lot of research and part sourcing, assembly, wiring, and etc.

OR:

Oftentimes, a used workstation can have unbeatable cost /performance:

HP Z820 Workstation M1H70UC E5-2667V2 16GB RAM 500GB HDD Win 7 > $3,125

And to that add a 2nd E5-2667 v2- about $1,000, 2nd CPU liquid cooler- $150, RAM to 128GB- $400, 2X Quadro P4000 $1,650, Drives, about $1,000 for a total of about $7,325.

But in this case, in my view, the "best deal" would be the dual Xeon E5-2640 V4 option.

Cheers,

BambiBoom

CAD / 3D Modeling / Graphic Design:

HP z420 (2015) (Rev 3) > Xeon E5-1660 v2 (6-core @ 3.7 / 4.0GHz) / 32GB DDR3 -1866 ECC RAM / Quadro K4200 (4GB) / Samsung SM951 M.2 256GB AHCI + Intel 730 480GB (9SSDSC2BP480G4R5) + Western Digital Black WD1003FZEX 1TB> M-Audio 192 sound card + Logitech z2300 2.1 speakers > 600W PSU> > Windows 7 Professional 64-bit >> 2X Dell Ultrasharp U2715H (2560 X 1440)
[ Passmark Rating = 5581 > CPU= 14046 / 2D= 838 / 3D= 4694 / Mem= 2777 / Disk= 11559] [6.12.16]
[Cinebench R15 > CPU = 1031cb / Single Core = 142 cb / OpenGL= 127.39 fps / MP Ratio = 7.24x] 3.2.17
[FryBench: 3:24 / Efficiency 2177.13] 3.11.17