Need advice on scientific computing workstation built with 2xK20 Tes

Toggle sidebar Toggle sidebar
B

bliko

Honorable
Dec 3, 2012
10
0
10,510
Hello homebuild experts,
I have recently acquired 2 Tesla K20 NVIDIA GPU and I need to host them.

My computational needs are:
*) LINUX.
*) running my own neural network (NN) code in C - usually i have 10s or even 100s of different independent neural networks to run.
*) a lot of text processing for preparing the input and output data files (100 megabyte max each) using sed, awk, grep and other unix command line cmds. Preprocessing is glued together with perl or bash scripts.
*) Plotting data using R or gnuplot.
*) Maybe MySQL database but that can go on separate dedicated box.
*) No fancy graphic needs or video streaming etc.

The most intensive app is the NN code which I plan to port it to the GPUs.

What I have in mind is to use the GPUs for running the NNs and the CPU for text-processing, R or gnuplot as thus:

The pipeline (as a perl/bash script) starts on a single thread on the CPU, process the data files and writes them to temporary files to the hard disk or Linux ramdisk, then runs a NN by spawning portions of code to the GPU, etc. (not sure yet about this). When NN finishes, results are processed again on the CPU and hard disk, ploted etc. and the pipeline finishes. I can have many pipelines running in parallel on the available threads of the CPU.

I don't use commercial, closed-source software, meaning that my pipelines/code can utilise all the threads of my CPU by either running separate versions of the same software on each thread.

Alternatively, I will run a single C program using pthreads to spawn functions in parallel (i say this because a lot of CPU benchmarks run single-thread tests and multi-core processors do badly for their price) which will need to read huge shared memory and sending back small-size results using say MPI.

So in either way, I have high utilisation of threads on CPU and have portions of code which can go to GPU. I write my own C programs and maybe I can get compiler to use the extra AMD instructions for more efficiency.

Since I am constrained on money and space and watt, I would like to build a single workstation on a single motherboard hosting 1 or 2 or (max CPUs?) high-core cpus (e.g. AMDs FX-8350, 8cores, 155 uk pounds) and large memory (max RAM?)

CPU:
For the reasons above I would like CPU to give me lots of threads, and be cheap. I thought the AMD FX-8350 had price i could afford for what I got according to the benchmarks (e.g. http://www.cpubenchmark.net/high_end_cpus.html#cpuvalue). The Intels were ridiculously expensive (actually the AMDs ridiculously cheap).
Do any of you experts of homebuild systems have some serious objections about the choice of CPU? Would you prefer an Intel 4-core, say i5,
which is only 10% more expensive?

Based on the choice of the AMD FX-8350 (or your suggestion), can you please suggest:

1) MOBO: a lean, mean, cheap one which can host 2 NVIDIA GPUs (SLI may come handy) and ideally to have on-board simple GPU for the needs of the OS (Linux, no games, no fancy apps, etc.), simple network and sound cards, support for 2,3 3TB hard disks (say sata 6GB/s?) and top-of-the-range USB data transfers for adding external data storage if need be AND max data-transfers to the GPU cards. Is there a mobo to be able to host 2 AMD CPUs? (no overclocking, no fancy gimmicks for me, just fast transfers and max slots).

2) PSU: each K20 GPU has max power consumption of 225W. The AMD FX-8350 is at 125W. So depending on 1 or 2 CPUs, I think 850W or 1050W? Then XFX or Corsair Enthusiast depending on cheap deals at the time. Any objections/suggestions?

3) COOLING: the GPU cards come with their own fan. Do I need extra cooling you think? Will computer case standard fans (say 3, plus PSU's) be enough? I can install extra fan. Do I need water cooling? I do intend to run the system in hot climate, summer is about 35C=95F. Is water cooling cheaper at the end, more quiet and *safe*?

4) Hard disk, good for money i thought is the Western Digital 3TB RED 6GB/s at 130 uk pounds, but I am open to suggestions, it also depends on if any deals come up.

5) RAM: I would like to go for the maximum RAM I can slot on that board and be fast ram at that (1066?). Of course price is still an issue, but I can utilise all the RAM by creating linux ramdisks and storing temp files there.

Please tell me your opinion about using 2 or more cpus on the same board. I guess some of you may suggest a cluster of workstations also in view of the fact of acquiring an extra 2 GPUs, but then we get into other problems of power and space. (is there a high-end solution, e.g. servers, to this which is affordable?)

I live in the UK and usually buy from http://www.dabs.com

BTW, I saw one of you mentioning in another thread, overclocking errors will kill the scientific computing results, so no over-clocking for me.

Many thanks,

Bliko
 
how does your k20s come with a cooling fan? they are designed for servers where they have these massssive fans that spin at ridculously high speeds to cool them down. not too sure how well you can keep them cool

and if you somehow acquire k20s, which are around 2000 dollars apiece, why would you only spend 200 dollars on a CPU. you can easily run into a bottlenecking situation. and no, consumer hardware only lets you use one CPU at a time.

and if you are tight on power, it makes more sense to get intel. it uses much less power

id get this. but then you need a video card. the k20 does not have display outputs from what i know
http://pcpartpicker.com/uk/p/qNZt

or get this
http://pcpartpicker.com/uk/p/qO0T
 
Hi BigTroll, Thanks for your answer and the partpicker links.

how does your k20s come with a cooling fan?
Click to expand...
In the spec (http://www.nvidia.com/content/PDF/kepler/Tesla-K20-Active-BD-06499-001-v02.pdf) it states "NVIDIA has designed an active fan sink (Figure 8) to cool the GPU, memories and power components. For fan and environmental specifications refer to Table 6."

Ok that does not mean that it will be enough though... it consumes 200+W at top load.

The K20s were donated for a project at the institute I work. Then obviously I want to do the job as cheaply as possible, but of course to avoid bottlenecks is priority over money.

and no, consumer hardware only lets you use one CPU at a time.
Click to expand...
Sorry I don't understand this. Do you mean that in a dual-cpu motherboard from say Asus or Gigabyte, only 1 CPU is working at any one time? Is there an alternative? What about those Xeon Blade Servers?

and if you are tight on power, it makes more sense to get intel. it uses much less power
Click to expand...

Got that thanks!

Is there a specific CPU benchmark I should be focused on for evaluating cpus for this particular setting?

As far as the two builts you suggested at the end. That's great thanks. I have an idea now. Also reminded me about an SSD. only thing is to decided about 1 or 2 cpus on mobo and whether intel i7 cpu or cheaper fx-8350.

Thanks,
bliko
 
no. consumer hardware will not ever offer dual motherboards. and consumer CPUs are limited to only one CPU per a motherboard. you would have to use a amd opteron system or a intel dual xeon system. those have server level chipsets and will offer boards that have 2 sockets

wait, is your CPU going to be just purely running threads or are they going to do some of the compute work
 


CPU will do text processing and some other heavy staff which can't be ported to GPU. So will be used for number crunching too.

I guess I can get the ASUS Z9PA-D8 *2xsocket 2011(http://www.newegg.com/Product/Product.aspx?Item=N82E16813131886) fior 2xE5-2600 intels. Is that what you mean by 'non-consumer hardware'? or an opteron-based one.

 
that is a intel server board. you would need to buy xeons to run 2 CPUs at once.

consumer level hard ware such as the core i series are only limited to be run a single CPu configuration. nothing more

server hardware such as xeons from intel and opterons from AMD meanwhile can run in systems that contain more than one CPU
 
Thanks BigTroll,
that makes things much clearer. I did not know the distinction between commercial and consumer cpus. A system of 2 cpus obviously can benefit from shared memory (RAM) between programs/threads. But apart from that (if i don't need that i mean), it may cost less if I do 2 PCs with one cpu/gpu each, on a consumer hardware basis.

Alternatively there is the HP Proliant and Lenovo ThinkStation [!]
Thanks again.
 
id just get a x79 system with one 6 core CPU with one card. or you can get 2 systems each with one card. just that the second option would cost more but if you are making money from this, the cost can repay itself

yes but their system specs are pretty similar to what you can build yourself
 


indeed I can, but all this obfuscated specs scare me.

I am not making any money from all these, in fact i am paying from my pocket so that I have something at home to replace the SGI Altix UV1000 (4T shared memory, 1000 cores!) we have here at work and I am loosing from next year when my contract finishes. I will just have to pedal faster that's all [o].

as for the results of the number crunching ... maybe a big pharmaceutical can rip them off, they are all published and open-source you may say.
 


true say but we dont know exactly how demanding the work on the CPU will be.
 


Hi all,

I will need the CPU for 3 things
1) run programs whose main block is run on GPU
2) run text-processing programs for pre-post-processing data
3) run programs which can not be implemented on GPU - so far my main programs can be implemented on GPU, e.g. Neural Network, clustering, permutation tests, graph traversal.

So cpu power demand is not that much. Rather is the number of threats I can have to run the various things. For example usually I want to run a program N times with N different datasets,
so I am using GNU parallel to spawn M parallel text-processing threads (for M available CPU cores). Then some kind of manager will spawn the GPU section of the programs to the GP units and wait for results.

Text-processing and bash/perl scripts can benefit from fast CPU but the hard-disk overheads, launching programs, shells, awk, etc. is going to keep speed down, even if my CPU is superfast. To avoid HD bottlenecks, I will use ramdisks - i write all temp files now in ramdisks. Maybe this can create another bottleneck with memory but i think less serious.

To get an idea of what I mean by text-processing :
input file is a row-column ascii file (like an excel spreadsheet), separate columns 4 and 7,
calculate mean and stdev on these columns and then write them into another file,
the remove from that file all rows which are outliers, then ...
usually all done using awk, grep, sed, perl one liners etc.

Thanks guys,
 
if you want to run tons of ram, x79 is the way to go since it can handle 8 dimms worth of memory, 64gb at most

but then it does not exactly seem that you need that much CPU power. i would then suggest getting a i7 3820 instead of the 3930k.
 
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom