Oil submerged Servers

[fooltain]

I have several 8 core 8+ HD Servers running at my House, I'm a consultant and do a lot of VMware images for my different sandboxes, and I make the servers available to my clients at times, thus they need to be on 24/7.

That said the spare bedroom is LOUD as **** and rather warm, mainly warm because we keep the door shut because of the noise.

So my goal is not to overclock but to kill the noise, but you over clockers seem to be the experts in this area. <bows head to the experts>

I would like to know is how long will the MB, PS, and cables last submerged in say mineral oil, has anyone run one these longer than the http://www.pugetsystems.com/submerged.php#update5 system?

I wish I could afford the SSDs so they could be submerged, but that would be cost prohibitive for my data volumes.

So I think a hybrid approach would be my best approach, MB -Mother Board and PS Power Supply submerged and use blocks like http://www.frozencpu.com/products/7500/ex-blc-487/Koolance_HD-60_Hard_Drive_Liquid_Cooling_Block.html?tl=g30c203 to cool the HD - Hard Drives.

I have experience working with Acrylic, and enjoy working with it as a hobby.

So anyone out there know of a system that has been running for years?

 

[505090]

the only practical use for oil systems is to show off, the mess, leaks, upgrading, disposal, repairs,....

have you considered a window ac unit for the room or upgrading the loud parts for quieter ones

rack mount systems can run passive heatsinks...

 

[fooltain]

Those were my thoughts exactly about liquid cooling. My community won't allow a windows unit. They're all rack unit servers, I don't think I can passively cool them.

I'm not after show off, I have a real business use case here.

 

[505090]

portable ac perhaps

 

[slo]

I think i've read on these forums a post about a system that was submerged for years in oil but could only run for about 12-14 hours before the whole tank of oil got too warm and started overheating and shutting down. Am not sure wheather such a system would even work. If the servers are on 24/7 and must maintain stability at all costs i would not recoment submerging them becuase this kind of cooling is just too new to base such systems on.

A room AC unit would be the best option in combination with heatsinks but i have no idea on the power consumption of AC units on 24/7 basis.

Good luck and keep us informed

 

[fooltain]

My goal is elimante the noise, my systems are running around 38C. I can mount a radiator in the AC closet for a heat exchanger. Adding a portable AC to the room will only create more noise, plus with an AC you have to beable to drain the condensate. This is a spare bedroom, I'd like to be able to have a guest sleep in it. There is no way anyone can sleep or be in that room.

 

[slo]

what about proper liquid cooling?
I know blocks would cost a lot but you dont sound like you cannot afford it.

 

[jnjkele]

As I understand it (i've done a bit of research in the past on this) the biggest thing with the mineral oil is that it breaks down over time. The heat issue with the hardware can be solved by either putting the system in a big enough tank of oil, and circulating the oil with a submersible pump, or by running the oil through a heat exchanger similar to the ones used by liquid coolers. The oil would need to be changed at regular intervals (with 24/7 operation, probably no more frequently than every 6 or so months). The bigges problem is that PSU's cannot be submerged, as the line voltages on the A/C side (120V) don't react well with the mineral oil (unless someone has found a way around that). HDD's are also non-submergible (as you noted) as are optical drives. That said, the cabling gets complicated, but not un-doable.

I've never seen this done with mineral oil on this scale, but there are high voltage power systems that use proprietary submerged cooling for extended periods. I think they use something called flourinert, which if I remember right, is about 300- 400 $ per gallon.

Any way you slice it, though you will still have the heat problem as the same amount of heat will be rejected, and it has to go somewhere. Without something to move it out of the bedroom (i.e. room A/C unit) it will still get warm in there.

 

[fooltain]

I thought about that, and before I thought about the hybrid approach it was proper blocks or build a closet in the garage and add a ductless AC unit to it.

Using all blocks and no fans would cost a fortune.

http://www.frozencpu.com/products/9638/psu-422/Koolance_1000_Liquid-Cooled_Power_Supply_PSU-1000ATX-12N_Available_NOW.html?tl=g11c26

 

[fooltain]

The heat is not the issue, the AC closet is adjacent to the room. For the cost to block one system I can get the AC unit, and two systems I can build the closet.

I was trying to NOT block everything but the harddrives...I was just reading about the flourinet.

 

[slo]

Circulation would probably fix the problem but you can never be sure. I guess your buisiness is not something you want to risk. Submerging might work well or might fail epiccaly. As you said, a system (computer system we are talking of) of that scale has not been noted anywhere.
If you were some enthusiast who would build the submerged system on your sunday evening using the old hardware you have sitting there I'd surely encourage you BUT as i understand a large portion of what you do for living depends on this.

Also on a server if you leave out PSUs, hard drives and optical drives. this leaves out just the cpu to be cooled (and controller cards i'd guess).
I'd still suggest going for proper liquid cooling.

 

[fooltain]

I'm afraid you're correct, a hybrid system where the MB and its compnetns submerged and the Power supply would be optimal but I'm not going to risk it if no one has had success at running a submerged system for a long time.

Thanks for all of the input.

 

[Shadow703793]

Why not just swap out fans? A bunch of YateLoon, Scythe Slipstream fans should do the trick.

Also a little FYI:
HDD blocks are not needed. EVER. They actually do more harm than good.

Budget?

 

[fooltain]

I have 4 1U, 1 3U, 1 4U or 5U servers.

The 3U has 8 hard drives, and the 5U has 12.

The 4 1U are IBM, and the other two are Supermicro.

AC Ductless AC installed 3K, plus materials for the room $750 (I can build the room).
This way I could add a few more servers.

So rationaly I think I need to come in way under $3750 for all 6 machines.

 

[Shadow703793]

Mind listing more specs? esp. the CPUs and what your current set up (for cooling) is like?

 

[fooltain]

Lets do one that is really noisy...I used Belarc to pull this info:

Supermicro X7DB8

2.67 gigahertz Intel Xeon (2 installed)
16 kilobyte primary memory cache
2048 kilobyte secondary memory cache
64-bit ready
Multi-core (4 total)
Hyper-threaded (8 total)

Adaptec RAID 3805 Card

 

[Conumdrum]

Going AC is your best bet. Watercooling would be in the thousands, and you still have to remove the heat. Think of maintenance and upgrades, so much easier to use AC.

Your probably going to need a 3 ton AC unit (my house in Vegas has a 5 ton) It doesn't weigh 5 tons, but a 3 ton is pretty big still.

And the electric bill, whew!

 

[richardscott]

as there servers they wont have there cpu's upgraded like ever, so why not try WC but instead of water in the loop id recommend ethanol this way a leak wouldn't fry your server, as for costs a duel 120mm rad for each cpu a block and a pump shouldnt cost any more then £200 per server. im not sure if you can buy water blocks for cpu's that fit a 1u server tho you might be able to use elbow joints

 

[slo]

a good water cooling setup would cost anywhere from 300-400 per server (not per processor) depending on the coolong capacity required (no. of procesors, usage etc). I really cant see any way to go over 3000 dollars for 6 servers. You can probably get away with 2000-2500 and still using the best parts in the market

 

[jitpublisher]

You can buy an in room air conditioner at WalMart for about $500. It is free standing, plugs into a 110 outlet, and will do a fine job of keeping the room nice and cool. It has a dryer type hose coming out the back where it exhausts the hot air, that must be vented out a window is the only slightly tough thing about it. Other than that, you just wheel it in, plug it in, and you are set. They come with a vent attachment/adapter type deal that will fit into almost any type or size of window for the exhaust tube to hook onto.
http://www.stacksandstacks.com/windchaser-air-conditioner-with-remote-control-pacr9?id=176&sku=109408&utm_medium=shoppingengine&utm_source=googlebase
I have one similar to this to cool my house, and it works fantastic. I live in the northwest, so I don't need a whole lot of cooling, just something for the occasional hot summer day. I have about 1600 square feet, and it does a decent job. Won't freeze you to death by any means, but it keeps the whole house in the low 70's when it is in the upper 90's outside, and it is pretty easy on the electric bill. For a single hot room full of servers, this would be the ticket I would think.
Oil cooling has so many drawbacks, it is a novel idea at best reserved for those who want to do it, just to say they have done it. The mess, my god what a mess it will make. Can you imagine having to work on it? Can you live with stink of gallons of hot oil in your house?

 

[Houndsteeth]

Forget oil unless you are willing to spend an inordinate amount of money and time setting up silicone seals and buying not just standard food-grade mineral oil, but actual electrical-grade mineral oil. The problem with the oil idea is

1. Your systems have to be submerged

a) Because of the high viscosity of oil, using cooling blocks that rely on impingement (or turbulence) for maximum heat transfer to the fluid in a cooling loop would be extremely difficult, as pumps that can do this would be louder than fans.

b) Only the mainboard and peripheral cards can be submerged. Hard drives and optical drives cannot be submerged.

c) Transformer-grade mineral oil is dielectric by nature, and has no problems with any electrical current, despite what a previous poster said. It has been used since the beginning of electrification to good effect as a heat transfer medium. So putting power supplies in oil should not be a problem.

2. The oil has to be circulated and cooled. Remember, oil is just a heat transfer medium. You still have to take the heat out of the oil in the end.

a) Because of oil's high viscosity (compared to water), you cannot use standard water cooling pumps and expect to get the same MTBF.

b) Any seals in the loop must be made from silicone. Mineral oil will break down carbon-based plastic or rubber seals.

c) Because of the lower flow rate of oil (see point 2a) you will need to use low flow radiators, which will not be as efficient as water-cooled high flow rate radiators.

In this situation, you could go oil, but you would be spending a lot of money on a solution that would, in essence, be a lot more difficult to maintain that going with water cooling. Especially if you are able to create a central cooling reservoir for all your systems and keep them in a closed loop, you would see a much higher return going with water cooling. Worried about leaks? Yes, they happen, but if you use deionized distilled water, it not the big catastrophe that folks here will worry you about. Deionized water does not conduct electricity.

IMHO, don't do oil unless you have experience and knowhow. Instead, you can plan a water loop. For the lines outside the systems, you can plan to use 3/4" PVC pipe. Have a tap at each box, one for water flowing in, and the other for water flowing out. You can purchase no-flow quick-style connectors for a reasonable amount of money and pull any single box off the loop. At the end of the loop, you have a single reservoir tied to a single large radiator and low speed high volume fan. You would still have the noise, but it would be the noise of a single large pump and a large fan. You still have to buy water blocks for all the processors, but you do not need to buy multiple pumps and reservoirs. And you might even be able to hire a local plumber to come in and help you with the build if you find that you don't have enough time. Some might even be willing to help you design such a system just for the experience.

 

[Shadow703793]

Going AC is your best bet. Watercooling would be in the thousands, and you still have to remove the heat. Think of maintenance and upgrades, so much easier to use AC.

Your probably going to need a 3 ton AC unit (my house in Vegas has a 5 ton) It doesn't weigh 5 tons, but a 3 ton is pretty big still.

And the electric bill, whew!

Either way, OP is lokking in to the $2k+ range with any solution. Imo, it's best to get water as it cost MUCH more for a AC down the road (aka electric bill).

@OP: What temps on the CPUs at 100% load (check when running Prime95)? Depending on the temps you are currently hitting, you MAY be able to get a fanless heatsink like this one: http://www.frozencpu.com/products/9212/cpu-dyn-34/Dynatron_G118_1U_Server_Passive_Vapor_Chamber_CPU_Cooler_-_Intel_Xeon_5500_Series_Nehalem_1366_EP_Processors.html
I hear the veporchamber tech works quite well.

I was just reading about the flourinet.

You have NO idea how much that stuff costs do you?
http://www.parallax-tech.com/fluorine.htm#price


At any rate, start by replacing the current fans with some Yate Loon or Scythe Slipstream fans.

 

[orangegator]

From what I gather, the OP has 2 problems.

1. Heat build up in the room.
2. Noisy fans in the computers

Easy fix.

1. Portable AC unit in the bedroom.
2. Replace noisy fans with quieter ones.

 

[fooltain]

Noise is the number one issue, if we could open the door the room would be a lot cooler.

Most server cases less that 4U don't have fans on the heat sinks, neither do any of my systems.

The loudest server has 16 hot swap HD bays, and 2 Xeon CPUs. These systems have extremely high volume fans, and this system has triple redundant power supply.

I'm getting quotes on AC for a closet in the garage, and going with water and SSD drives. I may also replace 4 of the older machines with one newer machine and virtualize it.

FYI I ping the guys over at Puget systems, and they report that the original system is still running strong.

Depending of how well the visualization goes with the SSD, I may Ebay the old and get new.

I appreciate everyone's input.

 

[richardscott]

well im glad you have gone with our ideas it shows were not all stupi