Question How to power 4x 3080s (NOT for mining)

[shmeckles]

I'm looking to build a rig with 4x 3080s, most likely using consumer motherboard and CPU (LGA 2066 or new LGA 1700 w/ PCIe 5.0). However, I'm not sure how to provide adequate power. I saw that I can purchase a 24-pin ATX splitter and use two PSUs in tandem, however, reading back on previous posts here say that it can be dangerous.

Since most of the posts that I saw were older (5+ years), I was curious if that was still the case. Or, are there high-end PSUs that would support such a system?

 

[USAFRet]

"How to power 4x 3080s (NOT for mining)"

OK, if not for mining, what IS the use case?

 

[shmeckles]

Primarily, in-house red team activities. We need PoC of password weakness for quarterly pentests that we will be running. In addition (and this is a rare occurrence), we have old archives that we have lost passwords to and need to crack. Traditionally, we have outsourced the cracking, but the amount that we have spent on this (all while trusting HIPAA files to a 3rd party) has equated to a greater cost than building a machine with 4x GPUs. So we would like to keep that in-house as well.

 

[USAFRet]

Primarily, in-house red team activities. We need PoC of password weakness for quarterly pentests that we will be running. In addition (and this is a rare occurrence), we have old archives that we have lost passwords to and need to crack. Traditionally, we have outsourced the cracking, but the amount that we have spent on this (all while trusting HIPAA files to a 3rd party) has equated to a greater cost than building a machine with 4x GPUs. So we would like to keep that in-house as well.

And all of that pentesting and 'password cracking' in no way relates to needing 4x 3080s in a single system.
Not from where I sit, anyway.

 

[shmeckles]

Well, feel free to debate the need for high-end GPUs with the folks over at Hashcat or any recognized PenTest company because I don't think I'd be able to change you mind.

That aside, are you not going to help me because you don't believe GPUs should be used in CyberSec?

 

[USAFRet]

Well, feel free to debate the need for high-end GPUs with the folks over at Hashcat or any recognized PenTest company because I don't think I'd be able to change you mind.

That aside, are you not going to help me because you don't believe GPUs should be used in CyberSec?

GPUs, sure. Massive processing power.
4x in a single system? Thats a whole other thing.

What other similar configs and systems have you found?
Mining or otherwise?

This WILL require more than one power supply, and the associated connections.

 

[DSzymborski]

Well, feel free to debate the need for high-end GPUs with the folks over at Hashcat or any recognized PenTest company because I don't think I'd be able to change you mind.

That aside, are you not going to help me because you don't believe GPUs should be used in CyberSec?

He's not judging, but knowing the use case is incredibly helpful to providing useful advice.

 

[USAFRet]

Even though this isn't a mining system, the hardware is similar.

 

[shmeckles]

What other similar configs and systems have you found?
Mining or otherwise?

Most password cracking rigs from dedicated pentest companies use upwards of 8 GPUs, such as this one. Here is one of the de facto 'what hardware you need' articles written by some of the best password folks out there that regularly give presentations at DefCon. Unfortunately, our company doesn't have budget for an 8x cracking rig.

I understand the suspicion for mining, but that's not what I'm trying to do. I'm simply trying to understand some things to build a rig that I was tasked with at work. And the current mining climate has made this quite frustrating. Help me or don't, but the constant scrutiny is...defeating.

 

[USAFRet]

Most password cracking rigs from dedicated pentest companies use upwards of 8 GPUs, such as this one. Here is one of the de facto 'what hardware you need' articles written by some of the best password folks out there that regularly give presentations at DefCon. Unfortunately, our company doesn't have budget for an 8x cracking rig.

I understand the suspicion for mining, but that's not what I'm trying to do. I'm simply trying to understand some things to build a rig that I was tasked with at work. And the current mining climate has made this quite frustrating. Help me or don't, but the constant scrutiny is...defeating.

And I just suggested looking at mining systems, for similar hardware and power requirements.

Same GPUs, etc.

Or, look at the pentesting operations that are already doing this.
You guys are not the first one.

What is everyone else already doing?

 

[shmeckles]

What is everyone else already doing?

Everyone else is using pre-built servers such as this one from SuperMicro, but they start at around $5,000 without GPUs. That's way over our budget, so I'm tasked with a DIY one which is why I'm here asking how to power a 4x 3080 rig.

So... Is using two PSUs in tandem via a splitter safe or is there a better way?

 

[shmeckles]

And I just suggested looking at mining systems, for similar hardware and power requirements.

I should have mentioned that mining systems are generally avoided for this specific use case. For example, some mining motherboards I've seen do support multiple PSUs, however, the boards are bad for my use case because they generally force all cards to x1 lanes, as well as don't support the amount of RAM that I need. I did, however, get the split PSU adapter idea from a mining blog. However, after reading more into it (specifically old threads on this site), I noticed that it carried risks. That's why I'm asking about it specifically.

 

[Juular]

If you power-limit them enough (or if the load isn't actually that close to 100% utilization per GPU) then one really high-wattage PSU like bqt DPP12 1.5kW, EVGA G2/P2/T2 1600W or Corsair AX1600i should be enough wattage wise (and you can just buy an extra cable set which would be cheaper than a whole PSU). But a 24-pin splitter and a second PSU is an option, sure, miners are doing the same thing after all. Or you can just jump the second PSU manually, if you don't startup-shutdown the system very frequently that would work just fine.

 

[tychotithonus]

[a password-cracking semi-necromancer appears]

Sounds like the OP already understands the differences with cracking workloads - which are, indeed, different from mining workloads. I'm not a hardware expert, and I haven't had to solve 30xx power constraints, but I can describe what has worked for me.

I've been running hashcat (and John the Ripper OpenCL as well) for a few years in a single open-air rig with 6x 1080s and 2 Corsair RM1000 PSUs and a 24-pin splitter. For whatever reason, the power aspects have been stable for me, but that may just be luck. I overbuilt it on purpose, and never get anywhere near maxing the PSUs out. The powered ribbon risers can sometimes be a little flaky every few months, but it's not that bad - reseating them usually resolves it. Some wordlist-heavy workloads are indeed slowed down by x1 PCIe, so if you can swing a minimum of x4, that's a good idea. And unlike most mining, you want to have as much system RAM as GPU memory, because hashcat reasons. I also recommend at least one core per GPU, with a little left over in case you want to be doing other kinds of wordlist processing while you have cracking jobs running. And even if the CPU is idle for your current attack, you can always throw hashcat CPU (or john, or even MDXfind) at those cores while hunting for other patterns to exploit. I've undersized CPUs on cracking rigs in the past - and always come to regret it.

Note also that you can throttle the cards to, say, 80 or 90% of their max power, and still have quite decent hashcat performance. atom (primary hashcat dev) did a study of this a while back and there is a sweet spot you can probably find with some testing. If ambient temperatures are stable, you can also lock the fans to 80% or something like that to combat fan-speed variability.

All that being said, you also have an entirely different option: just build two (or more) rigs, and team them with Hashtopolis. There are a few attacks that are trickier to formulate, but for broad scale-out, it's hard to beat for many of the standard attack stacks. And you get to entirely sidestep the "lots of GPUs in one rig" problem. It also makes it easy to temporarily scale out fast (via Vast.ai, etc.)