News Nvidia's H100 GPUs will consume more power than some countries — each GPU consumes 700W of power, 3.5 million are expected to be sold in the coming...

[Admin]

Power consumption of AI processors continue to rise: Total power consumption of Nvidia's H100 processors.

Nvidia's H100 GPUs will consume more power than some countries — each GPU consumes 700W of power, 3.5 million are expected to be sold in the coming... : Read more

 

[Colif]

wow, couldn't that headline be made about almost any GPU sold in the last 5 years that uses over 700watts?

 

[coromonadalix]

+1

 

[ElisDTrailz]

Once again another terribly wrong article from this place. Does Tom's ever check sources? The average American household uses over 30kWh which is WAY more than 700 watts.

 

[DSzymborski]

Once again another terribly wrong article from this place. Does Tom's ever check sources? The average American household uses over 30kWh which is WAY more than 700 watts.

That's 30,000 watt-hours per day. The comparison with the GPU would be the theoretical of 16,800 watt-hours per day. The article exaggerated, but this also compared apples to hand grenades.

 

[Conor Stewart]

Once again another terribly wrong article from this place. Does Tom's ever check sources? The average American household uses over 30kWh which is WAY more than 700 watts.

It seems you don't know what you are talking about either. Did you really say 30 kWh is more than 700 W? The units aren't even the same, you can't compare them. kWh is a measure of how much energy is used, W is a measure of power.

This is the actual quote in the article, "At a 61% annual utilization, it is equivalent to the power consumption of the average American household occupant (based on 2.51 people/household).", so it isn't comparing it to full households, but to individual people and how much energy they use.

30 kWh divided by 2.51 people is 11.95 kWh and that figure is per day anyway. 700 W at 61 % utilisation for a day uses 10.24 kWh or at full utilisation uses 16.8 kWh.

So whilst a few statements in the article may be wrong the quotes they use and the general idea are correct.

 

[jp7189]

That's 30,000 watt-hours per day. The comparison with the GPU would be the theoretical of 16,800 watt-hours per day. The article exaggerated, but this also compared apples to hand grenades.

Not to be nit picky, but the article states 61% utilization, so that's more like 10kwh per day per H100 or 1/3 that of the average American household. That's a far cry from the assertion that H100 uses more than the average American household.

 

[jp7189]

It seems you don't know what you are talking about either. Did you really say 30 kWh is more than 700 W? The units aren't even the same, you can't compare them. kWh is a measure of how much energy is used, W is a measure of power.

This is the actual quote in the article, "At a 61% annual utilization, it is equivalent to the power consumption of the average American household occupant (based on 2.51 people/household).", so it isn't comparing it to full households, but to individual people and how much energy they use.

30 kWh divided by 2.51 people is 11.95 kWh and that figure is per day anyway. 700 W at 61 % utilisation for a day uses 10.24 kWh or at full utilisation uses 16.8 kWh.

So whilst a few statements in the article may be wrong the quotes they use and the general idea are correct.

The article source (which you're quoting) is accurate. Tom's summary is not. That's the point trying to be made. Here a direct quote from the very first line: "...each consumes up to 700W of power, which is more than the average American household."

Tom's left out "per person".

 

[DSzymborski]

Not to be nit picky, but the article states 61% utilization, so that's more like 10kwh per day per H100 or 1/3 that of the average American household. That's a far cry from the assertion that H100 uses more than the average American household.

Yeah, it's off. I was pointing out that the objection made things even worse, not better.

 

[H4UnT3R]

Well... now just count consumption of banks... and compare with this one... or BTC ;-) Love these nonsense articles by some Greta lovers (and Greta supports terroristm).

 

[HansSchulze]

Peak power is needed during very intense operations that are compute bound. It is hard to achieve full peak power consumption in most apps unless they are hand tuned. Some workloads like inference are more memory bound, so GPU isn't running as hard, and memory power is a small fraction of the GPU power. Likewise 60% utilization is pulling a rabbit out of a hat, server centers run multiple workloads which could put them up against the wall for 24/7 (efficient) or bursty web traffic (GPT?) which would reduce lifetime due to temperature fluctuations. If the 60% number was what the fraction of peak power, I would believe that - easy to misread or misquote sources.
Server centers like AWS, Azure are designed to find 24/7 workloads for all their expensive GPUs. But the GPUs aren't the most expensive part, as the article correctly alludes to. The larger part of the operating budget is electricity, so the primary goal is to use the most recent chips with the highest perf per watt. So imagine using V100 or K180 class supercomputers instead of H100, it would take similar power and take 10x or 50x longer.
Then, ask yourself what the goal is. Do we need to train 59 different AI models? No. Do we need a way to compile models incrementally? Yes. Pancake more layers of knowledge into them without starting over each time? Yes. Do we need to find ways to simplify their representation? Yes.
Source: I am NV alumni.

 

[slightnitpick]

Re: percent utilization
https://blog.dshr.org/2023/12/why-worry-about-resources.html#more

Note that this estimate assumes 100% duty cycle but ignores the power needed to cool the servers, two effects which tend to cancel each other.

The link is about the A100, not the H100, but presumably the cooling requirement would be similar.

 

[mdccohen]

Anton, what are you talking about.?? This is pure nonsense: “Nvidia literally sells tons of its H100 AI GPUs, and each consumes up to 700W of power, which is more than the average American household.” I don’t know where to start, but what is 700w greater than now????

 

[evdjj3j]

wow, couldn't that headline be made about almost any GPU sold in the last 5 years that uses over 700watts?

I knew who the author was as soon as I saw the headline. They need to replace this guy with AI.

 

[HansSchulze]

Re: percent utilization
https://blog.dshr.org/2023/12/why-worry-about-resources.html#more

The link is about the A100, not the H100, but presumably the cooling requirement would be similar.

Note that this estimate assumes 100% duty cycle but ignores the power needed to cool the servers, [I]two effects which tend to cancel each other[/I].
I read that original paragraph, and as a power engineer, I don't understand any of it. The rest of the website seems reasonable, but skips over a lot of details. For example, Google is using TPUs which are better power consumption for their search "AI" than GPUs. Otherwise Google wouldn't have built 5 generations of them.

Data center cooling tech has come a long way in the last few years, there are ways to recycle part of the heat and get some energy back. But cancel each other???

 

[Skramblr]

It seems you don't know what you are talking about either. Did you really say 30 kWh is more than 700 W? The units aren't even the same, you can't compare them. kWh is a measure of how much energy is used, W is a measure of power.

This is the actual quote in the article, "At a 61% annual utilization, it is equivalent to the power consumption of the average American household occupant (based on 2.51 people/household).", so it isn't comparing it to full households, but to individual people and how much energy they use.

30 kWh divided by 2.51 people is 11.95 kWh and that figure is per day anyway. 700 W at 61 % utilisation for a day uses 10.24 kWh or at full utilisation uses 16.8 kWh.

So whilst a few statements in the article may be wrong the quotes they use and the general idea are correct.

I show similar math
700W x 24hr * 30 days = 504kw for a month <-- assuming the full 600W quoted in the article.
Per EIA data, US households are about 885kW per month. Other sites show 500 to 1000 kW.
But overall, it still looks like the article is incorrect. Especially if we applied a 61% utilization to the GPU numbers, then the article looks even worse.

Note - I don't agree with the assessment of dividing the household power by 2.51 people. The article clearly states per household (not per person).
"each consumes up to 700W of power, which is more than the average American household. "

 

[slightnitpick]

Note that this estimate assumes 100% duty cycle but ignores the power needed to cool the servers, [I]two effects which tend to cancel each other[/I].
I read that original paragraph, and as a power engineer, I don't understand any of it. The rest of the website seems reasonable, but skips over a lot of details. For example, Google is using TPUs which are better power consumption for their search "AI" than GPUs. Otherwise Google wouldn't have built 5 generations of them.

Data center cooling tech has come a long way in the last few years, there are ways to recycle part of the heat and get some energy back. But cancel each other???

From what I understood: assuming 100% utilization would result in a greater power figure than actually happens, and ignoring cooling costs would result in a lesser power figure than actually happens, so the plus and minus here would "tend to cancel each other out" and get closer to the actual power used.