News Nvidia Makes 1,000% Profit on H100 GPUs: Report

[Kamen Rider Blade]

The absolute LAST thing we need is Big Government bureaucrats in Washington DC and Brussels meddling into pricing mandates of the tech industry. A bunch of people who couldn't tell you what a video card does in a computer if their lives depended on it. Nothing would smother innovation faster. We vote with our wallets.

Speaking of which, the more I read stories like these as a long time Team Green person, the more angry I get. Specifically around how Nvidia's strong arming vendors almost certainly pushed EVGA out of the GPU market with the overpriced 4000 series - a vendor that had been a partner of Nvidia since 1999 mind you. That's unforgivably egregious to me and I'll speak with my wallet next time.

My overpriced 3080 Ti will be due for an upgrade next year after 3 years of ownership (that I was lucky enough to pay MSRP for in 2021 after winning a NewEgg shuffle purchase for one). It barely runs for my 4K needs now at solid 60FPS/60Hz locked in games like MSFS with high/max quality bar sliders. Nvidia has pushed me out as a 7x GPU customer since the Riva 128. My next GPU will be AMD. Here's to looking forward to 2024 and what AMD will bring on with their RDNA 4 RX series!

Welcome to Team Red =D

 

[Mattzun]

If either NVidia or AMD sold commercially viable AI cards for less than they do, there would either be scalping or other huge companies would just make more profit on AI. We should have learned this lesson from the crypo era

What could lower the price of GPUs is to do a refresh of the 3000 series using Samsung fabs. This doesn't use capacity that could be used for higher margin AI chips. A 3000 series refresh with 16GB 3070s and 20 GB 3080s would be nice. Unfortunately, PC gaming is such a small niche market that NVidia doesn't care.

Its really the same for AAA game development - most games are obviously designed for consoles and creating a PC port is a bit of an afterthought.

 

[vehekos]

That's incredible production of value.

 

[bit_user]

That's incredible production of value.

So was crypto currency & chips, until they weren't.

I'm not predicting a similar bust for AI, but just wanted to add some perspective. Value can be subjective and ephemeral.

 

[Jaxstarke9977]

Good for them. Businesses exist to make money. What are they supposed to do? Struggle and lose money?

 

[Jaxstarke9977]

This is more reason to "HEAVILY Regulate" them.

So success as a business is reason enough to regulate them?

 

[Kamen Rider Blade]

So success as a business is reason enough to regulate them?

To prevent a company from ripping off their customers.

Do you think 1,000% Profit on H100 is fair to the consumer / customer?

 

[Order 66]

4070 would never beat the 3090ti. Impossible for a 190w mid ranger to beat a prev gen card with absolutely no budget/specs/power limit. No next gen mid ranger would ever be designed to rival a prev gen 500w 1tb/s card. Just imagine 3090Ti was so off limits it required designing its own friggin connector. when gddr7 reaches 1tb/s on 256-bit, then xx90Ti will get 2-3tb/s on g7x

That's true, however, I was thinking that maybe it could have been possible since the 3070 was able to match or exceed a 2080ti at half the cost

 

[Peter Ferrari]

That's true, however, I was thinking that maybe it could have been possible since the 3070 was able to match or exceed a 2080ti at half the cost

2080ti and 3090ti are nothing alike.

 

[Order 66]

maybe the 5070 will beat the 3090 depending on how greedy NVidia is next-gen.

 

[Friesiansam]

Do you think 1,000% Profit on H100 is fair to the consumer / customer?

Remember this, Nvidia can only charge these prices, because there are enough customers prepared to pay them.

 

[Order 66]

Remember this, Nvidia can only charge these prices, because there are enough customers prepared to pay them.

which is exactly why nobody should buy the 40 series except for maybe the 4090.

 

[Peter Ferrari]

maybe the 5070 will beat the 3090 depending on how greedy NVidia is next-gen.

yeah. we'll see. 5070 will be gb205 128/192-bit gddr7 imo so nothing is for certain. it might lose at 4k.

which is exactly why nobody should buy the 40 series except for maybe the 4090.

qft. ad102 or nothing. I wouldn't mind a 13000-14000 cuda cut with 20G, but I'm not touching a 4080.

Do you think 1,000% Profit on H100 is fair to the consumer / customer?

Yes. It's not like they bought all cancer medication to resell it at 10x profit. No one needs AI cards to live.

 

[Order 66]

yeah. we'll see. 5070 will be gb205 128/192-bit gddr7 imo so nothing is for certain. it might lose at 4k.


qft. ad102 or nothing. I wouldn't mind a 13000-14000 cuda cut with 20G, but I'm not touching a 4080.

It would be great if AMD was as competitve with nvidia's GPUs as they are with Intel's CPUs, hopefully keeping nvvidia's prices competitive

 

[Lomskij]

Good for them, people can always buy Nvidia's shares to ride the AI bandwagon.

Having said that, H100 seems an incredibly overhyped product - as if you can't run inference on anything else, including CPUs. Don't get me wrong - H100 is an engineering marvel, scoring 4x higher in LLM benchmarks than 4090, despite having less raw power. However it costs 10x more, so from bang per buck perspective it's a rubbish product.

Regarding the government intervention - why? Like it or no, gaming GPU is a luxury product; would you want to regulate the prices of Chanel handbags as well?

 

[bit_user]

Having said that, H100 seems an incredibly overhyped product - as if you can't run inference on anything else, including CPUs. Don't get me wrong - H100 is an engineering marvel, scoring 4x higher in LLM benchmarks than 4090, despite having less raw power. However it costs 10x more, so from bang per buck perspective it's a rubbish product.

You're missing the point. People aren't buying them mainly for inference - they're buying them for training. And you can't train the kind of massive LLMs on 4090's that are all the rage.

I'm also curious about this 4x score you mention. I presume that's limited to models that even fit in the RTX 4090's memory? That won't get you very far, with GPT-3 or GPT-4 -level models.

Regarding the government intervention - why? Like it or no, gaming GPU is a luxury product; would you want to regulate the prices of Chanel handbags as well?

It's important that companies not abuse their market dominance. However, there are already laws on the books for that. I don't think simply charging several arms and legs for such a product crosses any red lines.

 

[Peter Ferrari]

Don't get me wrong - H100 is an engineering marvel, scoring 4x higher in LLM benchmarks than 4090, despite having less raw power.

space/power are expensive.
scalability is another reason a100/h100s stations sell.
plus you get a ready made solution, a complete station with support for all the AI software/framework you need.

 

[bit_user]

scalability is another reason a100/h100s stations sell.
plus you get a ready made solution, a complete station with support for all the AI software/framework you need.

Their main value proposition is about scalability - to multiple GPUs, machines, and even racks. For training huge models, you really need that.

On any other axis, the H100's price just doesn't make sense.

 

[vehekos]

Do you think 1,000% Profit on H100 is fair to the consumer / customer?

Absolutely yes.
The customer is voluntarily willing to pay for it, so it is fair.

If the customer pays for it, is because he values the hardware more than the money he's paying.

The real question is who should get the hardware: you, to play a game, or somebody who thinks he will create more wealth than the cost of the hardware?

Is like a piece of wood. Who should buy it? You may buy the wood, to make a barbecue, and profit 1$. Somebody else may buy the wood, to make a house, and profit 1000$

Society is better by giving the wood to the house builder, because it gives back to society 1000$ more than the cost, and you give back only 1$

The price informs that the good is better used by somebody else, and if somebody else gives back to society 1000% more than you do, they should get the good, and not you.

Nvidia does the right thing by selling it to the best bidder. You are the selfish greedy one. You want the wood, even if it costs a house to the rest of society.

 

[bit_user]

Nvidia does the right thing by selling it to the best bidder. You are the selfish greedy one. You want the wood, even if it costs a house to the rest of society.

So many people seem to think this is zero-sum, and I don't blame you for addressing that question because it's certainly the perception.

However, I read news about new packaging operations being setup, more interposer substrate being ordered, and manufacturers ramping up HBM production, and I'm left to wonder if the bottleneck on H100 production is really the TSMC 4N wafer supply, itself. I haven't followed the story that closely, but it's not obvious to me that TSMC doesn't currently have enough wafer capacity to supply products for both markets.

 

[Peter Ferrari]

Their main value proposition is about scalability - to multiple GPUs, machines, and even racks. For training huge models, you really need that.

On any other axis, the H100's price just doesn't make sense.

yup.
we all know nvidia have them huge racks witing for you as long as you're paying.

So many people seem to think this is zero-sum, and I don't blame you for addressing that question because it's certainly the perception.

However, I read news about new packaging operations being setup, more interposer substrate being ordered, and manufacturers ramping up HBM production, and I'm left to wonder if the bottleneck on H100 production is really the TSMC 4N wafer supply, itself. I haven't followed the story that closely, but it's not obvious to me that TSMC doesn't currently have enough wafer capacity to supply products for both markets.

wonder how long before nvidia starts building their own fab for their datacenter/ai lineup and only do geforce desktop/mobile + quadro on tsmc/samsung. it's inevitable imo, but will only happen after they introduce the copa design for compute-oriented products.

 

[thestryker]

Absolutely yes.
The customer is voluntarily willing to pay for it, so it is fair.

If the customer pays for it, is because he values the hardware more than the money he's paying.

The real question is who should get the hardware: you, to play a game, or somebody who thinks he will create more wealth than the cost of the hardware?

Is like a piece of wood. Who should buy it? You may buy the wood, to make a barbecue, and profit 1$. Somebody else may buy the wood, to make a house, and profit 1000$

Society is better by giving the wood to the house builder, because it gives back to society 1000$ more than the cost, and you give back only 1$

The price informs that the good is better used by somebody else, and if somebody else gives back to society 1000% more than you do, they should get the good, and not you.

Nvidia does the right thing by selling it to the best bidder. You are the selfish greedy one. You want the wood, even if it costs a house to the rest of society.

This is a shockingly ignorant post that doesn't seem to grasp how the real world works.

The price in this circumstance is because there's literally no other choice at that level which means nobody is "choosing" this so much as hedging their bets that it will pay off. The only choice they're actually making is that not buying it would be worse.

The companies buying this hardware aren't doing it for the betterment of anything but their wallets. It's not like we're talking about putting together supercomputers or specialized research hardware. This is just a crypto-esque money grab where rich people are looking to become moreso.

You say society, but I hate to break it to you society doesn't benefit from behavior like this. With profiteering the only people who end up actually benefiting are shareholders and the business itself. So disproportionately benefiting the wealthy is all you end up doing which is a net loss for society. Perhaps if there were guardrails preventing the hoarding of wealth you'd be onto something, but there isn't so you aren't.

 

[thestryker]

So many people seem to think this is zero-sum, and I don't blame you for addressing that question because it's certainly the perception.

However, I read news about new packaging operations being setup, more interposer substrate being ordered, and manufacturers ramping up HBM production, and I'm left to wonder if the bottleneck on H100 production is really the TSMC 4N wafer supply, itself. I haven't followed the story that closely, but it's not obvious to me that TSMC doesn't currently have enough wafer capacity to supply products for both markets.

From what I've been able to gather this isn't an either or situation so much as this generation of consumer cards aren't selling as well as past generations and they can sell every H100 they can make. I believe nvidia is likely also hedging against what happened with crypto where they were left holding the bag on a lot of high end. I wouldn't be surprised if their manufacture volume strategy was to just keep up with demand so they don't fall further behind, but also don't necessarily catch up either.

HBM stacks have also gotten higher, and with Intel using HBM successfully on the Max CPUs I think we're going to see it used in more places.

 

[JamesJones44]

Without direct access to that data (which is a closely guarded secret) it's hard to know where the profits actually end up. But the profit ratio claim is sourced within the first paragraph.

We do actually know what Nvidia makes in general terms. Since Nvidia is a public company we know from their earnings that profit margin last quarter was 28%, while their gross is 68%. Sadly, they don't break out hardware from services so it's hard to get a pure reading on those how much hardware itself contributes to those numbers. However, what it does tell you is that unless Nvidia loses money hand over fist on software, which is likely impossible, their margin on hardware while good aren't even close to 1000%.

I would bet large money in Vegas that the 28% margin is propped up by software services since those traditionally have much higher profit margins than hardware. However, even if we give Nvidia the benefit of the doubt their and say they earn 35% margin on hardware sales, that means there cost of hardware with everything factored in is still less than half of that 1000% number.

 

[Peter Ferrari]

their margin on hardware while good aren't even close to 1000x.

1000% is 10x.
Elementary school mathematics.
100% is one. 100%+100%+100%+100%+100%+100%+100%+100%+100%+100% is ten.
That'll be 15 dollars.