News Jim Keller suggests Nvidia should have used Ethernet to stitch together Blackwell GPUs — Nvidia could have saved billions

[Admin]

Jim Keller advises Nvidia to use Ethernet to stitch two Blackwell GPUs in GB200.

Jim Keller suggests Nvidia should have used Ethernet to stitch together Blackwell GPUs — Nvidia could have saved billions : Read more

 

[ezst036]

Nvidia certainly has become the poster child for vendor lock-in.

Putting Apple for a run for its money.

 

[hotaru251]

It would have also made it a bit easier for those customers to migrate their software to different hardware platforms, which Nvidia doesn't necessarily want.

nvidia likely did it specifically for that reason.
even if it was cheaper to do it another way they want you locked into their stuff forever.

Putting Apple for a run for its money.

if that leaked memo a while ago was anything to go by Jensen entirely wants to be like Apple.

 

[The Historical Fidelity]

nvidia likely did it specifically for that reason.
even if it was cheaper to do it another way they want you locked into their stuff forever.

if that leaked memo a while ago was anything to go by Jensen entirely wants to be like Apple.

I feel like companies like Apple and Nvidia are paving the way towards new Anti-Trust law interpretation. In that they are building roadblocks against another competitor having the chance to compete and win over the customer, thus creating a de facto monopoly in a certain market. Only the anti-trust case must be looked at on a per-capita basis being locked into the company’s ecosystem instead of simply from the classical “lack of competitor in the market-place” basis.

Ex: if Nvidia sells to 80% of the datacenter customers, and let’s say Intel or AMD come out with superior products incompatible with Nvidia’s required secondary hardware/connectivity/etc., if the only reason Intel and AMD cannot win over any part of the Nvidia 80% is because customers state they had to buy expensive proprietary Nvidia secondaries to use their current Nvidia hardware and abandoning Nvidia would incur losing all the investment in proprietary Nvidia secondaries, then that is a “DeFacto Monopoly” on 80% of the market, not because Nvidia is more competitive, but because of an artificial “ball and chain” arresting customers. All while their competitors offer better products using open source secondaries.

Maybe I’ve had too much coffee, but when I read this article, this idea popped in my head and I rolled with it lol.

 

[hotaru251]

with superior products incompatible with Nvidia’s required secondary hardware/connectivity/etc

I mean they do go out of there way to do so.
https://github.com/vosen/ZLUDA was made that let AMD gpu's (some not all) run CUDA on non nvidia gpu's...nvidia updated their licensing agreement to make that agasint the terms of use.

That "could" potentially be used in a case if it ever got to that point.

 

[thisisaname]

Was the next generation of Ethernet out when they where designing it?

 

[Alvar "Miles" Udell]

Keller contends this could have saved Nvidia and users of its hardware a lot of money.

Except it wouldn't save users any money, it'd just increase nVidia's profit margin if what he says is true.

 

[ezst036]

if that leaked memo a while ago was anything to go by Jensen entirely wants to be like Apple.

I agree. Though, to be fair, it is always the leaders who take the arrows. Nvidia and Apple respectively, are the unrefuted leaders in their domains.

The most hilarious thing I think though is how Apple for years refused to have any dealings whatsoever with Nvidia, only supporting AMD(ATi) video cards throughout the decade, beginning around 2010. Nvidia was entirely outmoded in Mohave.

Apple's schemes of vendor lock-in found a hard deadlock against Nvidia's schemes of vendor lock-in. hehehe

 

[edzieba]

Was the next generation of Ethernet out when they where designing it?

No. And back when NVLink was developed, Ethernet was still limited to 10 Gigabit (the 25 Gigabit alliance had not even formed yet) and lacked the latency and coherency requirements for inter-chip communication.

The headline could effectively read "Jim Keller suggests Nvidia could reduce chip costs using time travel".

 

[hotaru251]

Nvidia and Apple respectively, are the unrefuted leaders in their domains.

Apple isn't.
Apples a brand more than unmatched products.

Apple for years refused to have any dealings whatsoever with Nvidia

not really. that bad blood was due to business co-op blame game and made total sense.

 

[AmazingGoose]

Blackwell NVLink supports 1.8TBps. Is there an Ethernet equivalent to 14.4Tbps? Even with some form of bonded or multichannel link, I'd be surprised if such an Ethernet solution exists.

 

[JTWrenn]

This feels like a comparison of a first gen tech to a very mature one. Nvidia has a bad habit of going only custom and in house....but it is rarely not really good tech. This was not an investment in blackwell...it was an investment in their next 4 gens of ai chips.

 

[derekullo]

Blackwell NVLink supports 1.8TBps. Is there an Ethernet equivalent to 14.4Tbps? Even with some form of bonded or multichannel link, I'd be surprised if such an Ethernet solution exists.

Infiniband currently supports up to 12 - 200Gbe bonded for 2.4 TbE with plans for doubling that to 4.8Tbps with 12 - 400Gbe.
https://en.wikipedia.org/wiki/InfiniBand#Performance

Highest I can see for Ethernet is 8 - 200Gbe bonded for 1.6 TbE
https://en.wikipedia.org/wiki/Terabit_Ethernet#1.6T_port_types

 

[CmdrShepard]

Nvidia certainly has become the poster child for vendor lock-in.

Nobody is forcing you to use CUDA on NVIDIA hardware -- it's a value-add for customers (and a great value at that). People can dis it and hate it but that's all sour grapes talk.

I feel like companies like Apple and Nvidia are paving the way towards new Anti-Trust law interpretation. In that they are building roadblocks against another competitor having the chance to compete and win over the customer, thus creating a de facto monopoly in a certain market.

I feel like people today want everything someone else invents, designs, makes, and markets for free. Not only that, but they also want to dictate what others will support and how much they will charge for it.

Case in point -- Epic games lawsuit against Apple.

TL;DR version of that case -- Epic wants the benefits of:

- Well designed developed, and documented software platform
- Good distribution system
- Large user base with a lot of money
- Great visibility

But they don't want to pay a dime to Apple for any of that.

Nobody ever answered a straight question "if you don't agree 30% is fair, how much do you think it should be?" (not counting answers of 3% which is just card processing fees which don't cover the cost of iOS maintenance and development, store curation, notarization, digital distribution, not to mention all the free apps whose distribution has to be sponsored somehow).

Same thing is happening here with CUDA -- NVIDIA has poured a metric sh*t-ton of money and developer hours into creating CUDA to benefit customers using their hardware just like Apple did with iOS and app store to make the hardware (a phone) more appealing.

To top it off, both companies have solid if not great developer support and developers are naturally drawn to that. To try to ascribe malice to someone offering a well rounded product is asinine.

... not because NVIDIA is more competitive, but because of an artificial “ball and chain” arresting customers.

But NVIDIA IS more competitive. They offer more value for money. If I have to pay the same for top of the line AMD and NVIDIA card I am getting NVIDIA because I am getting CUDA, NPP library, NVJPEG library, NVIDIA Video SDK, Iray support, Physx, Visual Studio integration, performance profiler software, etc, etc. The fact that you don't care about those things doesn't make AMD a better deal just because they use and/or favor open-source.

For AMD, software and developer support wasn't even an afterthought -- they were primarily a hardware company. While they were selling Athlon chips Intel was working on their own C++ compiler and performance libraries. People were poking fun at Northwood and Prescott, yet those chips were used in professional settings (medical, etc) and were giving great performance with properly optimized software. Developers like me were writing optimized code for Intel, not for AMD because AMD didn't care about software at all. Their mantra was "with our CPU you don't need to optimize software" and that created a toxic mindset in the software industry, undoing of which took years.

Same goes for ATI -- Radeons were competitive with NVIDIA hardware of the time, but drivers and developer support were atrocious. Only now they are doing what NVIDIA is doing, but now they aren't first and it's too little, too late.

All while their competitors offer better products using open source secondaries.

So basically you are praising those competitors for socializing the costs and privatizing the profits through leeching from open-source instead of investing their own money into development of a superior product?

Maybe I’ve had too much coffee, but when I read this article, this idea popped in my head and I rolled with it lol.

You definitely had too much of something, and Jim Keller seems to have the same supplier.

There is a reason why Jim Keller, a legendary CPU designer and chief executive officer of Tenstorrent, an Nvidia rival, suggests that Nvidia should have used Ethernet instead of proprietary NVLink.

Yes there is, it's because he's apparently barking mad yet the press is singing odes to him as if he is at the minimum the second coming of Christ.

Why nobody ever mentions say Richard Sites or Rich Witek who made considerably more significant contributions than Jim Keller can ever hope is beyond me. Given his reckless statements Jim Keller looks more and more like a techbro instead of a serious engineer and techbros are associated in my head mostly with fraud, grift, and attention seeking.

 

[The Historical Fidelity]

Nobody is forcing you to use CUDA on NVIDIA hardware -- it's a value-add for customers (and a great value at that). People can dis it and hate it but that's all sour grapes talk.

I feel like people today want everything someone else invents, designs, makes, and markets for free. Not only that, but they also want to dictate what others will support and how much they will charge for it.

Case in point -- Epic games lawsuit against Apple.

TL;DR version of that case -- Epic wants the benefits of:

- Well designed developed, and documented software platform
- Good distribution system
- Large user base with a lot of money
- Great visibility

But they don't want to pay a dime to Apple for any of that.

Nobody ever answered a straight question "if you don't agree 30% is fair, how much do you think it should be?" (not counting answers of 3% which is just card processing fees which don't cover the cost of iOS maintenance and development, store curation, notarization, digital distribution, not to mention all the free apps whose distribution has to be sponsored somehow).

Same thing is happening here with CUDA -- NVIDIA has poured a metric sh*t-ton of money and developer hours into creating CUDA to benefit customers using their hardware just like Apple did with iOS and app store to make the hardware (a phone) more appealing.

To top it off, both companies have solid if not great developer support and developers are naturally drawn to that. To try to ascribe malice to someone offering a well rounded product is asinine.

But NVIDIA IS more competitive. They offer more value for money. If I have to pay the same for top of the line AMD and NVIDIA card I am getting NVIDIA because I am getting CUDA, NPP library, NVJPEG library, NVIDIA Video SDK, Iray support, Physx, Visual Studio integration, performance profiler software, etc, etc. The fact that you don't care about those things doesn't make AMD a better deal just because they use and/or favor open-source.

For AMD, software and developer support wasn't even an afterthought -- they were primarily a hardware company. While they were selling Athlon chips Intel was working on their own C++ compiler and performance libraries. People were poking fun at Northwood and Prescott, yet those chips were used in professional settings (medical, etc) and were giving great performance with properly optimized software. Developers like me were writing optimized code for Intel, not for AMD because AMD didn't care about software at all. Their mantra was "with our CPU you don't need to optimize software" and that created a toxic mindset in the software industry, undoing of which took years.

Same goes for ATI -- Radeons were competitive with NVIDIA hardware of the time, but drivers and developer support were atrocious. Only now they are doing what NVIDIA is doing, but now they aren't first and it's too little, too late.

So basically you are praising those competitors for socializing the costs and privatizing the profits through leeching from open-source instead of investing their own money into development of a superior product?

You definitely had too much of something, and Jim Keller seems to have the same supplier.

Yes there is, it's because he's apparently barking mad yet the press is singing odes to him as if he is at the minimum the second coming of Christ.

Why nobody ever mentions say Richard Sites or Rich Witek who made considerably more significant contributions than Jim Keller can ever hope is beyond me. Given his reckless statements Jim Keller looks more and more like a techbro instead of a serious engineer and techbros are associated in my head mostly with fraud, grift, and attention seeking.

I find your comments shallow and pedantic. I say good day to you sir!

 

[CmdrShepard]

I find your comments shallow and pedantic. I say good day to you sir!

I am not even sure what you mean by that.

 

[The Historical Fidelity]

I am not even sure what you mean by that.

Well can’t help you there bud

 

[Pete Mitchell]

I find your comments shallow and pedantic. I say good day to you

Your mother was a hamster and your father smelled of elderberries.​

 

[slightnitpick]

I find your comments shallow and pedantic. I say good day to you sir!

I am not even sure what you mean by that.

Here's a possible example (I basically stopped reading at this point because it conflicted with my beliefs on the free market):

I feel like people today want everything someone else invents, designs, makes, and markets for free. Not only that, but they also want to dictate what others will support and how much they will charge for it.

Case in point -- Epic games lawsuit against Apple.

TL;DR version of that case -- Epic wants the benefits of:

- Well designed developed, and documented software platform
- Good distribution system
- Large user base with a lot of money
- Great visibility

But they don't want to pay a dime to Apple for any of that.

All Epic wants is #3, the large user base. And that is not a value-add from Apple per se, it is a value-add from users who purchased from Apple. A side-loaded Epic Store can run on basically any platform (and is ultimately intended to do so), is its own distribution system, and only has the visibility it itself is able to get by encouraging iPhone users to side-load the store.

Now if you're just talking about the games that Epic distributes over the Apple Store, then Epic does pay Apple through direct app purchases. But once Epic (not Apple, re: your "and markets for free" above) has convinced the user to download the games, then for all in-app purchases Apple is adding no additional value.

In the case of in-app purchases it is Apple leveraging their monopoly over iPhone users to dictate to app developers what will be supported and what cut of the price they will charge for it.

Hobson's choice is not a free market. If you believe in monopolies beyond IP rights then that is your right, but you can't expect people in countries with free market laws to just accept your statements as to what should be the case. If Apple doesn't like functioning in a free market state then it can choose to do its business elsewhere.

 

[The Historical Fidelity]

Your mother was a hamster and your father smelled of elderberries.​

I fart in your general direction!

 

[ezst036]

Here's a possible example (I basically stopped reading at this point because it conflicted with my beliefs on the free market):

All Epic wants is #3, the large user base. And that is not a value-add from Apple per se, it is a value-add from users who purchased from Apple. A side-loaded Epic Store can run on basically any platform (and is ultimately intended to do so), is its own distribution system, and only has the visibility it itself is able to get by encouraging iPhone users to side-load the store.

Now if you're just talking about the games that Epic distributes over the Apple Store, then Epic does pay Apple through direct app purchases. But once Epic (not Apple, re: your "and markets for free" above) has convinced the user to download the games, then for all in-app purchases Apple is adding no additional value.

In the case of in-app purchases it is Apple leveraging their monopoly over iPhone users to dictate to app developers what will be supported and what cut of the price they will charge for it.

Hobson's choice is not a free market. If you believe in monopolies beyond IP rights then that is your right, but you can't expect people in countries with free market laws to just accept your statements as to what should be the case. If Apple doesn't like functioning in a free market state then it can choose to do its business elsewhere.

CmdrShepard does make many valid points. But with Epic vs Apple, there are no good guys. Epic's greed is only matched by Apple's greed, which is only then matched by Epic's greed and back to Apple greed again. I would have a lot more sympathy for Epic if they copied Valve. But Epic didn't do that. So their entire schtick is suspect.

While Epic vs Apple was going on, it allowed Valve to take a leadership position and make Linux now into the second best for gaming, wheras before it wasn't even a distant third. It was simply a joke. Let Apple keep fighting Epic, all it shows is that Apple is out for Apple first, and any attempts they make that might make things better for gaming are only so that they actually benefit Apple first, not the realm of games and the people who play them.

Apple for Apple. And Epic for Epic. They're not out for us, otherwise they would've put their game launcher store on Linux too like Valve did with Steam.

It's only a growing category. Linux gaming. What exactly would be stopping them? Companies that don't embrace openness and are only in it for sheer greed cannot be trusted.

 

[Findecanor]

I am concerned about the ethical issues of how this tech is used for so-called "AI": the massive power consumption, the massive copyright infringement and how it is disrupting the job market.

We are in fact in a climate crisis right now, and this industry is only fuelling the fire.

I wish Jim Keller's Tenstorrent instead focused on getting their rumoured "Ascalon" RISC-V CPU core out the door to consumers. If it is as good as they say, it should be competitive with Intel, AMD, Apple and Qualcomm, and we need good competition and diversity in the CPU space.

NVIdia are already guilty for having contributed to and profited off the crypto-currency bubble.

 

[Sleepy_Hollowed]

I mean, I can understand nvidia not using it if it did not meet all requirements of some initial heavy customers.

Ethernet is great, but some implementations are not great, and I can see them not wanting to spend money on Ethernet in-house if it did not meet all the requirements.

That being said, the main idea is good, it would be cross-compatible, and I think long term that'd be good at reducing costs for sure. They can even pitch engineering power to the standard.

 

[bit_user]

Nvidia's platforms use proprietary low-latency NVLink for chip-to-chip and server-to-server communications (which compete against PCIe with the CXL protocol on top)

No, that's not what CXL is. It doesn't sit atop PCIe! Rather, it shares a common PHY layer with PCIe, but they diverge at the protocol layer.

Also, CXL didn't support fabrics, initially. So, it's not accurate to say either PCIe or CXL are viable alternatives to NVLink... yet. Soon, however (I forget whether all the necessary features only arrive in CXL 3.0 or before).

some might say that Ethernet's capabilities don't cater to emerging AI and HPC workloads.

Well, Intel's Gaudi and Tenstorrent's newer products are both Ethernet-based. I'm sure I've read about other AI accelerators using it, as well.

the industry — spearheaded by AMD, Broadcom, Intel, Meta, Microsoft, and Oracle — is developing the Ultra Ethernet interconnection technology, poised to offer higher throughput and features for AI and HPC communications.

As far as I understand, they're just focusing on the software and protocol stack, but not the actual hardware. I believe their goal is to work on complementary developments to what the existing ISO/IEC working group is doing at the hardware level.

so Keller advises (or trolls) that Nvidia should adopt Ethernet.

His post is entirely self-serving. It's just trying to pitch Tenstorrent's use of Ethernet as an advantage. Whether or not it really is depends on the particulars, but I'm certain there are specific aspects that would count as "wins" in favor of NVLink.

For instance, if you want cache-coherency over Ethernet, you need to achieve that via an additional layer. NVLink is natively cache-coherent. I expect latency is another key benefit of NVLink. Given the way Tenstorrent accelerators work, I think these aren't big negatives, since dataflow architectures like theirs tend to be less latency-sensitive. Nvidia has to support a broader diversity of usage models, including ones where latency is very much a limiting factor.

 

[bit_user]

nvidia likely did it specifically for that reason.

When Nvidia developed NVLink, there was no viable open alternative. It's the same reason AMD and Intel developed their own GPU interconnect fabrics, as well.

However, Nvidia certainly dragged their feet on getting involved with CXL. I'm not sure if they've finally joined the working group, but they weren't for a long time.

Even now, there still isn't anything (in the wild) that has the same feature set and can provide equivalent performance to the latest iteration of NVLink. That's starting to change, however (see above post).