News Korean researchers power-shame Nvidia with new neural AI chip — claim 625 times less power, 41 times smaller

[Admin]

A team of scientists from KAIST detailed a new AI chip during the 2024 ISSCC. The new C-Transformer is claimed to be the world's first ultra-low power AI accelerator chip capable of LLM processing - and does so using 625 times less power than the Nvidia A100.

Korean researchers power-shame Nvidia with new neural AI chip — claim 625 times less power, 41 times smaller : Read more

 

[ekio]

Nvidia is the most overvalued company currently. They are rich because they lead temporarily in a very demanding market. Their solution is based on graphics processors doing an alternative job they were not fully intended for, full greedy premium price. Can’t wait to see competition bringing them back down to earth.

 

[derekullo]

Nvidia is the most overvalued company currently. They are rich because they lead temporarily in a very demanding market. Their solution is based on graphics processors doing an alternative job they were not fully intended for, full greedy premium price. Can’t wait to see competition bringing them back down to earth.

GPUs were initially developed for rendering graphics in video games but have become instrumental in AI due to their highly parallel structure. They can perform thousands of operations simultaneously. This aligns perfectly with the tasks required by neural networks, the critical technology in generative AI.

GPUs were never perfect for rendering graphics either ... that's why Nvidia releases newer more efficient models every other year.

 

[Unolocogringo]

Nvidia is the most overvalued company currently. They are rich because they lead temporarily in a very demanding market. Their solution is based on graphics processors doing an alternative job they were not fully intended for, full greedy premium price. Can’t wait to see competition bringing them back down to earth.

Maybe you do not understand the underlying technology of CUDA cores(NVIDIA)/SHADERS (AMD)?

Nvidia CUDA Cores Explained: How are they different?

Nvidia CUDA Cores and how are they different from CPU Cores and AMD's Stream Processors. Does having more CUDA Cores gives better performance?

www.techcenturion.com

https://www.amd.com/system/files/documents/rdna-whitepaper.pdf

Their are more, these are quick search primers.
They were already rich with their 80% (plus or minus a few points) gaming and professional graphics card market share.
It just so happens these highly parallel compute units work well for AI also.
They also work very well for Folding @ Home. Trying to solve humanities diseases/afflictions .

 

[edzieba]

claiming that the C-Transformer uses 625 times less power and is 41x smaller than the green team's A100 Tensor Core GPU

Or in other words, their chip is only ~6.5% of the power density of the A100. Since almost any even moderately high performing chip is power limited these days (CPUs, GPUs, even mobile SoCs now) their chip is either downclocked to hell for this particular my-number-is-biggest bullshotting exercise, or is just plain inefficient in terms of die usage - and with die area correlating to fab cost and fab cost dominating total cost, don't expect price/perf to be great.

 

[Amdlova]

Maybe now nvidia comes to gammer market with sorry indeed on mind.

 

[ekio]

The problem with Nvidia, is that their tech is relying on a single generalist GPU architecture rather than an extremely optimized GPU for graphics and an extremely optimized AI processor line.
When specialized processors are emerging, they will beat Nvidia like crazy.
Nvidia tripled their value thanks to mining then AI, they are just relying on trend where they are the easy solution, but they were never the best for long.

 

[DougMcC]

Nvidia is the most overvalued company currently. They are rich because they lead temporarily in a very demanding market. Their solution is based on graphics processors doing an alternative job they were not fully intended for, full greedy premium price. Can’t wait to see competition bringing them back down to earth.

Have you seen Tesla's valuation? They're valued as if they are not only the most valuable car company on earth, but also as if none of their competition exists.
Tesla: 500B valuation on <2M auto sales
GM: 50B valuation on > 6M auto sales.

 

[dlheliski]

Or in other words, their chip is only ~6.5% of the power density of the A100. Since almost any even moderately high performing chip is power limited these days (CPUs, GPUs, even mobile SoCs now) their chip is either downclocked to hell for this particular my-number-is-biggest bullshotting exercise, or is just plain inefficient in terms of die usage - and with die area correlating to fab cost and fab cost dominating total cost, don't expect price/perf to be great.

Exactly. This article is just hot garbage. 625 less power, 41 times smaller, ignoring the fact that it is 183 times slower. So TOPS/W is about 3X better, and TOPS/area is about 4X worse. No surprise, reduce Vdd and use more silicon. An idiot can do this on a bad day.

 

[hotaru251]

Maybe now nvidia comes to gammer market with sorry indeed on mind.

why?
They have no reason to. There are not enoguh ai chips in world to feed demand.


it sucks, but Nvidia has stated they arent a graphics card company anymore. Gaming will never become their focus again. There is sadly mroe $ in other stuff.

 

[bit_user]

At face value, that's 183x slower than the claimed 624 TOPS of the Nvidia A100 PCIe card (but the KAIST chip is claimed to use 625x less power).

So, it's only it's only 3.4 times as efficient as their previous generation GPU, which is less than 6 months away from being a two-generation old GPU???

Stopped reading right there. This is the epitome of a click-bait headline. It suggests equivalent performance at the stated size & efficiency. You should've either used metrics which normalize for the performance disparity or just gone ahead and actually included the performance data.

Korean researchers power-shame Nvidia with new neural AI chip — claim 625 times less power draw, 41 times smaller
​

Worse, the thing which makes A100 and H100 GPUs so potent is their training ability, which is something these Korean chips can't do. The article doesn't even mention the words "training" or "inference".

This "journalism" is really disappointing, guys.

 

[Alvar "Miles" Udell]

So, it's only it's only 3.4 times as efficient as their previous generation GPU, which is less than 6 months away from being a two-generation old GPU???

Stopped reading right there. This is the epitome of a click-bait headline. It suggests equivalent performance at the stated size & efficiency. You should've either used metrics which normalize for the performance disparity or just gone ahead and actually included the performance data.

​

Korean researchers power-shame Nvidia with new neural AI chip — claim 625 times less power draw, 41 times smaller​

​

Worse, the thing which makes A100 and H100 GPUs so potent is their training ability. The article doesn't even mention the words "training" or "inference".

This "journalism" is really disappointing, guys.

Like most articles written by "freelance writers" on this site lately, though at least this one cites their source.

Remember that this is a third generation product from KAIST, and I can't think of any big, or any at all, splashes their first two generations have made. And considering Google's Gemini Nano, a LLM which will run totally on-device, is inbound to Snapdragon and Exynos powered phones, and how this press release specifically mentions running LLMs on phones, I don't see how well this generation is going to go over either.

 

[usertests]

Good job parroting pointless numbers in the headline.

 

[watzupken]

The problem with Nvidia, is that their tech is relying on a single generalist GPU architecture rather than an extremely optimized GPU for graphics and an extremely optimized AI processor line.
When specialized processors are emerging, they will beat Nvidia like crazy.
Nvidia tripled their value thanks to mining then AI, they are just relying on trend where they are the easy solution, but they were never the best for long.

Strongly agree. Their GPU solutions so happen to work very well for such loads, and therefore, made their GPUs very desirable. But the truth is that they are not made/optimised specific for a use case, which is why it is super power inefficient, and will struggle to scale given the high power requirement. So their dominant position is actually very shaky in the mid to long term.

 

[bit_user]

Strongly agree. Their GPU solutions so happen to work very well for such loads, and therefore, made their GPUs very desirable. But the truth is that they are not made/optimised specific for a use case,

Really? If it's not an optimization, what do you call it when they added fp16 in the P100 (2016) or dp4a in the rest of the Pascal series (also 2016)? And then what do you call the Tensor cores they introduced in Volta (2017) and have been refining ever since? Or new data formats like BF16 and fp8? Also, what's with all the SRAM and HBM bandwidth they've been adding, in recent generations?

You guys are clearly in denial. The main reason Nvidia is owning AI is precisely because they got on board before most others and have been ahead of the game in optimizing their architectures to perform at scale like nobody else's. Furthermore, the generality of their programming model is a big asset while AI is evolving so rapidly, with new layer types and algorithms still emerging periodically.

which is why it is super power inefficient, and will struggle to scale given the high power requirement.

Dataflow architectures are no doubt better, but they require the hardware either to have quite substantial off-chip bandwidth or to scale with the networks. When dealing with networks having tens or hundreds of billions of parameters, the latter is probably a non-starter for most users. That's why Nvidia is on top, and not Cerebras.

So their dominant position is actually very shaky in the mid to long term.

It's funny to me how you seem to assume they can't design ever more streamlined architectures at least as fast as anyone else in the industry. They have the deepest pockets, the most experience, and can easily attract the best minds in the industry.

 

[derekullo]

The problem with Nvidia, is that their tech is relying on a single generalist GPU architecture rather than an extremely optimized GPU for graphics and an extremely optimized AI processor line.
When specialized processors are emerging, they will beat Nvidia like crazy.
Nvidia tripled their value thanks to mining then AI, they are just relying on trend where they are the easy solution, but they were never the best for long.

At some point relying on trend after trend for nearly a decade becomes more than just your competitors being too lazy to compete.
I'd imagine Nvidia is already working on dedicated AI hardware if it isn't out already for "important customers".
Any company that releases a product that challenged Nvidia in AI would most likely be acquired by Nvidia assuming it wasn't an already large company like Intel which would trigger antitrust issues.

 

[bit_user]

I'd imagine Nvidia is already working on dedicated AI hardware if it isn't out already for "important customers".

Years ago, they were already publishing papers about research chips they built to explore chiplet/tile-based architectures for AI.

https://www.tomshardware.com/news/nvidia-msm-inference-chip,39780.html
​

Intel also makes test chips, just to explore ideas. I read that Apple takes an interesting approach of prototyping new ideas into actual production silicon, but they make sure those features can be disabled without affecting the device's primary functionality. I have no idea what AMD does for prototyping - whether any of it is silicon-based or all via simulations.

Any company that releases a product that challenged Nvidia in AI would most likely be acquired by Nvidia assuming it wasn't an already large company like Intel which would trigger antitrust issues.

I'm not sure about that. Of all the recent acquisitions Nvidia has made, I can't think of one that was an actual competitor. They were all done to complement Nvidia's existing capabilities. ARM had a slight redundancy, in that their Mali iGPUs targeted some of the same markets as Nvidia's SoCs, but that was clearly incidental and not a serious competitive threat.

I do expect Nvidia to do some more acquisitions, while the share price is so high. It'll be interesting to see where they focus. My bet would be something robotics-oriented. Snatching up a couple AI software startups is probably a given.