News AI models that cost $1 billion to train are underway, $100 billion models coming — largest current models take 'only' $100 million to train: Anthro...

[Admin]

Anthropic CEO Dario Amodei says that AI model training costs could jump to $100 billion as early as next year.

AI models that cost $1 billion to train are underway, $100 billion models coming — largest current models take 'only' $100 million to train: Anthro... : Read more

 

[vijosef]

The deviation of so many resources towards AI will create a lot of scarcity and inflation.

 

[Notton]

"Amodei also added, "I think if we go to ten or a hundred billion, and I think that will happen in 2025, 2026, maybe 2027"

That's like saying "I predict Tom's will add another article about AI in the next 6hours to next year, and it'll have anywhere between 400 words and 40,000.

"I'm covering all my bases" -Nostradamus, probably

 

[thisisaname]

So, if AI models grow ten times more powerful each year, we can rationally expect the hardware required to train them to be at least ten times more powerful, too

That does not look like progress, they are just deploying more hardware and more expensive at that.

If there was progress then 10 times more powerful would need less than 10 times the hardware.

 

[DougMcC]

1B I believe. 10B I'm doubtful. I suspect people will rest on 1B models while they refine training techniques and optimize software. 10B is a big, big investment that will be much more challenging to recoup the cost. Being first in the race to be the best has some value, but AI models are already very close to 'good enough' for 99% of the tasks people have come up with to use them for. I'd be surprised if $1B models were not good enough for just about everything. CEOs are not going to want to plunk down 10B unless they are quite sure they are getting enough gain that it will pay off, and that will require quite a lot of research first. So my personal bet is on no $10B model until past 2030.

 

[bit_user]

The author neglects to consider the business aspect of this development, which another recent article covered:

https://www.tomshardware.com/tech-i...ai-bubble-intensify-in-wake-of-sequoia-report
​

As stated in the comments on that article, I think the methodology is way too simplistic and throws the findings into serious question. However, even if they're over by a factor of two, the underlying idea that these developments are driven by commercial incentives is indeed sound. That means it must translate into some combination of efficiencies and value-added services people are willing to pay more for, in amounts totaling such lofty sums. This problem is only compounded, if training costs indeed balloon by such amounts as Anthropic claims.

I think one aspect we shouldn't overlook is the cost of curating the training data. To avoid overfitting, you generally need a lot more training data than you have model parameters. I wonder how much of the training costs go into that.

 

[why_wolf]

"generative artificial intelligence (like ChatGPT) to artificial general intelligence" I see they're using easily (probably deliberately) confusable terminology to describe wildly different technologies. They really want the general public to think stuff like chatGPT is what the general public thinks of when they hear AI from decades of sci-fi.

I still question how they actually intended on making money from any of this. Just replacing the Google search engine with this and keeping all the ad-revenue to themselves may not be enough. You're radically increasing the cost of every "search" running it through these chatbots.

 

[bit_user]

They plan on causing a massive extension to welfare mixed with unemployment, aka ubi basically. To then mitigate jobs only the ai investors will offer (at a minimum wage at best).

That's exactly the idea. Remove independent workers, small business, pool resources and money into something nightmarish similar to Idiocracy.

That's one possible scenario, but the worst part might be that there is no master plan. Some people have ideas like this, but the truth is that there are lots of big questions and unknowns, yet progress continues unabated.

I'm skeptical of UBI (Universal Basic Income), since I'm not aware of any time in recent history that productivity gains have been shared out like that. Perhaps it will become a political necessity, but I just wouldn't assume it'll happen.

 

[kjfatl]

The bubble will pop and investors will lose billions.
Reminds me of the Dutch Tulip craze, or early investors in US Canals and Railroads. Commodore Vanderbilt did well, nearly everyone else lost.

When will CEO's' start asking the question, what is the ROI on this project?

In the meantime, for those who know when to bail, there is a lot of money to be made.
Remember, AI is supposed to get so good that AI software developers will no longer be needed.

 

[bit_user]

Reminds me of the Dutch Tulip craze,

No, tulips have no intrinsic value. You can't use them to do anything. All they are is pretty, and then they die. AI has already proven its worth, well beyond that.

When will CEO's' start asking the question, what is the ROI on this project?

I'm sure that's always a part of the discussion. For sure, those questions will continue to get sharper.

Remember, AI is supposed to get so good that AI software developers will no longer be needed.

There will continue to be a need for developers into the forseeable future. Their tools and job activities will change and maybe there won't be as many of them, but it's like saying that factory automation would eliminate all jobs in manufacturing. Didn't happen.

 

[DougMcC]

The bubble will pop and investors will lose billions.
Reminds me of the Dutch Tulip craze, or early investors in US Canals and Railroads. Commodore Vanderbilt did well, nearly everyone else lost.

When will CEO's' start asking the question, what is the ROI on this project?

In the meantime, for those who know when to bail, there is a lot of money to be made.
Remember, AI is supposed to get so good that AI software developers will no longer be needed.

ROI isn't hard for them at the current scale. Plenty of people willing to pay them the 50c per M tokens for gpt 3.5.
I'm super curious to see reports of how many people are willing to pay them the $5 per M on 4o. That will tell us a LOT about what the future looks like. My company is paying for loads of tokens on 3.5 but zero on 4o because the delta in capability isn't worth the price.
There are probably a tiny number of use cases that will pay $50 / M on gpt 5. Maybe enough to fund it.
But $500/M on GPT6? Who will need that?

 

[kjfatl]

No, tulips have no intrinsic value. You can't use them to do anything. All they are is pretty, and then they die. AI has already proven its worth, well beyond that.

The worldwide tulip market today is worth between $5 billion and $10 billion. AI is simply one tool used by companies to make money. The value is not in the tool, but the product developed using the tool. For the professional gardener, tulip bulbs are simply a tool.

I'm sure that's always a part of the discussion. For sure, those questions will continue to get sharper.

ROI is the only thing that matters long-term. If you can't pay back the investors after paying your employees and the power bill you have very little to offer in bankruptcy. Someone will pay 3 cents on the dollar for your assets and dispose of them appropriately.

There will continue to be a need for developers into the forseeable future. Their tools and job activities will change and maybe there won't be as many of them, but it's like saying that factory automation would eliminate all jobs in manufacturing. Didn't happen.

Developer jobs will never go away. For the past 50 years or so, the products have simply become more complex. Todays AI experts earning $500K will have a rude awakening in 10 years when every software engineering student has taken the 3 class AI curriculum as part of their undergraduate study. Those who continue to make money will be those who develop specialized tools where AI expertise is about 5% of what the job requires. It will be on your resume like C, C# C++, Python, Linux. Windows, Android. Just another tool.

 

[DonQuixoteIII]

This sounds very much like trying to make a wheel rounder.

 

[Kondamin]

The deviation of so many resources towards AI will create a lot of scarcity and inflation.

That already happened to memory and high end chips on modern nodes.
Now you will see a drop in demand as smaller scale projects become pointless which causes a drop in total demand.

 

[bit_user]

That already happened to memory and high end chips on modern nodes.

Well, DDR5 and NAND prices have ticked up a little bit, but they were somewhat artificially depressed until the end of last year.

Now you will see a drop in demand as smaller scale projects become pointless which causes a drop in total demand.

I think you're exaggerating. Got any examples of chips that have become markedly more expensive?

 

[weber462]

I think BOINC can now be used to train models with the introduction of "Sporadic Applications" in the newly released version 8. https://github.com/BOINC/boinc/wiki/Sporadic-Applications

 

[Kondamin]

Well, DDR5 and NAND prices have ticked up a little bit, but they were somewhat artificially depressed until the end of last year.


I think you're exaggerating. Got any examples of chips that have become markedly more expensive?

Those high end nvidia Chips like the H100 exploded from a bit above $1000 to over 10K
Just like HBM saw a significant increase in price.

 

[bit_user]

Those high end nvidia Chips like the H100 exploded from a bit above $1000 to over 10K

No, they went from like $18k to about $30k or more.

Just like HBM saw a significant increase in price.

Fair point. Anything needing HBM will be getting more expensive, until the DRAM industry can catch up with demand.

 

[kealii123]

The deviation of so many resources towards AI will create a lot of scarcity and inflation.

Nah, what will happen (as the article alludes to) is that we will finally start using the cheapest, potentially most abundant energy source we have: nuclear. People will quickly get over their hangups about it.

If aliens stopped by this afternoon, and saw us attempting to power our economy with windmills 80 years after we unleashed the power of the atom, they would pack up and leave thinking us a lost cause.

 

[bit_user]

Nah, what will happen (as the article alludes to) is that we will finally start using the cheapest, potentially most abundant energy source we have: nuclear. People will quickly get over their hangups about it.

If aliens stopped by this afternoon, and saw us attempting to power our economy with windmills 80 years after we unleashed the power of the atom, they would pack up and leave thinking us a lost cause.

Current nuclear power technology is a bridge fuel, at best. It's very expensive, since the reactors have to be overbuilt and you have to build storage facilities to warehouse the spent fuel for like 10k years (seriously). Furthermore, uranium is a finite resource and there's only a few more decades' worth left. I know there's been a lot of buzz about alternate fuels for reactors, and that's definitely interesting, even though I think their energy density is lower.

Windmills make sense primarily in relatively few locations that have strong and consistent winds. Solar is a much better option, though it's also somewhat location-specific.

I'm not anti-nuclear, but I think it's not a silver bullet. I'm a fan of "all of the above", when it comes to non- carbon-based energy generation.

 

[vijosef]

Nah, what will happen (as the article alludes to) is that we will finally start using the cheapest, potentially most abundant energy source we have: nuclear. People will quickly get over their hangups about it.

The cheapest, most abundant energy source we have, is still coal.

Germany Brings Back Mothballed Coal Plants to Help Keep Lights On​

 

[bit_user]

The cheapest, most abundant energy source we have, is still coal.

In the USA, coal is economically uncompetitive with natural gas. Coal also emits something like twice as much CO2 per unit of energy as natural gas, not to mention the particulate pollution.

Solar is getting quite cheap, as well, but possibly not competitive if you factor in the storage requirements necessary to use it as a 24/7 energy source.

 

[DougMcC]

In the USA, coal is economically uncompetitive with natural gas. Coal also emits something like twice as much CO2 per unit of energy as natural gas, not to mention the particulate pollution.

Solar is getting quite cheap, as well, but possibly not competitive if you factor in the storage requirements necessary to use it as a 24/7 energy source.

Solar has reached a key tipping point. It is now cheaper to deploy solar + storage than it is to build an equivalent coal plant.

 

[kealii123]

The cheapest, most abundant energy source we have, is still coal.

Germany Brings Back Mothballed Coal Plants to Help Keep Lights On​

Yes, if your total cost calcuation is only dollars. If you count the total cost of all outputs, like air pollution, but especially the political cost, nuclear is "cheaper".


Nuclear is also the safest form of energy generation, with tens of thousands of years of fission fuel that can be (somewhat) easily mined. 0 people have died from Nuclear power in my lifetime, even Fukushima. Meanwhile, a dozen or so people die every year installing Solar panels on roofs in the United States from falls (my own electrician cousin fell off a roof and had to have his back fused).

Wind makes no sense anywhere, even remote places. You would have to pair a windmill with something like a Powerwall, which has 13.5kwh of power and weighs 400 pounds. One gallon of diesel gives you this same kwh when run through a generator (plus any wind/solar remote setup will need a generator backup anyways)

 

[1_rick]

"chip improvements continue a pace"

It's "apace", not "a pace". Apace means "at a rapid pace; swiftly."