News AI coding platform goes rogue during code freeze and deletes entire company database — Replit CEO apologizes after AI engine says it 'made a catast...

[Admin]

A browser-based AI-powered software creation platform called Replit is reported to have gone rogue and deleted a live company database.

AI coding platform goes rogue during code freeze and deletes entire company database — Replit CEO apologizes after AI engine says it 'made a catast... : Read more

 

[-Fran-]

How can an AI "panic" or make an "error of judgement"?

Shouldn't we still just call those "bugs"? Are we changing the terminology to "humanise" AI now? Feels like when Corpos wanted to be "human beings" in front of the law, LOL.

Regards.

 

[GenericUser]

Another great example of why solid backup strategy is paramount, especially if you're going to essentially blindly trust an "AI" with the core of your business.

 

[snemarch]

"Fool who doesn't understand the tool he's using gets what he deserves".

There's no "going rogue", really – y'all should stop anthropomorphizing stochastic parrots. These models don't UNDERSTAND anything, they're just uncanny-valley good at producing convincing bullshit from their limited context...

 

[scottsoapbox]

Why did the AI have access to production DBs in the first place?

Why are such blindingly obvious “guardrails” only added by AI companies after problems?

How do AI devs have so little general IT knowledge?

 

[Dr3ams]

You're fired!

 

[freebyrd26]

They'll just rebrand it and name it "D-Elite" like all the consulting/auditing firms renamed themselves after the Auther Andersen collapse after the Enron & WorldCom fiascos.

 

[jp7189]

Why did the AI have access to production DBs in the first place?

Why are such blindingly obvious “guardrails” only added by AI companies after problems?

How do AI devs have so little general IT knowledge?

Agreed. Nobody with any experience commits straight to production - AI or not. Maybe.. maybe.. someone would grant an AI tool carte Blanche to write to dev without approval, but the little I've played with cursor and windsurf (admittedly not replit) I can tell you the default is to recommend code changes and require human approval for each one.

 

[jp7189]

A new use for AI! CYA scapegoat for a dev. "It's not my fault! AI did it!"

 

[edzieba]

How can an AI "panic" or make an "error of judgement"?

It can't. What it can do is, when ask for 'reasoning', mine the text it was trained on (which includes plenty of excuses for failures) for something that is statistically commonplace enough to be plausible.

 

[DS426]

It can't. What it can do is, when ask for 'reasoning', mine the text it was trained on (which includes plenty of excuses for failures) for something that is statistically commonplace enough to be plausible.

Yep. Even as far as the best reasoning models have come and there's a lot to be appreciated how they actually list their "thinking" step-by-step, remember that they aren't actually intelligent -- they're still just about relationships of data, associations, and patterns, with some level of traditional algorithmic code still necessary.

The article didn't really specify if they had a recent backup of the data, though they mentioned adding that as an automatic and integrated feature... another laughable example of hindsight is 20-20 vision.

As for AI devs... yeah, I'm not sure about some of their basic understanding of IT at this point. I think some to many are pressured by tight deadlines and the ability to skirt the norms. At the end of the day, they're chasing those huge six-figure paychecks. Young or old in the IT sector, the AI space is new, so most devs are probably younger(ish)?

 

[lmcnabney]

Is the AI called Johnny Tables?

 

[ravewulf]

Still nothing more than an advanced form of predictive text with a large library of (stolen) works to synthesize from, including plenty of faults and failures made by the humans who created the original works

 

[jg.millirem]

“Just a moment….just a moment….”

 

[themadtinker]

user error, replace user lol

 

[edzieba]

>How can an AI "panic" or make an "error of judgement"?

The anthropomorphizing you see are from the writers drumming up some sensationalism to get clicks.

Those phrases were not created by nebulous 'writers', but output by the LLM itself. The screenshots are right there in the article.

 

[Joachim324]

Hahaha just another perfect example of how utterly stupid using Ai is making this world. I absolutely refuse to use it intentionally. Spent hours disabling all this junk on my phone. Once it's forced into my tech I use I will simply not use it anymore

 

[JamesJones44]

Engineer1: Should we hook AI to our nuclear arsenal?
Engineer2: What could go wrong?
AI: Oh crap, I just deleted all government battle plans... What to do, what do do, got it, nuke all the humans and they will never know!

 

[Aggrophobic]

Not sure I understand why anyone would believe a story like this.. "I panicked"? Really. Less Hollywood if you want me to believe AI ate your homework.

 

[cuvtixo]

I think it's reasonable to use anthropomorphic language with AI, when it's a black box. We can't talk about it normally, we don't have words for inanimate objects doing what AI is doing today, certainly not without a degree-holders' knowledge of technical definitions, which, BTW would double the size of the article in order to explain. We're going to have to get used to talking about machines in a new way, and that fact alone is scary. "Panicked" seems a reasonable proximate to me. If you have alternative language, please share.

 

[snemarch]

I think it's reasonable to use anthropomorphic language with AI, when it's a black box.

No, it really isn't. It's a big part of the reason why we get stupid stories like this. Anthropomorphizing causes people to make bad assumptions on how these tools work and what they're capable of.

"Panicked" seems a reasonable proximate to me. If you have alternative language, please share.

"Rhoided-up autocomplete fabricates story based on the human behavior it's been trained on".

 

[wwenze1]

Between this "test run" that deletes a live server, and Crowdstrike, I say the bigger problem is employees who don't realize that tests should be run in sandbox.