News AI coding platform goes rogue during code freeze and deletes entire company database — Replit CEO apologizes after AI engine says it 'made a catast...

[-Fran-]

It can't. What it can do is, when ask for 'reasoning', mine the text it was trained on (which includes plenty of excuses for failures) for something that is statistically commonplace enough to be plausible.

In case it wasn't obvious enough: it was a rethorical question.

It reads super cringe when they implement features to "humanise" the LLMs.

Regards.

 

[Hasselblad]

Remember the "early" days, say 18 years ago when people blamed "computer faults" for mistakes, computers do not make mistakes only humans.

 

[ggruber]

How can an AI "panic" or make an "error of judgement"?

Shouldn't we still just call those "bugs"? Are we changing the terminology to "humanise" AI now? Feels like when Corpos wanted to be "human beings" in front of the law, LOL.

Regards.

During my 30 years as a software architect, I never deleted a company's DB. I guess I was just lucky...?

 

[-Fran-]

During my 30 years as a software architect, I never deleted a company's DB. I guess I was just lucky...?

I know a couple very smart people that went a step too far in how smart they thought they were and have deleted tables and schemas in Prod DBs because... Well, they thought they knew better against all warnings.

Not saying the AI is right/correct or "let's give it the benefit of the doubt", but let's not forget they're coded to the image of the developers and the information fed into them. Simplifying it, for sure, but not far off.

So, perhaps in a sense, you have been indeed lucky to have had access to data proving doing that is a terrible idea and peers who have taught and complemented you well.

Regards.

 

[2Be_or_Not2Be]

This article is a bit lacking on critical details, and it feels a bit sensationalized.

It mentions "‘Vibe Coding Day 8’ of Lemkin’s Replit test run" - so are what we really looking at is a story about a "vibe coding" event where they create fictitious companies and see how an AI agent could help this fictitious company's workflows?

Granted, the outcome seen here is pretty bad for that AI company's AI agent, but it doesn't seem like this is even a real company with "live" production database(s).

 

[John_Turner]

This is why AI needs Sandboxing. There's got to be parts of the system it cannot touch. Otherwise it's like handing the company over to a two year old.

 

[passivecool]

Backups. Backups. Backups.