News Toyota Shut Down 14 Factories Due to 'Insufficient Disk Space'

[Admin]

Toyota's shutdown of 14 factories in late August was due to an error that occurred because of "insufficient disk space" on its servers.

Toyota Shut Down 14 Factories Due to 'Insufficient Disk Space' : Read more

 

[gggplaya]

Toyota took Lean Engineering way too far.

 

[washmc]

The company further wrote that the servers were running on the same system as its backup, causing the same issue there, so the company couldn't make a switch.

This is the most hilarious part. Who knew I could do better sys admin work than whoever is in charge of all these Toyota plants?

 

[Elusive Ruse]

You'd think at some point they'd use the profits from their overpriced cars and invest it on their servers.

 

[vehekos]

Just in space failed

 

[PEnns]

What the ever-loving f......???

Toyota doesn't have one competent sys. admin?? Who runs out of disk space in this pathetic manner!!

 

[peachpuff]

Did they forget to empty the recycling bin?

 

[derekullo]

Yo dog we heard you liked virtualization!
So we virtualized all your servers so you can backup while you virtualize and virtualize while you backup!

 

[Sippincider]

I could forgive an early startup, with money tight and "Bring Your Own IT".

But a company with roots to 1924, and doing $286+ billion in annual revenue today?? Get the freaking storage you need.

 

[umeng2002_2]

Is this one of those cases where they need to install Windows 11 from thousands of floppy discs?

 

[neojack]

believe it or not, but it happens a lot really.

2 classic cases :
- logs on an SQL serveur. to clear the logs you have to merge the logs with the main DB file.
- snapshots on a Virtual Machine. It creates two files than needs to be merged eventually.

Those merging operations needs twice the space as the separate files. The process can start but if there is not enough space, it would fail and all the files would be corrupt. You need space to restore the backup but at this point there is any. it's a catch 22.
Restoring a backup needs space wich is a problem if the backup is on the same disk as it was the case here.

Also, by default, Microsoft SQL server stores it's files on the system partition. So if the company maintaning it didn't payed attention, the problem would be even worse, the OS would crash.

 

[Sluggotg]

Maybe I could loan them a couple of 1541 Disk Drives?

 

[sitehostplus]

This is the most hilarious part. Who knew I could do better sys admin work than whoever is in charge of all these Toyota plants?

Good thing they sell cars, and not computers. Then I would be really worried.

 

[Eximo]

Hopefully the admins have a chain of emails related to storage expansion that went unfulfilled due to budget restrictions.

 

[brandonjclark]

Did they forget to empty the recycling bin?

"Going forward, we will review our maintenance procedures and strengthen our efforts to prevent a recurrence, so that we can deliver as many vehicles to our customers as soon as possible," the unsigned note reads.

I think you're closer than you realize.

I bet it's something REALLY stupid like not clearing up IIS logs or some base-level datastore over-provisioning type of problem in vmwareland.

Or maybe a DB issue.

The fact that they call out a maintenance procedure pretty much says it all.

 

[umeng2002_2]

Good thing they sell cars, and not computers. Then I would be really worried.

Oh sweet summer child...

 

[ingwe13]

Oh sweet summer child...

This hits the nail on the head.

 

[fireaza]

Toyota's shutdown of 14 factories in late August was due to an error that occurred because of "insufficient disk space" on its servers.

Toyota Shut Down 14 Factories Due to 'Insufficient Disk Space' : Read more

As someone who lives in Japan, I can confirm new HDDs are a lot more expensive than in other countries, so I can't blame them for not wanting to buy them.