News 77TB of Research Data Lost Because of HPE Software Up

[Wlerin]

Offline backups won't save you when it is your broken backup script that is deleting files instead of actually backing them up.

Neither the old script nor the new script were broken.

It's the amalgamated horror created by updating a running bash script that broke.

They got screwed over by a buggy backup script. Their data would likely have been fine if they hadn't attempted to back it up with the "updated" backup script that ended up destroying two days worth of data before they realized something went wrong.

Or rather, their data would have been fine if they hadn't downloaded and installed the update while the backup script was running.

Or if the backup script were a bit more robust, or HPE had provided clearer updating guidelines, checks for running scripts, etc. etc.

Since it is two specific days of data that were lost, I'd guess it was the NEW DATA that was being destroyed by the borked script. So that data wouldn't exist in any other backups since it didn't exist yet when older backups were made and no longer existed for subsequent backups to pick up.

It wasn't two specific days. It was all files whose last update was prior to December 3, 2021 17:32. The translated article relies too heavily on Google. edit: Even the linked Stack article gets this, and a great many other things that Tom's Hardware misstates or omits, right.

 

[jasonkaler]

HPE admits a software bug that led to an accidental wipeout of 77TB of data.

77TB of Research Data Lost Because of HPE Software Up : Read more

Two huge human failures here:
1) Not having a proper backup strategy.
What they should have done was implement backup rotation strategy. Thats backup 101

2) Automated testing
That's software development 101.
Who on Earth deploys any form of code into a production environment without testing it first?

Also...
Whatever happened to the undelete command?
Is no one able to perform data recovery? Data doesn't cease to exist when it's deleted, only when its overwritten and surely they haven't overwritten that entire volume yet.

 

[Darkbreeze]

Who on Earth deploys any form of code into a production environment without testing it first?

Which is why I said earlier that there needs to be a "sure, go ahead", which generally happens after "Ok, let's test this and see if anything breaks."

 

[ThatMouse]

and who told you that they did not pay HP for the backup solution ? HP admitted its fault. But what makes me angry about this situation is the new generation of lazy people who dont write their own scripts for such a HUGE research Data backups !!! or at least check the codes and the scripts before applying them (Test them ) ... Universities should be better than this. Many MANY people should be fired after this.

Ya I agree. It comes from the "cloud" generation where people just think as long as it's in the cloud, it's totally backed up and safe. The reasoning is flawed: Well if X company does down then we're all in trouble! No, just the people who stored their only copy there is!