How to fix CrowdStrike BSODs in three minutes — fix requires manual changes, but they are simple

[rluker5]

I hope Delta does this.
I had my 6:40AM flights cancelled at 10:00PM last night and rescheduled for 6AM tomorrow.
Not as nice as a vacation siting here.

 

[CmdrShepard]

I hope Delta does this.
I had my 6:40AM flights cancelled at 10:00PM last night and rescheduled for 6AM tomorrow.
Not as nice as a vacation siting here.

Does this answer your hopes?

https://co.reddit.com/r/delta/comments/1e73d0r/manual_bitlocker_recovery_on_every_machine/

 

[dwd999]

Can you do this also with Bitlocker ?

No. This is also a good reason not to encode an entire disk with Bitlocker.

 

[vanadiel007]

they suppose to protect us from 0-day cyber security issue. not sure monthly freq would be enough.

How secure do you feel if they cannot protect their own software from a programming mistake that causes global mayhem?
This error on their part has caused more money and downtime than any cyber attack would. They already call this the largest IT outage in history...

 

[vanadiel007]

That's not a blind update -- CrowdStrike is an EDR which is similar to AV. What was updated was supposedly the equivalent of "definitions". The problem seems to be that the kernel driver they wrote did two things it shoudln't have:

1. It parsed downloaded definitions (so practically handling untrusted input in kernel ring 0 privilege)
2. It didn't have proper exception handling (like try/catch block) and crashed taking down the OS with it

So no, updating definitions (and even drivers if necessary) is acceptable, writing crappy code shouldn't be acceptable but it is not only acceptable but also paid well and made easy so that even idiots can do it nowadays with the help of various tools.

When we are at it, the CEO of CrowdStrike was the CTO at McAfee back in 2010 when they had their moment of fame with a similar issue. As long as incompetence is rewarded by giving even better positions with higher salaries instead of firing this is going to keep happening.

I would not feel "secure" with a solution that crashed my whole network and took down half of the world...

 

[CmdrShepard]

I would not feel "secure" with a solution that crashed my whole network and took down half of the world...

That's OK -- it's not you who should feel secure, it's the company's management who has outsourced security to a 3rd party. And they will feel absolutely secure when they get called in front of their shareholders since they "followed best practices" and paid for someone else to take the blame.

It's all a security theater, and other companies trust you based on who you work with, not on how you work.

 

[rluker5]

Does this answer your hopes?

https://co.reddit.com/r/delta/comments/1e73d0r/manual_bitlocker_recovery_on_every_machine/

Thank goodness I don't have to use those anymore because of phones. The Delta app also wasn't working right this morning, but the website was fine.
Edit: 2nd Delta flight cancelled as well. They might have something on Monday, but my return from NY was on Tuesday. About half of their stuff on the board here is cancelled with most of the rest delayed for now.
We didn't have a flight attendant available was the reason.

 

[mhmarefat]

I mean, they were lucky this was not a virus or backdoor

What do you mean? Clients had already installed the backdoor (Windows 10).

 

[vanadiel007]

That's OK -- it's not you who should feel secure, it's the company's management who has outsourced security to a 3rd party. And they will feel absolutely secure when they get called in front of their shareholders since they "followed best practices" and paid for someone else to take the blame.

It's all a security theater, and other companies trust you based on who you work with, not on how you work.

I disagree with that. I should feel secure when my data is stored by a bank or an airline.
Obviously that is not the case here. Case in point is the latest AT&T data security breach.

It's really time we out these companies with poor practices so we know who to stay away from.

 

[USAFRet]

It's really time we out these companies with poor practices so we know who to stay away from.

That list is LONG.
Mercedes, Yahoo, Apple, Prudential, Tesla, just to name a few.

And some 'companies' you have no option to stay away from. Experian and NHS, for instance.

 

[vanadiel007]

That list is LONG.
Mercedes, Yahoo, Apple, Prudential, Tesla, just to name a few.

And some 'companies' you have no option to stay away from. Experian and NHS, for instance.

Is it not terrible that they are so reliant on security software from a single vendor that does not even secure them from a global outage?

 

[USAFRet]

Is it not terrible that they are so reliant on security software from a single vendor that does not even secure them from a global outage?

Not getting your question.

Not all companies that have had a major data breach rely on the same single source for security.
The causes and reasons behind these breaches are many and varied.

 

[-Fran-]

No one will disagree that "single points of failure" are terrible by nature and design.

If anyone chooses to implement something while well aware of it, then they have to justify it very well.

In this particular case... Follow the money, I guess.

Regards.

 

[RyzenNoob]

Unles you're the poor IT guy who needs to do it on 10s of thousands of pcs, vms, atms and kiosks, this solution is ludicrous, it will cost companies A LOT.

Not really, this was a server based problem

 

[bill001g]

Not really, this was a server based problem

Be careful there was a microsoft one that happened almost at the same time.

The big one is the crowdstrike one. I am sure a huge number of servers were affected but it also was a massive number of end client machines. Many companies use crowdstrike to secure the laptops people use to work at home. They partially use crowdstrike to know if the employee installed some software...it seem they can also use this software to track employees usage.

The thing that makes this even a larger issue is many companies also used bitlocker in case the machines were stolen. Many of these laptops have to be physically taken into the office....or worse shipped in.

There were example on tv where they showed the technicians working on thing like airline kiosk. These are not really servers they are client that are used to access the airline network. They need them extremely secure because they are in a open airport where anyone can have physical access. When the machine got corrupted all the extra security greatly increased the recovery time. I suspect this will not be completely resolved for many days if not weeks. Too many device need to be touched.

 

[Ken Lynch]

Has anyone come up with a bootable USB that has admin access to the Windows directory on the boot drive? On all of my machines I had to know the local admin pwd before I could get the CMD to launch in Safe Mode. Many machines were built many years ago before me but thankfully I had the local admin and pwd. If I did not, these machines would be bricked.

I'm disappointed that all of these "simple instruction" posts don't consider that best practice is to not have the main user be the local admin.

I'm guessing that a dos usb boot disk would honor the admin rights imposed on C:\windows ... but maybe a Linux boot USB might be able to wiggle in just to delete the one file you need to get rid of to boot off the HD.

 

[rgd1101]

Yes I read that linux boot would work, unless there is bitlocker