News Global IT issue strikes Windows machines, cause of issue allegedly linked to CrowdStrike software update

[Admin]

An apparent IT update is allegedly linked to a global IT outage for Microsoft Cloud services and BSoD

Global IT issue strikes Windows machines, cause of issue allegedly linked to CrowdStrike software update : Read more

 

[Colif]

Its not always fun being first country in world to noitce

Glad I went out this morning before all the banks stopped working.

An update has been pushed out to fix the broken update.

https://www.abc.net.au/news/2024-07...-microsoft-banks-airlines-australia/104119960

wonder what BSOD it causes.

CrowdStrike CEO George Kurtz says the defect in the recent content update for Windows hosts causing the computing outages has been "identified, isolated and a fix has been deployed".

Microsoft & Crowdstrike combo caused it.

Internet was meant to be better than this.

Hate to be flying as its effected airports worldwide.

 

[nogames]

What a great day First waiting to get my own company issued machine working (5 hours) and now I can see, that around 35 of our 60 Azure VMs are unreachable due to this. It seems like I will be working all weekend.

 

[bill001g]

It will never happen but a photo of the group who had primary responsibility for the update would be nice. They always post photos when things are good maybe some public shaming on twitter would good.

Even if this company would pay for all the costs, and I bet the fine print says they pay nothing, this just causes the stock price to drop. Maybe that to a tiny extent hurts the top executives but they really need to do more so the end employees start to feel if they do a poor job there will be consequences.

Just a sign of things to come with all the forced updates. Just wait until every pc in the world is down because microsoft just had to patch in their latest update to the mircosoft xbox client pretending it was a critical security update.

 

[yoji]

I am not a security expert, but is it time for Microsoft to be allowed to harden the kernel? There is no way any external program should be able to crash the machine. I know many of the security companies insist on hooks into core system components and antitrust rules obliged MS to comply/provide... but is it time to revisit?
It wont protect us from MS messing up... but would be overall "better"?

 

[Pierce2623]

It will never happen but a photo of the group who had primary responsibility for the update would be nice. They always post photos when things are good maybe some public shaming on twitter would good.

Even if this company would pay for all the costs, and I bet the fine print says they pay nothing, this just causes the stock price to drop. Maybe that to a tiny extent hurts the top executives but they really need to do more so the end employees start to feel if they do a poor job there will be consequences.

Just a sign of things to come with all the forced updates. Just wait until every pc in the world is down because microsoft just had to patch in their latest update to the mircosoft xbox client pretending it was a critical security update.

In the modern world, that’s the sort of thing that gets people killed. Let’s not post a picture.

 

[James Roach]

This was a really, really well written article. Covered all the bases of good journalism that are consistently missing across mainstream media. Well done.

 

[das_stig]

"By the time Crowdshite became self-aware it had spread into millions of computer servers across the planet. Ordinary computers in office buildings, dorm rooms, everywhere. It was software in cyberspace. There was no system core. It could not be shut down" psst Microsoft here our systems just BSOD, so we're OK.

 

[Vanderlindemedia]

And nobody in their right mind would even test before pushing an update?

 

[brandonjclark]

The fact is, SOMEONE signed off on a change ticket, and they should be fired, at the least.

 

[Makaveli]

Thankfully we don't use this in our environment. I woke up to my teams chat going crazy this morning and had to remind everyone we are good.

 

[rgd1101]

we have this at work. IT support been up since 1am this morning.

 

[Blastomonas]

The old expression: Too many eggs in one basket... springs to mind.

Whilst it may be cheaper and efficient to use cloud services to replace on site infrastructure, there are some serious negatives too.

 

[rgd1101]

we use them for security. running on pc and server vm.

 

[Makaveli]

we have this at work. IT support been up since 1am this morning.

feel for your team been there done that.

 

[hotaru251]

another reason forced updates shouldn't be a thing.

 

[jeremyj_83]

we use them for security. running on pc and server vm.

We don't use Crowdstrike ourselves but where we get our cloud hosted email company does. We need email as we are remote consultants and everything comes in as tickets.

 

[bit_user]

It will never happen but a photo of the group who had primary responsibility for the update would be nice. They always post photos when things are good maybe some public shaming on twitter would good.

Even if we agreed that "mob justice" were appropriate, you can't know which individuals were responsible or what factors fed into this decisions behind this problem. As a developer, I often have disagreements with management about things like which bugs to fix before release, how things need to be tested, etc. I'm sure a manager could also tell you stories about lazy programmers or testers who took shortcuts, etc.

The best and most appropriate solution is to look for other vendors who provide an equivalent service. Vote with your $$$.

 

[frogr]

It will never happen but a photo of the group who had primary responsibility for the update would be nice. They always post photos when things are good maybe some public shaming on twitter would good.

Even if this company would pay for all the costs, and I bet the fine print says they pay nothing, this just causes the stock price to drop. Maybe that to a tiny extent hurts the top executives but they really need to do more so the end employees start to feel if they do a poor job there will be consequences.

Just a sign of things to come with all the forced updates. Just wait until every pc in the world is down because microsoft just had to patch in their latest update to the mircosoft xbox client pretending it was a critical security update.

"so the end employees start to feel if they do a poor job there will be consequences." Why blame the lowest level employees?
If the company has an effective / adequate prelease testing protocol and a corporate culture that quality overrides the release schedule, this shouldn't have happened. Responsibility for failure is clearly with upper management.

 

[bit_user]

feel for your team been there done that.

I've been on the other side - having a critical project deadline that I planned to try and meet by working for the entire weekend. Then, around 4 PM on Friday afternoon, the internal network goes down for the whole building. IT dude spends just enough time to confirm how broken it is and then checks out. Doesn't even start work on fixing it, until the usual start of his day, Monday morning.

The IT org was part of a different division, in the company. There was nothing my boss could do about it. Theoretically, it could've been escalated high enough, but for one reason or another it didn't happen.

 

[bit_user]

another reason forced updates shouldn't be a thing.

The flip side of this is a Day Zero hack. If their software needed some fix or enhancement to protect your machines, you wouldn't want to wait for the IT guy to get back from vacation and trigger the update, would you?

There's no perfect solution here. The best option is probably more rigorous testing. Depending on the nature of the issue, perhaps also better development standards.

 

[jeremyj_83]

I have a friend who works for Delta and is in Milwaukee today to help get people out from the RNC. He said more than 1/2 of world wide flights have been canceled and all airlines have put an embargo on non rev travel.

A colleague of mine found this on Twitter showing flights over the last 12 hours in the USA.

https://twitter.com/x/status/1814268813879206397

View: https://x.com/us_stormwatch/status/1814268813879206397

 

[Makaveli]

just seen the other article THG posted.

CrowdStrike's market cap plunges $12.5 billion in wake of global outage

Heads gonna roll for this.

 

[bill001g]

just seen the other article THG posted.

CrowdStrike's market cap plunges $12.5 billion in wake of global outage

Heads gonna roll for this.

Nope just things like teacher pensions funds losing their money. The top exec have their golden parachutes and they likely have 2 assistants to tell them what deleting a file means so top management is only symbolically to blame.

Everyone else in the company will point fingers at everyone else and they will agree nobody is to blame.

The only way is if there are enough lawsuits for damages that they have to cut payroll. Not likely they get the guys who really did it and hard to say if laying off their co employees would make these guys even think to do a better job next time.

What is so strange is almost every company I have worked for would have huge test labs that would simulate some of their major customers. They would test push updates and see if the simulated customers would still function.

Since this affected almost every machine it is almost like they didn't even test this.

 

[Makaveli]

Nope just things like teacher pensions funds losing their money. The top exec have their golden parachutes and they likely have 2 assistants to tell them what deleting a file means so top management is only symbolically to blame.

Everyone else in the company will point fingers at everyone else and they will agree nobody is to blame.

The only way is if there are enough lawsuits for damages that they have to cut payroll. Not likely they get the guys who really did it and hard to say if laying off their co employees would make these guys even think to do a better job next time.

What is so strange is almost every company I have worked for would have huge test labs that would simulate some of their major customers. They would test push updates and see if the simulated customers would still function.

Since this affected almost every machine it is almost like they didn't even test this.

No doubt c suite is always exempt from that.