News Global IT issue strikes Windows machines, cause of issue allegedly linked to CrowdStrike software update

[slightnitpick]

Or more open source solutions. I know it's not a panacea, but opening up the software means more people can do audits on it to find bugs and other problems. I'm not saying this would be manual - a lot of what people currently do is in the form of automated static and dynamic analysis.

Open source has issues too: https://blog.dshr.org/2020/06/supporting-open-source-software.html

Catalin Cimpanu illustrates the scale of the inadequate support problem in Vulnerabilities in popular open source projects doubled in 2019:

A study that analyzed the top 54 open source projects found that security vulnerabilities in these tools doubled in 2019, going from 421 bugs reported in 2018 to 968 last year.

According to RiskSense's "The Dark Reality of Open Source" report, released today, the company found 2,694 bugs reported in popular open source projects between 2015 and March 2020.

The report didn't include projects like Linux, WordPress, Drupal, and other super-popular free tools, since these projects are often monitored, and security bugs make the news, ensuring most of these security issues get patched fairly quickly.

Instead, RiskSense looked at other popular open source projects that aren't as well known but broadly adopted by the tech and software community. This included tools like Jenkins, MongoDB, Elasticsearch, Chef, GitLab, Spark, Puppet, and others.

RiskSense says that one of the main problems they found during their study was that a large number of the security bugs they analyzed had been reported to the National Vulnerability Database (NVD) many weeks after they've been publicly disclosed.

The company said it usually took on average around 54 days for bugs found in these 54 projects to be reported to the NVD, with PostgreSQL seeing reporting delays that amounted to eight months.

And the big deal with any software is marketshare. There are open source products that basically saturate their particular market. And the people rolling out the open source solution at any given organization aren't necessarily looking at the code. Open source does not mean that more white hats will be looking at the code, but it does mean that more black hats can.

Another option might be to have some sort of regulation or certification. Obviously, you couldn't certify individual releases of a security software package that needs to roll out promptly enough to deal with zero day attacks, but you could certify the development & testing process.

Maybe, but then the measuring stick becomes the target, so that when a new exploit happens the programmers and company can declare that they followed procedure.

 

[bit_user]

Open source has issues too: https://blog.dshr.org/2020/06/supporting-open-source-software.html

Let's not get on to a big digression about open source. I said it's not a panacea, but it's working quite well for a lot of major software packages and customers. It has issues, but not ones that can't be addressed by one way or another.

Maybe, but then the measuring stick becomes the target, so that when a new exploit happens the programmers and company can declare that they followed procedure.

For me, the question is this: do you want these providers to be a total black box, or would you (as a hypothetical or actual customer) like to have some insight into how they do the development and testing?

There's no perfect solution, but that doesn't mean there are no ways to improve the current status quo. I think we need only look at how existing mission-critical products and services are developed, in order to see that there are indeed tools and techniques that have a good track record.

 

[slightnitpick]

I think we need only look at how existing mission-critical products and services are developed, in order to see that there are indeed tools and techniques that have a good track record.

And apparently we now know those tools are Windows 3.1 and 95, and the techniques are "never upgrade". j/k

 

[palladin9479]

Is the cloud aspect even relevant here? This isn't a case of clients being unable to access some needed resource because it's in the cloud and that cloud is down. It was an update to a locally installed application, causing the local machines to crash.

Absolutely yes, the an employee of Crowdstrike pushed an update to production. Every customer running the Windows version of the Falcon agent immediately downloaded and installed that update, then soon after kernel panic and crashed. The owners of those critical systems had no choice in the matter, their entire enterprise network was effectively controlled by an unknown employee at a third party company. The software in question even has a version lock option where it will be one version behind production (n-1), which should of saved those systems, but it looks like that option is merely a suggestion and mothership can just override it, which it did.

To practical effect of these type of systems is that an employee at another company outranks and can override your CIO, CEO and even Board of Directors policies. This aspect of "another mans datacenter computing" is something that is chronically missed and deliberately avoided in such discussions. It is a risk that needs to be accounted for in architectures.

No joke, our previous Security Director had a hardon for Crowdstrike and wanted us to implement it. When we did the vendor assessment we asked about updates, they mentioned that they would handle them. We then asked if we could control them and were firmly told in an uncomfortable tone "umm no, we don't let customers do that". Turns out that one of their "selling" points is that they'll handle all updates for you and that you do not need to ever worry about managing agent or signature versions. That was a very hard no from our CIO and we went with Fire eye / Carbon Black instead. Both of those require us to manage the update process, which we do, but no surprise outages.

Ultimately it boils down to authority and responsibility and that while you can delegate authority to an outside entity responsibility can never be delegated.

 

[USAFRet]

Turns out that one of their "selling" points is that they'll handle all updates for you and that you do not need to ever worry about managing agent or signature versions.

ITAAS
IT As A Service.

Let someone else handle it. The C-level can reduce their inhouse workforce, and likely get a bonus for doing so.

And then something like this happens.

 

[palladin9479]

so wait, ms uses crowd strike for all their comercial programs?

That's a different issue entirely, Microsoft has a partner agreement with Crowdstrike to offer it as part of their Azure products. If you pay for it then it'll be automatically installed into all your Azure infrastructure and is effectively "invisible" to you. When Crowdstrike pushed their broke update, every agent on the planet, including those in Azure, immediately downloaded it, installed it, then turned on the kernel and it blew up.

It's not really a MS vs no-MS, it's more that Microsoft admins / engineers tend to prefer things that automagically update themselves with little to no intervention, while Unix/Linux admins / engineers innately do not trust such systems and want oversight. When was the last time a Fedora or Debian desktop forced an update and reboot on the user? Yet that is now an accepted policy with Windows 10/11.

 

[palladin9479]

ITAAS
IT As A Service.

Let someone else handle it. The C-level can reduce their inhouse workforce, and likely get a bonus for doing so.

And then something like this happens.

Yep people think they can push responsibility for something off to someone else and by carefully controlling the conversation the real obvious questions are not allowed to be asked. I think the sheer ridiculous amount of financial damage from this even tis going to be enough to cause everyone to rethink their production risk strategy.

 

[das_stig]

Crowdstrike lead developer exposed !!