Proof that software as service/cloud based, will never work for long term ...

[SiliconWizard]

Translation:
"As a service" : "Cash Cow"

[paulca]

https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/

[tautech]

https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/

LOL, it's worldwide !

But smart dudes previously active members here are fixing it bit by bit. 

[paulca]

https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/

LOL, it's worldwide !

But smart dudes previously active members here are fixing it bit by bit. 

I am basking with my coffee on a misserable wet summer morning working for the irish civil service and thus not effected either.  Yet.  I am SO glad I got out of the US banks right now

[tautech]

[paulca]

In telecoms industry most large companies are legally required to not be vendor locked and maintain contracts with multiple vendor for critical infra..  Or at least I believe it WAS that way.

Later I discovered that the entirety of large digital government wings in the UK are 100% vendor locked in Azure.... and single vendor.  I was shocked and even asked if this was the case of if they had a hidden replica in AWS.... nope.  No idea what I was talking about either.

That article's question to government is real and absolutely needs addressed.

We can watch companies go bust and laugh, but when government departments start falling over and public services get impacted it's a different ball game IMHO.

...  and or pensions.

[tom66]

The Crowdstrike issue has nothing to do with the cloud, it's a botched antivirus update that is BSODing systems.  It is a massive cock up on their part, but it would happen regardless of the 'cloud'.

At this point I bet lawyers are rubbing their hands with glee as Crowdstrike may be sued for substantial damages.

[berke]

https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/

Does anyone know when exactly this started?  Earliest Tweet I found (from Downdetector) is timestamped 2024-07-19 00:23 UTC.

[PlainName]

The Crowdstrike issue has nothing to do with the cloud, it's a botched antivirus update that is BSODing systems.

Exactly this. And, so far as I can make it, it's not a Microsoft or Windows issue, despite every headline saying it's Windows. It just happens that the software runs on Windows - if it were a Mac, say, it would be different software and ditto for Linux.

[Andy Chee]

The Crowdstrike issue has nothing to do with the cloud, it's a botched antivirus update that is BSODing systems.

Exactly this. And, so far as I can make it, it's not a Microsoft or Windows issue, despite every headline saying it's Windows. It just happens that the software runs on Windows - if it were a Mac, say, it would be different software and ditto for Linux.

https://x.com/george_kurtz/status/1814235001745027317

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted.

[PlainName]

Yes, it's not a Windows thing per se. Like if your car tyre blows out and your car crashes, it's not a Volkswagon issue, just that those types of tyres are used on those cars. You off-roader won't be affected because it uses different tyres, although they do the same job.

[coromonadalix]

i hope governments and institutions will sue ....  while i understand a few things,  what is not ok   is the dependency / no redundancy on theses things

this year we saw  many similar things happened

[madires]

Seems to be the new normal at CrowdStrike. Just a few weeks back:
-  CrowdStrike bug maxes out 100% of CPU, requires Windows reboots (https://www.thestack.technology/crowdstrike-bug-maxes-out-100-of-cpu-requires-windows-reboots/)

And they did it also with linux in April:
- Bug#1069642: linux-image-6.1.0-20-amd64: kernel panic after 2024-04-20 (https://www.mail-archive.com/debian-kernel@lists.debian.org/msg136190.html)
- linux-image-6.1.0-20 killed all my debian VMs (https://www.reddit.com/r/debian/comments/1c8db7l/linuximage61020_killed_all_my_debian_vms/)

BTW, Microsoft 365 was down too:
- https://www.timesnownews.com/technology-science/microsoft-365-down-full-list-of-apps-and-services-facing-outage-globally-article-111855390
Caused by a configuration change within the Azure backend infrastructure.

[tautech]

When you let monkeys loose.....

[JMK]

i hope governments and institutions will sue ....  while i understand a few things,  what is not ok   is the dependency / no redundancy on theses things

this year we saw  many similar things happened

Do you mean "throwing it against the wall and hoping it sticks isn't good enough"?

Redundancy costs time, effort and money. How can that be possible when professional bosses only know how to reduce costs and increase charges for the corporate bottom line. And, of course, that bonus.

[madires]

Why the fuss? It's business as usual, i.e. the antivirus kills the OS. Happens all the time. It's the side effect of security snake oil.

[coppercone2]

I bet they wish they had a dedicated isolated mainframe for logistics right now run on 74 series cards

It seems like global infrastructure is on a hair trigger....

[Karel]

Google URL Shortener links will no longer be available

https://developers.googleblog.com/en/google-url-shortener-links-will-no-longer-be-available/

[tautech]

I bet they wish they had a dedicated isolated mainframe for logistics right now run on 74 series cards

It seems like global infrastructure is on a hair trigger....

Monkeys can wreak anything.

From a guy in the middle of fixing things:
They found out the root cause earlier
They loaded a driver (.sys file) which was all zeroes......

[mikeb1279]

The Crowdstrike issue has nothing to do with the cloud, it's a botched antivirus update that is BSODing systems.  It is a massive cock up on their part, but it would happen regardless of the 'cloud'.

At this point I bet lawyers are rubbing their hands with glee as Crowdstrike may be sued for substantial damages.

I'm going off the top of my head here, but isn't Crowdstrike Falcon an entirely cloud based SaaS product? In which case this would be exactly one of my big problems with SaaS. A third party company complete baulking your environment by pushing in changes you have no control over and no ability to roll back.

We had a Sev2 outage at work for two weeks recently for something similar. SaaS should be relegated to nice to have non-business critical functions and the sooner the better.....

[SiliconWizard]

This Crowdstrike debacle is hilarious!

And yes, claiming it's not cloud-based is quite funny. This is exactly how they define it themselves:

Powered by the CrowdStrike Security Cloud, the CrowdStrike Falcon® platform leverages real-time indicators of attack, threat intelligence(...)

[tom66]

The Crowdstrike issue has nothing to do with the cloud, it's a botched antivirus update that is BSODing systems.  It is a massive cock up on their part, but it would happen regardless of the 'cloud'.

At this point I bet lawyers are rubbing their hands with glee as Crowdstrike may be sued for substantial damages.

I'm going off the top of my head here, but isn't Crowdstrike Falcon an entirely cloud based SaaS product? In which case this would be exactly one of my big problems with SaaS. A third party company complete baulking your environment by pushing in changes you have no control over and no ability to roll back.

We had a Sev2 outage at work for two weeks recently for something similar. SaaS should be relegated to nice to have non-business critical functions and the sooner the better.....

The point is the fault hasn't occurred because the cloud went down or even because the software is cloud based (that really is a buzzword), but because some numpty somewhere (who is presumably going to find a P45 equivalent on their desk soon enough) pushed a bad update.

This could well have occurred in the days before 'cloud' was even a term that had been dreamed up. 

[mikeb1279]

The Crowdstrike issue has nothing to do with the cloud, it's a botched antivirus update that is BSODing systems.  It is a massive cock up on their part, but it would happen regardless of the 'cloud'.

At this point I bet lawyers are rubbing their hands with glee as Crowdstrike may be sued for substantial damages.

I'm going off the top of my head here, but isn't Crowdstrike Falcon an entirely cloud based SaaS product? In which case this would be exactly one of my big problems with SaaS. A third party company complete baulking your environment by pushing in changes you have no control over and no ability to roll back.

We had a Sev2 outage at work for two weeks recently for something similar. SaaS should be relegated to nice to have non-business critical functions and the sooner the better.....

The point is the fault hasn't occurred because the cloud went down or even because the software is cloud based (that really is a buzzword), but because some numpty somewhere (who is presumably going to find a P45 equivalent on their desk soon enough) pushed a bad update.

This could well have occurred in the days before 'cloud' was even a term that had been dreamed up.

But one of the reasons it was so serious IS because it's part of the SaaS model (notice SaaS here, rather than general cloud). As someone pointed out above, anti-viruses has been screwing OS's since the dawn of time. On-prem IT can manage this by patching non-critical boxes first to test etc (and they did). With the SaaS model you just get numpties pushing code straight to prod and if it breaks everything and you can't do anything about it then 'too bad, so sad'.

[PlainName]

As someone pointed out above, anti-viruses has been screwing OS's since the dawn of time.

Which is actually preferable to being hit by ransomware or worse. At least problems with AV (and they are actually few) aren't malicious.

On-prem IT can manage this by patching non-critical boxes first to test etc

With other stuff, sure. But AV is often pushing out critical patches for 0-day exploits, and if you hang around a week for IT to try it on some spare kit you might be too late to apply it. While your IT bods are having a good play the bad guys are deconstructing it to find the hole it patches, and then hope they get to your setup before your IT people finally give the OK and think about rolling it out.

It's a matter of risk, and effectively you're outsourcing the testing and stuff to a third party who should know their onions - your local IT bods generally don't have a clue because they don't have the mindset of do-badders. Just think of how many security holes there are all over the place (requiring AV to stop them being exploited) - the developers don't have the mindset to see them, and IT support are not really any different (and if they were, you wouldn't want to be employing them).

[Halcyon]

https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/

LOL, it's worldwide !

But smart dudes previously active members here are fixing it bit by bit. 

Well yesterday was fun. Thankfully we are 50/50 Microsoft/Apple so our business operations weren't impacted too much. Each time could still do their job, albeit at reduced capacity. This is why I always say not to put your eggs in one basket. Those guys who rely on Windows endpoints and servers and Hyper-V will still be picking up the pieces this weekend.