Global IT Outage Caused by CrowdStrike Update Glitch

Major IT outages have hit industries across the world after a CrowdStrike update issue pushed global IT systems offline and prevented Windows systems from starting up. 

Flights are grounded in airports around the world, hospital systems are offline, and global TV channels and news broadcasters have been taken off air as a result of the incident. 

Everything from banks and payment systems in supermarkets is also offline, and companies from a range of industries have said that they would also see delays and technical issues throughout the day. 

Tracking website Down Detector has registered outages and problems from companies from a range of different industries around the world: Delta Airlines, Visa, Mastercard, Lloyds Bank, Santander, Amazon, RyanAir, Sky News, Ladbrokes, BT, and Microsoft Teams all show issues. It’s not yet known whether all of them are linked to the problem with PCs.

Many of the UK’s NHS systems are also offline. In a statement, the health service blamed the outage on an issue with EMIS, an appointment and patient record system, which is disrupting the majority of GP practices.

"The NHS has long-standing measures in place to manage the disruption, including using paper patient records and handwritten prescriptions, and the usual phone systems to contact your GP."

"There is currently no known impact on 999 or emergency services, so people should use these services as they usually would."

"Patients should attend appointments unless told otherwise. Only contact your GP if it’s urgent, and otherwise please use 111 online or call 111.". 

Microsoft has said a resolution for Windows devices is "forthcoming", but said a third party was at fault after it issued an update.

It said: "We are aware of an issue affecting Windows devices due to an update from a third-party software platform. In a separate statement, a spokesperson from the firm said: "We are aware of an issue affecting a subset of customers. 

"We acknowledge the impact this can have on customers, and we are working to restore services for those still experiencing disruptions as quickly as possible."

Microsoft, whose devices are the most impacted by the outage, said that it was investigating the problem and “continue to take mitigation actions.”

What is the cause of the Global IT outage?

The large-scale outage is reportedly related to a broken Crowdstrike cybersecurity update that left Windows computers unable to start up and has taken much of the world’s infrastructure offline.

The issues appear to relate to an issue at Crowdstrike’s Falcon Sensors, which is reporting to have experienced issues following an update that took place in the early hours of Friday morning. 

The cybersecurity firm said that it had identified the issue and rolled the update back – but those computers that were already affected do not appear to have been fixed.

On the company’s Reddit thread, representatives advised that the problem could be fixed by deleting the update and then restarting the computer. 

“CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor,” the firm told users in a Tech Alert on Crownstrike.com.

 Tech Alert from Crowdstrike

“Details Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. Current Action Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.”

“CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.”

This requires administrators to have access to the computer, however, which may not be immediately possible for those who are being used remotely.

“Multiple StickmanCyber security engineering and our 24x7/365 security operations teams across the country support reports that this outage is related to a CrowdStrike update, said StickmanCyber CEO Ajay Unni, one of Australia's largest cybersecurity services companies. 

“It is our understanding that any business running versions 7.15 and 7.16 are affected by the outage, but 7.17 seems to be ok. We are waiting on an official advisory from CrowdStrike on these findings but doing our best to help affected customers. It’s a lesson to always update your software, but obviously, this is an extreme example. IT security tools are all designed to ensure that companies can continue to operate in the worst-case scenario of a data breach, so to be the root cause of a global IT outage is an unmitigated disaster.
 
“Crowdstrike support is offering a workaround to customers. It claims users may be able to fix the issue by booting Windows in safe mode or the Windows Recovery Environment and deleting a file named “C-00000291*.sys”.   

'Not a Cyber Attack'

George Kurtz, the CEO of Crowdstrike, said: "Crowdstrike is actively working with customers impacted by a defect found in a single content update for Windows hosts." "Mac and Linux hosts are not impacted. This is not a security incident or cyber attack.

"The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.

"We further recommend organisations ensure they’re communicating with Crowdstrike representatives through official channels. "Our team is fully mobilised to ensure the security and stability of Crowdstrike customers."

What is Crowdstrike?

CrowdStrike is a cybersecurity company that focuses on protecting devices from cyber attacks. They offer cloud-based solutions to detect and respond to threats, going beyond just traditional antivirus software. 

The Texas-based firm was co-founded in 2011 by current CEO George Kurtz and Dmitri Alperovitch. The company has massive influence, even being called in by the US Democratic National Committee, to investigate a breach into its computer network in 2016.

“Crowdstrike has stated that they are aware of reports of crashes on Microsoft's Windows operating system relating to its Falcon sensor," Adam Pilton, Senior Cybersecurity Consultant at CyberSmart and former Detective Sergeant investigating cybercrime told EM360Tech.

"There are some suggestions that this is two major incidents running simultaneously: A service-wide Azure outage and CrowdStrike Falcon blue screens.”

Ralf - stock.adobe.com

CrowdStrike offers security software to many organizations. A problem on their end could create a domino effect, causing systems belonging to any organizations using their solution to malfunction, including Microsoft. 

Reports suggest that CrowdStrike has issued a software update that has caused critical crashes, known colloquially as ‘the blue screen of death’, to Windows devices, rendering them operable. 

“The outage impacting Windows devices this morning appears to have been caused by a driver update by CrowdStrike, bricking older Windows devices and servers, which will be worst hit. Unfortunately for CrowdStrike, if that is the case, it could be nauseating to fix,"  Tom Kidwell, Co-founder of Ecliptic Dynamics and former British Army and UK Government intelligence specialist, told EM360Tech.

"Due to the nature of the update, an individual from every organisation will need to boot into safe mode, remove the issue file/driver, and then either roll back or update to a new version, something CrowdStrike will need to release very quickly.”

“Incidents like this highlight the vulnerability in using a single supplier on such a vast scale, and why it’s critical that organizations have a backup plan. Best practice for vendors is to pressure test any updates before rollout, however this can be difficult when you serve 60-90% of the world.” Kidwell advised. 

CrowdStrike is massively influential, with its most recent report confirming that they have over 23,019 subscription customers, with a total revenue of $2.24 billion in 2023, a 54% increase, compared to 2022.

How to Fix Crowdstrike Outage Issues?

If you have been affected by the worldwide Crowdstrike IT Outage there are workaround steps you can take to fix the issue yourself. It's important to note these involve modifying system files, so proceed with caution and only if you’re comfortable:

1. Boot Windows into Safe Mode or the Windows Recovery Environment. This provides a limited environment to potentially fix the issue.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Restart your device normally. 

If you are uncomfortable modifying your own system files, experts warn consider waiting for an official fix from CrowdStrike or contacting your workplace IT department for assistance.

"Crowdstrike has blamed a sensor update for the global outage and claims to be fixing the problem themselves. Their current advice is to take no further action but to monitor updates until a resolution is found, Brian Higgins, Security Specialist at Comparitech told EM360Tech.

"Not massively helpful for all of the essential services affected but since there is nothing practical to be done by users at this stage there is little more to be said. I’m sure there will be plenty of post-mortem commentary about resilience models and redundancies etc"

"A verified solution from CrowdStrike is likely on the way, and it will minimize the risk of unintended consequences that could further delay getting your systems back online.

Tom Henson, Managing Director at Emerge Digital told EM360Tech: "There will be many highly skilled individuals working on the issue, especially due to its impact on global infrastructure. They should be able to quickly halt the delivery of the problematic update, to stop it affecting any more systems. If systems are still accessible, pushing out a new update will suffice."

However, if the faulty software causes systems to go offline entirely, the resolution could be lengthy, as each business would need to roll back manually rather than receive an update from the vendor. Offline systems cannot be updated.,"

"Given the complexity of technology, issues like this are inevitable. We frequently see isolated problems with large cloud platforms. If this is indeed a conflicting update issue, both applications being mainstream means it should not have slipped through. This incident is unlikely to be repeated by these vendors to this extent, but it highlights vulnerabilities in global infrastructure,” Henson concluded.