A flawed software update issued by security giant CrowdStrike caused a massive overnight outage to Windows computers around the world, disrupting businesses, airports, train stations, banks, broadcasting stations and the health sector.
CrowdStrike said the outage was not the result of a cyberattack, but rather a “flaw” in a software update for its flagship security product, Falcon Sensor, which caused all Windows computers with Falcon installed to crash before fully loading.
“The issue has been identified, isolated, and a fix has been distributed,” CrowdStrike said in a statement on Friday. Some businesses and organizations have begun to recover, but many expect the outage to continue through the weekend or next week, given the complexity of the fix. CrowdStrike CEO George Kurtz told NBC News that “some systems that do not automatically recover may take time.” In a subsequent tweet, Kurtz apologized for the outage.
Here's everything you need to know about the outage.
what happened?
Late Thursday night and into Friday, reports of IT issues began emerging of Windows computers hitting the infamous “Blue Screen of Death,” a bright blue error screen and message that appears when Windows encounters a critical failure, crashes, or is unable to load.
The outages were first confirmed in Australia early on Friday, with reports coming in from other parts of Asia and Europe, as well as the United States, soon after the day began.
CrowdStrike quickly discovered that a faulty software update for Falcon was causing Windows computers that had the software installed to crash. Falcon allows CrowdStrike to remotely analyze and check for malicious threats and malware on the computers it's installed on.
Around the same time, Microsoft reported a significant outage in one of its most used Azure cloud regions, which covers much of the central U.S. A Microsoft spokesperson told TechCrunch that the company's outage was unrelated to the CrowdStrike incident.
Around noon Eastern time on Friday, Microsoft CEO Satya Nadella posted on X that the company was aware of the CrowdStrike update failure and was “working closely with CrowdStrike and the industry at large to provide technical guidance and support to help customers get their systems back online safely.”
What is CrowdStrike and what does the Falcon Sensor do?
Founded in 2011, CrowdStrike has quickly grown into a cybersecurity giant, currently providing software and services to 29,000 enterprise customers, including nearly half of the Fortune 500 companies, 43 of the 50 U.S. states, and eight of the top 10 technology companies, according to its website.
The company's cybersecurity software, Falcon, is used by businesses to manage the security of millions of computers around the world, including large corporations, hospitals, transportation agencies and government agencies. Most consumer devices do not run Falcon and are not affected by this outage.
The company's biggest recent achievement was catching a Russian government hacker group breaking into the Democratic National Committee ahead of the 2016 US presidential election. CrowdStrike is also known for giving memorable animal-themed names to the hacker groups it tracks based on their nationality, such as Fancy Bear, believed to be part of Russia's Main Intelligence Directorate (GRU), Cozy Bear, believed to be part of Russia's Foreign Intelligence Service (SVR), Gothic Panda, believed to be a Chinese government-affiliated group, and Charming Kitten, believed to be an Iranian government-affiliated group. The company also produces action figures representing these groups, which it sells as souvenirs.
CrowdStrike is a very large company, one of the sponsors of the Mercedes F1 team, and this year they even became the first cybersecurity company to air an ad in the Super Bowl.
Who is affected by the outage?
Virtually anyone who interacts with computer systems running CrowdStrike software in their daily lives is affected, even if the computer is not their own.
These devices include grocery store cash registers, airport and train station departure boards, school computers, work-issued laptops and desktops, airport check-in systems, airlines' proprietary ticketing and scheduling platforms, healthcare networks, and more. Because of the pervasiveness of CrowdStrike's software, an outage would cause disruption in multiple ways around the world. Just one affected Windows computer in a fleet of systems can disrupt a network.
TechCrunch reporters around the world have seen and experienced outages in a variety of places, including travel destinations, doctor's offices, and online. Early Friday morning, the Federal Aviation Administration issued a grounding order citing disruptions, effectively halting all air travel across the U.S. So far, Amtrak's national rail network appears to be functioning normally.
What is the US government doing so far?
Given that the problem originates from the companies, there isn't much the US federal government can do. The investigation team's report states that President Biden has been informed of the CrowdStrike outage and that “his team has been in contact with CrowdStrike and the affected organizations.” This is primarily because the federal government is a CrowdStrike customer and is affected.
Several federal agencies were affected by the incident, including the Department of Education and the Social Security Administration, which both announced Friday that they had closed their offices as a result of the power outage.
The report said Biden's team was “coordinating across agencies, receiving sector-specific updates throughout the day, and standing by to provide assistance as needed.”
In a separate tweet, the Department of Homeland Security said it was working with the US cybersecurity agency CISA, CrowdStrike, Microsoft and federal, state, local and critical infrastructure partners to “thoroughly assess and address the outage.”
Government and congressional investigators will no doubt be asking questions of CrowdStrike (and, to some extent, Microsoft, whose unrelated outage caused overnight disruption to customers).
At this time, the focus is on restoring affected systems.
How can affected customers repair their Windows computers?
The big problem here is that CrowdStrike's Falcon Sensor software malfunctions and crashes Windows machines, and it's not easy to fix.
So far, CrowdStrike has released patches and also detailed workarounds that can help affected systems function normally until a permanent solution is found. One option is for users to ” [affected computer] “You will be given the opportunity to download the reverted channel files,” he said, referring to the modified files.
In a message to users, CrowdStrike detailed several steps that customers can take, one of which is that they will need physical access to the affected system to remove the flawed file: CrowdStrike said that users should boot their computer into Safe Mode or the Windows Recovery Environment, navigate to the CrowdStrike directory, and delete the flawed file “C-00000291*.sys.”
The broader issue of having to manually fix files can be a major headache for businesses and organizations that have large numbers of computers or Windows-based servers in data centers, other regions, or even locations in other countries entirely.
CISA warns that malicious actors are “exploiting” the outage
In a statement on Friday, CISA said the outage was the result of a bug in a CrowdStrike update and not a cyberattack. CISA is “working closely with CrowdStrike and our federal, state, local, tribal, territorial, critical infrastructure and international partners to assess the impact and assist with remediation efforts.”
But CISA noted that it has “observed threat actors leveraging this incident for phishing and other malicious activity.” The cybersecurity agency did not provide details but warned organizations to remain vigilant.
Malicious actors could take advantage of the confusion and chaos to launch their own cyberattacks. Rachel Toback, a social engineering expert and founder of cybersecurity firm Social Proof Security, said in a series of X posts that people should “verify that people are who they say they are before taking sensitive actions.”
“Criminals will be taking advantage of this IT outage to try to pose as IT people to you or pose as IT people to you to steal access, passwords, codes, etc.,” Toback said.
What do we know so far about misinformation?
It's no wonder some people assumed the outage was the result of a cyberattack – sudden outages, blue screens of death at airports, and office computers flooded with error messages creating chaos. As expected, there's already a fair amount of misinformation floating around, with social media sites incorrectly flagging trending topics like “cyberattack.”
Always check official news and sources – if it seems too good to be true, it probably is.
TechCrunch will continue to update this report throughout the day.
TechCrunch's Ram Iyer contributed to this report.