Carding Forum
Professional
- Messages
- 2,788
- Reaction score
- 1,176
- Points
- 113
The recent global outage, which affected several million Windows computers worldwide due to Crowdstrike, is probably not the first episode in the history of an information security vendor that calls into question the quality of its services. According to some reports, a similar, but less resonant incident in the spring of 2024 affected the infrastructure of the company's clients running Linux.
Repeating your own mistakes
It is not the first time that Crowdstrike, an American cybersecurity software provider, has released an update to its Falcon Sensor product, which leads to the failure of customer infrastructure, TechSpot writes.
Recall that on July 19, 2024, there was a global failure of information systems in various industries –because of it, in particular, some large American and European air carriers were forced to completely or partially "land" their air fleets. As it turned out, the problem was hidden in the updated version of the Falcon Sensor application of the information security company Crowdstrike, which caused the "blue screen of death" (BSoD) of Microsoft Windows.
Debian Server Incident
According to a Hacker News portal user under the pseudonym JackC, serious accidents caused by the spread of previously untested Crowdstrike updates have happened before. Moreover, he was personally involved in dealing with the consequences of an incident of this kind.
The user did not specify the name of the organization in which he works, only explaining that this is a technical laboratory belonging to the civil sector of the economy and managed by a large company. It was at the initiative of the management of the latter that the function of protecting the "motley" infrastructure of an organization running one of the most popular Linux distributions, Debian, was decided to be assigned to Crowdstrike software.
On April 19, 2024, Crowdstrike released the Falcon Sensor update, which turned out to be incompatible with the current version of the Debian Linux stable branch used in the JackC organization, so its specialists had to "patch" the OS, as they did in similar situations many times. At first glance, the procedures for applying the patch and updating the CrowdStrike software went completely smoothly and no anomalies were observed in the system. However, a week later, quite unexpectedly for administrators, the entire fleet of Linux machines in the organization "fell" at once, and attempts to make the failed computers work by rebooting were in vain.
Understanding the reasons for the failure, the company's specialists connected one of the system disks of the affected Linux machine to another computer and studied the logs (logs of system operations). Analysis of the logs revealed that the alleged culprit is the Crowdstrike software. As a result, it was decided to remove it from one of the affected servers in order to test this hypothesis. Deleting it really helped – the computer immediately booted up successfully without any problematic software. Repeated installation of Crowstrike software in the future again led to failure.
Interaction with the Crowdstrike support service
Only after making sure that the source of the problem is actually the Crowdstrike software, the company's specialists contacted the information security vendor's support service via the Internet. The response was received a day later-it offered to provide additional evidence that the failure in the organization was caused by incorrect operation of the Falcon Sensor due to the vendor's fault. As noted by JackC, the next day the support service recognized the fact that there was an error in the product. A week later, the vendor reported that the next version of Falcon Sensor was not tested for compatibility with the configuration of the software running the client's infrastructure.
As a result of the trial, JackC and colleagues came to the conclusion that it is not possible to prevent the occurrence of similar incidents in the future due to the vicious approach to delivering updates chosen by Crowdstrike, which he described with the following phrase: "We put software on your machines whenever we want, regardless of the degree of urgency, without testing it."
It is noteworthy that, as in the case of the recent global failure, the release of the "problematic" update of the Crowdstrike product took place on the eve of the weekend-on Friday, April 19, 2024.
Compatibility issues with RHEL and derivatives
The story of JackC, who does not specify his place of work, can and should be treated with a certain degree of skepticism – after all, the details of the incident are known only from his words. However, there are other facts that indicate that the CrowdStrike software update on Linux machines, released in the spring of 2024, may have caused serious failures.
For example, the Red Hat portal knowledge base mentions the possibility of a critical OS error (kernel panic) Red Hat Enterprise Linux (RHEL) version 9.4 when loading the 5.14.0-427.13.1.el9_4.x86_64 kernel caused by the falcon-sensor process. Users of Rocky Linux 9.4 (one of the many forks of RHEL) also complained about similar problems on the official forum of the project in mid-May 2024.
Intruders do not doze off
On July 19, 2024, the US Cybersecurity and Infrastructure Protection Agency (CISA) called on organizations affected by the Crowdstrike software incident to be particularly vigilant. According to the agency, against the background of the chaos that has arisen, cybercriminals have become more active, sending phishing emails on behalf of Crowdstrike specialists, which offer to "fix" failed systems for a certain amount in cryptocurrency.
Meanwhile, Crowdstrike released a fix for Falcon Agent, and Microsoft Corporation on July 20, 2024 offered a free tool that allows you to "fix" Windows machines that were disabled as a result of an incident by booting from a USB drive.
Repeating your own mistakes
It is not the first time that Crowdstrike, an American cybersecurity software provider, has released an update to its Falcon Sensor product, which leads to the failure of customer infrastructure, TechSpot writes.
Recall that on July 19, 2024, there was a global failure of information systems in various industries –because of it, in particular, some large American and European air carriers were forced to completely or partially "land" their air fleets. As it turned out, the problem was hidden in the updated version of the Falcon Sensor application of the information security company Crowdstrike, which caused the "blue screen of death" (BSoD) of Microsoft Windows.
Debian Server Incident
According to a Hacker News portal user under the pseudonym JackC, serious accidents caused by the spread of previously untested Crowdstrike updates have happened before. Moreover, he was personally involved in dealing with the consequences of an incident of this kind.
The user did not specify the name of the organization in which he works, only explaining that this is a technical laboratory belonging to the civil sector of the economy and managed by a large company. It was at the initiative of the management of the latter that the function of protecting the "motley" infrastructure of an organization running one of the most popular Linux distributions, Debian, was decided to be assigned to Crowdstrike software.
On April 19, 2024, Crowdstrike released the Falcon Sensor update, which turned out to be incompatible with the current version of the Debian Linux stable branch used in the JackC organization, so its specialists had to "patch" the OS, as they did in similar situations many times. At first glance, the procedures for applying the patch and updating the CrowdStrike software went completely smoothly and no anomalies were observed in the system. However, a week later, quite unexpectedly for administrators, the entire fleet of Linux machines in the organization "fell" at once, and attempts to make the failed computers work by rebooting were in vain.
Understanding the reasons for the failure, the company's specialists connected one of the system disks of the affected Linux machine to another computer and studied the logs (logs of system operations). Analysis of the logs revealed that the alleged culprit is the Crowdstrike software. As a result, it was decided to remove it from one of the affected servers in order to test this hypothesis. Deleting it really helped – the computer immediately booted up successfully without any problematic software. Repeated installation of Crowstrike software in the future again led to failure.
Interaction with the Crowdstrike support service
Only after making sure that the source of the problem is actually the Crowdstrike software, the company's specialists contacted the information security vendor's support service via the Internet. The response was received a day later-it offered to provide additional evidence that the failure in the organization was caused by incorrect operation of the Falcon Sensor due to the vendor's fault. As noted by JackC, the next day the support service recognized the fact that there was an error in the product. A week later, the vendor reported that the next version of Falcon Sensor was not tested for compatibility with the configuration of the software running the client's infrastructure.
As a result of the trial, JackC and colleagues came to the conclusion that it is not possible to prevent the occurrence of similar incidents in the future due to the vicious approach to delivering updates chosen by Crowdstrike, which he described with the following phrase: "We put software on your machines whenever we want, regardless of the degree of urgency, without testing it."
It is noteworthy that, as in the case of the recent global failure, the release of the "problematic" update of the Crowdstrike product took place on the eve of the weekend-on Friday, April 19, 2024.
Compatibility issues with RHEL and derivatives
The story of JackC, who does not specify his place of work, can and should be treated with a certain degree of skepticism – after all, the details of the incident are known only from his words. However, there are other facts that indicate that the CrowdStrike software update on Linux machines, released in the spring of 2024, may have caused serious failures.
For example, the Red Hat portal knowledge base mentions the possibility of a critical OS error (kernel panic) Red Hat Enterprise Linux (RHEL) version 9.4 when loading the 5.14.0-427.13.1.el9_4.x86_64 kernel caused by the falcon-sensor process. Users of Rocky Linux 9.4 (one of the many forks of RHEL) also complained about similar problems on the official forum of the project in mid-May 2024.
Intruders do not doze off
On July 19, 2024, the US Cybersecurity and Infrastructure Protection Agency (CISA) called on organizations affected by the Crowdstrike software incident to be particularly vigilant. According to the agency, against the background of the chaos that has arisen, cybercriminals have become more active, sending phishing emails on behalf of Crowdstrike specialists, which offer to "fix" failed systems for a certain amount in cryptocurrency.
Meanwhile, Crowdstrike released a fix for Falcon Agent, and Microsoft Corporation on July 20, 2024 offered a free tool that allows you to "fix" Windows machines that were disabled as a result of an incident by booting from a USB drive.
