Understanding the CrowdStrike Crash and Its Implications
In July 2024, one of the most significant and widespread disruptions in the cybersecurity world unfolded and impacted countless enterprises reliant on CrowdStrike for a number of their security needs. The issue, a seemingly simple code glitch in a recent CrowdStrike patch, led to widespread blue screens of death (BSODs) on Windows systems and caused considerable downtime and operational challenges for any device that downloaded and installed the update. As businesses scrambled to restore functionality, the incident underscored the critical importance of robust IT strategies and potential vulnerabilities inherent in automated cybersecurity updates.
So, what can you and your team do to avoid an unforeseeable issue like this in the future?
The Unavoidable Nature of Vendor-Side Issues
The CrowdStrike crash highlights an unfortunate reality of modern enterprise IT: vendor-side issues can sometimes be unavoidable. Most businesses, for simplicity and efficiency, configure their cybersecurity updates to apply automatically. Even at the enterprise level, many organizations don’t have the dedicated IT bandwidth and resources to test and manually apply updates, especially in a cyber world where day-zero vulnerabilities can result in massive threats to operational integrity.
Conventional wisdom holds that standardized same-day, automatic update models can help ensure systems are protected with the latest security measures without requiring manual intervention. However, the recent CrowdStrike patch glitch demonstrated how even minor coding errors can have significant repercussions, including but certainly not limited to lasting reputational damage and residual downtime.
The immediate fix for the CrowdStrike issue involved restoring systems to a previous restore point, removing the problematic patch, and then reapplying the corrected version. This procedure was relatively easy for single-device users, but what about enterprises with tens of thousands of devices that aren’t stored in-house?
While the fix was developed and deployed quickly and the remedy was seemingly straightforward, the incident itself was a stark reminder of the potential risks associated with automatic updates and the need for robust contingency plans. The troubling reality is that automatic updates cannot be blindly relied upon.
Lessons Learned and Best Practices for Future Updates
One key takeaway from the CrowdStrike incident is the importance of not immediately implementing updates across an entire organization. Instead, businesses should consider a phased approach to rolling out patches–a more time-consuming process, but with important safety benefits.
By waiting a few days or even weeks to observe if other organizations encounter any issues, companies can avoid potential widespread disruptions. A waiting period allows time to identify and address any glitches or bugs in the update before it is broadly applied.
Additionally, companies can adopt a more ad hoc “guinea pig” strategy of initially applying updates to a limited number of devices to test the patch's efficacy and stability. If the initial batch of devices operates without any noticeable problems after a defined testing period, the update can then be deployed across the organization with much greater confidence.
Either strategy can significantly mitigate the risk of encountering severe problems like the BSODs caused by the CrowdStrike patch, even if it requires more short-term IT proactivity and precaution. If Delta Airlines could go back in time, assuredly the airline would have taken one of the above paths before rolling out the patch worldwide.
Special Notes
In this particular incident, CrowdStrike bypassed the safety checks customers had in their environments. The underlying system that examines the OS kernel for nefarious signatures or behavior was updated regardless of client/user/organization portal update settings. Because the CrowdStrike update bypassed “automated phased update” safety settings, update processing would have proceeded.
So consumers of CrowdStrike had no way to stop this from happening.
Derive's Proactive Response and Client Support
Derive Technologies' healthcare clients were, fortunately, able to weather the CrowdStrike crash more effectively than many others, thanks to immediate on-the-ground assistance at critical hospitals and healthcare centers by its Incident Response Teams, ensuring their essential services quickly regained full operational capacity. This hands-on support was crucial in minimizing downtime and restoring functionality as swiftly as possible to organizations that absolutely cannot afford to be offline.
Moving forward, Derive will conduct extensive post-mortem incident analyses to better understand the various factors that influenced recovery times and success rates. Armed with this data, Derive IT service and support experts will continue to hone best practices for handling similar issues in the future, ensuring clients remain resilient in the face of unavoidable cybersecurity challenges.
The Importance of Careful Updates and Expert Support
The CrowdStrike crash incident underscores the reality that while cybersecurity is more important than ever, equally critical is the need to approach system and software updates with caution.
Organizations should carefully control the process of automated updates as part of their overall IT risk mitigation strategies. Partnering with a certified IT services and support provider like Derive Technologies can provide the expertise and guidance needed to navigate these challenges effectively, when and wherever they occur.
By employing phased update rollouts and with access to immediate, hands-on software engineering support in critical situations, businesses can enhance their resilience and ensure their IT infrastructure remains robust and secure. In an ever-evolving digital threat landscape, these actions are paramount to maintaining business continuity and reputation, operational integrity, and protecting sensitive data.
To learn more about how these IT solutions and IT consulting services can benefit your business, CONTACT US TODAY.
blog
