When you’re under attack, don’t lose sight of the big picture
Cyber incidents, especially those which have a substantial operational impact on an organisation – such as a ransomware attack – can be extremely stressful and chaotic events. Of course, preparation is key; the more comprehensive, actionable, and practiced your incident response plans are, the better equipped you will be to navigate a ransomware incident and its minefields. However, even the best prepared teams can develop tunnel-vision during a response. For example, conducting a forensic investigation into the incident is often deprioritised when response teams maintain a singular focus on restoring operational services. And this works to the detriment of the organisation and the overall response.
Indeed, it is a common misconception that the investigation is only helpful in providing a retrospective accounting of events after-the-fact. The reality is, though, that the investigation is a critical enabler for the response and should be undertaken in parallel to the restoration of services.
An organisation’s resiliency against large-scale ransomware events is dependent on four core elements:
When a ransomware event is first discovered, the situation is often extremely fluid, with an overwhelming number of unknowns. The overall impact of the incident is unclear and, adding to the chaos, there is a high level of uncertainty over whether the environment is being actively compromised. The first 12 - 24 hours of a response are critical. They dictate the tempo of the overall response and largely influence its eventual outcome.
The first 24 hours
The first order of business during a ransomware incident is to stem the bleeding. This often involves disconnecting all assets, including critical ones, from the network. Doing so within this timeframe is paramount for two reasons:
- It helps prevent unimpacted assets from being infected.
- It provides response teams with the opportunity to perform a thorough impact assessment on the environment without having to worry about a continued compromise.
But these actions in themselves raise legitimate cause for concern. By doing the above, you are in effect causing a business interruption. Yet there is a logic at play here. The interruption you are causing to perform the assessment is a calculated risk. When done effectively, it will avert the even longer interruption to the business caused by clean assets being impacted due to lack of action. Naturally there are always exceptions, and in some cases, it may simply not be feasible to cause any interruption of business, especially if doing so poses a potential risk to life and safety. At the end of the day, the decision to “unplug” or not has to be taken with all the associated risk factors in mind. The better prepared you are ahead of an incident, the better equipped you will be to make these difficult decisions in a time-sensitive environment.
A well-prepared organisation will enact its incident response plans within the first 24 hours following the detection of an incident. As part of these plans, the incident response team should be assembled, with a clear mandate and well-defined roles and responsibilities. Stakeholders for a ransomware incident expand well beyond the technical; senior leadership participation is crucial to make critical decisions quickly, input from across business units is important to establish the restoration plan, and legal representation is key to ensure that legal and regulatory obligations are met.
Once convened, the incident response team’s priority is to conduct a thorough assessment to gain a clear picture of which assets are impacted. Backups should crucially form part of this assessment. It is very common for ransomware operators to target backups for encryption or deletion as part of their attack. Doing so effectively cripples restoration activities, increasing the threat actor’s leverage when negotiating for a ransom payment with their victims. Evaluating the availability and integrity of recent backups for impacted assets is therefore a key priority during the initial response.
After 24 hours
At this stage, the impact assessment has been completed and the state of backups confirmed. Now the incident response team should establish a clear order of priority for the restoration of services. In essence, this translates into a checklist of activities that need to be completed to bring targeted systems back online and connected to a clean network.
An IT team’s competency and capacity play a crucial part in this process. During restoration, systems and services do not recover seamlessly, with Murphy’s Law ever present in these cases! The better equipped the IT team is to troubleshoot and resolve the inevitable issues that arise during the restoration process, the shorter the restoration timelines are and the lower the potential business interruption.
The best-case outcome in a bad ransomware scenario sees a victim organisation have unimpacted backups with minimal data loss that are readily available, and crucially, easily restorable. If this is not possible, there are two less ideal alternatives:
Rebuild systems from scratch, with substantial or total data loss.
Negotiate and pay a ransom amount for a decryptor tool to salvage data.
Losing substantial amounts of data can be operationally crippling and virtually unrealistic. Operations simply cannot be restored in certain cases without the availability of historic data. Rebuilding systems from scratch is a time-consuming and labour-intensive process, often taking weeks to complete with substantial financial costs and potential financial impacts.
Negotiating and paying ransoms and receiving decryptor tools is also a lengthy process. It is not uncommon for it to take a week or more before an organisation gets a decryptor tool in-hand. There are additional considerations to keep in mind when entertaining this option. For example, paying a ransom can have reputational, regulatory, and legal ramifications, and even with access to a decryptor, there is no guarantee that all data can be salvaged. Additionally, decryptor tools only unlock data; they do not sanitise assets.
Systems will still need to be rebuilt to ensure a safe restoration, and decrypting large amounts of data can take several days. All these factors extend the timelines of a response even further.
In practice, organisations often adopt a restoration approach that combines the above scenarios out of pure necessity. Often, backups are only available for a subset of critical systems and data. In such cases, organisations may elect to restore backups where they can, negotiate and obtain a decryptor tool to restore other critical services, and rebuild tertiary systems from scratch. No matter which option or combination of options is taken, restoring services is not an instantaneous process. In most cases, it may take an organisation several days to restore critical services, and perhaps weeks to restore full operations.
It is imperative that response teams incorporate forensic acquisition activities into the restoration process. Getting this right is a balancing act, and often requires a high level of coordination and communication between IT and forensic teams. As the IT teams move to restore systems, they should be working in tandem with the forensic teams to ensure that the required forensic artefacts are collected prior to restoration. These artefacts could include live forensic collections, which tend to be relatively quick endeavours, or full disk images, which could take several hours to complete for a single system. However, the forensic aspect of a response cannot be overstated. Performing a forensic investigation should be an organic part of the response, and provides an organisation with the opportunity to surface the root cause of the incident, Indicators of Compromise (IOCs) used by the threat actors, and whether the threat actor accessed or exfiltrated sensitive data.
Establishing the root cause of an incident is important. Doing so allows for appropriate measures to be taken as part of the response to ensure any open vectors of attack are addressed. Blocking threat actor IOCs and monitoring for their presence in the rebuilt environment is also key to preventing a reinfection of the environment. Finally, understanding threat actor activities within the environment, particularly whether they accessed or exfiltrated sensitive data, is critical. This dictates whether organisations have any notification obligations to the public, regulators, or law enforcement. Failure to notify when a requirement is present can be very costly, reputationally and financially. External law firms specialising in data breaches are exceptionally well-equipped to provide the appropriate legal advice in such cases, and an organisation would be well-advised to seek their engagement early on in the process.
In our experience…
Ransomware events are by nature an extreme stress-test of an organisation’s people, operational resilience, and response capabilities. Restoring services from this shock to the system is neither an easy nor quick task. Even in the best of cases, some level of business interruption is to be expected. An organisation’s level of preparedness and the initial first hours of the response are key factors that determine the duration of that business interruption.
Our response teams have witnessed both sides of the spectrum: we’ve seen unprepared organisations face business interruptions to critical services for weeks. But we’ve also seen those with robust incident response plans and a backup strategy for critical systems mitigate the damage and get back up and running quickly.
Finally, having the right external partners can be a force multiplier for an organisation’s resilience. Experienced breach coaches and digital forensic and incident response firms are faced with these situations on a day-to-day basis. Leveraging their expertise can help an organisation stem the bleeding, restore services quicker, and limit their legal risks substantially. Ultimately, a ransomware incident is never a good day, but with the right elements in place, you can minimise the impact on your organisation and get through to the other side of it.