In the Trenches

By: Matthew Otte

Our previous post detailed common pitfalls and ways to avoid them as you develop your organization’s incident response capabilities. This post is intended to address the next steps by covering concepts that may aid you during an incident response (these tips are also great to keep in mind while practicing your response capabilities!). Our hope is that you will never have to use what you learn in this article during a real-world response. In the event you do, these tips just might help you come out on top. Also, don’t forget to explore these concepts while practicing your response capabilities to help take your organization to the next level before an incident occurs.

*Note: This article is not meant to be used as a guide during a real-world incident response. If your organization is in a security incident and in need of help, please contact Soteria’s Detection And Response Team (DART) at (877) 281-3662.

Act Fast, but Know Your Limits

Once a potential incident is identified, the race is on to contain the threat and to return your organization to a functional state as soon as possible. Complete and thorough preparation of your organization’s incident response capabilities, while avoiding the pitfalls mentioned in our previous post, will help to ensure that your team can respond quickly and effectively to a threat. However, organizations that fail to prepare, or ones that do not prepare properly, quickly discover the limits of the response capabilities as we detail below.

At Soteria, most of our incident response calls come in on a Friday afternoon. This is not always because attacks happen during this window. Rather, this is often because an organization has experienced an incident, tried for hours or days to resolve it on their own, and only decided to engage experts as the weekend approaches. Why do organizations wait until the last minute to get help? Well, they certainly are not ignoring the issue in the hope that it will go away. In many cases, the security and/or IT team is performing an incident response for the first time and attempting to learn as they go. For others, the organization has relied upon a third party who cannot perform forensics or incident response. Finally, some organizations are reluctant to incur the cost of professional services.

Failure to realize the limits of your incident response capabilities can have devastating effects beyond extending the incident response timeline. While your organization is stagnant, your attacker is not. They will continue to pursue their objectives or find other means to monetize their position if their primary objective is already met; all while your organization may be crippled from the effects of the compromise. Further, the improper use of forensics tools and techniques could destroy evidence present on the affected machines and hinder forensics analysts’ capability to timeline the threat actor’s actions. Finally, ill-advised response actions could alert a threat actor to your knowledge of their presence within the environment. This could shift their timeline and drive them to immediately begin destructive actions within your environment.

The issues discussed above can be addressed by acting quickly and knowing when to get help. There is no shame in knowing where your gaps lie (see our previous post to help identify them during your preparation) and preventing them from destroying your organization. Your organization’s responsibilities to your clients, users, and regulatory bodies are not subject to the quality of your incident response capabilities. This means your burden is not lessened simply because an under-qualified response team failed to identify the entire scope of the incident.

Control Your Bias

When responding to an incident, it is easy to allow bias to dictate decisions and actions. This mistake can manifest itself in a number of ways, and frequently hinders investigations when it is not properly controlled. Our incident responders work hard to suppress this bias and identify bias affecting others’ decision making during engagements. Familiarizing yourself with what this practice looks like can help you avoid it within your own incident response.

A good incident responder will perform evidence chaining, where they pivot from each individual artifact to uncover a timeline of threat actor actions. This process is simple when ample evidence is available but becomes much more difficult when there is limited evidence or bias starts to take hold. In these cases, individuals will ‘lead the evidence’ instead of allowing it to lead them. For example, a responder may begin to make assumptions about the case that match previous cases they have worked. This same type of bias is seen when a network admin assumes the compromise came from a known vulnerability in the environment. This is not to say that one cannot reference previous experience or examine hunches, but they should always be enforced by evidence before committing to them.

Another common form of bias is denying the possibility of events based on perceived probability. This usually takes the form of something like business owners believing they are too small to be targeted, security personnel believing a solution is foolproof, or responders dismissing the testimony of victims. Regardless of how it manifests, biases nearly always result in the same outcome. Time and money are wasted chasing the ‘white whale’ bit of evidence to prove your case while the truth is hiding in plain sight. To avoid these mistakes, keep an open mind while investigating and be wary of instances where individuals, including yourself, start to make unrealistic stretches to have the evidence support their assumptions.

Be Honest

Security events, incidents, and breaches can be very emotionally challenging. Often, it is assumed that an individual’s involvement in an incident that is potentially damaging to the organization will ultimately lead to their dismissal. Though this mindset is not unfounded, fear-driven response is often detrimental. For example, a user getting phished could withhold their knowledge of the attack in hopes that they are not reprimanded for their unintentional involvement.

These fear-driven responses are not an easy thing to overcome. This is especially true in organizations that do not have the cultural groundwork established to foster honesty in their teams. Fortunately, there are ways to ensure individuals that being forthcoming with information should be the priority. It is important to remember that victims of social engineering or other techniques are just that: victims. They often feel guilty, embarrassed, or ashamed and layering on threats of further punishment is typically not helpful in resolving an incident. However, if there is reason to believe the individual was intentionally involved or practiced gross negligence, the appropriate organizational policies should be adhered to for these infringements in place of attempting a technique below.

Many organizations elect to “double-down” on the fear response associated with the individual’s involvement in the incident. This is done by informing personnel that the investigation will ultimately uncover the details of the incident, and anybody potentially involved can limit their punishment by being forthcoming ahead of the investigation’s results. The hope is that fear of a greater punishment inspires victims of the attack to come forward. You may also elect to offer amnesty or incentive to coax victims to share their knowledge. In this technique, the organization promises to forgo punishment or offer a reward for any knowledge that assists in the response.

The importance of honesty does not end with end users as potential victims. Incident responders are not without their own faults, and often have made mistakes that lead to the compromise or hinder its response. Remember that the same honesty you expect from your end user should be reciprocated by you, especially given that the information you may be withholding could have a much larger impact on the response effort.

Do Not Focus on Blame

As previously stated, an incident response will often be a very emotional event. In the heat of the moment, many responders find themselves assigning blame for aspects of the compromise as well as hindrances to the response. This type of behavior should be avoided for many reasons. Primarily, the effort placed into assigning blame does nothing to further your response or retroactively prevent the compromise from happening. The entire response team should maintain a dedication to the effort to limit negative effects to the organization from the incident.

Not only does focusing on blame result in misplaced energy, but it may also sow seeds of dissent throughout the ranks of your incident response team. Individuals who feel they are unjustly given blame for a compromise or hindrance to the response may disengage from the team when their role is vitally needed. Cohesion should be prioritized above blame.

Finally, hyper focusing on blame creates a toxic culture in which individuals are much less likely to be honest in the future. As previously stated, individuals fearing repercussions for their mistakes may hinder a response by withholding information. Avoiding the tendency to focus on blame is the first step to steering your organization away from this negative culture.

Once a response is complete, a debrief should be held. During the debrief, the opportunity to discuss fault should be allowed. It is vital to ensure that the focus of discussing fault is still not aimed at chastising the individual or assigning blame, but rather allowing any parties who faltered to identify what caused the mistakes. The goal is to improve your overall response capabilities and all parties involved should remain humble. Given a long enough timeline, every member of your response team will make a mistake.

Prioritize the End Goal

The final tip for success during a response effort is to prioritize your end goal. Many incident response engagements carry long hours and can quickly become exhausting to both resources and personnel. This is especially true with severe incidents that involve many parties. Often, organizations will find themselves continuing to drudge through these exhaustive engagements long past the point of diminishing returns. This is a mistake that is both financially costly as well as inflicting a toll on morale.

Organizations who find themselves in unnecessarily extended engagements typically do so because of a failure to prioritize end goals. For example, rather than containing, eradicating, and recovering from an incident, focus may shift to implementing corrective actions while simultaneously engaging in the response. We see this often when the leadership of a response team begins allocating their resources toward something like improving a vulnerability management or patch management program discovered to be a contributing factor for the incident while an investigation is still ongoing. This misguided prioritization often results in an extended incident response engagement and key personnel being removed from the response effort to engage in less vital tasks. Further, a morale drop usually follows as a high stress engagement begins to blur with everyday operations.

To prevent this pitfall, response team leads should be focused on containing, eradication, and recovering as quickly as possible. Maintaining this finite goal ensures that scope creep does not begin to expand the number of tasks, creating what feels to be an endless response. Only after an incident response is complete should effort be dedicated to implementing the corrective actions for contributing factors identified during the response that were not immediately addressed as part of the containment, eradication, or recovery efforts.

Our next post will cover tips to keep in mind while wrapping up your incident response. These tips can also be applied after practicing your response capabilities to help ensure you get the most out of that practice.