By: Matthew Otte

Throughout their many years of practice, Soterians have advised for, participated in, and commanded a large assortment of cyber security incident responses (IRs) in the form of both exercises and real-world events. Though no two response efforts are exactly alike, Soteria has discovered a pattern of common pitfalls which tend to present themselves during IRs and frequently result in disaster. With the Ponemon SMB Cybersecurity Survey showing nearly 60% of small businesses failing after a data breach, developing effective IR practices are vital for small businesses. These shortcomings, however, are not limited to just amateur responders, as many have been witnessed within well established, funded, and practiced teams. 

This post was written to assemble the list of pitfalls and the considerations to be made before, during, and after an IR to prevent individuals and teams from falling victim to those common mistakes. Our hope is that this guide can help prepare you with the confidence that you are making the correct choices. Further, with a direction to begin your journey, this post may assist in transitioning your incident response strategy from simply hoping an incident never occurs to being ready to face the big one. 

Given the many moving parts during an IR, this guide has been broken down into three posts. Our first post covers considerations to be made prior to an IR.

Incident Response Plan and Playbooks

“If You Fail to Plan, You Are Planning to Fail” — Benjamin Franklin

Benjamin Franklin’s quote has been proven right in more IRs witnessed by Soteria than we would hope to see. We’ve watched company after company stand up their war rooms, often for the first time, only to realize they were overwhelmed by the experience. This is where plans and playbooks can help to guide even the most experienced teams to ensure effective and consistent responses are made. Soteria recommends starting your incident response improvement journey with an incident response plan and playbooks, as they will often identify other opportunities for improvement as they are developed.

Incident Response Plan

Your Incident Response Plan (IRP) is the backbone of any incident response you will engage in. Your IRP will dictate the processes and functions that should occur throughout the lifecycle of an incident response, ensuring that the engagement flows as smoothly as possible. With such a pivotal role in developing an incident response capability, it is often overlooked, as many teams have poorly maintained, underutilized, or even no IRP at all. 

Soteria recommends developing an IRP with the following areas of coverage at a minimum:

Development and Distribution – Your IRP should designate the party responsible for developing and distributing the plan. The party responsible should not create the plan alone, as multiple business areas will be involved in the response effort. Representation across these groups is important in the drafting and reviewing of the IRP. Once developed, the party responsible should ensure all incident response team members have access to the plan.

Authority to operate – During an incident response, bureaucracy may delay vital response efforts. The IRP should dictate the authority the teams will operate under (Soteria recommends the CIO/CISO, or equivalent). The finalized IRP should be approved by this individual.

Designation of core and extended incident response teams (IRT) – Incident response efforts expand beyond the scope of just cyber security and IT teams but will not involve all company personnel. The specific job roles required during a routine response, and potentially needed for larger engagements, should be identified.

Defined communication protocols – During the incident response, a large amount of potentially sensitive information will flow up and down stream. Predefining communication protocols ensures efficient communication starts immediately with the appropriate individuals involved.

Defined incident severity matrix – Not all incidents will have the same potential for damage to your environment. Treating every response effort the same is a misuse of assets. A severity matrix will help define what level of effort to dedicate to the response. At Soteria, we use a matrix with incident type and value of assets affected to calculate the severity of incidents.

Roles and responsibilities – The IRP should clearly define which role is responsible for the various steps throughout the IR life cycle. The IRP should also define which individuals will fill which roles.

Points of contact (POCs) – The IRP should include contact information for all pertinent internal and external resources associated with the response effort.

While building your IRP, be sure to avoid the following common mistakes:

Single points of failure – Assigning roles or procedures to a single point a failure may cause an entire response effort to fail. Effort should be made to ensure alternates are available in any feasible applications.

Superfluous text – A long, convoluted IRP is a plan that is likely to be left on the shelf. The IRP should not go into excessive detail or cover topics not appropriate for the response effort. 

Lack of charts – RACI charts can aid in the readability of an IRP. Relying solely on text for your plan will decrease the ability of users to quickly understand their role in a scenario where timely response is needed most.

Playbooks

Playbooks are intended to catalog the high-level steps associated with handling specific incidents or completing specific tasks within an environment. When properly developed, playbooks ensure consistent, effective response actions are taken during an incident response. Further, developing playbooks prior to an incident allows thorough consideration for response efforts, which may prevent mistakes from stress or fatigue associated with the incident. Soteria recommends developing playbooks after completing your IRP. Many helpful templates can be found online for various playbooks, but the following common mistakes should be avoided.

One of the more common pitfalls Soteria experiences associated with playbook development is assuming that routine response actions are not worth cataloging in playbooks. To further the error, an organization will instead focus their effort into developing playbooks for unlikely scenarios. For example, a small business may develop an advanced persistent threat (APT) playbook while overlooking one for a routine scenario like phishing. When doing this, organizations are honing their response capabilities for less probable scenarios while allowing common response actions to persist in an inconsistent or ineffective manner. Most organizations would benefit from having playbooks for common threats like phishing, ransomware, general malware, business email compromise (BEC), and unauthorized access. 

Once the appropriate scenarios have been identified for your organization, playbook development should begin. During development, a second common pitfall Soteria has observed is excessive detail in the playbooks. A lengthy document detailing every individual user click required to perform a response action will be left on the bookshelf or will require significant effort to maintain due to changes in technology. Playbooks should instead list the response actions taken during each stage of your chosen incident response methodology without discussing the individual steps required to accomplish it. For example, Soteria’s general malware playbook simply highlights potential containment actions such as endpoint isolation, disabling accounts, or disabling services. The exact steps to accomplishing these actions are known by our technicians and detailed elsewhere if needed. This process should be repeated for all stages of the incident response lifecycle in all chosen scenarios while identifying your organization’s specific processes and technologies. 

Identify Key Players

Many organizations assume that a cyber security incident response effort is isolated to just the cyber security or information technology teams. This could not be further from reality as most incidents will require input from stakeholders from across the organization. Lack of familiarity with the incident response process across potentially key departments will often result in miscommunications, unknown points of contact, disagreements over hierarchy, and failure to cooperate. All these consequences lead to an overall ineffective incident response engagement and increase the impact to your organization. 

One Soterian managed to witness all the issues mentioned above in one incident response engagement. The organization had not practiced or communicated their incident response procedures and during a ransomware event, experienced many arguments amongst their teams. Specifically, different business units refused to cooperate with the incident lead as they were operating out of a different geographic division. Further, human resources, public relations, and legal were neglected when sharing information about the incident. This resulted in improper identification of employee PII, leaking of sensitive investigation details to clients, and issues with disclosure of breach data.

Soteria recommends identifying the following key player roles within your environment and socializing with the IRP.  Key roles may include:

• Incident Commander 
• Incident Responder 
• Network Administrator 
• Executive Leadership Representative 
• Development Team 
• Legal Counsel 
• Human Resources 
• Finance 
• Corporate Communication

Your organization may contain more or fewer key players as many business models are unique. Considerations should be made to what resources will be required during an incident response engagement and your unique hierarchy while populating this list. 

Practice, Practice, Practice

Practice is a common suggestion for improving incident response capabilities with good reason. A real-world scenario should not be the first time incident response procedures are socialized or attempted. This “trial by fire” method frequently results in a frustrated abandonment of any plans in exchange for a messy ad hoc response. Instead of facing this pitfall during an incident response engagement, effort should be put forth after the previous steps are accomplished to practice and hone the incident response plan and playbooks. Soteria recommends distributing the plan and playbooks to key players, discussing the intended response efforts among all key players, and regularly engaging in a tabletop or purple team exercise to test a finalized plan.

It is well defined that practice is valuable, but not all methods of practice return the same results. Soteria has performed many tabletop exercises in which participants attempt to run the exercise without incorporating their CSIRP or IR playbooks. Any exercise for incident response should be seen as an opportunity to test your plan and playbooks as well as identifying opportunities for improvements.

Another common pitfall seen with practice, mostly in tabletop exercises (TTXs), is making decisions based on unproven assumptions. Specifically, participants will assume a capability without ever having practiced or tested it. For example, a cloud recovery process that is assumed to be adequate but, when attempting to perform a real-world recovery of many servers, a bandwidth issue is discovered, which drags the process out several days. One can see how this discovery may be devastating during a real-world incident response. 

Many exercise and practice scenarios are also plagued with reinforcement of poor practices. For example, a process may have already been identified as a shortcoming but is still used as the team either has not developed a new process or is not comfortable using the new one. Though there are diminished returns for using exercise time to develop processes, practicing new ones is exactly what an exercise is intended to do. 

Foster Honesty in Your Security Program

Wouldn’t it be wonderful to hear about a successful phishing attempt by the user who was phished rather than discovering the devastating effects within your environment days later? Unfortunately, this is seldom the case as many users fear their mistake may be used against them for disciplinary actions. This leads to an all too frequent occurrence for Soterian investigators where users will endlessly swear to not have performed the actions leading to a compromise, even against digital evidence suggesting otherwise. In the best case, this causes frustration and in the worst scenarios, response efforts are wasted chasing leads that do not exist.

Soteria has had the privilege of working with several organizations where these challenges do not exist. In these unique organizations, a culture has been developed that focuses on corrective actions and not assigning blame. In a specific incident recently handled in one such organization, multiple users reported having been potential victims of a phish. Responders were able to investigate and identify a user whose account was accessed by an unauthorized actor. Prior to any threat actor actions on objective, responders were able to scope the incident to identify all affected parties and issue password changes for their accounts. This successful response can be attributed to the focus on correcting the problem without blame and treating incidents as training opportunities instead of a catalyst for disciplinary actions.

Once this culture has been established within your organization, users can be seen as an extension of your security program rather than what your security program is working against. Rewarding users for their active participation in improving the security program may have further benefits, such as  highlighting potential security controls that are being bypassed or are not effectively implemented.