Incident Response Planning: The Good, Bad, and Ugly

Brian Gardner, Ph.D., Chief Technology Officer & Information Security Officer, City of Dallas

Brian Gardner, Ph.D., Chief Technology Officer & Information Security Officer, City of Dallas

The reality in today’s world is, Municipalities are constantly facing security threats, cyberattacks, natural disasters, and other unpredictable events that can impact their operations, including in technology. It has become increasingly important for municipal government to have incident response planning in place and tested. A well-defined response removes a portion of chaos that surely will ensue during the event. Roles, responsibilities, and procedures all reduce the stress at the moment for your team members and improve ever-critical communication between your groups. So correct information can flow quickly and seamlessly between the teams and up to management. Misinformation can be just as effective at causing damage to the containment and recovery efforts.

I see the need for having an IRP as critical for several reasons. Firstly, there is a direct need to minimize the impact of an incident by providing a clear and organized response. When an incident occurs, having that plan in place helps your response teams to quickly identify and respond to the issue, reducing the blast radius of an incident. This prevents further damage, escalation, and hopefully reduces time to recovery. Which in municipal government disruption to critical infrastructure and public safety operations.

More importantly, it improves the efficiency of incident response efforts and forensic investigations. A well-defined plan can ensure that the response efforts are coordinated and consistent, and that all key players function as a unit. Preventing confusion and miscommunication and can ensure resources are allocated effectively. Thereby outlining the steps to be taken in response, coordinating the efforts centrally and preserving the evidence to understand the entirety of the event.

While having an IRP in place is important, I believe it’s equally important to regularly test and update the plan. Inevitably even a well thought out and tested plan will have failures along the way. However, sheer repetition will work in your favor, making your response team’s efforts easier, if there is such a place during a crisis event. Additionally, testing can work to identify some of those pesky gaps or weaknesses in the plan. Although like I stated, without a doubt, all plans are prone to failure along the way. The purpose is not to include every possibility, the purpose is to minimize the impact and hopefully provide a better outcome.

" Preventing confusion and miscommunication and can ensure resources are allocated effectively "

At the time, incidents can be one of the most stressful moments in any cyber-tech’s career. After the event, that knowledge provides you with some of the best information around, not only about the type of incident, but the organization’s culture. By experiencing and overcoming a crisis, you and your teams develop a stronger sense of teamwork and camaraderie. This builds a culture of preparedness within the organization, where team members, management, and the organization are more likely to take incident response planning seriously and proactive. There is no substitute for living through an actual event.

Although you may feel a stigma around events, the reality is every organization will experience one or more. I personally look at staff who have worked through security incidents as invaluable assets to an organization’s incident response efforts. These team members have firsthand experience dealing with the challenges and complexities of a security incident. They provide a level of expertise and knowledge that you cannot replicate through training, testing, or simulations alone.

The bad and ugly to any incident is in the moment, most will be difficult, painful, and no doubt stressful. The real takeaway and good is once the event has concluded, you, your teams, and the organization as a whole will learn exponentially. I personally seek out those that have that experience. Just as you would want an experienced surgeon, having those with a deeper understanding of the nuances and complexities of incident response provides real value. There is no substitute for experience to build confidence and trust.

A well-written and tested incident response plan, along with raw experience are critical factors in the modern cybersecurity landscape. Unfortunately, the reality is municipalities are huge targets, with no relief in sight to the growth of cyber[1]incidents. I recommend to become better at prioritizing incident response planning and investing in experience, and take every opportunity to perform a lesson learned. This will be the most valuable information you will find in your cyber career because protecting is only the beginning, response and recovery will be inevitable.