The topic for today is how to perform containment on a 'mision critical' service or system that your organization depends on, and cannot be shut down? There are many examples of systems that are in production, and under normal circumstances an operational team can't pull the plug. Web servers that are the primary sales vehicle, legacy systems that no one understands,  the backend database that has EVERYTHING on it, the email servers, and the list goes on. In the event that an incident occurs on a normal system normally we can pull the plug to stop the bleeding and move on. However, if downing that box will be career limiting, and allowing the incident to continue is almost as bad, what to do?

I'll summarise suggestions here.

Francois write in "This doesn't speak directly to security incidents, but ... This situation showcases a vulnerability on the business side, and highlights a flaw in business thinking. The vulnerability is neglecting to plan. Incidents like this are the reason we have disaster recovery and business continuity plans. Downtime is inevitable, the only controllable factor is how we respond.

We all see systems fail for one reason or another. Mostly these issues aren't security related. The flaws in business thinking: business continuity and disaster are the sole responsibility of security or IT. From the business owner's standpoint, it doesn't matter why systems go down. The business concern is staying in business. Which means that concern needs to be addressed by business people. It's not just for the security or IT department.

If a system is 100% business-critical with no downtime allowed, the business owners need to understand the ramifications and risks. You wouldn't build your office on a flood plain. You wouldn't trust your accounting books to a known crook. Neither should you risk your infrastructure - financial, IT or otherwise.

A businessperson doesn't need to be an IT expert to understand risk. They just need to understand that the risk is there, and solid risk management is needed.

In my view if a business can't manage risk, it doesn't matter where the risk falls, it's a flaw on the part of the decision makers - not the responder. Security staff is there to help manage those risks, not to absorb bullets."

From my own perspective and experience, containment has always been a business decision. So long as management are aware of the risks of remainin in operation rather than taking the system/network/application down then operational staff must abide by that choice.

Cheers,
Adrien de Beaupré
intru-shun.ca