One of the most interesting challenges of working as Chief Information Security Officer in a utility company is the variety of infrastructure types that supports the business process. I refer here to the infrastructure that supports real-time management systems for generation transmission and distribution of energy and the system that are responsible for coordinating the pumping of water to individual households and industries.

The implementation of a information security management system that includes this kind of critical infrastructure to the business processes provides a number of interesting challenges which are not covered in the conventional security model for IT processes:

• Information Security risks associated to the delivery process of energy and water utility services process, can lead to disruption of both services for a large number of people in a country. If errors in the handling of SCADA equipment have been responsible of cascading effects that collapse most of the electrical system of a country, what if someone is doing an identity theft in the energy SCADA system and performs tasks such as increasing the rotation of the generation turbines, increasing the energy flow exceeding the capacity of a transmission line or simply turning off the turbines of a power plant? Imagine the chaos that would plunge a country or region.
• What if in the water tanks of a city begins to overflow its maximum level and the pressure causes the pipes bursting in the streets? Imagine scenarios like the following in every city: http://www.youtube.com/watch?v=kbz_zxsJCfg&feature=related
• The cost of repairing damage of any of the above scenarios is enormous. If we add the inability of the company to generate money for generation, transmission and distribution of energy, how much time passes before the company cease to exist?

SCADA systems have a very particular operating environment. Because they are real-time systems, data monitoring and orders sent to the RTU should arrive in the shortest time possible, since an additional delay of even 10 ms can mean a massive blackout by activation of the protections of a substation. Similarly, suppliers of these systems tend to provide support on these only on a specific configuration, which is usually not too safe and lacks basic security controls such as security patches, data encryption, authentication and non default configurations.

 The architecture for a SCADA system is as follows:

The components are:

• Remote Terminal Unit (RTU): The RTU is defined as a communication device within the SCADA system and is located at the remote substation. The RTU gathers data from field devices in memory until the MTU request that information. It also process orders from the SCADA like switch off a transmission line.
• Master Terminal Unit (MTU): The MTU is defined as the heart of a SCADA system and is located at the main monitoring center. MTU initiates communication with remote units and interfaces with the DAS and the HMI.
• Data Acquisition System (DAS): The DAS gathers information from the MTU, generates and store alerts that needs attention from the operator because it can cause impact on the system. 
• Human Machine Interface (HMI): The HMI is defined as the interface where the operator logs on to monitor the variables of the system. It gathers information from the DAS. 

 Due to its criticality, SCADA operators are reluctant to implement any type of information security controls that can change the operating environment for the system. How to implement a security scheme that does not interfere with the functionality needed for the business process? We took the following items specified in the standards of North American Reliability Corp (NERC) Critical Infrastructure Protection (CIP) to implement controls for an Energy SCADA:

Project 2008-06 Cyber Security — Order 706

• CIP–002–2 — Critical Cyber Asset Identification
• CIP–003–2 — Security Management Controls
• CIP–004–2 — Personnel and Training
• CIP–005–2 — Electronic Security Perimeter(s)
• CIP–006-2a — Cyber Security — Physical Security
• CIP–007–2 — Systems Security Management
• CIP–008–2 — Incident Reporting and Response Planning
• CIP–009–2 — Recovery Plans for Critical Cyber Assets

For point number two, we took the same table to classify information assets for the corporate information security management system and applied it to the energy processes: 

From the previous table, we assigned controls to implement and ensure the security level for the asset. For point 3 and 4 we adopted all definitions from the corporate Information Security Management System. See all the required controls here: http://www.nerc.com/files/CIP-003-1.pdf and http://www.nerc.com/files/CIP-004-2.pdf.

The biggest issue here was authentication and clear-text traffic. Many devices from our SCADA system did not support authentication and also information was sent using cleartext protocols. Every time we tried to introduce a VPN or crypto level-2 devices, the network latency increased and functions of the system were degraded, which is why we had to remove those controls. When we asked our vendor for those controls as native functions for the system, we received a request to purchase the next version of the SCADA System.

The corporate antivirus didn't work because it consumed all the resources of the DAS and the HMI. Same happened with the Host IPS. The solution we found for the problem was SolidCore S3 product (http://www.solidcore.com/products/s3-control.html), as it was non-intrusive, did not add extra layers and virtual devices to the operating system and controlled very good the zero-day problems.

For configuration changes, we established a weekly maintenance schedule in which the service of the SCADA system would stop for three hours changing the operation mode to contingency, so the IT operators could perform screening for viruses, install security patches and modifying security baselines. If the change was not successful and the system is degraded, the changes were removed and tried again the following week. This was not an easy task, because the vendor would not support us and we had to learn a lot on how the system components worked.

For point 5, We tried to redraw the SCADA network so critical traffic would not mix with other type of traffic. For wireless devices, we managed to implement 802.1X authentication.  We divided the SCADA network into the following perimeters:

Cisco Firewall Service Module inside Catalyst 6509 with VSS supervisors (VS-6509E-S720-10G) gave us the required bandwith and no disruptions were presented within the SCADA environment. It also have IPS (IDSM-2) that sends the alerts along with the log firewalls to our RSA envision correlator.

For point 6, all the place has armored doors, CCTV, biometric authentication and security guards patrolling around the physical perimeter.

Now we are able to manage the security controls inside the corporate IT network and the SCADA systems. I still know that I have many things to do to to achieve the other points of NERC, but still will be an interesting and challenging goal.

-- Manuel Humberto Santander Peláez | http://twitter.com/manuelsantander | http://manuel.santander.name | msantand at isc dot sans dot org