My next class:
Network Monitoring and Threat Detection In-DepthSingaporeNov 18th - Nov 23rd 2024

When Lightning Strikes

Published: 2010-08-03. Last Updated: 2010-08-04 18:05:09 UTC
by Johannes Ullrich (Version: 1)
11 comment(s)

This weekend, I had a pretty bad lightning strike hit my house. The kind of where you see spark hitting the street in front of the house and your dog jumping in your lap lightning strike. Overall, lightning is a pretty common phenomenon around here. I live in Florida, which appears to be #1 in lightning strikes and casualties in the US [1] . For the 5+ years I live here, the power grid has actually been rather stable during lightning storms, but lately, I had a string of bad luck and would like to share some lessons learned:

So far, I had no damage to equipment completely protected by a UPS/surge protector. I use various types of UPSs, and all performed well so far. Some are rather old and have hardly any battery life left. But they do still work well enough for power spikes/dips as they show up during electrical storms.

The damage I had, in particular in the last storm, affected exclusively network equipment and networking interfaces. I assume that the surge entered the network. I lost two switches and the wired network interfaces in two PCs. Otherwise, the PCs work fine. So far I had not used any network surge protectors, but now started to use the surge protectors provided by the UPS. This appears to work fine, but in some cases, the network now works as "half duplex" and no longer in "duplex" mode. I looked into stand alone network surge protectors for some devices, and it turned out to be a bit hard to find one that supports gigabit ethernet. But they are available. The UPS network surge protection is only supposed to work up to 100 Base-T but synced fine at Gigabit (no duplex).

A thunderstorm a couple months ago, caused some "interesting" damage to my cable modem. I was only able to upload 1MByte in a single connection. This was very weird as it also applied to connections inside VPN tunnels, the cable modem shouldn't really "see" what was happening. But sure enough, swapping the modem fixed the problem. I added a surge protector for the cable line as well. One reason I had not done this before was that I had bad experience with surge protectors and cable modems in the past. But my new cable modem (like many others) provides a status screen and the signal-to-noise number did not suffer significantly after adding the surge protector. The surge protector replaced a simple straight through connector which may have caused a similar loss.

Couple other hints:

- do not plug surge protectors into a UPS. If the UPS runs on batteries it will usually generate a steep sine wave which may destroy surge protectors (in particular tricky to find power strips without surge protector)
- do not plug a UPS into a UPS (same reason as above)
- lightning damage can be subtle. None of my equipment has any visible damage
- proper grounding of all lines entering the house is important (around here, I find that utility companies are pretty good about that)
- once the power is out, turn off the main fuse to the house. But be aware the main fuse can be hard to "flip". Depending on the nature of the outage you may have some surges and unstable power until the damage is repaired (if you want to know when power comes back, just flip all the individual fuses other then one or two that only power lights)

If you consider a backup generator: I looked at many options, but haven't been able to justify one so far. This last outage was 10 hrs long and was by far the longest I have seen. My backup plan is a well charged laptop and a 3G data card to keep me connected. If you consider backup power for a server room, don't forget the AC! For the generators I looked at, the cost to install was almost as much if not more then the cost of the generator. If you do use a portable generator to power individual devices, make sure you do NOT plug the generator into your house wiring before disconnecting the main fuse.

As a quick summary: Surge protectors work. They will probably not save your equipment if the lightning storm rips the electrical wiring out of your walls, but they can help against some pretty nasty strikes. Unplugging your equipment (and WiFi :) ) is better, but not always feasible.

[1] http://www.srh.noaa.gov/mlb/?n=lightning_stats

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter

11 comment(s)
My next class:
Network Monitoring and Threat Detection In-DepthSingaporeNov 18th - Nov 23rd 2024

Comments

I've used these protectors before with 1G FDX.

http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=PNET1GB&tab=models
I suggest a whole house surge suppressor that you mount in your electric panel. They usually run a couple hundred, but can save you a lot in the long run.
Hey Dr J...

You wouldn't be in or near Tampa would you? If not, be thankful...as that part of FL is in fact the lightening strike capital of the world. Several universities use that area for electrical testing of engineering theories.
Great post, but there's a bit of myth floating around, here - there's a HUGE difference between a strike hitting a structure, and the strike hitting a tree/pole that's 40 feet from a structure.

First up, the "surge" from a true strike is ambient. Our old shop had a 100 foot tower attached to the building. It got struck twice, and I was charged with making us survive it. These are the realities:
1. Creating deliberate strike points, and CORRECT grounding of the strike points is key, lest you burn down your building. Having it grounded is not enough; run the cable wrong, and the cable will start a fire (or several fires) inside the walls and attic spaces.
2. When you see plasma flowing along the grids of your drop ceiling, you'll realize that the touting of surge protectors and "ground everything and it'll be fine" is a cute concept.
3. During an ACTUAL strike on the structure, the ambient step potential is several gazillion volts per foot for dozens of yards. Grounding does not mitigate this fact. Unplugging does not mitigate this fact.
4. Your hardware devices will live or die based on their shielding and orientation to the strikepoint/ground path, since every conductor in them is a low resistance path along that step potential. If there happens to be a little silicon in the way, well, there won't be when it's over. Note that we're talking KV per inch within a dozen yards of the strike point OR its grounding cable. Your UPS is not even relevant at this point; the grounding path is a huge inductor; every uncaged conductive sub-path in the area will have some amount of current induced, including inside the chip-level.
5. You will lose things like spare mice and keyboards that are not even plugged in, depending on orientation. And, you'll notice that the survival/loss is consistent with that orientation. :)
6. A faraday cage can work wonders, but only if it is done correctly. Many PCs with a cheap metal case will actually survive in some part, possibly enough to cannibalize. Plastic cased PCs will probably need to be removed from production unless the mainboard was exactly flat along the gradient; if they don't fail outright, they typically will before the month is over. Since most rack mounted devices have metal enclosures, the servers etc typically are ok regardless of the rack type, but connectivity may be lost depending on luck, cable shielding, etc. Fully enclosed (metallic all four sides) racks will generally fare slightly better as far as connectivity. Racks with plastic (or no) doors will typically lose NICs, switches, etc in bulk. As with any production, you already keep a stack of old NICs handy - so if lightning is likely, just keep them in a faraday cage of some type (metal storage box or foil wrap).

For hubs, routers and switches... plastic case = dead device, doesn't matter how you ground/surge-protect it or the Cat5/6.

So, revision of your quick summary:
Surge protectors work fair for NEARBY strikes; they become mostly useless as the strike becomes a direct hit. Mitigation of a direct hit requires a different type of engineering (shielding, etc), since you're dealing with a huge ambient EMF gradient, and induced current, neither of which cares about grounding.
You mention not to plug a generator into your house wiring unless you first disconnect the main fuse. House-wired generators, temporary or otherwise, require an isolating lockout mechanism that physically prevents the mains from being enabled while the generator is operating. Make sure you consult with a professional installer and make sure you are following local code. Not doing so can put utility provider workers at significant risk of life and limb!
You mention not to plug a generator into your house wiring unless you first disconnect the main fuse. House-wired generators, temporary or otherwise, require an isolating lockout mechanism that physically prevents the mains from being enabled while the generator is operating. Make sure you consult with a professional installer and make sure you are following local code. Not doing so can put utility provider workers at significant risk of life and limb!
Have any of you been made aware of a bigger threat.. which is coming tomorrow??

A CME is heading straight at us from the sun. This one is big, and has the potential to knock out satellites, power grids, communications and more. This is not a local lightning strike. This is big, and not FIOS :-)

You should have ample surge protection as power spikes do occur during events like this one. And don't go to the beach tomorrow (Aug 4, 2010). Not a good idea LOL..

-AL
I don't trust any surge protector except ZeroSurge.
They've proven themselves many times. During a lightning storm, we lost all the desktops. The server room equipment was on the ZeroSurge - never skipped a beat.
We use a Generlink device to hook up our portable generator in lieu of a transfer switch. We opted to spend the extra hundred bucks to get the model with the whole house surge suppression built in.
As someone noted already, there's not much you can do to protect against a direct strike. When protecting against a nearby strike, you're mostly trying to ensure that everything that's interconnected rises to the same potential; voltage differences between devices cause currents to flow in network cables and the like. It's vital that everything come to a single, low-impedance ground. A common problem is cable and phone system grounds not being tied to the power system ground stake; I've seen large voltage differences between cable TV shields and power system ground even in the absence of lightning.

Diary Archives