As this is Cyber Security Awareness Month lets discuss the topic of the day: File System Backups. 

Backups are one of the staples of the operations teams that is oft overlooked and even more often under rated.  Backup and recovery are essential to the organizations IT health.  That very statement should resound with our audience with a big DUH!  But through the years of consulting, I have been shocked at how little attention and operational practice is put towards proper backup and recovery.

First of all, imho, file system back ups are for data, not OS/Applications.  The two are different in my books because so often we run into the problem of recovering a compromised system with compromised backups.

Second: As more organizations move to virtual infrastructures, which some refer to as *gasp* applistructure, back up and recovery of systems becomes almost trivial.  Anyone familiar with Vmware ESX/Infrastructure and even Workstation sees the benefits of exporting virtual disks and bringing clones back online.

Given that its "Cyber Security Awareness Month" and this is the Internet Storm Center, I shouldn't have to beat the message of "thou shalt backup your systems properly" down your throats.

Tip #1: Back it up or lose it for ever.

We have all been there and done that when it comes to losing files on a system when it dies, because we were not as diligent as we should have been in regards to backups.

Tip #2: Test your recovery procedures at minimum 1 time per quarter

As a consultant, I have walked into a data center and asked if all the systems are properly backed up.  When the client says yes, I ask if they mind if we test the recovery procedure (as part of the scope of engagement of course).  They often get very squirrelly that that point.  Point is, you have to know that recovery is going to work, because you never know when you are going to need it.

Many of us in the industry were pleasantly surprised when more financial data was not lost during 9/11.  Financial organizations are required to have proper offsite backup/replication processes in place, and what do you know... they did!

Tip #3: Ensure that your backup software (agent and server) are properly patched

All one has to do is look at Metasploit's exploit list to see that backup software has had a rough couple of years.  Why go after each individual server when we can go after the backup server and the storage device.  At the ISC and at Intelguardians we have seen hundreds of large organizations get pwn3d via backup software.

Tip #4: Protect your backup tapes

On many occasions while visiting client data centers I encounter this bizarre situation:

Biometric cages to get access to systems, armed guards, firewalls, laser beams (well, no laser beams but it sounded cool) all protecting client systems.  Then, on the loading dock of the data center, a box with tapes labled: For Iron Mountain or similar.

Just think of what happened back in 2002:

Backup tapes stolen from group digitizing military medical records.

Backup tapes stolen from Japanese company van that was creating national ID cards.

Backup tapes stolen from a military shipment going through international customs.

All of these incidents happened within 2 months of each other.  Were they related... who the heck knows.  Point is, protect your backup tapes as if they were the actual systems they came from.

Tip #5: Use backup diffs to find rootkit file installations.

Infosec Guru Randy Marchany reminded me of an incident that happened at Virginia tech a couple of years back.  A large number of Solaris systems were compromised and they were not sure how or what files had changed.  Their brilliant network backup expert, Judy Albert ran out of the meeting and came back minutes later with a precise list of all the files that had changed on the system.  Since they were diligent about backups, they had a "pristine" state snapshot from the day before the incident and could diff the results of the current backup to see which files had changed.  Great thinking!

On a very sad note, Judy was killed in a Gyrocopter accident last Saturday.  Our thoughts and prayers go out to her family, the IT group at Virginia Tech and to the Infosec community.  We have lost one of our own, but her wisdom will be passed down to generations to come.

If you have any ideas, tips, or stories related to File System Backups and Cyber Security Awareness Month, please submit them through our contact page.  Please keep them PG-13 :-)

TTFN,

Mike Poor, Handler on Duty, Intelguardians, Inc