Do you know where your backup tape software is?

We have been talking about backups the last couple of days.  I am a big advocate of backups, having had one computer crash and burn a few years ago and left me high and dry with no backups.  I suddenly became obsessed with backups. 

I went to work for a large corporation who also was adament about backups. They expected them to be done on all systems containing any company information period.  We installed a server (Novell at that time) and set a policy that all company data was to be stored on the file server. Anyone caught with company data on a local hard drive could be written up or penalized.  The backups were done everyday. Tapes were stored in a fireproof/bomb proof vault. This was a reinforced room in the center of the office building that was tightly controlled for access. In this room were additional fire proof cabinets.  The tapes were kept in these cabinets in fire proof tape cases.  We had a 12 month rotation. We did daily backups Monday thru Thursday, weekly backups Friday 1 through 4, and monthly backups Month 1 through 12.  At the end of the fiscal year a backup was done that was kept forever (theoretically).  The year end tape was sent to the corporate IT department and was clearly marked for location and dates covered.

I thought we had a more than adequate backup plan.  And we did. However, here are a couple of things that I/we overlooked. 

1) We had a system failure and had to reload from scratch.  We had to replace the hard drives in the servers (this was more than15 years ago by the way, pre raid stuff).  After the drive was replaced, I had to reinstall the Novell OS from scratch.  After the install was done then I had to reload the tape software before I could restore the tapes.  Oh No, where did the tape software go, let's see where did we put it.  Software cabinet, hmmm, not there. desk drawer, nope. Vault, not there either.  Started asking guestions about where the software may have been stored.  My supervisor wasn't sure, the guy that I replaced had initially installed the system. He probably would know where it was at but we couldn't call him. He had left the company not of his own free will.  I didn't think he would be very quick to answer my questions. So now what do we do. I called the software vendor and explained the dilemma to them. They said that it was not a problem. I just needed to fax them a copy of the paperwork showing we owned a legal copy of the software with my license number on it.  No problem right.  Purchasing would have that.

Off to the purchasing office.  The Purchasing Manager had been with the company just a little longer than I had so he was not sure when the software was purchased or where it was purchased from.  We looked through the files and couldn't find anything. Luckily the previous purchasing manager had been transferred to another location within the company so we contacted him.  He had a "great little sponge in his head" and was able to tell us where it was purchased from, when, approximately it was purchase and which box in the fireproof vault we would fine the information in.  Thankfully we thought we were back on track.

We dug up the paperwork and faxed it to the company to get replacement for the software.  Thinking we were going to be ok we anxiously awaited word that the software was in route.  Umm - no the story does not end here.  I received a call back from the software company - umm that is a really old version of the software. We don't have that anymore. We will have to send you the new version and you will have to pay us I think it was $2500 for the new version.  I said ok - we have no choice - I will get a PO and fax to them.  Then they dropped the bomb on me. By the way - this is not a clean restore.  There is a process that needs to be done to restore the old format and convert for the new. No guarantee that all fo the files will restore. Oh my goodness, this just keeps getting worse.

We did get the new software, and we did go through the reinstall, and we did go through the conversion. I had to apply some updates to my Novell server that I hadn't planned on as a result of the software update, I had to do some heavy breathing and sweating. But 3 days (and nights) later we were back on line and had lost a minimal amount of information.  I had to reinstall my users manually, I could not restore the permissions from tape. That was a bit of a challenge and took a while to get everyone back to "normal". 

The moral of this story and my tip first tip for the day is:

Keep your tape software in a safe, secure location. Make sure you stay up to date and install new versions as they come out. Document your user's and security settings, document your system configuration so that if you have to reinstall and have problems restoring your settings you will know what they are.

Now for tip number 2:

What about your archival tapes?

Another issue with backups that I have dealt with happened about 5 years ago.  A customer that I was working with had been doing backups and had been doing archival backups at year due to government reporting requirements that they have to deal with.  They can for 10 years be mandated to produce information on demand.  They do backups that will allow them to recover the information that is required.  They had an adequate backup plan.  They stored the backups off site at the local bank.

They did overlook one thing.  They had replaced there backup drive.  The old drive was disposed of, it was obsolete and didn't work very well anymore.  They received a request for reproducing a report for a court case that was evolving.  They sent some one to the bank to get the tape to do the restore. No problem, except that the tape was for a drive they no longer had. How do they restore now?  Can we buy a tape drive? Do we know anyone that can restore it for us? How do we get to the data? 

Luckily I had the same type of tape drive that they needed to do the restore. I was able to recover the data for them and then it was backed up to the new tape device.  All of the other tapes were brought to me and I was able to recover the data from all of them and we moved them to the new media.

The moral to this story and second tip of the day is:

If you are replacing your current tape drives you need to restore any information in the old format and backup to the new format. Or you need to keep the old drive around as long as possible for retrieval purposes.