Unfortunately we work events and incidents every day.  Some are worse than others, but the one rule of incident handling is that every incident must be handled as if it were going to end up in court.  Gathering evidence should begin as soon as it is identified. 

Every incident handler should have a bound incident log book with numbered pages.  Once you begin to work an incident, record every detail into the journal.  Every handler should be recording their efforts, too.  This becomes collaberative evidence in court.  Make sure to date and initial every entry. 

The next evidence gathering technique is the bit-by-bit backup.  Before you start working on the system, it is imperative that a backup is made.  Some people are taking the hard drive out and replacing it, using the original as the backup.  This is probably the best method if you have spare hard drives, but if you don't you must make a binary backup.  A binary backup preserves all the evidence including deleted and fragmentary files. 

One of the most popular tools is "dd".  It comes in most Unix and Linux distributions.  A Windows version can be found here.   Be sure to practice with your incident response team and help desk personnel, as making a backup under pressure can be more difficult than you think.

If you have more tips for gathering evidence, send them to us here and we'll pass them along.

Mari Nichols   iMarSolutions

Update:  One of our readers Mikael wrote in and gave us a tip on Clonezilla, an OpenSource clone system solution that includes dd and lots of storage drivers.  Check it out here. 

Update 2:  If you aren't specifically interested in working with evidence, but rather malware, Drew wrote in suggesting to use CERT's Liveview utility.  He says it takes the dd image and generates a VMWare Workstation configuration wrapper to run it as a virtual machine.  Find more info on Liveview here.

Update 3:  Thanks to "alerter" for sending the following in:

There are a slew of "newbie" mistakes that will wind up defeating well intentioned efforts to "preserve" incident "evidence" for possible use in court action. First and foremost, there must be a written and verifiable chain of custody for everything and anything that might be considered evidence.  Do not take this lightly, otherwise everything that cannot show strictly observed chain of custody will wind up being for naught in court, at least in the US.

Before any HDs are "pulled" or "imaged," other volatile elements of system state also need to be captured:

 - contents of CPU "memory" (not just registers, but on file cache)
 - contents of dynamic system RAM
 - contents of system ROM (if present)
 - contents of sub-system ROMs (video, network, etc.)
 - contents of static RAM in connected routers/switches
 - contents of dynamic RAM in connected routers/switches

If ROMs can't be imaged, they should at least be hashed, with attention to using more than one good hash in order
to help defeat arguments against unintentional hash collisions. What's in volatile memory may be just as important, if not more so, that anything that may exist on disks.  Some attackware never writes any part of itself directly to any local disk.  Memory images need to be hashed, too.

Original disks need to be hashed (see above), before and after they have been bit imaged.  There should be
end-to-end sector-level hashes as well as various filesystem hashes taken on originals. Disk images not only need to be bit images of all physical disc sectors (not just clusters), images need to be hashed and then compared against original hashes. No compression should be used when taking images.When physically pulling a drive and imaging it, write block devices must be used to prove that no part of any attempted imaging procedure was able to write back to the original disk.

For networked systems, one nicely pre-rolled toolkit is the bootable Helix .iso image. Place a booted Helix notebook on the same network segment where the incident took place (or may still be still in progress), point the relevant Helix tools at the selected host(s) and start "preserving evidence."  The pre-rolled Helix imaging solutions can also handle the hashing of originals and images. More "incident responders" should take the time to play ith Helix in simulations/exercises;  because the preservation of evidence cannot be an afterthought.  It has to be an integral part of the original response.

Update 4: Thanks to Jason for writing in with these suggestions:

See http://www.afflib.org/ and http://www.forensicswiki.org/wiki/AFF for information about the Advanced Forensics Format.

You can create forensic hard disk drive images with built-in lzma compression, passphrase encryption, and progressive hashing during image creation. It also has user-definable metadata stored along with the image. I wish more tools supported it.  Last I checked, Sleuthkit could read them just fine and you could mount them in Linux, but trying to use Mount Image Pro for it in Windows didn't work.  I was told by their customer service that they are considering adding support at some time in the future.

The biggest problem I had (granted this was at least a year ago) with disk imaging was the storage required to keep disk image files.  I worked at a financial institution and worked on several hundred investigations.  Those disk drive images add up quickly.  Forensically sound compression was a huge boon. It's easy to say "never use compression / always use bit by bit copies" when you aren't working under a tight budget.