I posted a story a little over a week ago asking for reader input on their favorite log analysis tools and followed up with some of my own.  I promised that I'd post a summary of what you provided me.  I was hoping to do that last week, but life got in the way.  So in the spirit of "better late than never", here is my wrap-up.

The one open source tool that was mentioned most often was ossec and frankly, I'm not sure how that one slipped my mind when I did my own list.  I started using it a few months ago and really like it.  Daniel Cid, the maintainer, pointed out to me that there are quite a few rules for it that can be found at http://www.ossec.net/rules/ and they are updated/added to on a daily basis.

Beyond that, most of the folks who wrote in said that they wrote their own scripts to search/parse/summarize their logs because with experience they've learned what it is they want to look for.  I guess this points out one of the problems in the area though.  Folks with lots of experience, who have managed their machines/networks for a long time develop a feel for what is normal and what they need to watch for, but how much bad stuff happened on the way to developing all that experience?  Also, is their intuition, correct?  As I mentioned to fellow handler, Swa, when he wrote up his audit story last month, in some ways, automated summarization/reporting on logs based on experience is a lot like signature-based anti-virus or IDS, you'll catch the known stuff, but may miss the new stuff.  That's why it is important to also look at the unusual stuff.  Not just, the "top 10" reports, but also the "bottom 10".

I was kind of surprised that few of our readers wrote in about any of the commercial tools out there.  I don't know if that is because our readers all are strong believers in open source, or don't have experience with the commercial tools, or if the commercial tools just don't do what they need.  I personally have almost no experience with the commercial tools because in most of my paid jobs, there was no budget for log analysis, so we were stuck with open source, stuff I wrote, or doing without.

I'll wrap this up, by pointing you to a report that was released at the SANS Log Analysis Summit after SANSFIRE in DC in July.  I was able to attend part of the summit, including the talk by Chris Brenton and Mike Poor where they discussed the Top 5 Essential Log Reports.

Update 1:  (2006-09-18 19:00 UTC) Tor e-mailed me to point out that he has had a lot of success using Excel (and especially using it to create pivot tables) for log analysis.  He also pointed to Microsoft's LogParser (see the unofficial site at http://www.logparser.com) as a favorite tool of his.

-------------------------------
Jim Clausing, jclausing --at-- isc dot sans dot org