A few months ago I took over the Abuse Department for a small ISP in the Midwest.  Little did I realize when they asked me to take the abuse that it was really me that was going to be abused.  From disgruntled customers to disgruntled service providers I have dealt with them all.  Prior to taking on this responsibility I wondered why it was that there was so much spam, why is it that ISP’s aren’t taking control of the situation, why it can’t be stopped, after all how hard could it be.  I now understand, I totally get it.  For those who have never had to deal with the cleanup, have never had to deal with the customers who don’t understand the correlation between spam, viruses, and P2P programs, let me tell you it hasn’t been easy.  In this diary I am going to outline some of the lessons that I have learned and hope that some of you will share your lessons learned with us.

Lesson 1 – Your logs and Log reports can be your most valuable tool and can give you an advanced warning of mail server abuse.  We have a lot of servers and many of them are email servers.  I monitor the log files daily to look for any obvious problems.  I have been amazed at how many times I have detected a problem simply by looking at the logs.  We currently are using Logwatch Reporting.  The summarization in these reports is pretty good.  However, having to look at a report for each server does take a bit of time.  I am reviewing different Log Management programs right now looking for a way to simplify or consolidate the information. I have decided that this may well be my first line of defense.

Lesson 2 – Customer computer’s without anti-virus and/or firewall protection are a big target, not just for them but for their ISP as well.  It absolutely amazed me how quickly a computer can go from compromised to abused and used.  Over the July 4^th weekend while reviewing my logs I noticed that one of our IP addresses, a residential customer’s home computer was sending over 200,000 emails a day.  I quickly blocked the IP and determined who the customer was. In my conversation with the customer I asked them if they had an anti-virus program.  They said that they did, when I asked them how long ago they had purchased the license, they couldn’t remember.  It came with their computer and they bought their computer a few years ago.  They said that they updated it everyday. I explained to them that it has to be renewed every year.  They had no idea. It amazes me that people have no idea what it takes to protect their computer and perhaps their identity as well. 

Lesson 3 – A mail server, no matter how well protected is in danger of being blocklisted. And once blocklisted it is really hard to get it off the list.  As I indicated our customer over the 4^th of July weekend with a compromised computer was sending massive amounts of spam.  As soon as I discovered it I stopped the activity, however it was already too late.  The server had been blocklisted.  I attempted to contact the blocklists but found it literally impossible to do.  It took the best part of 3 days to get every thing returned to normal.  In the meantime, I had to deal with customers who were trying to send email's and they were unable to do so. They were angry and didn't understand that it is virtually out of my hands.  Once the blocklist is there, you are at the mercy of the listers. I really wish that there was a process or a better way to resolve these issues.

Lesson 4 – Many of our customers whose IP addresses have been identified with spamming have had 2 components in common.  They either had outdated anti-virus programs/or using free anti-virus programs and/or they were using programs to download music/movies from the Internet.  Many of the customers that had the music/movie programs had no idea that these programs were installed on the computer. (They had teenager computer users).  The ones that knew that the programs were there had no idea about the security risks that these programs created for their computer.  It amazes me how little people know about the programs or files installed on their computers.  They download that cute screen-saver or wallpaper program not realizing that they have just installed spyware or smutware, thus opening up their computer to the world of the bad guys.

Lesson 5 – We have had a few instances where our small business customers had put up web servers or email servers.  They either had bad advice given to them or they used out of box solutions and their web servers/mail servers had been compromised.  In one case they had been hosting a paypal phishing site.  When I contacted them, they did not even know that they had a web server running.  Upon investigation they discovered that not only was the web server service running (and not being used) but users had been installed on their server.  The bad guys were doing a bit more than hosting a paypal site.

At SansFire this year, one of the Sans@Night events was a panel discussion – Meet the Handler’s. A question came up about the education of the small business/home computer user and whose responsibility it was. One of the guests in the audience didn’t feel that it should be an IT responsibility.  I said then and I will say it again.  It is our responsibility and is to our benefit.  If we help to educate the end user, help them to understand the impact they have on the rest of the customers served by their Company, their ISP and the Internet, the ultimate outcome will be a better cleaner Internet for everyone.  A little education may result in increased understanding of the importance of firewalls and anti-virus/anti-spyware programs and OS updates which will lead to increased use of these programs.  The increased use of these programs will inevitably lead to the fewer compromised computers, fewer Botnets, and fewer security holes.

Who better to reach out to our communities, to our families and friends then those of us who know and understand? A little education may go a long way.

Let us know what you think?  What lessons have you learned?