I have tested a new Sguil agent released by Paul Halliday [1] last month to collect and store http traffic session into the Sguil database for web traffic analysis. If you are looking for a method to collect and mine web traffic session, this new agent is your tool. Here is an example how the httpry agent collects its log:

2011-07-13 00:36:47 192.168.48.138 50108 72.14.204.121 80 GET www.pintumbler.org /Code/dnsbl http://www.pintumbler.org/Code/hafs Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
2011-07-13 00:36:48 192.168.48.138 50108 72.14.204.121 80 GET www.pintumbler.org /_/rsrc/1303426214049/Code/dnsbl/dnsbh1.png http://www.pintumbler.org/Code/dnsbl Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
2011-07-13 00:36:48 192.168.48.138 50227 72.14.204.121 80 GET www.pintumbler.org /_/rsrc/1303426235351/Code/dnsbl/dnsbh2.png http://www.pintumbler.org/Code/dnsbl Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
2011-07-13 00:36:48 192.168.48.138 50228 72.14.204.121 80 GET www.pintumbler.org /_/rsrc/1303426262027/Code/dnsbl/dnsbh3.png http://www.pintumbler.org/Code/dnsbl Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

This Sguil client snapshot shows the traffic collected by the agent on a client and displayed in the order the web sites were accessed. Like other traffic collected by the Sguil framework, this traffic can request the packets to be analyzed with Wireshark.

Follow Paul's instructions [2] on how to install and configured the agent to have the traffic report to the Sguil database. I would also suggest to run a cronjob to rotate the httpry service once per day to empty the logfile, otherwise it will grow exponentially and the agent will stop processing. Consider adding sites you consider of no value to the /etc/ httpry_agent.exclude file to carefully select what you insert in your database.

[1] http://www.pintumbler.org/Code/hafs
[2] https://github.com/int13h/httpry_agent/blob/master/README.md
[3] http://isc.sans.org/diary.html?storyid=9295

 -----------

Guy Bruneau IPSS Inc. gbruneau at isc dot sans dot edu

Community SANS SEC 503 coming to Ottawa Sep 2011