Collecting IOCs from IMAP Folder
I've plenty of subscriptions to "cyber security" mailing lists that generate a lot of traffic. Even if we try to get rid of emails, that's a fact: email remains a key communication channel. Some mailing lists posts contain interesting indicators of compromize. So, I searched for a nice way to extract them in an automated way (and to correlate them with other data). I did not find a solution ready to use that matched my requirements:
- Connect to any mailbox (preferably via IMAP)
- Produce data easy to process (JSON)
- Be easy to deploy (Docker)
So, I built my own Docker image... It is based on the following components:
- procmail
- getmail
- some Python libraries
- The project es_mail_intel[1]
The last tool is an old project that achieves exactly why I expect: It extracts IOCs from emails and stores them in ElasticSearch. But, if you don't want ElasticSearch, it can also produce a JSON file! Parsing emails is a pain! So, I did not want to write my own parser.
Data are processed in this way: Emails are fetched via IMAP at regular intervals by getmail and pushed to procmail. It pushes them to the Python script that extracts interesting data.
IMAP data >> getmail >> procmail >> mail_parser2json_extract.py >> JSON data
Here is my Dockerfile:
FROM ubuntu:18.04 MAINTAINER Xavier Mertens <xavier@rootshell.be> RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \ ca-certificates \ getmail \ procmail \ git \ python \ python-ipaddress \ python-pdfminer \ python-elasticsearch \ python-xlrd \ && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* RUN mkdir -p /root/.getmail RUN git clone https://github.com/clverhack/es_email_intel.git /opt/es_email_intel COPY getmail.conf / RUN echo ":0" >>/procmailrc RUN echo "|/opt/es_email_intel/mail_parser2json_extract.py 2 >>/log/ioc.json" >>/procmailrc COPY run.sh / RUN chmod u+x /run.sh RUN touch /tmp/firstboot CMD ["/run.sh"]
It needs a getmail.conf with the parameters of the mailbox you'd like to monitor:
[retriever] type = SimpleIMAPSSLRetriever server = CONF_SERVER username = CONF_LOGIN password = CONF_PASSWORD [destination] type = MDA_external path = /usr/bin/procmail user = getmail group = getmail arguments = ('/procmailrc', ) [options] verbose=1 read_all=false delete=false delete_after=0 delete_bigger_than=0 max_bytes_per_session=0 max_message_size=0 max_messages_per_session=0 delivered_to=false received=false message_log=/log/getmail.log message_log_syslog=false message_log_verbose=true
getmail is a very powerful tool with plenty of options. Just have a look at the documentation[2] to find your best way to interact with your mailboxes. The script 'run.sh' will be executed by the container and, at first boot, configure your credentials:
#!/bin/bash if [ -r /tmp/firstboot ]; then sed -i "s|CONF_SERVER|$IMAP_SERVER|g" /getmail.conf sed -i "s|CONF_LOGIN|$IMAP_USER|g" /getmail.conf sed -i "s|CONF_PASSWORD|$IMAP_PASS|g" /getmail.conf groupadd getmail useradd -u $UID -g getmail -d /home/getmail getmail mkdir /home/getmail && chown getmail:getmail /home/getmail test -d /log || mkdir /log touch /log/getmail.log /log/ioc.json chown -R root:getmail /log chmod -R g+w /log rm /tmp/firstboot fi while true do /usr/bin/getmail -r /getmail.conf sleep $IMAP_WAIT done
And, finally, my docker-compose.yml file:
version: '3' services: iocollector: build: . image: "xme/iocollector" restart: always hostname: iocollector container_name: iocollector volumes: - /etc/localtime:/etc/localtime:ro - /data/iocollector/log:/log environment: - UID=1000 - IMAP_SERVER=<server_ip_or_fqdn> - IMAP_USER=<username> - IMAP_PASS=<password> - IMAP_WAIT=30 network_mode: bridge
Start your docker and it will populate the mapped /log directory with an 'ioc.json' file:
{ "bitcoin_wallet": [ "1ncgvmeqno77x5pqbp7yi4779si", "3k04tl9upoapvqrcbntcsnglvcr5yn", "366vraic2k6ocpol7q9ck8vs2o5by0dp", "1fgtytuwedagp1krbukpd1dkkpfs", "3pmusoxmgn1qmrzedtxsqbua6dua6t", "34pg4dq162op34bd1aospshvsyxwhsxyo", "38wvzcnz49gvuwo29iuf8vxyssvmfi66w", "1nvgzipjvq5hdd16iszjdx9jzvf3", "1igrmo3w477lnb4e3tsz3gfdrciwo", "1yenq1jfsx54ptvvdtbplcqumu2dz3we", "3wusjglqxkkro3nfzd1mkwx1qqf", "1tjkoizmzrof4kgkok3jciwysmrxprj7" ], "ctime": "Thu Mar 5 17:54:23 2020", "domain": [ "xpressvpngoodforpsdgo.duckdns.org", "windowsfirewallsecurityauthorise.duckdns.org", "cartonal.net", "asco.az", "duckdns.org", "dnverificationfailedbymicrosoftthanks.duckdns.org", "saranadinamika.co.id", ... "windowsdefenderserversecuresofficeiq.duckdns.org", "tecnol2.com", "synergiespakistan.com", "chxsndy3manufacturingandinsurancebusines.duckdns.org", "ecomglobalpurchasecentercom.duckdns.org", "globalsharesecurefilesgood.duckdns.org", "ewrt55.com", "matriximportsusa.com", "pstv-energy.com", "msofficecloudtransferfileprotocolsys.duckdns.org", "workshop002.duckdns.org", "globalfbdnsaddressgoogle.duckdns.org", "www.virustotal.com", "marroiq.com", "challengerevertprocessupdate.duckdns.org", "windowsdefenderserversecureserver.duckdns.org", "windowsdefenderserversecuresofficew.duckdns.org" ], "email": [ "1042@home7.solarlog-web.de", ... ], "epoch": "1583427263", "filename": [ "64443.exe", "soa.xlsx", "copy.xlsx", "p.o.xlsx", "agenttesla.xlsx", "vbc.exe", "products.xlsx", "quote.pdf" ], "ipv4": [ ... ], "md5": [], "message_text": "...", "mutex": [], "sha1": [], "sha256": [ "3a0965bd363e46e8c30590d7ddfd22bfc353728c6311c3ca61479da6f874bd5f", "f3a98da7e0a2c7cfe0907f5b0b769a4ff3bad45d7dce7ed905d6c3c2bf835201", "cb01d834237a37d3825f4075961cf6eda5f856af13526d699e4f5c93edf00c15", "2ef88915bb08f6d0d4c5fb2a90777930ad4e0d0b757e4b2b51988813338470f9", "df852235358fe361ba0577ad4c064b642148b2ac52d5857316d20bca33ea3d04", "172542fe3f7aae11deaf6960652d0074a1a7d83ae5596951ad69ba151a9f9630" ], "ssdeep": [], "url": [ "windowsdefenderserversecureserver.duckdns.org/mich/vbc.exe", "globalsharesecurefilesgood.duckdns.org/love/vbc.exe", "windowsfirewallsecurityauthorise.duckdns.org/big/vbc.exe", "workshop002.duckdns.org/vbc.exe", "windowsdefenderserversecuresofficew.duckdns.org/windows/vbc.exe", "dnverificationfailedbymicrosoftthanks.duckdns.org/lee/vbc.exe", "dnverificationfailedbymicrosoftthanks.duckdns.org/lavin/vbc.exe", "ecomglobalpurchasecentercom.duckdns.org/love/vbc.exe", "windowsdefenderserversecuresofficew.duckdns.org/reg/vbc.exe", ... "globalfbdnsaddressgoogle.duckdns.org/lvc/vbc.exe" ] }
Note: The complete is email is parsed. You will find in the JSON file all SMTP headers, the email body, etc. Less relevant for IOC's but still interesting in some cases (by example, to analyze spam).
Here is a recap of the data flow:
[1] https://github.com/clverhack/es_email_intel.git
[2] http://pyropus.ca/software/getmail/
Xavier Mertens (@xme)
Senior ISC Handler - Freelance Cyber Security Consultant
PGP Key
Reverse-Engineering Malware: Advanced Code Analysis | Singapore | Nov 18th - Nov 22nd 2024 |
Comments