mail_to_misp/README.md

206 lines
8.4 KiB
Markdown
Raw Normal View History

2021-09-30 02:32:04 +02:00
[![Build Status](https://app.travis-ci.com/MISP/mail_to_misp.svg?branch=main)](https://app.travis-ci.com/MISP/mail_to_misp)
2020-06-16 13:13:25 +02:00
[![codecov](https://codecov.io/gh/MISP/mail_to_misp/branch/main/graph/badge.svg)](https://codecov.io/gh/MISP/mail_to_misp)
2018-05-15 05:13:13 +02:00
2017-04-27 14:32:31 +02:00
# mail_to_misp
2017-05-24 16:09:04 +02:00
Connect your mail infrastructure to [MISP](https://github.com/MISP/MISP) in order to create events based on the information contained within mails.
2017-04-27 14:32:31 +02:00
## Features
- Extraction of URLs and IP addresses (and port numbers) from free text emails
- Extraction of hostnames from URLs
2017-04-28 10:00:45 +02:00
- Extraction of hashes (MD5, SHA1, SHA256)
2018-05-02 19:08:22 +02:00
- DNS expansion
2017-04-27 14:32:31 +02:00
- Custom filter list for lines containing specific words
- Subject filters
- Respecting TLP classification mentioned in free text (including optional spelling robustness)
- Refanging of URLs ('hxxp://...')
- Add tags automatically based on key words (configurable)
- Add tags automatically depending on the presence of other tags (configurable)
2017-05-22 09:48:42 +02:00
- Add tags automatically depending on presence of hashes (e.g. for automatic expansion)
2017-04-27 14:32:31 +02:00
- Ignore 'whitelisted' domains (configurable)
2017-05-22 09:48:42 +02:00
- Specify a stop word term to no further process input
2017-04-28 10:00:45 +02:00
- Configurable list of attributes not to enable the IDS flag
2017-04-27 14:32:31 +02:00
- Automatically create 'external analysis' links based on filter list (e.g. VirusTotal, malwr.com)
2017-05-29 17:09:44 +02:00
- Automatically create 'internal reference' links based on filter list
- Detection of forwarded messages
2018-04-12 10:31:44 +02:00
- Process attachments as malware samples or specify that they are processed as benign files (`m2m_attachment_keyword`)
You can send mails with attachments to mail_to_misp and tell it, to treat the attachment as a benign document (in contrast to the default behaviour: treating it as a malware sample). You need to set a keyword in the configuration:
2018-04-12 10:31:44 +02:00
`m2m_attachment_keyword = 'attachment:benign'`
2017-05-30 12:12:26 +02:00
- Logging to syslog
2017-05-30 12:33:49 +02:00
- Remove "[tags]", "Re:" and "Fwd:" from subjects
2017-06-01 15:17:51 +02:00
- Optionally attach entire mail to event
- Contains now a fake-smtpd spamtrap which delivers IoCs/mails to MISP
2018-04-12 10:31:44 +02:00
- Automatically filter out attributes that are on a server side warning list (`enforcewarninglist=True`)
- Support for value sighting (`sighting=True`, `sighting_source="YOUR_MAIL_TO_MISP_IDENTIFIER"`)
- Auto-publish when `key:yourkey` is specified in mail (configurable, `m2m_key`, `m2m_auto_distribution`)
The `m2m_key configuration` is used to specify a secret only you and your users know. If you know the key, you can send a mail to your mail_to_misp instance, and when this key is present in the body of the message, it will automatically publish the event. So let's assume your config says: `m2m_key = 'ABCDEFGHIJKLMN0PQRSTUVWXYZ'`
If you send a mail to mail_to_misp containing: `key:ABCDEFGHIJKLMN0PQRSTUVWXYZ` the event is automatically published.
2018-04-12 09:40:49 +02:00
If you don't want to use this feature, just don't put it in the message body.
2018-04-12 10:31:44 +02:00
The distribution is defined in the configuration as well: `m2m_auto_distribution = '3' # 3 = All communities`
For OSINT collection purposes (like collecting URLs to OSINT reports), you can tell `mail_to_misp` to only extract URLs (`--urlsonly`) and append them to a predefined MISP event (`--event N`). The subject of such a mail goes into the comment field of the value.
Example:
```
osinturlcollection: "|/path/to/mail_to_misp.py --urlsonly --event 12345 -"
```
2018-08-02 14:02:10 +02:00
# Pass parameters in the email body
2018-08-02 12:02:50 +02:00
```
m2m:<parameter>:<Value>
# Examples
m2m:attachment:benign # Email attachment considered benign (attachment in MISP, malware-sample by default)
m2m:attach_original_mail:1 # Attach the full original email to the MISP Event (may contain private information)
m2m:m2mkey:YOUSETYOURKEYHERE # Key required for some actions
# The following key are ignored if M2M:m2mkey is invalid
m2m:distribution:<0-3,5> # Note: impossible to pass a sharing group yet.
m2m:threat_level:<0-2>
m2m:analysis:<0-3>
m2m:publish:1 # Autopublish
```
2017-04-27 14:32:31 +02:00
2017-04-27 14:58:25 +02:00
## Implementation
2017-12-14 11:32:52 +01:00
The implemented workflow is mainly for mail servers like Postfix. Client side implementations exist but are no longer supported:
2017-04-27 14:58:25 +02:00
2017-06-30 08:45:25 +02:00
1. Postfix and others
`Email -> mail_to_misp`
2. Apple Mail [unmaintained]
2017-05-03 09:47:25 +02:00
2017-05-23 15:17:19 +02:00
`Email -> Apple Mail -> Mail rule -> AppleScript -> mail_to_misp -> PyMISP -> MISP`
2017-05-03 09:47:25 +02:00
3. Mozilla Thunderbird [unmaintained]
2017-05-03 09:47:25 +02:00
2017-05-23 15:17:19 +02:00
`Email -> Thunderbird -> Mail rule -> filterscript -> thunderbird_wrapper -> mail_to_misp -> PyMISP -> MISP`
2018-05-02 19:08:22 +02:00
2017-05-23 15:17:19 +02:00
2017-06-30 08:45:25 +02:00
## Installation
2017-05-23 15:17:19 +02:00
2017-06-30 08:45:25 +02:00
### Postfix (or other MTA) - preferred method
2017-04-27 14:58:25 +02:00
2017-06-30 08:45:25 +02:00
1. Setup a new email address in the aliases file (e.g. /etc/aliases) and configure the correct path:
`misp_handler: "|/path/to/mail_to_misp.py -"`
2017-06-30 08:45:25 +02:00
2. Rebuild the DB:
`$ sudo newaliases`
3. Configure mail_to_misp_config.py
You should now be able to send your IoC-containing mails to misp_handler@YOURDOMAIN.
#### Bonus: Fake-SMTPD spamtrap
If you want to process all incoming junk mails automatically and collect the contained information in a separate throw-away MISP instance, you could use the fake_smtp.py script. It listens on port 25, accepts all mails and pushes them through mail_to_misp to a MISP instance.
It can also be configured to listen on an SSL port. (465)
2017-06-30 08:45:25 +02:00
1. Configure mail_to_misp_config.py
2. cp mail_to_misp_config.py fake_smtp_config.py
2021-09-30 02:06:35 +02:00
3. Make port 25 accessible to normal users
2017-06-30 08:45:25 +02:00
2021-09-30 02:06:35 +02:00
```
$ sudo apt install authbind
$ sudo touch /etc/authbind/byport/25
$ sudo chown misp:misp /etc/authbind/byport/25
$ sudo chmod 770 /etc/authbind/byport/25
```
4. Run fake_smtp.py
`$ python3 fake_smtp.py`
2017-04-27 14:58:25 +02:00
### Apple Mail [unmaintained]
2017-04-27 14:58:25 +02:00
1. Mail rule script
- git clone this repository
- open the AppleScript file MUA/Apple/Mail/MISP Mail Rule Action.txt in Apple's 'Script Editor'
- adjust the path to the python installation and location of the mail_to_misp.py script
- save it in ~/Library/Application Scripts/com.apple.mail/
2. Create a mail rule based on your needs, executing the AppleScript defined before
3. Configure mail_to_misp_config.py
### Thunderbird [unmaintained]
2017-05-03 06:33:41 +02:00
1. Git clone https://github.com/rommelfs/filterscript and install plugin (instructions within the project description)
2. Mail rule script
- git clone this repository
- open the bash script MUA/Mozilla/Thunderbird/thunderbird_wrapper.sh and adujst the paths
- adjust the path to the python installation and location of the mail_to_misp.py script
3. Create a mail rule based on your needs, executing the thunderbird_wrapper.sh script
4. Configure mail_to_misp_config.py
2017-04-27 14:58:25 +02:00
You should be able to create MISP events now.
### Outlook [unmaintained]
2017-05-03 09:47:25 +02:00
Outlook is not implemented due to lack of test environment. However, it should be feasible to do it this way:
```
import win32com.client
import pythoncom
2018-05-02 19:08:22 +02:00
2017-05-03 09:47:25 +02:00
class Handler_Class(object):
def OnNewMailEx(self, receivedItemsIDs):
for ID in receivedItemsIDs.split(","):
# Microsoft.Office.Interop.Outlook _MailItem properties:
# https://msdn.microsoft.com/en-us/library/microsoft.office.interop.outlook._mailitem_properties.aspx
mailItem = outlook.Session.GetItemFromID(ID)
print "Subj: " + mailItem.Subject
print "Body: " + mailItem.Body.encode( 'ascii', 'ignore' )
print "========"
2018-05-02 19:08:22 +02:00
2017-05-03 09:47:25 +02:00
outlook = win32com.client.DispatchWithEvents("Outlook.Application", Handler_Class)
pythoncom.PumpMessages()
```
(from: https://blog.matthewurch.ca/?p=236)
2018-05-02 19:08:22 +02:00
Obviously, you would like to filter mails based on subject or from address and pass subject and body to mail_to_misp.py in order to do something useful. Pull-requests welcome for actual implementations :)
2017-04-27 14:58:25 +02:00
2017-05-23 15:18:47 +02:00
2017-04-27 14:32:31 +02:00
## Requirements
2018-05-08 15:35:01 +02:00
### The easy way
```bash
pip install --user poetry
2019-01-21 14:52:10 +01:00
2018-05-08 15:35:01 +02:00
# Install other python requirements
poetry install -E fileobjects -E openioc -E virustotal -E email -E url
2018-05-08 15:35:01 +02:00
# Test if the script is working
./mail_to_misp.py -h
```
2017-05-03 06:33:41 +02:00
### General
2017-04-27 15:14:06 +02:00
- mail_to_misp requires access to a MISP instance (via API).
- Python >=3.6
2017-06-01 09:22:37 +02:00
- dnspython
2018-05-02 19:08:22 +02:00
- PyMISP
2017-06-01 09:04:16 +02:00
- faup from https://github.com/stricaud/faup
2017-04-27 15:14:06 +02:00
- urlmarker from https://github.com/rcompton/ryancompton.net/blob/master/assets/praw_drugs/urlmarker.py (contained in this project)
2017-05-31 14:57:34 +02:00
- ftfy from https://github.com/LuminosoInsight/python-ftfy (to fix unicode text)
2018-05-02 19:08:22 +02:00
- defang from https://github.com/Rafiot/defang.git (fork of: https://bitbucket.org/johannestaas/defang)
2017-04-27 14:32:31 +02:00
### Thunderbird [unmaintained]
2017-04-27 14:32:31 +02:00
2017-05-03 06:33:41 +02:00
- https://github.com/rommelfs/filterscript (modified fork from https://github.com/adamnew123456/filterscript)
2017-10-13 17:01:10 +02:00
## License
This software is licensed under [GNU Affero General Public License version 3](http://www.gnu.org/licenses/agpl-3.0.html)
* Copyright (C) 2017 - 2019 Sascha Rommelfangen, Raphaël Vinot
2021-04-02 03:35:55 +02:00
* Copyright (C) 2017 - 2021 CIRCL - Computer Incident Response Center Luxembourg