mirror of https://github.com/MISP/misp-modules
Merge github.com:MISP/misp-modules into 8ear-add-docker-capabilitites
8ear
commit
d55331fc1c
13
.travis.yml
13
.travis.yml
|
|
@ -14,8 +14,6 @@ install:
|
|||
- sudo apt-get install libzbar0 libzbar-dev libpoppler-cpp-dev
|
||||
- pip install pipenv
|
||||
- pipenv install --dev
|
||||
# MKDOCS
|
||||
- pip install -r docs/REQUIREMENTS.txt
|
||||
|
||||
script:
|
||||
- pipenv run coverage run -m --parallel-mode --source=misp_modules misp_modules.__init__ -l 127.0.0.1 &
|
||||
|
|
@ -35,14 +33,3 @@ script:
|
|||
after_success:
|
||||
- pipenv run coverage combine .coverage*
|
||||
- pipenv run codecov
|
||||
# MKDOCS
|
||||
- make ci_generate_docs
|
||||
|
||||
deploy:
|
||||
provider: pages
|
||||
local-dir: site
|
||||
skip-cleanup: true
|
||||
github-token: $GITHUB_TOKEN # Set in the settings page of your repository, as a secure variable
|
||||
keep-history: true
|
||||
on:
|
||||
branch: master
|
||||
546
README.md
546
README.md
|
|
@ -3,24 +3,552 @@
|
|||
[](https://travis-ci.org/MISP/misp-modules)
|
||||
[](https://coveralls.io/github/MISP/misp-modules?branch=master)
|
||||
[](https://codecov.io/gh/MISP/misp-modules)
|
||||
[](https://app.fossa.io/projects/git%2Bgithub.com%2F8ear%2Fmisp-modules?ref=badge_shield)
|
||||
|
||||
## About
|
||||
|
||||
MISP modules are autonomous modules that can be used for expansion and other services in [MISP](https://github.com/MISP/MISP).
|
||||
|
||||
The modules are written in Python 3 following a simple API interface. The objective is to ease the extensions of MISP functionalities
|
||||
without modifying core components. The API is available via a simple REST API which is independent from MISP installation or configuration.
|
||||
|
||||
MISP modules support is included in MISP starting from version `2.4.28`.
|
||||
MISP modules support is included in MISP starting from version 2.4.28.
|
||||
|
||||
For more information: [Extending MISP with Python modules](https://www.circl.lu/assets/files/misp-training/switch2016/2-misp-modules.pdf) slides from MISP training.
|
||||
For more information: [Extending MISP with Python modules](https://www.misp-project.org/misp-training/3.1-misp-modules.pdf) slides from MISP training.
|
||||
|
||||
## Existing MISP modules
|
||||
|
||||
### Expansion modules
|
||||
|
||||
* [Backscatter.io](misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations.
|
||||
* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description, its history, and position in BGP Ranking.
|
||||
* [BTC scam check](misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused.
|
||||
* [BTC transactions](misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP.
|
||||
* [CIRCL Passive DNS](misp_modules/modules/expansion/circl_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information.
|
||||
* [CIRCL Passive SSL](misp_modules/modules/expansion/circl_passivessl.py) - a hover and expansion module to expand IP addresses with the X.509 certificate seen.
|
||||
* [countrycode](misp_modules/modules/expansion/countrycode.py) - a hover module to tell you what country a URL belongs to.
|
||||
* [CrowdStrike Falcon](misp_modules/modules/expansion/crowdstrike_falcon.py) - an expansion module to expand using CrowdStrike Falcon Intel Indicator API.
|
||||
* [CVE](misp_modules/modules/expansion/cve.py) - a hover module to give more information about a vulnerability (CVE).
|
||||
* [CVE advanced](misp_modules/modules/expansion/cve_advanced.py) - An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).
|
||||
* [Cuckoo submit](misp_modules/modules/expansion/cuckoo_submit.py) - A hover module to submit malware sample, url, attachment, domain to Cuckoo Sandbox.
|
||||
* [DBL Spamhaus](misp_modules/modules/expansion/dbl_spamhaus.py) - a hover module to check Spamhaus DBL for a domain name.
|
||||
* [DNS](misp_modules/modules/expansion/dns.py) - a simple module to resolve MISP attributes like hostname and domain to expand IP addresses attributes.
|
||||
* [docx-enrich](misp_modules/modules/expansion/docx-enrich.py) - an enrichment module to get text out of Word document into MISP (using free-text parser).
|
||||
* [DomainTools](misp_modules/modules/expansion/domaintools.py) - a hover and expansion module to get information from [DomainTools](http://www.domaintools.com/) whois.
|
||||
* [EUPI](misp_modules/modules/expansion/eupi.py) - a hover and expansion module to get information about an URL from the [Phishing Initiative project](https://phishing-initiative.eu/?lang=en).
|
||||
* [Farsight DNSDB Passive DNS](misp_modules/modules/expansion/farsight_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information.
|
||||
* [GeoIP](misp_modules/modules/expansion/geoip_country.py) - a hover and expansion module to get GeoIP information from geolite/maxmind.
|
||||
* [Greynoise](misp_modules/modules/expansion/greynoise.py) - a hover to get information from greynoise.
|
||||
* [hashdd](misp_modules/modules/expansion/hashdd.py) - a hover module to check file hashes against [hashdd.com](http://www.hashdd.com) including NSLR dataset.
|
||||
* [hibp](misp_modules/modules/expansion/hibp.py) - a hover module to lookup against Have I Been Pwned?
|
||||
* [intel471](misp_modules/modules/expansion/intel471.py) - an expansion module to get info from [Intel471](https://intel471.com).
|
||||
* [IPASN](misp_modules/modules/expansion/ipasn.py) - a hover and expansion to get the BGP ASN of an IP address.
|
||||
* [iprep](misp_modules/modules/expansion/iprep.py) - an expansion module to get IP reputation from packetmail.net.
|
||||
* [Joe Sandbox submit](misp_modules/modules/expansion/joesandbox_submit.py) - Submit files and URLs to Joe Sandbox.
|
||||
* [Joe Sandbox query](misp_modules/modules/expansion/joesandbox_query.py) - Query Joe Sandbox with the link of an analysis and get the parsed data.
|
||||
* [macaddress.io](misp_modules/modules/expansion/macaddress_io.py) - a hover module to retrieve vendor details and other information regarding a given MAC address or an OUI from [MAC address Vendor Lookup](https://macaddress.io). See [integration tutorial here](https://macaddress.io/integrations/MISP-module).
|
||||
* [macvendors](misp_modules/modules/expansion/macvendors.py) - a hover module to retrieve mac vendor information.
|
||||
* [ocr-enrich](misp_modules/modules/expansion/ocr-enrich.py) - an enrichment module to get OCRized data from images into MISP.
|
||||
* [ods-enrich](misp_modules/modules/expansion/ods-enrich.py) - an enrichment module to get text out of OpenOffice spreadsheet document into MISP (using free-text parser).
|
||||
* [odt-enrich](misp_modules/modules/expansion/odt-enrich.py) - an enrichment module to get text out of OpenOffice document into MISP (using free-text parser).
|
||||
* [onyphe](misp_modules/modules/expansion/onyphe.py) - a modules to process queries on Onyphe.
|
||||
* [onyphe_full](misp_modules/modules/expansion/onyphe_full.py) - a modules to process full queries on Onyphe.
|
||||
* [OTX](misp_modules/modules/expansion/otx.py) - an expansion module for [OTX](https://otx.alienvault.com/).
|
||||
* [passivetotal](misp_modules/modules/expansion/passivetotal.py) - a [passivetotal](https://www.passivetotal.org/) module that queries a number of different PassiveTotal datasets.
|
||||
* [pdf-enrich](misp_modules/modules/expansion/pdf-enrich.py) - an enrichment module to extract text from PDF into MISP (using free-text parser).
|
||||
* [pptx-enrich](misp_modules/modules/expansion/pptx-enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser).
|
||||
* [qrcode](misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values.
|
||||
* [rbl](misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute.
|
||||
* [reversedns](misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes.
|
||||
* [securitytrails](misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/).
|
||||
* [shodan](misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module.
|
||||
* [Sigma queries](misp_modules/modules/expansion/sigma_queries.py) - Experimental expansion module querying a sigma rule to convert it into all the available SIEM signatures.
|
||||
* [Sigma syntax validator](misp_modules/modules/expansion/sigma_syntax_validator.py) - Sigma syntax validator.
|
||||
* [sourcecache](misp_modules/modules/expansion/sourcecache.py) - a module to cache a specific link from a MISP instance.
|
||||
* [STIX2 pattern syntax validator](misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax.
|
||||
* [ThreatCrowd](misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/).
|
||||
* [threatminer](misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/).
|
||||
* [urlhaus](misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url.
|
||||
* [urlscan](misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io).
|
||||
* [virustotal](misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://developers.virustotal.com/reference))
|
||||
* [virustotal_public](misp_modules/modules/expansion/virustotal_public.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a public key and a low request rate limit. (More details about the API: [here](https://developers.virustotal.com/reference))
|
||||
* [VMray](misp_modules/modules/expansion/vmray_submit.py) - a module to submit a sample to VMray.
|
||||
* [VulnDB](misp_modules/modules/expansion/vulndb.py) - a module to query [VulnDB](https://www.riskbasedsecurity.com/).
|
||||
* [Vulners](misp_modules/modules/expansion/vulners.py) - an expansion module to expand information about CVEs using Vulners API.
|
||||
* [whois](misp_modules/modules/expansion/whois.py) - a module to query a local instance of [uwhois](https://github.com/rafiot/uwhoisd).
|
||||
* [wikidata](misp_modules/modules/expansion/wiki.py) - a [wikidata](https://www.wikidata.org) expansion module.
|
||||
* [xforce](misp_modules/modules/expansion/xforceexchange.py) - an IBM X-Force Exchange expansion module.
|
||||
* [xlsx-enrich](misp_modules/modules/expansion/xlsx-enrich.py) - an enrichment module to get text out of an Excel document into MISP (using free-text parser).
|
||||
* [YARA query](misp_modules/modules/expansion/yara_query.py) - a module to create YARA rules from single hash attributes.
|
||||
* [YARA syntax validator](misp_modules/modules/expansion/yara_syntax_validator.py) - YARA syntax validator.
|
||||
|
||||
### Export modules
|
||||
|
||||
* [CEF](misp_modules/modules/export_mod/cef_export.py) module to export Common Event Format (CEF).
|
||||
* [Cisco FireSight Manager ACL rule](misp_modules/modules/export_mod/cisco_firesight_manager_ACL_rule_export.py) module to export as rule for the Cisco FireSight manager ACL.
|
||||
* [GoAML export](misp_modules/modules/export_mod/goamlexport.py) module to export in [GoAML format](http://goaml.unodc.org/goaml/en/index.html).
|
||||
* [Lite Export](misp_modules/modules/export_mod/liteexport.py) module to export a lite event.
|
||||
* [PDF export](misp_modules/modules/export_mod/pdfexport.py) module to export an event in PDF.
|
||||
* [Nexthink query format](misp_modules/modules/export_mod/nexthinkexport.py) module to export in Nexthink query format.
|
||||
* [osquery](misp_modules/modules/export_mod/osqueryexport.py) module to export in [osquery](https://osquery.io/) query format.
|
||||
* [ThreatConnect](misp_modules/modules/export_mod/threat_connect_export.py) module to export in ThreatConnect CSV format.
|
||||
* [ThreatStream](misp_modules/modules/export_mod/threatStream_misp_export.py) module to export in ThreatStream format.
|
||||
|
||||
### Import modules
|
||||
|
||||
* [CSV import](misp_modules/modules/import_mod/csvimport.py) Customizable CSV import module.
|
||||
* [Cuckoo JSON](misp_modules/modules/import_mod/cuckooimport.py) Cuckoo JSON import.
|
||||
* [Email Import](misp_modules/modules/import_mod/email_import.py) Email import module for MISP to import basic metadata.
|
||||
* [GoAML import](misp_modules/modules/import_mod/goamlimport.py) Module to import [GoAML](http://goaml.unodc.org/goaml/en/index.html) XML format.
|
||||
* [Joe Sandbox import](misp_modules/modules/import_mod/joe_import.py) Parse data from a Joe Sandbox json report.
|
||||
* [OCR](misp_modules/modules/import_mod/ocr.py) Optical Character Recognition (OCR) module for MISP to import attributes from images, scan or faxes.
|
||||
* [OpenIOC](misp_modules/modules/import_mod/openiocimport.py) OpenIOC import based on PyMISP library.
|
||||
* [ThreatAnalyzer](misp_modules/modules/import_mod/threatanalyzer_import.py) - An import module to process ThreatAnalyzer archive.zip/analysis.json sandbox exports.
|
||||
* [VMRay](misp_modules/modules/import_mod/vmray_import.py) - An import module to process VMRay export.
|
||||
|
||||
## How to install and start MISP modules in a Python virtualenv? (recommended)
|
||||
|
||||
~~~~bash
|
||||
sudo apt-get install python3-dev python3-pip libpq5 libjpeg-dev tesseract-ocr libpoppler-cpp-dev imagemagick virtualenv libopencv-dev zbar-tools libzbar0 libzbar-dev libfuzzy-dev -y
|
||||
sudo -u www-data virtualenv -p python3 /var/www/MISP/venv
|
||||
cd /usr/local/src/
|
||||
sudo git clone https://github.com/MISP/misp-modules.git
|
||||
cd misp-modules
|
||||
sudo -u www-data /var/www/MISP/venv/bin/pip install -I -r REQUIREMENTS
|
||||
sudo -u www-data /var/www/MISP/venv/bin/pip install .
|
||||
# Start misp-modules as a service
|
||||
sudo cp etc/systemd/system/misp-modules.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now misp-modules
|
||||
/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s & #to start the modules
|
||||
~~~~
|
||||
|
||||
## How to install and start MISP modules on RHEL-based distributions ?
|
||||
As of this writing, the official RHEL repositories only contain Ruby 2.0.0 and Ruby 2.1 or higher is required. As such, this guide installs Ruby 2.2 from the [SCL](https://access.redhat.com/documentation/en-us/red_hat_software_collections/3/html/3.2_release_notes/chap-installation#sect-Installation-Subscribe) repository.
|
||||
|
||||
~~~~bash
|
||||
sudo yum install rh-ruby22
|
||||
sudo yum install openjpeg-devel
|
||||
sudo yum install rubygem-rouge rubygem-asciidoctor zbar-devel opencv-devel gcc-c++ pkgconfig poppler-cpp-devel python-devel redhat-rpm-config
|
||||
cd /var/www/MISP
|
||||
git clone https://github.com/MISP/misp-modules.git
|
||||
cd misp-modules
|
||||
sudo -u apache /usr/bin/scl enable rh-python36 "virtualenv -p python3 /var/www/MISP/venv"
|
||||
sudo -u apache /var/www/MISP/venv/bin/pip install -U -I -r REQUIREMENTS
|
||||
sudo -u apache /var/www/MISP/venv/bin/pip install -U .
|
||||
~~~~
|
||||
|
||||
Create the service file /etc/systemd/system/misp-modules.service :
|
||||
~~~~
|
||||
echo "[Unit]
|
||||
Description=MISP's modules
|
||||
After=misp-workers.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=apache
|
||||
Group=apache
|
||||
ExecStart=/usr/bin/scl enable rh-python36 rh-ruby22 '/var/www/MISP/venv/bin/misp-modules –l 127.0.0.1 –s'
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target" | sudo tee /etc/systemd/system/misp-modules.service
|
||||
~~~~
|
||||
|
||||
The `After=misp-workers.service` must be changed or removed if you have not created a misp-workers service.
|
||||
Then, enable the misp-modules service and start it:
|
||||
~~~~bash
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now misp-modules
|
||||
~~~~
|
||||
|
||||
## How to add your own MISP modules?
|
||||
|
||||
Create your module in [misp_modules/modules/expansion/](misp_modules/modules/expansion/), [misp_modules/modules/export_mod/](misp_modules/modules/export_mod/), or [misp_modules/modules/import_mod/](misp_modules/modules/import_mod/). The module should have at minimum three functions:
|
||||
|
||||
* **introspection** function that returns a dict of the supported attributes (input and output) by your expansion module.
|
||||
* **handler** function which accepts a JSON document to expand the values and return a dictionary of the expanded values.
|
||||
* **version** function that returns a dict with the version and the associated meta-data including potential configurations required of the module.
|
||||
|
||||
Don't forget to return an error key and value if an error is raised to propagate it to the MISP user-interface.
|
||||
|
||||
Your module's script name should also be added in the `__all__` list of `<module type folder>/__init__.py` in order for it to be loaded.
|
||||
|
||||
~~~python
|
||||
...
|
||||
# Checking for required value
|
||||
if not request.get('ip-src'):
|
||||
# Return an error message
|
||||
return {'error': "A source IP is required"}
|
||||
...
|
||||
~~~
|
||||
|
||||
|
||||
### introspection
|
||||
|
||||
The function that returns a dict of the supported attributes (input and output) by your expansion module.
|
||||
|
||||
~~~python
|
||||
mispattributes = {'input': ['link', 'url'],
|
||||
'output': ['attachment', 'malware-sample']}
|
||||
|
||||
def introspection():
|
||||
return mispattributes
|
||||
~~~
|
||||
|
||||
### version
|
||||
|
||||
The function that returns a dict with the version and the associated meta-data including potential configurations required of the module.
|
||||
|
||||
|
||||
### Additional Configuration Values
|
||||
|
||||
If your module requires additional configuration (to be exposed via the MISP user-interface), you can define those in the moduleconfig value returned by the version function.
|
||||
|
||||
~~~python
|
||||
# config fields that your code expects from the site admin
|
||||
moduleconfig = ["apikey", "event_limit"]
|
||||
|
||||
def version():
|
||||
moduleinfo['config'] = moduleconfig
|
||||
return moduleinfo
|
||||
~~~
|
||||
|
||||
|
||||
When you do this a config array is added to the meta-data output containing all the potential configuration values:
|
||||
|
||||
~~~
|
||||
"meta": {
|
||||
"description": "PassiveTotal expansion service to expand values with multiple Passive DNS sources",
|
||||
"config": [
|
||||
"username",
|
||||
"password"
|
||||
],
|
||||
"module-type": [
|
||||
"expansion",
|
||||
"hover"
|
||||
],
|
||||
|
||||
...
|
||||
~~~
|
||||
|
||||
|
||||
If you want to use the configuration values set in the web interface they are stored in the key `config` in the JSON object passed to the handler.
|
||||
|
||||
~~~
|
||||
def handler(q=False):
|
||||
|
||||
# Check if we were given a configuration
|
||||
config = q.get("config", {})
|
||||
|
||||
# Find out if there is a username field
|
||||
username = config.get("username", None)
|
||||
~~~
|
||||
|
||||
|
||||
### handler
|
||||
|
||||
The function which accepts a JSON document to expand the values and return a dictionary of the expanded values.
|
||||
|
||||
~~~python
|
||||
def handler(q=False):
|
||||
"Fully functional rot-13 encoder"
|
||||
if q is False:
|
||||
return False
|
||||
request = json.loads(q)
|
||||
src = request.get('ip-src')
|
||||
if src is None:
|
||||
# Return an error message
|
||||
return {'error': "A source IP is required"}
|
||||
else:
|
||||
return {'results':
|
||||
codecs.encode(src, "rot-13")}
|
||||
~~~
|
||||
|
||||
#### export module
|
||||
|
||||
For an export module, the `request["data"]` object corresponds to a list of events (dictionaries) to handle.
|
||||
|
||||
Iterating over events attributes is performed using their `Attribute` key.
|
||||
|
||||
~~~python
|
||||
...
|
||||
for event in request["data"]:
|
||||
for attribute in event["Attribute"]:
|
||||
# do stuff w/ attribute['type'], attribute['value'], ...
|
||||
...
|
||||
|
||||
### Returning Binary Data
|
||||
|
||||
If you want to return a file or other data you need to add a data attribute.
|
||||
|
||||
~~~python
|
||||
{"results": {"values": "filename.txt",
|
||||
"types": "attachment",
|
||||
"data" : base64.b64encode(<ByteIO>) # base64 encode your data first
|
||||
"comment": "This is an attachment"}}
|
||||
~~~
|
||||
|
||||
If the binary file is malware you can use 'malware-sample' as the type. If you do this the malware sample will be automatically zipped and password protected ('infected') after being uploaded.
|
||||
|
||||
|
||||
~~~python
|
||||
{"results": {"values": "filename.txt",
|
||||
"types": "malware-sample",
|
||||
"data" : base64.b64encode(<ByteIO>) # base64 encode your data first
|
||||
"comment": "This is an attachment"}}
|
||||
~~~
|
||||
|
||||
[To learn more about how data attributes are processed you can read the processing code here.](https://github.com/MISP/PyMISP/blob/4f230c9299ad9d2d1c851148c629b61a94f3f117/pymisp/mispevent.py#L185-L200)
|
||||
|
||||
|
||||
### Module type
|
||||
|
||||
A MISP module can be of four types:
|
||||
|
||||
- **expansion** - service related to an attribute that can be used to extend and update an existing event.
|
||||
- **hover** - service related to an attribute to provide additional information to the users without updating the event.
|
||||
- **import** - service related to importing and parsing an external object that can be used to extend an existing event.
|
||||
- **export** - service related to exporting an object, event, or data.
|
||||
|
||||
module-type is an array where the list of supported types can be added.
|
||||
|
||||
## Testing your modules?
|
||||
|
||||
MISP uses the **modules** function to discover the available MISP modules and their supported MISP attributes:
|
||||
|
||||
~~~
|
||||
% curl -s http://127.0.0.1:6666/modules | jq .
|
||||
[
|
||||
{
|
||||
"name": "passivetotal",
|
||||
"type": "expansion",
|
||||
"mispattributes": {
|
||||
"input": [
|
||||
"hostname",
|
||||
"domain",
|
||||
"ip-src",
|
||||
"ip-dst"
|
||||
],
|
||||
"output": [
|
||||
"ip-src",
|
||||
"ip-dst",
|
||||
"hostname",
|
||||
"domain"
|
||||
]
|
||||
},
|
||||
"meta": {
|
||||
"description": "PassiveTotal expansion service to expand values with multiple Passive DNS sources",
|
||||
"config": [
|
||||
"username",
|
||||
"password"
|
||||
],
|
||||
"author": "Alexandre Dulaunoy",
|
||||
"version": "0.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "sourcecache",
|
||||
"type": "expansion",
|
||||
"mispattributes": {
|
||||
"input": [
|
||||
"link"
|
||||
],
|
||||
"output": [
|
||||
"link"
|
||||
]
|
||||
},
|
||||
"meta": {
|
||||
"description": "Module to cache web pages of analysis reports, OSINT sources. The module returns a link of the cached page.",
|
||||
"author": "Alexandre Dulaunoy",
|
||||
"version": "0.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "dns",
|
||||
"type": "expansion",
|
||||
"mispattributes": {
|
||||
"input": [
|
||||
"hostname",
|
||||
"domain"
|
||||
],
|
||||
"output": [
|
||||
"ip-src",
|
||||
"ip-dst"
|
||||
]
|
||||
},
|
||||
"meta": {
|
||||
"description": "Simple DNS expansion service to resolve IP address from MISP attributes",
|
||||
"author": "Alexandre Dulaunoy",
|
||||
"version": "0.1"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
~~~
|
||||
|
||||
The MISP module service returns the available modules in a JSON array containing each module name along with their supported input attributes.
|
||||
|
||||
Based on this information, a query can be built in a JSON format and saved as body.json:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"hostname": "www.foo.be",
|
||||
"module": "dns"
|
||||
}
|
||||
~~~
|
||||
|
||||
Then you can POST this JSON format query towards the MISP object server:
|
||||
|
||||
~~~bash
|
||||
curl -s http://127.0.0.1:6666/query -H "Content-Type: application/json" --data @body.json -X POST
|
||||
~~~
|
||||
|
||||
The module should output the following JSON:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"types": [
|
||||
"ip-src",
|
||||
"ip-dst"
|
||||
],
|
||||
"values": [
|
||||
"188.65.217.78"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
~~~
|
||||
|
||||
It is also possible to restrict the category options of the resolved attributes by passing a list of categories along (optional):
|
||||
|
||||
~~~json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"types": [
|
||||
"ip-src",
|
||||
"ip-dst"
|
||||
],
|
||||
"values": [
|
||||
"188.65.217.78"
|
||||
],
|
||||
"categories": [
|
||||
"Network activity",
|
||||
"Payload delivery"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
~~~
|
||||
|
||||
For both the type and the category lists, the first item in the list will be the default setting on the interface.
|
||||
|
||||
### Enable your module in the web interface
|
||||
|
||||
For a module to be activated in the MISP web interface it must be enabled in the "Plugin Settings.
|
||||
|
||||
Go to "Administration > Server Settings" in the top menu
|
||||
- Go to "Plugin Settings" in the top "tab menu bar"
|
||||
- Click on the name of the type of module you have created to expand the list of plugins to show your module.
|
||||
- Find the name of your plugin's "enabled" value in the Setting Column.
|
||||
"Plugin.[MODULE NAME]_enabled"
|
||||
- Double click on its "Value" column
|
||||
|
||||
~~~
|
||||
Priority Setting Value Description Error Message
|
||||
Recommended Plugin.Import_ocr_enabled false Enable or disable the ocr module. Value not set.
|
||||
~~~
|
||||
|
||||
- Use the drop-down to set the enabled value to 'true'
|
||||
|
||||
~~~
|
||||
Priority Setting Value Description Error Message
|
||||
Recommended Plugin.Import_ocr_enabled true Enable or disable the ocr module. Value not set.
|
||||
~~~
|
||||
|
||||
### Set any other required settings for your module
|
||||
|
||||
In this same menu set any other plugin settings that are required for testing.
|
||||
|
||||
## Install misp-module on an offline instance.
|
||||
First, you need to grab all necessary packages for example like this :
|
||||
|
||||
Use pip wheel to create an archive
|
||||
~~~
|
||||
mkdir misp-modules-offline
|
||||
pip3 wheel -r REQUIREMENTS shodan --wheel-dir=./misp-modules-offline
|
||||
tar -cjvf misp-module-bundeled.tar.bz2 ./misp-modules-offline/*
|
||||
~~~
|
||||
On offline machine :
|
||||
~~~
|
||||
mkdir misp-modules-bundle
|
||||
tar xvf misp-module-bundeled.tar.bz2 -C misp-modules-bundle
|
||||
cd misp-modules-bundle
|
||||
ls -1|while read line; do sudo pip3 install --force-reinstall --ignore-installed --upgrade --no-index --no-deps ${line};done
|
||||
~~~
|
||||
Next you can follow standard install procedure.
|
||||
|
||||
## How to contribute your own module?
|
||||
|
||||
Fork the project, add your module, test it and make a pull-request. Modules can be also private as you can add a module in your own MISP installation.
|
||||
|
||||
|
||||
## Tips for developers creating modules
|
||||
|
||||
Download a pre-built virtual image from the [MISP training materials](https://www.circl.lu/services/misp-training-materials/).
|
||||
|
||||
- Create a Host-Only adapter in VirtualBox
|
||||
- Set your Misp OVA to that Host-Only adapter
|
||||
- Start the virtual machine
|
||||
- Get the IP address of the virtual machine
|
||||
- SSH into the machine (Login info on training page)
|
||||
- Go into the misp-modules directory
|
||||
|
||||
~~~bash
|
||||
cd /usr/local/src/misp-modules
|
||||
~~~
|
||||
|
||||
Set the git repo to your fork and checkout your development branch. If you SSH'ed in as the misp user you will have to use sudo.
|
||||
|
||||
~~~bash
|
||||
sudo git remote set-url origin https://github.com/YourRepo/misp-modules.git
|
||||
sudo git pull
|
||||
sudo git checkout MyModBranch
|
||||
~~~
|
||||
|
||||
Remove the contents of the build directory and re-install misp-modules.
|
||||
|
||||
~~~bash
|
||||
sudo rm -fr build/*
|
||||
sudo -u www-data /var/www/MISP/venv/bin/pip install --upgrade .
|
||||
~~~
|
||||
|
||||
SSH in with a different terminal and run `misp-modules` with debugging enabled.
|
||||
|
||||
~~~bash
|
||||
# In case misp-modules is not a service do:
|
||||
# sudo killall misp-modules
|
||||
sudo systemctl disable --now misp-modules
|
||||
sudo -u www-data /var/www/MISP/venv/bin/misp-modules -d
|
||||
~~~
|
||||
|
||||
|
||||
In your original terminal you can now run your tests manually and see any errors that arrive
|
||||
|
||||
~~~bash
|
||||
cd tests/
|
||||
curl -s http://127.0.0.1:6666/query -H "Content-Type: application/json" --data @MY_TEST_FILE.json -X POST
|
||||
cd ../
|
||||
~~~
|
||||
|
||||
## Documentation
|
||||
|
||||
The new documentation can found [here](https://misp.github.io/misp-modules).
|
||||
In order to provide documentation about some modules that require specific input / output / configuration, the [doc](doc) directory contains detailed information about the general purpose, requirements, features, input and ouput of each of these modules:
|
||||
|
||||
|
||||
## License
|
||||
[](https://app.fossa.io/projects/git%2Bgithub.com%2F8ear%2Fmisp-modules?ref=badge_large)
|
||||
- ***description** - quick description of the general purpose of the module, as the one given by the moduleinfo
|
||||
- **requirements** - special libraries needed to make the module work
|
||||
- **features** - description of the way to use the module, with the required MISP features to make the module give the intended result
|
||||
- **references** - link(s) giving additional information about the format concerned in the module
|
||||
- **input** - description of the format of data used in input
|
||||
- **output** - description of the format given as the result of the module execution
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
-e git+https://github.com/D4-project/BGP-Ranking.git/@429cea9c0787876820984a2df4e982449a84c10e#egg=pybgpranking&subdirectory=client
|
||||
-e git+https://github.com/D4-project/IPASN-History.git/@47cd0f2658ab172fce42126ff3a1dbcddfb0b5fb#egg=pyipasnhistory&subdirectory=client
|
||||
-e git+https://github.com/MISP/PyIntel471.git@0df8d51f1c1425de66714b3a5a45edb69b8cc2fc#egg=pyintel471
|
||||
-e git+https://github.com/MISP/PyMISP.git@583fb6592495ea358aad47a8a1ec92d43c13348a#egg=pymisp
|
||||
-e git+https://github.com/MISP/PyMISP.git@3ad351380055f0a655ed529b9c79b242a9227b84#egg=pymisp
|
||||
-e git+https://github.com/Rafiot/uwhoisd.git@411572840eba4c72dc321c549b36a54ed5cea9de#egg=uwhois&subdirectory=client
|
||||
-e git+https://github.com/cartertemm/ODTReader.git/@49d6938693f6faa3ff09998f86dba551ae3a996b#egg=odtreader
|
||||
-e git+https://github.com/sebdraven/pydnstrails@48c1f740025c51289f43a24863d1845ff12fd21a#egg=pydnstrails
|
||||
|
|
|
|||
|
|
@ -322,8 +322,9 @@ In order to provide documentation about some modules that require specific input
|
|||
- **input** - description of the format of data used in input
|
||||
- **output** - description of the format given as the result of the module execution
|
||||
|
||||
In addition to the modul documentation please add your module to [docs/index.md](https://github.com/MISP/misp-modules/tree/master/docs/index.md).
|
||||
In addition to the module documentation please add your module to [docs/index.md](https://github.com/MISP/misp-modules/tree/master/docs/index.md).
|
||||
|
||||
There are also [complementary slides](https://www.misp-project.org/misp-training/3.1-misp-modules.pdf) for the creation of MISP modules.
|
||||
|
||||
|
||||
## Tips for developers creating modules
|
||||
|
|
|
|||
|
|
@ -405,7 +405,7 @@ class JoeParser():
|
|||
def finalize_results(self):
|
||||
if self.references:
|
||||
self.build_references()
|
||||
event = json.loads(self.misp_event.to_json())['Event']
|
||||
event = json.loads(self.misp_event.to_json())
|
||||
self.results = {key: event[key] for key in ('Attribute', 'Object', 'Tag') if (key in event and event[key])}
|
||||
|
||||
@staticmethod
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
from collections import defaultdict
|
||||
from pymisp import MISPEvent, MISPObject
|
||||
import json
|
||||
import requests
|
||||
|
|
@ -12,17 +13,25 @@ cveapi_url = 'https://cve.circl.lu/api/cve/'
|
|||
|
||||
|
||||
class VulnerabilityParser():
|
||||
def __init__(self, vulnerability):
|
||||
def __init__(self, attribute, vulnerability):
|
||||
self.attribute = attribute
|
||||
self.vulnerability = vulnerability
|
||||
self.misp_event = MISPEvent()
|
||||
self.misp_event.add_attribute(**attribute)
|
||||
self.references = defaultdict(list)
|
||||
self.capec_features = ('id', 'name', 'summary', 'prerequisites', 'solutions')
|
||||
self.vulnerability_mapping = {
|
||||
'id': ('text', 'id'), 'summary': ('text', 'summary'),
|
||||
'vulnerable_configuration_cpe_2_2': ('text', 'vulnerable_configuration'),
|
||||
'Modified': ('datetime', 'modified'), 'Published': ('datetime', 'published'),
|
||||
'references': ('link', 'references'), 'cvss': ('float', 'cvss-score')}
|
||||
self.weakness_mapping = {'name': 'name', 'description_summary': 'description',
|
||||
'status': 'status', 'weaknessabs': 'weakness-abs'}
|
||||
|
||||
def get_result(self):
|
||||
event = json.loads(self.misp_event.to_json())['Event']
|
||||
if self.references:
|
||||
self.__build_references()
|
||||
event = json.loads(self.misp_event.to_json())
|
||||
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
|
||||
return {'results': results}
|
||||
|
||||
|
|
@ -41,7 +50,50 @@ class VulnerabilityParser():
|
|||
attribute_type, relation = self.vulnerability_mapping[feature]
|
||||
for value in self.vulnerability[feature]:
|
||||
vulnerability_object.add_attribute(relation, **{'type': attribute_type, 'value': value})
|
||||
vulnerability_object.add_reference(self.attribute['uuid'], 'related-to')
|
||||
self.misp_event.add_object(**vulnerability_object)
|
||||
if 'cwe' in self.vulnerability and self.vulnerability['cwe'] != 'Unknown':
|
||||
self.__parse_weakness(vulnerability_object.uuid)
|
||||
if 'capec' in self.vulnerability:
|
||||
self.__parse_capec(vulnerability_object.uuid)
|
||||
|
||||
def __build_references(self):
|
||||
for object_uuid, references in self.references.items():
|
||||
for misp_object in self.misp_event.objects:
|
||||
if misp_object.uuid == object_uuid:
|
||||
for reference in references:
|
||||
misp_object.add_reference(**reference)
|
||||
break
|
||||
|
||||
def __parse_capec(self, vulnerability_uuid):
|
||||
attribute_type = 'text'
|
||||
for capec in self.vulnerability['capec']:
|
||||
capec_object = MISPObject('attack-pattern')
|
||||
for feature in self.capec_features:
|
||||
capec_object.add_attribute(feature, **dict(type=attribute_type, value=capec[feature]))
|
||||
for related_weakness in capec['related_weakness']:
|
||||
attribute = dict(type='weakness', value="CWE-{}".format(related_weakness))
|
||||
capec_object.add_attribute('related-weakness', **attribute)
|
||||
self.misp_event.add_object(**capec_object)
|
||||
self.references[vulnerability_uuid].append(dict(referenced_uuid=capec_object.uuid,
|
||||
relationship_type='targeted-by'))
|
||||
|
||||
def __parse_weakness(self, vulnerability_uuid):
|
||||
attribute_type = 'text'
|
||||
cwe_string, cwe_id = self.vulnerability['cwe'].split('-')
|
||||
cwes = requests.get(cveapi_url.replace('/cve/', '/cwe'))
|
||||
if cwes.status_code == 200:
|
||||
for cwe in cwes.json():
|
||||
if cwe['id'] == cwe_id:
|
||||
weakness_object = MISPObject('weakness')
|
||||
weakness_object.add_attribute('id', **dict(type=attribute_type, value='-'.join([cwe_string, cwe_id])))
|
||||
for feature, relation in self.weakness_mapping.items():
|
||||
if cwe.get(feature):
|
||||
weakness_object.add_attribute(relation, **dict(type=attribute_type, value=cwe[feature]))
|
||||
self.misp_event.add_object(**weakness_object)
|
||||
self.references[vulnerability_uuid].append(dict(referenced_uuid=weakness_object.uuid,
|
||||
relationship_type='weakened-by'))
|
||||
break
|
||||
|
||||
|
||||
def handler(q=False):
|
||||
|
|
@ -61,7 +113,7 @@ def handler(q=False):
|
|||
else:
|
||||
misperrors['error'] = 'cve.circl.lu API not accessible'
|
||||
return misperrors['error']
|
||||
parser = VulnerabilityParser(vulnerability)
|
||||
parser = VulnerabilityParser(attribute, vulnerability)
|
||||
parser.parse_vulnerability_information()
|
||||
return parser.get_result()
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ class URLhaus():
|
|||
return vt_object
|
||||
|
||||
def get_result(self):
|
||||
event = json.loads(self.misp_event.to_json())['Event']
|
||||
event = json.loads(self.misp_event.to_json())
|
||||
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
|
||||
return {'results': results}
|
||||
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ class VirusTotalParser(object):
|
|||
return self.input_types_mapping[self.attribute.type](self.attribute.value, recurse=True)
|
||||
|
||||
def get_result(self):
|
||||
event = json.loads(self.misp_event.to_json())['Event']
|
||||
event = json.loads(self.misp_event.to_json())
|
||||
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
|
||||
return {'results': results}
|
||||
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ class VirusTotalParser():
|
|||
self.apikey = apikey
|
||||
|
||||
def get_result(self):
|
||||
event = json.loads(self.misp_event.to_json())['Event']
|
||||
event = json.loads(self.misp_event.to_json())
|
||||
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
|
||||
return {'results': results}
|
||||
|
||||
|
|
|
|||
|
|
@ -194,7 +194,7 @@ class CsvParser():
|
|||
return list2pop, misp, list(reversed(head))
|
||||
|
||||
def finalize_results(self):
|
||||
event = json.loads(self.misp_event.to_json())['Event']
|
||||
event = json.loads(self.misp_event.to_json())
|
||||
self.results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,198 +1,744 @@
|
|||
import json
|
||||
import base64
|
||||
import io
|
||||
import logging
|
||||
import posixpath
|
||||
import stat
|
||||
import tarfile
|
||||
import zipfile
|
||||
from pymisp import MISPEvent, MISPObject, MISPAttribute
|
||||
from pymisp.tools import make_binary_objects
|
||||
from collections import OrderedDict
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
misperrors = {'error': 'Error'}
|
||||
userConfig = {}
|
||||
inputSource = ['file']
|
||||
|
||||
moduleinfo = {'version': '0.1', 'author': 'Victor van der Stoep',
|
||||
'description': 'Cuckoo JSON import',
|
||||
'module-type': ['import']}
|
||||
moduleinfo = {
|
||||
'version': '1.1',
|
||||
'author': 'Pierre-Jean Grenier',
|
||||
'description': "Import a Cuckoo archive (zipfile or bzip2 tarball), "
|
||||
"either downloaded manually or exported from the "
|
||||
"API (/tasks/report/{task_id}/all).",
|
||||
'module-type': ['import'],
|
||||
}
|
||||
|
||||
moduleconfig = []
|
||||
|
||||
mispattributes = {
|
||||
'inputSource': ['file'],
|
||||
'output': ['MISP objects', 'malware-sample'],
|
||||
'format': 'misp_standard',
|
||||
}
|
||||
|
||||
# Attributes for which we can set the "Artifacts dropped"
|
||||
# category if we want to
|
||||
ARTIFACTS_DROPPED = (
|
||||
"filename",
|
||||
"md5",
|
||||
"sha1",
|
||||
"sha256",
|
||||
"sha512",
|
||||
"malware-sample",
|
||||
"mimetype",
|
||||
"ssdeep",
|
||||
)
|
||||
|
||||
# Same for the category "Payload delivery"
|
||||
PAYLOAD_DELIVERY = ARTIFACTS_DROPPED
|
||||
|
||||
|
||||
class PrettyDict(OrderedDict):
|
||||
"""
|
||||
This class is just intended for a pretty print
|
||||
of its keys and values.
|
||||
"""
|
||||
MAX_SIZE = 30
|
||||
|
||||
def __str__(self):
|
||||
tmp = []
|
||||
for k, v in self.items():
|
||||
v = str(v)
|
||||
if len(v) > self.MAX_SIZE:
|
||||
k += ',cut'
|
||||
v = v[:self.MAX_SIZE]
|
||||
v.replace('\n', ' ')
|
||||
tmp.append((k, v))
|
||||
return "; ".join(f"({k}) {v}" for k, v in tmp)
|
||||
|
||||
|
||||
def search_objects(event, name, attributes=[]):
|
||||
"""
|
||||
Search for objects in event, which name is `name` and
|
||||
contain at least the attributes given.
|
||||
Return a generator.
|
||||
@ param attributes: a list of (object_relation, value)
|
||||
"""
|
||||
match = filter(
|
||||
lambda obj: all(
|
||||
obj.name == name
|
||||
and (obj_relation, str(attr_value)) in map(
|
||||
lambda attr: (attr.object_relation, str(attr.value)),
|
||||
obj.attributes
|
||||
)
|
||||
for obj_relation, attr_value in attributes
|
||||
), event.objects
|
||||
)
|
||||
return match
|
||||
|
||||
|
||||
def find_process_by_pid(event, pid):
|
||||
"""
|
||||
Find a 'process' MISPObject by its PID. If multiple objects are found,
|
||||
only return the first one.
|
||||
@ param pid: integer or str
|
||||
"""
|
||||
generator = search_objects(
|
||||
event,
|
||||
"process",
|
||||
(('pid', pid),)
|
||||
)
|
||||
return next(generator, None)
|
||||
|
||||
|
||||
class CuckooParser():
|
||||
# This dict is used to generate the userConfig and link the different
|
||||
# options to the corresponding method of the parser. This way, we avoid
|
||||
# redundancy and make future changes easier (instead of for instance
|
||||
# defining all the options in userConfig directly, and then making a
|
||||
# switch when running the parser).
|
||||
# Careful about the order here, as we create references between
|
||||
# MISPObjects/MISPAttributes at the same time we generate them.
|
||||
# Hence when we create object B, which we want to reference to
|
||||
# object A, we should already have created object A.
|
||||
# TODO create references only after all parsing is done
|
||||
options = {
|
||||
"Sandbox info": {
|
||||
"method": lambda self: self.add_sandbox_info(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add info related to the sandbox",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"Upload sample": {
|
||||
"method": lambda self: self.add_sample(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Upload the sample",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"Processes": {
|
||||
"method": lambda self: self.add_process_tree(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add info related to the processes",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"DNS": {
|
||||
"method": lambda self: self.add_dns(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add DNS queries/answers",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"TCP": {
|
||||
"method": lambda self: self.add_network("tcp"),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add TCP connections",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"UDP": {
|
||||
"method": lambda self: self.add_network("udp"),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add UDP connections",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"HTTP": {
|
||||
"method": lambda self: self.add_http(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add HTTP requests",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"Signatures": {
|
||||
"method": lambda self: self.add_signatures(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Add Cuckoo's triggered signatures",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"Screenshots": {
|
||||
"method": lambda self: self.add_screenshots(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Upload the screenshots",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"Dropped files": {
|
||||
"method": lambda self: self.add_dropped_files(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Upload the dropped files",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
"Dropped buffers": {
|
||||
"method": lambda self: self.add_dropped_buffers(),
|
||||
"userConfig": {
|
||||
'type': 'Boolean',
|
||||
'message': "Upload the dropped buffers",
|
||||
'checked': 'true',
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def __init__(self, config):
|
||||
self.event = MISPEvent()
|
||||
self.files = None
|
||||
self.malware_binary = None
|
||||
self.report = None
|
||||
self.config = {
|
||||
# if an option is missing (we receive None as a value),
|
||||
# fall back to the default specified in the options
|
||||
key: int(
|
||||
on if on is not None
|
||||
else self.options[key]["userConfig"]["checked"] == 'true'
|
||||
)
|
||||
for key, on in config.items()
|
||||
}
|
||||
|
||||
def get_file(self, relative_filepath):
|
||||
"""Return an io.BufferedIOBase for the corresponding relative_filepath
|
||||
in the Cuckoo archive. If not found, return an empty io.BufferedReader
|
||||
to avoid fatal errors."""
|
||||
blackhole = io.BufferedReader(open('/dev/null', 'rb'))
|
||||
res = self.files.get(relative_filepath, blackhole)
|
||||
if res == blackhole:
|
||||
log.debug(f"Did not find file {relative_filepath}, "
|
||||
f"returned an empty file instead")
|
||||
return res
|
||||
|
||||
def read_archive(self, archive_encoded):
|
||||
"""Read the archive exported from Cuckoo and initialize the class"""
|
||||
# archive_encoded is base 64 encoded content
|
||||
# we extract the info about each file but do not retrieve
|
||||
# it automatically, as it may take too much space in memory
|
||||
buf_io = io.BytesIO(base64.b64decode(archive_encoded))
|
||||
if zipfile.is_zipfile(buf_io):
|
||||
# the archive was probably downloaded from the WebUI
|
||||
buf_io.seek(0) # don't forget this not to read an empty buffer
|
||||
z = zipfile.ZipFile(buf_io, 'r')
|
||||
self.files = {
|
||||
info.filename: z.open(info)
|
||||
for info in z.filelist
|
||||
# only extract the regular files and dirs, we don't
|
||||
# want any symbolic link
|
||||
if stat.S_ISREG(info.external_attr >> 16)
|
||||
or stat.S_ISDIR(info.external_attr >> 16)
|
||||
}
|
||||
else:
|
||||
# the archive was probably downloaded from the API
|
||||
buf_io.seek(0) # don't forget this not to read an empty buffer
|
||||
f = tarfile.open(fileobj=buf_io, mode='r:bz2')
|
||||
self.files = {
|
||||
info.name: f.extractfile(info)
|
||||
for info in f.getmembers()
|
||||
# only extract the regular files and dirs, we don't
|
||||
# want any symbolic link
|
||||
if info.isreg() or info.isdir()
|
||||
}
|
||||
|
||||
# We want to keep the order of the keys of sub-dicts in the report,
|
||||
# eg. the signatures have marks with unknown keys such as
|
||||
# {'marks': [
|
||||
# {"suspicious_features": "Connection to IP address",
|
||||
# "suspicious_request": "OPTIONS http://85.20.18.18/doc"}
|
||||
# ]}
|
||||
# To render those marks properly, we can only hope the developpers
|
||||
# thought about the order in which they put the keys, and keep this
|
||||
# order so that the signature makes sense to the reader.
|
||||
# We use PrettyDict, a customization of OrderedDict to do so.
|
||||
# It will be instanced iteratively when parsing the json (ie. subdicts
|
||||
# will also be instanced as PrettyDict)
|
||||
self.report = json.load(
|
||||
self.get_file("reports/report.json"),
|
||||
object_pairs_hook=PrettyDict,
|
||||
)
|
||||
|
||||
def read_malware(self):
|
||||
self.malware_binary = self.get_file("binary").read()
|
||||
if not self.malware_binary:
|
||||
log.warn("No malware binary found")
|
||||
|
||||
def add_sandbox_info(self):
|
||||
info = self.report.get("info", {})
|
||||
if not info:
|
||||
log.warning("The 'info' field was not found "
|
||||
"in the report, skipping")
|
||||
return False
|
||||
|
||||
o = MISPObject(name='sandbox-report')
|
||||
o.add_attribute('score', info['score'])
|
||||
o.add_attribute('sandbox-type', 'on-premise')
|
||||
o.add_attribute('on-premise-sandbox', 'cuckoo')
|
||||
o.add_attribute('raw-report',
|
||||
f'started on:{info["machine"]["started_on"]} '
|
||||
f'duration:{info["duration"]}s '
|
||||
f'vm:{info["machine"]["name"]}/'
|
||||
f'{info["machine"]["label"]}')
|
||||
self.event.add_object(o)
|
||||
|
||||
def add_sample(self):
|
||||
"""Add the sample/target of the analysis"""
|
||||
target = self.report.get("target", {})
|
||||
category = target.get("category", "")
|
||||
if not category:
|
||||
log.warning("Could not find info about the sample "
|
||||
"in the report, skipping")
|
||||
return False
|
||||
|
||||
if category == "file":
|
||||
log.debug("Sample is a file, uploading it")
|
||||
self.read_malware()
|
||||
file_o, bin_type_o, bin_section_li = make_binary_objects(
|
||||
pseudofile=io.BytesIO(self.malware_binary),
|
||||
filename=target["file"]["name"],
|
||||
)
|
||||
|
||||
file_o.comment = "Submitted sample"
|
||||
# fix categories
|
||||
for obj in filter(None, (file_o, bin_type_o, *bin_section_li,)):
|
||||
for attr in obj.attributes:
|
||||
if attr.type in PAYLOAD_DELIVERY:
|
||||
attr.category = "Payload delivery"
|
||||
self.event.add_object(obj)
|
||||
|
||||
elif category == "url":
|
||||
log.debug("Sample is a URL")
|
||||
o = MISPObject(name='url')
|
||||
o.add_attribute('url', target['url'])
|
||||
o.add_attribute('text', "Submitted URL")
|
||||
self.event.add_object(o)
|
||||
|
||||
def add_http(self):
|
||||
"""Add the HTTP requests"""
|
||||
network = self.report.get("network", [])
|
||||
http = network.get("http", [])
|
||||
if not http:
|
||||
log.info("No HTTP connection found in the report, skipping")
|
||||
return False
|
||||
|
||||
for request in http:
|
||||
o = MISPObject(name='http-request')
|
||||
o.add_attribute('host', request['host'])
|
||||
o.add_attribute('method', request['method'])
|
||||
o.add_attribute('uri', request['uri'])
|
||||
o.add_attribute('user-agent', request['user-agent'])
|
||||
o.add_attribute('text', f"count:{request['count']} "
|
||||
f"port:{request['port']}")
|
||||
self.event.add_object(o)
|
||||
|
||||
def add_network(self, proto=None):
|
||||
"""
|
||||
Add UDP/TCP traffic
|
||||
proto must be one of "tcp", "udp"
|
||||
"""
|
||||
network = self.report.get("network", [])
|
||||
li_conn = network.get(proto, [])
|
||||
if not li_conn:
|
||||
log.info(f"No {proto} connection found in the report, skipping")
|
||||
return False
|
||||
|
||||
from_to = []
|
||||
# sort by time to get the "first packet seen" right
|
||||
li_conn.sort(key=lambda x: x["time"])
|
||||
for conn in li_conn:
|
||||
src = conn['src']
|
||||
dst = conn['dst']
|
||||
sport = conn['sport']
|
||||
dport = conn['dport']
|
||||
if (src, sport, dst, dport) in from_to:
|
||||
continue
|
||||
|
||||
from_to.append((src, sport, dst, dport))
|
||||
|
||||
o = MISPObject(name='network-connection')
|
||||
o.add_attribute('ip-src', src)
|
||||
o.add_attribute('ip-dst', dst)
|
||||
o.add_attribute('src-port', sport)
|
||||
o.add_attribute('dst-port', dport)
|
||||
o.add_attribute('layer3-protocol', "IP")
|
||||
o.add_attribute('layer4-protocol', proto.upper())
|
||||
o.add_attribute('first-packet-seen', conn['time'])
|
||||
self.event.add_object(o)
|
||||
|
||||
def add_dns(self):
|
||||
"""Add DNS records"""
|
||||
network = self.report.get("network", [])
|
||||
dns = network.get("dns", [])
|
||||
if not dns:
|
||||
log.info("No DNS connection found in the report, skipping")
|
||||
return False
|
||||
|
||||
for record in dns:
|
||||
o = MISPObject(name='dns-record')
|
||||
o.add_attribute('text', f"request type:{record['type']}")
|
||||
o.add_attribute('queried-domain', record['request'])
|
||||
for answer in record.get("answers", []):
|
||||
if answer["type"] in ("A", "AAAA"):
|
||||
o.add_attribute('a-record', answer['data'])
|
||||
# TODO implement MX/NS
|
||||
|
||||
self.event.add_object(o)
|
||||
|
||||
def _get_marks_str(self, marks):
|
||||
marks_strings = []
|
||||
for m in marks:
|
||||
m_type = m.pop("type") # temporarily remove the type
|
||||
|
||||
if m_type == "generic":
|
||||
marks_strings.append(str(m))
|
||||
|
||||
elif m_type == "ioc":
|
||||
marks_strings.append(m['ioc'])
|
||||
|
||||
elif m_type == "call":
|
||||
call = m["call"]
|
||||
arguments = call.get("arguments", {})
|
||||
flags = call.get("flags", {})
|
||||
info = ""
|
||||
for details in (arguments, flags):
|
||||
info += f" {details}"
|
||||
marks_strings.append(f"Call API '{call['api']}'%s" % info)
|
||||
|
||||
else:
|
||||
logging.debug(f"Unknown mark type '{m_type}', skipping")
|
||||
|
||||
m["type"] = m_type # restore key 'type'
|
||||
# TODO implemented marks 'config' and 'volatility'
|
||||
return marks_strings
|
||||
|
||||
def _add_ttp(self, attribute, ttp_short, ttp_num):
|
||||
"""
|
||||
Internal wrapper to add the TTP tag from the MITRE galaxy.
|
||||
@ params
|
||||
- attribute: MISPAttribute
|
||||
- ttp_short: short description of the TTP
|
||||
(eg. "Credential Dumping")
|
||||
- ttp_num: formatted as "T"+int
|
||||
(eg. T1003)
|
||||
"""
|
||||
attribute.add_tag(f'misp-galaxy:mitre-attack-pattern='
|
||||
f'"{ttp_short} - {ttp_num}"')
|
||||
|
||||
def add_signatures(self):
|
||||
"""Add the Cuckoo signatures, with as many details as possible
|
||||
regarding the marks"""
|
||||
signatures = self.report.get("signatures", [])
|
||||
if not signatures:
|
||||
log.info("No signature found in the report")
|
||||
return False
|
||||
|
||||
o = MISPObject(name='sb-signature')
|
||||
o.add_attribute('software', "Cuckoo")
|
||||
|
||||
for sign in signatures:
|
||||
marks = sign["marks"]
|
||||
marks_strings = self._get_marks_str(marks)
|
||||
summary = sign['description']
|
||||
if marks_strings:
|
||||
summary += "\n---\n"
|
||||
|
||||
marks_strings = set(marks_strings)
|
||||
description = summary + "\n".join(marks_strings)
|
||||
|
||||
a = MISPAttribute()
|
||||
a.from_dict(type='text', value=description)
|
||||
for ttp_num, desc in sign.get("ttp", {}).items():
|
||||
ttp_short = desc["short"]
|
||||
self._add_ttp(a, ttp_short, ttp_num)
|
||||
|
||||
# this signature was triggered by the processes with the following
|
||||
# PIDs, we can create references
|
||||
triggered_by_pids = filter(
|
||||
None,
|
||||
(m.get("pid", None) for m in marks)
|
||||
)
|
||||
# remove redundancy
|
||||
triggered_by_pids = set(triggered_by_pids)
|
||||
for pid in triggered_by_pids:
|
||||
process_o = find_process_by_pid(self.event, pid)
|
||||
if process_o:
|
||||
process_o.add_reference(a, "triggers")
|
||||
|
||||
o.add_attribute('signature', **a)
|
||||
|
||||
self.event.add_object(o)
|
||||
|
||||
def _handle_process(self, proc, accu):
|
||||
"""
|
||||
This is an internal recursive function to handle one process
|
||||
from a process tree and then iterate on its children.
|
||||
List the objects to be added, based on the tree, into the `accu` list.
|
||||
The `accu` list uses a DFS-like order.
|
||||
"""
|
||||
o = MISPObject(name='process')
|
||||
accu.append(o)
|
||||
o.add_attribute('pid', proc['pid'])
|
||||
o.add_attribute('command-line', proc['command_line'])
|
||||
o.add_attribute('name', proc['process_name'])
|
||||
o.add_attribute('parent-pid', proc['ppid'])
|
||||
for child in proc.get('children', []):
|
||||
pos_child = len(accu)
|
||||
o.add_attribute('child-pid', child['pid'])
|
||||
self._handle_process(child, accu)
|
||||
child_obj = accu[pos_child]
|
||||
child_obj.add_reference(o, 'child-of')
|
||||
|
||||
return o
|
||||
|
||||
def add_process_tree(self):
|
||||
"""Add process tree from the report, as separated process objects"""
|
||||
behavior = self.report.get("behavior", {})
|
||||
tree = behavior.get("processtree", [])
|
||||
if not tree:
|
||||
log.warning("No process tree found in the report, skipping")
|
||||
return False
|
||||
|
||||
for proc in tree:
|
||||
objs = []
|
||||
self._handle_process(proc, objs)
|
||||
for o in objs:
|
||||
self.event.add_object(o)
|
||||
|
||||
def get_relpath(self, path):
|
||||
"""
|
||||
Transform an absolute or relative path into a path relative to the
|
||||
correct cuckoo analysis directory, without knowing the cuckoo
|
||||
working directory.
|
||||
Return an empty string if the path given does not refer to a
|
||||
file from the analysis directory.
|
||||
"""
|
||||
head, tail = posixpath.split(path)
|
||||
if not tail:
|
||||
return ""
|
||||
prev = self.get_relpath(head)
|
||||
longer = posixpath.join(prev, tail)
|
||||
if longer in self.files:
|
||||
return longer
|
||||
elif tail in self.files:
|
||||
return tail
|
||||
else:
|
||||
return ""
|
||||
|
||||
def add_screenshots(self):
|
||||
"""Add the screenshots taken by Cuckoo in a sandbox-report object"""
|
||||
screenshots = self.report.get('screenshots', [])
|
||||
if not screenshots:
|
||||
log.info("No screenshot found in the report, skipping")
|
||||
return False
|
||||
|
||||
o = MISPObject(name='sandbox-report')
|
||||
o.add_attribute('sandbox-type', 'on-premise')
|
||||
o.add_attribute('on-premise-sandbox', "cuckoo")
|
||||
for shot in screenshots:
|
||||
# The path given by Cuckoo is an absolute path, but we need a path
|
||||
# relative to the analysis folder.
|
||||
path = self.get_relpath(shot['path'])
|
||||
img = self.get_file(path)
|
||||
# .decode('utf-8') in order to avoid the b'' format
|
||||
img_data = base64.b64encode(img.read()).decode('utf-8')
|
||||
filename = posixpath.basename(path)
|
||||
|
||||
o.add_attribute(
|
||||
"sandbox-file", value=filename,
|
||||
data=img_data, type='attachment',
|
||||
category="External analysis",
|
||||
)
|
||||
|
||||
self.event.add_object(o)
|
||||
|
||||
def _get_dropped_objs(self, path, filename=None, comment=None):
|
||||
"""
|
||||
Internal wrapper to get dropped files/buffers as file objects
|
||||
@ params
|
||||
- path: relative to the cuckoo analysis directory
|
||||
- filename: if not specified, deduced from the path
|
||||
"""
|
||||
if not filename:
|
||||
filename = posixpath.basename(path)
|
||||
|
||||
dropped_file = self.get_file(path)
|
||||
dropped_binary = io.BytesIO(dropped_file.read())
|
||||
# create ad hoc objects
|
||||
file_o, bin_type_o, bin_section_li = make_binary_objects(
|
||||
pseudofile=dropped_binary, filename=filename,
|
||||
)
|
||||
|
||||
if comment:
|
||||
file_o.comment = comment
|
||||
# fix categories
|
||||
for obj in filter(None, (file_o, bin_type_o, *bin_section_li,)):
|
||||
for attr in obj.attributes:
|
||||
if attr.type in ARTIFACTS_DROPPED:
|
||||
attr.category = "Artifacts dropped"
|
||||
|
||||
return file_o, bin_type_o, bin_section_li
|
||||
|
||||
def _add_yara(self, obj, yara_dict):
|
||||
"""Internal wrapper to add Yara matches to an MISPObject"""
|
||||
for yara in yara_dict:
|
||||
description = yara.get("meta", {}).get("description", "")
|
||||
name = yara.get("name", "")
|
||||
obj.add_attribute(
|
||||
"text",
|
||||
f"Yara match\n(name) {name}\n(description) {description}",
|
||||
comment="Yara match"
|
||||
)
|
||||
|
||||
def add_dropped_files(self):
|
||||
"""Upload the dropped files as file objects"""
|
||||
dropped = self.report.get("dropped", [])
|
||||
if not dropped:
|
||||
log.info("No dropped file found, skipping")
|
||||
return False
|
||||
|
||||
for d in dropped:
|
||||
# Cuckoo logs three things that are of interest for us:
|
||||
# - 'filename' which is not the original name of the file
|
||||
# but is formatted as follow:
|
||||
# 8 first bytes of SHA265 + _ + original name in lower case
|
||||
# - 'filepath' which is the original filepath on the VM,
|
||||
# where the file was dropped
|
||||
# - 'path' which is the local path of the stored file,
|
||||
# in the cuckoo archive
|
||||
filename = d.get("name", "")
|
||||
original_path = d.get("filepath", "")
|
||||
sha256 = d.get("sha256", "")
|
||||
if original_path and sha256:
|
||||
log.debug(f"Will now try to restore original filename from "
|
||||
f"path {original_path}")
|
||||
try:
|
||||
s = filename.split("_")
|
||||
if not s:
|
||||
raise Exception("unexpected filename read "
|
||||
"in the report")
|
||||
sha256_first_8_bytes = s[0]
|
||||
original_name = s[1]
|
||||
# check our assumptions are valid, if so we can safely
|
||||
# restore the filename, if not the format may have changed
|
||||
# so we'll keep the filename of the report
|
||||
if sha256.startswith(sha256_first_8_bytes) and \
|
||||
original_path.lower().endswith(original_name) and \
|
||||
filename not in original_path.lower():
|
||||
# we can restore the original case of the filename
|
||||
position = original_path.lower().rindex(original_name)
|
||||
filename = original_path[position:]
|
||||
log.debug(f"Successfully restored original filename: "
|
||||
f"{filename}")
|
||||
else:
|
||||
raise Exception("our assumptions were wrong, "
|
||||
"filename format may have changed")
|
||||
except Exception as e:
|
||||
log.debug(f"Cannot restore filename: {e}")
|
||||
|
||||
if not filename:
|
||||
filename = "NO NAME FOUND IN THE REPORT"
|
||||
log.warning(f'No filename found for dropped file! '
|
||||
f'Will use "{filename}"')
|
||||
|
||||
file_o, bin_type_o, bin_section_o = self._get_dropped_objs(
|
||||
self.get_relpath(d['path']),
|
||||
filename=filename,
|
||||
comment="Dropped file"
|
||||
)
|
||||
|
||||
self._add_yara(file_o, d.get("yara", []))
|
||||
|
||||
file_o.add_attribute("fullpath", original_path,
|
||||
category="Artifacts dropped")
|
||||
|
||||
# why is this a list? for when various programs drop the same file?
|
||||
for pid in d.get("pids", []):
|
||||
# if we have an object for the process that dropped the file,
|
||||
# we can link the two (we just take the first result from
|
||||
# the search)
|
||||
process_o = find_process_by_pid(self.event, pid)
|
||||
if process_o:
|
||||
file_o.add_reference(process_o, "dropped-by")
|
||||
|
||||
self.event.add_object(file_o)
|
||||
|
||||
def add_dropped_buffers(self):
|
||||
""""Upload the dropped buffers as file objects"""
|
||||
buffer = self.report.get("buffer", [])
|
||||
if not buffer:
|
||||
log.info("No dropped buffer found, skipping")
|
||||
return False
|
||||
|
||||
for i, buf in enumerate(buffer):
|
||||
file_o, bin_type_o, bin_section_o = self._get_dropped_objs(
|
||||
self.get_relpath(buf['path']),
|
||||
filename=f"buffer {i}",
|
||||
comment="Dropped buffer"
|
||||
)
|
||||
self._add_yara(file_o, buf.get("yara", []))
|
||||
self.event.add_object(file_o)
|
||||
|
||||
def parse(self):
|
||||
"""Run the parsing"""
|
||||
for name, active in self.config.items():
|
||||
if active:
|
||||
self.options[name]["method"](self)
|
||||
|
||||
def get_misp_event(self):
|
||||
log.debug("Running MISP expansions")
|
||||
self.event.run_expansions()
|
||||
return self.event
|
||||
|
||||
|
||||
def handler(q=False):
|
||||
# Just in case we have no data
|
||||
# In case there's no data
|
||||