MISP
/
misp-modules
mirror of https://github.com/MISP/misp-modules
2
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'main' of github.com:MISP/misp-modules into main

pull/431/head
Raphaël Vinot 2020-10-09 14:24:30 +02:00
parent a907613ce2 0072e04627
commit 608bad1542
48 changed files with 1248 additions and 138 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@ -22,7 +22,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
* [AssemblyLine submit](misp_modules/modules/expansion/assemblyline_submit.py) - an expansion module to submit samples and urls to AssemblyLine.
* [AssemblyLine query](misp_modules/modules/expansion/assemblyline_query.py) - an expansion module to query AssemblyLine and parse the full submission report.
* [Backscatter.io](misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations.
* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description, its history, and position in BGP Ranking.
* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description and its ranking and position in BGP Ranking.
* [RansomcoinDB check](misp_modules/modules/expansion/ransomcoindb.py) - An expansion hover module to query the [ransomcoinDB](https://ransomcoindb.concinnity-risks.com): it contains mapping between BTC addresses and malware hashes. Enrich MISP by querying for BTC -> hash or hash -> BTC addresses.
* [BTC scam check](misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused.
* [BTC transactions](misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP.
@ -69,6 +69,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
* [pptx-enrich](misp_modules/modules/expansion/pptx_enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser).
* [qrcode](misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values.
* [rbl](misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute.
* [recordedfuture](misp_modules/modules/expansion/recordedfuture.py) - a hover and expansion module for enriching MISP attributes with threat intelligence from Recorded Future.
* [reversedns](misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes.
* [securitytrails](misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/).
* [shodan](misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module.
@ -79,6 +80,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
* [STIX2 pattern syntax validator](misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax.
* [ThreatCrowd](misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/).
* [threatminer](misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/).
* [TruSTAR Enrich](misp_modules/modules/expansion/trustar_enrich.py) - an expansion module to enrich MISP data with [TruSTAR](https://www.trustar.co/).
* [urlhaus](misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url.
* [urlscan](misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io).
* [virustotal](misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://developers.virustotal.com/reference))

181
doc/README.md
View File

@ -108,13 +108,13 @@ Query backscatter.io (https://backscatter.io/).
Query BGP Ranking (https://bgpranking-ng.circl.lu/).
- **features**:
>The module takes an AS number attribute as input and displays its description and history, and position in BGP Ranking.
>The module takes an AS number attribute as input and displays its description as well as its ranking position in BGP Ranking for a given day.
>
>
- **input**:
>Autonomous system number.
- **output**:
>Text containing a description of the ASN, its history, and the position in BGP Ranking.
>An asn object with its related bgp-ranking object.
- **references**:
>https://github.com/D4-project/BGP-Ranking/
- **requirements**:
@ -311,6 +311,26 @@ An expansion hover module to expand information about CVE id.
-----
#### [cve_advanced](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/cve_advanced.py)
<img src=logos/cve.png height=60>
An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).
- **features**:
>The module takes a vulnerability attribute as input and queries the CIRCL CVE search API to gather additional information.
>
>The result of the query is then parsed to return additional information about the vulnerability, like its cvss score or some references, as well as the potential related weaknesses and attack patterns.
>
>The vulnerability additional data is returned in a vulnerability MISP object, and the related additional information are put into weakness and attack-pattern MISP objects.
- **input**:
>Vulnerability attribute.
- **output**:
>Additional information about the vulnerability, such as its cvss score, some references, or the related weaknesses and attack patterns.
- **references**:
>https://cve.circl.lu, https://cve/mitre.org/
-----
#### [cytomic_orion](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/cytomic_orion.py)
<img src=logos/cytomic_orion.png height=60>
@ -369,7 +389,7 @@ A simple DNS expansion service to resolve IP address from domain MISP attributes
-----
#### [docx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/docx-enrich.py)
#### [docx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/docx_enrich.py)
<img src=logos/docx.png height=60>
@ -476,6 +496,42 @@ Module to access Farsight DNSDB Passive DNS.
-----
#### [geoip_asn](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_asn.py)
<img src=logos/maxmind.png height=60>
- **descrption**:
>An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about its related AS number.
- **features**:
>The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the related AS number.
- **input**:
>An IP address MISP attribute.
- **output**:
>Text containing information about the AS number of the IP address.
- **references**:
>https://www.maxmind.com/en/home
- **requirements**:
>A local copy of Maxmind's Geolite database
-----
#### [geoip_city](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_city.py)
<img src=logos/maxmind.png height=60>
An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located.
- **features**:
>The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the city where this IP address is located.
- **input**:
>An IP address MISP attribute.
- **output**:
>Text containing information about the city where the IP address is located.
- **references**:
>https://www.maxmind.com/en/home
- **requirements**:
>A local copy of Maxmind's Geolite database
-----
#### [geoip_country](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_country.py)
<img src=logos/maxmind.png height=60>
@ -496,19 +552,39 @@ Module to query a local copy of Maxmind's Geolite database.
-----
#### [google_search](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/google_search.py)
<img src=logos/google.png height=60>
- **descrption**:
>A hover module to get information about an url using a Google search.
- **features**:
>The module takes an url as input to query the Google search API. The result of the query is then return as raw text.
- **input**:
>An url attribute.
- **output**:
>Text containing the result of a Google search on the input url.
- **references**:
>https://github.com/abenassi/Google-Search-API
- **requirements**:
>The python Google Search API library
-----
#### [greynoise](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/greynoise.py)
<img src=logos/greynoise.png height=60>
Module to access GreyNoise.io API
- **features**:
>The module takes an IP address as input and queries Greynoise for some additional information about it. The result is returned as text.
>The module takes an IP address as input and queries Greynoise for some additional information about it: basically it checks whether a given IP address is “Internet background noise”, or has been observed scanning or attacking devices across the Internet. The result is returned as text.
- **input**:
>An IP address.
- **output**:
>Additional information about the IP fetched from Greynoise API.
- **references**:
>https://greynoise.io/, https://github.com/GreyNoise-Intelligence/api.greynoise.io
- **requirements**:
>A Greynoise API key.
-----
@ -542,6 +618,37 @@ Module to access haveibeenpwned.com API.
-----
#### [intel471](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/intel471.py)
<img src=logos/intel471.png height=60>
- **descrption**:
>An expansion module to query Intel471 in order to get additional information about a domain, ip address, email address, url or hash.
- **features**:
>The module uses the Intel471 python library to query the Intel471 API with the value of the input attribute. The result of the query is then returned as freetext so the Freetext import parses it.
- **input**:
>A MISP attribute whose type is included in the following list:
>- hostname
>- domain
>- url
>- ip-src
>- ip-dst
>- email-src
>- email-dst
>- target-email
>- whois-registrant-email
>- whois-registrant-name
>- md5
>- sha1
>- sha256
- **output**:
>Freetext
- **references**:
>https://public.intel471.com/
- **requirements**:
>The intel471 python library
-----
#### [intelmq_eventdb](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/intelmq_eventdb.py)
<img src=logos/intelmq.png height=60>
@ -731,7 +838,7 @@ Query the MALWAREbazaar API to get additional information about the input hash a
-----
#### [ocr-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ocr-enrich.py)
#### [ocr_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ocr_enrich.py)
Module to process some optical character recognition on pictures.
- **features**:
@ -745,7 +852,7 @@ Module to process some optical character recognition on pictures.
-----
#### [ods-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ods-enrich.py)
#### [ods_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ods_enrich.py)
<img src=logos/ods.png height=60>
@ -761,7 +868,7 @@ Module to extract freetext from a .ods document.
-----
#### [odt-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/odt-enrich.py)
#### [odt_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/odt_enrich.py)
<img src=logos/odt.png height=60>
@ -900,7 +1007,7 @@ Module to get information from AlienVault OTX.
-----
#### [pdf-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pdf-enrich.py)
#### [pdf_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pdf_enrich.py)
<img src=logos/pdf.jpg height=60>
@ -916,7 +1023,7 @@ Module to extract freetext from a PDF document.
-----
#### [pptx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pptx-enrich.py)
#### [pptx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pptx_enrich.py)
<img src=logos/pptx.png height=60>
@ -946,6 +1053,24 @@ Module to decode QR codes.
-----
#### [ransomcoindb](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ransomcoindb.py)
- **descrption**:
>Module to access the ransomcoinDB with a hash or btc address attribute and get the associated btc address of hashes.
- **features**:
>The module takes either a hash attribute or a btc attribute as input to query the ransomcoinDB API for some additional data.
>
>If the input is a btc address, we will get the associated hashes returned in a file MISP object. If we query ransomcoinDB with a hash, the response contains the associated btc addresses returned as single MISP btc attributes.
- **input**:
>A hash (md5, sha1 or sha256) or btc attribute.
- **output**:
>Hashes associated to a btc address or btc addresses associated to a hash.
- **references**:
>https://ransomcoindb.concinnity-risks.com
- **requirements**:
>A ransomcoinDB API key.
-----
#### [rbl](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/rbl.py)
Module to check an IPv4 address against known RBLs.
@ -964,6 +1089,24 @@ Module to check an IPv4 address against known RBLs.
-----
#### [recordedfuture](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/recordedfuture.py)
<img src=logos/recordedfuture.png height=60>
Module to enrich attributes with threat intelligence from Recorded Future.
- **features**:
>Enrich an attribute to add a custom enrichment object to the event. The object contains a copy of the enriched attribute with added tags presenting risk score and triggered risk rules from Recorded Future. Malware and Threat Actors related to the enriched indicator in Recorded Future is matched against MISP's galaxy clusters and applied as galaxy tags. The custom enrichment object also includes a list of related indicators from Recorded Future (IP's, domains, hashes, URL's and vulnerabilities) added as additional attributes.
- **input**:
>A MISP attribute of one of the following types: ip, ip-src, ip-dst, domain, hostname, md5, sha1, sha256, uri, url, vulnerability, weakness.
- **output**:
>A MISP object containing a copy of the enriched attribute with added tags from Recorded Future and a list of new attributes related to the enriched attribute.
- **references**:
>https://www.recordedfuture.com/
- **requirements**:
>A Recorded Future API token.
-----
#### [reversedns](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/reversedns.py)
Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes.
@ -1071,6 +1214,24 @@ An expansion hover module to perform a syntax check on sigma rules.
-----
#### [sophoslabs_intelix](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/sophoslabs_intelix.py)
<img src=logos/sophoslabs_intelix.svg height=60>
An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute.
- **features**:
>The module takes an ip address, url, domain or sha256 attribute and queries the SophosLabs Intelix API with the attribute value. The result of this query is a SophosLabs Intelix hash report, or an ip or url lookup, that is then parsed and returned in a MISP object.
- **input**:
>An ip address, url, domain or sha256 attribute.
- **output**:
>SophosLabs Intelix report and lookup objects
- **references**:
>https://aws.amazon.com/marketplace/pp/B07SLZPMCS
- **requirements**:
>A client_id and client_secret pair to authenticate to the SophosLabs Intelix API
-----
#### [sourcecache](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/sourcecache.py)
Module to cache web pages of analysis reports, OSINT sources. The module returns a link of the cached page.
@ -1422,7 +1583,7 @@ An expansion module for IBM X-Force Exchange.
-----
#### [xlsx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/xlsx-enrich.py)
#### [xlsx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/xlsx_enrich.py)
<img src=logos/xlsx.png height=60>

4
doc/expansion/bgpranking.json
View File

@ -1,8 +1,8 @@
{
"description": "Query BGP Ranking (https://bgpranking-ng.circl.lu/).",
"requirements": ["pybgpranking python library"],
"features": "The module takes an AS number attribute as input and displays its description and history, and position in BGP Ranking.\n\n",
"features": "The module takes an AS number attribute as input and displays its description as well as its ranking position in BGP Ranking for a given day.\n\n",
"references": ["https://github.com/D4-project/BGP-Ranking/"],
"input": "Autonomous system number.",
"output": "Text containing a description of the ASN, its history, and the position in BGP Ranking."
"output": "An asn object with its related bgp-ranking object."
}

8
doc/expansion/cve_advanced.json Normal file
View File

@ -0,0 +1,8 @@
{
"description": "An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).",
"logo": "logos/cve.png",
"input": "Vulnerability attribute.",
"output": "Additional information about the vulnerability, such as its cvss score, some references, or the related weaknesses and attack patterns.",
"references": ["https://cve.circl.lu", "https://cve/mitre.org/"],
"features": "The module takes a vulnerability attribute as input and queries the CIRCL CVE search API to gather additional information.\n\nThe result of the query is then parsed to return additional information about the vulnerability, like its cvss score or some references, as well as the potential related weaknesses and attack patterns.\n\nThe vulnerability additional data is returned in a vulnerability MISP object, and the related additional information are put into weakness and attack-pattern MISP objects."
}

0
doc/expansion/docx-enrich.json → doc/expansion/docx_enrich.json
View File

9
doc/expansion/geoip_asn.json Normal file
View File

@ -0,0 +1,9 @@
{
"descrption": "An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about its related AS number.",
"logo": "logos/maxmind.png",
"requirements": ["A local copy of Maxmind's Geolite database"],
"input": "An IP address MISP attribute.",
"output": "Text containing information about the AS number of the IP address.",
"references": ["https://www.maxmind.com/en/home"],
"features": "The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the related AS number."
}

9
doc/expansion/geoip_city.json Normal file
View File

@ -0,0 +1,9 @@
{
"description": "An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located.",
"logo": "logos/maxmind.png",
"requirements": ["A local copy of Maxmind's Geolite database"],
"input": "An IP address MISP attribute.",
"output": "Text containing information about the city where the IP address is located.",
"references": ["https://www.maxmind.com/en/home"],
"features": "The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the city where this IP address is located."
}

9
doc/expansion/google_search.json Normal file
View File

@ -0,0 +1,9 @@
{
"descrption": "A hover module to get information about an url using a Google search.",
"logo": "logos/google.png",
"requirements": ["The python Google Search API library"],
"input": "An url attribute.",
"output": "Text containing the result of a Google search on the input url.",
"references": ["https://github.com/abenassi/Google-Search-API"],
"features": "The module takes an url as input to query the Google search API. The result of the query is then return as raw text."
}

4
doc/expansion/greynoise.json
View File

@ -1,9 +1,9 @@
{
"description": "Module to access GreyNoise.io API",
"logo": "logos/greynoise.png",
"requirements": [],
"requirements": ["A Greynoise API key."],
"input": "An IP address.",
"output": "Additional information about the IP fetched from Greynoise API.",
"references": ["https://greynoise.io/", "https://github.com/GreyNoise-Intelligence/api.greynoise.io"],
"features": "The module takes an IP address as input and queries Greynoise for some additional information about it. The result is returned as text."
"features": "The module takes an IP address as input and queries Greynoise for some additional information about it: basically it checks whether a given IP address is “Internet background noise”, or has been observed scanning or attacking devices across the Internet. The result is returned as text."
}

9
doc/expansion/intel471.json Normal file
View File

@ -0,0 +1,9 @@
{
"descrption": "An expansion module to query Intel471 in order to get additional information about a domain, ip address, email address, url or hash.",
"logo": "logos/intel471.png",
"requirements": ["The intel471 python library"],
"input": "A MISP attribute whose type is included in the following list:\n- hostname\n- domain\n- url\n- ip-src\n- ip-dst\n- email-src\n- email-dst\n- target-email\n- whois-registrant-email\n- whois-registrant-name\n- md5\n- sha1\n- sha256",
"output": "Freetext",
"references": ["https://public.intel471.com/"],
"features": "The module uses the Intel471 python library to query the Intel471 API with the value of the input attribute. The result of the query is then returned as freetext so the Freetext import parses it."
}

0
doc/expansion/ocr-enrich.json → doc/expansion/ocr_enrich.json
View File

0
doc/expansion/ods-enrich.json → doc/expansion/ods_enrich.json
View File

0
doc/expansion/odt-enrich.json → doc/expansion/odt_enrich.json
View File

0
doc/expansion/pdf-enrich.json → doc/expansion/pdf_enrich.json
View File

0
doc/expansion/pptx-enrich.json → doc/expansion/pptx_enrich.json
View File

8
doc/expansion/ransomcoindb.json Normal file
View File

@ -0,0 +1,8 @@
{
"descrption": "Module to access the ransomcoinDB with a hash or btc address attribute and get the associated btc address of hashes.",
"requirements": ["A ransomcoinDB API key."],
"input": "A hash (md5, sha1 or sha256) or btc attribute.",
"output": "Hashes associated to a btc address or btc addresses associated to a hash.",
"references": ["https://ransomcoindb.concinnity-risks.com"],
"features": "The module takes either a hash attribute or a btc attribute as input to query the ransomcoinDB API for some additional data.\n\nIf the input is a btc address, we will get the associated hashes returned in a file MISP object. If we query ransomcoinDB with a hash, the response contains the associated btc addresses returned as single MISP btc attributes."
}

9
doc/expansion/recordedfuture.json Normal file
View File

@ -0,0 +1,9 @@
{
"description": "Module to enrich attributes with threat intelligence from Recorded Future.",
"logo": "logos/recordedfuture.png",
"requirements": ["A Recorded Future API token."],
"input": "A MISP attribute of one of the following types: ip, ip-src, ip-dst, domain, hostname, md5, sha1, sha256, uri, url, vulnerability, weakness.",
"output": "A MISP object containing a copy of the enriched attribute with added tags from Recorded Future and a list of new attributes related to the enriched attribute.",
"references": ["https://www.recordedfuture.com/"],
"features": "Enrich an attribute to add a custom enrichment object to the event. The object contains a copy of the enriched attribute with added tags presenting risk score and triggered risk rules from Recorded Future. Malware and Threat Actors related to the enriched indicator in Recorded Future is matched against MISP's galaxy clusters and applied as galaxy tags. The custom enrichment object also includes a list of related indicators from Recorded Future (IP's, domains, hashes, URL's and vulnerabilities) added as additional attributes."
}

9
doc/expansion/sophoslabs_intelix.json Normal file
View File

@ -0,0 +1,9 @@
{
"description": "An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute.",
"logo": "logos/sophoslabs_intelix.svg",
"requirements": ["A client_id and client_secret pair to authenticate to the SophosLabs Intelix API"],
"input": "An ip address, url, domain or sha256 attribute.",
"output": "SophosLabs Intelix report and lookup objects",
"references": ["https://aws.amazon.com/marketplace/pp/B07SLZPMCS"],
"features": "The module takes an ip address, url, domain or sha256 attribute and queries the SophosLabs Intelix API with the attribute value. The result of this query is a SophosLabs Intelix hash report, or an ip or url lookup, that is then parsed and returned in a MISP object."
}

0
doc/expansion/xlsx-enrich.json → doc/expansion/xlsx_enrich.json
View File

BIN
doc/logos/google.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

BIN
doc/logos/intel471.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.6 KiB

BIN
doc/logos/recordedfuture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

32
doc/logos/sophoslabs_intelix.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

12
misp_modules/modules/expansion/__init__.py
View File

@ -18,4 +18,14 @@ __all__ = ['cuckoo_submit', 'vmray_submit', 'bgpranking', 'circl_passivedns', 'c
'virustotal_public', 'apiosintds', 'urlscan', 'securitytrails', 'apivoid',
'assemblyline_submit', 'assemblyline_query', 'ransomcoindb', 'malwarebazaar',
'lastline_query', 'lastline_submit', 'sophoslabs_intelix', 'cytomic_orion', 'censys_enrich',
'trustar_enrich']
'trustar_enrich', 'recordedfuture']
minimum_required_fields = ('type', 'uuid', 'value')
checking_error = 'containing at least a "type" field and a "value" field'
standard_error_message = 'This module requires an "attribute" field as input'
def check_input_attribute(attribute, requirements=minimum_required_fields):
return all(feature in attribute for feature in requirements)

7
misp_modules/modules/expansion/apivoid.py
View File

@ -1,5 +1,6 @@
import json
import requests
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
misperrors = {'error': 'Error'}
@ -74,7 +75,11 @@ def handler(q=False):
request = json.loads(q)
if not request.get('config', {}).get('apikey'):
return {'error': 'An API key for APIVoid is required.'}
attribute = request.get('attribute')
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if attribute['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
apikey = request['config']['apikey']
apivoid_parser = APIVoidParser(attribute)
apivoid_parser.parse_domain(apikey)

5
misp_modules/modules/expansion/assemblyline_query.py
View File

@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
import json
from . import check_input_attribute, standard_error_message
from assemblyline_client import Client, ClientError
from collections import defaultdict
from pymisp import MISPAttribute, MISPEvent, MISPObject
@ -139,6 +140,10 @@ def handler(q=False):
if q is False:
return False
request = json.loads(q)
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
if request['attribute']['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
if not request.get('config'):
return {"error": "Missing configuration."}
if not request['config'].get('apiurl'):

72
misp_modules/modules/expansion/bgpranking.py
View File

@ -1,13 +1,15 @@
# -*- coding: utf-8 -*-
import json
from datetime import date, timedelta
from . import check_input_attribute, standard_error_message
from datetime import date, datetime, timedelta
from pybgpranking import BGPRanking
from pymisp import MISPAttribute, MISPEvent, MISPObject
misperrors = {'error': 'Error'}
mispattributes = {'input': ['AS'], 'output': ['freetext']}
mispattributes = {'input': ['AS'], 'format': 'misp_standard'}
moduleinfo = {'version': '0.1', 'author': 'Raphaël Vinot',
'description': 'Query an ASN Description history service (https://github.com/CIRCL/ASN-Description-History.git)',
'description': 'Query BGP Ranking to get the ranking of an Autonomous System number.',
'module-type': ['expansion', 'hover']}
@ -15,19 +17,65 @@ def handler(q=False):
if q is False:
return False
request = json.loads(q)
if request.get('AS'):
toquery = request['AS']
else:
misperrors['error'] = "Unsupported attributes type"
return misperrors
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
toquery = request['attribute']
if toquery['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
bgpranking = BGPRanking()
values = bgpranking.query(toquery, date=(date.today() - timedelta(1)).isoformat())
value_toquery = int(toquery['value'][2:]) if toquery['value'].startswith('AS') else int(toquery['value'])
values = bgpranking.query(value_toquery, date=(date.today() - timedelta(1)).isoformat())
if not values:
misperrors['error'] = 'Unable to find the ASN in BGP Ranking'
if not values['response'] or not values['response']['asn_description']:
misperrors['error'] = 'There is no result about this ASN in BGP Ranking'
return misperrors
return {'results': [{'types': mispattributes['output'], 'values': values}]}
event = MISPEvent()
attribute = MISPAttribute()
attribute.from_dict(**toquery)
event.add_attribute(**attribute)
asn_object = MISPObject('asn')
asn_object.add_attribute(**{
'type': 'AS',
'object_relation': 'asn',
'value': values['meta']['asn']
})
description, country = values['response']['asn_description'].split(', ')
for relation, value in zip(('description', 'country'), (description, country)):
asn_object.add_attribute(**{
'type': 'text',
'object_relation': relation,
'value': value
})
mapping = {
'address_family': {'type': 'text', 'object_relation': 'address-family'},
'date': {'type': 'datetime', 'object_relation': 'date'},
'position': {'type': 'float', 'object_relation': 'position'},
'rank': {'type': 'float', 'object_relation': 'ranking'}
}
bgp_object = MISPObject('bgp-ranking')
for feature in ('rank', 'position'):
bgp_attribute = {'value': values['response']['ranking'][feature]}
bgp_attribute.update(mapping[feature])
bgp_object.add_attribute(**bgp_attribute)
date_attribute = {'value': datetime.strptime(values['meta']['date'], '%Y-%m-%d')}
date_attribute.update(mapping['date'])
bgp_object.add_attribute(**date_attribute)
address_attribute = {'value': values['meta']['address_family']}
address_attribute.update(mapping['address_family'])
bgp_object.add_attribute(**address_attribute)
asn_object.add_reference(attribute.uuid, 'describes')
asn_object.add_reference(bgp_object.uuid, 'ranked-with')
event.add_object(asn_object)
event.add_object(bgp_object)
event = json.loads(event.to_json())
results = {key: event[key] for key in ('Attribute', 'Object')}
return {'results': results}
def introspection():

7
misp_modules/modules/expansion/censys_enrich.py
View File

@ -3,6 +3,7 @@ import json
import base64
import codecs
from dateutil.parser import isoparse
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
try:
import censys.base
@ -36,11 +37,11 @@ def handler(q=False):
api_id = request['config']['api_id']
api_secret = request['config']['api_secret']
if not request.get('attribute'):
return {'error': 'Unsupported input.'}
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
return {'error': 'Unsupported attributes type'}
return {'error': 'Unsupported attribute type.'}
attribute = MISPAttribute()
attribute.from_dict(**request['attribute'])

7
misp_modules/modules/expansion/circl_passivedns.py
View File

@ -1,5 +1,6 @@
import json
import pypdns
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
mispattributes = {'input': ['hostname', 'domain', 'ip-src', 'ip-dst', 'ip-src|port', 'ip-dst|port'], 'format': 'misp_standard'}
@ -58,11 +59,11 @@ def handler(q=False):
if not request['config'].get('username') or not request['config'].get('password'):
return {'error': 'CIRCL Passive DNS authentication is incomplete, please provide your username and password.'}
authentication = (request['config']['username'], request['config']['password'])
if not request.get('attribute'):
return {'error': 'Unsupported input.'}
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
return {'error': 'Unsupported attributes type'}
return {'error': 'Unsupported attribute type.'}
pdns_parser = PassiveDNSParser(attribute, authentication)
pdns_parser.parse()
return pdns_parser.get_results()

7
misp_modules/modules/expansion/circl_passivessl.py
View File

@ -1,5 +1,6 @@
import json
import pypssl
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
mispattributes = {'input': ['ip-src', 'ip-dst', 'ip-src|port', 'ip-dst|port'], 'format': 'misp_standard'}
@ -83,11 +84,11 @@ def handler(q=False):
if not request['config'].get('username') or not request['config'].get('password'):
return {'error': 'CIRCL Passive SSL authentication is incomplete, please provide your username and password.'}
authentication = (request['config']['username'], request['config']['password'])
if not request.get('attribute'):
return {'error': 'Unsupported input.'}
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
return {'error': 'Unsupported attributes type'}
return {'error': 'Unsupported attribute type.'}
pssl_parser = PassiveSSLParser(attribute, authentication)
pssl_parser.parse()
return pssl_parser.get_results()

15
misp_modules/modules/expansion/cve_advanced.py
View File

@ -1,7 +1,8 @@
from collections import defaultdict
from pymisp import MISPEvent, MISPObject
import json
import requests
from . import check_input_attribute, standard_error_message
from collections import defaultdict
from pymisp import MISPEvent, MISPObject
misperrors = {'error': 'Error'}
mispattributes = {'input': ['vulnerability'], 'format': 'misp_standard'}
@ -55,7 +56,7 @@ class VulnerabilityParser():
value = value['title']
vulnerability_object.add_attribute(relation, **{'type': attribute_type, 'value': value})
vulnerability_object.add_reference(self.attribute['uuid'], 'related-to')
self.misp_event.add_object(**vulnerability_object)
self.misp_event.add_object(vulnerability_object)
if 'cwe' in self.vulnerability and self.vulnerability['cwe'] not in ('Unknown', 'NVD-CWE-noinfo'):
self.__parse_weakness(vulnerability_object.uuid)
if 'capec' in self.vulnerability:
@ -78,7 +79,7 @@ class VulnerabilityParser():
for related_weakness in capec['related_weakness']:
attribute = dict(type='weakness', value="CWE-{}".format(related_weakness))
capec_object.add_attribute('related-weakness', **attribute)
self.misp_event.add_object(**capec_object)
self.misp_event.add_object(capec_object)
self.references[vulnerability_uuid].append(dict(referenced_uuid=capec_object.uuid,
relationship_type='targeted-by'))
@ -94,7 +95,7 @@ class VulnerabilityParser():
for feature, relation in self.weakness_mapping.items():
if cwe.get(feature):
weakness_object.add_attribute(relation, **dict(type=attribute_type, value=cwe[feature]))
self.misp_event.add_object(**weakness_object)
self.misp_event.add_object(weakness_object)
self.references[vulnerability_uuid].append(dict(referenced_uuid=weakness_object.uuid,
relationship_type='weakened-by'))
break
@ -108,7 +109,9 @@ def handler(q=False):
if q is False:
return False
request = json.loads(q)
attribute = request.get('attribute')
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if attribute.get('type') != 'vulnerability':
misperrors['error'] = 'Vulnerability id missing.'
return misperrors

5
misp_modules/modules/expansion/cytomic_orion.py
View File

@ -7,6 +7,7 @@ An expansion module to enrich attributes in MISP and share indicators of comprom
'''
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
import json
import requests
@ -146,9 +147,11 @@ def handler(q=False):
if not request.get('attribute'):
return {'error': 'Unsupported input.'}
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
return {'error': 'Unsupported attributes type'}
return {'error': 'Unsupported attribute type.'}
if not request.get('config'):
return {'error': 'Missing configuration'}

56
misp_modules/modules/expansion/greynoise.py
View File

@ -3,35 +3,59 @@ import json
misperrors = {'error': 'Error'}
mispattributes = {'input': ['ip-dst', 'ip-src'], 'output': ['text']}
moduleinfo = {'version': '0.1', 'author': 'Aurélien Schwab <aurelien.schwab+dev@gmail.com>', 'description': 'Module to access GreyNoise.io API.', 'module-type': ['hover']}
moduleconfig = ['user-agent'] # TODO take this into account in the code
moduleinfo = {
'version': '0.2',
'author': 'Aurélien Schwab <aurelien.schwab+dev@gmail.com>',
'description': 'Module to access GreyNoise.io API.',
'module-type': ['hover']
}
moduleconfig = ['api_key']
greynoise_api_url = 'http://api.greynoise.io:8888/v1/query/ip'
default_user_agent = 'MISP-Module'
greynoise_api_url = 'https://api.greynoise.io/v2/noise/quick/'
codes_mapping = {
'0x00': 'The IP has never been observed scanning the Internet',
'0x01': 'The IP has been observed by the GreyNoise sensor network',
'0x02': 'The IP has been observed scanning the GreyNoise sensor network, but has not completed a full connection, meaning this can be spoofed',
'0x03': 'The IP is adjacent to another host that has been directly observed by the GreyNoise sensor network',
'0x04': 'Reserved',
'0x05': 'This IP is commonly spoofed in Internet-scan activity',
'0x06': 'This IP has been observed as noise, but this host belongs to a cloud provider where IPs can be cycled frequently',
'0x07': 'This IP is invalid',
'0x08': 'This IP was classified as noise, but has not been observed engaging in Internet-wide scans or attacks in over 60 days'
}
def handler(q=False):
if q is False:
return False
request = json.loads(q)
if not request.get('config') or not request['config'].get('api_key'):
return {'error': 'Missing Greynoise API key.'}
headers = {
'Accept': 'application/json',
'key': request['config']['api_key']
}
for input_type in mispattributes['input']:
if input_type in request:
ip = request[input_type]
break
else:
misperrors['error'] = "Unsupported attributes type"
misperrors['error'] = "Unsupported attributes type."
return misperrors
data = {'ip': ip}
r = requests.post(greynoise_api_url, data=data, headers={'user-agent': default_user_agent}) # Real request
if r.status_code == 200: # OK (record found)
response = r.text
if response:
return {'results': [{'types': mispattributes['output'], 'values': response}]}
elif r.status_code == 404: # Not found (not an error)
return {'results': [{'types': mispattributes['output'], 'values': 'No data'}]}
else: # Real error
misperrors['error'] = 'GreyNoise API not accessible (HTTP ' + str(r.status_code) + ')'
return misperrors['error']
response = requests.get(f'{greynoise_api_url}{ip}', headers=headers) # Real request
if response.status_code == 200: # OK (record found)
return {'results': [{'types': mispattributes['output'], 'values': codes_mapping[response.json()['code']]}]}
# There is an error
errors = {
400: "Bad request.",
401: "Unauthorized. Please check your API key.",
429: "Too many requests. You've hit the rate-limit."
}
try:
misperrors['error'] = errors[response.status_code]
except KeyError:
misperrors['error'] = f'GreyNoise API not accessible (HTTP {response.status_code})'
return misperrors['error']
def introspection():

11
misp_modules/modules/expansion/ipasn.py
View File

@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
import json
from . import check_input_attribute, standard_error_message
from pyipasnhistory import IPASNHistory
from pymisp import MISPAttribute, MISPEvent, MISPObject
@ -34,11 +35,11 @@ def handler(q=False):
if q is False:
return False
request = json.loads(q)
if request.get('attribute') and request['attribute'].get('type') in mispattributes['input']:
toquery = request['attribute']['value']
else:
misperrors['error'] = "Unsupported attributes type"
return misperrors
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
if request['attribute']['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
toquery = request['attribute']['value']
ipasn = IPASNHistory()
values = ipasn.query(toquery)

5
misp_modules/modules/expansion/joesandbox_query.py
View File

@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
import jbxapi
import json
from . import check_input_attribute, checking_error, standard_error_message
from joe_parser import JoeParser
misperrors = {'error': 'Error'}
@ -27,6 +28,10 @@ def handler(q=False):
if not apikey:
return {'error': 'No API key provided'}
if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
return {'error': f'{standard_error_message}, {checking_error} that is the link to the Joe Sandbox report.'}
if request['attribute']['type'] != 'link':
return {'error': 'Unsupported attribute type.'}
url = request['attribute']['value']
if "/submissions/" not in url:
return {'error': "The URL does not point to a Joe Sandbox analysis."}

4
misp_modules/modules/expansion/lastline_query.py
View File

@ -3,8 +3,8 @@
Module (type "expansion") to query a Lastline report from an analysis link.
"""
import json
import lastline_api
from . import check_input_attribute, checking_error, standard_error_message
misperrors = {
@ -52,6 +52,8 @@ def handler(q=False):
try:
config = request["config"]
auth_data = lastline_api.LastlineAbstractClient.get_login_params_from_dict(config)
if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
return {'error': f'{standard_error_message}, {checking_error} that is the link to a Lastline analysis.'}
analysis_link = request['attribute']['value']
# The API url changes based on the analysis link host name
api_url = lastline_api.get_portal_url_from_task_link(analysis_link)

5
misp_modules/modules/expansion/malwarebazaar.py
View File

@ -1,5 +1,6 @@
import json
import requests
from . import check_input_attribute, checking_error, standard_error_message
from pymisp import MISPEvent, MISPObject
mispattributes = {'input': ['md5', 'sha1', 'sha256'],
@ -34,7 +35,11 @@ def handler(q=False):
if q is False:
return False
request = json.loads(q)
if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
return {'error': f'{standard_error_message}, {checking_error} that is the hash to submit to Malware Bazaar.'}
attribute = request['attribute']
if attribute['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
url = 'https://mb-api.abuse.ch/api/v1/'
response = requests.post(url, data={'query': 'get_info', 'hash': attribute['value']}).json()
query_status = response['query_status']

5
misp_modules/modules/expansion/ransomcoindb.py
View File

@ -1,4 +1,5 @@
import json
from . import check_input_attribute, checking_error, standard_error_message
from ._ransomcoindb import ransomcoindb
from pymisp import MISPObject
@ -28,6 +29,10 @@ def handler(q=False):
q = json.loads(q)
if "config" not in q or "api-key" not in q["config"]:
return {"error": "Ransomcoindb API key is missing"}
if not q.get('attribute') or not check_input_attribute(q['attribute'], requirements=('type', 'value')):
return {'error': f'{standard_error_message}, {checking_error}.'}
if q['attribute']['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
api_key = q["config"]["api-key"]
r = {"results": []}

396
misp_modules/modules/expansion/recordedfuture.py Normal file
View File

@ -0,0 +1,396 @@
import json
import logging
import requests
from requests.exceptions import HTTPError, ProxyError,\
InvalidURL, ConnectTimeout, ConnectionError
from . import check_input_attribute, checking_error, standard_error_message
import platform
import os
from urllib.parse import quote, urlparse
from pymisp import MISPAttribute, MISPEvent, MISPTag, MISPObject
moduleinfo = {
'version': '1.0.1',
'author': 'Recorded Future',
'description': 'Module to retrieve data from Recorded Future',
'module-type': ['expansion', 'hover']
}
moduleconfig = ['token', 'proxy_host', 'proxy_port', 'proxy_username', 'proxy_password']
misperrors = {'error': 'Error'}
ATTRIBUTES = [
'ip',
'ip-src',
'ip-dst',
'domain',
'hostname',
'md5',
'sha1',
'sha256',
'uri',
'url',
'vulnerability',
'weakness'
]
mispattributes = {
'input': ATTRIBUTES,
'output': ATTRIBUTES + ['email-src', 'text'],
'format': 'misp_standard'
}
LOGGER = logging.getLogger('recorded_future')
LOGGER.setLevel(logging.INFO)
class RequestHandler:
"""A class for handling any outbound requests from this module."""
def __init__(self):
self.session = requests.Session()
self.app_id = f'{os.path.basename(__file__)}/{moduleinfo["version"]} ({platform.platform()}) ' \
f'misp_enrichment/{moduleinfo["version"]} python-requests/{requests.__version__}'
self.proxies = None
self.rf_token = None
def get(self, url: str, headers: dict = None) -> requests.Response:
"""General get method with proxy error handling."""
try:
timeout = 7 if self.proxies else None
response = self.session.get(url, headers=headers, proxies=self.proxies, timeout=timeout)
response.raise_for_status()
return response
except (ConnectTimeout, ProxyError, InvalidURL) as error:
msg = 'Error connecting with proxy, please check the Recorded Future app proxy settings.'
LOGGER.error(f'{msg} Error: {error}')
misperrors['error'] = msg
raise
def rf_lookup(self, category: str, ioc: str) -> requests.Response:
"""Do a lookup call using Recorded Future's ConnectAPI."""
parsed_ioc = quote(ioc, safe='')
url = f'https://api.recordedfuture.com/v2/{category}/{parsed_ioc}?fields=risk%2CrelatedEntities'
headers = {'X-RFToken': self.rf_token,
'User-Agent': self.app_id}
try:
response = self.get(url, headers)
except HTTPError as error:
msg = f'Error when requesting data from Recorded Future. {error.response}: {error.response.reason}'
LOGGER.error(msg)
misperrors['error'] = msg
raise
return response
GLOBAL_REQUEST_HANDLER = RequestHandler()
class GalaxyFinder:
"""A class for finding MISP galaxy matches to Recorded Future data."""
def __init__(self):
self.session = requests.Session()
self.sources = {
'RelatedThreatActor': [
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/threat-actor.json'
],
'RelatedMalware': [
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/banker.json',
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/botnet.json',
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/exploit-kit.json',
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/rat.json',
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/ransomware.json',
'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/malpedia.json'
]
}
self.galaxy_clusters = {}
def pull_galaxy_cluster(self, related_type: str) -> None:
"""Fetches galaxy clusters for the related_type from the remote json files specified as self.sources."""
# Only fetch clusters if not fetched previously
if not self.galaxy_clusters.get(related_type):
for source in self.sources.get(related_type):
try:
response = GLOBAL_REQUEST_HANDLER.get(source)
name = source.split('/')[-1].split('.')[0]
self.galaxy_clusters[related_type] = {name: response.json()}
except ConnectionError as error:
LOGGER.warning(f'pull_galaxy_cluster failed for source: {source}, with error: {error}.')
def find_galaxy_match(self, indicator: str, related_type: str) -> str:
"""Searches the clusters of the related_type for a match with the indicator.
:returns the first matching galaxy string or an empty string if no galaxy match is found.
"""
self.pull_galaxy_cluster(related_type)
for cluster_name, cluster in self.galaxy_clusters.get(related_type, {}).items():
for value in cluster['values']:
try:
if indicator in value['meta']['synonyms'] or indicator in value['value']:
value = value['value']
return f'misp-galaxy:{cluster_name}="{value}"'
except KeyError:
pass
return ''
class RFColors:
"""Class for setting signature RF-colors."""
def __init__(self):
self.rf_white = '#CCCCCC'
self.rf_yellow = '#FFCE00'
self.rf_red = '#CF0A2C'
def riskscore_color(self, risk_score: int) -> str:
"""Returns appropriate hex-colors according to risk score."""
risk_score = int(risk_score)
if risk_score < 25:
return self.rf_white
elif risk_score < 65:
return self.rf_yellow
else:
return self.rf_red
def riskrule_color(self, risk_rule_criticality: int) -> str:
"""Returns appropriate hex-colors according to risk rule criticality."""
risk_rule_criticality = int(risk_rule_criticality)
if risk_rule_criticality == 1:
return self.rf_white
elif risk_rule_criticality == 2:
return self.rf_yellow
else: # risk_rule_criticality == 3 or 4
return self.rf_red
class RFEnricher:
"""Class for enriching an attribute with data from Recorded Future.
The enrichment data is returned as a custom MISP object.
"""
def __init__(self, attribute_props: dict):
self.event = MISPEvent()
self.enrichment_object = MISPObject('Recorded Future Enrichment')
description = (
'An object containing the enriched attribute and '
'related entities from Recorded Future.'
)
self.enrichment_object.from_dict(**{
'meta-category': 'misc',
'description': description,
'distribution': 0
})
# Create a copy of enriched attribute to add tags to
temp_attr = MISPAttribute()
temp_attr.from_dict(**attribute_props)
self.enriched_attribute = MISPAttribute()
self.enriched_attribute.from_dict(**{
'value': temp_attr.value,
'type': temp_attr.type,
'distribution': 0
})
self.related_attributes = []
self.color_picker = RFColors()
self.galaxy_finder = GalaxyFinder()
# Mapping from MISP-type to RF-type
self.type_to_rf_category = {
'ip': 'ip',
'ip-src': 'ip',
'ip-dst': 'ip',
'domain': 'domain',
'hostname': 'domain',
'md5': 'hash',
'sha1': 'hash',
'sha256': 'hash',
'uri': 'url',
'url': 'url',
'vulnerability': 'vulnerability',
'weakness': 'vulnerability'
}
# Related entities from RF portrayed as related attributes in MISP
self.related_attribute_types = [
'RelatedIpAddress', 'RelatedInternetDomainName', 'RelatedHash',
'RelatedEmailAddress', 'RelatedCyberVulnerability'
]
# Related entities from RF portrayed as tags in MISP
self.galaxy_tag_types = ['RelatedMalware', 'RelatedThreatActor']
def enrich(self) -> None:
"""Run the enrichment."""
category = self.type_to_rf_category.get(self.enriched_attribute.type)
json_response = GLOBAL_REQUEST_HANDLER.rf_lookup(category, self.enriched_attribute.value)
response = json.loads(json_response.content)
try:
# Add risk score and risk rules as tags to the enriched attribute
risk_score = response['data']['risk']['score']
hex_color = self.color_picker.riskscore_color(risk_score)
tag_name = f'recorded-future:risk-score="{risk_score}"'
self.add_tag(tag_name, hex_color)
for evidence in response['data']['risk']['evidenceDetails']:
risk_rule = evidence['rule']
criticality = evidence['criticality']
hex_color = self.color_picker.riskrule_color(criticality)
tag_name = f'recorded-future:risk-rule="{risk_rule}"'
self.add_tag(tag_name, hex_color)
# Retrieve related entities
for related_entity in response['data']['relatedEntities']:
related_type = related_entity['type']
if related_type in self.related_attribute_types:
# Related entities returned as additional attributes
for related in related_entity['entities']:
if int(related["count"]) > 4:
indicator = related['entity']['name']
self.add_related_attribute(indicator, related_type)
elif related_type in self.galaxy_tag_types:
# Related entities added as galaxy-tags to the enriched attribute
galaxy_tags = []
for related in related_entity['entities']:
if int(related["count"]) > 4:
indicator = related['entity']['name']
galaxy = self.galaxy_finder.find_galaxy_match(indicator, related_type)
# Handle deduplication of galaxy tags
if galaxy and galaxy not in galaxy_tags:
galaxy_tags.append(galaxy)
for galaxy in galaxy_tags:
self.add_tag(galaxy)
except KeyError:
misperrors['error'] = 'Unexpected format in Recorded Future api response.'
raise
def add_related_attribute(self, indicator: str, related_type: str) -> None:
"""Helper method for adding an indicator to the related attribute list."""
out_type = self.get_output_type(related_type, indicator)
attribute = MISPAttribute()
attribute.from_dict(**{'value': indicator, 'type': out_type, 'distribution': 0})
self.related_attributes.append((related_type, attribute))
def add_tag(self, tag_name: str, hex_color: str = None) -> None:
"""Helper method for adding a tag to the enriched attribute."""
tag = MISPTag()
tag_properties = {'name': tag_name}
if hex_color:
tag_properties['colour'] = hex_color
tag.from_dict(**tag_properties)
self.enriched_attribute.add_tag(tag)
def get_output_type(self, related_type: str, indicator: str) -> str:
"""Helper method for translating a Recorded Future related type to a MISP output type."""
output_type = 'text'
if related_type == 'RelatedIpAddress':
output_type = 'ip-dst'
elif related_type == 'RelatedInternetDomainName':
output_type = 'domain'
elif related_type == 'RelatedHash':
hash_len = len(indicator)
if hash_len == 64:
output_type = 'sha256'
elif hash_len == 40:
output_type = 'sha1'
elif hash_len == 32:
output_type = 'md5'
elif related_type == 'RelatedEmailAddress':
output_type = 'email-src'
elif related_type == 'RelatedCyberVulnerability':
signature = indicator.split('-')[0]
if signature == 'CVE':
output_type = 'vulnerability'
elif signature == 'CWE':
output_type = 'weakness'
return output_type
def get_results(self) -> dict:
"""Build and return the enrichment results."""
self.enrichment_object.add_attribute('Enriched attribute', **self.enriched_attribute)
for related_type, attribute in self.related_attributes:
self.enrichment_object.add_attribute(related_type, **attribute)
self.event.add_object(**self.enrichment_object)
event = json.loads(self.event.to_json())
result = {key: event[key] for key in ['Object'] if key in event}
return {'results': result}
def get_proxy_settings(config: dict) -> dict:
"""Returns proxy settings in the requests format.
If no proxy settings are set, return None."""
proxies = None
host = config.get('proxy_host')
port = config.get('proxy_port')
username = config.get('proxy_username')
password = config.get('proxy_password')
if host:
if not port:
misperrors['error'] = 'The recordedfuture_proxy_host config is set, ' \
'please also set the recordedfuture_proxy_port.'
raise KeyError
parsed = urlparse(host)
if 'http' in parsed.scheme:
scheme = 'http'
else:
scheme = parsed.scheme
netloc = parsed.netloc
host = f'{netloc}:{port}'
if username:
if not password:
misperrors['error'] = 'The recordedfuture_proxy_username config is set, ' \
'please also set the recordedfuture_proxy_password.'
raise KeyError
auth = f'{username}:{password}'
host = auth + '@' + host
proxies = {
'http': f'{scheme}://{host}',
'https': f'{scheme}://{host}'
}
LOGGER.info(f'Proxy settings: {proxies}')
return proxies
def handler(q=False):
"""Handle enrichment."""
if q is False:
return False
request = json.loads(q)
config = request.get('config')
if config and config.get('token'):
GLOBAL_REQUEST_HANDLER.rf_token = config.get('token')
else:
misperrors['error'] = 'Missing Recorded Future token.'
return misperrors
if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
return {'error': f'{standard_error_message}, {checking_error}.'}
if request['attribute']['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
try:
GLOBAL_REQUEST_HANDLER.proxies = get_proxy_settings(config)
except KeyError:
return misperrors
input_attribute = request.get('attribute')
rf_enricher = RFEnricher(input_attribute)
try:
rf_enricher.enrich()
except (HTTPError, ConnectTimeout, ProxyError, InvalidURL, KeyError):
return misperrors
return rf_enricher.get_results()
def introspection():
"""Returns a dict of the supported attributes."""
return mispattributes
def version():
"""Returns a dict with the version and the associated meta-data
including potential configurations required of the module."""
moduleinfo['config'] = moduleconfig
return moduleinfo

227
misp_modules/modules/expansion/shodan.py
View File

@ -5,38 +5,225 @@ try:
import shodan
except ImportError:
print("shodan module not installed.")
from . import check_input_attribute, standard_error_message
from datetime import datetime
from pymisp import MISPAttribute, MISPEvent, MISPObject
misperrors = {'error': 'Error'}
mispattributes = {'input': ['ip-src', 'ip-dst'], 'output': ['freetext']}
moduleinfo = {'version': '0.1', 'author': 'Raphaël Vinot',
mispattributes = {'input': ['ip-src', 'ip-dst'],
'format': 'misp_standard'}
moduleinfo = {'version': '0.2', 'author': 'Raphaël Vinot',
'description': 'Query on Shodan',
'module-type': ['expansion']}
moduleconfig = ['apikey']
class ShodanParser():
def __init__(self, attribute):
self.misp_event = MISPEvent()
self.attribute = MISPAttribute()
self.attribute.from_dict(**attribute)
self.misp_event.add_attribute(**self.attribute)
self.ip_address_mapping = {
'asn': {'type': 'AS', 'object_relation': 'asn'},
'city': {'type': 'text', 'object_relation': 'city'},
'country_code': {'type': 'text', 'object_relation': 'country-code'},
'country_name': {'type': 'text', 'object_relation': 'country'},
'isp': {'type': 'text', 'object_relation': 'ISP'},
'latitude': {'type': 'float', 'object_relation': 'latitude'},
'longitude': {'type': 'float', 'object_relation': 'longitude'},
'org': {'type': 'text', 'object_relation': 'organization'},
'postal_code': {'type': 'text', 'object_relation': 'zipcode'},
'region_code': {'type': 'text', 'object_relation': 'region-code'}
}
self.ip_port_mapping = {
'domains': {'type': 'domain', 'object_relation': 'domain'},
'hostnames': {'type': 'hostname', 'object_relation': 'hostname'}
}
self.vulnerability_mapping = {
'cvss': {'type': 'float', 'object_relation': 'cvss-score'},
'summary': {'type': 'text', 'object_relation': 'summary'}
}
self.x509_mapping = {
'bits': {'type': 'text', 'object_relation': 'pubkey-info-size'},
'expires': {'type': 'datetime', 'object_relation': 'validity-not-after'},
'issued': {'type': 'datetime', 'object_relation': 'validity-not-before'},
'issuer': {'type': 'text', 'object_relation': 'issuer'},
'serial': {'type': 'text', 'object_relation': 'serial-number'},
'sig_alg': {'type': 'text', 'object_relation': 'signature_algorithm'},
'subject': {'type': 'text', 'object_relation': 'subject'},
'type': {'type': 'text', 'object_relation': 'pubkey-info-algorithm'},
'version': {'type': 'text', 'object_relation': 'version'}
}
def query_shodan(self, apikey):
# Query Shodan and get the results in a json blob
api = shodan.Shodan(apikey)
query_results = api.host(self.attribute.value)
# Parse the information about the IP address used as input
ip_address_attributes = []
for feature, mapping in self.ip_address_mapping.items():
if query_results.get(feature):
attribute = {'value': query_results[feature]}
attribute.update(mapping)
ip_address_attributes.append(attribute)
if ip_address_attributes:
ip_address_object = MISPObject('ip-api-address')
for attribute in ip_address_attributes:
ip_address_object.add_attribute(**attribute)
ip_address_object.add_attribute(**self._get_source_attribute())
ip_address_object.add_reference(self.attribute.uuid, 'describes')
self.misp_event.add_object(ip_address_object)
# Parse the hostnames / domains and ports associated with the IP address
if query_results.get('ports'):
ip_port_object = MISPObject('ip-port')
ip_port_object.add_attribute(**self._get_source_attribute())
feature = self.attribute.type.split('-')[1]
for port in query_results['ports']:
attribute = {
'type': 'port',
'object_relation': f'{feature}-port',
'value': port
}
ip_port_object.add_attribute(**attribute)
for feature, mapping in self.ip_port_mapping.items():
for value in query_results.get(feature, []):
attribute = {'value': value}
attribute.update(mapping)
ip_port_object.add_attribute(**attribute)
ip_port_object.add_reference(self.attribute.uuid, 'extends')
self.misp_event.add_object(ip_port_object)
else:
if any(query_results.get(feature) for feature in ('domains', 'hostnames')):
domain_ip_object = MISPObject('domain-ip')
domain_ip_object.add_attribute(**self._get_source_attribute())
for feature in ('domains', 'hostnames'):
for value in query_results[feature]:
attribute = {
'type': 'domain',
'object_relation': 'domain',
'value': value
}
domain_ip_object.add_attribute(**attribute)
domain_ip_object.add_reference(self.attribute.uuid, 'extends')
self.misp_event.add_object(domain_ip_object)
# Parse data within the "data" field
if query_results.get('vulns'):
vulnerabilities = {}
for data in query_results['data']:
# Parse vulnerabilities
if data.get('vulns'):
for cve, vulnerability in data['vulns'].items():
if cve not in vulnerabilities:
vulnerabilities[cve] = vulnerability
# Also parse the certificates
if data.get('ssl'):
self._parse_cert(data['ssl'])
for cve, vulnerability in vulnerabilities.items():
vulnerability_object = MISPObject('vulnerability')
vulnerability_object.add_attribute(**{
'type': 'vulnerability',
'object_relation': 'id',
'value': cve
})
for feature, mapping in self.vulnerability_mapping.items():
if vulnerability.get(feature):
attribute = {'value': vulnerability[feature]}
attribute.update(mapping)
vulnerability_object.add_attribute(**attribute)
if vulnerability.get('references'):
for reference in vulnerability['references']:
vulnerability_object.add_attribute(**{
'type': 'link',
'object_relation': 'references',
'value': reference
})
vulnerability_object.add_reference(self.attribute.uuid, 'vulnerability-of')
self.misp_event.add_object(vulnerability_object)
for cve_id in query_results['vulns']:
if cve_id not in vulnerabilities:
attribute = {
'type': 'vulnerability',
'value': cve_id
}
self.misp_event.add_attribute(**attribute)
else:
# We have no vulnerability data, we only check if we have
# certificates within the "data" field
for data in query_results['data']:
if data.get('ssl'):
self._parse_cert(data['ssl']['cert'])
def get_result(self):
event = json.loads(self.misp_event.to_json())
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
return {'results': results}
# When we want to add the IP address information in objects such as the
# domain-ip or ip-port objects referencing the input IP address attribute
def _get_source_attribute(self):
return {
'type': self.attribute.type,
'object_relation': self.attribute.type,
'value': self.attribute.value
}
def _parse_cert(self, certificate):
x509_object = MISPObject('x509')
for feature in ('serial', 'sig_alg', 'version'):
if certificate.get(feature):
attribute = {'value': certificate[feature]}
attribute.update(self.x509_mapping[feature])
x509_object.add_attribute(**attribute)
# Parse issuer and subject value
for feature in ('issuer', 'subject'):
if certificate.get(feature):
attribute_value = (f'{identifier}={value}' for identifier, value in certificate[feature].items())
attribute = {'value': f'/{"/".join(attribute_value)}'}
attribute.update(self.x509_mapping[feature])
x509_object.add_attribute(**attribute)
# Parse datetime attributes
for feature in ('expires', 'issued'):
if certificate.get(feature):
attribute = {'value': datetime.strptime(certificate[feature], '%Y%m%d%H%M%SZ')}
attribute.update(self.x509_mapping[feature])
x509_object.add_attribute(**attribute)
# Parse fingerprints
if certificate.get('fingerprint'):
for hash_type, hash_value in certificate['fingerprint'].items():
x509_object.add_attribute(**{
'type': f'x509-fingerprint-{hash_type}',
'object_relation': f'x509-fingerprint-{hash_type}',
'value': hash_value
})
# Parse public key related info
if certificate.get('pubkey'):
for feature, value in certificate['pubkey'].items():
attribute = {'value': value}
attribute.update(self.x509_mapping[feature])
x509_object.add_attribute(**attribute)
x509_object.add_reference(self.attribute.uuid, 'identifies')
self.misp_event.add_object(x509_object)
def handler(q=False):
if q is False:
return False
request = json.loads(q)
if request.get('ip-src'):
toquery = request['ip-src']
elif request.get('ip-dst'):
toquery = request['ip-dst']
else:
misperrors['error'] = "Unsupported attributes type"
return misperrors
if not request.get('config') or not request['config'].get('apikey'):
misperrors['error'] = 'Shodan authentication is missing'
return misperrors
api = shodan.Shodan(request['config'].get('apikey'))
return handle_expansion(api, toquery)
def handle_expansion(api, domain):
return {'results': [{'types': mispattributes['output'], 'values': json.dumps(api.host(domain))}]}
if not request.get('config', {}).get('apikey'):
return {'error': 'Shodan authentication is missing'}
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if attribute['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
shodan_parser = ShodanParser(attribute)
shodan_parser.query_shodan(request['config']['apikey'])
return shodan_parser.get_result()
def introspection():

27
misp_modules/modules/expansion/sophoslabs_intelix.py
View File

@ -1,7 +1,8 @@
from pymisp import MISPEvent, MISPObject
import json
import requests
import base64
from . import check_input_attribute, checking_error, standard_error_message
from pymisp import MISPEvent, MISPObject
from urllib.parse import quote
moduleinfo = {'version': '1.0',
@ -105,13 +106,25 @@ def handler(q=False):
misperrors['error'] = "Missing client_id or client_secret value for SOPHOSLabs Intelix. \
It's free to sign up here https://aws.amazon.com/marketplace/pp/B07SLZPMCS."
return misperrors
to_check = (('type', 'value'), ('type', 'value1'))
if not j.get('attribute') or not any(check_input_attribute(j['attribute'], requirements=check) for check in to_check):
return {'error': f'{standard_error_message}, {checking_error}.'}
attribute = j['attribute']
if attribute['type'] not in misp_types_in:
return {'error': 'Unsupported attribute type.'}
client = SophosLabsApi(j['config']['client_id'], j['config']['client_secret'])
if j['attribute']['type'] == "sha256":
client.hash_lookup(j['attribute']['value1'])
if j['attribute']['type'] in ['ip-dst', 'ip-src', 'ip']:
client.ip_lookup(j["attribute"]["value1"])
if j['attribute']['type'] in ['uri', 'url', 'domain', 'hostname']:
client.url_lookup(j["attribute"]["value1"])
mapping = {
'sha256': 'hash_lookup',
'ip-dst': 'ip_lookup',
'ip-src': 'ip_lookup',
'ip': 'ip_lookup',
'uri': 'url_lookup',
'url': 'url_lookup',
'domain': 'url_lookup',
'hostname': 'url_lookup'
}
attribute_value = attribute['value'] if 'value' in attribute else attribute['value1']
getattr(client, mapping[attribute['type']])(attribute_value)
return client.get_result()

148
misp_modules/modules/expansion/trustar_enrich.py
View File

@ -1,7 +1,11 @@
import json
import pymisp
from base64 import b64encode
from collections import OrderedDict
from . import check_input_attribute, checking_error, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
from trustar import TruStar
from trustar import TruStar, Indicator
from urllib.parse import quote
misperrors = {'error': "Error"}
mispattributes = {
@ -33,9 +37,13 @@ class TruSTARParser:
'SHA256': "sha256"
}
# Relevant fields from each TruSTAR endpoint
SUMMARY_FIELDS = ["severityLevel", "source", "score", "attributes"]
METADATA_FIELDS = ["sightings", "first_seen", "last_seen", "tags"]
REPORT_BASE_URL = "https://station.trustar.co/constellation/reports/{}"
CLIENT_METATAG = "MISP-{}".format(pymisp.__version__)
CLIENT_METATAG = f"MISP-{pymisp.__version__}"
def __init__(self, attribute, config):
config['enclave_ids'] = config.get('enclave_ids', "").strip().split(',')
@ -51,45 +59,111 @@ class TruSTARParser:
"""
Returns the MISP Event enriched with TruSTAR indicator summary data.
"""
event = json.loads(self.misp_event.to_json())
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
return {'results': results}
try:
event = json.loads(self.misp_event.to_json())
results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
return {'results': results}
except Exception as e:
misperrors['error'] += f" -- Encountered issue serializing enrichment data -- {e}"
return misperrors
def generate_trustar_links(self, entity_value):
def generate_trustar_link(self, entity_type, entity_value):
"""
Generates links to TruSTAR reports if they exist.
Generates link to TruSTAR report of entity.
:param entity_type: <str> Type of entity.
:param entity_value: <str> Value of entity.
:return: <str> Link to indicator report in TruSTAR platform.
"""
report_links = list()
trustar_reports = self.ts_client.search_reports(entity_value)
for report in trustar_reports:
report_links.append(self.REPORT_BASE_URL.format(report.id))
report_id = b64encode(quote(f"{entity_type}|{entity_value}").encode()).decode()
return report_links
return self.REPORT_BASE_URL.format(report_id)
def parse_indicator_summary(self, summaries):
@staticmethod
def extract_tags(enrichment_report):
"""
Converts a response from the TruSTAR /1.3/indicators/summaries endpoint
a MISP trustar_report object and adds the summary data and links as attributes.
Extracts tags from the enrichment report in order to add them
to the TruSTAR MISP Object. Removes tags from report to avoid
redundancy.
:param summaries: <generator> A TruSTAR Python SDK Page.generator object for generating
indicator summaries pages.
:param: <OrderedDict> Enrichment data.
:return: <list> List of tags.
"""
if enrichment_report and enrichment_report.get('tags'):
return [tag.get('name') for tag in enrichment_report.pop('tags')]
return None
def generate_enrichment_report(self, summary, metadata):
"""
Extracts desired fields from summary and metadata reports and
generates an enrichment report.
:param summary: <trustar.IndicatorSummary> Indicator summary report.
:param metadata: <trustar.Indicator> Indicator metadata report.
:return: <str> Enrichment report.
"""
# Preserve order of fields as they exist in SUMMARY_FIELDS and METADATA_FIELDS
enrichment_report = OrderedDict()
if summary:
summary_dict = summary.to_dict()
enrichment_report.update(
{field: summary_dict[field] for field in self.SUMMARY_FIELDS if summary_dict.get(field)})
if metadata:
metadata_dict = metadata.to_dict()
enrichment_report.update(
{field: metadata_dict[field] for field in self.METADATA_FIELDS if metadata_dict.get(field)})
return enrichment_report
def parse_indicator_summary(self, indicator, summary, metadata):
"""
Pulls enrichment data from the TruSTAR /indicators/summaries and /indicators/metadata endpoints
and creates a MISP trustar_report.
:param indicator: <str> Value of the attribute
:summary: <trustar.IndicatorSummary> Indicator summary response object.
:metadata: <trustar.Indicator> Indicator response object.
"""
for summary in summaries:
trustar_obj = MISPObject('trustar_report')
# Verify that the indicator type is supported by TruSTAR
if summary and summary.indicator_type in self.ENTITY_TYPE_MAPPINGS:
indicator_type = summary.indicator_type
indicator_value = summary.value
if indicator_type in self.ENTITY_TYPE_MAPPINGS:
elif metadata and metadata.type in self.ENTITY_TYPE_MAPPINGS:
indicator_type = metadata.type
else:
misperrors['error'] += " -- Attribute not found or not supported"
raise Exception
try:
# Extract most relevant fields from indicator summary and metadata responses
enrichment_report = self.generate_enrichment_report(summary, metadata)
tags = self.extract_tags(enrichment_report)
if enrichment_report:
# Create MISP trustar_report object and populate it with enrichment data
trustar_obj = MISPObject('trustar_report')
trustar_obj.add_attribute(indicator_type, attribute_type=self.ENTITY_TYPE_MAPPINGS[indicator_type],
value=indicator_value)
value=indicator)
trustar_obj.add_attribute("INDICATOR_SUMMARY", attribute_type="text",
value=json.dumps(summary.to_dict(), sort_keys=True, indent=4))
report_links = self.generate_trustar_links(indicator_value)
for link in report_links:
trustar_obj.add_attribute("REPORT_LINK", attribute_type="link", value=link)
value=json.dumps(enrichment_report, indent=4))
report_link = self.generate_trustar_link(indicator_type, indicator)
trustar_obj.add_attribute("REPORT_LINK", attribute_type="link", value=report_link)
self.misp_event.add_object(**trustar_obj)
elif not tags:
# If enrichment report is empty and there are no tags, nothing to add to attribute
raise Exception("No relevant data found")
if tags:
for tag in tags:
self.misp_event.add_attribute_tag(tag, indicator)
except Exception as e:
misperrors['error'] += f" -- Error enriching attribute {indicator} -- {e}"
raise e
def handler(q=False):
@ -110,17 +184,31 @@ def handler(q=False):
misperrors['error'] = "Your TruSTAR API key and secret are required for indicator enrichment."
return misperrors
if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
return {'error': f'{standard_error_message}, {checking_error}.'}
attribute = request['attribute']
if attribute['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
trustar_parser = TruSTARParser(attribute, config)
metadata = None
summary = None
try:
summaries = list(
trustar_parser.ts_client.get_indicator_summaries([attribute['value']], page_size=MAX_PAGE_SIZE))
metadata = trustar_parser.ts_client.get_indicators_metadata([Indicator(value=attribute['value'])])[0]
except Exception as e:
misperrors['error'] = "Unable to retrieve TruSTAR summary data: {}".format(e)
misperrors['error'] += f" -- Could not retrieve indicator metadata from TruSTAR {e}"
try:
summary = list(
trustar_parser.ts_client.get_indicator_summaries([attribute['value']], page_size=MAX_PAGE_SIZE))[0]
except Exception as e:
misperrors['error'] += f" -- Unable to retrieve TruSTAR summary data: {e}"
try:
trustar_parser.parse_indicator_summary(attribute['value'], summary, metadata)
except Exception:
return misperrors
trustar_parser.parse_indicator_summary(summaries)
return trustar_parser.get_results()

8
misp_modules/modules/expansion/urlhaus.py
View File

@ -1,6 +1,8 @@
from pymisp import MISPAttribute, MISPEvent, MISPObject
# -*- coding: utf-8 -*-
import json
import requests
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
misperrors = {'error': 'Error'}
mispattributes = {'input': ['domain', 'hostname', 'ip-src', 'ip-dst', 'md5', 'sha256', 'url'],
@ -134,7 +136,11 @@ def handler(q=False):
if q is False:
return False
request = json.loads(q)
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if attribute['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
urlhaus_parser = _misp_type_mapping[attribute['type']](attribute)
return urlhaus_parser.query_api()

12
misp_modules/modules/expansion/virustotal.py
View File

@ -1,6 +1,7 @@
from pymisp import MISPAttribute, MISPEvent, MISPObject
import json
import requests
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
misperrors = {'error': 'Error'}
mispattributes = {'input': ['hostname', 'domain', "ip-src", "ip-dst", "md5", "sha1", "sha256", "url"],
@ -143,7 +144,7 @@ class VirusTotalParser(object):
def parse_resolutions(self, resolutions, subdomains=None, uuids=None):
domain_ip_object = MISPObject('domain-ip')
if self.attribute.type == 'domain':
if self.attribute.type in ('domain', 'hostname'):
domain_ip_object.add_attribute('domain', type='domain', value=self.attribute.value)
attribute_type, relation, key = ('ip-dst', 'ip', 'ip_address')
else:
@ -174,7 +175,7 @@ class VirusTotalParser(object):
vt_object = MISPObject('virustotal-report')
vt_object.add_attribute('permalink', type='link', value=query_result['permalink'])
detection_ratio = '{}/{}'.format(query_result['positives'], query_result['total'])
vt_object.add_attribute('detection-ratio', type='text', value=detection_ratio)
vt_object.add_attribute('detection-ratio', type='text', value=detection_ratio, disable_correlation=True)
self.misp_event.add_object(**vt_object)
return vt_object.uuid
@ -195,6 +196,11 @@ def handler(q=False):
if not request.get('config') or not request['config'].get('apikey'):
misperrors['error'] = "A VirusTotal api key is required for this module."
return misperrors
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
if request['attribute']['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
event_limit = request['config'].get('event_limit')
if not isinstance(event_limit, int):
event_limit = 5

9
misp_modules/modules/expansion/virustotal_public.py
View File

@ -1,6 +1,7 @@
from pymisp import MISPAttribute, MISPEvent, MISPObject
import json
import requests
from . import check_input_attribute, standard_error_message
from pymisp import MISPAttribute, MISPEvent, MISPObject
misperrors = {'error': 'Error'}
mispattributes = {'input': ['hostname', 'domain', "ip-src", "ip-dst", "md5", "sha1", "sha256", "url"],
@ -36,7 +37,7 @@ class VirusTotalParser():
def parse_resolutions(self, resolutions, subdomains=None, uuids=None):
domain_ip_object = MISPObject('domain-ip')
if self.attribute.type == 'domain':
if self.attribute.type in ('domain', 'hostname'):
domain_ip_object.add_attribute('domain', type='domain', value=self.attribute.value)
attribute_type, relation, key = ('ip-dst', 'ip', 'ip_address')
else:
@ -174,7 +175,11 @@ def handler(q=False):
if not request.get('config') or not request['config'].get('apikey'):
misperrors['error'] = "A VirusTotal api key is required for this module."
return misperrors
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
attribute = request['attribute']
if attribute['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
query_type, to_call = misp_type_mapping[attribute['type']]
parser = to_call(request['config']['apikey'], attribute)
query_result = parser.get_query_result(query_type)

5
misp_modules/modules/expansion/xforceexchange.py
View File

@ -1,6 +1,7 @@
import requests
import json
import sys
from . import check_input_attribute, standard_error_message
from collections import defaultdict
from pymisp import MISPAttribute, MISPEvent, MISPObject
from requests.auth import HTTPBasicAuth
@ -160,6 +161,10 @@ def handler(q=False):
return misperrors
key = request["config"]["apikey"]
password = request['config']['apipassword']
if not request.get('attribute') or not check_input_attribute(request['attribute']):
return {'error': f'{standard_error_message} which should contain at least a type, a value and an uuid.'}
if request['attribute']['type'] not in mispattributes['input']:
return {'error': 'Unsupported attribute type.'}
parser = XforceExchange(request['attribute'], key, password)
parser.parse()
return parser.get_result()

2
misp_modules/modules/import_mod/email_import.py
View File

@ -42,7 +42,7 @@ def handler(q=False):
# request data is always base 64 byte encoded
data = base64.b64decode(request["data"])
email_object = EMailObject(pseudofile=BytesIO(data), attach_original_mail=True, standalone=False)
email_object = EMailObject(pseudofile=BytesIO(data), attach_original_email=True, standalone=False)
# Check if we were given a configuration
config = request.get("config", {})

34
tests/test_expansions.py
View File

@ -97,9 +97,16 @@ class TestExpansions(unittest.TestCase):
self.assertEqual(self.get_errors(response), 'An API key for APIVoid is required.')
def test_bgpranking(self):
query = {"module": "bgpranking", "AS": "13335"}
query = {
"module": "bgpranking",
"attribute": {
"type": "AS",
"value": "13335",
"uuid": "ea89a33b-4ab7-4515-9f02-922a0bee333d"
}
}
response = self.misp_modules_post(query)
self.assertEqual(self.get_values(response)['response']['asn_description'], 'CLOUDFLARENET, US')
self.assertEqual(self.get_object(response), 'asn')
def test_btc_steroids(self):
query = {"module": "btc_steroids", "btc": "1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA"}
@ -229,11 +236,24 @@ class TestExpansions(unittest.TestCase):
self.assertEqual(to_check, 'OK (Not Found)', response)
def test_greynoise(self):
query = {"module": "greynoise", "ip-dst": "1.1.1.1"}
response = self.misp_modules_post(query)
value = self.get_values(response)
if value != 'GreyNoise API not accessible (HTTP 429)':
self.assertTrue(value.startswith('{"ip":"1.1.1.1","status":"ok"'))
module_name = 'greynoise'
query = {"module": module_name, "ip-dst": "1.1.1.1"}
if module_name in self.configs:
query['config'] = self.configs[module_name]
response = self.misp_modules_post(query)
try:
self.assertEqual(self.get_values(response), 'This IP is commonly spoofed in Internet-scan activity')
except Exception:
self.assertIn(
self.get_errors(reponse),
(
"Unauthorized. Please check your API key.",
"Too many requests. You've hit the rate-limit."
)
)
else:
response = self.misp_modules_post(query)
self.assertEqual(self.get_errors(response), 'Missing Greynoise API key.')
def test_ipasn(self):
query = {"module": "ipasn",