Merge branch 'main' of github.com:MISP/misp-modules into main

README.md

@@ -22,7 +22,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
 * [AssemblyLine submit](misp_modules/modules/expansion/assemblyline_submit.py) - an expansion module to submit samples and urls to AssemblyLine.
 * [AssemblyLine query](misp_modules/modules/expansion/assemblyline_query.py) - an expansion module to query AssemblyLine and parse the full submission report.
 * [Backscatter.io](misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations.
-* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description, its history, and position in BGP Ranking.
+* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description and its ranking and position in BGP Ranking.
 * [RansomcoinDB check](misp_modules/modules/expansion/ransomcoindb.py) - An expansion hover module to query the [ransomcoinDB](https://ransomcoindb.concinnity-risks.com): it contains mapping between BTC addresses and malware hashes. Enrich MISP by querying for BTC -> hash or hash -> BTC addresses.
 * [BTC scam check](misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused.
 * [BTC transactions](misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP.
@@ -69,6 +69,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
 * [pptx-enrich](misp_modules/modules/expansion/pptx_enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser).
 * [qrcode](misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values.
 * [rbl](misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute.
+* [recordedfuture](misp_modules/modules/expansion/recordedfuture.py) - a hover and expansion module for enriching MISP attributes with threat intelligence from Recorded Future.
 * [reversedns](misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes.
 * [securitytrails](misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/).
 * [shodan](misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module.
@@ -79,6 +80,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
 * [STIX2 pattern syntax validator](misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax.
 * [ThreatCrowd](misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/).
 * [threatminer](misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/).
+* [TruSTAR Enrich](misp_modules/modules/expansion/trustar_enrich.py) - an expansion module to enrich MISP data with [TruSTAR](https://www.trustar.co/).
 * [urlhaus](misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url.
 * [urlscan](misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io).
 * [virustotal](misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://developers.virustotal.com/reference))

doc/README.md

@@ -108,13 +108,13 @@ Query backscatter.io (https://backscatter.io/).
 
 Query BGP Ranking (https://bgpranking-ng.circl.lu/).
 - **features**:
->The module takes an AS number attribute as input and displays its description and history, and position in BGP Ranking.
+>The module takes an AS number attribute as input and displays its description as well as its ranking position in BGP Ranking for a given day.
 >
 >
 - **input**:
 >Autonomous system number.
 - **output**:
->Text containing a description of the ASN, its history, and the position in BGP Ranking.
+>An asn object with its related bgp-ranking object.
 - **references**:
 >https://github.com/D4-project/BGP-Ranking/
 - **requirements**:
@@ -311,6 +311,26 @@ An expansion hover module to expand information about CVE id.
 
 -----
 
+#### [cve_advanced](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/cve_advanced.py)
+
+<img src=logos/cve.png height=60>
+
+An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).
+- **features**:
+>The module takes a vulnerability attribute as input and queries the CIRCL CVE search API to gather additional information.
+>
+>The result of the query is then parsed to return additional information about the vulnerability, like its cvss score or some references, as well as the potential related weaknesses and attack patterns.
+>
+>The vulnerability additional data is returned in a vulnerability MISP object, and the related additional information are put into weakness and attack-pattern MISP objects.
+- **input**:
+>Vulnerability attribute.
+- **output**:
+>Additional information about the vulnerability, such as its cvss score, some references, or the related weaknesses and attack patterns.
+- **references**:
+>https://cve.circl.lu, https://cve/mitre.org/
+
+-----
+
 #### [cytomic_orion](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/cytomic_orion.py)
 
 <img src=logos/cytomic_orion.png height=60>
@@ -369,7 +389,7 @@ A simple DNS expansion service to resolve IP address from domain MISP attributes
 
 -----
 
-#### [docx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/docx-enrich.py)
+#### [docx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/docx_enrich.py)
 
 <img src=logos/docx.png height=60>
 
@@ -476,6 +496,42 @@ Module to access Farsight DNSDB Passive DNS.
 
 -----
 
+#### [geoip_asn](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_asn.py)
+
+<img src=logos/maxmind.png height=60>
+- **descrption**:
+>An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about its related AS number.
+- **features**:
+>The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the related AS number.
+- **input**:
+>An IP address MISP attribute.
+- **output**:
+>Text containing information about the AS number of the IP address.
+- **references**:
+>https://www.maxmind.com/en/home
+- **requirements**:
+>A local copy of Maxmind's Geolite database
+
+-----
+
+#### [geoip_city](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_city.py)
+
+<img src=logos/maxmind.png height=60>
+
+An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located.
+- **features**:
+>The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the city where this IP address is located.
+- **input**:
+>An IP address MISP attribute.
+- **output**:
+>Text containing information about the city where the IP address is located.
+- **references**:
+>https://www.maxmind.com/en/home
+- **requirements**:
+>A local copy of Maxmind's Geolite database
+
+-----
+
 #### [geoip_country](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_country.py)
 
 <img src=logos/maxmind.png height=60>
@@ -496,19 +552,39 @@ Module to query a local copy of Maxmind's Geolite database.
 
 -----
 
+#### [google_search](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/google_search.py)
+
+<img src=logos/google.png height=60>
+- **descrption**:
+>A hover module to get information about an url using a Google search.
+- **features**:
+>The module takes an url as input to query the Google search API. The result of the query is then return as raw text.
+- **input**:
+>An url attribute.
+- **output**:
+>Text containing the result of a Google search on the input url.
+- **references**:
+>https://github.com/abenassi/Google-Search-API
+- **requirements**:
+>The python Google Search API library
+
+-----
+
 #### [greynoise](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/greynoise.py)
 
 <img src=logos/greynoise.png height=60>
 
 Module to access GreyNoise.io API
 - **features**:
->The module takes an IP address as input and queries Greynoise for some additional information about it. The result is returned as text.
+>The module takes an IP address as input and queries Greynoise for some additional information about it: basically it checks whether a given IP address is “Internet background noise”, or has been observed scanning or attacking devices across the Internet. The result is returned as text.
 - **input**:
 >An IP address.
 - **output**:
 >Additional information about the IP fetched from Greynoise API.
 - **references**:
 >https://greynoise.io/, https://github.com/GreyNoise-Intelligence/api.greynoise.io
+- **requirements**:
+>A Greynoise API key.
 
 -----
 
@@ -542,6 +618,37 @@ Module to access haveibeenpwned.com API.
 
 -----
 
+#### [intel471](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/intel471.py)
+
+<img src=logos/intel471.png height=60>
+- **descrption**:
+>An expansion module to query Intel471 in order to get additional information about a domain, ip address, email address, url or hash.
+- **features**:
+>The module uses the Intel471 python library to query the Intel471 API with the value of the input attribute. The result of the query is then returned as freetext so the Freetext import parses it.
+- **input**:
+>A MISP attribute whose type is included in the following list:
+>- hostname
+>- domain
+>- url
+>- ip-src
+>- ip-dst
+>- email-src
+>- email-dst
+>- target-email
+>- whois-registrant-email
+>- whois-registrant-name
+>- md5
+>- sha1
+>- sha256
+- **output**:
+>Freetext
+- **references**:
+>https://public.intel471.com/
+- **requirements**:
+>The intel471 python library
+
+-----
+
 #### [intelmq_eventdb](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/intelmq_eventdb.py)
 
 <img src=logos/intelmq.png height=60>
@@ -731,7 +838,7 @@ Query the MALWAREbazaar API to get additional information about the input hash a
 
 -----
 
-#### [ocr-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ocr-enrich.py)
+#### [ocr_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ocr_enrich.py)
 
 Module to process some optical character recognition on pictures.
 - **features**:
@@ -745,7 +852,7 @@ Module to process some optical character recognition on pictures.
 
 -----
 
-#### [ods-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ods-enrich.py)
+#### [ods_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ods_enrich.py)
 
 <img src=logos/ods.png height=60>
 
@@ -761,7 +868,7 @@ Module to extract freetext from a .ods document.
 
 -----
 
-#### [odt-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/odt-enrich.py)
+#### [odt_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/odt_enrich.py)
 
 <img src=logos/odt.png height=60>
 
@@ -900,7 +1007,7 @@ Module to get information from AlienVault OTX.
 
 -----
 
-#### [pdf-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pdf-enrich.py)
+#### [pdf_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pdf_enrich.py)
 
 <img src=logos/pdf.jpg height=60>
 
@@ -916,7 +1023,7 @@ Module to extract freetext from a PDF document.
 
 -----
 
-#### [pptx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pptx-enrich.py)
+#### [pptx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pptx_enrich.py)
 
 <img src=logos/pptx.png height=60>
 
@@ -946,6 +1053,24 @@ Module to decode QR codes.
 
 -----
 
+#### [ransomcoindb](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ransomcoindb.py)
+- **descrption**:
+>Module to access the ransomcoinDB with a hash or btc address attribute and get the associated btc address of hashes.
+- **features**:
+>The module takes either a hash attribute or a btc attribute as input to query the ransomcoinDB API for some additional data.
+>
+>If the input is a btc address, we will get the associated hashes returned in a file MISP object. If we query ransomcoinDB with a hash, the response contains the associated btc addresses returned as single MISP btc attributes.
+- **input**:
+>A hash (md5, sha1 or sha256) or btc attribute.
+- **output**:
+>Hashes associated to a btc address or btc addresses associated to a hash.
+- **references**:
+>https://ransomcoindb.concinnity-risks.com
+- **requirements**:
+>A ransomcoinDB API key.
+
+-----
+
 #### [rbl](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/rbl.py)
 
 Module to check an IPv4 address against known RBLs.
@@ -964,6 +1089,24 @@ Module to check an IPv4 address against known RBLs.
 
 -----
 
+#### [recordedfuture](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/recordedfuture.py)
+
+<img src=logos/recordedfuture.png height=60>
+
+Module to enrich attributes with threat intelligence from Recorded Future.
+- **features**:
+>Enrich an attribute to add a custom enrichment object to the event. The object contains a copy of the enriched attribute with added tags presenting risk score and triggered risk rules from Recorded Future. Malware and Threat Actors related to the enriched indicator in Recorded Future is matched against MISP's galaxy clusters and applied as galaxy tags. The custom enrichment object also includes a list of related indicators from Recorded Future (IP's, domains, hashes, URL's and vulnerabilities) added as additional attributes.
+- **input**:
+>A MISP attribute of one of the following types: ip, ip-src, ip-dst, domain, hostname, md5, sha1, sha256, uri, url, vulnerability, weakness.
+- **output**:
+>A MISP object containing a copy of the enriched attribute with added tags from Recorded Future and a list of new attributes related to the enriched attribute.
+- **references**:
+>https://www.recordedfuture.com/
+- **requirements**:
+>A Recorded Future API token.
+
+-----
+
 #### [reversedns](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/reversedns.py)
 
 Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes.
@@ -1071,6 +1214,24 @@ An expansion hover module to perform a syntax check on sigma rules.
 
 -----
 
+#### [sophoslabs_intelix](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/sophoslabs_intelix.py)
+
+<img src=logos/sophoslabs_intelix.svg height=60>
+
+An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute.
+- **features**:
+>The module takes an ip address, url, domain or sha256 attribute and queries the SophosLabs Intelix API with the attribute value. The result of this query is a SophosLabs Intelix hash report, or an ip or url lookup, that is then parsed and returned in a MISP object.
+- **input**:
+>An ip address, url, domain or sha256 attribute.
+- **output**:
+>SophosLabs Intelix report and lookup objects
+- **references**:
+>https://aws.amazon.com/marketplace/pp/B07SLZPMCS
+- **requirements**:
+>A client_id and client_secret pair to authenticate to the SophosLabs Intelix API
+
+-----
+
 #### [sourcecache](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/sourcecache.py)
 
 Module to cache web pages of analysis reports, OSINT sources. The module returns a link of the cached page.
@@ -1422,7 +1583,7 @@ An expansion module for IBM X-Force Exchange.
 
 -----
 
-#### [xlsx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/xlsx-enrich.py)
+#### [xlsx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/xlsx_enrich.py)
 
 <img src=logos/xlsx.png height=60>
 

doc/expansion/bgpranking.json

@@ -1,8 +1,8 @@
 {
     "description": "Query BGP Ranking (https://bgpranking-ng.circl.lu/).",
     "requirements": ["pybgpranking python library"],
-    "features": "The module takes an AS number attribute as input and displays its description and history, and position in BGP Ranking.\n\n",
+    "features": "The module takes an AS number attribute as input and displays its description as well as its ranking position in BGP Ranking for a given day.\n\n",
     "references": ["https://github.com/D4-project/BGP-Ranking/"],
     "input": "Autonomous system number.",
-    "output": "Text containing a description of the ASN, its history, and the position in BGP Ranking."
+    "output": "An asn object with its related bgp-ranking object."
 }

doc/expansion/cve_advanced.json

@@ -0,0 +1,8 @@
+{
+    "description": "An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).",
+    "logo": "logos/cve.png",
+    "input": "Vulnerability attribute.",
+    "output": "Additional information about the vulnerability, such as its cvss score, some references, or the related weaknesses and attack patterns.",
+    "references": ["https://cve.circl.lu", "https://cve/mitre.org/"],
+    "features": "The module takes a vulnerability attribute as input and queries the CIRCL CVE search API to gather additional information.\n\nThe result of the query is then parsed to return additional information about the vulnerability, like its cvss score or some references, as well as the potential related weaknesses and attack patterns.\n\nThe vulnerability additional data is returned in a vulnerability MISP object, and the related additional information are put into weakness and attack-pattern MISP objects."
+}

doc/expansion/geoip_asn.json

@@ -0,0 +1,9 @@
+{
+    "descrption": "An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about its related AS number.",
+    "logo": "logos/maxmind.png",
+    "requirements": ["A local copy of Maxmind's Geolite database"],
+    "input": "An IP address MISP attribute.",
+    "output": "Text containing information about the AS number of the IP address.",
+    "references": ["https://www.maxmind.com/en/home"],
+    "features": "The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the related AS number."
+}

doc/expansion/geoip_city.json

@@ -0,0 +1,9 @@
+{
+    "description": "An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located.",
+    "logo": "logos/maxmind.png",
+    "requirements": ["A local copy of Maxmind's Geolite database"],
+    "input": "An IP address MISP attribute.",
+    "output": "Text containing information about the city where the IP address is located.",
+    "references": ["https://www.maxmind.com/en/home"],
+    "features": "The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the city where this IP address is located."
+}

doc/expansion/google_search.json

@@ -0,0 +1,9 @@
+{
+    "descrption": "A hover module to get information about an url using a Google search.",
+    "logo": "logos/google.png",
+    "requirements": ["The python Google Search API library"],
+    "input": "An url attribute.",
+    "output": "Text containing the result of a Google search on the input url.",
+    "references": ["https://github.com/abenassi/Google-Search-API"],
+    "features": "The module takes an url as input to query the Google search API. The result of the query is then return as raw text."
+}

doc/expansion/greynoise.json

@@ -1,9 +1,9 @@
 {
     "description": "Module to access GreyNoise.io API",
     "logo": "logos/greynoise.png",
-    "requirements": [],
+    "requirements": ["A Greynoise API key."],
     "input": "An IP address.",
     "output": "Additional information about the IP fetched from Greynoise API.",
     "references": ["https://greynoise.io/", "https://github.com/GreyNoise-Intelligence/api.greynoise.io"],
-    "features": "The module takes an IP address as input and queries Greynoise for some additional information about it. The result is returned as text."
+    "features": "The module takes an IP address as input and queries Greynoise for some additional information about it: basically it checks whether a given IP address is “Internet background noise”, or has been observed scanning or attacking devices across the Internet. The result is returned as text."
 }

doc/expansion/intel471.json

@@ -0,0 +1,9 @@
+{
+    "descrption": "An expansion module to query Intel471 in order to get additional information about a domain, ip address, email address, url or hash.",
+    "logo": "logos/intel471.png",
+    "requirements": ["The intel471 python library"],
+    "input": "A MISP attribute whose type is included in the following list:\n- hostname\n- domain\n- url\n- ip-src\n- ip-dst\n- email-src\n- email-dst\n- target-email\n- whois-registrant-email\n- whois-registrant-name\n- md5\n- sha1\n- sha256",
+    "output": "Freetext",
+    "references": ["https://public.intel471.com/"],
+    "features": "The module uses the Intel471 python library to query the Intel471 API with the value of the input attribute. The result of the query is then returned as freetext so the Freetext import parses it."
+}

doc/expansion/ransomcoindb.json

@@ -0,0 +1,8 @@
+{
+    "descrption": "Module to access the ransomcoinDB with a hash or btc address attribute and get the associated btc address of hashes.",
+    "requirements": ["A ransomcoinDB API key."],
+    "input": "A hash (md5, sha1 or sha256) or btc attribute.",
+    "output": "Hashes associated to a btc address or btc addresses associated to a hash.",
+    "references": ["https://ransomcoindb.concinnity-risks.com"],
+    "features": "The module takes either a hash attribute or a btc attribute as input to query the ransomcoinDB API for some additional data.\n\nIf the input is a btc address, we will get the associated hashes returned in a file MISP object. If we query ransomcoinDB with a hash, the response contains the associated btc addresses returned as single MISP btc attributes."
+}

doc/expansion/recordedfuture.json

@@ -0,0 +1,9 @@
+{
+    "description": "Module to enrich attributes with threat intelligence from Recorded Future.",
+    "logo": "logos/recordedfuture.png",
+    "requirements": ["A Recorded Future API token."],
+    "input": "A MISP attribute of one of the following types: ip, ip-src, ip-dst, domain, hostname, md5, sha1, sha256, uri, url, vulnerability, weakness.",
+    "output": "A MISP object containing a copy of the enriched attribute with added tags from Recorded Future and a list of new attributes related to the enriched attribute.",
+    "references": ["https://www.recordedfuture.com/"],
+    "features": "Enrich an attribute to add a custom enrichment object to the event. The object contains a copy of the enriched attribute with added tags presenting risk score and triggered risk rules from Recorded Future. Malware and Threat Actors related to the enriched indicator in Recorded Future is matched against MISP's galaxy clusters and applied as galaxy tags. The custom enrichment object also includes a list of related indicators from Recorded Future (IP's, domains, hashes, URL's and vulnerabilities) added as additional attributes."
+}

doc/expansion/sophoslabs_intelix.json

@@ -0,0 +1,9 @@
+{
+    "description": "An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute.",
+    "logo": "logos/sophoslabs_intelix.svg",
+    "requirements": ["A client_id and client_secret pair to authenticate to the SophosLabs Intelix API"],
+    "input": "An ip address, url, domain or sha256 attribute.",
+    "output": "SophosLabs Intelix report and lookup objects",
+    "references": ["https://aws.amazon.com/marketplace/pp/B07SLZPMCS"],
+    "features": "The module takes an ip address, url, domain or sha256 attribute and queries the SophosLabs Intelix API with the attribute value. The result of this query is a SophosLabs Intelix hash report, or an ip or url lookup, that is then parsed and returned in a MISP object."
+}

misp_modules/modules/expansion/__init__.py

@@ -18,4 +18,14 @@ __all__ = ['cuckoo_submit', 'vmray_submit', 'bgpranking', 'circl_passivedns', 'c
            'virustotal_public', 'apiosintds', 'urlscan', 'securitytrails', 'apivoid',
            'assemblyline_submit', 'assemblyline_query', 'ransomcoindb', 'malwarebazaar',
            'lastline_query', 'lastline_submit', 'sophoslabs_intelix', 'cytomic_orion', 'censys_enrich',
-           'trustar_enrich']
+           'trustar_enrich', 'recordedfuture']
+
+
+minimum_required_fields = ('type', 'uuid', 'value')
+
+checking_error = 'containing at least a "type" field and a "value" field'
+standard_error_message = 'This module requires an "attribute" field as input'
+
+
+def check_input_attribute(attribute, requirements=minimum_required_fields):
+    return all(feature in attribute for feature in requirements)

misp_modules/modules/expansion/apivoid.py

@@ -1,5 +1,6 @@
 import json
 import requests
+from . import check_input_attribute, standard_error_message
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
@@ -74,7 +75,11 @@ def handler(q=False):
     request = json.loads(q)
     if not request.get('config', {}).get('apikey'):
         return {'error': 'An API key for APIVoid is required.'}
-    attribute = request.get('attribute')
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
+    attribute = request['attribute']
+    if attribute['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     apikey = request['config']['apikey']
     apivoid_parser = APIVoidParser(attribute)
     apivoid_parser.parse_domain(apikey)

misp_modules/modules/expansion/assemblyline_query.py

@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 import json
+from . import check_input_attribute, standard_error_message
 from assemblyline_client import Client, ClientError
 from collections import defaultdict
 from pymisp import MISPAttribute, MISPEvent, MISPObject
@@ -139,6 +140,10 @@ def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
+    if request['attribute']['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     if not request.get('config'):
         return {"error": "Missing configuration."}
     if not request['config'].get('apiurl'):

misp_modules/modules/expansion/bgpranking.py

@@ -1,13 +1,15 @@
 # -*- coding: utf-8 -*-
 
 import json
-from datetime import date, timedelta
+from . import check_input_attribute, standard_error_message
+from datetime import date, datetime, timedelta
 from pybgpranking import BGPRanking
+from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
-mispattributes = {'input': ['AS'], 'output': ['freetext']}
+mispattributes = {'input': ['AS'], 'format': 'misp_standard'}
 moduleinfo = {'version': '0.1', 'author': 'Raphaël Vinot',
-              'description': 'Query an ASN Description history service (https://github.com/CIRCL/ASN-Description-History.git)',
+              'description': 'Query BGP Ranking to get the ranking of an Autonomous System number.',
               'module-type': ['expansion', 'hover']}
 
 
@@ -15,19 +17,65 @@ def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
-    if request.get('AS'):
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
-        toquery = request['AS']
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
-    else:
+    toquery = request['attribute']
-        misperrors['error'] = "Unsupported attributes type"
+    if toquery['type'] not in mispattributes['input']:
-        return misperrors
+        return {'error': 'Unsupported attribute type.'}
 
     bgpranking = BGPRanking()
-    values = bgpranking.query(toquery, date=(date.today() - timedelta(1)).isoformat())
+    value_toquery = int(toquery['value'][2:]) if toquery['value'].startswith('AS') else int(toquery['value'])
+    values = bgpranking.query(value_toquery, date=(date.today() - timedelta(1)).isoformat())
 
-    if not values:
+    if not values['response'] or not values['response']['asn_description']:
-        misperrors['error'] = 'Unable to find the ASN in BGP Ranking'
+        misperrors['error'] = 'There is no result about this ASN in BGP Ranking'
         return misperrors
-    return {'results': [{'types': mispattributes['output'], 'values': values}]}
+
+    event = MISPEvent()
+    attribute = MISPAttribute()
+    attribute.from_dict(**toquery)
+    event.add_attribute(**attribute)
+
+    asn_object = MISPObject('asn')
+    asn_object.add_attribute(**{
+        'type': 'AS',
+        'object_relation': 'asn',
+        'value': values['meta']['asn']
+    })
+    description, country = values['response']['asn_description'].split(', ')
+    for relation, value in zip(('description', 'country'), (description, country)):
+        asn_object.add_attribute(**{
+            'type': 'text',
+            'object_relation': relation,
+            'value': value
+        })
+
+    mapping = {
+        'address_family': {'type': 'text', 'object_relation': 'address-family'},
+        'date': {'type': 'datetime', 'object_relation': 'date'},
+        'position': {'type': 'float', 'object_relation': 'position'},
+        'rank': {'type': 'float', 'object_relation': 'ranking'}
+    }
+    bgp_object = MISPObject('bgp-ranking')
+    for feature in ('rank', 'position'):
+        bgp_attribute = {'value': values['response']['ranking'][feature]}
+        bgp_attribute.update(mapping[feature])
+        bgp_object.add_attribute(**bgp_attribute)
+    date_attribute = {'value': datetime.strptime(values['meta']['date'], '%Y-%m-%d')}
+    date_attribute.update(mapping['date'])
+    bgp_object.add_attribute(**date_attribute)
+    address_attribute = {'value': values['meta']['address_family']}
+    address_attribute.update(mapping['address_family'])
+    bgp_object.add_attribute(**address_attribute)
+
+    asn_object.add_reference(attribute.uuid, 'describes')
+    asn_object.add_reference(bgp_object.uuid, 'ranked-with')
+    event.add_object(asn_object)
+    event.add_object(bgp_object)
+
+    event = json.loads(event.to_json())
+    results = {key: event[key] for key in ('Attribute', 'Object')}
+    return {'results': results}
 
 
 def introspection():

misp_modules/modules/expansion/censys_enrich.py

@@ -3,6 +3,7 @@ import json
 import base64
 import codecs
 from dateutil.parser import isoparse
+from . import check_input_attribute, standard_error_message
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 try:
     import censys.base
@@ -36,11 +37,11 @@ def handler(q=False):
     api_id = request['config']['api_id']
     api_secret = request['config']['api_secret']
 
-    if not request.get('attribute'):
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
-        return {'error': 'Unsupported input.'}
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
     attribute = request['attribute']
     if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
-        return {'error': 'Unsupported attributes type'}
+        return {'error': 'Unsupported attribute type.'}
 
     attribute = MISPAttribute()
     attribute.from_dict(**request['attribute'])

misp_modules/modules/expansion/circl_passivedns.py

@@ -1,5 +1,6 @@
 import json
 import pypdns
+from . import check_input_attribute, standard_error_message
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 mispattributes = {'input': ['hostname', 'domain', 'ip-src', 'ip-dst', 'ip-src|port', 'ip-dst|port'], 'format': 'misp_standard'}
@@ -58,11 +59,11 @@ def handler(q=False):
     if not request['config'].get('username') or not request['config'].get('password'):
         return {'error': 'CIRCL Passive DNS authentication is incomplete, please provide your username and password.'}
     authentication = (request['config']['username'], request['config']['password'])
-    if not request.get('attribute'):
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
-        return {'error': 'Unsupported input.'}
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
     attribute = request['attribute']
     if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
-        return {'error': 'Unsupported attributes type'}
+        return {'error': 'Unsupported attribute type.'}
     pdns_parser = PassiveDNSParser(attribute, authentication)
     pdns_parser.parse()
     return pdns_parser.get_results()

misp_modules/modules/expansion/circl_passivessl.py

@@ -1,5 +1,6 @@
 import json
 import pypssl
+from . import check_input_attribute, standard_error_message
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 mispattributes = {'input': ['ip-src', 'ip-dst', 'ip-src|port', 'ip-dst|port'], 'format': 'misp_standard'}
@@ -83,11 +84,11 @@ def handler(q=False):
     if not request['config'].get('username') or not request['config'].get('password'):
         return {'error': 'CIRCL Passive SSL authentication is incomplete, please provide your username and password.'}
     authentication = (request['config']['username'], request['config']['password'])
-    if not request.get('attribute'):
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
-        return {'error': 'Unsupported input.'}
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
     attribute = request['attribute']
     if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
-        return {'error': 'Unsupported attributes type'}
+        return {'error': 'Unsupported attribute type.'}
     pssl_parser = PassiveSSLParser(attribute, authentication)
     pssl_parser.parse()
     return pssl_parser.get_results()

misp_modules/modules/expansion/cve_advanced.py

@@ -1,7 +1,8 @@
-from collections import defaultdict
-from pymisp import MISPEvent, MISPObject
 import json
 import requests
+from . import check_input_attribute, standard_error_message
+from collections import defaultdict
+from pymisp import MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
 mispattributes = {'input': ['vulnerability'], 'format': 'misp_standard'}
@@ -55,7 +56,7 @@ class VulnerabilityParser():
                         value = value['title']
                     vulnerability_object.add_attribute(relation, **{'type': attribute_type, 'value': value})
         vulnerability_object.add_reference(self.attribute['uuid'], 'related-to')
-        self.misp_event.add_object(**vulnerability_object)
+        self.misp_event.add_object(vulnerability_object)
         if 'cwe' in self.vulnerability and self.vulnerability['cwe'] not in ('Unknown', 'NVD-CWE-noinfo'):
             self.__parse_weakness(vulnerability_object.uuid)
         if 'capec' in self.vulnerability:
@@ -78,7 +79,7 @@ class VulnerabilityParser():
             for related_weakness in capec['related_weakness']:
                 attribute = dict(type='weakness', value="CWE-{}".format(related_weakness))
                 capec_object.add_attribute('related-weakness', **attribute)
-            self.misp_event.add_object(**capec_object)
+            self.misp_event.add_object(capec_object)
             self.references[vulnerability_uuid].append(dict(referenced_uuid=capec_object.uuid,
                                                             relationship_type='targeted-by'))
 
@@ -94,7 +95,7 @@ class VulnerabilityParser():
                     for feature, relation in self.weakness_mapping.items():
                         if cwe.get(feature):
                             weakness_object.add_attribute(relation, **dict(type=attribute_type, value=cwe[feature]))
-                    self.misp_event.add_object(**weakness_object)
+                    self.misp_event.add_object(weakness_object)
                     self.references[vulnerability_uuid].append(dict(referenced_uuid=weakness_object.uuid,
                                                                     relationship_type='weakened-by'))
                     break
@@ -108,7 +109,9 @@ def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
-    attribute = request.get('attribute')
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
+    attribute = request['attribute']
     if attribute.get('type') != 'vulnerability':
         misperrors['error'] = 'Vulnerability id missing.'
         return misperrors

misp_modules/modules/expansion/cytomic_orion.py

@@ -7,6 +7,7 @@ An expansion module to enrich attributes in MISP and share indicators of comprom
 
 '''
 
+from . import check_input_attribute, standard_error_message
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 import json
 import requests
@@ -146,9 +147,11 @@ def handler(q=False):
     if not request.get('attribute'):
         return {'error': 'Unsupported input.'}
 
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
     attribute = request['attribute']
     if not any(input_type == attribute['type'] for input_type in mispattributes['input']):
-        return {'error': 'Unsupported attributes type'}
+        return {'error': 'Unsupported attribute type.'}
 
     if not request.get('config'):
         return {'error': 'Missing configuration'}

misp_modules/modules/expansion/greynoise.py

@@ -3,35 +3,59 @@ import json
 
 misperrors = {'error': 'Error'}
 mispattributes = {'input': ['ip-dst', 'ip-src'], 'output': ['text']}
-moduleinfo = {'version': '0.1', 'author': 'Aurélien Schwab <aurelien.schwab+dev@gmail.com>', 'description': 'Module to access GreyNoise.io API.', 'module-type': ['hover']}
+moduleinfo = {
-moduleconfig = ['user-agent']  # TODO take this into account in the code
+    'version': '0.2',
+    'author': 'Aurélien Schwab <aurelien.schwab+dev@gmail.com>',
+    'description': 'Module to access GreyNoise.io API.',
+    'module-type': ['hover']
+}
+moduleconfig = ['api_key']
 
-greynoise_api_url = 'http://api.greynoise.io:8888/v1/query/ip'
+greynoise_api_url = 'https://api.greynoise.io/v2/noise/quick/'
-default_user_agent = 'MISP-Module'
+codes_mapping = {
+    '0x00': 'The IP has never been observed scanning the Internet',
+    '0x01': 'The IP has been observed by the GreyNoise sensor network',
+    '0x02': 'The IP has been observed scanning the GreyNoise sensor network, but has not completed a full connection, meaning this can be spoofed',
+    '0x03': 'The IP is adjacent to another host that has been directly observed by the GreyNoise sensor network',
+    '0x04': 'Reserved',
+    '0x05': 'This IP is commonly spoofed in Internet-scan activity',
+    '0x06': 'This IP has been observed as noise, but this host belongs to a cloud provider where IPs can be cycled frequently',
+    '0x07': 'This IP is invalid',
+    '0x08': 'This IP was classified as noise, but has not been observed engaging in Internet-wide scans or attacks in over 60 days'
+}
 
 
 def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
+    if not request.get('config') or not request['config'].get('api_key'):
+        return {'error': 'Missing Greynoise API key.'}
+    headers = {
+        'Accept': 'application/json',
+        'key': request['config']['api_key']
+    }
     for input_type in mispattributes['input']:
         if input_type in request:
             ip = request[input_type]
             break
     else:
-        misperrors['error'] = "Unsupported attributes type"
+        misperrors['error'] = "Unsupported attributes type."
         return misperrors
-    data = {'ip': ip}
+    response = requests.get(f'{greynoise_api_url}{ip}', headers=headers)  # Real request
-    r = requests.post(greynoise_api_url, data=data, headers={'user-agent': default_user_agent})  # Real request
+    if response.status_code == 200:  # OK (record found)
-    if r.status_code == 200:  # OK (record found)
+        return {'results': [{'types': mispattributes['output'], 'values': codes_mapping[response.json()['code']]}]}
-        response = r.text
+    # There is an error
-        if response:
+    errors = {
-            return {'results': [{'types': mispattributes['output'], 'values': response}]}
+        400: "Bad request.",
-    elif r.status_code == 404:  # Not found (not an error)
+        401: "Unauthorized. Please check your API key.",
-        return {'results': [{'types': mispattributes['output'], 'values': 'No data'}]}
+        429: "Too many requests. You've hit the rate-limit."
-    else:  # Real error
+    }
-        misperrors['error'] = 'GreyNoise API not accessible (HTTP ' + str(r.status_code) + ')'
+    try:
-        return misperrors['error']
+        misperrors['error'] = errors[response.status_code]
+    except KeyError:
+        misperrors['error'] = f'GreyNoise API not accessible (HTTP {response.status_code})'
+    return misperrors['error']
 
 
 def introspection():

misp_modules/modules/expansion/ipasn.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 
 import json
+from . import check_input_attribute, standard_error_message
 from pyipasnhistory import IPASNHistory
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 
@@ -34,11 +35,11 @@ def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
-    if request.get('attribute') and request['attribute'].get('type') in mispattributes['input']:
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
-        toquery = request['attribute']['value']
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
-    else:
+    if request['attribute']['type'] not in mispattributes['input']:
-        misperrors['error'] = "Unsupported attributes type"
+        return {'error': 'Unsupported attribute type.'}
-        return misperrors
+    toquery = request['attribute']['value']
 
     ipasn = IPASNHistory()
     values = ipasn.query(toquery)

misp_modules/modules/expansion/joesandbox_query.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 import jbxapi
 import json
+from . import check_input_attribute, checking_error, standard_error_message
 from joe_parser import JoeParser
 
 misperrors = {'error': 'Error'}
@@ -27,6 +28,10 @@ def handler(q=False):
     if not apikey:
         return {'error': 'No API key provided'}
 
+    if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
+        return {'error': f'{standard_error_message}, {checking_error} that is the link to the Joe Sandbox report.'}
+    if request['attribute']['type'] != 'link':
+        return {'error': 'Unsupported attribute type.'}
     url = request['attribute']['value']
     if "/submissions/" not in url:
         return {'error': "The URL does not point to a Joe Sandbox analysis."}

misp_modules/modules/expansion/lastline_query.py

@@ -3,8 +3,8 @@
 Module (type "expansion") to query a Lastline report from an analysis link.
 """
 import json
-
 import lastline_api
+from . import check_input_attribute, checking_error, standard_error_message
 
 
 misperrors = {
@@ -52,6 +52,8 @@ def handler(q=False):
     try:
         config = request["config"]
         auth_data = lastline_api.LastlineAbstractClient.get_login_params_from_dict(config)
+        if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
+            return {'error': f'{standard_error_message}, {checking_error} that is the link to a Lastline analysis.'}
         analysis_link = request['attribute']['value']
         # The API url changes based on the analysis link host name
         api_url = lastline_api.get_portal_url_from_task_link(analysis_link)

misp_modules/modules/expansion/malwarebazaar.py

@@ -1,5 +1,6 @@
 import json
 import requests
+from . import check_input_attribute, checking_error, standard_error_message
 from pymisp import MISPEvent, MISPObject
 
 mispattributes = {'input': ['md5', 'sha1', 'sha256'],
@@ -34,7 +35,11 @@ def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
+    if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
+        return {'error': f'{standard_error_message}, {checking_error} that is the hash to submit to Malware Bazaar.'}
     attribute = request['attribute']
+    if attribute['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     url = 'https://mb-api.abuse.ch/api/v1/'
     response = requests.post(url, data={'query': 'get_info', 'hash': attribute['value']}).json()
     query_status = response['query_status']

misp_modules/modules/expansion/ransomcoindb.py

@@ -1,4 +1,5 @@
 import json
+from . import check_input_attribute, checking_error, standard_error_message
 from ._ransomcoindb import ransomcoindb
 from pymisp import MISPObject
 
@@ -28,6 +29,10 @@ def handler(q=False):
     q = json.loads(q)
     if "config" not in q or "api-key" not in q["config"]:
         return {"error": "Ransomcoindb API key is missing"}
+    if not q.get('attribute') or not check_input_attribute(q['attribute'], requirements=('type', 'value')):
+        return {'error': f'{standard_error_message}, {checking_error}.'}
+    if q['attribute']['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     api_key = q["config"]["api-key"]
     r = {"results": []}
 

misp_modules/modules/expansion/recordedfuture.py

@@ -0,0 +1,396 @@
+import json
+import logging
+import requests
+from requests.exceptions import HTTPError, ProxyError,\
+    InvalidURL, ConnectTimeout, ConnectionError
+from . import check_input_attribute, checking_error, standard_error_message
+import platform
+import os
+from urllib.parse import quote, urlparse
+from pymisp import MISPAttribute, MISPEvent, MISPTag, MISPObject
+
+moduleinfo = {
+    'version': '1.0.1',
+    'author': 'Recorded Future',
+    'description': 'Module to retrieve data from Recorded Future',
+    'module-type': ['expansion', 'hover']
+}
+
+moduleconfig = ['token', 'proxy_host', 'proxy_port', 'proxy_username', 'proxy_password']
+
+misperrors = {'error': 'Error'}
+
+ATTRIBUTES = [
+    'ip',
+    'ip-src',
+    'ip-dst',
+    'domain',
+    'hostname',
+    'md5',
+    'sha1',
+    'sha256',
+    'uri',
+    'url',
+    'vulnerability',
+    'weakness'
+]
+
+mispattributes = {
+    'input': ATTRIBUTES,
+    'output': ATTRIBUTES + ['email-src', 'text'],
+    'format': 'misp_standard'
+}
+
+LOGGER = logging.getLogger('recorded_future')
+LOGGER.setLevel(logging.INFO)
+
+
+class RequestHandler:
+    """A class for handling any outbound requests from this module."""
+    def __init__(self):
+        self.session = requests.Session()
+        self.app_id = f'{os.path.basename(__file__)}/{moduleinfo["version"]} ({platform.platform()}) ' \
+                      f'misp_enrichment/{moduleinfo["version"]} python-requests/{requests.__version__}'
+        self.proxies = None
+        self.rf_token = None
+
+    def get(self, url: str, headers: dict = None) -> requests.Response:
+        """General get method with proxy error handling."""
+        try:
+            timeout = 7 if self.proxies else None
+            response = self.session.get(url, headers=headers, proxies=self.proxies, timeout=timeout)
+            response.raise_for_status()
+            return response
+        except (ConnectTimeout, ProxyError, InvalidURL) as error:
+            msg = 'Error connecting with proxy, please check the Recorded Future app proxy settings.'
+            LOGGER.error(f'{msg} Error: {error}')
+            misperrors['error'] = msg
+            raise
+
+    def rf_lookup(self, category: str, ioc: str) -> requests.Response:
+        """Do a lookup call using Recorded Future's ConnectAPI."""
+        parsed_ioc = quote(ioc, safe='')
+        url = f'https://api.recordedfuture.com/v2/{category}/{parsed_ioc}?fields=risk%2CrelatedEntities'
+        headers = {'X-RFToken': self.rf_token,
+                   'User-Agent': self.app_id}
+        try:
+            response = self.get(url, headers)
+        except HTTPError as error:
+            msg = f'Error when requesting data from Recorded Future. {error.response}: {error.response.reason}'
+            LOGGER.error(msg)
+            misperrors['error'] = msg
+            raise
+        return response
+
+
+GLOBAL_REQUEST_HANDLER = RequestHandler()
+
+
+class GalaxyFinder:
+    """A class for finding MISP galaxy matches to Recorded Future data."""
+    def __init__(self):
+        self.session = requests.Session()
+        self.sources = {
+            'RelatedThreatActor': [
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/threat-actor.json'
+            ],
+            'RelatedMalware': [
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/banker.json',
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/botnet.json',
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/exploit-kit.json',
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/rat.json',
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/ransomware.json',
+                'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/malpedia.json'
+            ]
+        }
+        self.galaxy_clusters = {}
+
+    def pull_galaxy_cluster(self, related_type: str) -> None:
+        """Fetches galaxy clusters for the related_type from the remote json files specified as self.sources."""
+        # Only fetch clusters if not fetched previously
+        if not self.galaxy_clusters.get(related_type):
+            for source in self.sources.get(related_type):
+                try:
+                    response = GLOBAL_REQUEST_HANDLER.get(source)
+                    name = source.split('/')[-1].split('.')[0]
+                    self.galaxy_clusters[related_type] = {name: response.json()}
+                except ConnectionError as error:
+                    LOGGER.warning(f'pull_galaxy_cluster failed for source: {source}, with error: {error}.')
+
+    def find_galaxy_match(self, indicator: str, related_type: str) -> str:
+        """Searches the clusters of the related_type for a match with the indicator.
+           :returns the first matching galaxy string or an empty string if no galaxy match is found.
+        """
+        self.pull_galaxy_cluster(related_type)
+        for cluster_name, cluster in self.galaxy_clusters.get(related_type, {}).items():
+            for value in cluster['values']:
+                try:
+                    if indicator in value['meta']['synonyms'] or indicator in value['value']:
+                        value = value['value']
+                        return f'misp-galaxy:{cluster_name}="{value}"'
+                except KeyError:
+                    pass
+        return ''
+
+
+class RFColors:
+    """Class for setting signature RF-colors."""
+    def __init__(self):
+        self.rf_white = '#CCCCCC'
+        self.rf_yellow = '#FFCE00'
+        self.rf_red = '#CF0A2C'
+
+    def riskscore_color(self, risk_score: int) -> str:
+        """Returns appropriate hex-colors according to risk score."""
+        risk_score = int(risk_score)
+        if risk_score < 25:
+            return self.rf_white
+        elif risk_score < 65:
+            return self.rf_yellow
+        else:
+            return self.rf_red
+
+    def riskrule_color(self, risk_rule_criticality: int) -> str:
+        """Returns appropriate hex-colors according to risk rule criticality."""
+        risk_rule_criticality = int(risk_rule_criticality)
+        if risk_rule_criticality == 1:
+            return self.rf_white
+        elif risk_rule_criticality == 2:
+            return self.rf_yellow
+        else:  # risk_rule_criticality == 3 or 4
+            return self.rf_red
+
+
+class RFEnricher:
+    """Class for enriching an attribute with data from Recorded Future.
+       The enrichment data is returned as a custom MISP object.
+    """
+    def __init__(self, attribute_props: dict):
+        self.event = MISPEvent()
+        self.enrichment_object = MISPObject('Recorded Future Enrichment')
+        description = (
+            'An object containing the enriched attribute and '
+            'related entities from Recorded Future.'
+        )
+        self.enrichment_object.from_dict(**{
+            'meta-category': 'misc',
+            'description': description,
+            'distribution': 0
+        })
+
+        # Create a copy of enriched attribute to add tags to
+        temp_attr = MISPAttribute()
+        temp_attr.from_dict(**attribute_props)
+        self.enriched_attribute = MISPAttribute()
+        self.enriched_attribute.from_dict(**{
+            'value': temp_attr.value,
+            'type': temp_attr.type,
+            'distribution': 0
+        })
+
+        self.related_attributes = []
+        self.color_picker = RFColors()
+        self.galaxy_finder = GalaxyFinder()
+
+        # Mapping from MISP-type to RF-type
+        self.type_to_rf_category = {
+            'ip': 'ip',
+            'ip-src': 'ip',
+            'ip-dst': 'ip',
+            'domain': 'domain',
+            'hostname': 'domain',
+            'md5': 'hash',
+            'sha1': 'hash',
+            'sha256': 'hash',
+            'uri': 'url',
+            'url': 'url',
+            'vulnerability': 'vulnerability',
+            'weakness': 'vulnerability'
+        }
+
+        # Related entities from RF portrayed as related attributes in MISP
+        self.related_attribute_types = [
+            'RelatedIpAddress', 'RelatedInternetDomainName', 'RelatedHash',
+            'RelatedEmailAddress', 'RelatedCyberVulnerability'
+        ]
+        # Related entities from RF portrayed as tags in MISP
+        self.galaxy_tag_types = ['RelatedMalware', 'RelatedThreatActor']
+
+    def enrich(self) -> None:
+        """Run the enrichment."""
+        category = self.type_to_rf_category.get(self.enriched_attribute.type)
+        json_response = GLOBAL_REQUEST_HANDLER.rf_lookup(category, self.enriched_attribute.value)
+        response = json.loads(json_response.content)
+
+        try:
+            # Add risk score and risk rules as tags to the enriched attribute
+            risk_score = response['data']['risk']['score']
+            hex_color = self.color_picker.riskscore_color(risk_score)
+            tag_name = f'recorded-future:risk-score="{risk_score}"'
+            self.add_tag(tag_name, hex_color)
+            for evidence in response['data']['risk']['evidenceDetails']:
+                risk_rule = evidence['rule']
+                criticality = evidence['criticality']
+                hex_color = self.color_picker.riskrule_color(criticality)
+                tag_name = f'recorded-future:risk-rule="{risk_rule}"'
+                self.add_tag(tag_name, hex_color)
+
+            # Retrieve related entities
+            for related_entity in response['data']['relatedEntities']:
+                related_type = related_entity['type']
+                if related_type in self.related_attribute_types:
+                    # Related entities returned as additional attributes
+                    for related in related_entity['entities']:
+                        if int(related["count"]) > 4:
+                            indicator = related['entity']['name']
+                            self.add_related_attribute(indicator, related_type)
+                elif related_type in self.galaxy_tag_types:
+                    # Related entities added as galaxy-tags to the enriched attribute
+                    galaxy_tags = []
+                    for related in related_entity['entities']:
+                        if int(related["count"]) > 4:
+                            indicator = related['entity']['name']
+                            galaxy = self.galaxy_finder.find_galaxy_match(indicator, related_type)
+                            # Handle deduplication of galaxy tags
+                            if galaxy and galaxy not in galaxy_tags:
+                                galaxy_tags.append(galaxy)
+                    for galaxy in galaxy_tags:
+                        self.add_tag(galaxy)
+        except KeyError:
+            misperrors['error'] = 'Unexpected format in Recorded Future api response.'
+            raise
+
+    def add_related_attribute(self, indicator: str, related_type: str) -> None:
+        """Helper method for adding an indicator to the related attribute list."""
+        out_type = self.get_output_type(related_type, indicator)
+        attribute = MISPAttribute()
+        attribute.from_dict(**{'value': indicator, 'type': out_type, 'distribution': 0})
+        self.related_attributes.append((related_type, attribute))
+
+    def add_tag(self, tag_name: str, hex_color: str = None) -> None:
+        """Helper method for adding a tag to the enriched attribute."""
+        tag = MISPTag()
+        tag_properties = {'name': tag_name}
+        if hex_color:
+            tag_properties['colour'] = hex_color
+        tag.from_dict(**tag_properties)
+        self.enriched_attribute.add_tag(tag)
+
+    def get_output_type(self, related_type: str, indicator: str) -> str:
+        """Helper method for translating a Recorded Future related type to a MISP output type."""
+        output_type = 'text'
+        if related_type == 'RelatedIpAddress':
+            output_type = 'ip-dst'
+        elif related_type == 'RelatedInternetDomainName':
+            output_type = 'domain'
+        elif related_type == 'RelatedHash':
+            hash_len = len(indicator)
+            if hash_len == 64:
+                output_type = 'sha256'
+            elif hash_len == 40:
+                output_type = 'sha1'
+            elif hash_len == 32:
+                output_type = 'md5'
+        elif related_type == 'RelatedEmailAddress':
+            output_type = 'email-src'
+        elif related_type == 'RelatedCyberVulnerability':
+            signature = indicator.split('-')[0]
+            if signature == 'CVE':
+                output_type = 'vulnerability'
+            elif signature == 'CWE':
+                output_type = 'weakness'
+        return output_type
+
+    def get_results(self) -> dict:
+        """Build and return the enrichment results."""
+        self.enrichment_object.add_attribute('Enriched attribute', **self.enriched_attribute)
+        for related_type, attribute in self.related_attributes:
+            self.enrichment_object.add_attribute(related_type, **attribute)
+        self.event.add_object(**self.enrichment_object)
+        event = json.loads(self.event.to_json())
+        result = {key: event[key] for key in ['Object'] if key in event}
+        return {'results': result}
+
+
+def get_proxy_settings(config: dict) -> dict:
+    """Returns proxy settings in the requests format.
+       If no proxy settings are set, return None."""
+    proxies = None
+    host = config.get('proxy_host')
+    port = config.get('proxy_port')
+    username = config.get('proxy_username')
+    password = config.get('proxy_password')
+
+    if host:
+        if not port:
+            misperrors['error'] = 'The recordedfuture_proxy_host config is set, ' \
+                                  'please also set the recordedfuture_proxy_port.'
+            raise KeyError
+        parsed = urlparse(host)
+        if 'http' in parsed.scheme:
+            scheme = 'http'
+        else:
+            scheme = parsed.scheme
+        netloc = parsed.netloc
+        host = f'{netloc}:{port}'
+
+        if username:
+            if not password:
+                misperrors['error'] = 'The recordedfuture_proxy_username config is set, ' \
+                                      'please also set the recordedfuture_proxy_password.'
+                raise KeyError
+            auth = f'{username}:{password}'
+            host = auth + '@' + host
+
+        proxies = {
+            'http': f'{scheme}://{host}',
+            'https': f'{scheme}://{host}'
+        }
+
+    LOGGER.info(f'Proxy settings: {proxies}')
+    return proxies
+
+
+def handler(q=False):
+    """Handle enrichment."""
+    if q is False:
+        return False
+    request = json.loads(q)
+
+    config = request.get('config')
+    if config and config.get('token'):
+        GLOBAL_REQUEST_HANDLER.rf_token = config.get('token')
+    else:
+        misperrors['error'] = 'Missing Recorded Future token.'
+        return misperrors
+    if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
+        return {'error': f'{standard_error_message}, {checking_error}.'}
+    if request['attribute']['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
+
+    try:
+        GLOBAL_REQUEST_HANDLER.proxies = get_proxy_settings(config)
+    except KeyError:
+        return misperrors
+
+    input_attribute = request.get('attribute')
+    rf_enricher = RFEnricher(input_attribute)
+
+    try:
+        rf_enricher.enrich()
+    except (HTTPError, ConnectTimeout, ProxyError, InvalidURL, KeyError):
+        return misperrors
+
+    return rf_enricher.get_results()
+
+
+def introspection():
+    """Returns a dict of the supported attributes."""
+    return mispattributes
+
+
+def version():
+    """Returns a dict with the version and the associated meta-data
+    including potential configurations required of the module."""
+    moduleinfo['config'] = moduleconfig
+    return moduleinfo

misp_modules/modules/expansion/shodan.py

@@ -5,38 +5,225 @@ try:
     import shodan
 except ImportError:
     print("shodan module not installed.")
+from . import check_input_attribute, standard_error_message
+from datetime import datetime
+from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
-mispattributes = {'input': ['ip-src', 'ip-dst'], 'output': ['freetext']}
+mispattributes = {'input': ['ip-src', 'ip-dst'],
-moduleinfo = {'version': '0.1', 'author': 'Raphaël Vinot',
+                  'format': 'misp_standard'}
+moduleinfo = {'version': '0.2', 'author': 'Raphaël Vinot',
               'description': 'Query on Shodan',
               'module-type': ['expansion']}
 
 moduleconfig = ['apikey']
 
 
+class ShodanParser():
+    def __init__(self, attribute):
+        self.misp_event = MISPEvent()
+        self.attribute = MISPAttribute()
+        self.attribute.from_dict(**attribute)
+        self.misp_event.add_attribute(**self.attribute)
+        self.ip_address_mapping = {
+            'asn': {'type': 'AS', 'object_relation': 'asn'},
+            'city': {'type': 'text', 'object_relation': 'city'},
+            'country_code': {'type': 'text', 'object_relation': 'country-code'},
+            'country_name': {'type': 'text', 'object_relation': 'country'},
+            'isp': {'type': 'text', 'object_relation': 'ISP'},
+            'latitude': {'type': 'float', 'object_relation': 'latitude'},
+            'longitude': {'type': 'float', 'object_relation': 'longitude'},
+            'org': {'type': 'text', 'object_relation': 'organization'},
+            'postal_code': {'type': 'text', 'object_relation': 'zipcode'},
+            'region_code': {'type': 'text', 'object_relation': 'region-code'}
+        }
+        self.ip_port_mapping = {
+            'domains': {'type': 'domain', 'object_relation': 'domain'},
+            'hostnames': {'type': 'hostname', 'object_relation': 'hostname'}
+        }
+        self.vulnerability_mapping = {
+            'cvss': {'type': 'float', 'object_relation': 'cvss-score'},
+            'summary': {'type': 'text', 'object_relation': 'summary'}
+        }
+        self.x509_mapping = {
+            'bits': {'type': 'text', 'object_relation': 'pubkey-info-size'},
+            'expires': {'type': 'datetime', 'object_relation': 'validity-not-after'},
+            'issued': {'type': 'datetime', 'object_relation': 'validity-not-before'},
+            'issuer': {'type': 'text', 'object_relation': 'issuer'},
+            'serial': {'type': 'text', 'object_relation': 'serial-number'},
+            'sig_alg': {'type': 'text', 'object_relation': 'signature_algorithm'},
+            'subject': {'type': 'text', 'object_relation': 'subject'},
+            'type': {'type': 'text', 'object_relation': 'pubkey-info-algorithm'},
+            'version': {'type': 'text', 'object_relation': 'version'}
+        }
+
+    def query_shodan(self, apikey):
+        # Query Shodan and get the results in a json blob
+        api = shodan.Shodan(apikey)
+        query_results = api.host(self.attribute.value)
+
+        # Parse the information about the IP address used as input
+        ip_address_attributes = []
+        for feature, mapping in self.ip_address_mapping.items():
+            if query_results.get(feature):
+                attribute = {'value': query_results[feature]}
+                attribute.update(mapping)
+                ip_address_attributes.append(attribute)
+        if ip_address_attributes:
+            ip_address_object = MISPObject('ip-api-address')
+            for attribute in ip_address_attributes:
+                ip_address_object.add_attribute(**attribute)
+            ip_address_object.add_attribute(**self._get_source_attribute())
+            ip_address_object.add_reference(self.attribute.uuid, 'describes')
+            self.misp_event.add_object(ip_address_object)
+
+        # Parse the hostnames / domains and ports associated with the IP address
+        if query_results.get('ports'):
+            ip_port_object = MISPObject('ip-port')
+            ip_port_object.add_attribute(**self._get_source_attribute())
+            feature = self.attribute.type.split('-')[1]
+            for port in query_results['ports']:
+                attribute = {
+                    'type': 'port',
+                    'object_relation': f'{feature}-port',
+                    'value': port
+                }
+                ip_port_object.add_attribute(**attribute)
+            for feature, mapping in self.ip_port_mapping.items():
+                for value in query_results.get(feature, []):
+                    attribute = {'value': value}
+                    attribute.update(mapping)
+                    ip_port_object.add_attribute(**attribute)
+            ip_port_object.add_reference(self.attribute.uuid, 'extends')
+            self.misp_event.add_object(ip_port_object)
+        else:
+            if any(query_results.get(feature) for feature in ('domains', 'hostnames')):
+                domain_ip_object = MISPObject('domain-ip')
+                domain_ip_object.add_attribute(**self._get_source_attribute())
+                for feature in ('domains', 'hostnames'):
+                    for value in query_results[feature]:
+                        attribute = {
+                            'type': 'domain',
+                            'object_relation': 'domain',
+                            'value': value
+                        }
+                        domain_ip_object.add_attribute(**attribute)
+                domain_ip_object.add_reference(self.attribute.uuid, 'extends')
+                self.misp_event.add_object(domain_ip_object)
+
+        # Parse data within the "data" field
+        if query_results.get('vulns'):
+            vulnerabilities = {}
+            for data in query_results['data']:
+                # Parse vulnerabilities
+                if data.get('vulns'):
+                    for cve, vulnerability in data['vulns'].items():
+                        if cve not in vulnerabilities:
+                            vulnerabilities[cve] = vulnerability
+                # Also parse the certificates
+                if data.get('ssl'):
+                    self._parse_cert(data['ssl'])
+            for cve, vulnerability in vulnerabilities.items():
+                vulnerability_object = MISPObject('vulnerability')
+                vulnerability_object.add_attribute(**{
+                    'type': 'vulnerability',
+                    'object_relation': 'id',
+                    'value': cve
+                })
+                for feature, mapping in self.vulnerability_mapping.items():
+                    if vulnerability.get(feature):
+                        attribute = {'value': vulnerability[feature]}
+                        attribute.update(mapping)
+                        vulnerability_object.add_attribute(**attribute)
+                if vulnerability.get('references'):
+                    for reference in vulnerability['references']:
+                        vulnerability_object.add_attribute(**{
+                            'type': 'link',
+                            'object_relation': 'references',
+                            'value': reference
+                        })
+                vulnerability_object.add_reference(self.attribute.uuid, 'vulnerability-of')
+                self.misp_event.add_object(vulnerability_object)
+            for cve_id in query_results['vulns']:
+                if cve_id not in vulnerabilities:
+                    attribute = {
+                        'type': 'vulnerability',
+                        'value': cve_id
+                    }
+                    self.misp_event.add_attribute(**attribute)
+        else:
+            # We have no vulnerability data, we only check if we have
+            # certificates within the "data" field
+            for data in query_results['data']:
+                if data.get('ssl'):
+                    self._parse_cert(data['ssl']['cert'])
+
+    def get_result(self):
+        event = json.loads(self.misp_event.to_json())
+        results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
+        return {'results': results}
+
+    # When we want to add the IP address information in objects such as the
+    # domain-ip or ip-port objects referencing the input IP address attribute
+    def _get_source_attribute(self):
+        return {
+            'type': self.attribute.type,
+            'object_relation': self.attribute.type,
+            'value': self.attribute.value
+        }
+
+    def _parse_cert(self, certificate):
+        x509_object = MISPObject('x509')
+        for feature in ('serial', 'sig_alg', 'version'):
+            if certificate.get(feature):
+                attribute = {'value': certificate[feature]}
+                attribute.update(self.x509_mapping[feature])
+                x509_object.add_attribute(**attribute)
+        # Parse issuer and subject value
+        for feature in ('issuer', 'subject'):
+            if certificate.get(feature):
+                attribute_value = (f'{identifier}={value}' for identifier, value in certificate[feature].items())
+                attribute = {'value': f'/{"/".join(attribute_value)}'}
+                attribute.update(self.x509_mapping[feature])
+                x509_object.add_attribute(**attribute)
+        # Parse datetime attributes
+        for feature in ('expires', 'issued'):
+            if certificate.get(feature):
+                attribute = {'value': datetime.strptime(certificate[feature], '%Y%m%d%H%M%SZ')}
+                attribute.update(self.x509_mapping[feature])
+                x509_object.add_attribute(**attribute)
+        # Parse fingerprints
+        if certificate.get('fingerprint'):
+            for hash_type, hash_value in certificate['fingerprint'].items():
+                x509_object.add_attribute(**{
+                    'type': f'x509-fingerprint-{hash_type}',
+                    'object_relation': f'x509-fingerprint-{hash_type}',
+                    'value': hash_value
+                })
+        # Parse public key related info
+        if certificate.get('pubkey'):
+            for feature, value in certificate['pubkey'].items():
+                attribute = {'value': value}
+                attribute.update(self.x509_mapping[feature])
+                x509_object.add_attribute(**attribute)
+        x509_object.add_reference(self.attribute.uuid, 'identifies')
+        self.misp_event.add_object(x509_object)
+
+
 def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
-    if request.get('ip-src'):
+    if not request.get('config', {}).get('apikey'):
-        toquery = request['ip-src']
+        return {'error': 'Shodan authentication is missing'}
-    elif request.get('ip-dst'):
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
-        toquery = request['ip-dst']
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
-    else:
+    attribute = request['attribute']
-        misperrors['error'] = "Unsupported attributes type"
+    if attribute['type'] not in mispattributes['input']:
-        return misperrors
+        return {'error': 'Unsupported attribute type.'}
-
+    shodan_parser = ShodanParser(attribute)
-    if not request.get('config') or not request['config'].get('apikey'):
+    shodan_parser.query_shodan(request['config']['apikey'])
-        misperrors['error'] = 'Shodan authentication is missing'
+    return shodan_parser.get_result()
-        return misperrors
-    api = shodan.Shodan(request['config'].get('apikey'))
-
-    return handle_expansion(api, toquery)
-
-
-def handle_expansion(api, domain):
-    return {'results': [{'types': mispattributes['output'], 'values': json.dumps(api.host(domain))}]}
 
 
 def introspection():

misp_modules/modules/expansion/sophoslabs_intelix.py

@@ -1,7 +1,8 @@
-from pymisp import MISPEvent, MISPObject
 import json
 import requests
 import base64
+from . import check_input_attribute, checking_error, standard_error_message
+from pymisp import MISPEvent, MISPObject
 from urllib.parse import quote
 
 moduleinfo = {'version': '1.0',
@@ -105,13 +106,25 @@ def handler(q=False):
         misperrors['error'] = "Missing client_id or client_secret value for SOPHOSLabs Intelix. \
             It's free to sign up here https://aws.amazon.com/marketplace/pp/B07SLZPMCS."
         return misperrors
+    to_check = (('type', 'value'), ('type', 'value1'))
+    if not j.get('attribute') or not any(check_input_attribute(j['attribute'], requirements=check) for check in to_check):
+        return {'error': f'{standard_error_message}, {checking_error}.'}
+    attribute = j['attribute']
+    if attribute['type'] not in misp_types_in:
+        return {'error': 'Unsupported attribute type.'}
     client = SophosLabsApi(j['config']['client_id'], j['config']['client_secret'])
-    if j['attribute']['type'] == "sha256":
+    mapping = {
-        client.hash_lookup(j['attribute']['value1'])
+        'sha256': 'hash_lookup',
-    if j['attribute']['type'] in ['ip-dst', 'ip-src', 'ip']:
+        'ip-dst': 'ip_lookup',
-        client.ip_lookup(j["attribute"]["value1"])
+        'ip-src': 'ip_lookup',
-    if j['attribute']['type'] in ['uri', 'url', 'domain', 'hostname']:
+        'ip': 'ip_lookup',
-        client.url_lookup(j["attribute"]["value1"])
+        'uri': 'url_lookup',
+        'url': 'url_lookup',
+        'domain': 'url_lookup',
+        'hostname': 'url_lookup'
+    }
+    attribute_value = attribute['value'] if 'value' in attribute else attribute['value1']
+    getattr(client, mapping[attribute['type']])(attribute_value)
     return client.get_result()
 
 

misp_modules/modules/expansion/trustar_enrich.py

@@ -1,7 +1,11 @@
 import json
 import pymisp
+from base64 import b64encode
+from collections import OrderedDict
+from . import check_input_attribute, checking_error, standard_error_message
 from pymisp import MISPAttribute, MISPEvent, MISPObject
-from trustar import TruStar
+from trustar import TruStar, Indicator
+from urllib.parse import quote
 
 misperrors = {'error': "Error"}
 mispattributes = {
@@ -33,9 +37,13 @@ class TruSTARParser:
         'SHA256': "sha256"
     }
 
+    # Relevant fields from each TruSTAR endpoint
+    SUMMARY_FIELDS = ["severityLevel", "source", "score", "attributes"]
+    METADATA_FIELDS = ["sightings", "first_seen", "last_seen", "tags"]
+
     REPORT_BASE_URL = "https://station.trustar.co/constellation/reports/{}"
 
-    CLIENT_METATAG = "MISP-{}".format(pymisp.__version__)
+    CLIENT_METATAG = f"MISP-{pymisp.__version__}"
 
     def __init__(self, attribute, config):
         config['enclave_ids'] = config.get('enclave_ids', "").strip().split(',')
@@ -51,45 +59,111 @@ class TruSTARParser:
         """
         Returns the MISP Event enriched with TruSTAR indicator summary data.
         """
-        event = json.loads(self.misp_event.to_json())
+        try:
-        results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
+            event = json.loads(self.misp_event.to_json())
-        return {'results': results}
+            results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])}
+            return {'results': results}
+        except Exception as e:
+            misperrors['error'] += f" -- Encountered issue serializing enrichment data -- {e}"
+            return misperrors
 
-    def generate_trustar_links(self, entity_value):
+    def generate_trustar_link(self, entity_type, entity_value):
         """
-        Generates links to TruSTAR reports if they exist.
+        Generates link to TruSTAR report of entity.
 
+        :param entity_type: <str> Type of entity.
         :param entity_value: <str> Value of entity.
+        :return: <str> Link to indicator report in TruSTAR platform.
         """
-        report_links = list()
+        report_id = b64encode(quote(f"{entity_type}|{entity_value}").encode()).decode()
-        trustar_reports = self.ts_client.search_reports(entity_value)
-        for report in trustar_reports:
-            report_links.append(self.REPORT_BASE_URL.format(report.id))
 
-        return report_links
+        return self.REPORT_BASE_URL.format(report_id)
 
-    def parse_indicator_summary(self, summaries):
+    @staticmethod
+    def extract_tags(enrichment_report):
         """
-        Converts a response from the TruSTAR /1.3/indicators/summaries endpoint
+        Extracts tags from the enrichment report in order to add them
-        a MISP trustar_report object and adds the summary data and links as attributes.
+        to the TruSTAR MISP Object. Removes tags from report to avoid
+        redundancy.
 
-        :param summaries: <generator> A TruSTAR Python SDK Page.generator object for generating
+        :param: <OrderedDict> Enrichment data.
-                          indicator summaries pages.
+        :return: <list> List of tags.
+        """
+        if enrichment_report and enrichment_report.get('tags'):
+            return [tag.get('name') for tag in enrichment_report.pop('tags')]
+        return None
+
+    def generate_enrichment_report(self, summary, metadata):
+        """
+        Extracts desired fields from summary and metadata reports and
+        generates an enrichment report.
+
+        :param summary: <trustar.IndicatorSummary> Indicator summary report.
+        :param metadata: <trustar.Indicator> Indicator metadata report.
+        :return: <str> Enrichment report.
+        """
+        # Preserve order of fields as they exist in SUMMARY_FIELDS and METADATA_FIELDS
+        enrichment_report = OrderedDict()
+
+        if summary:
+            summary_dict = summary.to_dict()
+            enrichment_report.update(
+                {field: summary_dict[field] for field in self.SUMMARY_FIELDS if summary_dict.get(field)})
+
+        if metadata:
+            metadata_dict = metadata.to_dict()
+            enrichment_report.update(
+                {field: metadata_dict[field] for field in self.METADATA_FIELDS if metadata_dict.get(field)})
+
+        return enrichment_report
+
+    def parse_indicator_summary(self, indicator, summary, metadata):
+        """
+        Pulls enrichment data from the TruSTAR /indicators/summaries and /indicators/metadata endpoints
+        and creates a MISP trustar_report.
+
+        :param indicator: <str> Value of the attribute
+        :summary: <trustar.IndicatorSummary> Indicator summary response object.
+        :metadata: <trustar.Indicator> Indicator response object.
         """
 
-        for summary in summaries:
+        # Verify that the indicator type is supported by TruSTAR
-            trustar_obj = MISPObject('trustar_report')
+        if summary and summary.indicator_type in self.ENTITY_TYPE_MAPPINGS:
             indicator_type = summary.indicator_type
-            indicator_value = summary.value
+        elif metadata and metadata.type in self.ENTITY_TYPE_MAPPINGS:
-            if indicator_type in self.ENTITY_TYPE_MAPPINGS:
+            indicator_type = metadata.type
+        else:
+            misperrors['error'] += " -- Attribute not found or not supported"
+            raise Exception
+
+        try:
+            # Extract most relevant fields from indicator summary and metadata responses
+            enrichment_report = self.generate_enrichment_report(summary, metadata)
+            tags = self.extract_tags(enrichment_report)
+
+            if enrichment_report:
+                # Create MISP trustar_report object and populate it with enrichment data
+                trustar_obj = MISPObject('trustar_report')
                 trustar_obj.add_attribute(indicator_type, attribute_type=self.ENTITY_TYPE_MAPPINGS[indicator_type],
-                                          value=indicator_value)
+                                          value=indicator)
                 trustar_obj.add_attribute("INDICATOR_SUMMARY", attribute_type="text",
-                                          value=json.dumps(summary.to_dict(), sort_keys=True, indent=4))
+                                          value=json.dumps(enrichment_report, indent=4))
-                report_links = self.generate_trustar_links(indicator_value)
+
-                for link in report_links:
+                report_link = self.generate_trustar_link(indicator_type, indicator)
-                    trustar_obj.add_attribute("REPORT_LINK", attribute_type="link", value=link)
+                trustar_obj.add_attribute("REPORT_LINK", attribute_type="link", value=report_link)
+
                 self.misp_event.add_object(**trustar_obj)
+            elif not tags:
+                # If enrichment report is empty and there are no tags, nothing to add to attribute
+                raise Exception("No relevant data found")
+
+            if tags:
+                for tag in tags:
+                    self.misp_event.add_attribute_tag(tag, indicator)
+
+        except Exception as e:
+            misperrors['error'] += f" -- Error enriching attribute {indicator} -- {e}"
+            raise e
 
 
 def handler(q=False):
@@ -110,17 +184,31 @@ def handler(q=False):
         misperrors['error'] = "Your TruSTAR API key and secret are required for indicator enrichment."
         return misperrors
 
+    if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')):
+        return {'error': f'{standard_error_message}, {checking_error}.'}
     attribute = request['attribute']
+    if attribute['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     trustar_parser = TruSTARParser(attribute, config)
+    metadata = None
+    summary = None
 
     try:
-        summaries = list(
+        metadata = trustar_parser.ts_client.get_indicators_metadata([Indicator(value=attribute['value'])])[0]
-            trustar_parser.ts_client.get_indicator_summaries([attribute['value']], page_size=MAX_PAGE_SIZE))
     except Exception as e:
-        misperrors['error'] = "Unable to retrieve TruSTAR summary data: {}".format(e)
+        misperrors['error'] += f" -- Could not retrieve indicator metadata from TruSTAR {e}"
+
+    try:
+        summary = list(
+            trustar_parser.ts_client.get_indicator_summaries([attribute['value']], page_size=MAX_PAGE_SIZE))[0]
+    except Exception as e:
+        misperrors['error'] += f" -- Unable to retrieve TruSTAR summary data: {e}"
+
+    try:
+        trustar_parser.parse_indicator_summary(attribute['value'], summary, metadata)
+    except Exception:
         return misperrors
 
-    trustar_parser.parse_indicator_summary(summaries)
     return trustar_parser.get_results()
 
 

misp_modules/modules/expansion/urlhaus.py

@@ -1,6 +1,8 @@
-from pymisp import MISPAttribute, MISPEvent, MISPObject
+# -*- coding: utf-8 -*-
 import json
 import requests
+from . import check_input_attribute, standard_error_message
+from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
 mispattributes = {'input': ['domain', 'hostname', 'ip-src', 'ip-dst', 'md5', 'sha256', 'url'],
@@ -134,7 +136,11 @@ def handler(q=False):
     if q is False:
         return False
     request = json.loads(q)
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
     attribute = request['attribute']
+    if attribute['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     urlhaus_parser = _misp_type_mapping[attribute['type']](attribute)
     return urlhaus_parser.query_api()
 

misp_modules/modules/expansion/virustotal.py

@@ -1,6 +1,7 @@
-from pymisp import MISPAttribute, MISPEvent, MISPObject
 import json
 import requests
+from . import check_input_attribute, standard_error_message
+from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
 mispattributes = {'input': ['hostname', 'domain', "ip-src", "ip-dst", "md5", "sha1", "sha256", "url"],
@@ -143,7 +144,7 @@ class VirusTotalParser(object):
 
     def parse_resolutions(self, resolutions, subdomains=None, uuids=None):
         domain_ip_object = MISPObject('domain-ip')
-        if self.attribute.type == 'domain':
+        if self.attribute.type in ('domain', 'hostname'):
             domain_ip_object.add_attribute('domain', type='domain', value=self.attribute.value)
             attribute_type, relation, key = ('ip-dst', 'ip', 'ip_address')
         else:
@@ -174,7 +175,7 @@ class VirusTotalParser(object):
             vt_object = MISPObject('virustotal-report')
             vt_object.add_attribute('permalink', type='link', value=query_result['permalink'])
             detection_ratio = '{}/{}'.format(query_result['positives'], query_result['total'])
-            vt_object.add_attribute('detection-ratio', type='text', value=detection_ratio)
+            vt_object.add_attribute('detection-ratio', type='text', value=detection_ratio, disable_correlation=True)
             self.misp_event.add_object(**vt_object)
             return vt_object.uuid
 
@@ -195,6 +196,11 @@ def handler(q=False):
     if not request.get('config') or not request['config'].get('apikey'):
         misperrors['error'] = "A VirusTotal api key is required for this module."
         return misperrors
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
+    if request['attribute']['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
+
     event_limit = request['config'].get('event_limit')
     if not isinstance(event_limit, int):
         event_limit = 5

misp_modules/modules/expansion/virustotal_public.py

@@ -1,6 +1,7 @@
-from pymisp import MISPAttribute, MISPEvent, MISPObject
 import json
 import requests
+from . import check_input_attribute, standard_error_message
+from pymisp import MISPAttribute, MISPEvent, MISPObject
 
 misperrors = {'error': 'Error'}
 mispattributes = {'input': ['hostname', 'domain', "ip-src", "ip-dst", "md5", "sha1", "sha256", "url"],
@@ -36,7 +37,7 @@ class VirusTotalParser():
 
     def parse_resolutions(self, resolutions, subdomains=None, uuids=None):
         domain_ip_object = MISPObject('domain-ip')
-        if self.attribute.type == 'domain':
+        if self.attribute.type in ('domain', 'hostname'):
             domain_ip_object.add_attribute('domain', type='domain', value=self.attribute.value)
             attribute_type, relation, key = ('ip-dst', 'ip', 'ip_address')
         else:
@@ -174,7 +175,11 @@ def handler(q=False):
     if not request.get('config') or not request['config'].get('apikey'):
         misperrors['error'] = "A VirusTotal api key is required for this module."
         return misperrors
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'}
     attribute = request['attribute']
+    if attribute['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     query_type, to_call = misp_type_mapping[attribute['type']]
     parser = to_call(request['config']['apikey'], attribute)
     query_result = parser.get_query_result(query_type)

misp_modules/modules/expansion/xforceexchange.py

@@ -1,6 +1,7 @@
 import requests
 import json
 import sys
+from . import check_input_attribute, standard_error_message
 from collections import defaultdict
 from pymisp import MISPAttribute, MISPEvent, MISPObject
 from requests.auth import HTTPBasicAuth
@@ -160,6 +161,10 @@ def handler(q=False):
         return misperrors
     key = request["config"]["apikey"]
     password = request['config']['apipassword']
+    if not request.get('attribute') or not check_input_attribute(request['attribute']):
+        return {'error': f'{standard_error_message} which should contain at least a type, a value and an uuid.'}
+    if request['attribute']['type'] not in mispattributes['input']:
+        return {'error': 'Unsupported attribute type.'}
     parser = XforceExchange(request['attribute'], key, password)
     parser.parse()
     return parser.get_result()

misp_modules/modules/import_mod/email_import.py

@@ -42,7 +42,7 @@ def handler(q=False):
     # request data is always base 64 byte encoded
     data = base64.b64decode(request["data"])
 
-    email_object = EMailObject(pseudofile=BytesIO(data), attach_original_mail=True, standalone=False)
+    email_object = EMailObject(pseudofile=BytesIO(data), attach_original_email=True, standalone=False)
 
     # Check if we were given a configuration
     config = request.get("config", {})

tests/test_expansions.py

@@ -97,9 +97,16 @@ class TestExpansions(unittest.TestCase):
             self.assertEqual(self.get_errors(response), 'An API key for APIVoid is required.')
 
     def test_bgpranking(self):
-        query = {"module": "bgpranking", "AS": "13335"}
+        query = {
+            "module": "bgpranking",
+            "attribute": {
+                "type": "AS",
+                "value": "13335",
+                "uuid": "ea89a33b-4ab7-4515-9f02-922a0bee333d"
+            }
+        }
         response = self.misp_modules_post(query)
-        self.assertEqual(self.get_values(response)['response']['asn_description'], 'CLOUDFLARENET, US')
+        self.assertEqual(self.get_object(response), 'asn')
 
     def test_btc_steroids(self):
         query = {"module": "btc_steroids", "btc": "1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA"}
@@ -229,11 +236,24 @@ class TestExpansions(unittest.TestCase):
         self.assertEqual(to_check, 'OK (Not Found)', response)
 
     def test_greynoise(self):
-        query = {"module": "greynoise", "ip-dst": "1.1.1.1"}
+        module_name = 'greynoise'
-        response = self.misp_modules_post(query)
+        query = {"module": module_name, "ip-dst": "1.1.1.1"}
-        value = self.get_values(response)
+        if module_name in self.configs:
-        if value != 'GreyNoise API not accessible (HTTP 429)':
+            query['config'] = self.configs[module_name]
-            self.assertTrue(value.startswith('{"ip":"1.1.1.1","status":"ok"'))
+            response = self.misp_modules_post(query)
+            try:
+                self.assertEqual(self.get_values(response), 'This IP is commonly spoofed in Internet-scan activity')
+            except Exception:
+                self.assertIn(
+                    self.get_errors(reponse),
+                    (
+                        "Unauthorized. Please check your API key.",
+                        "Too many requests. You've hit the rate-limit."
+                    )
+                )
+        else:
+            response = self.misp_modules_post(query)
+            self.assertEqual(self.get_errors(response), 'Missing Greynoise API key.')
 
     def test_ipasn(self):
         query = {"module": "ipasn",