diff --git a/README.md b/README.md index 67573da..26dce03 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj * [AssemblyLine submit](misp_modules/modules/expansion/assemblyline_submit.py) - an expansion module to submit samples and urls to AssemblyLine. * [AssemblyLine query](misp_modules/modules/expansion/assemblyline_query.py) - an expansion module to query AssemblyLine and parse the full submission report. * [Backscatter.io](misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations. -* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description, its history, and position in BGP Ranking. +* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description and its ranking and position in BGP Ranking. * [RansomcoinDB check](misp_modules/modules/expansion/ransomcoindb.py) - An expansion hover module to query the [ransomcoinDB](https://ransomcoindb.concinnity-risks.com): it contains mapping between BTC addresses and malware hashes. Enrich MISP by querying for BTC -> hash or hash -> BTC addresses. * [BTC scam check](misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused. * [BTC transactions](misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP. @@ -69,6 +69,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj * [pptx-enrich](misp_modules/modules/expansion/pptx_enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser). * [qrcode](misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values. * [rbl](misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute. +* [recordedfuture](misp_modules/modules/expansion/recordedfuture.py) - a hover and expansion module for enriching MISP attributes with threat intelligence from Recorded Future. * [reversedns](misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes. * [securitytrails](misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/). * [shodan](misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module. @@ -79,6 +80,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj * [STIX2 pattern syntax validator](misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax. * [ThreatCrowd](misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/). * [threatminer](misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/). +* [TruSTAR Enrich](misp_modules/modules/expansion/trustar_enrich.py) - an expansion module to enrich MISP data with [TruSTAR](https://www.trustar.co/). * [urlhaus](misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url. * [urlscan](misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io). * [virustotal](misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://developers.virustotal.com/reference)) diff --git a/doc/README.md b/doc/README.md index e173ad4..1225780 100644 --- a/doc/README.md +++ b/doc/README.md @@ -108,13 +108,13 @@ Query backscatter.io (https://backscatter.io/). Query BGP Ranking (https://bgpranking-ng.circl.lu/). - **features**: ->The module takes an AS number attribute as input and displays its description and history, and position in BGP Ranking. +>The module takes an AS number attribute as input and displays its description as well as its ranking position in BGP Ranking for a given day. > > - **input**: >Autonomous system number. - **output**: ->Text containing a description of the ASN, its history, and the position in BGP Ranking. +>An asn object with its related bgp-ranking object. - **references**: >https://github.com/D4-project/BGP-Ranking/ - **requirements**: @@ -311,6 +311,26 @@ An expansion hover module to expand information about CVE id. ----- +#### [cve_advanced](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/cve_advanced.py) + + + +An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE). +- **features**: +>The module takes a vulnerability attribute as input and queries the CIRCL CVE search API to gather additional information. +> +>The result of the query is then parsed to return additional information about the vulnerability, like its cvss score or some references, as well as the potential related weaknesses and attack patterns. +> +>The vulnerability additional data is returned in a vulnerability MISP object, and the related additional information are put into weakness and attack-pattern MISP objects. +- **input**: +>Vulnerability attribute. +- **output**: +>Additional information about the vulnerability, such as its cvss score, some references, or the related weaknesses and attack patterns. +- **references**: +>https://cve.circl.lu, https://cve/mitre.org/ + +----- + #### [cytomic_orion](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/cytomic_orion.py) @@ -369,7 +389,7 @@ A simple DNS expansion service to resolve IP address from domain MISP attributes ----- -#### [docx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/docx-enrich.py) +#### [docx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/docx_enrich.py) @@ -476,6 +496,42 @@ Module to access Farsight DNSDB Passive DNS. ----- +#### [geoip_asn](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_asn.py) + + +- **descrption**: +>An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about its related AS number. +- **features**: +>The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the related AS number. +- **input**: +>An IP address MISP attribute. +- **output**: +>Text containing information about the AS number of the IP address. +- **references**: +>https://www.maxmind.com/en/home +- **requirements**: +>A local copy of Maxmind's Geolite database + +----- + +#### [geoip_city](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_city.py) + + + +An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located. +- **features**: +>The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the city where this IP address is located. +- **input**: +>An IP address MISP attribute. +- **output**: +>Text containing information about the city where the IP address is located. +- **references**: +>https://www.maxmind.com/en/home +- **requirements**: +>A local copy of Maxmind's Geolite database + +----- + #### [geoip_country](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/geoip_country.py) @@ -496,19 +552,39 @@ Module to query a local copy of Maxmind's Geolite database. ----- +#### [google_search](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/google_search.py) + + +- **descrption**: +>A hover module to get information about an url using a Google search. +- **features**: +>The module takes an url as input to query the Google search API. The result of the query is then return as raw text. +- **input**: +>An url attribute. +- **output**: +>Text containing the result of a Google search on the input url. +- **references**: +>https://github.com/abenassi/Google-Search-API +- **requirements**: +>The python Google Search API library + +----- + #### [greynoise](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/greynoise.py) Module to access GreyNoise.io API - **features**: ->The module takes an IP address as input and queries Greynoise for some additional information about it. The result is returned as text. +>The module takes an IP address as input and queries Greynoise for some additional information about it: basically it checks whether a given IP address is “Internet background noise”, or has been observed scanning or attacking devices across the Internet. The result is returned as text. - **input**: >An IP address. - **output**: >Additional information about the IP fetched from Greynoise API. - **references**: >https://greynoise.io/, https://github.com/GreyNoise-Intelligence/api.greynoise.io +- **requirements**: +>A Greynoise API key. ----- @@ -542,6 +618,37 @@ Module to access haveibeenpwned.com API. ----- +#### [intel471](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/intel471.py) + + +- **descrption**: +>An expansion module to query Intel471 in order to get additional information about a domain, ip address, email address, url or hash. +- **features**: +>The module uses the Intel471 python library to query the Intel471 API with the value of the input attribute. The result of the query is then returned as freetext so the Freetext import parses it. +- **input**: +>A MISP attribute whose type is included in the following list: +>- hostname +>- domain +>- url +>- ip-src +>- ip-dst +>- email-src +>- email-dst +>- target-email +>- whois-registrant-email +>- whois-registrant-name +>- md5 +>- sha1 +>- sha256 +- **output**: +>Freetext +- **references**: +>https://public.intel471.com/ +- **requirements**: +>The intel471 python library + +----- + #### [intelmq_eventdb](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/intelmq_eventdb.py) @@ -731,7 +838,7 @@ Query the MALWAREbazaar API to get additional information about the input hash a ----- -#### [ocr-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ocr-enrich.py) +#### [ocr_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ocr_enrich.py) Module to process some optical character recognition on pictures. - **features**: @@ -745,7 +852,7 @@ Module to process some optical character recognition on pictures. ----- -#### [ods-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ods-enrich.py) +#### [ods_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ods_enrich.py) @@ -761,7 +868,7 @@ Module to extract freetext from a .ods document. ----- -#### [odt-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/odt-enrich.py) +#### [odt_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/odt_enrich.py) @@ -900,7 +1007,7 @@ Module to get information from AlienVault OTX. ----- -#### [pdf-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pdf-enrich.py) +#### [pdf_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pdf_enrich.py) @@ -916,7 +1023,7 @@ Module to extract freetext from a PDF document. ----- -#### [pptx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pptx-enrich.py) +#### [pptx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/pptx_enrich.py) @@ -946,6 +1053,24 @@ Module to decode QR codes. ----- +#### [ransomcoindb](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/ransomcoindb.py) +- **descrption**: +>Module to access the ransomcoinDB with a hash or btc address attribute and get the associated btc address of hashes. +- **features**: +>The module takes either a hash attribute or a btc attribute as input to query the ransomcoinDB API for some additional data. +> +>If the input is a btc address, we will get the associated hashes returned in a file MISP object. If we query ransomcoinDB with a hash, the response contains the associated btc addresses returned as single MISP btc attributes. +- **input**: +>A hash (md5, sha1 or sha256) or btc attribute. +- **output**: +>Hashes associated to a btc address or btc addresses associated to a hash. +- **references**: +>https://ransomcoindb.concinnity-risks.com +- **requirements**: +>A ransomcoinDB API key. + +----- + #### [rbl](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/rbl.py) Module to check an IPv4 address against known RBLs. @@ -964,6 +1089,24 @@ Module to check an IPv4 address against known RBLs. ----- +#### [recordedfuture](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/recordedfuture.py) + + + +Module to enrich attributes with threat intelligence from Recorded Future. +- **features**: +>Enrich an attribute to add a custom enrichment object to the event. The object contains a copy of the enriched attribute with added tags presenting risk score and triggered risk rules from Recorded Future. Malware and Threat Actors related to the enriched indicator in Recorded Future is matched against MISP's galaxy clusters and applied as galaxy tags. The custom enrichment object also includes a list of related indicators from Recorded Future (IP's, domains, hashes, URL's and vulnerabilities) added as additional attributes. +- **input**: +>A MISP attribute of one of the following types: ip, ip-src, ip-dst, domain, hostname, md5, sha1, sha256, uri, url, vulnerability, weakness. +- **output**: +>A MISP object containing a copy of the enriched attribute with added tags from Recorded Future and a list of new attributes related to the enriched attribute. +- **references**: +>https://www.recordedfuture.com/ +- **requirements**: +>A Recorded Future API token. + +----- + #### [reversedns](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/reversedns.py) Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes. @@ -1071,6 +1214,24 @@ An expansion hover module to perform a syntax check on sigma rules. ----- +#### [sophoslabs_intelix](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/sophoslabs_intelix.py) + + + +An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute. +- **features**: +>The module takes an ip address, url, domain or sha256 attribute and queries the SophosLabs Intelix API with the attribute value. The result of this query is a SophosLabs Intelix hash report, or an ip or url lookup, that is then parsed and returned in a MISP object. +- **input**: +>An ip address, url, domain or sha256 attribute. +- **output**: +>SophosLabs Intelix report and lookup objects +- **references**: +>https://aws.amazon.com/marketplace/pp/B07SLZPMCS +- **requirements**: +>A client_id and client_secret pair to authenticate to the SophosLabs Intelix API + +----- + #### [sourcecache](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/sourcecache.py) Module to cache web pages of analysis reports, OSINT sources. The module returns a link of the cached page. @@ -1422,7 +1583,7 @@ An expansion module for IBM X-Force Exchange. ----- -#### [xlsx-enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/xlsx-enrich.py) +#### [xlsx_enrich](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/xlsx_enrich.py) diff --git a/doc/expansion/bgpranking.json b/doc/expansion/bgpranking.json index a98b780..4695aa1 100644 --- a/doc/expansion/bgpranking.json +++ b/doc/expansion/bgpranking.json @@ -1,8 +1,8 @@ { "description": "Query BGP Ranking (https://bgpranking-ng.circl.lu/).", "requirements": ["pybgpranking python library"], - "features": "The module takes an AS number attribute as input and displays its description and history, and position in BGP Ranking.\n\n", + "features": "The module takes an AS number attribute as input and displays its description as well as its ranking position in BGP Ranking for a given day.\n\n", "references": ["https://github.com/D4-project/BGP-Ranking/"], "input": "Autonomous system number.", - "output": "Text containing a description of the ASN, its history, and the position in BGP Ranking." + "output": "An asn object with its related bgp-ranking object." } diff --git a/doc/expansion/cve_advanced.json b/doc/expansion/cve_advanced.json new file mode 100644 index 0000000..a4b2ac6 --- /dev/null +++ b/doc/expansion/cve_advanced.json @@ -0,0 +1,8 @@ +{ + "description": "An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).", + "logo": "logos/cve.png", + "input": "Vulnerability attribute.", + "output": "Additional information about the vulnerability, such as its cvss score, some references, or the related weaknesses and attack patterns.", + "references": ["https://cve.circl.lu", "https://cve/mitre.org/"], + "features": "The module takes a vulnerability attribute as input and queries the CIRCL CVE search API to gather additional information.\n\nThe result of the query is then parsed to return additional information about the vulnerability, like its cvss score or some references, as well as the potential related weaknesses and attack patterns.\n\nThe vulnerability additional data is returned in a vulnerability MISP object, and the related additional information are put into weakness and attack-pattern MISP objects." +} diff --git a/doc/expansion/docx-enrich.json b/doc/expansion/docx_enrich.json similarity index 100% rename from doc/expansion/docx-enrich.json rename to doc/expansion/docx_enrich.json diff --git a/doc/expansion/geoip_asn.json b/doc/expansion/geoip_asn.json new file mode 100644 index 0000000..98189c7 --- /dev/null +++ b/doc/expansion/geoip_asn.json @@ -0,0 +1,9 @@ +{ + "descrption": "An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about its related AS number.", + "logo": "logos/maxmind.png", + "requirements": ["A local copy of Maxmind's Geolite database"], + "input": "An IP address MISP attribute.", + "output": "Text containing information about the AS number of the IP address.", + "references": ["https://www.maxmind.com/en/home"], + "features": "The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the related AS number." +} diff --git a/doc/expansion/geoip_city.json b/doc/expansion/geoip_city.json new file mode 100644 index 0000000..bf6d8fa --- /dev/null +++ b/doc/expansion/geoip_city.json @@ -0,0 +1,9 @@ +{ + "description": "An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located.", + "logo": "logos/maxmind.png", + "requirements": ["A local copy of Maxmind's Geolite database"], + "input": "An IP address MISP attribute.", + "output": "Text containing information about the city where the IP address is located.", + "references": ["https://www.maxmind.com/en/home"], + "features": "The module takes an IP address attribute as input and queries a local copy of the Maxmind's Geolite database to get information about the city where this IP address is located." +} diff --git a/doc/expansion/google_search.json b/doc/expansion/google_search.json new file mode 100644 index 0000000..a3caddf --- /dev/null +++ b/doc/expansion/google_search.json @@ -0,0 +1,9 @@ +{ + "descrption": "A hover module to get information about an url using a Google search.", + "logo": "logos/google.png", + "requirements": ["The python Google Search API library"], + "input": "An url attribute.", + "output": "Text containing the result of a Google search on the input url.", + "references": ["https://github.com/abenassi/Google-Search-API"], + "features": "The module takes an url as input to query the Google search API. The result of the query is then return as raw text." +} diff --git a/doc/expansion/greynoise.json b/doc/expansion/greynoise.json index f1f1003..49ba481 100644 --- a/doc/expansion/greynoise.json +++ b/doc/expansion/greynoise.json @@ -1,9 +1,9 @@ { "description": "Module to access GreyNoise.io API", "logo": "logos/greynoise.png", - "requirements": [], + "requirements": ["A Greynoise API key."], "input": "An IP address.", "output": "Additional information about the IP fetched from Greynoise API.", "references": ["https://greynoise.io/", "https://github.com/GreyNoise-Intelligence/api.greynoise.io"], - "features": "The module takes an IP address as input and queries Greynoise for some additional information about it. The result is returned as text." + "features": "The module takes an IP address as input and queries Greynoise for some additional information about it: basically it checks whether a given IP address is “Internet background noise”, or has been observed scanning or attacking devices across the Internet. The result is returned as text." } diff --git a/doc/expansion/intel471.json b/doc/expansion/intel471.json new file mode 100644 index 0000000..72dbaba --- /dev/null +++ b/doc/expansion/intel471.json @@ -0,0 +1,9 @@ +{ + "descrption": "An expansion module to query Intel471 in order to get additional information about a domain, ip address, email address, url or hash.", + "logo": "logos/intel471.png", + "requirements": ["The intel471 python library"], + "input": "A MISP attribute whose type is included in the following list:\n- hostname\n- domain\n- url\n- ip-src\n- ip-dst\n- email-src\n- email-dst\n- target-email\n- whois-registrant-email\n- whois-registrant-name\n- md5\n- sha1\n- sha256", + "output": "Freetext", + "references": ["https://public.intel471.com/"], + "features": "The module uses the Intel471 python library to query the Intel471 API with the value of the input attribute. The result of the query is then returned as freetext so the Freetext import parses it." +} diff --git a/doc/expansion/ocr-enrich.json b/doc/expansion/ocr_enrich.json similarity index 100% rename from doc/expansion/ocr-enrich.json rename to doc/expansion/ocr_enrich.json diff --git a/doc/expansion/ods-enrich.json b/doc/expansion/ods_enrich.json similarity index 100% rename from doc/expansion/ods-enrich.json rename to doc/expansion/ods_enrich.json diff --git a/doc/expansion/odt-enrich.json b/doc/expansion/odt_enrich.json similarity index 100% rename from doc/expansion/odt-enrich.json rename to doc/expansion/odt_enrich.json diff --git a/doc/expansion/pdf-enrich.json b/doc/expansion/pdf_enrich.json similarity index 100% rename from doc/expansion/pdf-enrich.json rename to doc/expansion/pdf_enrich.json diff --git a/doc/expansion/pptx-enrich.json b/doc/expansion/pptx_enrich.json similarity index 100% rename from doc/expansion/pptx-enrich.json rename to doc/expansion/pptx_enrich.json diff --git a/doc/expansion/ransomcoindb.json b/doc/expansion/ransomcoindb.json new file mode 100644 index 0000000..bc4e2ab --- /dev/null +++ b/doc/expansion/ransomcoindb.json @@ -0,0 +1,8 @@ +{ + "descrption": "Module to access the ransomcoinDB with a hash or btc address attribute and get the associated btc address of hashes.", + "requirements": ["A ransomcoinDB API key."], + "input": "A hash (md5, sha1 or sha256) or btc attribute.", + "output": "Hashes associated to a btc address or btc addresses associated to a hash.", + "references": ["https://ransomcoindb.concinnity-risks.com"], + "features": "The module takes either a hash attribute or a btc attribute as input to query the ransomcoinDB API for some additional data.\n\nIf the input is a btc address, we will get the associated hashes returned in a file MISP object. If we query ransomcoinDB with a hash, the response contains the associated btc addresses returned as single MISP btc attributes." +} diff --git a/doc/expansion/recordedfuture.json b/doc/expansion/recordedfuture.json new file mode 100644 index 0000000..2fec7eb --- /dev/null +++ b/doc/expansion/recordedfuture.json @@ -0,0 +1,9 @@ +{ + "description": "Module to enrich attributes with threat intelligence from Recorded Future.", + "logo": "logos/recordedfuture.png", + "requirements": ["A Recorded Future API token."], + "input": "A MISP attribute of one of the following types: ip, ip-src, ip-dst, domain, hostname, md5, sha1, sha256, uri, url, vulnerability, weakness.", + "output": "A MISP object containing a copy of the enriched attribute with added tags from Recorded Future and a list of new attributes related to the enriched attribute.", + "references": ["https://www.recordedfuture.com/"], + "features": "Enrich an attribute to add a custom enrichment object to the event. The object contains a copy of the enriched attribute with added tags presenting risk score and triggered risk rules from Recorded Future. Malware and Threat Actors related to the enriched indicator in Recorded Future is matched against MISP's galaxy clusters and applied as galaxy tags. The custom enrichment object also includes a list of related indicators from Recorded Future (IP's, domains, hashes, URL's and vulnerabilities) added as additional attributes." +} diff --git a/doc/expansion/sophoslabs_intelix.json b/doc/expansion/sophoslabs_intelix.json new file mode 100644 index 0000000..18dd7c1 --- /dev/null +++ b/doc/expansion/sophoslabs_intelix.json @@ -0,0 +1,9 @@ +{ + "description": "An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute.", + "logo": "logos/sophoslabs_intelix.svg", + "requirements": ["A client_id and client_secret pair to authenticate to the SophosLabs Intelix API"], + "input": "An ip address, url, domain or sha256 attribute.", + "output": "SophosLabs Intelix report and lookup objects", + "references": ["https://aws.amazon.com/marketplace/pp/B07SLZPMCS"], + "features": "The module takes an ip address, url, domain or sha256 attribute and queries the SophosLabs Intelix API with the attribute value. The result of this query is a SophosLabs Intelix hash report, or an ip or url lookup, that is then parsed and returned in a MISP object." +} diff --git a/doc/expansion/xlsx-enrich.json b/doc/expansion/xlsx_enrich.json similarity index 100% rename from doc/expansion/xlsx-enrich.json rename to doc/expansion/xlsx_enrich.json diff --git a/doc/logos/google.png b/doc/logos/google.png new file mode 100644 index 0000000..492f44c Binary files /dev/null and b/doc/logos/google.png differ diff --git a/doc/logos/intel471.png b/doc/logos/intel471.png new file mode 100644 index 0000000..08264e9 Binary files /dev/null and b/doc/logos/intel471.png differ diff --git a/doc/logos/recordedfuture.png b/doc/logos/recordedfuture.png new file mode 100644 index 0000000..a208c04 Binary files /dev/null and b/doc/logos/recordedfuture.png differ diff --git a/doc/logos/sophoslabs_intelix.svg b/doc/logos/sophoslabs_intelix.svg new file mode 100644 index 0000000..9fe952f --- /dev/null +++ b/doc/logos/sophoslabs_intelix.svg @@ -0,0 +1,32 @@ + + + + CC812F0D-F9F0-4D68-9347-3579CDA181A3 + Created with sketchtool. + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/misp_modules/modules/expansion/__init__.py b/misp_modules/modules/expansion/__init__.py index dbd3473..1b6d2bb 100644 --- a/misp_modules/modules/expansion/__init__.py +++ b/misp_modules/modules/expansion/__init__.py @@ -18,4 +18,14 @@ __all__ = ['cuckoo_submit', 'vmray_submit', 'bgpranking', 'circl_passivedns', 'c 'virustotal_public', 'apiosintds', 'urlscan', 'securitytrails', 'apivoid', 'assemblyline_submit', 'assemblyline_query', 'ransomcoindb', 'malwarebazaar', 'lastline_query', 'lastline_submit', 'sophoslabs_intelix', 'cytomic_orion', 'censys_enrich', - 'trustar_enrich'] + 'trustar_enrich', 'recordedfuture'] + + +minimum_required_fields = ('type', 'uuid', 'value') + +checking_error = 'containing at least a "type" field and a "value" field' +standard_error_message = 'This module requires an "attribute" field as input' + + +def check_input_attribute(attribute, requirements=minimum_required_fields): + return all(feature in attribute for feature in requirements) diff --git a/misp_modules/modules/expansion/apivoid.py b/misp_modules/modules/expansion/apivoid.py index 5d6395e..a71b5e6 100755 --- a/misp_modules/modules/expansion/apivoid.py +++ b/misp_modules/modules/expansion/apivoid.py @@ -1,5 +1,6 @@ import json import requests +from . import check_input_attribute, standard_error_message from pymisp import MISPAttribute, MISPEvent, MISPObject misperrors = {'error': 'Error'} @@ -74,7 +75,11 @@ def handler(q=False): request = json.loads(q) if not request.get('config', {}).get('apikey'): return {'error': 'An API key for APIVoid is required.'} - attribute = request.get('attribute') + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + attribute = request['attribute'] + if attribute['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} apikey = request['config']['apikey'] apivoid_parser = APIVoidParser(attribute) apivoid_parser.parse_domain(apikey) diff --git a/misp_modules/modules/expansion/assemblyline_query.py b/misp_modules/modules/expansion/assemblyline_query.py index 226e4dd..67fce45 100644 --- a/misp_modules/modules/expansion/assemblyline_query.py +++ b/misp_modules/modules/expansion/assemblyline_query.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import json +from . import check_input_attribute, standard_error_message from assemblyline_client import Client, ClientError from collections import defaultdict from pymisp import MISPAttribute, MISPEvent, MISPObject @@ -139,6 +140,10 @@ def handler(q=False): if q is False: return False request = json.loads(q) + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + if request['attribute']['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} if not request.get('config'): return {"error": "Missing configuration."} if not request['config'].get('apiurl'): diff --git a/misp_modules/modules/expansion/bgpranking.py b/misp_modules/modules/expansion/bgpranking.py index b01088d..c021d62 100755 --- a/misp_modules/modules/expansion/bgpranking.py +++ b/misp_modules/modules/expansion/bgpranking.py @@ -1,13 +1,15 @@ # -*- coding: utf-8 -*- import json -from datetime import date, timedelta +from . import check_input_attribute, standard_error_message +from datetime import date, datetime, timedelta from pybgpranking import BGPRanking +from pymisp import MISPAttribute, MISPEvent, MISPObject misperrors = {'error': 'Error'} -mispattributes = {'input': ['AS'], 'output': ['freetext']} +mispattributes = {'input': ['AS'], 'format': 'misp_standard'} moduleinfo = {'version': '0.1', 'author': 'Raphaël Vinot', - 'description': 'Query an ASN Description history service (https://github.com/CIRCL/ASN-Description-History.git)', + 'description': 'Query BGP Ranking to get the ranking of an Autonomous System number.', 'module-type': ['expansion', 'hover']} @@ -15,19 +17,65 @@ def handler(q=False): if q is False: return False request = json.loads(q) - if request.get('AS'): - toquery = request['AS'] - else: - misperrors['error'] = "Unsupported attributes type" - return misperrors + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + toquery = request['attribute'] + if toquery['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} bgpranking = BGPRanking() - values = bgpranking.query(toquery, date=(date.today() - timedelta(1)).isoformat()) + value_toquery = int(toquery['value'][2:]) if toquery['value'].startswith('AS') else int(toquery['value']) + values = bgpranking.query(value_toquery, date=(date.today() - timedelta(1)).isoformat()) - if not values: - misperrors['error'] = 'Unable to find the ASN in BGP Ranking' + if not values['response'] or not values['response']['asn_description']: + misperrors['error'] = 'There is no result about this ASN in BGP Ranking' return misperrors - return {'results': [{'types': mispattributes['output'], 'values': values}]} + + event = MISPEvent() + attribute = MISPAttribute() + attribute.from_dict(**toquery) + event.add_attribute(**attribute) + + asn_object = MISPObject('asn') + asn_object.add_attribute(**{ + 'type': 'AS', + 'object_relation': 'asn', + 'value': values['meta']['asn'] + }) + description, country = values['response']['asn_description'].split(', ') + for relation, value in zip(('description', 'country'), (description, country)): + asn_object.add_attribute(**{ + 'type': 'text', + 'object_relation': relation, + 'value': value + }) + + mapping = { + 'address_family': {'type': 'text', 'object_relation': 'address-family'}, + 'date': {'type': 'datetime', 'object_relation': 'date'}, + 'position': {'type': 'float', 'object_relation': 'position'}, + 'rank': {'type': 'float', 'object_relation': 'ranking'} + } + bgp_object = MISPObject('bgp-ranking') + for feature in ('rank', 'position'): + bgp_attribute = {'value': values['response']['ranking'][feature]} + bgp_attribute.update(mapping[feature]) + bgp_object.add_attribute(**bgp_attribute) + date_attribute = {'value': datetime.strptime(values['meta']['date'], '%Y-%m-%d')} + date_attribute.update(mapping['date']) + bgp_object.add_attribute(**date_attribute) + address_attribute = {'value': values['meta']['address_family']} + address_attribute.update(mapping['address_family']) + bgp_object.add_attribute(**address_attribute) + + asn_object.add_reference(attribute.uuid, 'describes') + asn_object.add_reference(bgp_object.uuid, 'ranked-with') + event.add_object(asn_object) + event.add_object(bgp_object) + + event = json.loads(event.to_json()) + results = {key: event[key] for key in ('Attribute', 'Object')} + return {'results': results} def introspection(): diff --git a/misp_modules/modules/expansion/censys_enrich.py b/misp_modules/modules/expansion/censys_enrich.py index 0fc61ae..d5823ff 100644 --- a/misp_modules/modules/expansion/censys_enrich.py +++ b/misp_modules/modules/expansion/censys_enrich.py @@ -3,6 +3,7 @@ import json import base64 import codecs from dateutil.parser import isoparse +from . import check_input_attribute, standard_error_message from pymisp import MISPAttribute, MISPEvent, MISPObject try: import censys.base @@ -36,11 +37,11 @@ def handler(q=False): api_id = request['config']['api_id'] api_secret = request['config']['api_secret'] - if not request.get('attribute'): - return {'error': 'Unsupported input.'} + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} attribute = request['attribute'] if not any(input_type == attribute['type'] for input_type in mispattributes['input']): - return {'error': 'Unsupported attributes type'} + return {'error': 'Unsupported attribute type.'} attribute = MISPAttribute() attribute.from_dict(**request['attribute']) diff --git a/misp_modules/modules/expansion/circl_passivedns.py b/misp_modules/modules/expansion/circl_passivedns.py index d278a85..5f98314 100755 --- a/misp_modules/modules/expansion/circl_passivedns.py +++ b/misp_modules/modules/expansion/circl_passivedns.py @@ -1,5 +1,6 @@ import json import pypdns +from . import check_input_attribute, standard_error_message from pymisp import MISPAttribute, MISPEvent, MISPObject mispattributes = {'input': ['hostname', 'domain', 'ip-src', 'ip-dst', 'ip-src|port', 'ip-dst|port'], 'format': 'misp_standard'} @@ -58,11 +59,11 @@ def handler(q=False): if not request['config'].get('username') or not request['config'].get('password'): return {'error': 'CIRCL Passive DNS authentication is incomplete, please provide your username and password.'} authentication = (request['config']['username'], request['config']['password']) - if not request.get('attribute'): - return {'error': 'Unsupported input.'} + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} attribute = request['attribute'] if not any(input_type == attribute['type'] for input_type in mispattributes['input']): - return {'error': 'Unsupported attributes type'} + return {'error': 'Unsupported attribute type.'} pdns_parser = PassiveDNSParser(attribute, authentication) pdns_parser.parse() return pdns_parser.get_results() diff --git a/misp_modules/modules/expansion/circl_passivessl.py b/misp_modules/modules/expansion/circl_passivessl.py index 102bed8..65783d7 100755 --- a/misp_modules/modules/expansion/circl_passivessl.py +++ b/misp_modules/modules/expansion/circl_passivessl.py @@ -1,5 +1,6 @@ import json import pypssl +from . import check_input_attribute, standard_error_message from pymisp import MISPAttribute, MISPEvent, MISPObject mispattributes = {'input': ['ip-src', 'ip-dst', 'ip-src|port', 'ip-dst|port'], 'format': 'misp_standard'} @@ -83,11 +84,11 @@ def handler(q=False): if not request['config'].get('username') or not request['config'].get('password'): return {'error': 'CIRCL Passive SSL authentication is incomplete, please provide your username and password.'} authentication = (request['config']['username'], request['config']['password']) - if not request.get('attribute'): - return {'error': 'Unsupported input.'} + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} attribute = request['attribute'] if not any(input_type == attribute['type'] for input_type in mispattributes['input']): - return {'error': 'Unsupported attributes type'} + return {'error': 'Unsupported attribute type.'} pssl_parser = PassiveSSLParser(attribute, authentication) pssl_parser.parse() return pssl_parser.get_results() diff --git a/misp_modules/modules/expansion/cve_advanced.py b/misp_modules/modules/expansion/cve_advanced.py index 86cba8c..d15711f 100644 --- a/misp_modules/modules/expansion/cve_advanced.py +++ b/misp_modules/modules/expansion/cve_advanced.py @@ -1,7 +1,8 @@ -from collections import defaultdict -from pymisp import MISPEvent, MISPObject import json import requests +from . import check_input_attribute, standard_error_message +from collections import defaultdict +from pymisp import MISPEvent, MISPObject misperrors = {'error': 'Error'} mispattributes = {'input': ['vulnerability'], 'format': 'misp_standard'} @@ -55,7 +56,7 @@ class VulnerabilityParser(): value = value['title'] vulnerability_object.add_attribute(relation, **{'type': attribute_type, 'value': value}) vulnerability_object.add_reference(self.attribute['uuid'], 'related-to') - self.misp_event.add_object(**vulnerability_object) + self.misp_event.add_object(vulnerability_object) if 'cwe' in self.vulnerability and self.vulnerability['cwe'] not in ('Unknown', 'NVD-CWE-noinfo'): self.__parse_weakness(vulnerability_object.uuid) if 'capec' in self.vulnerability: @@ -78,7 +79,7 @@ class VulnerabilityParser(): for related_weakness in capec['related_weakness']: attribute = dict(type='weakness', value="CWE-{}".format(related_weakness)) capec_object.add_attribute('related-weakness', **attribute) - self.misp_event.add_object(**capec_object) + self.misp_event.add_object(capec_object) self.references[vulnerability_uuid].append(dict(referenced_uuid=capec_object.uuid, relationship_type='targeted-by')) @@ -94,7 +95,7 @@ class VulnerabilityParser(): for feature, relation in self.weakness_mapping.items(): if cwe.get(feature): weakness_object.add_attribute(relation, **dict(type=attribute_type, value=cwe[feature])) - self.misp_event.add_object(**weakness_object) + self.misp_event.add_object(weakness_object) self.references[vulnerability_uuid].append(dict(referenced_uuid=weakness_object.uuid, relationship_type='weakened-by')) break @@ -108,7 +109,9 @@ def handler(q=False): if q is False: return False request = json.loads(q) - attribute = request.get('attribute') + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + attribute = request['attribute'] if attribute.get('type') != 'vulnerability': misperrors['error'] = 'Vulnerability id missing.' return misperrors diff --git a/misp_modules/modules/expansion/cytomic_orion.py b/misp_modules/modules/expansion/cytomic_orion.py index 9723ed6..c13b254 100755 --- a/misp_modules/modules/expansion/cytomic_orion.py +++ b/misp_modules/modules/expansion/cytomic_orion.py @@ -7,6 +7,7 @@ An expansion module to enrich attributes in MISP and share indicators of comprom ''' +from . import check_input_attribute, standard_error_message from pymisp import MISPAttribute, MISPEvent, MISPObject import json import requests @@ -146,9 +147,11 @@ def handler(q=False): if not request.get('attribute'): return {'error': 'Unsupported input.'} + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} attribute = request['attribute'] if not any(input_type == attribute['type'] for input_type in mispattributes['input']): - return {'error': 'Unsupported attributes type'} + return {'error': 'Unsupported attribute type.'} if not request.get('config'): return {'error': 'Missing configuration'} diff --git a/misp_modules/modules/expansion/greynoise.py b/misp_modules/modules/expansion/greynoise.py index dd54158..4cd89d5 100644 --- a/misp_modules/modules/expansion/greynoise.py +++ b/misp_modules/modules/expansion/greynoise.py @@ -3,35 +3,59 @@ import json misperrors = {'error': 'Error'} mispattributes = {'input': ['ip-dst', 'ip-src'], 'output': ['text']} -moduleinfo = {'version': '0.1', 'author': 'Aurélien Schwab ', 'description': 'Module to access GreyNoise.io API.', 'module-type': ['hover']} -moduleconfig = ['user-agent'] # TODO take this into account in the code +moduleinfo = { + 'version': '0.2', + 'author': 'Aurélien Schwab ', + 'description': 'Module to access GreyNoise.io API.', + 'module-type': ['hover'] +} +moduleconfig = ['api_key'] -greynoise_api_url = 'http://api.greynoise.io:8888/v1/query/ip' -default_user_agent = 'MISP-Module' +greynoise_api_url = 'https://api.greynoise.io/v2/noise/quick/' +codes_mapping = { + '0x00': 'The IP has never been observed scanning the Internet', + '0x01': 'The IP has been observed by the GreyNoise sensor network', + '0x02': 'The IP has been observed scanning the GreyNoise sensor network, but has not completed a full connection, meaning this can be spoofed', + '0x03': 'The IP is adjacent to another host that has been directly observed by the GreyNoise sensor network', + '0x04': 'Reserved', + '0x05': 'This IP is commonly spoofed in Internet-scan activity', + '0x06': 'This IP has been observed as noise, but this host belongs to a cloud provider where IPs can be cycled frequently', + '0x07': 'This IP is invalid', + '0x08': 'This IP was classified as noise, but has not been observed engaging in Internet-wide scans or attacks in over 60 days' +} def handler(q=False): if q is False: return False request = json.loads(q) + if not request.get('config') or not request['config'].get('api_key'): + return {'error': 'Missing Greynoise API key.'} + headers = { + 'Accept': 'application/json', + 'key': request['config']['api_key'] + } for input_type in mispattributes['input']: if input_type in request: ip = request[input_type] break else: - misperrors['error'] = "Unsupported attributes type" + misperrors['error'] = "Unsupported attributes type." return misperrors - data = {'ip': ip} - r = requests.post(greynoise_api_url, data=data, headers={'user-agent': default_user_agent}) # Real request - if r.status_code == 200: # OK (record found) - response = r.text - if response: - return {'results': [{'types': mispattributes['output'], 'values': response}]} - elif r.status_code == 404: # Not found (not an error) - return {'results': [{'types': mispattributes['output'], 'values': 'No data'}]} - else: # Real error - misperrors['error'] = 'GreyNoise API not accessible (HTTP ' + str(r.status_code) + ')' - return misperrors['error'] + response = requests.get(f'{greynoise_api_url}{ip}', headers=headers) # Real request + if response.status_code == 200: # OK (record found) + return {'results': [{'types': mispattributes['output'], 'values': codes_mapping[response.json()['code']]}]} + # There is an error + errors = { + 400: "Bad request.", + 401: "Unauthorized. Please check your API key.", + 429: "Too many requests. You've hit the rate-limit." + } + try: + misperrors['error'] = errors[response.status_code] + except KeyError: + misperrors['error'] = f'GreyNoise API not accessible (HTTP {response.status_code})' + return misperrors['error'] def introspection(): diff --git a/misp_modules/modules/expansion/ipasn.py b/misp_modules/modules/expansion/ipasn.py index 3c6867c..3a32358 100755 --- a/misp_modules/modules/expansion/ipasn.py +++ b/misp_modules/modules/expansion/ipasn.py @@ -1,6 +1,7 @@ # -*- coding: utf-8 -*- import json +from . import check_input_attribute, standard_error_message from pyipasnhistory import IPASNHistory from pymisp import MISPAttribute, MISPEvent, MISPObject @@ -34,11 +35,11 @@ def handler(q=False): if q is False: return False request = json.loads(q) - if request.get('attribute') and request['attribute'].get('type') in mispattributes['input']: - toquery = request['attribute']['value'] - else: - misperrors['error'] = "Unsupported attributes type" - return misperrors + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + if request['attribute']['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} + toquery = request['attribute']['value'] ipasn = IPASNHistory() values = ipasn.query(toquery) diff --git a/misp_modules/modules/expansion/joesandbox_query.py b/misp_modules/modules/expansion/joesandbox_query.py index 1ace259..b9c4987 100644 --- a/misp_modules/modules/expansion/joesandbox_query.py +++ b/misp_modules/modules/expansion/joesandbox_query.py @@ -1,6 +1,7 @@ # -*- coding: utf-8 -*- import jbxapi import json +from . import check_input_attribute, checking_error, standard_error_message from joe_parser import JoeParser misperrors = {'error': 'Error'} @@ -27,6 +28,10 @@ def handler(q=False): if not apikey: return {'error': 'No API key provided'} + if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')): + return {'error': f'{standard_error_message}, {checking_error} that is the link to the Joe Sandbox report.'} + if request['attribute']['type'] != 'link': + return {'error': 'Unsupported attribute type.'} url = request['attribute']['value'] if "/submissions/" not in url: return {'error': "The URL does not point to a Joe Sandbox analysis."} diff --git a/misp_modules/modules/expansion/lastline_query.py b/misp_modules/modules/expansion/lastline_query.py index 4ce4e47..dbfdf14 100644 --- a/misp_modules/modules/expansion/lastline_query.py +++ b/misp_modules/modules/expansion/lastline_query.py @@ -3,8 +3,8 @@ Module (type "expansion") to query a Lastline report from an analysis link. """ import json - import lastline_api +from . import check_input_attribute, checking_error, standard_error_message misperrors = { @@ -52,6 +52,8 @@ def handler(q=False): try: config = request["config"] auth_data = lastline_api.LastlineAbstractClient.get_login_params_from_dict(config) + if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')): + return {'error': f'{standard_error_message}, {checking_error} that is the link to a Lastline analysis.'} analysis_link = request['attribute']['value'] # The API url changes based on the analysis link host name api_url = lastline_api.get_portal_url_from_task_link(analysis_link) diff --git a/misp_modules/modules/expansion/malwarebazaar.py b/misp_modules/modules/expansion/malwarebazaar.py index 4574b75..60739e8 100644 --- a/misp_modules/modules/expansion/malwarebazaar.py +++ b/misp_modules/modules/expansion/malwarebazaar.py @@ -1,5 +1,6 @@ import json import requests +from . import check_input_attribute, checking_error, standard_error_message from pymisp import MISPEvent, MISPObject mispattributes = {'input': ['md5', 'sha1', 'sha256'], @@ -34,7 +35,11 @@ def handler(q=False): if q is False: return False request = json.loads(q) + if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')): + return {'error': f'{standard_error_message}, {checking_error} that is the hash to submit to Malware Bazaar.'} attribute = request['attribute'] + if attribute['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} url = 'https://mb-api.abuse.ch/api/v1/' response = requests.post(url, data={'query': 'get_info', 'hash': attribute['value']}).json() query_status = response['query_status'] diff --git a/misp_modules/modules/expansion/ransomcoindb.py b/misp_modules/modules/expansion/ransomcoindb.py index 2b9b566..0e05855 100644 --- a/misp_modules/modules/expansion/ransomcoindb.py +++ b/misp_modules/modules/expansion/ransomcoindb.py @@ -1,4 +1,5 @@ import json +from . import check_input_attribute, checking_error, standard_error_message from ._ransomcoindb import ransomcoindb from pymisp import MISPObject @@ -28,6 +29,10 @@ def handler(q=False): q = json.loads(q) if "config" not in q or "api-key" not in q["config"]: return {"error": "Ransomcoindb API key is missing"} + if not q.get('attribute') or not check_input_attribute(q['attribute'], requirements=('type', 'value')): + return {'error': f'{standard_error_message}, {checking_error}.'} + if q['attribute']['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} api_key = q["config"]["api-key"] r = {"results": []} diff --git a/misp_modules/modules/expansion/recordedfuture.py b/misp_modules/modules/expansion/recordedfuture.py new file mode 100644 index 0000000..ccea31b --- /dev/null +++ b/misp_modules/modules/expansion/recordedfuture.py @@ -0,0 +1,396 @@ +import json +import logging +import requests +from requests.exceptions import HTTPError, ProxyError,\ + InvalidURL, ConnectTimeout, ConnectionError +from . import check_input_attribute, checking_error, standard_error_message +import platform +import os +from urllib.parse import quote, urlparse +from pymisp import MISPAttribute, MISPEvent, MISPTag, MISPObject + +moduleinfo = { + 'version': '1.0.1', + 'author': 'Recorded Future', + 'description': 'Module to retrieve data from Recorded Future', + 'module-type': ['expansion', 'hover'] +} + +moduleconfig = ['token', 'proxy_host', 'proxy_port', 'proxy_username', 'proxy_password'] + +misperrors = {'error': 'Error'} + +ATTRIBUTES = [ + 'ip', + 'ip-src', + 'ip-dst', + 'domain', + 'hostname', + 'md5', + 'sha1', + 'sha256', + 'uri', + 'url', + 'vulnerability', + 'weakness' +] + +mispattributes = { + 'input': ATTRIBUTES, + 'output': ATTRIBUTES + ['email-src', 'text'], + 'format': 'misp_standard' +} + +LOGGER = logging.getLogger('recorded_future') +LOGGER.setLevel(logging.INFO) + + +class RequestHandler: + """A class for handling any outbound requests from this module.""" + def __init__(self): + self.session = requests.Session() + self.app_id = f'{os.path.basename(__file__)}/{moduleinfo["version"]} ({platform.platform()}) ' \ + f'misp_enrichment/{moduleinfo["version"]} python-requests/{requests.__version__}' + self.proxies = None + self.rf_token = None + + def get(self, url: str, headers: dict = None) -> requests.Response: + """General get method with proxy error handling.""" + try: + timeout = 7 if self.proxies else None + response = self.session.get(url, headers=headers, proxies=self.proxies, timeout=timeout) + response.raise_for_status() + return response + except (ConnectTimeout, ProxyError, InvalidURL) as error: + msg = 'Error connecting with proxy, please check the Recorded Future app proxy settings.' + LOGGER.error(f'{msg} Error: {error}') + misperrors['error'] = msg + raise + + def rf_lookup(self, category: str, ioc: str) -> requests.Response: + """Do a lookup call using Recorded Future's ConnectAPI.""" + parsed_ioc = quote(ioc, safe='') + url = f'https://api.recordedfuture.com/v2/{category}/{parsed_ioc}?fields=risk%2CrelatedEntities' + headers = {'X-RFToken': self.rf_token, + 'User-Agent': self.app_id} + try: + response = self.get(url, headers) + except HTTPError as error: + msg = f'Error when requesting data from Recorded Future. {error.response}: {error.response.reason}' + LOGGER.error(msg) + misperrors['error'] = msg + raise + return response + + +GLOBAL_REQUEST_HANDLER = RequestHandler() + + +class GalaxyFinder: + """A class for finding MISP galaxy matches to Recorded Future data.""" + def __init__(self): + self.session = requests.Session() + self.sources = { + 'RelatedThreatActor': [ + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/threat-actor.json' + ], + 'RelatedMalware': [ + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/banker.json', + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/botnet.json', + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/exploit-kit.json', + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/rat.json', + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/ransomware.json', + 'https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/malpedia.json' + ] + } + self.galaxy_clusters = {} + + def pull_galaxy_cluster(self, related_type: str) -> None: + """Fetches galaxy clusters for the related_type from the remote json files specified as self.sources.""" + # Only fetch clusters if not fetched previously + if not self.galaxy_clusters.get(related_type): + for source in self.sources.get(related_type): + try: + response = GLOBAL_REQUEST_HANDLER.get(source) + name = source.split('/')[-1].split('.')[0] + self.galaxy_clusters[related_type] = {name: response.json()} + except ConnectionError as error: + LOGGER.warning(f'pull_galaxy_cluster failed for source: {source}, with error: {error}.') + + def find_galaxy_match(self, indicator: str, related_type: str) -> str: + """Searches the clusters of the related_type for a match with the indicator. + :returns the first matching galaxy string or an empty string if no galaxy match is found. + """ + self.pull_galaxy_cluster(related_type) + for cluster_name, cluster in self.galaxy_clusters.get(related_type, {}).items(): + for value in cluster['values']: + try: + if indicator in value['meta']['synonyms'] or indicator in value['value']: + value = value['value'] + return f'misp-galaxy:{cluster_name}="{value}"' + except KeyError: + pass + return '' + + +class RFColors: + """Class for setting signature RF-colors.""" + def __init__(self): + self.rf_white = '#CCCCCC' + self.rf_yellow = '#FFCE00' + self.rf_red = '#CF0A2C' + + def riskscore_color(self, risk_score: int) -> str: + """Returns appropriate hex-colors according to risk score.""" + risk_score = int(risk_score) + if risk_score < 25: + return self.rf_white + elif risk_score < 65: + return self.rf_yellow + else: + return self.rf_red + + def riskrule_color(self, risk_rule_criticality: int) -> str: + """Returns appropriate hex-colors according to risk rule criticality.""" + risk_rule_criticality = int(risk_rule_criticality) + if risk_rule_criticality == 1: + return self.rf_white + elif risk_rule_criticality == 2: + return self.rf_yellow + else: # risk_rule_criticality == 3 or 4 + return self.rf_red + + +class RFEnricher: + """Class for enriching an attribute with data from Recorded Future. + The enrichment data is returned as a custom MISP object. + """ + def __init__(self, attribute_props: dict): + self.event = MISPEvent() + self.enrichment_object = MISPObject('Recorded Future Enrichment') + description = ( + 'An object containing the enriched attribute and ' + 'related entities from Recorded Future.' + ) + self.enrichment_object.from_dict(**{ + 'meta-category': 'misc', + 'description': description, + 'distribution': 0 + }) + + # Create a copy of enriched attribute to add tags to + temp_attr = MISPAttribute() + temp_attr.from_dict(**attribute_props) + self.enriched_attribute = MISPAttribute() + self.enriched_attribute.from_dict(**{ + 'value': temp_attr.value, + 'type': temp_attr.type, + 'distribution': 0 + }) + + self.related_attributes = [] + self.color_picker = RFColors() + self.galaxy_finder = GalaxyFinder() + + # Mapping from MISP-type to RF-type + self.type_to_rf_category = { + 'ip': 'ip', + 'ip-src': 'ip', + 'ip-dst': 'ip', + 'domain': 'domain', + 'hostname': 'domain', + 'md5': 'hash', + 'sha1': 'hash', + 'sha256': 'hash', + 'uri': 'url', + 'url': 'url', + 'vulnerability': 'vulnerability', + 'weakness': 'vulnerability' + } + + # Related entities from RF portrayed as related attributes in MISP + self.related_attribute_types = [ + 'RelatedIpAddress', 'RelatedInternetDomainName', 'RelatedHash', + 'RelatedEmailAddress', 'RelatedCyberVulnerability' + ] + # Related entities from RF portrayed as tags in MISP + self.galaxy_tag_types = ['RelatedMalware', 'RelatedThreatActor'] + + def enrich(self) -> None: + """Run the enrichment.""" + category = self.type_to_rf_category.get(self.enriched_attribute.type) + json_response = GLOBAL_REQUEST_HANDLER.rf_lookup(category, self.enriched_attribute.value) + response = json.loads(json_response.content) + + try: + # Add risk score and risk rules as tags to the enriched attribute + risk_score = response['data']['risk']['score'] + hex_color = self.color_picker.riskscore_color(risk_score) + tag_name = f'recorded-future:risk-score="{risk_score}"' + self.add_tag(tag_name, hex_color) + for evidence in response['data']['risk']['evidenceDetails']: + risk_rule = evidence['rule'] + criticality = evidence['criticality'] + hex_color = self.color_picker.riskrule_color(criticality) + tag_name = f'recorded-future:risk-rule="{risk_rule}"' + self.add_tag(tag_name, hex_color) + + # Retrieve related entities + for related_entity in response['data']['relatedEntities']: + related_type = related_entity['type'] + if related_type in self.related_attribute_types: + # Related entities returned as additional attributes + for related in related_entity['entities']: + if int(related["count"]) > 4: + indicator = related['entity']['name'] + self.add_related_attribute(indicator, related_type) + elif related_type in self.galaxy_tag_types: + # Related entities added as galaxy-tags to the enriched attribute + galaxy_tags = [] + for related in related_entity['entities']: + if int(related["count"]) > 4: + indicator = related['entity']['name'] + galaxy = self.galaxy_finder.find_galaxy_match(indicator, related_type) + # Handle deduplication of galaxy tags + if galaxy and galaxy not in galaxy_tags: + galaxy_tags.append(galaxy) + for galaxy in galaxy_tags: + self.add_tag(galaxy) + except KeyError: + misperrors['error'] = 'Unexpected format in Recorded Future api response.' + raise + + def add_related_attribute(self, indicator: str, related_type: str) -> None: + """Helper method for adding an indicator to the related attribute list.""" + out_type = self.get_output_type(related_type, indicator) + attribute = MISPAttribute() + attribute.from_dict(**{'value': indicator, 'type': out_type, 'distribution': 0}) + self.related_attributes.append((related_type, attribute)) + + def add_tag(self, tag_name: str, hex_color: str = None) -> None: + """Helper method for adding a tag to the enriched attribute.""" + tag = MISPTag() + tag_properties = {'name': tag_name} + if hex_color: + tag_properties['colour'] = hex_color + tag.from_dict(**tag_properties) + self.enriched_attribute.add_tag(tag) + + def get_output_type(self, related_type: str, indicator: str) -> str: + """Helper method for translating a Recorded Future related type to a MISP output type.""" + output_type = 'text' + if related_type == 'RelatedIpAddress': + output_type = 'ip-dst' + elif related_type == 'RelatedInternetDomainName': + output_type = 'domain' + elif related_type == 'RelatedHash': + hash_len = len(indicator) + if hash_len == 64: + output_type = 'sha256' + elif hash_len == 40: + output_type = 'sha1' + elif hash_len == 32: + output_type = 'md5' + elif related_type == 'RelatedEmailAddress': + output_type = 'email-src' + elif related_type == 'RelatedCyberVulnerability': + signature = indicator.split('-')[0] + if signature == 'CVE': + output_type = 'vulnerability' + elif signature == 'CWE': + output_type = 'weakness' + return output_type + + def get_results(self) -> dict: + """Build and return the enrichment results.""" + self.enrichment_object.add_attribute('Enriched attribute', **self.enriched_attribute) + for related_type, attribute in self.related_attributes: + self.enrichment_object.add_attribute(related_type, **attribute) + self.event.add_object(**self.enrichment_object) + event = json.loads(self.event.to_json()) + result = {key: event[key] for key in ['Object'] if key in event} + return {'results': result} + + +def get_proxy_settings(config: dict) -> dict: + """Returns proxy settings in the requests format. + If no proxy settings are set, return None.""" + proxies = None + host = config.get('proxy_host') + port = config.get('proxy_port') + username = config.get('proxy_username') + password = config.get('proxy_password') + + if host: + if not port: + misperrors['error'] = 'The recordedfuture_proxy_host config is set, ' \ + 'please also set the recordedfuture_proxy_port.' + raise KeyError + parsed = urlparse(host) + if 'http' in parsed.scheme: + scheme = 'http' + else: + scheme = parsed.scheme + netloc = parsed.netloc + host = f'{netloc}:{port}' + + if username: + if not password: + misperrors['error'] = 'The recordedfuture_proxy_username config is set, ' \ + 'please also set the recordedfuture_proxy_password.' + raise KeyError + auth = f'{username}:{password}' + host = auth + '@' + host + + proxies = { + 'http': f'{scheme}://{host}', + 'https': f'{scheme}://{host}' + } + + LOGGER.info(f'Proxy settings: {proxies}') + return proxies + + +def handler(q=False): + """Handle enrichment.""" + if q is False: + return False + request = json.loads(q) + + config = request.get('config') + if config and config.get('token'): + GLOBAL_REQUEST_HANDLER.rf_token = config.get('token') + else: + misperrors['error'] = 'Missing Recorded Future token.' + return misperrors + if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')): + return {'error': f'{standard_error_message}, {checking_error}.'} + if request['attribute']['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} + + try: + GLOBAL_REQUEST_HANDLER.proxies = get_proxy_settings(config) + except KeyError: + return misperrors + + input_attribute = request.get('attribute') + rf_enricher = RFEnricher(input_attribute) + + try: + rf_enricher.enrich() + except (HTTPError, ConnectTimeout, ProxyError, InvalidURL, KeyError): + return misperrors + + return rf_enricher.get_results() + + +def introspection(): + """Returns a dict of the supported attributes.""" + return mispattributes + + +def version(): + """Returns a dict with the version and the associated meta-data + including potential configurations required of the module.""" + moduleinfo['config'] = moduleconfig + return moduleinfo diff --git a/misp_modules/modules/expansion/shodan.py b/misp_modules/modules/expansion/shodan.py index 5a4b792..f295deb 100755 --- a/misp_modules/modules/expansion/shodan.py +++ b/misp_modules/modules/expansion/shodan.py @@ -5,38 +5,225 @@ try: import shodan except ImportError: print("shodan module not installed.") +from . import check_input_attribute, standard_error_message +from datetime import datetime +from pymisp import MISPAttribute, MISPEvent, MISPObject misperrors = {'error': 'Error'} -mispattributes = {'input': ['ip-src', 'ip-dst'], 'output': ['freetext']} -moduleinfo = {'version': '0.1', 'author': 'Raphaël Vinot', +mispattributes = {'input': ['ip-src', 'ip-dst'], + 'format': 'misp_standard'} +moduleinfo = {'version': '0.2', 'author': 'Raphaël Vinot', 'description': 'Query on Shodan', 'module-type': ['expansion']} moduleconfig = ['apikey'] +class ShodanParser(): + def __init__(self, attribute): + self.misp_event = MISPEvent() + self.attribute = MISPAttribute() + self.attribute.from_dict(**attribute) + self.misp_event.add_attribute(**self.attribute) + self.ip_address_mapping = { + 'asn': {'type': 'AS', 'object_relation': 'asn'}, + 'city': {'type': 'text', 'object_relation': 'city'}, + 'country_code': {'type': 'text', 'object_relation': 'country-code'}, + 'country_name': {'type': 'text', 'object_relation': 'country'}, + 'isp': {'type': 'text', 'object_relation': 'ISP'}, + 'latitude': {'type': 'float', 'object_relation': 'latitude'}, + 'longitude': {'type': 'float', 'object_relation': 'longitude'}, + 'org': {'type': 'text', 'object_relation': 'organization'}, + 'postal_code': {'type': 'text', 'object_relation': 'zipcode'}, + 'region_code': {'type': 'text', 'object_relation': 'region-code'} + } + self.ip_port_mapping = { + 'domains': {'type': 'domain', 'object_relation': 'domain'}, + 'hostnames': {'type': 'hostname', 'object_relation': 'hostname'} + } + self.vulnerability_mapping = { + 'cvss': {'type': 'float', 'object_relation': 'cvss-score'}, + 'summary': {'type': 'text', 'object_relation': 'summary'} + } + self.x509_mapping = { + 'bits': {'type': 'text', 'object_relation': 'pubkey-info-size'}, + 'expires': {'type': 'datetime', 'object_relation': 'validity-not-after'}, + 'issued': {'type': 'datetime', 'object_relation': 'validity-not-before'}, + 'issuer': {'type': 'text', 'object_relation': 'issuer'}, + 'serial': {'type': 'text', 'object_relation': 'serial-number'}, + 'sig_alg': {'type': 'text', 'object_relation': 'signature_algorithm'}, + 'subject': {'type': 'text', 'object_relation': 'subject'}, + 'type': {'type': 'text', 'object_relation': 'pubkey-info-algorithm'}, + 'version': {'type': 'text', 'object_relation': 'version'} + } + + def query_shodan(self, apikey): + # Query Shodan and get the results in a json blob + api = shodan.Shodan(apikey) + query_results = api.host(self.attribute.value) + + # Parse the information about the IP address used as input + ip_address_attributes = [] + for feature, mapping in self.ip_address_mapping.items(): + if query_results.get(feature): + attribute = {'value': query_results[feature]} + attribute.update(mapping) + ip_address_attributes.append(attribute) + if ip_address_attributes: + ip_address_object = MISPObject('ip-api-address') + for attribute in ip_address_attributes: + ip_address_object.add_attribute(**attribute) + ip_address_object.add_attribute(**self._get_source_attribute()) + ip_address_object.add_reference(self.attribute.uuid, 'describes') + self.misp_event.add_object(ip_address_object) + + # Parse the hostnames / domains and ports associated with the IP address + if query_results.get('ports'): + ip_port_object = MISPObject('ip-port') + ip_port_object.add_attribute(**self._get_source_attribute()) + feature = self.attribute.type.split('-')[1] + for port in query_results['ports']: + attribute = { + 'type': 'port', + 'object_relation': f'{feature}-port', + 'value': port + } + ip_port_object.add_attribute(**attribute) + for feature, mapping in self.ip_port_mapping.items(): + for value in query_results.get(feature, []): + attribute = {'value': value} + attribute.update(mapping) + ip_port_object.add_attribute(**attribute) + ip_port_object.add_reference(self.attribute.uuid, 'extends') + self.misp_event.add_object(ip_port_object) + else: + if any(query_results.get(feature) for feature in ('domains', 'hostnames')): + domain_ip_object = MISPObject('domain-ip') + domain_ip_object.add_attribute(**self._get_source_attribute()) + for feature in ('domains', 'hostnames'): + for value in query_results[feature]: + attribute = { + 'type': 'domain', + 'object_relation': 'domain', + 'value': value + } + domain_ip_object.add_attribute(**attribute) + domain_ip_object.add_reference(self.attribute.uuid, 'extends') + self.misp_event.add_object(domain_ip_object) + + # Parse data within the "data" field + if query_results.get('vulns'): + vulnerabilities = {} + for data in query_results['data']: + # Parse vulnerabilities + if data.get('vulns'): + for cve, vulnerability in data['vulns'].items(): + if cve not in vulnerabilities: + vulnerabilities[cve] = vulnerability + # Also parse the certificates + if data.get('ssl'): + self._parse_cert(data['ssl']) + for cve, vulnerability in vulnerabilities.items(): + vulnerability_object = MISPObject('vulnerability') + vulnerability_object.add_attribute(**{ + 'type': 'vulnerability', + 'object_relation': 'id', + 'value': cve + }) + for feature, mapping in self.vulnerability_mapping.items(): + if vulnerability.get(feature): + attribute = {'value': vulnerability[feature]} + attribute.update(mapping) + vulnerability_object.add_attribute(**attribute) + if vulnerability.get('references'): + for reference in vulnerability['references']: + vulnerability_object.add_attribute(**{ + 'type': 'link', + 'object_relation': 'references', + 'value': reference + }) + vulnerability_object.add_reference(self.attribute.uuid, 'vulnerability-of') + self.misp_event.add_object(vulnerability_object) + for cve_id in query_results['vulns']: + if cve_id not in vulnerabilities: + attribute = { + 'type': 'vulnerability', + 'value': cve_id + } + self.misp_event.add_attribute(**attribute) + else: + # We have no vulnerability data, we only check if we have + # certificates within the "data" field + for data in query_results['data']: + if data.get('ssl'): + self._parse_cert(data['ssl']['cert']) + + def get_result(self): + event = json.loads(self.misp_event.to_json()) + results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])} + return {'results': results} + + # When we want to add the IP address information in objects such as the + # domain-ip or ip-port objects referencing the input IP address attribute + def _get_source_attribute(self): + return { + 'type': self.attribute.type, + 'object_relation': self.attribute.type, + 'value': self.attribute.value + } + + def _parse_cert(self, certificate): + x509_object = MISPObject('x509') + for feature in ('serial', 'sig_alg', 'version'): + if certificate.get(feature): + attribute = {'value': certificate[feature]} + attribute.update(self.x509_mapping[feature]) + x509_object.add_attribute(**attribute) + # Parse issuer and subject value + for feature in ('issuer', 'subject'): + if certificate.get(feature): + attribute_value = (f'{identifier}={value}' for identifier, value in certificate[feature].items()) + attribute = {'value': f'/{"/".join(attribute_value)}'} + attribute.update(self.x509_mapping[feature]) + x509_object.add_attribute(**attribute) + # Parse datetime attributes + for feature in ('expires', 'issued'): + if certificate.get(feature): + attribute = {'value': datetime.strptime(certificate[feature], '%Y%m%d%H%M%SZ')} + attribute.update(self.x509_mapping[feature]) + x509_object.add_attribute(**attribute) + # Parse fingerprints + if certificate.get('fingerprint'): + for hash_type, hash_value in certificate['fingerprint'].items(): + x509_object.add_attribute(**{ + 'type': f'x509-fingerprint-{hash_type}', + 'object_relation': f'x509-fingerprint-{hash_type}', + 'value': hash_value + }) + # Parse public key related info + if certificate.get('pubkey'): + for feature, value in certificate['pubkey'].items(): + attribute = {'value': value} + attribute.update(self.x509_mapping[feature]) + x509_object.add_attribute(**attribute) + x509_object.add_reference(self.attribute.uuid, 'identifies') + self.misp_event.add_object(x509_object) + + def handler(q=False): if q is False: return False request = json.loads(q) - if request.get('ip-src'): - toquery = request['ip-src'] - elif request.get('ip-dst'): - toquery = request['ip-dst'] - else: - misperrors['error'] = "Unsupported attributes type" - return misperrors - - if not request.get('config') or not request['config'].get('apikey'): - misperrors['error'] = 'Shodan authentication is missing' - return misperrors - api = shodan.Shodan(request['config'].get('apikey')) - - return handle_expansion(api, toquery) - - -def handle_expansion(api, domain): - return {'results': [{'types': mispattributes['output'], 'values': json.dumps(api.host(domain))}]} + if not request.get('config', {}).get('apikey'): + return {'error': 'Shodan authentication is missing'} + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + attribute = request['attribute'] + if attribute['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} + shodan_parser = ShodanParser(attribute) + shodan_parser.query_shodan(request['config']['apikey']) + return shodan_parser.get_result() def introspection(): diff --git a/misp_modules/modules/expansion/sophoslabs_intelix.py b/misp_modules/modules/expansion/sophoslabs_intelix.py index 017683a..4d7c413 100644 --- a/misp_modules/modules/expansion/sophoslabs_intelix.py +++ b/misp_modules/modules/expansion/sophoslabs_intelix.py @@ -1,7 +1,8 @@ -from pymisp import MISPEvent, MISPObject import json import requests import base64 +from . import check_input_attribute, checking_error, standard_error_message +from pymisp import MISPEvent, MISPObject from urllib.parse import quote moduleinfo = {'version': '1.0', @@ -105,13 +106,25 @@ def handler(q=False): misperrors['error'] = "Missing client_id or client_secret value for SOPHOSLabs Intelix. \ It's free to sign up here https://aws.amazon.com/marketplace/pp/B07SLZPMCS." return misperrors + to_check = (('type', 'value'), ('type', 'value1')) + if not j.get('attribute') or not any(check_input_attribute(j['attribute'], requirements=check) for check in to_check): + return {'error': f'{standard_error_message}, {checking_error}.'} + attribute = j['attribute'] + if attribute['type'] not in misp_types_in: + return {'error': 'Unsupported attribute type.'} client = SophosLabsApi(j['config']['client_id'], j['config']['client_secret']) - if j['attribute']['type'] == "sha256": - client.hash_lookup(j['attribute']['value1']) - if j['attribute']['type'] in ['ip-dst', 'ip-src', 'ip']: - client.ip_lookup(j["attribute"]["value1"]) - if j['attribute']['type'] in ['uri', 'url', 'domain', 'hostname']: - client.url_lookup(j["attribute"]["value1"]) + mapping = { + 'sha256': 'hash_lookup', + 'ip-dst': 'ip_lookup', + 'ip-src': 'ip_lookup', + 'ip': 'ip_lookup', + 'uri': 'url_lookup', + 'url': 'url_lookup', + 'domain': 'url_lookup', + 'hostname': 'url_lookup' + } + attribute_value = attribute['value'] if 'value' in attribute else attribute['value1'] + getattr(client, mapping[attribute['type']])(attribute_value) return client.get_result() diff --git a/misp_modules/modules/expansion/trustar_enrich.py b/misp_modules/modules/expansion/trustar_enrich.py index efe7c53..ab472af 100644 --- a/misp_modules/modules/expansion/trustar_enrich.py +++ b/misp_modules/modules/expansion/trustar_enrich.py @@ -1,7 +1,11 @@ import json import pymisp +from base64 import b64encode +from collections import OrderedDict +from . import check_input_attribute, checking_error, standard_error_message from pymisp import MISPAttribute, MISPEvent, MISPObject -from trustar import TruStar +from trustar import TruStar, Indicator +from urllib.parse import quote misperrors = {'error': "Error"} mispattributes = { @@ -33,9 +37,13 @@ class TruSTARParser: 'SHA256': "sha256" } + # Relevant fields from each TruSTAR endpoint + SUMMARY_FIELDS = ["severityLevel", "source", "score", "attributes"] + METADATA_FIELDS = ["sightings", "first_seen", "last_seen", "tags"] + REPORT_BASE_URL = "https://station.trustar.co/constellation/reports/{}" - CLIENT_METATAG = "MISP-{}".format(pymisp.__version__) + CLIENT_METATAG = f"MISP-{pymisp.__version__}" def __init__(self, attribute, config): config['enclave_ids'] = config.get('enclave_ids', "").strip().split(',') @@ -51,45 +59,111 @@ class TruSTARParser: """ Returns the MISP Event enriched with TruSTAR indicator summary data. """ - event = json.loads(self.misp_event.to_json()) - results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])} - return {'results': results} + try: + event = json.loads(self.misp_event.to_json()) + results = {key: event[key] for key in ('Attribute', 'Object') if (key in event and event[key])} + return {'results': results} + except Exception as e: + misperrors['error'] += f" -- Encountered issue serializing enrichment data -- {e}" + return misperrors - def generate_trustar_links(self, entity_value): + def generate_trustar_link(self, entity_type, entity_value): """ - Generates links to TruSTAR reports if they exist. + Generates link to TruSTAR report of entity. + :param entity_type: Type of entity. :param entity_value: Value of entity. + :return: Link to indicator report in TruSTAR platform. """ - report_links = list() - trustar_reports = self.ts_client.search_reports(entity_value) - for report in trustar_reports: - report_links.append(self.REPORT_BASE_URL.format(report.id)) + report_id = b64encode(quote(f"{entity_type}|{entity_value}").encode()).decode() - return report_links + return self.REPORT_BASE_URL.format(report_id) - def parse_indicator_summary(self, summaries): + @staticmethod + def extract_tags(enrichment_report): """ - Converts a response from the TruSTAR /1.3/indicators/summaries endpoint - a MISP trustar_report object and adds the summary data and links as attributes. + Extracts tags from the enrichment report in order to add them + to the TruSTAR MISP Object. Removes tags from report to avoid + redundancy. - :param summaries: A TruSTAR Python SDK Page.generator object for generating - indicator summaries pages. + :param: Enrichment data. + :return: List of tags. + """ + if enrichment_report and enrichment_report.get('tags'): + return [tag.get('name') for tag in enrichment_report.pop('tags')] + return None + + def generate_enrichment_report(self, summary, metadata): + """ + Extracts desired fields from summary and metadata reports and + generates an enrichment report. + + :param summary: Indicator summary report. + :param metadata: Indicator metadata report. + :return: Enrichment report. + """ + # Preserve order of fields as they exist in SUMMARY_FIELDS and METADATA_FIELDS + enrichment_report = OrderedDict() + + if summary: + summary_dict = summary.to_dict() + enrichment_report.update( + {field: summary_dict[field] for field in self.SUMMARY_FIELDS if summary_dict.get(field)}) + + if metadata: + metadata_dict = metadata.to_dict() + enrichment_report.update( + {field: metadata_dict[field] for field in self.METADATA_FIELDS if metadata_dict.get(field)}) + + return enrichment_report + + def parse_indicator_summary(self, indicator, summary, metadata): + """ + Pulls enrichment data from the TruSTAR /indicators/summaries and /indicators/metadata endpoints + and creates a MISP trustar_report. + + :param indicator: Value of the attribute + :summary: Indicator summary response object. + :metadata: Indicator response object. """ - for summary in summaries: - trustar_obj = MISPObject('trustar_report') + # Verify that the indicator type is supported by TruSTAR + if summary and summary.indicator_type in self.ENTITY_TYPE_MAPPINGS: indicator_type = summary.indicator_type - indicator_value = summary.value - if indicator_type in self.ENTITY_TYPE_MAPPINGS: + elif metadata and metadata.type in self.ENTITY_TYPE_MAPPINGS: + indicator_type = metadata.type + else: + misperrors['error'] += " -- Attribute not found or not supported" + raise Exception + + try: + # Extract most relevant fields from indicator summary and metadata responses + enrichment_report = self.generate_enrichment_report(summary, metadata) + tags = self.extract_tags(enrichment_report) + + if enrichment_report: + # Create MISP trustar_report object and populate it with enrichment data + trustar_obj = MISPObject('trustar_report') trustar_obj.add_attribute(indicator_type, attribute_type=self.ENTITY_TYPE_MAPPINGS[indicator_type], - value=indicator_value) + value=indicator) trustar_obj.add_attribute("INDICATOR_SUMMARY", attribute_type="text", - value=json.dumps(summary.to_dict(), sort_keys=True, indent=4)) - report_links = self.generate_trustar_links(indicator_value) - for link in report_links: - trustar_obj.add_attribute("REPORT_LINK", attribute_type="link", value=link) + value=json.dumps(enrichment_report, indent=4)) + + report_link = self.generate_trustar_link(indicator_type, indicator) + trustar_obj.add_attribute("REPORT_LINK", attribute_type="link", value=report_link) + self.misp_event.add_object(**trustar_obj) + elif not tags: + # If enrichment report is empty and there are no tags, nothing to add to attribute + raise Exception("No relevant data found") + + if tags: + for tag in tags: + self.misp_event.add_attribute_tag(tag, indicator) + + except Exception as e: + misperrors['error'] += f" -- Error enriching attribute {indicator} -- {e}" + raise e def handler(q=False): @@ -110,17 +184,31 @@ def handler(q=False): misperrors['error'] = "Your TruSTAR API key and secret are required for indicator enrichment." return misperrors + if not request.get('attribute') or not check_input_attribute(request['attribute'], requirements=('type', 'value')): + return {'error': f'{standard_error_message}, {checking_error}.'} attribute = request['attribute'] + if attribute['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} trustar_parser = TruSTARParser(attribute, config) + metadata = None + summary = None try: - summaries = list( - trustar_parser.ts_client.get_indicator_summaries([attribute['value']], page_size=MAX_PAGE_SIZE)) + metadata = trustar_parser.ts_client.get_indicators_metadata([Indicator(value=attribute['value'])])[0] except Exception as e: - misperrors['error'] = "Unable to retrieve TruSTAR summary data: {}".format(e) + misperrors['error'] += f" -- Could not retrieve indicator metadata from TruSTAR {e}" + + try: + summary = list( + trustar_parser.ts_client.get_indicator_summaries([attribute['value']], page_size=MAX_PAGE_SIZE))[0] + except Exception as e: + misperrors['error'] += f" -- Unable to retrieve TruSTAR summary data: {e}" + + try: + trustar_parser.parse_indicator_summary(attribute['value'], summary, metadata) + except Exception: return misperrors - trustar_parser.parse_indicator_summary(summaries) return trustar_parser.get_results() diff --git a/misp_modules/modules/expansion/urlhaus.py b/misp_modules/modules/expansion/urlhaus.py index baaaaf6..ed13b77 100644 --- a/misp_modules/modules/expansion/urlhaus.py +++ b/misp_modules/modules/expansion/urlhaus.py @@ -1,6 +1,8 @@ -from pymisp import MISPAttribute, MISPEvent, MISPObject +# -*- coding: utf-8 -*- import json import requests +from . import check_input_attribute, standard_error_message +from pymisp import MISPAttribute, MISPEvent, MISPObject misperrors = {'error': 'Error'} mispattributes = {'input': ['domain', 'hostname', 'ip-src', 'ip-dst', 'md5', 'sha256', 'url'], @@ -134,7 +136,11 @@ def handler(q=False): if q is False: return False request = json.loads(q) + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} attribute = request['attribute'] + if attribute['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} urlhaus_parser = _misp_type_mapping[attribute['type']](attribute) return urlhaus_parser.query_api() diff --git a/misp_modules/modules/expansion/virustotal.py b/misp_modules/modules/expansion/virustotal.py index b09de81..f5f29c5 100644 --- a/misp_modules/modules/expansion/virustotal.py +++ b/misp_modules/modules/expansion/virustotal.py @@ -1,6 +1,7 @@ -from pymisp import MISPAttribute, MISPEvent, MISPObject import json import requests +from . import check_input_attribute, standard_error_message +from pymisp import MISPAttribute, MISPEvent, MISPObject misperrors = {'error': 'Error'} mispattributes = {'input': ['hostname', 'domain', "ip-src", "ip-dst", "md5", "sha1", "sha256", "url"], @@ -143,7 +144,7 @@ class VirusTotalParser(object): def parse_resolutions(self, resolutions, subdomains=None, uuids=None): domain_ip_object = MISPObject('domain-ip') - if self.attribute.type == 'domain': + if self.attribute.type in ('domain', 'hostname'): domain_ip_object.add_attribute('domain', type='domain', value=self.attribute.value) attribute_type, relation, key = ('ip-dst', 'ip', 'ip_address') else: @@ -174,7 +175,7 @@ class VirusTotalParser(object): vt_object = MISPObject('virustotal-report') vt_object.add_attribute('permalink', type='link', value=query_result['permalink']) detection_ratio = '{}/{}'.format(query_result['positives'], query_result['total']) - vt_object.add_attribute('detection-ratio', type='text', value=detection_ratio) + vt_object.add_attribute('detection-ratio', type='text', value=detection_ratio, disable_correlation=True) self.misp_event.add_object(**vt_object) return vt_object.uuid @@ -195,6 +196,11 @@ def handler(q=False): if not request.get('config') or not request['config'].get('apikey'): misperrors['error'] = "A VirusTotal api key is required for this module." return misperrors + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} + if request['attribute']['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} + event_limit = request['config'].get('event_limit') if not isinstance(event_limit, int): event_limit = 5 diff --git a/misp_modules/modules/expansion/virustotal_public.py b/misp_modules/modules/expansion/virustotal_public.py index e7c2e96..989e48d 100644 --- a/misp_modules/modules/expansion/virustotal_public.py +++ b/misp_modules/modules/expansion/virustotal_public.py @@ -1,6 +1,7 @@ -from pymisp import MISPAttribute, MISPEvent, MISPObject import json import requests +from . import check_input_attribute, standard_error_message +from pymisp import MISPAttribute, MISPEvent, MISPObject misperrors = {'error': 'Error'} mispattributes = {'input': ['hostname', 'domain', "ip-src", "ip-dst", "md5", "sha1", "sha256", "url"], @@ -36,7 +37,7 @@ class VirusTotalParser(): def parse_resolutions(self, resolutions, subdomains=None, uuids=None): domain_ip_object = MISPObject('domain-ip') - if self.attribute.type == 'domain': + if self.attribute.type in ('domain', 'hostname'): domain_ip_object.add_attribute('domain', type='domain', value=self.attribute.value) attribute_type, relation, key = ('ip-dst', 'ip', 'ip_address') else: @@ -174,7 +175,11 @@ def handler(q=False): if not request.get('config') or not request['config'].get('apikey'): misperrors['error'] = "A VirusTotal api key is required for this module." return misperrors + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message}, which should contain at least a type, a value and an uuid.'} attribute = request['attribute'] + if attribute['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} query_type, to_call = misp_type_mapping[attribute['type']] parser = to_call(request['config']['apikey'], attribute) query_result = parser.get_query_result(query_type) diff --git a/misp_modules/modules/expansion/xforceexchange.py b/misp_modules/modules/expansion/xforceexchange.py index 7999ce2..936917f 100644 --- a/misp_modules/modules/expansion/xforceexchange.py +++ b/misp_modules/modules/expansion/xforceexchange.py @@ -1,6 +1,7 @@ import requests import json import sys +from . import check_input_attribute, standard_error_message from collections import defaultdict from pymisp import MISPAttribute, MISPEvent, MISPObject from requests.auth import HTTPBasicAuth @@ -160,6 +161,10 @@ def handler(q=False): return misperrors key = request["config"]["apikey"] password = request['config']['apipassword'] + if not request.get('attribute') or not check_input_attribute(request['attribute']): + return {'error': f'{standard_error_message} which should contain at least a type, a value and an uuid.'} + if request['attribute']['type'] not in mispattributes['input']: + return {'error': 'Unsupported attribute type.'} parser = XforceExchange(request['attribute'], key, password) parser.parse() return parser.get_result() diff --git a/misp_modules/modules/import_mod/email_import.py b/misp_modules/modules/import_mod/email_import.py index 114f8c9..7453dcd 100644 --- a/misp_modules/modules/import_mod/email_import.py +++ b/misp_modules/modules/import_mod/email_import.py @@ -42,7 +42,7 @@ def handler(q=False): # request data is always base 64 byte encoded data = base64.b64decode(request["data"]) - email_object = EMailObject(pseudofile=BytesIO(data), attach_original_mail=True, standalone=False) + email_object = EMailObject(pseudofile=BytesIO(data), attach_original_email=True, standalone=False) # Check if we were given a configuration config = request.get("config", {}) diff --git a/tests/test_expansions.py b/tests/test_expansions.py index b853c25..1aa0f7a 100644 --- a/tests/test_expansions.py +++ b/tests/test_expansions.py @@ -97,9 +97,16 @@ class TestExpansions(unittest.TestCase): self.assertEqual(self.get_errors(response), 'An API key for APIVoid is required.') def test_bgpranking(self): - query = {"module": "bgpranking", "AS": "13335"} + query = { + "module": "bgpranking", + "attribute": { + "type": "AS", + "value": "13335", + "uuid": "ea89a33b-4ab7-4515-9f02-922a0bee333d" + } + } response = self.misp_modules_post(query) - self.assertEqual(self.get_values(response)['response']['asn_description'], 'CLOUDFLARENET, US') + self.assertEqual(self.get_object(response), 'asn') def test_btc_steroids(self): query = {"module": "btc_steroids", "btc": "1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA"} @@ -229,11 +236,24 @@ class TestExpansions(unittest.TestCase): self.assertEqual(to_check, 'OK (Not Found)', response) def test_greynoise(self): - query = {"module": "greynoise", "ip-dst": "1.1.1.1"} - response = self.misp_modules_post(query) - value = self.get_values(response) - if value != 'GreyNoise API not accessible (HTTP 429)': - self.assertTrue(value.startswith('{"ip":"1.1.1.1","status":"ok"')) + module_name = 'greynoise' + query = {"module": module_name, "ip-dst": "1.1.1.1"} + if module_name in self.configs: + query['config'] = self.configs[module_name] + response = self.misp_modules_post(query) + try: + self.assertEqual(self.get_values(response), 'This IP is commonly spoofed in Internet-scan activity') + except Exception: + self.assertIn( + self.get_errors(reponse), + ( + "Unauthorized. Please check your API key.", + "Too many requests. You've hit the rate-limit." + ) + ) + else: + response = self.misp_modules_post(query) + self.assertEqual(self.get_errors(response), 'Missing Greynoise API key.') def test_ipasn(self): query = {"module": "ipasn",