From ea14fb4a525cbb496e1ab85fc979af030d78edb1 Mon Sep 17 00:00:00 2001 From: Christophe Vandeplas Date: Sat, 10 Aug 2024 09:36:56 +0200 Subject: [PATCH] fix: [doc] align static documentation pages --- README.md | 428 +++++++++++++++++++---------- documentation/mkdocs/contribute.md | 61 ++-- documentation/mkdocs/index.md | 106 ++++--- documentation/mkdocs/install.md | 35 ++- 4 files changed, 405 insertions(+), 225 deletions(-) diff --git a/README.md b/README.md index c9e6b619..7b347926 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # MISP modules -[![Python package](https://github.com/MISP/misp-modules/actions/workflows/python-package.yml/badge.svg)](https://github.com/MISP/misp-modules/actions/workflows/python-package.yml)[![Coverage Status](https://coveralls.io/repos/github/MISP/misp-modules/badge.svg?branch=main)](https://coveralls.io/github/MISP/misp-modules?branch=main) +[![Build status](https://github.com/MISP/misp-modules/actions/workflows/python-package.yml/badge.svg)](https://github.com/MISP/misp-modules/actions/workflows/python-package.yml)[![Coverage Status](https://coveralls.io/repos/github/MISP/misp-modules/badge.svg?branch=main)](https://coveralls.io/github/MISP/misp-modules?branch=main) [![codecov](https://codecov.io/gh/MISP/misp-modules/branch/main/graph/badge.svg)](https://codecov.io/gh/MISP/misp-modules) MISP modules are autonomous modules that can be used to extend [MISP](https://github.com/MISP/MISP) for new services such as expansion, import, export and workflow action. @@ -15,165 +15,227 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj ## Existing MISP modules ### Expansion modules -* [apiosintDS](misp_modules/modules/expansion/apiosintds.py) - a hover and expansion module to query the [OSINT.digitalside.it](https://osint.digitalside.it) API. [Documentation](https://apiosintds.readthedocs.io/en/latest/userguidemisp.html). -* [API Void](misp_modules/modules/expansion/apivoid.py) - an expansion and hover module to query API Void with a domain attribute. -* [AssemblyLine submit](misp_modules/modules/expansion/assemblyline_submit.py) - an expansion module to submit samples and urls to AssemblyLine. -* [AssemblyLine query](misp_modules/modules/expansion/assemblyline_query.py) - an expansion module to query AssemblyLine and parse the full submission report. -* [Backscatter.io](misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations. -* [BGP Ranking](misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description and its ranking and position in BGP Ranking. -* [RansomcoinDB check](misp_modules/modules/expansion/ransomcoindb.py) - An expansion hover module to query the [ransomcoinDB](https://ransomcoindb.concinnity-risks.com): it contains mapping between BTC addresses and malware hashes. Enrich MISP by querying for BTC -> hash or hash -> BTC addresses. -* [BTC scam check](misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused. -* [BTC transactions](misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP. -* [Censys-enrich](misp_modules/modules/expansion/censys_enrich.py) - An expansion and module to retrieve information from censys.io about a particular IP or certificate. -* [CIRCL Passive DNS](misp_modules/modules/expansion/circl_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information. -* [CIRCL Passive SSL](misp_modules/modules/expansion/circl_passivessl.py) - a hover and expansion module to expand IP addresses with the X.509 certificate(s) seen. -* [countrycode](misp_modules/modules/expansion/countrycode.py) - a hover module to tell you what country a URL belongs to. -* [CrowdSec](misp_modules/modules/expansion/crowdsec.py) - a hover module to expand using CrowdSec's CTI API. -* [CrowdStrike Falcon](misp_modules/modules/expansion/crowdstrike_falcon.py) - an expansion module to expand using CrowdStrike Falcon Intel Indicator API. -* [CPE](misp_modules/modules/expansion/cpe.py) - An expansion module to query the CVE Search API with a cpe code, to get its related vulnerabilities. -* [CVE](misp_modules/modules/expansion/cve.py) - a hover module to give more information about a vulnerability (CVE). -* [CVE advanced](misp_modules/modules/expansion/cve_advanced.py) - An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE). -* [Cuckoo submit](misp_modules/modules/expansion/cuckoo_submit.py) - A hover module to submit malware sample, url, attachment, domain to Cuckoo Sandbox. -* [Cytomic Orion](misp_modules/modules/expansion/cytomic_orion.py) - An expansion module to enrich attributes in MISP and share indicators of compromise with Cytomic Orion. -* [DBL Spamhaus](misp_modules/modules/expansion/dbl_spamhaus.py) - a hover module to check Spamhaus DBL for a domain name. -* [DNS](misp_modules/modules/expansion/dns.py) - a simple module to resolve MISP attributes like hostname and domain to expand IP addresses attributes. -* [docx-enrich](misp_modules/modules/expansion/docx_enrich.py) - an enrichment module to get text out of Word document into MISP (using free-text parser). -* [DomainTools](misp_modules/modules/expansion/domaintools.py) - a hover and expansion module to get information from [DomainTools](http://www.domaintools.com/) whois. -* [EQL](misp_modules/modules/expansion/eql.py) - an expansion module to generate event query language (EQL) from an attribute. [Event Query Language](https://eql.readthedocs.io/en/latest/) -* [EUPI](misp_modules/modules/expansion/eupi.py) - a hover and expansion module to get information about an URL from the [Phishing Initiative project](https://phishing-initiative.eu/?lang=en). -* [Farsight DNSDB Passive DNS](misp_modules/modules/expansion/farsight_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information. -* [GeoIP](misp_modules/modules/expansion/geoip_country.py) - a hover and expansion module to get GeoIP information from geolite/maxmind. -* [GeoIP_City](misp_modules/modules/expansion/geoip_city.py) - a hover and expansion module to get GeoIP City information from geolite/maxmind. -* [GeoIP_ASN](misp_modules/modules/expansion/geoip_asn.py) - a hover and expansion module to get GeoIP ASN information from geolite/maxmind. +* [apiosintDS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/apiosintds.py) - a hover and expansion module to query the [OSINT.digitalside.it](https://osint.digitalside.it) API. [Documentation](https://apiosintds.readthedocs.io/en/latest/userguidemisp.html). +* [API Void](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/apivoid.py) - an expansion and hover module to query API Void with a domain attribute. +* [AssemblyLine submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/assemblyline_submit.py) - an expansion module to submit samples and urls to AssemblyLine. +* [AssemblyLine query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/assemblyline_query.py) - an expansion module to query AssemblyLine and parse the full submission report. +* [Backscatter.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations. +* [BGP Ranking](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description, its history, and position in BGP Ranking. +* [RansomcoinDB check](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ransomcoindb.py) - An expansion hover module to query the [ransomcoinDB](https://ransomcoindb.concinnity-risks.com): it contains mapping between BTC addresses and malware hashes. Enrich MISP by querying for BTC -> hash or hash -> BTC addresses. +* [BTC scam check](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused. +* [BTC transactions](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP. +* [Censys-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/censys_enrich.py) - An expansion and module to retrieve information from censys.io about a particular IP or certificate. +* [CIRCL Passive DNS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/circl_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information. +* [CIRCL Passive SSL](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/circl_passivessl.py) - a hover and expansion module to expand IP addresses with the X.509 certificates seen. +* [countrycode](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/countrycode.py) - a hover module to tell you what country a URL belongs to. +* [CrowdSec](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/crowdsec.py) - a hover module to expand using CrowdSec's CTI API. +* [CrowdStrike Falcon](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/crowdstrike_falcon.py) - an expansion module to expand using CrowdStrike Falcon Intel Indicator API. +* [CPE](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cpe.py) - An expansion module to query the CVE Search API with a cpe code, to get its related vulnerabilities. +* [CVE](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cve.py) - a hover module to give more information about a vulnerability (CVE). +* [CVE advanced](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cve_advanced.py) - An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE). +* [Cuckoo submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cuckoo_submit.py) - A hover module to submit malware sample, url, attachment, domain to Cuckoo Sandbox. +* [Cytomic Orion](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cytomic_orion.py) - An expansion module to enrich attributes in MISP and share indicators of compromise with Cytomic Orion. +* [DBL Spamhaus](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/dbl_spamhaus.py) - a hover module to check Spamhaus DBL for a domain name. +* [DNS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/dns.py) - a simple module to resolve MISP attributes like hostname and domain to expand IP addresses attributes. +* [docx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/docx_enrich.py) - an enrichment module to get text out of Word document into MISP (using free-text parser). +* [DomainTools](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/domaintools.py) - a hover and expansion module to get information from [DomainTools](http://www.domaintools.com/) whois. +* [EQL](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/eql.py) - an expansion module to generate event query language (EQL) from an attribute. [Event Query Language](https://eql.readthedocs.io/en/latest/) +* [EUPI](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/eupi.py) - a hover and expansion module to get information about an URL from the [Phishing Initiative project](https://phishing-initiative.eu/?lang=en). +* [Farsight DNSDB Passive DNS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/farsight_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information. +* [GeoIP](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/geoip_country.py) - a hover and expansion module to get GeoIP information from geolite/maxmind. +* [GeoIP_City](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/geoip_city.py) - a hover and expansion module to get GeoIP City information from geolite/maxmind. +* [GeoIP_ASN](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/geoip_asn.py) - a hover and expansion module to get GeoIP ASN information from geolite/maxmind. * [Google Threat Intelligence] (https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/google_threat_intelligence.py) - An expansion module to have the observable's threat score assessed by Google Threat Intelligence. -* [GreyNoise](misp_modules/modules/expansion/greynoise.py) - a hover and expansion module to get IP and CVE information from GreyNoise. -* [hashdd](misp_modules/modules/expansion/hashdd.py) - a hover module to check file hashes against [hashdd.com](http://www.hashdd.com) including NSLR dataset. -* [Hashlookup](misp_modules/modules/expansion/hashlookup.py) - An expansion module to enrich a file hash with hashlookup.circl.lu services (NSRL and other sources) -* [hibp](misp_modules/modules/expansion/hibp.py) - a hover module to lookup against Have I Been Pwned? -* [html_to_markdown](misp_modules/modules/expansion/html_to_markdown.py) - Simple HTML to markdown converter -* [HYAS Insight](misp_modules/modules/expansion/hyasinsight.py) - a hover and expansion module to get information from [HYAS Insight](https://www.hyas.com/hyas-insight). -* [intel471](misp_modules/modules/expansion/intel471.py) - an expansion module to get info from [Intel471](https://intel471.com). -* [IP2Location.io](misp_modules/modules/expansion/ip2locationio.py) - an expansion module to get additional information on an IP address using the IP2Location.io API -* [IPASN](misp_modules/modules/expansion/ipasn.py) - a hover and expansion to get the BGP ASN of an IP address. -* [ipinfo.io](misp_modules/modules/expansion/ipinfo.py) - an expansion module to get additional information on an IP address using the ipinfo.io API -* [iprep](misp_modules/modules/expansion/iprep.py) - an expansion module to get IP reputation from packetmail.net. -* [Joe Sandbox submit](misp_modules/modules/expansion/joesandbox_submit.py) - Submit files and URLs to Joe Sandbox. -* [Joe Sandbox query](misp_modules/modules/expansion/joesandbox_query.py) - Query Joe Sandbox with the link of an analysis and get the parsed data. -* [Lastline submit](misp_modules/modules/expansion/lastline_submit.py) - Submit files and URLs to Lastline. -* [Lastline query](misp_modules/modules/expansion/lastline_query.py) - Query Lastline with the link to an analysis and parse the report. -* [macaddress.io](misp_modules/modules/expansion/macaddress_io.py) - a hover module to retrieve vendor details and other information regarding a given MAC address or an OUI from [MAC address Vendor Lookup](https://macaddress.io). See [integration tutorial here](https://macaddress.io/integrations/MISP-module). -* [macvendors](misp_modules/modules/expansion/macvendors.py) - a hover module to retrieve mac vendor information. -* [MALWAREbazaar](misp_modules/modules/expansion/malwarebazaar.py) - an expansion module to query MALWAREbazaar with some payload. -* [McAfee MVISION Insights](misp_modules/modules/expansion/mcafee_insights_enrich.py) - an expansion module enrich IOCs with McAfee MVISION Insights. -* [Mmdb server lookup](misp_modules/modules/expansion/mmdb_lookup.py) - an expansion module to enrich an ip with geolocation information from an mmdb server such as ip.circl.lu. -* [ocr-enrich](misp_modules/modules/expansion/ocr_enrich.py) - an enrichment module to get OCRized data from images into MISP. -* [ods-enrich](misp_modules/modules/expansion/ods_enrich.py) - an enrichment module to get text out of OpenOffice spreadsheet document into MISP (using free-text parser). -* [odt-enrich](misp_modules/modules/expansion/odt_enrich.py) - an enrichment module to get text out of OpenOffice document into MISP (using free-text parser). -* [onyphe](misp_modules/modules/expansion/onyphe.py) - a modules to process queries on Onyphe. -* [onyphe_full](misp_modules/modules/expansion/onyphe_full.py) - a modules to process full queries on Onyphe. -* [OTX](misp_modules/modules/expansion/otx.py) - an expansion module for [OTX](https://otx.alienvault.com/). -* [passivetotal](misp_modules/modules/expansion/passivetotal.py) - a [passivetotal](https://www.passivetotal.org/) module that queries a number of different PassiveTotal datasets. -* [pdf-enrich](misp_modules/modules/expansion/pdf_enrich.py) - an enrichment module to extract text from PDF into MISP (using free-text parser). -* [pptx-enrich](misp_modules/modules/expansion/pptx_enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser). -* [qrcode](misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values. -* [rbl](misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute. -* [recordedfuture](misp_modules/modules/expansion/recordedfuture.py) - a hover and expansion module for enriching MISP attributes with threat intelligence from Recorded Future. -* [reversedns](misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes. -* [securitytrails](misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/). -* [shodan](misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module. -* [Sigma queries](misp_modules/modules/expansion/sigma_queries.py) - Experimental expansion module querying a sigma rule to convert it into all the available SIEM signatures. -* [Sigma syntax validator](misp_modules/modules/expansion/sigma_syntax_validator.py) - Sigma syntax validator. -* [Socialscan](misp_modules/modules/expansion/socialscan.py) - a hover module to check if an email address or a username is used on different online platforms, using the [socialscan](https://github.com/iojw/socialscan) python library -* [SophosLabs Intelix](misp_modules/modules/expansion/sophoslabs_intelix.py) - SophosLabs Intelix is an API for Threat Intelligence and Analysis (free tier available). [SophosLabs](https://aws.amazon.com/marketplace/pp/B07SLZPMCS) -* [sourcecache](misp_modules/modules/expansion/sourcecache.py) - a module to cache a specific link from a MISP instance. -* [stairwell](misp_modules/modules/expansion/stairwell.py) - an expansion module to enrich hash observables with the Stairwell API -* [STIX2 pattern syntax validator](misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax. -* [ThreatCrowd](misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/). -* [threatminer](misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/). -* [TruSTAR Enrich](misp_modules/modules/expansion/trustar_enrich.py) - an expansion module to enrich MISP data with [TruSTAR](https://www.trustar.co/). -* [urlhaus](misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url. -* [urlscan](misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io). -* [variotdbs](misp_modules/modules/expansion/variotdbs.py) - an expansion module to query the [VARIoT db](https://www.variotdbs.pl) API to get more information about a Vulnerability -* [virustotal](misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://docs.virustotal.com/reference/overview)) -* [virustotal_public](misp_modules/modules/expansion/virustotal_public.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a public key and a low request rate limit. (More details about the API: [here](https://docs.virustotal.com/reference/overview)) -* [VMray](misp_modules/modules/expansion/vmray_submit.py) - a module to submit a sample to VMray. -* [VMware NSX](misp_modules/modules/expansion/vmware_nsx.py) - a module to enrich a file or URL with VMware NSX Defender. -* [VulnDB](misp_modules/modules/expansion/vulndb.py) - a module to query [VulnDB](https://www.riskbasedsecurity.com/). -* [Vulners](misp_modules/modules/expansion/vulners.py) - an expansion module to expand information about CVEs using Vulners API. -* [Vysion](misp_modules/modules/expansion/vysion.py) - an expansion module to add dark web intelligence using Vysion API. -* [whois](misp_modules/modules/expansion/whois.py) - a module to query a local instance of [uwhois](https://github.com/rafiot/uwhoisd). -* [whoisfreaks](misp_modules/modules/expansion/whoisfreaks.py) - An expansion module for [whoisfreaks](https://whoisfreaks.com/) that will provide an enriched analysis of the provided domain, including WHOIS and DNS information. -* [wikidata](misp_modules/modules/expansion/wiki.py) - a [wikidata](https://www.wikidata.org) expansion module. -* [xforce](misp_modules/modules/expansion/xforceexchange.py) - an IBM X-Force Exchange expansion module. -* [xlsx-enrich](misp_modules/modules/expansion/xlsx_enrich.py) - an enrichment module to get text out of an Excel document into MISP (using free-text parser). -* [YARA query](misp_modules/modules/expansion/yara_query.py) - a module to create YARA rules from single hash attributes. -* [YARA syntax validator](misp_modules/modules/expansion/yara_syntax_validator.py) - YARA syntax validator. +* [GreyNoise](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/greynoise.py) - a hover and expansion module to get IP and CVE information from GreyNoise. +* [hashdd](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hashdd.py) - a hover module to check file hashes against [hashdd.com](http://www.hashdd.com) including NSLR dataset. +* [Hashlookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hashlookup.py) - An expansion module to enrich a file hash with hashlookup.circl.lu services (NSRL and other sources) +* [hibp](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hibp.py) - a hover module to lookup against Have I Been Pwned? +* [html_to_markdown](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/html_to_markdown.py) - Simple HTML to markdown converter +* [HYAS Insight](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hyasinsight.py) - a hover and expansion module to get information from [HYAS Insight](https://www.hyas.com/hyas-insight). +* [intel471](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/intel471.py) - an expansion module to get info from [Intel471](https://intel471.com). +* [IP2Location.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ip2locationio.py) - an expansion module to get additional information on an IP address using the IP2Location.io API +* [IPASN](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ipasn.py) - a hover and expansion to get the BGP ASN of an IP address. +* [ipinfo.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ipinfo.py) - an expansion module to get additional information on an IP address using the ipinfo.io API +* [iprep](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/iprep.py) - an expansion module to get IP reputation from packetmail.net. +* [Joe Sandbox submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/joesandbox_submit.py) - Submit files and URLs to Joe Sandbox. +* [Joe Sandbox query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/joesandbox_query.py) - Query Joe Sandbox with the link of an analysis and get the parsed data. +* [Lastline submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/lastline_submit.py) - Submit files and URLs to Lastline. +* [Lastline query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/lastline_query.py) - Query Lastline with the link to an analysis and parse the report. +* [macaddress.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/macaddress_io.py) - a hover module to retrieve vendor details and other information regarding a given MAC address or an OUI from [MAC address Vendor Lookup](https://macaddress.io). See [integration tutorial here](https://macaddress.io/integrations/MISP-module). +* [macvendors](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/macvendors.py) - a hover module to retrieve mac vendor information. +* [MALWAREbazaar](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/malwarebazaar.py) - an expansion module to query MALWAREbazaar with some payload. +* [McAfee MVISION Insights](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/mcafee_insights_enrich.py) - an expansion module enrich IOCs with McAfee MVISION Insights. +* [Mmdb server lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/mmdb_lookup.py) - an expansion module to enrich an ip with geolocation information from an mmdb server such as ip.circl.lu. +* [ocr-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ocr_enrich.py) - an enrichment module to get OCRized data from images into MISP. +* [ods-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ods_enrich.py) - an enrichment module to get text out of OpenOffice spreadsheet document into MISP (using free-text parser). +* [odt-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/odt_enrich.py) - an enrichment module to get text out of OpenOffice document into MISP (using free-text parser). +* [onyphe](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/onyphe.py) - a modules to process queries on Onyphe. +* [onyphe_full](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/onyphe_full.py) - a modules to process full queries on Onyphe. +* [OTX](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/otx.py) - an expansion module for [OTX](https://otx.alienvault.com/). +* [passivetotal](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/passivetotal.py) - a [passivetotal](https://www.passivetotal.org/) module that queries a number of different PassiveTotal datasets. +* [pdf-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/pdf_enrich.py) - an enrichment module to extract text from PDF into MISP (using free-text parser). +* [pptx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/pptx_enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser). +* [qrcode](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values. +* [rbl](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute. +* [recordedfuture](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/recordedfuture.py) - a hover and expansion module for enriching MISP attributes with threat intelligence from Recorded Future. +* [reversedns](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes. +* [securitytrails](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/). +* [shodan](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module. +* [Sigma queries](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sigma_queries.py) - Experimental expansion module querying a sigma rule to convert it into all the available SIEM signatures. +* [Sigma syntax validator](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sigma_syntax_validator.py) - Sigma syntax validator. +* [Socialscan](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/socialscan.py) - a hover module to check if an email address or a username is used on different online platforms, using the [socialscan](https://github.com/iojw/socialscan) python library +* [SophosLabs Intelix](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sophoslabs_intelix.py) - SophosLabs Intelix is an API for Threat Intelligence and Analysis (free tier available). [SophosLabs](https://aws.amazon.com/marketplace/pp/B07SLZPMCS) +* [sourcecache](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sourcecache.py) - a module to cache a specific link from a MISP instance. +* [stairwell](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/stairwell.py) - an expansion module to enrich hash observables with the Stairwell API +* [STIX2 pattern syntax validator](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax. +* [ThreatCrowd](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/). +* [threatminer](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/). +* [TruSTAR Enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/trustar_enrich.py) - an expansion module to enrich MISP data with [TruSTAR](https://www.trustar.co/). +* [urlhaus](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url. +* [urlscan](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io). +* [variotdbs](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/variotdbs.py) - an expansion module to query the [VARIoT db](https://www.variotdbs.pl) API to get more information about a Vulnerability +* [virustotal](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://docs.virustotal.com/reference/overview)) +* [virustotal_public](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/virustotal_public.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a public key and a low request rate limit. (More details about the API: [here](https://docs.virustotal.com/reference/overview)) +* [VMray](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vmray_submit.py) - a module to submit a sample to VMray. +* [VMware NSX](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vmware_nsx.py) - a module to enrich a file or URL with VMware NSX Defender. +* [VulnDB](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vulndb.py) - a module to query [VulnDB](https://www.riskbasedsecurity.com/). +* [Vulners](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vulners.py) - an expansion module to expand information about CVEs using Vulners API. +* [Vysion](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vysion.py) - an expansion module to add dark web intelligence using Vysion API. +* [whois](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whois.py) - a module to query a local instance of [uwhois](https://github.com/rafiot/uwhoisd). +* [whoisfreaks](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whoisfreaks.py) - An expansion module for [whoisfreaks](https://whoisfreaks.com/) that will provide an enriched analysis of the provided domain, including WHOIS and DNS information. +* [wikidata](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/wiki.py) - a [wikidata](https://www.wikidata.org) expansion module. +* [xforce](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/xforceexchange.py) - an IBM X-Force Exchange expansion module. +* [xlsx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/xlsx_enrich.py) - an enrichment module to get text out of an Excel document into MISP (using free-text parser). +* [YARA query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/yara_query.py) - a module to create YARA rules from single hash attributes. +* [YARA syntax validator](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/yara_syntax_validator.py) - YARA syntax validator. ### Export modules -* [CEF](misp_modules/modules/export_mod/cef_export.py) - module to export Common Event Format (CEF). -* [Cisco FireSight Manager ACL rule](misp_modules/modules/export_mod/cisco_firesight_manager_ACL_rule_export.py) - module to export as rule for the Cisco FireSight manager ACL. -* [GoAML export](misp_modules/modules/export_mod/goamlexport.py) - module to export in [GoAML format](http://goaml.unodc.org/goaml/en/index.html). -* [Lite Export](misp_modules/modules/export_mod/liteexport.py) - module to export a lite event. -* [PDF export](misp_modules/modules/export_mod/pdfexport.py) - module to export an event in PDF. -* [Mass EQL Export](misp_modules/modules/export_mod/mass_eql_export.py) - module to export applicable attributes from an event to a mass EQL query. -* [Nexthink query format](misp_modules/modules/export_mod/nexthinkexport.py) - module to export in Nexthink query format. -* [osquery](misp_modules/modules/export_mod/osqueryexport.py) - module to export in [osquery](https://osquery.io/) query format. -* [ThreatConnect](misp_modules/modules/export_mod/threat_connect_export.py) - module to export in ThreatConnect CSV format. -* [ThreatStream](misp_modules/modules/export_mod/threatStream_misp_export.py) - module to export in ThreatStream format. -* [VirusTotal Graph](misp_modules/modules/export_mod/vt_graph.py) - Module to create a VirusTotal graph out of an event. +* [CEF](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cef_export.py) - module to export Common Event Format (CEF). +* [Cisco FireSight Manager ACL rule](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cisco_firesight_manager_ACL_rule_export.py) - module to export as rule for the Cisco FireSight manager ACL. +* [GoAML export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/goamlexport.py) - module to export in [GoAML format](http://goaml.unodc.org/goaml/en/index.html). +* [Lite Export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/liteexport.py) - module to export a lite event. +* [PDF export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/pdfexport.py) - module to export an event in PDF. +* [Mass EQL Export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/mass_eql_export.py) - module to export applicable attributes from an event to a mass EQL query. +* [Nexthink query format](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/nexthinkexport.py) - module to export in Nexthink query format. +* [osquery](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/osqueryexport.py) - module to export in [osquery](https://osquery.io/) query format. +* [ThreatConnect](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/threat_connect_export.py) - module to export in ThreatConnect CSV format. +* [ThreatStream](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/threatStream_misp_export.py) - module to export in ThreatStream format. +* [VirusTotal Graph](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/vt_graph.py) - Module to create a VirusTotal graph out of an event. ### Import modules -* [CSV import](misp_modules/modules/import_mod/csvimport.py) - Customizable CSV import module. -* [Cuckoo JSON](misp_modules/modules/import_mod/cuckooimport.py) - Cuckoo JSON import. -* [Email Import](misp_modules/modules/import_mod/email_import.py) - Email import module for MISP to import basic metadata. -* [GoAML import](misp_modules/modules/import_mod/goamlimport.py) - Module to import [GoAML](http://goaml.unodc.org/goaml/en/index.html) XML format. -* [Joe Sandbox import](misp_modules/modules/import_mod/joe_import.py) - Parse data from a Joe Sandbox json report. -* [Lastline import](misp_modules/modules/import_mod/lastline_import.py) - Module to import Lastline analysis reports. -* [OCR](misp_modules/modules/import_mod/ocr.py) - Optical Character Recognition (OCR) module for MISP to import attributes from images, scan or faxes. -* [OpenIOC](misp_modules/modules/import_mod/openiocimport.py) - OpenIOC import based on PyMISP library. -* [ThreatAnalyzer](misp_modules/modules/import_mod/threatanalyzer_import.py) - An import module to process ThreatAnalyzer archive.zip/analysis.json sandbox exports. -* [VMRay](misp_modules/modules/import_mod/vmray_import.py) - An import module to process VMRay export. +* [CSV import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/csvimport.py) - Customizable CSV import module. +* [Cuckoo JSON](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/cuckooimport.py) - Cuckoo JSON import. +* [Email Import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/email_import.py) - Email import module for MISP to import basic metadata. +* [GoAML import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/goamlimport.py) - Module to import [GoAML](http://goaml.unodc.org/goaml/en/index.html) XML format. +* [Joe Sandbox import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/joe_import.py) - Parse data from a Joe Sandbox json report. +* [Lastline import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/lastline_import.py) - Module to import Lastline analysis reports. +* [OCR](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/ocr.py) - Optical Character Recognition (OCR) module for MISP to import attributes from images, scan or faxes. +* [OpenIOC](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/openiocimport.py) - OpenIOC import based on PyMISP library. +* [ThreatAnalyzer](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/threatanalyzer_import.py) - An import module to process ThreatAnalyzer archive.zip/analysis.json sandbox exports. +* [VMRay](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/vmray_import.py) - An import module to process VMRay export. ## How to install and start MISP modules in a Python virtualenv? (recommended) ***Be sure to run the latest version of `pip`***. To install the latest version of pip, `pip install --upgrade pip` will do the job. ~~~~bash -sudo apt-get install python3-dev python3-pip libpq5 libjpeg-dev tesseract-ocr libpoppler-cpp-dev imagemagick virtualenv libopencv-dev zbar-tools libzbar0 libzbar-dev libfuzzy-dev build-essential -y -sudo -u www-data virtualenv -p python3 /var/www/MISP/venv +SUDO_WWW="sudo -u www-data" + +sudo apt-get install -y \ + python3-dev \ + python3-pip \ + git \ + libpq5 \ + libjpeg-dev \ + tesseract-ocr \ + libpoppler-cpp-dev \ + imagemagick virtualenv \ + libopencv-dev \ + zbar-tools \ + libzbar0 \ + libzbar-dev \ + libfuzzy-dev \ + libcaca-dev \ + build-essential + +# BEGIN with virtualenv: +$SUDO_WWW virtualenv -p python3 /var/www/MISP/venv +# END with virtualenv + cd /usr/local/src/ -sudo chown -R www-data: . -sudo -u www-data git clone https://github.com/MISP/misp-modules.git -cd misp-modules -sudo -u www-data /var/www/MISP/venv/bin/pip install -I -r REQUIREMENTS -sudo -u www-data /var/www/MISP/venv/bin/pip install . +# Ideally you add your user to the staff group and make /usr/local/src group writeable, below follows an example with user misp +sudo adduser misp staff +sudo chmod 2775 /usr/local/src +sudo chown root:staff /usr/local/src +git clone https://github.com/MISP/misp-modules.git +git clone git://github.com/stricaud/faup.git faup +git clone git://github.com/stricaud/gtcaca.git gtcaca + +# Install gtcaca/faup +cd gtcaca +mkdir -p build +cd build +cmake .. && make +sudo make install +cd ../../faup +mkdir -p build +cd build +cmake .. && make +sudo make install +sudo ldconfig + +cd ../../misp-modules + +# BEGIN with virtualenv: +$SUDO_WWW /var/www/MISP/venv/bin/pip install -I -r REQUIREMENTS +$SUDO_WWW /var/www/MISP/venv/bin/pip install . +# END with virtualenv + +# BEGIN without virtualenv: +sudo pip install -I -r REQUIREMENTS +sudo pip install . +# END without virtualenv + # Start misp-modules as a service sudo cp etc/systemd/system/misp-modules.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now misp-modules -sudo service misp-modules start #or -/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 & #to start the modules +sudo service misp-modules start # or +/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s & # to start the modules manually ~~~~ ## How to install and start MISP modules on RHEL-based distributions ? -As of this writing, the official RHEL repositories only contain Ruby 2.0.0 and Ruby 2.1 or higher is required. As such, this guide installs Ruby 2.2 from the [SCL](https://access.redhat.com/documentation/en-us/red_hat_software_collections/3/html/3.2_release_notes/chap-installation#sect-Installation-Subscribe) repository. + +As of this writing, the official RHEL repositories only contain Ruby 2.0.0 and Ruby 2.1 or higher is required. As such, this guide installs Ruby 2.2 from the [SCL](https://access.redhat.com/documentation/en-us/red_hat_software_collections/3/html/3.2_release_notes/chap-installation#sect-Installation-Subscribe) repository. ~~~~bash -sudo yum install rh-python36 rh-ruby22 -sudo yum install openjpeg-devel -sudo yum install rubygem-rouge rubygem-asciidoctor zbar-devel opencv-devel gcc-c++ pkgconfig poppler-cpp-devel python-devel redhat-rpm-config +SUDO_WWW="sudo -u apache" +sudo yum install \ + rh-python36 \ + rh-ruby22 \ + openjpeg-devel \ + rubygem-rouge \ + rubygem-asciidoctor \ + zbar-devel \ + opencv-devel \ + gcc-c++ \ + pkgconfig \ + poppler-cpp-devel \ + python-devel \ + redhat-rpm-config cd /var/www/MISP -git clone https://github.com/MISP/misp-modules.git +$SUDO_WWW git clone https://github.com/MISP/misp-modules.git cd misp-modules -sudo -u apache /usr/bin/scl enable rh-python36 "virtualenv -p python3 /var/www/MISP/venv" -sudo -u apache /var/www/MISP/venv/bin/pip install -U -I -r REQUIREMENTS -sudo -u apache /var/www/MISP/venv/bin/pip install -U . +$SUDO_WWW /usr/bin/scl enable rh-python36 "virtualenv -p python3 /var/www/MISP/venv" +$SUDO_WWW /var/www/MISP/venv/bin/pip install -U -I -r REQUIREMENTS +$SUDO_WWW /var/www/MISP/venv/bin/pip install -U . ~~~~ Create the service file /etc/systemd/system/misp-modules.service : -~~~~ + +~~~~bash echo "[Unit] Description=MISP's modules After=misp-workers.service @@ -182,7 +244,7 @@ After=misp-workers.service Type=simple User=apache Group=apache -ExecStart=/usr/bin/scl enable rh-python36 rh-ruby22 '/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1' +ExecStart=/usr/bin/scl enable rh-python36 rh-ruby22 '/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s' Restart=always RestartSec=10 @@ -197,9 +259,86 @@ systemctl daemon-reload systemctl enable --now misp-modules ~~~~ +## How to use an MISP modules Docker container + +### Docker build + +~~~~bash +docker build -t misp-modules \ + --build-arg BUILD_DATE=$(date -u +"%Y-%m-%d") \ + docker/ +~~~~ + +### Docker run + +~~~~bash +# Start Redis +docker run --rm -d --name=misp-redis redis:alpine +# Start MISP-modules +docker run \ + --rm -d --name=misp-modules \ + -e REDIS_BACKEND=misp-redis \ + -e REDIS_PORT="6379" \ + -e REDIS_PW="" \ + -e REDIS_DATABASE="245" \ + -e MISP_MODULES_DEBUG="false" \ + dcso/misp-dockerized-misp-modules +~~~~ + +### Docker-compose + +~~~~yml +services: + misp-modules: + # https://hub.docker.com/r/dcso/misp-dockerized-misp-modules + image: dcso/misp-dockerized-misp-modules:3 + + # Local image: + #image: misp-modules + #build: + # context: docker/ + + environment: + # Redis + REDIS_BACKEND: misp-redis + REDIS_PORT: "6379" + REDIS_DATABASE: "245" + # System PROXY (OPTIONAL) + http_proxy: + https_proxy: + no_proxy: 0.0.0.0 + # Timezone (OPTIONAL) + TZ: Europe/Berlin + # MISP-Modules (OPTIONAL) + MISP_MODULES_DEBUG: "false" + # Logging options (OPTIONAL) + LOG_SYSLOG_ENABLED: "no" + misp-redis: + # https://hub.docker.com/_/redis or alternative https://hub.docker.com/r/dcso/misp-dockerized-redis/ + image: redis:alpine +~~~~ + +## Install misp-module on an offline instance. +First, you need to grab all necessary packages for example like this : + +Use pip wheel to create an archive +~~~ +mkdir misp-modules-offline +pip3 wheel -r REQUIREMENTS shodan --wheel-dir=./misp-modules-offline +tar -cjvf misp-module-bundeled.tar.bz2 ./misp-modules-offline/* +~~~ +On offline machine : +~~~ +mkdir misp-modules-bundle +tar xvf misp-module-bundeled.tar.bz2 -C misp-modules-bundle +cd misp-modules-bundle +ls -1|while read line; do sudo pip3 install --force-reinstall --ignore-installed --upgrade --no-index --no-deps ${line};done +~~~ +Next you can follow standard install procedure. + ## How to add your own MISP modules? -Create your module in [misp_modules/modules/expansion/](misp_modules/modules/expansion/), [misp_modules/modules/export_mod/](misp_modules/modules/export_mod/), or [misp_modules/modules/import_mod/](misp_modules/modules/import_mod/). The module should have at minimum three functions: +Create your module in [misp_modules/modules/expansion/](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/), [misp_modules/modules/export_mod/](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/), or [misp_modules/modules/import_mod/](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/). The module should have at minimum three functions: * **introspection** function that returns a dict of the supported attributes (input and output) by your expansion module. * **handler** function which accepts a JSON document to expand the values and return a dictionary of the expanded values. @@ -508,27 +647,11 @@ Recommended Plugin.Import_ocr_enabled true Enable or disable the ocr In this same menu set any other plugin settings that are required for testing. -## Install misp-module on an offline instance. -First, you need to grab all necessary packages for example like this : - -Use pip wheel to create an archive -~~~ -mkdir misp-modules-offline -pip3 wheel -r REQUIREMENTS shodan --wheel-dir=./misp-modules-offline -tar -cjvf misp-module-bundeled.tar.bz2 ./misp-modules-offline/* -~~~ -On offline machine : -~~~ -mkdir misp-modules-bundle -tar xvf misp-module-bundeled.tar.bz2 -C misp-modules-bundle -cd misp-modules-bundle -ls -1|while read line; do sudo pip3 install --force-reinstall --ignore-installed --upgrade --no-index --no-deps ${line};done -~~~ -Next you can follow standard install procedure. ## How to contribute your own module? Fork the project, add your module, test it and make a pull-request. Modules can be also private as you can add a module in your own MISP installation. +For further information please see [Contribute](contribute/). ## Tips for developers creating modules @@ -589,3 +712,6 @@ In order to provide documentation about some modules that require specific input - **references** - link(s) giving additional information about the format concerned in the module - **input** - description of the format of data used in input - **output** - description of the format given as the result of the module execution + +## Licenses +For further Information see also the [license file](license/). diff --git a/documentation/mkdocs/contribute.md b/documentation/mkdocs/contribute.md index ef312f6a..3d39596a 100644 --- a/documentation/mkdocs/contribute.md +++ b/documentation/mkdocs/contribute.md @@ -1,6 +1,6 @@ ## How to add your own MISP modules? -Create your module in [misp_modules/modules/expansion/](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/expansion/), [misp_modules/modules/export_mod/](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/export_mod/), or [misp_modules/modules/import_mod/](https://github.com/MISP/misp-modules/tree/master/misp_modules/modules/import_mod/). The module should have at minimum three functions: +Create your module in [misp_modules/modules/expansion/](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/), [misp_modules/modules/export_mod/](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/), or [misp_modules/modules/import_mod/](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/). The module should have at minimum three functions: * **introspection** function that returns a dict of the supported attributes (input and output) by your expansion module. * **handler** function which accepts a JSON document to expand the values and return a dictionary of the expanded values. @@ -309,22 +309,27 @@ Recommended Plugin.Import_ocr_enabled true Enable or disable the ocr In this same menu set any other plugin settings that are required for testing. +## Install misp-module on an offline instance. +First, you need to grab all necessary packages for example like this : +Use pip wheel to create an archive +~~~ +mkdir misp-modules-offline +pip3 wheel -r REQUIREMENTS shodan --wheel-dir=./misp-modules-offline +tar -cjvf misp-module-bundeled.tar.bz2 ./misp-modules-offline/* +~~~ +On offline machine : +~~~ +mkdir misp-modules-bundle +tar xvf misp-module-bundeled.tar.bz2 -C misp-modules-bundle +cd misp-modules-bundle +ls -1|while read line; do sudo pip3 install --force-reinstall --ignore-installed --upgrade --no-index --no-deps ${line};done +~~~ +Next you can follow standard install procedure. -## Documentation +## How to contribute your own module? -In order to provide documentation about some modules that require specific input / output / configuration, the [doc](https://github.com/MISP/misp-modules/tree/master/doc) directory contains detailed information about the general purpose, requirements, features, input and output of each of these modules: - -- ***description** - quick description of the general purpose of the module, as the one given by the moduleinfo -- **requirements** - special libraries needed to make the module work -- **features** - description of the way to use the module, with the required MISP features to make the module give the intended result -- **references** - link(s) giving additional information about the format concerned in the module -- **input** - description of the format of data used in input -- **output** - description of the format given as the result of the module execution - -In addition to the module documentation please add your module to [docs/index.md](https://github.com/MISP/misp-modules/tree/master/docs/index.md). - -There are also [complementary slides](https://www.misp-project.org/misp-training/3.1-misp-modules.pdf) for the creation of MISP modules. +Fork the project, add your module, test it and make a pull-request. Modules can be also private as you can add a module in your own MISP installation. ## Tips for developers creating modules @@ -334,7 +339,7 @@ Download a pre-built virtual image from the [MISP training materials](https://ww - Create a Host-Only adapter in VirtualBox - Set your Misp OVA to that Host-Only adapter - Start the virtual machine -- Get the IP address of the virutal machine +- Get the IP address of the virtual machine - SSH into the machine (Login info on training page) - Go into the misp-modules directory @@ -352,16 +357,18 @@ sudo git checkout MyModBranch Remove the contents of the build directory and re-install misp-modules. -~~~python +~~~bash sudo rm -fr build/* -sudo pip3 install --upgrade . +sudo -u www-data /var/www/MISP/venv/bin/pip install --upgrade . ~~~ SSH in with a different terminal and run `misp-modules` with debugging enabled. -~~~python -sudo killall misp-modules -misp-modules -d +~~~bash +# In case misp-modules is not a service do: +# sudo killall misp-modules +sudo systemctl disable --now misp-modules +sudo -u www-data /var/www/MISP/venv/bin/misp-modules -d ~~~ @@ -372,3 +379,17 @@ cd tests/ curl -s http://127.0.0.1:6666/query -H "Content-Type: application/json" --data @MY_TEST_FILE.json -X POST cd ../ ~~~ + +## Documentation + +In order to provide documentation about some modules that require specific input / output / configuration, the [index.md](docs/index.md) file within the [docs](docs) directory contains detailed information about the general purpose, requirements, features, input and ouput of each of these modules: + +- ***description** - quick description of the general purpose of the module, as the one given by the moduleinfo +- **requirements** - special libraries needed to make the module work +- **features** - description of the way to use the module, with the required MISP features to make the module give the intended result +- **references** - link(s) giving additional information about the format concerned in the module +- **input** - description of the format of data used in input +- **output** - description of the format given as the result of the module execution + +## Licenses +For further Information see also the [license file](license/). diff --git a/documentation/mkdocs/index.md b/documentation/mkdocs/index.md index ca6179f3..ecc41366 100644 --- a/documentation/mkdocs/index.md +++ b/documentation/mkdocs/index.md @@ -1,113 +1,143 @@ # Home -[![Build Status](https://travis-ci.org/MISP/misp-modules.svg?branch=master)](https://travis-ci.org/MISP/misp-modules) -[![Coverage Status](https://coveralls.io/repos/github/MISP/misp-modules/badge.svg?branch=master)](https://coveralls.io/github/MISP/misp-modules?branch=master) +[![Build status](https://github.com/MISP/misp-modules/actions/workflows/python-package.yml/badge.svg)](https://github.com/MISP/misp-modules/actions/workflows/python-package.yml)[![Coverage Status](https://coveralls.io/repos/github/MISP/misp-modules/badge.svg?branch=main)](https://coveralls.io/github/MISP/misp-modules?branch=main) [![codecov](https://codecov.io/gh/MISP/misp-modules/branch/main/graph/badge.svg)](https://codecov.io/gh/MISP/misp-modules) -[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%MISP%2Fmisp-modules.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2FMISP%2Fmisp-modules?ref=badge_shield) -MISP modules are autonomous modules that can be used for expansion and other services in [MISP](https://github.com/MISP/MISP). +MISP modules are autonomous modules that can be used to extend [MISP](https://github.com/MISP/MISP) for new services such as expansion, import, export and workflow action. + +MISP modules can be also installed and used without MISP as a [standalone tool accessible via a convenient web interface](./website). The modules are written in Python 3 following a simple API interface. The objective is to ease the extensions of MISP functionalities -without modifying core components. The API is available via a simple REST API which is independent from MISP installation or configuration. - -MISP modules support is included in MISP starting from version `2.4.28`. - -For more information: [Extending MISP with Python modules](https://www.circl.lu/assets/files/misp-training/switch2016/2-misp-modules.pdf) slides from MISP training. +without modifying core components. The API is available via a simple REST API which is independent from MISP installation or configuration and can be used with other tools. +For more information: [Extending MISP with Python modules](https://www.misp-project.org/misp-training/3.1-misp-modules.pdf) slides from [MISP training](https://github.com/MISP/misp-training). ## Existing MISP modules ### Expansion modules - +* [apiosintDS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/apiosintds.py) - a hover and expansion module to query the [OSINT.digitalside.it](https://osint.digitalside.it) API. [Documentation](https://apiosintds.readthedocs.io/en/latest/userguidemisp.html). +* [API Void](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/apivoid.py) - an expansion and hover module to query API Void with a domain attribute. +* [AssemblyLine submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/assemblyline_submit.py) - an expansion module to submit samples and urls to AssemblyLine. +* [AssemblyLine query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/assemblyline_query.py) - an expansion module to query AssemblyLine and parse the full submission report. * [Backscatter.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/backscatter_io.py) - a hover and expansion module to expand an IP address with mass-scanning observations. * [BGP Ranking](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/bgpranking.py) - a hover and expansion module to expand an AS number with the ASN description, its history, and position in BGP Ranking. +* [RansomcoinDB check](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ransomcoindb.py) - An expansion hover module to query the [ransomcoinDB](https://ransomcoindb.concinnity-risks.com): it contains mapping between BTC addresses and malware hashes. Enrich MISP by querying for BTC -> hash or hash -> BTC addresses. * [BTC scam check](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/btc_scam_check.py) - An expansion hover module to instantly check if a BTC address has been abused. * [BTC transactions](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/btc_steroids.py) - An expansion hover module to get a blockchain balance and the transactions from a BTC address in MISP. +* [Censys-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/censys_enrich.py) - An expansion and module to retrieve information from censys.io about a particular IP or certificate. * [CIRCL Passive DNS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/circl_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information. -* [CIRCL Passive SSL](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/circl_passivessl.py) - a hover and expansion module to expand IP addresses with the X.509 certificate seen. +* [CIRCL Passive SSL](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/circl_passivessl.py) - a hover and expansion module to expand IP addresses with the X.509 certificates seen. * [countrycode](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/countrycode.py) - a hover module to tell you what country a URL belongs to. +* [CrowdSec](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/crowdsec.py) - a hover module to expand using CrowdSec's CTI API. * [CrowdStrike Falcon](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/crowdstrike_falcon.py) - an expansion module to expand using CrowdStrike Falcon Intel Indicator API. +* [CPE](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cpe.py) - An expansion module to query the CVE Search API with a cpe code, to get its related vulnerabilities. * [CVE](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cve.py) - a hover module to give more information about a vulnerability (CVE). * [CVE advanced](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cve_advanced.py) - An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE). * [Cuckoo submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cuckoo_submit.py) - A hover module to submit malware sample, url, attachment, domain to Cuckoo Sandbox. +* [Cytomic Orion](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/cytomic_orion.py) - An expansion module to enrich attributes in MISP and share indicators of compromise with Cytomic Orion. * [DBL Spamhaus](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/dbl_spamhaus.py) - a hover module to check Spamhaus DBL for a domain name. * [DNS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/dns.py) - a simple module to resolve MISP attributes like hostname and domain to expand IP addresses attributes. -* [docx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/docx-enrich.py) - an enrichment module to get text out of Word document into MISP (using free-text parser). +* [docx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/docx_enrich.py) - an enrichment module to get text out of Word document into MISP (using free-text parser). * [DomainTools](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/domaintools.py) - a hover and expansion module to get information from [DomainTools](http://www.domaintools.com/) whois. +* [EQL](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/eql.py) - an expansion module to generate event query language (EQL) from an attribute. [Event Query Language](https://eql.readthedocs.io/en/latest/) * [EUPI](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/eupi.py) - a hover and expansion module to get information about an URL from the [Phishing Initiative project](https://phishing-initiative.eu/?lang=en). -* [EQL](misp_modules/modules/expansion/eql.py) - an expansion module to generate event query language (EQL) from an attribute. [Event Query Language](https://eql.readthedocs.io/en/latest/) * [Farsight DNSDB Passive DNS](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/farsight_passivedns.py) - a hover and expansion module to expand hostname and IP addresses with passive DNS information. * [GeoIP](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/geoip_country.py) - a hover and expansion module to get GeoIP information from geolite/maxmind. -* [Greynoise](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/greynoise.py) - a hover to get information from greynoise. +* [GeoIP_City](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/geoip_city.py) - a hover and expansion module to get GeoIP City information from geolite/maxmind. +* [GeoIP_ASN](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/geoip_asn.py) - a hover and expansion module to get GeoIP ASN information from geolite/maxmind. +* [Google Threat Intelligence] (https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/google_threat_intelligence.py) - An expansion module to have the observable's threat score assessed by Google Threat Intelligence. +* [GreyNoise](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/greynoise.py) - a hover and expansion module to get IP and CVE information from GreyNoise. * [hashdd](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hashdd.py) - a hover module to check file hashes against [hashdd.com](http://www.hashdd.com) including NSLR dataset. +* [Hashlookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hashlookup.py) - An expansion module to enrich a file hash with hashlookup.circl.lu services (NSRL and other sources) * [hibp](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hibp.py) - a hover module to lookup against Have I Been Pwned? +* [html_to_markdown](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/html_to_markdown.py) - Simple HTML to markdown converter +* [HYAS Insight](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/hyasinsight.py) - a hover and expansion module to get information from [HYAS Insight](https://www.hyas.com/hyas-insight). * [intel471](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/intel471.py) - an expansion module to get info from [Intel471](https://intel471.com). +* [IP2Location.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ip2locationio.py) - an expansion module to get additional information on an IP address using the IP2Location.io API * [IPASN](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ipasn.py) - a hover and expansion to get the BGP ASN of an IP address. +* [ipinfo.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ipinfo.py) - an expansion module to get additional information on an IP address using the ipinfo.io API * [iprep](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/iprep.py) - an expansion module to get IP reputation from packetmail.net. * [Joe Sandbox submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/joesandbox_submit.py) - Submit files and URLs to Joe Sandbox. * [Joe Sandbox query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/joesandbox_query.py) - Query Joe Sandbox with the link of an analysis and get the parsed data. +* [Lastline submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/lastline_submit.py) - Submit files and URLs to Lastline. +* [Lastline query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/lastline_query.py) - Query Lastline with the link to an analysis and parse the report. * [macaddress.io](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/macaddress_io.py) - a hover module to retrieve vendor details and other information regarding a given MAC address or an OUI from [MAC address Vendor Lookup](https://macaddress.io). See [integration tutorial here](https://macaddress.io/integrations/MISP-module). * [macvendors](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/macvendors.py) - a hover module to retrieve mac vendor information. -* [ocr-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ocr-enrich.py) - an enrichment module to get OCRized data from images into MISP. -* [ods-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ods-enrich.py) - an enrichment module to get text out of OpenOffice spreadsheet document into MISP (using free-text parser). -* [odt-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/odt-enrich.py) - an enrichment module to get text out of OpenOffice document into MISP (using free-text parser). +* [MALWAREbazaar](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/malwarebazaar.py) - an expansion module to query MALWAREbazaar with some payload. +* [McAfee MVISION Insights](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/mcafee_insights_enrich.py) - an expansion module enrich IOCs with McAfee MVISION Insights. +* [Mmdb server lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/mmdb_lookup.py) - an expansion module to enrich an ip with geolocation information from an mmdb server such as ip.circl.lu. +* [ocr-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ocr_enrich.py) - an enrichment module to get OCRized data from images into MISP. +* [ods-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/ods_enrich.py) - an enrichment module to get text out of OpenOffice spreadsheet document into MISP (using free-text parser). +* [odt-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/odt_enrich.py) - an enrichment module to get text out of OpenOffice document into MISP (using free-text parser). * [onyphe](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/onyphe.py) - a modules to process queries on Onyphe. * [onyphe_full](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/onyphe_full.py) - a modules to process full queries on Onyphe. * [OTX](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/otx.py) - an expansion module for [OTX](https://otx.alienvault.com/). * [passivetotal](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/passivetotal.py) - a [passivetotal](https://www.passivetotal.org/) module that queries a number of different PassiveTotal datasets. -* [pdf-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/pdf-enrich.py) - an enrichment module to extract text from PDF into MISP (using free-text parser). -* [pptx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/pptx-enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser). +* [pdf-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/pdf_enrich.py) - an enrichment module to extract text from PDF into MISP (using free-text parser). +* [pptx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/pptx_enrich.py) - an enrichment module to get text out of PowerPoint document into MISP (using free-text parser). * [qrcode](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/qrcode.py) - a module decode QR code, barcode and similar codes from an image and enrich with the decoded values. * [rbl](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/rbl.py) - a module to get RBL (Real-Time Blackhost List) values from an attribute. +* [recordedfuture](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/recordedfuture.py) - a hover and expansion module for enriching MISP attributes with threat intelligence from Recorded Future. * [reversedns](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/reversedns.py) - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes. * [securitytrails](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/securitytrails.py) - an expansion module for [securitytrails](https://securitytrails.com/). * [shodan](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/shodan.py) - a minimal [shodan](https://www.shodan.io/) expansion module. * [Sigma queries](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sigma_queries.py) - Experimental expansion module querying a sigma rule to convert it into all the available SIEM signatures. * [Sigma syntax validator](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sigma_syntax_validator.py) - Sigma syntax validator. +* [Socialscan](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/socialscan.py) - a hover module to check if an email address or a username is used on different online platforms, using the [socialscan](https://github.com/iojw/socialscan) python library +* [SophosLabs Intelix](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sophoslabs_intelix.py) - SophosLabs Intelix is an API for Threat Intelligence and Analysis (free tier available). [SophosLabs](https://aws.amazon.com/marketplace/pp/B07SLZPMCS) * [sourcecache](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/sourcecache.py) - a module to cache a specific link from a MISP instance. +* [stairwell](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/stairwell.py) - an expansion module to enrich hash observables with the Stairwell API * [STIX2 pattern syntax validator](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/stix2_pattern_syntax_validator.py) - a module to check a STIX2 pattern syntax. * [ThreatCrowd](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/threatcrowd.py) - an expansion module for [ThreatCrowd](https://www.threatcrowd.org/). * [threatminer](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/threatminer.py) - an expansion module to expand from [ThreatMiner](https://www.threatminer.org/). +* [TruSTAR Enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/trustar_enrich.py) - an expansion module to enrich MISP data with [TruSTAR](https://www.trustar.co/). * [urlhaus](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/urlhaus.py) - Query urlhaus to get additional data about a domain, hash, hostname, ip or url. * [urlscan](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/urlscan.py) - an expansion module to query [urlscan.io](https://urlscan.io). +* [variotdbs](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/variotdbs.py) - an expansion module to query the [VARIoT db](https://www.variotdbs.pl) API to get more information about a Vulnerability * [virustotal](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/virustotal.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a high request rate limit required. (More details about the API: [here](https://docs.virustotal.com/reference/overview)) * [virustotal_public](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/virustotal_public.py) - an expansion module to query the [VirusTotal](https://www.virustotal.com/gui/home) API with a public key and a low request rate limit. (More details about the API: [here](https://docs.virustotal.com/reference/overview)) * [VMray](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vmray_submit.py) - a module to submit a sample to VMray. +* [VMware NSX](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vmware_nsx.py) - a module to enrich a file or URL with VMware NSX Defender. * [VulnDB](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vulndb.py) - a module to query [VulnDB](https://www.riskbasedsecurity.com/). * [Vulners](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vulners.py) - an expansion module to expand information about CVEs using Vulners API. +* [Vysion](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vysion.py) - an expansion module to add dark web intelligence using Vysion API. * [whois](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whois.py) - a module to query a local instance of [uwhois](https://github.com/rafiot/uwhoisd). +* [whoisfreaks](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whoisfreaks.py) - An expansion module for [whoisfreaks](https://whoisfreaks.com/) that will provide an enriched analysis of the provided domain, including WHOIS and DNS information. * [wikidata](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/wiki.py) - a [wikidata](https://www.wikidata.org) expansion module. * [xforce](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/xforceexchange.py) - an IBM X-Force Exchange expansion module. -* [xlsx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/xlsx-enrich.py) - an enrichment module to get text out of an Excel document into MISP (using free-text parser). +* [xlsx-enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/xlsx_enrich.py) - an enrichment module to get text out of an Excel document into MISP (using free-text parser). * [YARA query](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/yara_query.py) - a module to create YARA rules from single hash attributes. * [YARA syntax validator](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/yara_syntax_validator.py) - YARA syntax validator. ### Export modules -* [CEF](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cef_export.py) module to export Common Event Format (CEF). -* [Cisco FireSight Manager ACL rule](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cisco_firesight_manager_ACL_rule_export.py) module to export as rule for the Cisco FireSight manager ACL. -* [GoAML export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/goamlexport.py) module to export in [GoAML format](http://goaml.unodc.org/goaml/en/index.html). -* [Lite Export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/liteexport.py) module to export a lite event. -* [Mass EQL Export](misp_modules/modules/export_mod/mass_eql_export.py) module to export applicable attributes from an event to a mass EQL query. -* [PDF export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/pdfexport.py) module to export an event in PDF. -* [Nexthink query format](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/nexthinkexport.py) module to export in Nexthink query format. -* [osquery](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/osqueryexport.py) module to export in [osquery](https://osquery.io/) query format. -* [ThreatConnect](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/threat_connect_export.py) module to export in ThreatConnect CSV format. -* [ThreatStream](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/threatStream_misp_export.py) module to export in ThreatStream format. +* [CEF](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cef_export.py) - module to export Common Event Format (CEF). +* [Cisco FireSight Manager ACL rule](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/cisco_firesight_manager_ACL_rule_export.py) - module to export as rule for the Cisco FireSight manager ACL. +* [GoAML export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/goamlexport.py) - module to export in [GoAML format](http://goaml.unodc.org/goaml/en/index.html). +* [Lite Export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/liteexport.py) - module to export a lite event. +* [PDF export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/pdfexport.py) - module to export an event in PDF. +* [Mass EQL Export](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/mass_eql_export.py) - module to export applicable attributes from an event to a mass EQL query. +* [Nexthink query format](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/nexthinkexport.py) - module to export in Nexthink query format. +* [osquery](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/osqueryexport.py) - module to export in [osquery](https://osquery.io/) query format. +* [ThreatConnect](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/threat_connect_export.py) - module to export in ThreatConnect CSV format. +* [ThreatStream](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/threatStream_misp_export.py) - module to export in ThreatStream format. +* [VirusTotal Graph](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/export_mod/vt_graph.py) - Module to create a VirusTotal graph out of an event. ### Import modules -* [CSV import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/csvimport.py) Customizable CSV import module. -* [Cuckoo JSON](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/cuckooimport.py) Cuckoo JSON import. -* [Email Import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/email_import.py) Email import module for MISP to import basic metadata. -* [GoAML import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/goamlimport.py) Module to import [GoAML](http://goaml.unodc.org/goaml/en/index.html) XML format. -* [Joe Sandbox import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/joe_import.py) Parse data from a Joe Sandbox json report. -* [OCR](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/ocr.py) Optical Character Recognition (OCR) module for MISP to import attributes from images, scan or faxes. -* [OpenIOC](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/openiocimport.py) OpenIOC import based on PyMISP library. +* [CSV import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/csvimport.py) - Customizable CSV import module. +* [Cuckoo JSON](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/cuckooimport.py) - Cuckoo JSON import. +* [Email Import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/email_import.py) - Email import module for MISP to import basic metadata. +* [GoAML import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/goamlimport.py) - Module to import [GoAML](http://goaml.unodc.org/goaml/en/index.html) XML format. +* [Joe Sandbox import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/joe_import.py) - Parse data from a Joe Sandbox json report. +* [Lastline import](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/lastline_import.py) - Module to import Lastline analysis reports. +* [OCR](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/ocr.py) - Optical Character Recognition (OCR) module for MISP to import attributes from images, scan or faxes. +* [OpenIOC](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/openiocimport.py) - OpenIOC import based on PyMISP library. * [ThreatAnalyzer](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/threatanalyzer_import.py) - An import module to process ThreatAnalyzer archive.zip/analysis.json sandbox exports. * [VMRay](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/import_mod/vmray_import.py) - An import module to process VMRay export. + ## How to contribute your own module? Fork the project, add your module, test it and make a pull-request. Modules can be also private as you can add a module in your own MISP installation. @@ -115,6 +145,4 @@ For further information please see [Contribute](contribute/). ## Licenses -[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%MISP%2Fmisp-modules.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2FMISP%2Fmisp-modules?ref=badge_large) - For further Information see also the [license file](license/). diff --git a/documentation/mkdocs/install.md b/documentation/mkdocs/install.md index 3eed0f49..1cb7cb88 100644 --- a/documentation/mkdocs/install.md +++ b/documentation/mkdocs/install.md @@ -4,6 +4,8 @@ SUDO_WWW="sudo -u www-data" sudo apt-get install -y \ + python3-dev \ + python3-pip \ git \ libpq5 \ libjpeg-dev \ @@ -15,9 +17,10 @@ sudo apt-get install -y \ libzbar0 \ libzbar-dev \ libfuzzy-dev \ - libcaca-dev + libcaca-dev \ + build-essential -# BEGIN with virtualenv: +# BEGIN with virtualenv: $SUDO_WWW virtualenv -p python3 /var/www/MISP/venv # END with virtualenv @@ -45,12 +48,12 @@ sudo ldconfig cd ../../misp-modules -# BEGIN with virtualenv: +# BEGIN with virtualenv: $SUDO_WWW /var/www/MISP/venv/bin/pip install -I -r REQUIREMENTS $SUDO_WWW /var/www/MISP/venv/bin/pip install . # END with virtualenv -# BEGIN without virtualenv: +# BEGIN without virtualenv: sudo pip install -I -r REQUIREMENTS sudo pip install . # END without virtualenv @@ -59,16 +62,18 @@ sudo pip install . sudo cp etc/systemd/system/misp-modules.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now misp-modules -/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s & #to start the modules +sudo service misp-modules start # or +/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s & # to start the modules manually ~~~~ ## How to install and start MISP modules on RHEL-based distributions ? -As of this writing, the official RHEL repositories only contain Ruby 2.0.0 and Ruby 2.1 or higher is required. As such, this guide installs Ruby 2.2 from the SCL repository. +As of this writing, the official RHEL repositories only contain Ruby 2.0.0 and Ruby 2.1 or higher is required. As such, this guide installs Ruby 2.2 from the [SCL](https://access.redhat.com/documentation/en-us/red_hat_software_collections/3/html/3.2_release_notes/chap-installation#sect-Installation-Subscribe) repository. ~~~~bash SUDO_WWW="sudo -u apache" sudo yum install \ + rh-python36 \ rh-ruby22 \ openjpeg-devel \ rubygem-rouge \ @@ -80,8 +85,8 @@ sudo yum install \ poppler-cpp-devel \ python-devel \ redhat-rpm-config -cd /usr/local/src/ -sudo git clone https://github.com/MISP/misp-modules.git +cd /var/www/MISP +$SUDO_WWW git clone https://github.com/MISP/misp-modules.git cd misp-modules $SUDO_WWW /usr/bin/scl enable rh-python36 "virtualenv -p python3 /var/www/MISP/venv" $SUDO_WWW /var/www/MISP/venv/bin/pip install -U -I -r REQUIREMENTS @@ -99,7 +104,7 @@ After=misp-workers.service Type=simple User=apache Group=apache -ExecStart=/usr/bin/scl enable rh-python36 rh-ruby22 '/var/www/MISP/venv/bin/misp-modules –l 127.0.0.1 –s' +ExecStart=/usr/bin/scl enable rh-python36 rh-ruby22 '/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s' Restart=always RestartSec=10 @@ -107,8 +112,8 @@ RestartSec=10 WantedBy=multi-user.target" | sudo tee /etc/systemd/system/misp-modules.service ~~~~ -The After=misp-workers.service must be changed or removed if you have not created a misp-workers service. Then, enable the misp-modules service and start it: - +The `After=misp-workers.service` must be changed or removed if you have not created a misp-workers service. +Then, enable the misp-modules service and start it: ~~~~bash systemctl daemon-reload systemctl enable --now misp-modules @@ -147,20 +152,20 @@ services: misp-modules: # https://hub.docker.com/r/dcso/misp-dockerized-misp-modules image: dcso/misp-dockerized-misp-modules:3 - + # Local image: #image: misp-modules #build: # context: docker/ - + environment: # Redis REDIS_BACKEND: misp-redis REDIS_PORT: "6379" REDIS_DATABASE: "245" # System PROXY (OPTIONAL) - http_proxy: - https_proxy: + http_proxy: + https_proxy: no_proxy: 0.0.0.0 # Timezone (OPTIONAL) TZ: Europe/Berlin