Merge pull request #688 from MISP/karenyousefi-main

chg: [merge] updated
pull/689/head
Alexandre Dulaunoy 2024-08-22 15:17:15 +02:00 committed by GitHub
commit 1587d19217
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
10 changed files with 482 additions and 9 deletions

View File

@ -90,6 +90,7 @@ For further Information see the [license file](https://misp.github.io/misp-modul
* [Lastline Submit](https://misp.github.io/misp-modules/expansion/#lastline-submit) - Deprecation notice: this module will be deprecated by December 2021, please use vmware_nsx module. Module to submit a file or URL to Lastline.
* [Macaddress.io Lookup](https://misp.github.io/misp-modules/expansion/#macaddress.io-lookup) - MISP hover module for macaddress.io
* [Macvendors Lookup](https://misp.github.io/misp-modules/expansion/#macvendors-lookup) - Module to access Macvendors API.
* [Malshare Upload](https://misp.github.io/misp-modules/expansion/#malshare-upload) - Module to push malware samples to malshare.com .
* [Malware Bazaar Lookup](https://misp.github.io/misp-modules/expansion/#malware-bazaar-lookup) - Query Malware Bazaar to get additional information about the input hash.
* [McAfee MVISION Insights Lookup](https://misp.github.io/misp-modules/expansion/#mcafee-mvision-insights-lookup) - Lookup McAfee MVISION Insights Details
* [GeoIP Enrichment](https://misp.github.io/misp-modules/expansion/#geoip-enrichment) - A hover and expansion module to enrich an ip with geolocation and ASN information from an mmdb server instance, such as CIRCL's ip.circl.lu.
@ -123,18 +124,21 @@ For further Information see the [license file](https://misp.github.io/misp-modul
* [ThreatCrowd Lookup](https://misp.github.io/misp-modules/expansion/#threatcrowd-lookup) - Module to get information from ThreatCrowd.
* [ThreadFox Lookup](https://misp.github.io/misp-modules/expansion/#threadfox-lookup) - Module to search for an IOC on ThreatFox by abuse.ch.
* [ThreatMiner Lookup](https://misp.github.io/misp-modules/expansion/#threatminer-lookup) - Module to get information from ThreatMiner.
* [Triage Submit](https://misp.github.io/misp-modules/expansion/#triage-submit) - Module to submit samples to tria.ge .
* [TruSTAR Enrich](https://misp.github.io/misp-modules/expansion/#trustar-enrich) - Module to get enrich indicators with TruSTAR.
* [URLhaus Lookup](https://misp.github.io/misp-modules/expansion/#urlhaus-lookup) - Query of the URLhaus API to get additional information about the input attribute.
* [URLScan Lookup](https://misp.github.io/misp-modules/expansion/#urlscan-lookup) - An expansion module to query urlscan.io.
* [VARIoT db Lookup](https://misp.github.io/misp-modules/expansion/#variot-db-lookup) - An expansion module to query the VARIoT db API for more information about a vulnerability.
* [VirusTotal v3 Lookup](https://misp.github.io/misp-modules/expansion/#virustotal-v3-lookup) - Enrich observables with the VirusTotal v3 API
* [VirusTotal Public API Lookup](https://misp.github.io/misp-modules/expansion/#virustotal-public-api-lookup) - Enrich observables with the VirusTotal v3 public API
* [VirusTotal Upload](https://misp.github.io/misp-modules/expansion/#virustotal-upload) - Module to push malware samples to VirusTotal v3 public API
* [VMRay Submit](https://misp.github.io/misp-modules/expansion/#vmray-submit) - Module to submit a sample to VMRay.
* [VMware NSX Defender Enrich](https://misp.github.io/misp-modules/expansion/#vmware-nsx-defender-enrich) - Module to enrich a file or URL with VMware NSX Defender.
* [VulnDB Lookup](https://misp.github.io/misp-modules/expansion/#vulndb-lookup) - Module to query VulnDB (RiskBasedSecurity.com).
* [Vulnerability Lookup](https://misp.github.io/misp-modules/expansion/#vulnerability-lookup) - An expansion module to query Vulnerability Lookup
* [Vulners Lookup](https://misp.github.io/misp-modules/expansion/#vulners-lookup) - An expansion hover module to expand information about CVE id using Vulners API.
* [Vysion Enrich](https://misp.github.io/misp-modules/expansion/#vysion-enrich) - Module to enrich the information by making use of the Vysion API.
* [Whois Lookup](https://misp.github.io/misp-modules/expansion/#whois-lookup) - Module to query a local instance of uwhois (https://github.com/rafiot/uwhoisd).
* [WhoisFreaks Lookup](https://misp.github.io/misp-modules/expansion/#whoisfreaks-lookup) - An expansion module for https://whoisfreaks.com/ that will provide an enriched analysis of the provided domain, including WHOIS and DNS information.
* [Wikidata Lookup](https://misp.github.io/misp-modules/expansion/#wikidata-lookup) - An expansion hover module to extract information from Wikidata to have additional information about particular term for analysis.
* [IBM X-Force Exchange Lookup](https://misp.github.io/misp-modules/expansion/#ibm-x-force-exchange-lookup) - An expansion module for IBM X-Force Exchange.
@ -181,5 +185,3 @@ For further Information see the [license file](https://misp.github.io/misp-modul
* [Mattermost](https://misp.github.io/misp-modules/action_mod/#mattermost) - Simplistic module to send message to a Mattermost channel.
* [Slack](https://misp.github.io/misp-modules/action_mod/#slack) - Simplistic module to send messages to a Slack channel.
* [Test action](https://misp.github.io/misp-modules/action_mod/#test-action) - This module is merely a test, always returning true. Triggers on event publishing.

View File

@ -114,6 +114,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
* [Vulnerability Lookup](https://misp.github.io/misp-modules/expansion/#vulnerability-lookup) - An expansion module to query Vulnerability Lookup
* [Vulners Lookup](https://misp.github.io/misp-modules/expansion/#vulners-lookup) - An expansion hover module to expand information about CVE id using Vulners API.
* [Vysion Enrich](https://misp.github.io/misp-modules/expansion/#vysion-enrich) - Module to enrich the information by making use of the Vysion API.
* [Whois Lookup](https://misp.github.io/misp-modules/expansion/#whois-lookup) - Module to query a local instance of uwhois (https://github.com/rafiot/uwhoisd).
* [WhoisFreaks Lookup](https://misp.github.io/misp-modules/expansion/#whoisfreaks-lookup) - An expansion module for https://whoisfreaks.com/ that will provide an enriched analysis of the provided domain, including WHOIS and DNS information.
* [Wikidata Lookup](https://misp.github.io/misp-modules/expansion/#wikidata-lookup) - An expansion hover module to extract information from Wikidata to have additional information about particular term for analysis.
* [IBM X-Force Exchange Lookup](https://misp.github.io/misp-modules/expansion/#ibm-x-force-exchange-lookup) - An expansion module for IBM X-Force Exchange.

View File

@ -1,6 +1,3 @@
IMPORTANT NOTE: we will soon be publishing `misp-modules` on PyPI.
## Install from pip
It is strongly recommended to use a virtual environment (see here for instructions https://docs.python.org/3/tutorial/venv.html).
@ -11,6 +8,8 @@ Once the virtual environment is loaded just use the command:
pip install misp-modules
~~~~
Note: this install method might not yet be available.
## Install from cloned repository
@ -83,7 +82,7 @@ Inside you will find three targets:
- `test-docs`: run a local server exposing the newly built documentation.
Note that you can either run the targets using `poetry` (default), or using the `squidfunk/mkdocs-material` by setting the environment variable `USE_DOCKER=true`.
Note: you can either run the targets using `poetry` (default), or using the Docker image `squidfunk/mkdocs-material` by setting the environment variable `USE_DOCKER=true`.
## Run MISP modules
@ -93,7 +92,7 @@ If you installed it using pip, you just need to execute the command `misp-module
## Run MISP modules in Docker
You can find an up-to-date container image and related documentation at the following repository:m https://github.com/MISP/misp-docker
You can find an up-to-date container image and related documentation at the following repository: https://github.com/MISP/misp-docker .
## Install misp-module on an offline instance
@ -137,7 +136,13 @@ Just follow those instructions but replace the package `misp-modules` with `-r r
Before doing so you need to generate the `requirements.txt` file. Due to the fact we are still supporting Python 3.8 and that Poetry still has some limitations (soon to be resolved) you need to need to replace the line `python = ">=3.8.*,<3.13"` inside `pyproject.toml` with your exact version (just run `python --version`).
Once you have done that, run the following commands to generate your very own `requirements.txt`.
The following `sed` command does everything for you.
~~~~bash
sed -i "s/^python = .*/python = \"$(python -c 'import platform; print(platform.python_version())')\"/" pyproject.toml
~~~~
Then, run the following commands to generate your very own `requirements.txt`.
~~~~bash
poetry lock
@ -145,3 +150,10 @@ poetry install
poetry self add poetry-plugin-export
poetry export --without-hashes -f requirements.txt -o requirements.txt
~~~~
Note that `misp-modules` will not be part of the `requirements.txt` file and you will need to create the wheel yourself:
~~~~bash
poetry build --output ./wheels
~~~~

View File

@ -2821,6 +2821,32 @@ Module to enrich the information by making use of the Vysion API.
-----
#### [Whois Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whois.py)
Module to query a local instance of uwhois (https://github.com/rafiot/uwhoisd).
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whois.py)]
- **features**:
>This module takes a domain or IP address attribute as input and queries a 'Univseral Whois proxy server' to get the correct details of the Whois query on the input value (check the references for more details about this whois server).
- **config**:
> - server
> - port
- **input**:
>A domain or IP address attribute.
- **output**:
>Text describing the result of a whois request for the input value.
- **references**:
>https://github.com/Lookyloo/uwhoisd
- **requirements**:
>uwhois: A whois python library
-----
#### [WhoisFreaks Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whoisfreaks.py)
<img src=logos/whoisfreaks.png height=60>

View File

@ -1539,6 +1539,29 @@ Module to access Macvendors API.
-----
#### [Malshare Upload](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/malshare_upload.py)
Module to push malware samples to MalShare.com
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/malshare_upload.py)]
- **features**:
>The module requires a MalShare API key to upload files, and returns the link of the MalShare analysis.
- **config**:
>api_key
- **input**:
>Attachment or malware sample
- **output**:
>Link attribute that points to the sample at the MalShare analysis instance.
- **references**:
> - https://malshare.com/
> - https://malshare.com/doc.php
-----
#### [Malware Bazaar Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/malwarebazaar.py)
Query Malware Bazaar to get additional information about the input hash.
@ -2432,6 +2455,42 @@ Module to get information from ThreatMiner.
- **references**:
>https://www.threatminer.org/
-----
#### [Triage Submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/triage_submit.py)
Module to submit samples to tria.ge
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/triage_submit.py)]
- **features**:
> Upload files, and returns the link of the uploaded analysis.
>
>The module can submit URLs to retrieve and analyze them directly in the browser or fetch and execute files in the sandbox.
- **config**:
>apikey
>
>url_mode ( 'submit' or 'fetch' )
- **input**:
>A MISP attribute included in the following list:
>- Attachment
>- malware-sample
>- url
- **output**:
>Link attribute that points to the sample at the Triage analysis instance.
- **references**:
> - https://tria.ge/
> - https://tria.ge/docs/cloud-api/submit/
- **requirements**:
>An access to the Triage API (apikey)
-----
#### [TruSTAR Enrich](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/trustar_enrich.py)
@ -2631,6 +2690,35 @@ Enrich observables with the VirusTotal v3 public API
- **requirements**:
>An access to the VirusTotal API (apikey)
-----
#### [VirusTotal Upload](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/virustotal_upload.py)
<img src=../logos/virustotal.png height=60>
Module to push malware samples to VirusTotal v3 public API
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/virustotal_upload.py)]
- **features**:
>The module requires a VirusTotal API key to Upload files, and returns the link of the uploaded analysis.
- **config**:
> - apikey
- **input**:
>Attachment or malware sample
- **output**:
>Link attribute that points to the sample at the VirusTotal analysis instance.
- **references**:
> - https://www.virustotal.com
> - https://docs.virustotal.com/reference/overview
- **requirements**:
>An access to the VirusTotal API (apikey)
-----
#### [VMRay Submit](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/vmray_submit.py)
@ -2818,6 +2906,32 @@ Module to enrich the information by making use of the Vysion API.
-----
#### [Whois Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whois.py)
Module to query a local instance of uwhois (https://github.com/rafiot/uwhoisd).
[[source code](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whois.py)]
- **features**:
>This module takes a domain or IP address attribute as input and queries a 'Univseral Whois proxy server' to get the correct details of the Whois query on the input value (check the references for more details about this whois server).
- **config**:
> - server
> - port
- **input**:
>A domain or IP address attribute.
- **output**:
>Text describing the result of a whois request for the input value.
- **references**:
>https://github.com/Lookyloo/uwhoisd
- **requirements**:
>uwhois: A whois python library
-----
#### [WhoisFreaks Lookup](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/whoisfreaks.py)
<img src=../logos/whoisfreaks.png height=60>

View File

@ -114,6 +114,7 @@ For more information: [Extending MISP with Python modules](https://www.misp-proj
* [Vulnerability Lookup](https://misp.github.io/misp-modules/expansion/#vulnerability-lookup) - An expansion module to query Vulnerability Lookup
* [Vulners Lookup](https://misp.github.io/misp-modules/expansion/#vulners-lookup) - An expansion hover module to expand information about CVE id using Vulners API.
* [Vysion Enrich](https://misp.github.io/misp-modules/expansion/#vysion-enrich) - Module to enrich the information by making use of the Vysion API.
* [Whois Lookup](https://misp.github.io/misp-modules/expansion/#whois-lookup) - Module to query a local instance of uwhois (https://github.com/rafiot/uwhoisd).
* [WhoisFreaks Lookup](https://misp.github.io/misp-modules/expansion/#whoisfreaks-lookup) - An expansion module for https://whoisfreaks.com/ that will provide an enriched analysis of the provided domain, including WHOIS and DNS information.
* [Wikidata Lookup](https://misp.github.io/misp-modules/expansion/#wikidata-lookup) - An expansion hover module to extract information from Wikidata to have additional information about particular term for analysis.
* [IBM X-Force Exchange Lookup](https://misp.github.io/misp-modules/expansion/#ibm-x-force-exchange-lookup) - An expansion module for IBM X-Force Exchange.

View File

@ -23,7 +23,7 @@ __all__ = ['cuckoo_submit', 'vmray_submit', 'circl_passivedns', 'circl_passivess
'extract_url_components', 'ipinfo', 'whoisfreaks', 'ip2locationio', 'stairwell',
'google_threat_intelligence', 'vulnerability_lookup', 'vysion', 'mcafee_insights_enrich',
'threatfox', 'yeti', 'abuseipdb', 'vmware_nsx', 'sigmf_expand', 'google_safe_browsing',
'google_search', 'whois']
'google_search', 'whois', 'triage_submit', 'virustotal_upload', 'malshare_upload' ]
minimum_required_fields = ('type', 'uuid', 'value')

View File

@ -0,0 +1,107 @@
import json
import sys
import base64
import io
import zipfile
import requests
import hashlib
import re
misperrors = {'error': 'Error'}
mispattributes = {'input': ['attachment', 'malware-sample'], 'output': ['link']}
moduleinfo = {
'version': '1',
'author': 'Karen Yousefi',
'description': 'Module to push malware samples to MalShare',
'module-type': ['expansion'],
'name': 'MalShare Upload',
'requirements': ['requests library'],
'logo': '',
}
moduleconfig = ['malshare_apikey']
def handler(q=False):
if q is False:
return False
request = json.loads(q)
try:
data = request.get("data")
if 'malware-sample' in request:
sample_filename = request.get("malware-sample").split("|", 1)[0]
data = base64.b64decode(data)
fl = io.BytesIO(data)
zf = zipfile.ZipFile(fl)
sample_hashname = zf.namelist()[0]
data = zf.read(sample_hashname, b"infected")
zf.close()
elif 'attachment' in request:
sample_filename = request.get("attachment")
data = base64.b64decode(data)
else:
misperrors['error'] = "No malware sample or attachment supplied"
return misperrors
except Exception:
misperrors['error'] = "Unable to process submitted sample data"
return misperrors
if request["config"].get("malshare_apikey") is None:
misperrors["error"] = "Missing MalShare API key"
return misperrors
malshare_apikey = request["config"].get("malshare_apikey")
try:
url = "https://malshare.com/api.php"
params = {'api_key': malshare_apikey, 'action': 'upload'}
files = {"upload": (sample_filename, data)}
response = requests.post(url, params=params, files=files)
response.raise_for_status()
response_text = response.text.strip()
# Calculate SHA256 of the file
sha256 = hashlib.sha256(data).hexdigest()
if response_text.startswith("Success"):
# If upload was successful or file already exists
malshare_link = (
f"https://malshare.com/sample.php?action=detail&hash={sha256}"
)
elif "sample already exists" in response_text:
# If file already exists, extract SHA256 from response
match = re.search(r'([a-fA-F0-9]{64})', response_text)
if match:
sha256 = match.group(1)
malshare_link = (
f"https://malshare.com/sample.php?action=detail&hash={sha256}"
)
else:
# If there's any other error
raise Exception(f"Upload failed: {response_text}")
except Exception as e:
misperrors['error'] = f"Unable to send sample to MalShare: {str(e)}"
return misperrors
r = {
'results': [
{
'types': 'link',
'values': malshare_link,
'comment': 'Link to MalShare analysis',
}
]
}
return r
def introspection():
return mispattributes
def version():
moduleinfo['config'] = moduleconfig
return moduleinfo

View File

@ -0,0 +1,119 @@
import json
import requests
import base64
import io
import zipfile
misperrors = {'error': 'Error'}
mispattributes = {'input': ['attachment', 'malware-sample', 'url'], 'output': ['link']}
moduleinfo = {
'version': '1',
'author': 'Karen Yousefi',
'description': 'Module to submit samples to tria.ge',
'module-type': ['expansion', 'hover'],
'name': 'Triage Submit',
'logo': '',
}
moduleconfig = ['apikey', 'url_mode']
def handler(q=False):
if q is False:
return False
request = json.loads(q)
if request.get('config', {}).get('apikey') is None:
misperrors['error'] = 'tria.ge API key is missing'
return misperrors
api_key = request['config']['apikey']
url_mode = request['config'].get('url_mode', 'submit') # 'submit' or 'fetch'
base_url = 'https://tria.ge/api/v0/samples'
headers = {'Authorization': f'Bearer {api_key}'}
if 'attachment' in request:
data = request['data']
filename = request['attachment']
return submit_file(headers, base_url, data, filename)
elif 'malware-sample' in request:
data = request['data']
filename = request['malware-sample'].split('|')[0]
return submit_file(headers, base_url, data, filename, is_malware_sample=True)
elif 'url' in request:
url = request['url']
return submit_url(headers, base_url, url, url_mode)
else:
misperrors['error'] = 'Unsupported input type'
return misperrors
def submit_file(headers, base_url, data, filename, is_malware_sample=False):
try:
if is_malware_sample:
file_data = base64.b64decode(data)
zip_file = zipfile.ZipFile(io.BytesIO(file_data))
file_data = zip_file.read(zip_file.namelist()[0], pwd=b'infected')
else:
file_data = base64.b64decode(data)
files = {'file': (filename, file_data)}
response = requests.post(base_url, headers=headers, files=files)
response.raise_for_status()
result = response.json()
sample_id = result['id']
sample_url = f'https://tria.ge/{sample_id}'
return {
'results': [
{
'types': 'link',
'values': sample_url,
'comment': 'Link to tria.ge analysis',
}
]
}
except Exception as e:
misperrors['error'] = f'Error submitting to tria.ge: {str(e)}'
return misperrors
def submit_url(headers, base_url, url, mode):
try:
if mode == 'fetch':
data = {'kind': 'fetch', 'url': url}
else: # submit
data = {'kind': 'url', 'url': url}
response = requests.post(base_url, headers=headers, json=data)
response.raise_for_status()
result = response.json()
sample_id = result['id']
sample_url = f'https://tria.ge/{sample_id}'
return {
'results': [
{
'types': 'link',
'values': sample_url,
'comment': f'Link to tria.ge analysis ({mode} mode)',
}
]
}
except Exception as e:
misperrors['error'] = f'Error submitting to tria.ge: {str(e)}'
return misperrors
def introspection():
return mispattributes
def version():
moduleinfo['config'] = moduleconfig
return moduleinfo

View File

@ -0,0 +1,91 @@
import json
import sys
import base64
import io
import zipfile
import requests
import hashlib
misperrors = {'error': 'Error'}
mispattributes = {'input': ['attachment', 'malware-sample'], 'output': ['link']}
moduleinfo = {
'version': '1',
'author': 'Karen Yousefi',
'description': 'Module to push malware samples to VirusTotal',
'module-type': ['expansion'],
'name': 'VirusTotal Upload',
'requirements': ['requests library'],
'logo': 'virustotal.png',
}
moduleconfig = ['virustotal_apikey']
def handler(q=False):
if q is False:
return False
request = json.loads(q)
try:
data = request.get("data")
if 'malware-sample' in request:
sample_filename = request.get("malware-sample").split("|", 1)[0]
data = base64.b64decode(data)
fl = io.BytesIO(data)
zf = zipfile.ZipFile(fl)
sample_hashname = zf.namelist()[0]
data = zf.read(sample_hashname, b"infected")
zf.close()
elif 'attachment' in request:
sample_filename = request.get("attachment")
data = base64.b64decode(data)
else:
misperrors['error'] = "No malware sample or attachment supplied"
return misperrors
except Exception:
misperrors['error'] = "Unable to process submitted sample data"
return misperrors
if request["config"].get("virustotal_apikey") is None:
misperrors["error"] = "Missing VirusTotal API key"
return misperrors
virustotal_apikey = request["config"].get("virustotal_apikey")
try:
url = "https://www.virustotal.com/api/v3/files"
headers = {
"accept": "application/json",
"x-apikey": virustotal_apikey,
}
files = {"file": (sample_filename, data)}
response = requests.post(url, headers=headers, files=files)
response.raise_for_status()
# Calculate SHA256 of the file
sha256 = hashlib.sha256(data).hexdigest()
virustotal_link = f"https://www.virustotal.com/gui/file/{sha256}"
except Exception as e:
misperrors['error'] = f"Unable to send sample to VirusTotal: {str(e)}"
return misperrors
r = {
'results': [
{
'types': 'link',
'values': virustotal_link,
'comment': 'Link to VirusTotal analysis',
}
]
}
return r
def introspection():
return mispattributes
def version():
moduleinfo['config'] = moduleconfig
return moduleinfo