+
- **descrption**:
>A hover module to get information about an url using a Google search.
- **features**:
@@ -655,6 +656,27 @@ Module to query a local copy of Maxmind's Geolite database.
-----
+#### [google_threat_intelligence](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/google_threat_intelligence.py)
+
+
+
+- **description**:
+An expansion module to have the observable's threat score assessed by Google Threat Intelligence.
+- **features**:
+>The module gives the Google Threat Intelligence assessment including a verdict for the given obsevable. [Example screeshot](https://github.com/MISP/MISP/assets/4747608/e275db2f-bb1e-4413-8cc0-ec3cb05e0414)
+]
+- **input**:
+>'hostname', 'domain', 'ip-src', 'ip-dst', 'md5', 'sha1', 'sha256', 'url'.
+- **output**:
+>Text fields containing the threat score, the severity, the verdict and the threat label of the observable inspected.
+- **references**:
+>https://gtidocs.virustotal.com/reference
+- **requirements**:
+>- pymisp
+>- vt
+
+-----
+
#### [greynoise](https://github.com/MISP/misp-modules/tree/main/misp_modules/modules/expansion/greynoise.py)
@@ -745,7 +767,7 @@ Expansion module to fetch the html content from an url and convert it into markd
HYAS Insight integration to MISP provides direct, high volume access to HYAS Insight data. It enables investigators and analysts to understand and defend against cyber adversaries and their infrastructure.
- **features**:
>This Module takes the IP Address, Domain, URL, Email, Phone Number, MD5, SHA1, Sha256, SHA512 MISP Attributes as input to query the HYAS Insight API.
-> The results of the HYAS Insight API are than are then returned and parsed into Hyas Insight Objects.
+> The results of the HYAS Insight API are than are then returned and parsed into Hyas Insight Objects.
>
>An API key is required to submit queries to the HYAS Insight API.
>
@@ -819,9 +841,9 @@ Module to access intelmqs eventdb.
An expansion module to query IP2Location.io to gather more information on a given IP address.
- **features**:
->The module takes an IP address attribute as input and queries the IP2Location.io API.
->Free plan user will get the basic geolocation informaiton, and different subsription plan will get more information on the IP address.
-> Refer to [pricing page](https://www.ip2location.io/pricing) for more information on data available for each plan.
+>The module takes an IP address attribute as input and queries the IP2Location.io API.
+>Free plan user will get the basic geolocation informaiton, and different subsription plan will get more information on the IP address.
+> Refer to [pricing page](https://www.ip2location.io/pricing) for more information on data available for each plan.
>
>More information on the responses content is available in the [documentation](https://www.ip2location.io/ip2location-documentation).
- **input**:
@@ -857,7 +879,7 @@ Module to query an IP ASN history service (https://github.com/D4-project/IPASN-H
An expansion module to query ipinfo.io to gather more information on a given IP address.
- **features**:
->The module takes an IP address attribute as input and queries the ipinfo.io API.
+>The module takes an IP address attribute as input and queries the ipinfo.io API.
>The geolocation information on the IP address is always returned.
>
>Depending on the subscription plan, the API returns different pieces of information then:
@@ -883,7 +905,7 @@ An expansion module to query ipinfo.io to gather more information on a given IP
IPQualityScore MISP Expansion Module for IP reputation, Email Validation, Phone Number Validation, Malicious Domain and Malicious URL Scanner.
- **features**:
>This Module takes the IP Address, Domain, URL, Email and Phone Number MISP Attributes as input to query the IPQualityScore API.
-> The results of the IPQualityScore API are than returned as IPQS Fraud and Risk Scoring Object.
+> The results of the IPQualityScore API are than returned as IPQS Fraud and Risk Scoring Object.
> The object contains a copy of the enriched attribute with added tags presenting the verdict based on fraud score,risk score and other attributes from IPQualityScore.
- **input**:
>A MISP attribute of type IP Address(ip-src, ip-dst), Domain(hostname, domain), URL(url, uri), Email Address(email, email-src, email-dst, target-email, whois-registrant-email) and Phone Number(phone-number, whois-registrant-phone).
@@ -1222,7 +1244,7 @@ Module to get information from AlienVault OTX.
An expansion module to query the CIRCL Passive SSH.
- **features**:
>The module queries the Passive SSH service from CIRCL.
->
+>
> The module can be used an hover module but also an expansion model to add related MISP objects.
>
- **input**:
@@ -1965,7 +1987,7 @@ Module to query a local instance of uwhois (https://github.com/rafiot/uwhoisd).
An expansion module for https://whoisfreaks.com/ that will provide an enriched analysis of the provided domain, including WHOIS and DNS information.
-Our Whois service, DNS Lookup API, and SSL analysis, equips organizations with comprehensive threat intelligence and attack surface analysis capabilities for enhanced security.
+Our Whois service, DNS Lookup API, and SSL analysis, equips organizations with comprehensive threat intelligence and attack surface analysis capabilities for enhanced security.
Explore our website's product section at https://whoisfreaks.com/ for a wide range of additional services catering to threat intelligence and attack surface analysis needs.
- **features**:
>The module takes a domain as input and queries the Whoisfreaks API with it.
@@ -2104,7 +2126,7 @@ Module to process a query on Yeti.
> - https://github.com/sebdraven/pyeti
- **requirements**:
> - pyeti
-> - API key
+> - API key
-----
@@ -2261,7 +2283,7 @@ Simple export of a MISP event to PDF.
> 'Activate_galaxy_description' is a boolean (True or void) to activate the description of event related galaxies.
> 'Activate_related_events' is a boolean (True or void) to activate the description of related event. Be aware this might leak information on confidential events linked to the current event !
> 'Activate_internationalization_fonts' is a boolean (True or void) to activate Noto fonts instead of default fonts (Helvetica). This allows the support of CJK alphabet. Be sure to have followed the procedure to download Noto fonts (~70Mo) in the right place (/tools/pdf_fonts/Noto_TTF), to allow PyMisp to find and use them during PDF generation.
-> 'Custom_fonts_path' is a text (path or void) to the TTF file of your choice, to create the PDF with it. Be aware the PDF won't support bold/italic/special style anymore with this option
+> 'Custom_fonts_path' is a text (path or void) to the TTF file of your choice, to create the PDF with it. Be aware the PDF won't support bold/italic/special style anymore with this option
- **input**:
>MISP Event
- **output**:
diff --git a/documentation/website/expansion/google_threat_intelligence.json b/documentation/website/expansion/google_threat_intelligence.json
new file mode 100644
index 0000000..89f4bff
--- /dev/null
+++ b/documentation/website/expansion/google_threat_intelligence.json
@@ -0,0 +1,14 @@
+{
+ "description": "An expansion module to have the observable's threat score assessed by Google Threat Intelligence.",
+ "logo": "google_threat_intelligence.png",
+ "requirements": [
+ "An access to the Google Threat Intelligence API (apikey), with a high request rate limit."
+ ],
+ "input": "A domain, hash (md5, sha1, sha256 or sha512), hostname or IP address attribute.",
+ "output": "Text fields containing the threat score, the severity, the verdict and the threat label of the observable inspected.",
+ "references": [
+ "https://www.virustotal.com/",
+ "https://gtidocs.virustotal.com/reference"
+ ],
+ "features": "GTI assessment for the given observable, this include information about level of severity, a clear verdict (malicious, suspicious, undetected and bening) and additional information provided by the Mandiant expertise combined with the VirusTotal database.\n\n[Output example screeshot](https://github.com/MISP/MISP/assets/4747608/e275db2f-bb1e-4413-8cc0-ec3cb05e0414)"
+}
diff --git a/misp_modules/modules/expansion/__init__.py b/misp_modules/modules/expansion/__init__.py
index 8cb745a..b76ed93 100644
--- a/misp_modules/modules/expansion/__init__.py
+++ b/misp_modules/modules/expansion/__init__.py
@@ -20,7 +20,8 @@ __all__ = ['cuckoo_submit', 'vmray_submit', 'bgpranking', 'circl_passivedns', 'c
'trustar_enrich', 'recordedfuture', 'html_to_markdown', 'socialscan', 'passive-ssh',
'qintel_qsentry', 'mwdb', 'hashlookup', 'mmdb_lookup', 'ipqs_fraud_and_risk_scoring',
'clamav', 'jinja_template_rendering','hyasinsight', 'variotdbs', 'crowdsec',
- 'extract_url_components', 'ipinfo', 'whoisfreaks', 'ip2locationio', 'vysion', 'stairwell']
+ 'extract_url_components', 'ipinfo', 'whoisfreaks', 'ip2locationio', 'vysion', 'stairwell',
+ 'google_threat_intelligence']
minimum_required_fields = ('type', 'uuid', 'value')
diff --git a/misp_modules/modules/expansion/google_threat_intelligence.py b/misp_modules/modules/expansion/google_threat_intelligence.py
new file mode 100644
index 0000000..220b96f
--- /dev/null
+++ b/misp_modules/modules/expansion/google_threat_intelligence.py
@@ -0,0 +1,322 @@
+#!/usr/local/bin/python
+# Copyright © 2024 The Google Threat Intelligence authors. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Google Threat Intelligence MISP expansion module."""
+
+from urllib import parse
+import vt
+import pymisp
+
+
+MISP_ATTRIBUTES = {
+ 'input': [
+ 'hostname',
+ 'domain',
+ 'ip-src',
+ 'ip-dst',
+ 'md5',
+ 'sha1',
+ 'sha256',
+ 'url',
+ ],
+ 'format': 'misp_standard',
+}
+
+MODULE_INFO = {
+ 'version': '1',
+ 'author': 'Google Threat Intelligence team',
+ 'description': ('An expansion module to have the observable\'s threat'
+ ' score assessed by Google Threat Intelligence.'),
+ 'module-type': ['expansion'],
+ 'config': [
+ 'apikey',
+ 'event_limit',
+ 'proxy_host',
+ 'proxy_port',
+ 'proxy_username',
+ 'proxy_password'
+ ]
+}
+
+DEFAULT_RESULTS_LIMIT = 10
+
+
+class GoogleThreatIntelligenceParser:
+ """Main parser class to create the MISP event."""
+ def __init__(self, client: vt.Client, limit: int) -> None:
+ self.client = client
+ self.limit = limit or DEFAULT_RESULTS_LIMIT
+ self.misp_event = pymisp.MISPEvent()
+ self.attribute = pymisp.MISPAttribute()
+ self.parsed_objects = {}
+ self.input_types_mapping = {
+ 'ip-src': self.parse_ip,
+ 'ip-dst': self.parse_ip,
+ 'domain': self.parse_domain,
+ 'hostname': self.parse_domain,
+ 'md5': self.parse_hash,
+ 'sha1': self.parse_hash,
+ 'sha256': self.parse_hash,
+ 'url': self.parse_url
+ }
+ self.proxies = None
+
+ def query_api(self, attribute: dict) -> None:
+ """Get data from the API and parse it."""
+ self.attribute.from_dict(**attribute)
+ self.input_types_mapping[self.attribute.type](self.attribute.value)
+
+ def get_results(self) -> dict:
+ """Serialize the MISP event."""
+ event = self.misp_event.to_dict()
+ results = {
+ key: event[key] for key in ('Attribute', 'Object') \
+ if (key in event and event[key])
+ }
+ return {'results': results}
+
+ def create_gti_report_object(self, report):
+ """Create GTI report object."""
+ report = report.to_dict()
+ permalink = ('https://www.virustotal.com/gui/'
+ f"{report['type']}/{report['id']}")
+ report_object = pymisp.MISPObject('Google-Threat-Intel-report')
+ report_object.add_attribute('permalink', type='link', value=permalink)
+ report_object.add_attribute(
+ 'Threat Score', type='text',
+ value=get_key(
+ report, 'attributes.gti_assessment.threat_score.value'))
+ report_object.add_attribute(
+ 'Verdict', type='text',
+ value=get_key(
+ report, 'attributes.gti_assessment.verdict.value').replace(
+ 'VERDICT_', ''))
+ report_object.add_attribute(
+ 'Severity', type='text',
+ value=get_key(
+ report, 'attributes.gti_assessment.severity.value').replace(
+ 'SEVERITY_', ''))
+ report_object.add_attribute(
+ 'Threat Label', type='text',
+ value=get_key(
+ report, ('attributes.popular_threat_classification'
+ '.suggested_threat_label')))
+ self.misp_event.add_object(**report_object)
+ return report_object.uuid
+
+ def parse_domain(self, domain: str) -> str:
+ """Create domain MISP object."""
+ domain_report = self.client.get_object(f'/domains/{domain}')
+
+ # DOMAIN
+ domain_object = pymisp.MISPObject('domain-ip')
+ domain_object.add_attribute(
+ 'domain', type='domain', value=domain_report.id)
+
+ report_uuid = self.create_gti_report_object(domain_report)
+ domain_object.add_reference(report_uuid, 'analyzed-with')
+ self.misp_event.add_object(**domain_object)
+ return domain_object.uuid
+
+ def parse_hash(self, file_hash: str) -> str:
+ """Create hash MISP object."""
+ file_report = self.client.get_object(f'/files/{file_hash}')
+ file_object = pymisp.MISPObject('file')
+ for hash_type in ('md5', 'sha1', 'sha256'):
+ file_object.add_attribute(
+ hash_type,
+ **{'type': hash_type, 'value': file_report.get(hash_type)})
+
+ report_uuid = self.create_gti_report_object(file_report)
+ file_object.add_reference(report_uuid, 'analyzed-with')
+ self.misp_event.add_object(**file_object)
+ return file_object.uuid
+
+ def parse_ip(self, ip: str) -> str:
+ """Create ip MISP object."""
+ ip_report = self.client.get_object(f'/ip_addresses/{ip}')
+
+ # IP
+ ip_object = pymisp.MISPObject('domain-ip')
+ ip_object.add_attribute('ip', type='ip-dst', value=ip_report.id)
+
+ report_uuid = self.create_gti_report_object(ip_report)
+ ip_object.add_reference(report_uuid, 'analyzed-with')
+ self.misp_event.add_object(**ip_object)
+ return ip_object.uuid
+
+ def parse_url(self, url: str) -> str:
+ """Create URL MISP object."""
+ url_id = vt.url_id(url)
+ url_report = self.client.get_object(f'/urls/{url_id}')
+
+ url_object = pymisp.MISPObject('url')
+ url_object.add_attribute('url', type='url', value=url_report.url)
+
+ report_uuid = self.create_gti_report_object(url_report)
+ url_object.add_reference(report_uuid, 'analyzed-with')
+ self.misp_event.add_object(**url_object)
+ return url_object.uuid
+
+
+def get_key(dictionary, key, default_value=''):
+ """Get value from nested dictionaries."""
+ dictionary = dictionary or {}
+ keys = key.split('.')
+ field_name = keys.pop()
+ for k in keys:
+ if k not in dictionary:
+ return default_value
+ dictionary = dictionary[k]
+ return dictionary.get(field_name, default_value)
+
+
+def get_proxy_settings(config: dict) -> dict:
+ """Returns proxy settings in the requests format or None if not set up."""
+ proxies = None
+ host = config.get('proxy_host')
+ port = config.get('proxy_port')
+ username = config.get('proxy_username')
+ password = config.get('proxy_password')
+
+ if host:
+ if not port:
+ raise KeyError(
+ ('The google_threat_intelligence_proxy_host config is set, '
+ 'please also set the virustotal_proxy_port.'))
+ parsed = parse.urlparse(host)
+ if 'http' in parsed.scheme:
+ scheme = 'http'
+ else:
+ scheme = parsed.scheme
+ netloc = parsed.netloc
+ host = f'{netloc}:{port}'
+
+ if username:
+ if not password:
+ raise KeyError(('The google_threat_intelligence_'
+ ' proxy_host config is set, please also'
+ ' set the virustotal_proxy_password.'))
+ auth = f'{username}:{password}'
+ host = auth + '@' + host
+
+ proxies = {
+ 'http': f'{scheme}://{host}',
+ 'https': f'{scheme}://{host}'
+ }
+ return proxies
+
+
+def dict_handler(request: dict):
+ """MISP entry point fo the module."""
+ if not request.get('config') or not request['config'].get('apikey'):
+ return {
+ 'error': ('A Google Threat Intelligence api '
+ 'key is required for this module.')
+ }
+
+ if not request.get('attribute'):
+ return {
+ 'error': ('This module requires an "attribute" field as input,'
+ ' which should contain at least a type, a value and an'
+ ' uuid.')
+ }
+
+ if request['attribute']['type'] not in MISP_ATTRIBUTES['input']:
+ return {'error': 'Unsupported attribute type.'}
+
+ event_limit = request['config'].get('event_limit')
+ attribute = request['attribute']
+
+ try:
+ proxy_settings = get_proxy_settings(request.get('config'))
+ client = vt.Client(
+ request['config']['apikey'],
+ headers={
+ 'x-tool': 'MISPModuleGTIExpansion',
+ },
+ proxy=proxy_settings['http'] if proxy_settings else None)
+ parser = GoogleThreatIntelligenceParser(
+ client, int(event_limit) if event_limit else None)
+ parser.query_api(attribute)
+ except vt.APIError as ex:
+ return {'error': ex.message}
+ except KeyError as ex:
+ return {'error': str(ex)}
+
+ return parser.get_results()
+
+
+def introspection():
+ """Returns the module input attributes required."""
+ return MISP_ATTRIBUTES
+
+
+def version():
+ """Returns the module metadata."""
+ return MODULE_INFO
+
+
+if __name__ == '__main__':
+ # Testing/debug calls.
+ import os
+ api_key = os.getenv('GTI_API_KEY')
+ # File
+ request_data = {
+ 'config': {'apikey': api_key},
+ 'attribute': {
+ 'type': 'sha256',
+ 'value': ('ed01ebfbc9eb5bbea545af4d01bf5f10'
+ '71661840480439c6e5babe8e080e41aa')
+ }
+ }
+ response = dict_handler(request_data)
+ report_obj = response['results']['Object'][0]
+ print(report_obj.to_dict())
+
+ # URL
+ request_data = {
+ 'config': {'apikey': api_key},
+ 'attribute': {
+ 'type': 'url',
+ 'value': 'http://47.21.48.182:60813/Mozi.a'
+ }
+ }
+ response = dict_handler(request_data)
+ report_obj = response['results']['Object'][0]
+ print(report_obj.to_dict())
+
+ # Ip
+ request_data = {
+ 'config': {'apikey': api_key},
+ 'attribute': {
+ 'type': 'ip-src',
+ 'value': '180.72.148.38'
+ }
+ }
+ response = dict_handler(request_data)
+ report_obj = response['results']['Object'][0]
+ print(report_obj.to_dict())
+
+ # Domain
+ request_data = {
+ 'config': {'apikey': api_key},
+ 'attribute': {
+ 'type': 'domain',
+ 'value': 'qexyhuv.com'
+ }
+ }
+ response = dict_handler(request_data)
+ report_obj = response['results']['Object'][0]
+ print(report_obj.to_dict())
diff --git a/website/README.md b/website/README.md
index f79774b..5b3e608 100644
--- a/website/README.md
+++ b/website/README.md
@@ -19,6 +19,8 @@ git submodule init && git submodule update ## Initialize misp-objects submodul
python3 app.py -i ## Initialize db
```
+Don't forget to install **misp-modules**...
+
## Config
Edit `config.py`
@@ -35,8 +37,6 @@ Edit `config.py`
- `ADMIN_PASSWORD`: Password for Admin user if `ADMIN_USER` is True
-
-
Rename `config.cfg.sample` to `config.cfg` then edit it:
- `ADMIN_USER`: If True, config page will not be accessible
diff --git a/website/app/__init__.py b/website/app/__init__.py
index 6cf0a3c..2996726 100644
--- a/website/app/__init__.py
+++ b/website/app/__init__.py
@@ -37,6 +37,7 @@ def create_app():
app.register_blueprint(home_blueprint, url_prefix="/")
app.register_blueprint(history_blueprint, url_prefix="/")
app.register_blueprint(account_blueprint, url_prefix="/")
+ csrf.exempt(home_blueprint)
return app
diff --git a/website/app/db_class/db.py b/website/app/db_class/db.py
index 924b0fc..89ec6ee 100644
--- a/website/app/db_class/db.py
+++ b/website/app/db_class/db.py
@@ -38,7 +38,7 @@ class Session_db(db.Model):
"id": self.id,
"uuid": self.uuid,
"modules": json.loads(self.modules_list),
- "query_enter": self.query_enter,
+ "query_enter": json.loads(self.query_enter),
"input_query": self.input_query,
"config_module": json.loads(self.config_module),
"result": json.loads(self.result),
@@ -51,7 +51,7 @@ class Session_db(db.Model):
json_dict = {
"uuid": self.uuid,
"modules": json.loads(self.modules_list),
- "query": self.query_enter,
+ "query": json.loads(self.query_enter),
"input": self.input_query,
"query_date": self.query_date.strftime('%Y-%m-%d %H:%M')
}
diff --git a/website/app/history/history_core.py b/website/app/history/history_core.py
index baa2b1e..200a96b 100644
--- a/website/app/history/history_core.py
+++ b/website/app/history/history_core.py
@@ -146,8 +146,8 @@ def util_remove_node_session(node_uuid, parent, parent_path):
child = parent["children"][i]
if child["uuid"] == node_uuid:
del parent_path["children"][i]
- return
- elif child["children"]:
+ return True
+ elif "children" in child and child["children"]:
return util_remove_node_session(node_uuid, child, parent_path["children"][i])
def remove_node_session(node_uuid):
@@ -160,7 +160,9 @@ def remove_node_session(node_uuid):
loc = i
break
elif q_value["children"]:
- return util_remove_node_session(node_uuid, q_value, sess[keys_list[i]])
+ if util_remove_node_session(node_uuid, q_value, sess[keys_list[i]]):
+ loc = i
+ break
if loc:
del sess[keys_list[i]]
diff --git a/website/app/home.py b/website/app/home.py
index 6c4e669..7a675cb 100644
--- a/website/app/home.py
+++ b/website/app/home.py
@@ -1,9 +1,10 @@
+import ast
import json
-from flask import Blueprint, render_template, request, jsonify, session as sess
+from flask import Blueprint, redirect, render_template, request, jsonify, session as sess
from flask_login import current_user
from . import session_class as SessionModel
from . import home_core as HomeModel
-from .utils.utils import admin_user_active
+from .utils.utils import admin_user_active, FLOWINTEL_URL
home_blueprint = Blueprint(
'home',
@@ -13,18 +14,35 @@ home_blueprint = Blueprint(
)
-@home_blueprint.route("/")
+@home_blueprint.route("/", methods=["GET", "POST"])
def home():
+ try:
+ del sess["query"]
+ except:
+ pass
sess["admin_user"] = bool(admin_user_active())
if "query" in request.args:
- return render_template("home.html", query=request.args.get("query"))
+ sess["query"] = ast.literal_eval(request.args.get("query"))
+ if "query" in request.form:
+ sess["query"] = json.loads(request.form.get("query"))
return render_template("home.html")
+@home_blueprint.route("/get_query", methods=['GET', 'POST'])
+def get_query():
+ """Get result from flowintel"""
+ if "query" in sess:
+ return {"query": sess.get("query")}
+ return {"message": "No query"}
+
@home_blueprint.route("/home/Input Attribute:
diff --git a/website/app/static/js/mispParser.js b/website/app/static/js/mispParser.js index f3901e4..4521f48 100644 --- a/website/app/static/js/mispParser.js +++ b/website/app/static/js/mispParser.js @@ -10,13 +10,15 @@ function parseMispObject(misp_object, query_url, functionToCall){ if(query_url){ $query=$("").attr("href", query_url+v.value).text("query").css("margin-left", "10px") } - // `_${functionToCall.name}('${v.value}')` refer to 'window._query_as_same = query_as_same' in my vue file - $query_same = $("