Merge branch 'main' of github.com:misp/pymisp

pull/1097/head
Christian Studer 2023-11-09 16:12:29 +01:00
commit f7ca4af380
15 changed files with 1471 additions and 1070 deletions

View File

@ -38,7 +38,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL

View File

@ -17,7 +17,7 @@ jobs:
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
submodules: recursive

View File

@ -1,14 +1,14 @@
version: 2
build:
os: "ubuntu-22.04"
tools:
python: "3"
python:
version: 3.8
install:
- method: pip
path: .
extra_requirements:
- docs
build:
image: latest
formats: all

View File

@ -2,6 +2,159 @@ Changelog
=========
v2.4.178 (2023-10-24)
---------------------
New
~~~
- Run tests on python 3.12 too. [Raphaël Vinot]
Changes
~~~~~~~
- Bump version, make __version__ dynamic. [Raphaël Vinot]
- Bump deps, allow older jsonschema for compatibility. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Make mypy happy. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Disable search logs tests for now. [Raphaël Vinot]
- Disable fastmode, reenable fetching files. [Raphaël Vinot]
- Try to speedup tests by not importing galaxies, taxos, ... [Raphaël
Vinot]
- Do not clone repo from test. [Raphaël Vinot]
Fix
~~~
- Make other fieldnames in CSV also valid... [Raphaël Vinot]
- Make fieldnames actually valid. [Raphaël Vinot]
- Remove CI for python 3.12, waiting for pydeep wheels. [Raphaël Vinot]
- Allow object-relation names with uppercase characters defined in the
templates. [Raphaël Vinot]
- Check if path exists in tests. [Raphaël Vinot]
Other
~~~~~
- Ch: Bump deps. [Raphaël Vinot]
v2.4.176 (2023-09-15)
---------------------
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump version, deps. [Raphaël Vinot]
- Bump objects. [Raphaël Vinot]
- Bump deps, objects. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
Fix
~~~
- Avoid exception when data is an empty iterator. [Raphaël Vinot]
Fix #1053
Other
~~~~~
- Revert "build(deps): bump codecov/codecov-action from 3 to 4" [Raphaël
Vinot]
This reverts commit b7bb6b74317b70613ed42ea234eaafb00da6e5c6.
- Build(deps): bump codecov/codecov-action from 3 to 4.
[dependabot[bot]]
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3 to 4.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v3...v4)
---
updated-dependencies:
- dependency-name: codecov/codecov-action
dependency-type: direct:production
update-type: version-update:semver-major
...
- Build(deps): bump actions/checkout from 3 to 4. [dependabot[bot]]
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
v2.4.175 (2023-08-23)
---------------------
Changes
~~~~~~~
- Bump objects, missed that. [Raphaël Vinot]
- Bump changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump deps, readthedocs config. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
Fix
~~~
- Update Sharing group info from full object. [Raphaël Vinot]
Fix #1049
- Changes in msg-extract strip a character. [Raphaël Vinot]
v2.4.174 (2023-07-31)
---------------------
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump version, templates. [Raphaël Vinot]
- Bump deps, fix code accordingly. [Raphaël Vinot]
Fix
~~~
- Push code changes related to deps upgrade... [Raphaël Vinot]
Other
~~~~~
- Git: Bump deps. [Raphaël Vinot]
v2.4.173 (2023-07-10)
---------------------
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump objects. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump objects. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
Fix
~~~
- Maybe fixing a CakePHP issue. [Raphaël Vinot]
Maybe fixing #1014
- Use proper endpoint to unpublish event. [Raphaël Vinot]
Fix #1012
Other
~~~~~
- Feat: introduce setter for galaxies. [Sura De Silva]
v2.4.172 (2023-06-08)
---------------------
@ -17,6 +170,7 @@ Changes
Fix
~~~
- Proper changelog bump. [Raphaël Vinot]
- Properly bump version. [Raphaël Vinot]
Other

2208
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,10 +1,13 @@
__version__ = '2.4.172'
import logging
import sys
import warnings
import importlib.metadata
logger = logging.getLogger(__name__)
__version__ = importlib.metadata.version("pymisp")
def warning_2024():
if sys.version_info < (3, 10):

View File

@ -33,7 +33,7 @@ if sys.platform == 'linux':
# Enable TCP keepalive by default on every requests
import socket
from urllib3.connection import HTTPConnection
HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [
HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [ # type: ignore
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1), # enable keepalive
(socket.SOL_TCP, socket.TCP_KEEPIDLE, 30), # Start pinging after 30s of idle time
(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10), # ping every 10s
@ -265,9 +265,9 @@ class PyMISP:
@property
def pymisp_version_main(self) -> Dict:
"""Get the most recent version of PyMISP from github"""
r = requests.get('https://raw.githubusercontent.com/MISP/PyMISP/main/pymisp/__init__.py')
r = requests.get('https://raw.githubusercontent.com/MISP/PyMISP/main/pyproject.toml')
if r.status_code == 200:
version = re.findall("__version__ = '(.*)'", r.text)
version = re.findall('version = "(.*)"', r.text)
return {'version': version[0]}
return {'error': 'Impossible to retrieve the version of the main branch.'}
@ -2046,6 +2046,7 @@ class PyMISP:
sid = get_uuid_or_id_from_abstract_misp(sharing_group)
else:
sid = get_uuid_or_id_from_abstract_misp(sharing_group_id)
sharing_group.pop('modified', None) # Quick fix for https://github.com/MISP/PyMISP/issues/1049 - remove when fixed in MISP.
r = self._prepare_request('POST', f'sharing_groups/edit/{sid}', data=sharing_group)
updated_sharing_group = self._check_json_response(r)
if not (self.global_pythonify or pythonify) or 'errors' in updated_sharing_group:
@ -3698,8 +3699,9 @@ class PyMISP:
def __repr__(self):
return f'<{self.__class__.__name__}(url={self.root_url})'
def _prepare_request(self, request_type: str, url: str, data: Union[Iterable, Mapping, AbstractMISP, bytes] = {}, params: Mapping = {},
kw_params: Mapping = {}, output_type: str = 'json', content_type: str = 'json') -> requests.Response:
def _prepare_request(self, request_type: str, url: str, data: Optional[Union[Iterable, Mapping, AbstractMISP, bytes]] = None,
params: Mapping = {}, kw_params: Mapping = {},
output_type: str = 'json', content_type: str = 'json') -> requests.Response:
'''Prepare a request for python-requests'''
if url[0] == '/':
# strip it: it will fail if MISP is in a sub directory
@ -3708,13 +3710,15 @@ class PyMISP:
# so we need to make it a + instead and hope for the best
url = url.replace(' ', '+')
url = urljoin(self.root_url, url)
if data == {} or isinstance(data, bytes):
d = data
elif data:
if isinstance(data, dict): # Else, we can directly json encode.
# Remove None values.
data = {k: v for k, v in data.items() if v is not None}
d = json.dumps(data, default=pymisp_json_default)
d: Optional[Union[bytes, str]] = None
if data is not None:
if isinstance(data, bytes):
d = data
else:
if isinstance(data, dict):
# Remove None values.
data = {k: v for k, v in data.items() if v is not None}
d = json.dumps(data, default=pymisp_json_default)
logger.debug(f'{request_type} - {url}')
if d is not None:

@ -1 +1 @@
Subproject commit 2ca2667d7668067f906e9601e0c97a79d4c7ccf1
Subproject commit 5feb0527321ecfc5a7028df5db561c95d0fb4798

View File

@ -1721,6 +1721,13 @@ class MISPEvent(AbstractMISP):
def galaxies(self) -> List[MISPGalaxy]:
return self.Galaxy
@galaxies.setter
def galaxies(self, galaxies: List[MISPGalaxy]):
if all(isinstance(x, MISPGalaxy) for x in galaxies):
self.Galaxy = galaxies
else:
raise PyMISPError('All the attributes have to be of type MISPGalaxy.')
@property
def objects(self) -> List[MISPObject]:
return self.Object

View File

@ -2,21 +2,25 @@
# -*- coding: utf-8 -*-
from pathlib import Path
from typing import List, Optional
import csv
from pymisp import MISPObject
class CSVLoader():
def __init__(self, template_name: str, csv_path: Path, fieldnames: list = [], has_fieldnames=False,
def __init__(self, template_name: str, csv_path: Path, fieldnames: Optional[List[str]] = None, has_fieldnames=False,
delimiter: str = ',', quotechar: str = '"'):
self.template_name = template_name
self.delimiter = delimiter
self.quotechar = quotechar
self.csv_path = csv_path
self.fieldnames = [f.strip().lower() for f in fieldnames]
self.fieldnames = []
if fieldnames:
self.fieldnames = [f.strip() for f in fieldnames]
if not self.fieldnames:
# If the user doesn't pass fieldnames, we assume the CSV has them.
# If the user doesn't pass fieldnames, they must be in the CSV.
self.has_fieldnames = True
else:
self.has_fieldnames = has_fieldnames
@ -28,22 +32,23 @@ class CSVLoader():
with open(self.csv_path, newline='') as csvfile:
reader = csv.reader(csvfile, delimiter=self.delimiter, quotechar=self.quotechar)
if self.has_fieldnames:
# The file has fieldnames, we either ignore it, or validate its validity
fieldnames = [f.strip().lower() for f in reader.__next__()]
# The file has fieldnames, we either ignore it, or use them as object-relation
fieldnames = [f.strip() for f in reader.__next__()]
if not self.fieldnames:
self.fieldnames = fieldnames
if not self.fieldnames:
raise Exception('No fieldnames, impossible to create objects.')
else:
# Check if the CSV file has a header, and if it matches with the object template
tmp_object = MISPObject(self.template_name)
if not tmp_object._definition['attributes']:
raise Exception(f'Unable to find the object template ({self.template_name}), impossible to create objects.')
allowed_fieldnames = list(tmp_object._definition['attributes'].keys())
for fieldname in self.fieldnames:
if fieldname not in allowed_fieldnames:
raise Exception(f'{fieldname} is not a valid object relation for {self.template_name}: {allowed_fieldnames}')
# Check if the CSV file has a header, and if it matches with the object template
tmp_object = MISPObject(self.template_name)
if not tmp_object._definition['attributes']:
raise Exception(f'Unable to find the object template ({self.template_name}), impossible to create objects.')
allowed_fieldnames = list(tmp_object._definition['attributes'].keys())
for fieldname in self.fieldnames:
if fieldname not in allowed_fieldnames:
raise Exception(f'{fieldname} is not a valid object relation for {self.template_name}: {allowed_fieldnames}')
for row in reader:
tmp_object = MISPObject(self.template_name)

View File

@ -11,7 +11,9 @@ from io import BytesIO
from pathlib import Path
from typing import Union, List, Tuple, Dict, cast, Any, Optional
from extract_msg import openMsg, MessageBase
from extract_msg import openMsg
from extract_msg.msg_classes import MessageBase
from extract_msg.properties import FixedLengthProp
from RTFDE.exceptions import MalformedEncapsulatedRtf, NotEncapsulatedRtf # type: ignore
from RTFDE.deencapsulate import DeEncapsulator # type: ignore
from oletools.common.codepages import codepage2codec # type: ignore
@ -111,8 +113,11 @@ class EMailObject(AbstractMISPObjectGenerator):
"cte": "base64"}
if msg_obj.htmlBody is not None:
try:
_html_encoding_raw = msg_obj.props['3FDE0003'].value
_html_encoding = codepage2codec(_html_encoding_raw)
if isinstance(msg_obj.props['3FDE0003'], FixedLengthProp):
_html_encoding_raw = msg_obj.props['3FDE0003'].value
_html_encoding = codepage2codec(_html_encoding_raw)
else:
_html_encoding = msg_obj.stringEncoding
except KeyError:
_html_encoding = msg_obj.stringEncoding
body['html'] = {'obj': msg_obj.htmlBody.decode(),

View File

@ -1,6 +1,6 @@
[tool.poetry]
name = "pymisp"
version = "2.4.172"
version = "2.4.178"
description = "Python API for MISP."
authors = ["Raphaël Vinot <raphael.vinot@circl.lu>"]
license = "BSD-2-Clause"
@ -27,7 +27,6 @@ classifiers=[
include = [
"CHANGELOG.txt",
"README.md",
"pymisp/data/*.json",
"pymisp/data/misp-objects/schema_objects.json",
"pymisp/data/misp-objects/schema_relationships.json",
@ -44,44 +43,48 @@ include = [
python = "^3.8"
requests = "^2.31.0"
python-dateutil = "^2.8.2"
jsonschema = "^4.17.3"
jsonschema = ">=4.17.3"
deprecated = "^1.2.14"
extract_msg = {version = "^0.41.2", optional = true}
RTFDE = {version = "^0.0.2", optional = true}
extract_msg = {version = "^0.45.0", optional = true}
RTFDE = {version = "^0.1.0", optional = true}
oletools = {version = "^0.60.1", optional = true}
python-magic = {version = "^0.4.27", optional = true}
pydeep2 = {version = "^0.5.1", optional = true}
lief = {version = "^0.13.1", optional = true}
lief = {version = "^0.13.2", optional = true}
beautifulsoup4 = {version = "^4.12.2", optional = true}
validators = {version = "^0.20.0", optional = true}
sphinx-autodoc-typehints = {version = "^1.23.0", optional = true}
validators = {version = "^0.22.0", optional = true}
sphinx-autodoc-typehints = {version = "^1.24.0", optional = true}
recommonmark = {version = "^0.7.1", optional = true}
reportlab = {version = "^4.0.4", optional = true}
reportlab = {version = "^4.0.6", optional = true}
pyfaup = {version = "^1.2", optional = true}
publicsuffixlist = {version = "^0.10.0.20230608", optional = true}
publicsuffixlist = {version = "^0.10.0.20231022", optional = true}
urllib3 = {extras = ["brotli"], version = "*", optional = true}
Sphinx = [
{version = "<7.2", python = "<3.9", optional = true},
{version = "^7.2", python = ">=3.9", optional = true}
]
[tool.poetry.extras]
fileobjects = ['python-magic', 'pydeep2', 'lief']
openioc = ['beautifulsoup4']
virustotal = ['validators']
docs = ['sphinx-autodoc-typehints', 'recommonmark']
docs = ['sphinx-autodoc-typehints', 'recommonmark', 'sphinx']
pdfexport = ['reportlab']
url = ['pyfaup']
email = ['extract_msg', "RTFDE", "oletools"]
brotli = ['urllib3']
[tool.poetry.group.dev.dependencies]
requests-mock = "^1.10.0"
mypy = "^1.3.0"
requests-mock = "^1.11.0"
mypy = "^1.6.1"
ipython = [
{version = "<8.13.0", python = "<3.9"},
{version = "^8.13.0", python = ">=3.9"}
]
jupyterlab = "^4.0.1"
types-requests = "^2.31.0.1"
types-python-dateutil = "^2.8.19.13"
types-redis = "^4.5.5.2"
jupyterlab = "^4.0.7"
types-requests = "^2.31.0.10"
types-python-dateutil = "^2.8.19.14"
types-redis = "^4.6.0.7"
types-Flask = "^1.1.6"
pytest-cov = "^4.1.0"

View File

@ -1,4 +1,4 @@
MD5, SHA1, SHA256
md5, sha1, sha256
644087ccca16d2a728ef7685a4106f09, eabd6974ac71efd72d9e0688d5a6131f336d169c, 385e31c97e3a07bbb81513f0cd0979e64e6b014943902efd002f57b21eadd41e
34187a34d0a3c5d63016c26346371b54, ce8209ff9828aa8cb095bd7d1589fc4d394c298c, 5f815b8a8e77731c9ca2b3a07a27f880ef24d54e458d77bdabbbaf2269fe96c3
871aa15f4d61c85e1284e1be3f99f705, 236eac0b19f91117b27f1b198a4d8490d99ec2e5, b434bccf0a5ff75b27184e661df751466aef69f35fbd7b8b8692302b8b886262

1 MD5 md5 SHA1 sha1 SHA256 sha256
2 644087ccca16d2a728ef7685a4106f09 644087ccca16d2a728ef7685a4106f09 eabd6974ac71efd72d9e0688d5a6131f336d169c eabd6974ac71efd72d9e0688d5a6131f336d169c 385e31c97e3a07bbb81513f0cd0979e64e6b014943902efd002f57b21eadd41e 385e31c97e3a07bbb81513f0cd0979e64e6b014943902efd002f57b21eadd41e
3 34187a34d0a3c5d63016c26346371b54 34187a34d0a3c5d63016c26346371b54 ce8209ff9828aa8cb095bd7d1589fc4d394c298c ce8209ff9828aa8cb095bd7d1589fc4d394c298c 5f815b8a8e77731c9ca2b3a07a27f880ef24d54e458d77bdabbbaf2269fe96c3 5f815b8a8e77731c9ca2b3a07a27f880ef24d54e458d77bdabbbaf2269fe96c3
4 871aa15f4d61c85e1284e1be3f99f705 871aa15f4d61c85e1284e1be3f99f705 236eac0b19f91117b27f1b198a4d8490d99ec2e5 236eac0b19f91117b27f1b198a4d8490d99ec2e5 b434bccf0a5ff75b27184e661df751466aef69f35fbd7b8b8692302b8b886262 b434bccf0a5ff75b27184e661df751466aef69f35fbd7b8b8692302b8b886262

View File

@ -64,7 +64,9 @@ class TestEmailObject(unittest.TestCase):
self._get_values(eml_email_object, "to")[0])
self.assertEqual(self._get_values(email_object, "from")[0],
self._get_values(eml_email_object, "from")[0])
self.assertEqual(self._get_values(email_object, "from-display-name")[0],
dirty_display_name = self._get_values(email_object, "from-display-name")[0]
dirty_display_name = dirty_display_name[:-2] + dirty_display_name[-1]
self.assertEqual(dirty_display_name,
self._get_values(eml_email_object, "from-display-name")[0])
self.assertEqual(len(self._get_values(email_object, "email-body")), 2)

View File

@ -51,7 +51,11 @@ urllib3.disable_warnings()
fast_mode = False
if not Path('tests/viper-test-files').exists():
test_file_path = Path('tests/viper-test-files')
print(test_file_path, 'exists: ', test_file_path.exists())
if not test_file_path.exists():
print('The test files are missing, pulling it.')
os.system('git clone https://github.com/viper-framework/viper-test-files.git tests/viper-test-files')
@ -1838,7 +1842,7 @@ class TestComprehensive(unittest.TestCase):
event.add_object(**o)
csv2 = CSVLoader(template_name='file', csv_path=Path('tests/csv_testfiles/invalid_fieldnames.csv'),
fieldnames=['SHA1', 'fileName', 'size-in-bytes'], has_fieldnames=True)
fieldnames=['sha1', 'filename', 'size-in-bytes'], has_fieldnames=True)
try:
first = self.user_misp_connector.add_event(event)
for o in csv2.load():
@ -2101,6 +2105,7 @@ class TestComprehensive(unittest.TestCase):
self.admin_misp_connector.delete_event(second)
self.admin_misp_connector.delete_event(third)
@unittest.skip("Not very important, skip for now.")
def test_search_logs(self):
r = self.admin_misp_connector.update_user({'email': 'testusr-changed@user.local'}, self.test_usr)
r = self.admin_misp_connector.search_logs(model='User', created=date.today(), pythonify=True)
@ -2111,14 +2116,13 @@ class TestComprehensive(unittest.TestCase):
self.assertEqual(entry.action, 'edit')
self.admin_misp_connector.update_user({'email': 'testusr@user.local'}, self.test_usr)
page = 1
while True:
r = self.admin_misp_connector.search_logs(model='User', limit=1, page=page, created=date.today(), pythonify=True)
if not r:
break
page += 1
time.sleep(5)
r = self.admin_misp_connector.search_logs(model='User', limit=1, page=1, created=date.today(), pythonify=True)
if r:
last_change = r[0]
self.assertEqual(last_change['change'], 'email (testusr-changed@user.local) => (testusr@user.local)', last_change)
self.assertEqual(last_change['change'], 'email (testusr-changed@user.local) => (testusr@user.local)', last_change)
else:
raise Exception('Unable to find log entry after updating the user')
def test_db_schema(self):
diag = self.admin_misp_connector.db_schema_diagnostic()
@ -2271,10 +2275,14 @@ class TestComprehensive(unittest.TestCase):
self.assertEqual(sharing_group.releasability, 'Testing')
# Change releasability
r = self.admin_misp_connector.update_sharing_group({"releasability": "Testing updated"}, sharing_group, pythonify=True)
self.assertEqual(r.releasability, 'Testing updated')
r = self.admin_misp_connector.update_sharing_group({"releasability": "Testing updated - 2"}, sharing_group)
self.assertEqual(r['SharingGroup']['releasability'], 'Testing updated - 2')
r = self.admin_misp_connector.update_sharing_group({"releasability": "Testing updated"}, sharing_group)
self.assertEqual(r['SharingGroup']['releasability'], 'Testing updated')
r = self.admin_misp_connector.update_sharing_group({"releasability": "Testing updated - 2"}, sharing_group, pythonify=True)
self.assertEqual(r.releasability, 'Testing updated - 2')
# Change name
r.name = 'Testcases SG - new name'
r = self.admin_misp_connector.update_sharing_group(r, pythonify=True)
self.assertEqual(r.name, 'Testcases SG - new name')
# Test `sharing_group_exists` method
self.assertTrue(self.admin_misp_connector.sharing_group_exists(sharing_group))
@ -2293,7 +2301,7 @@ class TestComprehensive(unittest.TestCase):
# Get list
sharing_groups = self.admin_misp_connector.sharing_groups(pythonify=True)
self.assertTrue(isinstance(sharing_groups, list))
self.assertEqual(sharing_groups[0].name, 'Testcases SG')
self.assertEqual(sharing_groups[0].name, 'Testcases SG - new name')
# Use the SG
@ -2307,7 +2315,7 @@ class TestComprehensive(unittest.TestCase):
try:
first = self.user_misp_connector.add_event(first)
first = self.admin_misp_connector.change_sharing_group_on_entity(first, sharing_group.id, pythonify=True)
self.assertEqual(first.SharingGroup['name'], 'Testcases SG')
self.assertEqual(first.SharingGroup['name'], 'Testcases SG - new name')
first_object = self.admin_misp_connector.change_sharing_group_on_entity(first.objects[0], sharing_group.id, pythonify=True)
self.assertEqual(first_object.sharing_group_id, sharing_group.id)