MISP
/
PyMISP
mirror of https://github.com/MISP/PyMISP
2
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'main' of github.com:misp/pymisp

main
Christian Studer 2024-10-17 14:03:49 +02:00
parent 185417982d f4438dbe3a
commit 7316c3e040
No known key found for this signature in database
GPG Key ID: 6BBED1B63A6D639F
12 changed files with 1033 additions and 678 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,16 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
exclude: "tests/data"
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/asottile/pyupgrade
rev: v3.17.0
hooks:
- id: pyupgrade
args: [--py38-plus]

167
CHANGELOG.txt
View File

@ -2,11 +2,178 @@ Changelog
=========
v2.5.0 (2024-10-04)
-------------------
Changes
~~~~~~~
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- [tests] misp_instance_version_master now uses the 2.5 branch.
[iglocska]
Fix
~~~
- Make mypy happy. [Raphaël Vinot]
v2.4.198 (2024-09-13)
---------------------
Changes
~~~~~~~
- Re-Bump changelog. [Raphaël Vinot]
- Bump changelog. [Raphaël Vinot]
- Bump deps, version. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Only include the changelog in the sdist package. [Raphaël Vinot]
Related #1295
- [data] describeTypes.json updated. [Alexandre Dulaunoy]
Other
~~~~~
- Openioc.py is not a script, but had exec bit. [Sebastian Wagner]
the file openioc can only be used as module and as part of a package,
has no instructions for direct execution and is therefor not a script
for direct execution
this removes the executable bit from the file
v2.4.197 (2024-09-02)
---------------------
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump deps, version, templates. [Raphaël Vinot]
- [misp-objects] updated to the latest version. [Alexandre Dulaunoy]
Fix
~~~
- Avoid printing huge log when a request fails. [Raphaël Vinot]
fix #1286
v2.4.196 (2024-08-21)
---------------------
New
~~~
- Add pre-commit file. [Raphaël Vinot]
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
Fix
~~~
- Remove broken config. [Raphaël Vinot]
v2.4.195 (2024-07-27)
---------------------
New
~~~
- Add delete role, test suite for roles. [Raphaël Vinot]
- Test publish & search. [Raphaël Vinot]
- Add delete role, test suite for roles. [Raphaël Vinot]
- Test publish & search. [Raphaël Vinot]
Changes
~~~~~~~
- Bump Changelog. [Raphaël Vinot]
- Bump objects. [Raphaël Vinot]
- Bump Changelog (issue with template) [Raphaël Vinot]
- Bump changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- [publish tests] further debugging. [iglocska]
- [publish test] check if the publishing actually worked as intended.
[iglocska]
- [tests] speculative fix for the published search. [iglocska]
- locally it seems to work as intended, curious what is going on here
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]
- [publish tests] further debugging. [iglocska]
- [publish test] check if the publishing actually worked as intended.
[iglocska]
- [tests] speculative fix for the published search. [iglocska]
- locally it seems to work as intended, curious what is going on here
- Bump deps. [Raphaël Vinot]
Fix
~~~
- Bump objects (invalid template) [Raphaël Vinot]
- Do not let a user pass a full dict as tagname. [Raphaël Vinot]
- [publish tests] fixed invalid setting name for disabling background
processing. [iglocska]
- [publish test] invalid path for the publishing outcome in the
response. [iglocska]
- [publish test] fixed. [iglocska]
- was incorrect as it triggered a background processed publishing, which can take time
- Do not let a user pass a full dict as tagname. [Raphaël Vinot]
- Do not let a user pass a full dict as tagname. [Raphaël Vinot]
- [publish tests] fixed invalid setting name for disabling background
processing. [iglocska]
- [publish test] invalid path for the publishing outcome in the
response. [iglocska]
- [publish test] fixed. [iglocska]
- was incorrect as it triggered a background processed publishing, which can take time
Other
~~~~~
- Re-naming variables to make tests happy. [Tobias Mainka]
- Added support to add or update a MISP role. [Tobias Mainka]
- Update tests. [Raphaël Vinot]
- Build(deps): bump certifi from 2024.6.2 to 2024.7.4. [dependabot[bot]]
Bumps [certifi](https://github.com/certifi/python-certifi) from 2024.6.2 to 2024.7.4.
- [Commits](https://github.com/certifi/python-certifi/compare/2024.06.02...2024.07.04)
---
updated-dependencies:
- dependency-name: certifi
dependency-type: indirect
...
- MANIFEST.in does not seem to have an effect any longer. [Ulrik Haugen]
- Include docs, examples and tests in sdist. [Ulrik Haugen]
- Re-naming variables to make tests happy. [Tobias Mainka]
- Added support to add or update a MISP role. [Tobias Mainka]
- Update tests. [Raphaël Vinot]
- Build(deps): bump certifi from 2024.6.2 to 2024.7.4. [dependabot[bot]]
Bumps [certifi](https://github.com/certifi/python-certifi) from 2024.6.2 to 2024.7.4.
- [Commits](https://github.com/certifi/python-certifi/compare/2024.06.02...2024.07.04)
---
updated-dependencies:
- dependency-name: certifi
dependency-type: indirect
...
- Feat: Adds methods to get attribute by id/uuid. [Sura De Silva]
v2.4.194 (2024-06-21)
---------------------
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump deps. [Raphaël Vinot]

10
MANIFEST.in
View File

@ -1,10 +0,0 @@
graft docs
graft examples
graft tests
include CHANGELOG.txt
include LICENSE
include pymisp/data/*.json
include pymisp/data/misp-objects/*.json
include pymisp/data/misp-objects/objects/*/definition.json
include pymisp/data/misp-objects/relationships/definition.json
include README.md

5
docs/source/conf.py
View File

@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# PyMISP documentation build configuration file, created by
# sphinx-quickstart on Fri Aug 26 11:39:17 2016.
@ -444,7 +443,3 @@ epub_exclude_files = ['search.html']
# If false, no index is generated.
#
# epub_use_index = True
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}

1357
poetry.lock generated
View File

File diff suppressed because it is too large Load Diff

54
pymisp/api.py
View File

@ -40,7 +40,7 @@ if sys.platform == 'linux':
# Enable TCP keepalive by default on every requests
import socket
from urllib3.connection import HTTPConnection
HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [ # type: ignore
HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1), # enable keepalive
(socket.SOL_TCP, socket.TCP_KEEPIDLE, 30), # Start pinging after 30s of idle time
(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10), # ping every 10s
@ -315,7 +315,7 @@ class PyMISP:
@property
def misp_instance_version_master(self) -> dict[str, Any] | list[dict[str, Any]]:
"""Get the most recent version from github"""
r = requests.get('https://raw.githubusercontent.com/MISP/MISP/2.4/VERSION.json')
r = requests.get('https://raw.githubusercontent.com/MISP/MISP/2.5/VERSION.json')
if r.status_code == 200:
master_version = loads(r.content)
return {'version': '{}.{}.{}'.format(master_version['major'], master_version['minor'], master_version['hotfix'])}
@ -2638,7 +2638,7 @@ class PyMISP:
for _r in self.roles(pythonify=True):
if not isinstance(_r, MISPRole):
continue
if _r.default_role: # type: ignore
if _r.default_role:
role_id = get_uuid_or_id_from_abstract_misp(_r)
break
else:
@ -2696,6 +2696,40 @@ class PyMISP:
to_return.append(nr)
return to_return
def add_role(self, role: MISPRole, pythonify: bool = False) -> dict[str, Any] | MISPRole:
"""Add a new role
:param role: role to add
:param pythonify: Returns a PyMISP Object instead of the plain json output
"""
r = self._prepare_request('POST', 'admin/roles/add', data=role)
role_j = self._check_json_response(r)
if not (self.global_pythonify or pythonify) or 'errors' in role_j:
return role_j
new_misp_role = MISPRole()
new_misp_role.from_dict(**role_j)
return new_misp_role
def update_role(self, role: MISPRole, role_id: int | None = None, pythonify: bool = False) -> dict[str, Any] | MISPRole:
"""Update a role on a MISP instance
:param role: role to update
:param role_id: id to update
:param pythonify: Returns a PyMISP Object instead of the plain json output
"""
if role_id is None:
uid = get_uuid_or_id_from_abstract_misp(role)
else:
uid = get_uuid_or_id_from_abstract_misp(role_id)
url = f'admin/roles/edit/{uid}'
r = self._prepare_request('POST', url, data=role)
updated_role = self._check_json_response(r)
if not (self.global_pythonify or pythonify) or 'errors' in updated_role:
return updated_role
updated_misp_role = MISPRole()
updated_misp_role.from_dict(**updated_role)
return updated_misp_role
def set_default_role(self, role: MISPRole | int | str | UUID) -> dict[str, Any] | list[dict[str, Any]]:
"""Set a default role for the new user accounts
@ -2706,6 +2740,15 @@ class PyMISP:
response = self._prepare_request('POST', url)
return self._check_json_response(response)
def delete_role(self, role: MISPRole | int | str | UUID) -> dict[str, Any] | list[dict[str, Any]]:
"""Delete a role
:param role: role to delete
"""
role_id = get_uuid_or_id_from_abstract_misp(role)
response = self._prepare_request('POST', f'admin/roles/delete/{role_id}')
return self._check_json_response(response)
# ## END Role ###
# ## BEGIN Decaying Models ###
@ -3942,7 +3985,10 @@ class PyMISP:
"""Check if the response from the server is not an unexpected error"""
if response.status_code >= 500:
headers_without_auth = {h_name: h_value for h_name, h_value in response.request.headers.items() if h_value != self.key}
logger.critical(everything_broken.format(headers_without_auth, response.request.body, response.text))
if logger.level == logging.DEBUG:
logger.debug(everything_broken.format(headers_without_auth, response.request.body, response.text))
else:
logger.critical(everything_broken.format(headers_without_auth, response.request.body, f'{response.text[:1000]}... (enable debug mode for more details)'))
raise MISPServerError(f'Error code 500:\n{response.text}')
if 400 <= response.status_code < 500:

11
pymisp/data/describeTypes.json
View File

@ -149,6 +149,10 @@
"default_category": "Network activity",
"to_ids": 1
},
"dom-hash": {
"default_category": "Network activity",
"to_ids": 1
},
"pattern-in-file": {
"default_category": "Payload installation",
"to_ids": 1
@ -796,6 +800,7 @@
"bro",
"zeek",
"community-id",
"dom-hash",
"pattern-in-file",
"pattern-in-traffic",
"pattern-in-memory",
@ -1293,7 +1298,8 @@
"favicon-mmh3",
"dkim",
"dkim-signature",
"ssh-fingerprint"
"ssh-fingerprint",
"dom-hash"
],
"Payload type": [
"comment",
@ -1377,7 +1383,8 @@
"other",
"cortex",
"anonymised",
"community-id"
"community-id",
"dom-hash"
],
"Financial fraud": [
"btc",

2
pymisp/data/misp-objects

@ -1 +1 @@
Subproject commit e3288ef6e516624e3e335939a2b7fe4aef5ce510
Subproject commit 3eaeaa30d1f0511740b2d9d2f65ee6a7a9103b32

38
pymisp/mispevent.py
View File

@ -2260,14 +2260,48 @@ class MISPRole(AbstractMISP):
def __init__(self, **kwargs: dict[str, Any]) -> None:
super().__init__(**kwargs)
self.perm_admin: int
self.perm_site_admin: int
self.name: str
self.perm_add: bool
self.perm_modify: bool
self.perm_modify_org: bool
self.perm_publish: bool
self.perm_delegate: bool
self.perm_sync: bool
self.perm_admin: bool
self.perm_audit: bool
self.perm_auth: bool
self.perm_site_admin: bool
self.perm_regexp_access: bool
self.perm_tagger: bool
self.perm_template: bool
self.perm_sharing_group: bool
self.perm_tag_editor: bool
self.perm_sighting: bool
self.perm_object_template: bool
self.default_role: bool
self.memory_limit: str | int
self.max_execution_time: str | int
self.restricted_to_site_admin: bool
self.perm_publish_zmq: bool
self.perm_publish_kafka: bool
self.perm_decaying: bool
self.enforce_rate_limit: bool
self.rate_limit_count: str | int
self.perm_galaxy_editor: bool
self.perm_warninglist: bool
self.perm_view_feed_correlations: bool
self.perm_analyst_data: bool
self.permission: str
self.permission_description: str
def from_dict(self, **kwargs) -> None: # type: ignore[no-untyped-def]
if 'Role' in kwargs:
kwargs = kwargs['Role']
super().from_dict(**kwargs)
def __repr__(self) -> str:
return '<{self.__class__.__name__}({self.name})'.format(self=self)
class MISPServer(AbstractMISP):

0
pymisp/tools/openioc.py Executable file → Normal file
View File

33
pyproject.toml
View File

@ -1,6 +1,6 @@
[tool.poetry]
name = "pymisp"
version = "2.4.194"
version = "2.5.0"
description = "Python API for MISP."
authors = ["Raphaël Vinot <raphael.vinot@circl.lu>"]
license = "BSD-2-Clause"
@ -27,13 +27,16 @@ classifiers=[
]
include = [
"CHANGELOG.txt",
{path = "CHANGELOG.txt", format = "sdist"},
"pymisp/data/*.json",
"pymisp/data/misp-objects/schema_objects.json",
"pymisp/data/misp-objects/schema_relationships.json",
"pymisp/data/misp-objects/objects/*/definition.json",
"pymisp/data/misp-objects/relationships/definition.json",
"pymisp/tools/pdf_fonts/Noto_TTF/*"
"pymisp/tools/pdf_fonts/Noto_TTF/*",
"docs",
"examples",
"tests",
]
[tool.poetry.urls]
@ -45,22 +48,24 @@ python = "^3.8"
requests = "^2.32.3"
python-dateutil = "^2.9.0.post0"
deprecated = "^1.2.14"
extract_msg = {version = "^0.48.5", optional = true}
extract_msg = {version = "^0.49", optional = true}
RTFDE = {version = "^0.1.1", optional = true}
oletools = {version = "^0.60.1", optional = true}
python-magic = {version = "^0.4.27", optional = true}
pydeep2 = {version = "^0.5.1", optional = true}
lief = {version = "^0.15.0", optional = true}
beautifulsoup4 = {version = "^4.12.3", optional = true}
validators = {version = "^0.33.0", optional = true}
sphinx-autodoc-typehints = {version = "^2.2.3", optional = true, python = ">=3.9"}
validators = {version = "^0.34.0", optional = true}
sphinx-autodoc-typehints = {version = "^2.4.4", optional = true, python = ">=3.10"}
docutils = {version = "^0.21.1", optional = true, python = ">=3.9"}
recommonmark = {version = "^0.7.1", optional = true, python = ">=3.9"}
reportlab = {version = "^4.2.2", optional = true}
reportlab = {version = "^4.2.5", optional = true}
pyfaup = {version = "^1.2", optional = true}
publicsuffixlist = {version = "^1.0.2.20240723", optional = true}
publicsuffixlist = {version = "^1.0.2.20241003", optional = true}
urllib3 = {extras = ["brotli"], version = "*", optional = true}
Sphinx = {version = "^7.4.7", python = ">=3.9", optional = true}
Sphinx = [
{version = "^8", python = ">=3.10", optional = true}
]
[tool.poetry.extras]
fileobjects = ['python-magic', 'pydeep2', 'lief']
@ -74,16 +79,16 @@ brotli = ['urllib3']
[tool.poetry.group.dev.dependencies]
requests-mock = "^1.12.1"
mypy = "^1.11.0"
mypy = "^1.11.2"
ipython = [
{version = "<8.13.0", python = "<3.9"},
{version = "^8.18.0", python = ">=3.9"},
{version = "^8.19.0", python = ">=3.10"}
]
jupyterlab = "^4.2.4"
types-requests = "^2.32.0.20240712"
types-python-dateutil = "^2.9.0.20240316"
types-redis = "^4.6.0.20240425"
jupyterlab = "^4.2.5"
types-requests = "^2.32.0.20240914"
types-python-dateutil = "^2.9.0.20241003"
types-redis = "^4.6.0.20241004"
types-Flask = "^1.1.6"
pytest-cov = "^5.0.0"

18
tests/testlive_comprehensive.py
View File

@ -26,7 +26,7 @@ try:
MISPSharingGroup, MISPFeed, MISPServer, MISPUserSetting,
MISPEventReport, MISPCorrelationExclusion, MISPGalaxyCluster,
MISPGalaxy, MISPOrganisationBlocklist, MISPEventBlocklist,
MISPNote)
MISPNote, MISPRole)
from pymisp.tools import CSVLoader, DomainIPObject, ASNObject, GenericObjectGenerator
except ImportError:
raise
@ -36,7 +36,7 @@ try:
verifycert = False
except ImportError as e:
print(e)
url = 'https://10.197.206.83'
url = 'https://10.197.206.84'
key = 'OdzzuBSnH83tEjvZbf7SFejC1kC3gS11Cnj2wxLk'
verifycert = False
@ -79,6 +79,7 @@ class TestComprehensive(unittest.TestCase):
cls.admin_misp_connector.set_server_setting('debug', 1, force=True)
if not fast_mode:
r = cls.admin_misp_connector.update_misp()
print(r)
# Creates an org
organisation = MISPOrganisation()
organisation.name = 'Test Org'
@ -2168,6 +2169,19 @@ class TestComprehensive(unittest.TestCase):
self.admin_misp_connector.set_default_role(3)
roles = self.admin_misp_connector.roles(pythonify=True)
self.assertTrue(isinstance(roles, list))
try:
# Create a new role
new_role = MISPRole()
new_role.name = 'testrole'
new_role = self.admin_misp_connector.add_role(new_role, pythonify=True)
self.assertFalse(new_role.perm_sighting)
new_role.perm_sighting = True
new_role.max_execution_time = 1234
updated_role = self.admin_misp_connector.update_role(new_role, pythonify=True)
self.assertTrue(updated_role.perm_sighting)
self.assertEqual(updated_role.max_execution_time, '1234')
finally:
self.admin_misp_connector.delete_role(new_role)
def test_describe_types(self) -> None:
remote = self.admin_misp_connector.describe_types_remote