Merge branch 'main' of github.com:misp/pymisp

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+exclude: "tests/data"
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml
+    -   id: check-added-large-files
+-   repo: https://github.com/asottile/pyupgrade
+    rev: v3.17.0
+    hooks:
+    -   id: pyupgrade
+        args: [--py38-plus]

CHANGELOG.txt

@@ -2,11 +2,178 @@ Changelog
 =========
 
 
+v2.5.0 (2024-10-04)
+-------------------
+
+Changes
+~~~~~~~
+- Bump deps. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- [tests] misp_instance_version_master now uses the 2.5 branch.
+  [iglocska]
+
+Fix
+~~~
+- Make mypy happy. [Raphaël Vinot]
+
+
+v2.4.198 (2024-09-13)
+---------------------
+
+Changes
+~~~~~~~
+- Re-Bump changelog. [Raphaël Vinot]
+- Bump changelog. [Raphaël Vinot]
+- Bump deps, version. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- Only include the changelog in the sdist package. [Raphaël Vinot]
+
+  Related #1295
+- [data] describeTypes.json updated. [Alexandre Dulaunoy]
+
+Other
+~~~~~
+- Openioc.py is not a script, but had exec bit. [Sebastian Wagner]
+
+  the file openioc can only be used as module and as part of a package,
+  has no instructions for direct execution and is therefor not a script
+  for direct execution
+
+  this removes the executable bit from the file
+
+
+v2.4.197 (2024-09-02)
+---------------------
+
+Changes
+~~~~~~~
+- Bump changelog. [Raphaël Vinot]
+- Bump deps, version, templates. [Raphaël Vinot]
+- [misp-objects] updated to the latest version. [Alexandre Dulaunoy]
+
+Fix
+~~~
+- Avoid printing huge log when a request fails. [Raphaël Vinot]
+
+  fix #1286
+
+
+v2.4.196 (2024-08-21)
+---------------------
+
+New
+~~~
+- Add pre-commit file. [Raphaël Vinot]
+
+Changes
+~~~~~~~
+- Bump changelog. [Raphaël Vinot]
+- Bump version. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+
+Fix
+~~~
+- Remove broken config. [Raphaël Vinot]
+
+
+v2.4.195 (2024-07-27)
+---------------------
+
+New
+~~~
+- Add delete role, test suite for roles. [Raphaël Vinot]
+- Test publish & search. [Raphaël Vinot]
+- Add delete role, test suite for roles. [Raphaël Vinot]
+- Test publish & search. [Raphaël Vinot]
+
+Changes
+~~~~~~~
+- Bump Changelog. [Raphaël Vinot]
+- Bump objects. [Raphaël Vinot]
+- Bump Changelog (issue with template) [Raphaël Vinot]
+- Bump changelog. [Raphaël Vinot]
+- Bump version. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- [publish tests] further debugging. [iglocska]
+- [publish test] check if the publishing actually worked as intended.
+  [iglocska]
+- [tests] speculative fix for the published search. [iglocska]
+
+  - locally it seems to work as intended, curious what is going on here
+- Bump deps. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- Bump deps. [Raphaël Vinot]
+- [publish tests] further debugging. [iglocska]
+- [publish test] check if the publishing actually worked as intended.
+  [iglocska]
+- [tests] speculative fix for the published search. [iglocska]
+
+  - locally it seems to work as intended, curious what is going on here
+- Bump deps. [Raphaël Vinot]
+
+Fix
+~~~
+- Bump objects (invalid template) [Raphaël Vinot]
+- Do not let a user pass a full dict as tagname. [Raphaël Vinot]
+- [publish tests] fixed invalid setting name for disabling background
+  processing. [iglocska]
+- [publish test] invalid path for the publishing outcome in the
+  response. [iglocska]
+- [publish test] fixed. [iglocska]
+
+  - was incorrect as it triggered a background processed publishing, which can take time
+- Do not let a user pass a full dict as tagname. [Raphaël Vinot]
+- Do not let a user pass a full dict as tagname. [Raphaël Vinot]
+- [publish tests] fixed invalid setting name for disabling background
+  processing. [iglocska]
+- [publish test] invalid path for the publishing outcome in the
+  response. [iglocska]
+- [publish test] fixed. [iglocska]
+
+  - was incorrect as it triggered a background processed publishing, which can take time
+
+Other
+~~~~~
+- Re-naming variables to make tests happy. [Tobias Mainka]
+- Added support to add or update a MISP role. [Tobias Mainka]
+- Update tests. [Raphaël Vinot]
+- Build(deps): bump certifi from 2024.6.2 to 2024.7.4. [dependabot[bot]]
+
+  Bumps [certifi](https://github.com/certifi/python-certifi) from 2024.6.2 to 2024.7.4.
+  - [Commits](https://github.com/certifi/python-certifi/compare/2024.06.02...2024.07.04)
+
+  ---
+  updated-dependencies:
+  - dependency-name: certifi
+    dependency-type: indirect
+  ...
+- MANIFEST.in does not seem to have an effect any longer. [Ulrik Haugen]
+- Include docs, examples and tests in sdist. [Ulrik Haugen]
+- Re-naming variables to make tests happy. [Tobias Mainka]
+- Added support to add or update a MISP role. [Tobias Mainka]
+- Update tests. [Raphaël Vinot]
+- Build(deps): bump certifi from 2024.6.2 to 2024.7.4. [dependabot[bot]]
+
+  Bumps [certifi](https://github.com/certifi/python-certifi) from 2024.6.2 to 2024.7.4.
+  - [Commits](https://github.com/certifi/python-certifi/compare/2024.06.02...2024.07.04)
+
+  ---
+  updated-dependencies:
+  - dependency-name: certifi
+    dependency-type: indirect
+  ...
+- Feat: Adds methods to get attribute by id/uuid. [Sura De Silva]
+
+
 v2.4.194 (2024-06-21)
 ---------------------
 
 Changes
 ~~~~~~~
+- Bump changelog. [Raphaël Vinot]
 - Bump version. [Raphaël Vinot]
 - Bump deps. [Raphaël Vinot]
 

MANIFEST.in

@@ -1,10 +0,0 @@
-graft docs
-graft examples
-graft tests
-include CHANGELOG.txt
-include LICENSE
-include pymisp/data/*.json
-include pymisp/data/misp-objects/*.json
-include pymisp/data/misp-objects/objects/*/definition.json
-include pymisp/data/misp-objects/relationships/definition.json
-include README.md

docs/source/conf.py

@@ -1,5 +1,4 @@
 #!/usr/bin/env python3
-# -*- coding: utf-8 -*-
 #
 # PyMISP documentation build configuration file, created by
 # sphinx-quickstart on Fri Aug 26 11:39:17 2016.
@@ -444,7 +443,3 @@ epub_exclude_files = ['search.html']
 # If false, no index is generated.
 #
 # epub_use_index = True
-
-
-# Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'https://docs.python.org/': None}

pymisp/api.py

@@ -40,7 +40,7 @@ if sys.platform == 'linux':
     # Enable TCP keepalive by default on every requests
     import socket
     from urllib3.connection import HTTPConnection
-    HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [  # type: ignore
+    HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [
         (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),  # enable keepalive
         (socket.SOL_TCP, socket.TCP_KEEPIDLE, 30),  # Start pinging after 30s of idle time
         (socket.SOL_TCP, socket.TCP_KEEPINTVL, 10),  # ping every 10s
@@ -315,7 +315,7 @@ class PyMISP:
     @property
     def misp_instance_version_master(self) -> dict[str, Any] | list[dict[str, Any]]:
         """Get the most recent version from github"""
-        r = requests.get('https://raw.githubusercontent.com/MISP/MISP/2.4/VERSION.json')
+        r = requests.get('https://raw.githubusercontent.com/MISP/MISP/2.5/VERSION.json')
         if r.status_code == 200:
             master_version = loads(r.content)
             return {'version': '{}.{}.{}'.format(master_version['major'], master_version['minor'], master_version['hotfix'])}
@@ -2638,7 +2638,7 @@ class PyMISP:
             for _r in self.roles(pythonify=True):
                 if not isinstance(_r, MISPRole):
                     continue
-                if _r.default_role:  # type: ignore
+                if _r.default_role:
                     role_id = get_uuid_or_id_from_abstract_misp(_r)
                     break
             else:
@@ -2696,6 +2696,40 @@ class PyMISP:
             to_return.append(nr)
         return to_return
 
+    def add_role(self, role: MISPRole, pythonify: bool = False) -> dict[str, Any] | MISPRole:
+        """Add a new role
+
+        :param role: role to add
+        :param pythonify: Returns a PyMISP Object instead of the plain json output
+        """
+        r = self._prepare_request('POST', 'admin/roles/add', data=role)
+        role_j = self._check_json_response(r)
+        if not (self.global_pythonify or pythonify) or 'errors' in role_j:
+            return role_j
+        new_misp_role = MISPRole()
+        new_misp_role.from_dict(**role_j)
+        return new_misp_role
+
+    def update_role(self, role: MISPRole, role_id: int | None = None, pythonify: bool = False) -> dict[str, Any] | MISPRole:
+        """Update a role on a MISP instance
+
+        :param role: role to update
+        :param role_id: id to update
+        :param pythonify: Returns a PyMISP Object instead of the plain json output
+        """
+        if role_id is None:
+            uid = get_uuid_or_id_from_abstract_misp(role)
+        else:
+            uid = get_uuid_or_id_from_abstract_misp(role_id)
+        url = f'admin/roles/edit/{uid}'
+        r = self._prepare_request('POST', url, data=role)
+        updated_role = self._check_json_response(r)
+        if not (self.global_pythonify or pythonify) or 'errors' in updated_role:
+            return updated_role
+        updated_misp_role = MISPRole()
+        updated_misp_role.from_dict(**updated_role)
+        return updated_misp_role
+
     def set_default_role(self, role: MISPRole | int | str | UUID) -> dict[str, Any] | list[dict[str, Any]]:
         """Set a default role for the new user accounts
 
@@ -2706,6 +2740,15 @@ class PyMISP:
         response = self._prepare_request('POST', url)
         return self._check_json_response(response)
 
+    def delete_role(self, role: MISPRole | int | str | UUID) -> dict[str, Any] | list[dict[str, Any]]:
+        """Delete a role
+
+        :param role: role to delete
+        """
+        role_id = get_uuid_or_id_from_abstract_misp(role)
+        response = self._prepare_request('POST', f'admin/roles/delete/{role_id}')
+        return self._check_json_response(response)
+
     # ## END Role ###
 
     # ## BEGIN Decaying Models ###
@@ -3942,7 +3985,10 @@ class PyMISP:
         """Check if the response from the server is not an unexpected error"""
         if response.status_code >= 500:
             headers_without_auth = {h_name: h_value for h_name, h_value in response.request.headers.items() if h_value != self.key}
-            logger.critical(everything_broken.format(headers_without_auth, response.request.body, response.text))
+            if logger.level == logging.DEBUG:
+                logger.debug(everything_broken.format(headers_without_auth, response.request.body, response.text))
+            else:
+                logger.critical(everything_broken.format(headers_without_auth, response.request.body, f'{response.text[:1000]}... (enable debug mode for more details)'))
             raise MISPServerError(f'Error code 500:\n{response.text}')
 
         if 400 <= response.status_code < 500:

pymisp/data/describeTypes.json

@@ -149,6 +149,10 @@
                 "default_category": "Network activity",
                 "to_ids": 1
             },
+            "dom-hash": {
+                "default_category": "Network activity",
+                "to_ids": 1
+            },
             "pattern-in-file": {
                 "default_category": "Payload installation",
                 "to_ids": 1
@@ -796,6 +800,7 @@
             "bro",
             "zeek",
             "community-id",
+            "dom-hash",
             "pattern-in-file",
             "pattern-in-traffic",
             "pattern-in-memory",
@@ -1293,7 +1298,8 @@
                 "favicon-mmh3",
                 "dkim",
                 "dkim-signature",
-                "ssh-fingerprint"
+                "ssh-fingerprint",
+                "dom-hash"
             ],
             "Payload type": [
                 "comment",
@@ -1377,7 +1383,8 @@
                 "other",
                 "cortex",
                 "anonymised",
-                "community-id"
+                "community-id",
+                "dom-hash"
             ],
             "Financial fraud": [
                 "btc",

pymisp/data/misp-objects

@@ -1 +1 @@
-Subproject commit e3288ef6e516624e3e335939a2b7fe4aef5ce510
+Subproject commit 3eaeaa30d1f0511740b2d9d2f65ee6a7a9103b32

pymisp/mispevent.py

@@ -2260,14 +2260,48 @@ class MISPRole(AbstractMISP):
 
     def __init__(self, **kwargs: dict[str, Any]) -> None:
         super().__init__(**kwargs)
-        self.perm_admin: int
+        self.name: str
-        self.perm_site_admin: int
+        self.perm_add: bool
+        self.perm_modify: bool
+        self.perm_modify_org: bool
+        self.perm_publish: bool
+        self.perm_delegate: bool
+        self.perm_sync: bool
+        self.perm_admin: bool
+        self.perm_audit: bool
+        self.perm_auth: bool
+        self.perm_site_admin: bool
+        self.perm_regexp_access: bool
+        self.perm_tagger: bool
+        self.perm_template: bool
+        self.perm_sharing_group: bool
+        self.perm_tag_editor: bool
+        self.perm_sighting: bool
+        self.perm_object_template: bool
+        self.default_role: bool
+        self.memory_limit: str | int
+        self.max_execution_time: str | int
+        self.restricted_to_site_admin: bool
+        self.perm_publish_zmq: bool
+        self.perm_publish_kafka: bool
+        self.perm_decaying: bool
+        self.enforce_rate_limit: bool
+        self.rate_limit_count: str | int
+        self.perm_galaxy_editor: bool
+        self.perm_warninglist: bool
+        self.perm_view_feed_correlations: bool
+        self.perm_analyst_data: bool
+        self.permission: str
+        self.permission_description: str
 
     def from_dict(self, **kwargs) -> None:  # type: ignore[no-untyped-def]
         if 'Role' in kwargs:
             kwargs = kwargs['Role']
         super().from_dict(**kwargs)
 
+    def __repr__(self) -> str:
+        return '<{self.__class__.__name__}({self.name})'.format(self=self)
+
 
 class MISPServer(AbstractMISP):
 

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pymisp"
-version = "2.4.194"
+version = "2.5.0"
 description = "Python API for MISP."
 authors = ["Raphaël Vinot <raphael.vinot@circl.lu>"]
 license = "BSD-2-Clause"
@@ -27,13 +27,16 @@ classifiers=[
 ]
 
 include = [
-    "CHANGELOG.txt",
+    {path = "CHANGELOG.txt", format = "sdist"},
     "pymisp/data/*.json",
     "pymisp/data/misp-objects/schema_objects.json",
     "pymisp/data/misp-objects/schema_relationships.json",
     "pymisp/data/misp-objects/objects/*/definition.json",
     "pymisp/data/misp-objects/relationships/definition.json",
-    "pymisp/tools/pdf_fonts/Noto_TTF/*"
+    "pymisp/tools/pdf_fonts/Noto_TTF/*",
+    "docs",
+    "examples",
+    "tests",
 ]
 
 [tool.poetry.urls]
@@ -45,22 +48,24 @@ python = "^3.8"
 requests = "^2.32.3"
 python-dateutil = "^2.9.0.post0"
 deprecated = "^1.2.14"
-extract_msg = {version = "^0.48.5", optional = true}
+extract_msg = {version = "^0.49", optional = true}
 RTFDE = {version = "^0.1.1", optional = true}
 oletools = {version = "^0.60.1", optional = true}
 python-magic = {version = "^0.4.27", optional = true}
 pydeep2 = {version = "^0.5.1", optional = true}
 lief = {version = "^0.15.0", optional = true}
 beautifulsoup4 = {version = "^4.12.3", optional = true}
-validators = {version = "^0.33.0", optional = true}
+validators = {version = "^0.34.0", optional = true}
-sphinx-autodoc-typehints = {version = "^2.2.3", optional = true, python = ">=3.9"}
+sphinx-autodoc-typehints = {version = "^2.4.4", optional = true, python = ">=3.10"}
 docutils = {version = "^0.21.1", optional = true, python = ">=3.9"}
 recommonmark = {version = "^0.7.1", optional = true, python = ">=3.9"}
-reportlab = {version = "^4.2.2", optional = true}
+reportlab = {version = "^4.2.5", optional = true}
 pyfaup = {version = "^1.2", optional = true}
-publicsuffixlist = {version = "^1.0.2.20240723", optional = true}
+publicsuffixlist = {version = "^1.0.2.20241003", optional = true}
 urllib3 = {extras = ["brotli"], version = "*", optional = true}
-Sphinx =  {version = "^7.4.7", python = ">=3.9", optional = true}
+Sphinx = [
+    {version = "^8", python = ">=3.10", optional = true}
+]
 
 [tool.poetry.extras]
 fileobjects = ['python-magic', 'pydeep2', 'lief']
@@ -74,16 +79,16 @@ brotli = ['urllib3']
 
 [tool.poetry.group.dev.dependencies]
 requests-mock = "^1.12.1"
-mypy = "^1.11.0"
+mypy = "^1.11.2"
 ipython = [
     {version = "<8.13.0", python = "<3.9"},
     {version = "^8.18.0", python = ">=3.9"},
     {version = "^8.19.0", python = ">=3.10"}
 ]
-jupyterlab = "^4.2.4"
+jupyterlab = "^4.2.5"
-types-requests = "^2.32.0.20240712"
+types-requests = "^2.32.0.20240914"
-types-python-dateutil = "^2.9.0.20240316"
+types-python-dateutil = "^2.9.0.20241003"
-types-redis = "^4.6.0.20240425"
+types-redis = "^4.6.0.20241004"
 types-Flask = "^1.1.6"
 pytest-cov = "^5.0.0"
 

tests/testlive_comprehensive.py

@@ -26,7 +26,7 @@ try:
                         MISPSharingGroup, MISPFeed, MISPServer, MISPUserSetting,
                         MISPEventReport, MISPCorrelationExclusion, MISPGalaxyCluster,
                         MISPGalaxy, MISPOrganisationBlocklist, MISPEventBlocklist,
-                        MISPNote)
+                        MISPNote, MISPRole)
     from pymisp.tools import CSVLoader, DomainIPObject, ASNObject, GenericObjectGenerator
 except ImportError:
     raise
@@ -36,7 +36,7 @@ try:
     verifycert = False
 except ImportError as e:
     print(e)
-    url = 'https://10.197.206.83'
+    url = 'https://10.197.206.84'
     key = 'OdzzuBSnH83tEjvZbf7SFejC1kC3gS11Cnj2wxLk'
     verifycert = False
 
@@ -79,6 +79,7 @@ class TestComprehensive(unittest.TestCase):
         cls.admin_misp_connector.set_server_setting('debug', 1, force=True)
         if not fast_mode:
             r = cls.admin_misp_connector.update_misp()
+            print(r)
         # Creates an org
         organisation = MISPOrganisation()
         organisation.name = 'Test Org'
@@ -2168,6 +2169,19 @@ class TestComprehensive(unittest.TestCase):
         self.admin_misp_connector.set_default_role(3)
         roles = self.admin_misp_connector.roles(pythonify=True)
         self.assertTrue(isinstance(roles, list))
+        try:
+            # Create a new role
+            new_role = MISPRole()
+            new_role.name = 'testrole'
+            new_role = self.admin_misp_connector.add_role(new_role, pythonify=True)
+            self.assertFalse(new_role.perm_sighting)
+            new_role.perm_sighting = True
+            new_role.max_execution_time = 1234
+            updated_role = self.admin_misp_connector.update_role(new_role, pythonify=True)
+            self.assertTrue(updated_role.perm_sighting)
+            self.assertEqual(updated_role.max_execution_time, '1234')
+        finally:
+            self.admin_misp_connector.delete_role(new_role)
 
     def test_describe_types(self) -> None:
         remote = self.admin_misp_connector.describe_types_remote