mirror of https://github.com/MISP/PyMISP
Merge remote-tracking branch 'upstream/master'
commit
545d5cc1b7
|
@ -2,6 +2,103 @@ Changelog
|
||||||
=========
|
=========
|
||||||
|
|
||||||
|
|
||||||
|
v2.4.99 (2018-12-06)
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
New
|
||||||
|
~~~
|
||||||
|
- Auto generate doc for PyMISPExpanded. [Raphaël Vinot]
|
||||||
|
- Search_index in ExpandedPyMISP, cleanup, update jupyter. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
- Add log search. [Raphaël Vinot]
|
||||||
|
- Add test for pushing an event to ZMQ. [Raphaël Vinot]
|
||||||
|
- Change_distribution method. [Raphaël Vinot]
|
||||||
|
- Add test cases for sightings, cleanup. [Raphaël Vinot]
|
||||||
|
- [example] Added sighting rest search example. [Sami Mokaddem]
|
||||||
|
- [sighting] Added support of sighting REST API. [Sami Mokaddem]
|
||||||
|
- Allow to pass csv to return_format in search. [Raphaël Vinot]
|
||||||
|
- Page/limit in search. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Changes
|
||||||
|
~~~~~~~
|
||||||
|
- Bump Changelog. [Raphaël Vinot]
|
||||||
|
- Bump version. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects & describeTypes. [Raphaël Vinot]
|
||||||
|
- Bump Changelog. [Raphaël Vinot]
|
||||||
|
- Version bump. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Add test cases for default distribution levels. [Raphaël Vinot]
|
||||||
|
- Include proposals in attributes search. [Dawid Czarnecki]
|
||||||
|
|
||||||
|
Add includeProposals param to the search method
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Update readme to document testing. [Raphaël Vinot]
|
||||||
|
- Fixes & update Jupyter. [Raphaël Vinot]
|
||||||
|
- [tuto] Update search. [Raphaël Vinot]
|
||||||
|
- Add a script to load the API key from the file system (training VM)
|
||||||
|
[Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Add print in testlive to debug travis. [Raphaël Vinot]
|
||||||
|
- Bump objects. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Auto generate doc for PyMISPExpanded. [Raphaël Vinot]
|
||||||
|
- Test failing on travis... [Raphaël Vinot]
|
||||||
|
- Properly handle errors on event creation/update. [Raphaël Vinot]
|
||||||
|
- Test case. [Raphaël Vinot]
|
||||||
|
- Do not run the zmq test on travis. [Raphaël Vinot]
|
||||||
|
- Type of quick_filter. [Raphaël Vinot]
|
||||||
|
- Quick_filter was broken. [Raphaël Vinot]
|
||||||
|
- Properly initialize the config when jupyter runs on the VM. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
- Travis run. [Raphaël Vinot]
|
||||||
|
- Readme update + python3 + pep8. [Christophe Vandeplas]
|
||||||
|
|
||||||
|
align python path to readme specifying python3
|
||||||
|
- Feed-generator gitignore. [Christophe Vandeplas]
|
||||||
|
- Test cases. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #310 from DragonDev1906/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Added get_object & get_attribute (by ID)
|
||||||
|
- Dded get_object & get_attribute. [DragonDev1906]
|
||||||
|
- Merge pull request #307 from garanews/patch-1. [Raphaël Vinot]
|
||||||
|
|
||||||
|
fix for last pymisp version
|
||||||
|
- Fix for last pymisp version. [garanews]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #305 from dawid-
|
||||||
|
czarnecki/feature/include_proposals. [Raphaël Vinot]
|
||||||
|
|
||||||
|
chg: Include proposals in attributes search
|
||||||
|
- Merge pull request #301 from deralexxx/patch-7. [Raphaël Vinot]
|
||||||
|
|
||||||
|
mention virtualenv
|
||||||
|
- Mention virtualenv. [Alexander J]
|
||||||
|
|
||||||
|
mide make sense for people who want to use it with virtualenv
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Be more precise with the supported time indicators. [Sascha
|
||||||
|
Rommelfangen]
|
||||||
|
- Fixed documentation bug. [Sascha Rommelfangen]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #295 from 3c7/fix/search_index_date. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
|
||||||
|
Fixes date parameters for search_index() function
|
||||||
|
- Fixes date parameters for search_index() function. [Nils Kuhnert]
|
||||||
|
- Merge branch 'sightingAPI' [Raphaël Vinot]
|
||||||
|
- Merge branch 'master' into sightingAPI. [Raphaël Vinot]
|
||||||
|
- Merge pull request #285 from juju4/devel. [Raphaël Vinot]
|
||||||
|
|
||||||
|
align examples on custom usage of misp_verifycert
|
||||||
|
- Align examples on custom usage of misp_verifycert. [juju4]
|
||||||
|
|
||||||
|
|
||||||
v2.4.96 (2018-10-12)
|
v2.4.96 (2018-10-12)
|
||||||
--------------------
|
--------------------
|
||||||
|
|
||||||
|
@ -22,6 +119,7 @@ New
|
||||||
|
|
||||||
Changes
|
Changes
|
||||||
~~~~~~~
|
~~~~~~~
|
||||||
|
- Bump changelog. [Raphaël Vinot]
|
||||||
- Bump version. [Raphaël Vinot]
|
- Bump version. [Raphaël Vinot]
|
||||||
- Bump misp-objects. [Raphaël Vinot]
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
- Allow to pass a json string to direct_call. [Raphaël Vinot]
|
- Allow to pass a json string to direct_call. [Raphaël Vinot]
|
||||||
|
@ -32,6 +130,7 @@ Changes
|
||||||
|
|
||||||
Fix
|
Fix
|
||||||
~~~
|
~~~
|
||||||
|
- Test cases sample files. [Raphaël Vinot]
|
||||||
- Prevent checking length on a integer. [Sami Mokaddem]
|
- Prevent checking length on a integer. [Sami Mokaddem]
|
||||||
- Direct call & add example. [Raphaël Vinot]
|
- Direct call & add example. [Raphaël Vinot]
|
||||||
- Disable test for travis, take 2. [Raphaël Vinot]
|
- Disable test for travis, take 2. [Raphaël Vinot]
|
||||||
|
|
|
@ -14,6 +14,12 @@ PyMISP
|
||||||
.. autoclass:: PyMISP
|
.. autoclass:: PyMISP
|
||||||
:members:
|
:members:
|
||||||
|
|
||||||
|
PyMISPExpanded (Python 3.6+ only)
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
|
.. autoclass:: PyMISPExpanded
|
||||||
|
:members:
|
||||||
|
|
||||||
MISPAbstract
|
MISPAbstract
|
||||||
------------
|
------------
|
||||||
|
|
||||||
|
|
|
@ -20,10 +20,17 @@ if __name__ == '__main__':
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)
|
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)
|
||||||
|
template = pymisp.get_object_templates_list()
|
||||||
|
if 'response' in template.keys():
|
||||||
|
template = template['response']
|
||||||
try:
|
try:
|
||||||
template_id = [x['ObjectTemplate']['id'] for x in pymisp.get_object_templates_list() if x['ObjectTemplate']['name'] == args.type][0]
|
template_ids = [x['ObjectTemplate']['id'] for x in template if x['ObjectTemplate']['name'] == args.type]
|
||||||
|
if len(template_ids) > 0:
|
||||||
|
template_id = template_ids[0]
|
||||||
|
else:
|
||||||
|
raise IndexError
|
||||||
except IndexError:
|
except IndexError:
|
||||||
valid_types = ", ".join([x['ObjectTemplate']['name'] for x in pymisp.get_object_templates_list()])
|
valid_types = ", ".join([x['ObjectTemplate']['name'] for x in template])
|
||||||
print ("Template for type %s not found! Valid types are: %s" % (args.type, valid_types))
|
print ("Template for type %s not found! Valid types are: %s" % (args.type, valid_types))
|
||||||
exit()
|
exit()
|
||||||
|
|
||||||
|
|
|
@ -1,8 +1,12 @@
|
||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
from pymisp import PyMISP
|
from pymisp import ExpandedPyMISP
|
||||||
from keys import url, key
|
try:
|
||||||
|
from keys import url, key
|
||||||
|
except ImportError:
|
||||||
|
url = 'http://localhost:8080'
|
||||||
|
key = '8h0gHbhS0fv6JUOlTED0AznLXFbf83TYtQrCycqb'
|
||||||
import argparse
|
import argparse
|
||||||
import tools
|
import tools
|
||||||
|
|
||||||
|
@ -13,7 +17,7 @@ if __name__ == '__main__':
|
||||||
parser.add_argument("-a", "--attribute", type=int, help="Number of attributes per event (default 3000)")
|
parser.add_argument("-a", "--attribute", type=int, help="Number of attributes per event (default 3000)")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
misp = PyMISP(url, key, True, 'json')
|
misp = ExpandedPyMISP(url, key, True)
|
||||||
|
|
||||||
if args.limit is None:
|
if args.limit is None:
|
||||||
args.limit = 1
|
args.limit = 1
|
||||||
|
|
|
@ -4,6 +4,7 @@
|
||||||
import random
|
import random
|
||||||
from random import randint
|
from random import randint
|
||||||
import string
|
import string
|
||||||
|
from pymisp import MISPEvent
|
||||||
|
|
||||||
|
|
||||||
def randomStringGenerator(size, chars=string.ascii_lowercase + string.digits):
|
def randomStringGenerator(size, chars=string.ascii_lowercase + string.digits):
|
||||||
|
@ -63,12 +64,16 @@ def create_dummy_event(misp):
|
||||||
|
|
||||||
|
|
||||||
def create_massive_dummy_events(misp, nbattribute):
|
def create_massive_dummy_events(misp, nbattribute):
|
||||||
event = misp.new_event(0, 4, 0, 'massive dummy event')
|
event = MISPEvent()
|
||||||
eventid = event['Event']['id']
|
event.info = 'massive dummy event'
|
||||||
|
event = misp.add_event(event)
|
||||||
|
print(event)
|
||||||
|
eventid = event.id
|
||||||
|
distribution = '0'
|
||||||
functions = [floodtxt, floodip, flooddomain, flooddomainip, floodemail, floodattachment]
|
functions = [floodtxt, floodip, flooddomain, flooddomainip, floodemail, floodattachment]
|
||||||
for i in range(nbattribute):
|
for i in range(nbattribute):
|
||||||
choice = randint(0, 5)
|
choice = randint(0, 5)
|
||||||
if choice == 5:
|
if choice == 5:
|
||||||
floodattachment(misp, eventid, event['Event']['distribution'], False, 'Payload delivery', '', event['Event']['info'], event['Event']['analysis'], event['Event']['threat_level_id'])
|
floodattachment(misp, eventid, distribution, False, 'Payload delivery', '', event.info, event.analysis, event.threat_level_id)
|
||||||
else:
|
else:
|
||||||
functions[choice](misp, event)
|
functions[choice](misp, event)
|
||||||
|
|
|
@ -1,9 +1,9 @@
|
||||||
#!/usr/bin/env python
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
|
||||||
from pymisp import PyMISP
|
from pymisp import ExpandedPyMISP
|
||||||
from keys import misp_url, misp_key, misp_verifycert
|
from keys import misp_url, misp_key, misp_verifycert
|
||||||
|
|
||||||
|
|
||||||
|
@ -14,12 +14,20 @@ if __name__ == '__main__':
|
||||||
parser.add_argument("-o", "--object_attribute", nargs='+', help="Object attribute column names")
|
parser.add_argument("-o", "--object_attribute", nargs='+', help="Object attribute column names")
|
||||||
parser.add_argument("-t", "--misp_types", nargs='+', help="MISP types to fetch (ip-src, hostname, ...)")
|
parser.add_argument("-t", "--misp_types", nargs='+', help="MISP types to fetch (ip-src, hostname, ...)")
|
||||||
parser.add_argument("-c", "--context", action='store_true', help="Add event level context (tags...)")
|
parser.add_argument("-c", "--context", action='store_true', help="Add event level context (tags...)")
|
||||||
parser.add_argument("-i", "--ignore", action='store_true', help="Returns the attributes even if the event isn't published, or the attribute doesn't have the to_ids flag")
|
|
||||||
parser.add_argument("-f", "--outfile", help="Output file to write the CSV.")
|
parser.add_argument("-f", "--outfile", help="Output file to write the CSV.")
|
||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
pymisp = PyMISP(misp_url, misp_key, misp_verifycert, debug=True)
|
pymisp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert, debug=True)
|
||||||
response = pymisp.get_csv(args.event_id, args.attribute, args.object_attribute, args.misp_types, args.context, args.ignore)
|
attr = []
|
||||||
|
if args.attribute:
|
||||||
|
attr += args.attribute
|
||||||
|
if args.object_attribute:
|
||||||
|
attr += args.object_attribute
|
||||||
|
if not attr:
|
||||||
|
attr = None
|
||||||
|
print(args.context)
|
||||||
|
response = pymisp.search(return_format='csv', eventid=args.event_id, requested_attributes=attr,
|
||||||
|
type_attribute=args.misp_types, include_context=args.context)
|
||||||
|
|
||||||
if args.outfile:
|
if args.outfile:
|
||||||
with open(args.outfile, 'w') as f:
|
with open(args.outfile, 'w') as f:
|
||||||
|
|
|
@ -1,4 +1,4 @@
|
||||||
__version__ = '2.4.96'
|
__version__ = '2.4.99'
|
||||||
import logging
|
import logging
|
||||||
import functools
|
import functools
|
||||||
import warnings
|
import warnings
|
||||||
|
@ -32,7 +32,7 @@ def deprecated(func):
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from .exceptions import PyMISPError, NewEventError, NewAttributeError, MissingDependency, NoURL, NoKey, InvalidMISPObject, UnknownMISPObjectTemplate, PyMISPInvalidFormat, MISPServerError, PyMISPNotImplementedYet, PyMISPUnexpectedResponse # noqa
|
from .exceptions import PyMISPError, NewEventError, NewAttributeError, MissingDependency, NoURL, NoKey, InvalidMISPObject, UnknownMISPObjectTemplate, PyMISPInvalidFormat, MISPServerError, PyMISPNotImplementedYet, PyMISPUnexpectedResponse, PyMISPEmptyResponse # noqa
|
||||||
from .api import PyMISP # noqa
|
from .api import PyMISP # noqa
|
||||||
from .abstract import AbstractMISP, MISPEncode, MISPTag, Distribution, ThreatLevel, Analysis # noqa
|
from .abstract import AbstractMISP, MISPEncode, MISPTag, Distribution, ThreatLevel, Analysis # noqa
|
||||||
from .mispevent import MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, MISPObject, MISPUser, MISPOrganisation, MISPSighting, MISPLog # noqa
|
from .mispevent import MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, MISPObject, MISPUser, MISPOrganisation, MISPSighting, MISPLog # noqa
|
||||||
|
|
|
@ -15,7 +15,7 @@ from io import BytesIO, open
|
||||||
import zipfile
|
import zipfile
|
||||||
|
|
||||||
from . import __version__, deprecated
|
from . import __version__, deprecated
|
||||||
from .exceptions import PyMISPError, SearchError, NoURL, NoKey
|
from .exceptions import PyMISPError, SearchError, NoURL, NoKey, PyMISPEmptyResponse
|
||||||
from .mispevent import MISPEvent, MISPAttribute, MISPUser, MISPOrganisation, MISPSighting, MISPFeed, MISPObject
|
from .mispevent import MISPEvent, MISPAttribute, MISPUser, MISPOrganisation, MISPSighting, MISPFeed, MISPObject
|
||||||
from .abstract import AbstractMISP, MISPEncode
|
from .abstract import AbstractMISP, MISPEncode
|
||||||
|
|
||||||
|
@ -157,6 +157,11 @@ class PyMISP(object):
|
||||||
if data is None:
|
if data is None:
|
||||||
req = requests.Request(request_type, url)
|
req = requests.Request(request_type, url)
|
||||||
else:
|
else:
|
||||||
|
if not isinstance(data, str):
|
||||||
|
if isinstance(data, dict):
|
||||||
|
# Remove None values.
|
||||||
|
data = {k: v for k, v in data.items() if v is not None}
|
||||||
|
data = json.dumps(data)
|
||||||
req = requests.Request(request_type, url, data=data)
|
req = requests.Request(request_type, url, data=data)
|
||||||
if self.asynch and background_callback is not None:
|
if self.asynch and background_callback is not None:
|
||||||
local_session = FuturesSession
|
local_session = FuturesSession
|
||||||
|
@ -227,6 +232,8 @@ class PyMISP(object):
|
||||||
json_response = response.json()
|
json_response = response.json()
|
||||||
except ValueError:
|
except ValueError:
|
||||||
# If the server didn't return a JSON blob, we've a problem.
|
# If the server didn't return a JSON blob, we've a problem.
|
||||||
|
if not len(response.text):
|
||||||
|
raise PyMISPEmptyResponse('The server returned an empty response. \n{}\n{}\n'.format(response.request.headers, response.request.body))
|
||||||
raise PyMISPError(everything_broken.format(response.request.headers, response.request.body, response.text))
|
raise PyMISPError(everything_broken.format(response.request.headers, response.request.body, response.text))
|
||||||
|
|
||||||
errors = []
|
errors = []
|
||||||
|
@ -336,6 +343,24 @@ class PyMISP(object):
|
||||||
response = self._prepare_request('GET', url)
|
response = self._prepare_request('GET', url)
|
||||||
return self._check_response(response)
|
return self._check_response(response)
|
||||||
|
|
||||||
|
def get_object(self, obj_id):
|
||||||
|
"""Get an object
|
||||||
|
|
||||||
|
:param obj_id: Object id to get
|
||||||
|
"""
|
||||||
|
url = urljoin(self.root_url, 'objects/view/{}'.format(obj_id))
|
||||||
|
response = self._prepare_request('GET', url)
|
||||||
|
return self._check_response(response)
|
||||||
|
|
||||||
|
def get_attribute(self, att_id):
|
||||||
|
"""Get an attribute
|
||||||
|
|
||||||
|
:param att_id: Attribute id to get
|
||||||
|
"""
|
||||||
|
url = urljoin(self.root_url, 'attributes/view/{}'.format(att_id))
|
||||||
|
response = self._prepare_request('GET', url)
|
||||||
|
return self._check_response(response)
|
||||||
|
|
||||||
def add_event(self, event):
|
def add_event(self, event):
|
||||||
"""Add a new event
|
"""Add a new event
|
||||||
|
|
||||||
|
@ -1212,12 +1237,14 @@ class PyMISP(object):
|
||||||
query['event_timestamp'] = kwargs.pop('event_timestamp', None)
|
query['event_timestamp'] = kwargs.pop('event_timestamp', None)
|
||||||
query['includeProposals'] = kwargs.pop('includeProposals', None)
|
query['includeProposals'] = kwargs.pop('includeProposals', None)
|
||||||
|
|
||||||
|
if kwargs:
|
||||||
|
logger.info('Some unknown parameters are in kwargs. appending as-is: {}'.format(', '.join(kwargs.keys())))
|
||||||
|
# Add all other keys as-is.
|
||||||
|
query.update({k: v for k, v in kwargs.items()})
|
||||||
|
|
||||||
# Cleanup
|
# Cleanup
|
||||||
query = {k: v for k, v in query.items() if v is not None}
|
query = {k: v for k, v in query.items() if v is not None}
|
||||||
|
|
||||||
if kwargs:
|
|
||||||
raise SearchError('Unused parameter: {}'.format(', '.join(kwargs.keys())))
|
|
||||||
|
|
||||||
# Create a session, make it async if and only if we have a callback
|
# Create a session, make it async if and only if we have a callback
|
||||||
return self.__query('restSearch/download', query, controller, async_callback)
|
return self.__query('restSearch/download', query, controller, async_callback)
|
||||||
|
|
||||||
|
@ -1410,6 +1437,15 @@ class PyMISP(object):
|
||||||
response = self._prepare_request('GET', url)
|
response = self._prepare_request('GET', url)
|
||||||
return self._check_response(response)
|
return self._check_response(response)
|
||||||
|
|
||||||
|
def get_users_statistics(self, context='data'):
|
||||||
|
"""Get users statistics from the MISP instance"""
|
||||||
|
availables_contexts = ['data', 'orgs', 'users', 'tags', 'attributehistogram', 'sightings', 'attackMatrix']
|
||||||
|
if context not in availables_contexts:
|
||||||
|
context = 'data'
|
||||||
|
url = urljoin(self.root_url, 'users/statistics/{}.json'.format(context))
|
||||||
|
response = self._prepare_request('GET', url)
|
||||||
|
return self._check_response(response)
|
||||||
|
|
||||||
# ############## Sightings ##################
|
# ############## Sightings ##################
|
||||||
|
|
||||||
def sighting_per_id(self, attribute_id):
|
def sighting_per_id(self, attribute_id):
|
||||||
|
@ -2270,11 +2306,9 @@ class PyMISP(object):
|
||||||
|
|
||||||
def get_object_template_id(self, object_uuid):
|
def get_object_template_id(self, object_uuid):
|
||||||
"""Gets the template ID corresponting the UUID passed as parameter"""
|
"""Gets the template ID corresponting the UUID passed as parameter"""
|
||||||
templates = self.get_object_templates_list()
|
url = urljoin(self.root_url, 'objectTemplates/view/{}'.format(object_uuid))
|
||||||
for t in templates:
|
response = self._prepare_request('GET', url)
|
||||||
if t['ObjectTemplate']['uuid'] == object_uuid:
|
return self._check_response(response)
|
||||||
return t['ObjectTemplate']['id']
|
|
||||||
raise Exception('Unable to find template uuid {} on the MISP instance'.format(object_uuid))
|
|
||||||
|
|
||||||
def update_object_templates(self):
|
def update_object_templates(self):
|
||||||
url = urljoin(self.root_url, '/objectTemplates/update')
|
url = urljoin(self.root_url, '/objectTemplates/update')
|
||||||
|
|
|
@ -6,7 +6,6 @@ from .api import PyMISP, everything_broken
|
||||||
from .mispevent import MISPEvent, MISPAttribute, MISPSighting, MISPLog
|
from .mispevent import MISPEvent, MISPAttribute, MISPSighting, MISPLog
|
||||||
from typing import TypeVar, Optional, Tuple, List, Dict
|
from typing import TypeVar, Optional, Tuple, List, Dict
|
||||||
from datetime import date, datetime
|
from datetime import date, datetime
|
||||||
import json
|
|
||||||
import csv
|
import csv
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
@ -70,7 +69,7 @@ class ExpandedPyMISP(PyMISP):
|
||||||
# The server returns a json message with the error details
|
# The server returns a json message with the error details
|
||||||
error_message = response.json()
|
error_message = response.json()
|
||||||
logger.error(f'Something went wrong ({response.status_code}): {error_message}')
|
logger.error(f'Something went wrong ({response.status_code}): {error_message}')
|
||||||
return {'errors': [(response.status_code, error_message)]}
|
return {'errors': (response.status_code, error_message)}
|
||||||
|
|
||||||
# At this point, we had no error.
|
# At this point, we had no error.
|
||||||
|
|
||||||
|
@ -80,11 +79,15 @@ class ExpandedPyMISP(PyMISP):
|
||||||
logger.debug(response)
|
logger.debug(response)
|
||||||
if isinstance(response, dict) and response.get('response') is not None:
|
if isinstance(response, dict) and response.get('response') is not None:
|
||||||
# Cleanup.
|
# Cleanup.
|
||||||
return response.get('response')
|
response = response['response']
|
||||||
return response
|
return response
|
||||||
except Exception:
|
except Exception:
|
||||||
if logger.isEnabledFor(logging.DEBUG):
|
if logger.isEnabledFor(logging.DEBUG):
|
||||||
logger.debug(response.text)
|
logger.debug(response.text)
|
||||||
|
if not len(response.content):
|
||||||
|
# Empty response
|
||||||
|
logger.error('Got an empty response.')
|
||||||
|
return {'errors': 'The response is empty.'}
|
||||||
return response.text
|
return response.text
|
||||||
|
|
||||||
def get_event(self, event_id: int):
|
def get_event(self, event_id: int):
|
||||||
|
@ -97,6 +100,8 @@ class ExpandedPyMISP(PyMISP):
|
||||||
created_event = super().add_event(event)
|
created_event = super().add_event(event)
|
||||||
if isinstance(created_event, str):
|
if isinstance(created_event, str):
|
||||||
raise NewEventError(f'Unexpected response from server: {created_event}')
|
raise NewEventError(f'Unexpected response from server: {created_event}')
|
||||||
|
elif 'errors' in created_event:
|
||||||
|
return created_event
|
||||||
e = MISPEvent()
|
e = MISPEvent()
|
||||||
e.load(created_event)
|
e.load(created_event)
|
||||||
return e
|
return e
|
||||||
|
@ -105,6 +110,8 @@ class ExpandedPyMISP(PyMISP):
|
||||||
updated_event = super().update_event(event.uuid, event)
|
updated_event = super().update_event(event.uuid, event)
|
||||||
if isinstance(updated_event, str):
|
if isinstance(updated_event, str):
|
||||||
raise UpdateEventError(f'Unexpected response from server: {updated_event}')
|
raise UpdateEventError(f'Unexpected response from server: {updated_event}')
|
||||||
|
elif 'errors' in updated_event:
|
||||||
|
return updated_event
|
||||||
e = MISPEvent()
|
e = MISPEvent()
|
||||||
e.load(updated_event)
|
e.load(updated_event)
|
||||||
return e
|
return e
|
||||||
|
@ -113,6 +120,8 @@ class ExpandedPyMISP(PyMISP):
|
||||||
updated_attribute = super().update_attribute(attribute.uuid, attribute)
|
updated_attribute = super().update_attribute(attribute.uuid, attribute)
|
||||||
if isinstance(updated_attribute, str):
|
if isinstance(updated_attribute, str):
|
||||||
raise UpdateAttributeError(f'Unexpected response from server: {updated_attribute}')
|
raise UpdateAttributeError(f'Unexpected response from server: {updated_attribute}')
|
||||||
|
elif 'errors' in updated_attribute:
|
||||||
|
return updated_attribute
|
||||||
a = MISPAttribute()
|
a = MISPAttribute()
|
||||||
a.from_dict(**updated_attribute)
|
a.from_dict(**updated_attribute)
|
||||||
return a
|
return a
|
||||||
|
@ -172,10 +181,7 @@ class ExpandedPyMISP(PyMISP):
|
||||||
query['includeEvent'] = include_event_meta
|
query['includeEvent'] = include_event_meta
|
||||||
|
|
||||||
url = urljoin(self.root_url, url_path)
|
url = urljoin(self.root_url, url_path)
|
||||||
# Remove None values.
|
response = self._prepare_request('POST', url, data=query)
|
||||||
# TODO: put that in self._prepare_request
|
|
||||||
query = {k: v for k, v in query.items() if v is not None}
|
|
||||||
response = self._prepare_request('POST', url, data=json.dumps(query))
|
|
||||||
normalized_response = self._check_response(response)
|
normalized_response = self._check_response(response)
|
||||||
if isinstance(normalized_response, str) or (isinstance(normalized_response, dict) and
|
if isinstance(normalized_response, str) or (isinstance(normalized_response, dict) and
|
||||||
normalized_response.get('errors')):
|
normalized_response.get('errors')):
|
||||||
|
@ -348,10 +354,7 @@ class ExpandedPyMISP(PyMISP):
|
||||||
query['includeContext'] = include_context
|
query['includeContext'] = include_context
|
||||||
query['headerless'] = headerless
|
query['headerless'] = headerless
|
||||||
url = urljoin(self.root_url, f'{controller}/restSearch')
|
url = urljoin(self.root_url, f'{controller}/restSearch')
|
||||||
# Remove None values.
|
response = self._prepare_request('POST', url, data=query)
|
||||||
# TODO: put that in self._prepare_request
|
|
||||||
query = {k: v for k, v in query.items() if v is not None}
|
|
||||||
response = self._prepare_request('POST', url, data=json.dumps(query))
|
|
||||||
normalized_response = self._check_response(response)
|
normalized_response = self._check_response(response)
|
||||||
if return_format == 'csv' and pythonify and not headerless:
|
if return_format == 'csv' and pythonify and not headerless:
|
||||||
return self._csv_to_dict(normalized_response)
|
return self._csv_to_dict(normalized_response)
|
||||||
|
@ -420,10 +423,7 @@ class ExpandedPyMISP(PyMISP):
|
||||||
query['id'] = query.pop('log_id')
|
query['id'] = query.pop('log_id')
|
||||||
|
|
||||||
url = urljoin(self.root_url, 'admin/logs/index')
|
url = urljoin(self.root_url, 'admin/logs/index')
|
||||||
# Remove None values.
|
response = self._prepare_request('POST', url, data=query)
|
||||||
# TODO: put that in self._prepare_request
|
|
||||||
query = {k: v for k, v in query.items() if v is not None}
|
|
||||||
response = self._prepare_request('POST', url, data=json.dumps(query))
|
|
||||||
normalized_response = self._check_response(response)
|
normalized_response = self._check_response(response)
|
||||||
if not pythonify:
|
if not pythonify:
|
||||||
return normalized_response
|
return normalized_response
|
||||||
|
@ -477,10 +477,7 @@ class ExpandedPyMISP(PyMISP):
|
||||||
query['timestamp'] = self.make_timestamp(timestamp)
|
query['timestamp'] = self.make_timestamp(timestamp)
|
||||||
|
|
||||||
url = urljoin(self.root_url, 'events/index')
|
url = urljoin(self.root_url, 'events/index')
|
||||||
# Remove None values.
|
response = self._prepare_request('POST', url, data=query)
|
||||||
# TODO: put that in self._prepare_request
|
|
||||||
query = {k: v for k, v in query.items() if v is not None}
|
|
||||||
response = self._prepare_request('POST', url, data=json.dumps(query))
|
|
||||||
normalized_response = self._check_response(response)
|
normalized_response = self._check_response(response)
|
||||||
|
|
||||||
if not pythonify:
|
if not pythonify:
|
||||||
|
|
File diff suppressed because it is too large
Load Diff
|
@ -1 +1 @@
|
||||||
Subproject commit 7fe77c02affc0abe14cc67fe9f14400e8b72561c
|
Subproject commit b25388c406e2f8dfcba94fe742a3ca90360315fd
|
|
@ -67,3 +67,7 @@ class PyMISPNotImplementedYet(PyMISPError):
|
||||||
|
|
||||||
class PyMISPUnexpectedResponse(PyMISPError):
|
class PyMISPUnexpectedResponse(PyMISPError):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class PyMISPEmptyResponse(PyMISPError):
|
||||||
|
pass
|
||||||
|
|
|
@ -464,13 +464,7 @@ class MISPEvent(AbstractMISP):
|
||||||
event = json_event
|
event = json_event
|
||||||
if not event:
|
if not event:
|
||||||
raise PyMISPError('Invalid event')
|
raise PyMISPError('Invalid event')
|
||||||
# Invalid event created by MISP up to 2.4.52 (attribute_count is none instead of '0')
|
self.from_dict(**event)
|
||||||
if (event.get('Event') and
|
|
||||||
'attribute_count' in event.get('Event') and
|
|
||||||
event.get('Event').get('attribute_count') is None):
|
|
||||||
event['Event']['attribute_count'] = '0'
|
|
||||||
e = event.get('Event')
|
|
||||||
self.from_dict(**e)
|
|
||||||
if validate:
|
if validate:
|
||||||
jsonschema.validate(json.loads(self.to_json()), self.__json_schema)
|
jsonschema.validate(json.loads(self.to_json()), self.__json_schema)
|
||||||
|
|
||||||
|
@ -489,17 +483,19 @@ class MISPEvent(AbstractMISP):
|
||||||
raise NewEventError('Invalid format for the date: {} - {}'.format(date, type(date)))
|
raise NewEventError('Invalid format for the date: {} - {}'.format(date, type(date)))
|
||||||
|
|
||||||
def from_dict(self, **kwargs):
|
def from_dict(self, **kwargs):
|
||||||
|
if kwargs.get('Event'):
|
||||||
|
kwargs = kwargs.get('Event')
|
||||||
# Required value
|
# Required value
|
||||||
self.info = kwargs.pop('info', None)
|
self.info = kwargs.pop('info', None)
|
||||||
if self.info is None:
|
if self.info is None:
|
||||||
raise NewAttributeError('The info field of the new event is required.')
|
raise NewEventError('The info field of the new event is required.')
|
||||||
|
|
||||||
# Default values for a valid event to send to a MISP instance
|
# Default values for a valid event to send to a MISP instance
|
||||||
self.distribution = kwargs.pop('distribution', None)
|
self.distribution = kwargs.pop('distribution', None)
|
||||||
if self.distribution is not None:
|
if self.distribution is not None:
|
||||||
self.distribution = int(self.distribution)
|
self.distribution = int(self.distribution)
|
||||||
if self.distribution not in [0, 1, 2, 3, 4]:
|
if self.distribution not in [0, 1, 2, 3, 4]:
|
||||||
raise NewAttributeError('{} is invalid, the distribution has to be in 0, 1, 2, 3, 4'.format(self.distribution))
|
raise NewEventError('{} is invalid, the distribution has to be in 0, 1, 2, 3, 4'.format(self.distribution))
|
||||||
|
|
||||||
if kwargs.get('threat_level_id') is not None:
|
if kwargs.get('threat_level_id') is not None:
|
||||||
self.threat_level_id = int(kwargs.pop('threat_level_id'))
|
self.threat_level_id = int(kwargs.pop('threat_level_id'))
|
||||||
|
@ -1005,6 +1001,8 @@ class MISPObject(AbstractMISP):
|
||||||
raise PyMISPError('All the attributes have to be of type MISPObjectReference.')
|
raise PyMISPError('All the attributes have to be of type MISPObjectReference.')
|
||||||
|
|
||||||
def from_dict(self, **kwargs):
|
def from_dict(self, **kwargs):
|
||||||
|
if kwargs.get('Object'):
|
||||||
|
kwargs = kwargs.get('Object')
|
||||||
if self._known_template:
|
if self._known_template:
|
||||||
if kwargs.get('template_uuid') and kwargs['template_uuid'] != self.template_uuid:
|
if kwargs.get('template_uuid') and kwargs['template_uuid'] != self.template_uuid:
|
||||||
if self._strict:
|
if self._strict:
|
||||||
|
@ -1017,6 +1015,12 @@ class MISPObject(AbstractMISP):
|
||||||
else:
|
else:
|
||||||
self._known_template = False
|
self._known_template = False
|
||||||
|
|
||||||
|
if 'distribution' in kwargs and kwargs['distribution'] is not None:
|
||||||
|
self.distribution = kwargs.pop('distribution')
|
||||||
|
self.distribution = int(self.distribution)
|
||||||
|
if self.distribution not in [0, 1, 2, 3, 4, 5]:
|
||||||
|
raise NewAttributeError('{} is invalid, the distribution has to be in 0, 1, 2, 3, 4, 5'.format(self.distribution))
|
||||||
|
|
||||||
if kwargs.get('timestamp'):
|
if kwargs.get('timestamp'):
|
||||||
if sys.version_info >= (3, 3):
|
if sys.version_info >= (3, 3):
|
||||||
self.timestamp = datetime.datetime.fromtimestamp(int(kwargs.pop('timestamp')), datetime.timezone.utc)
|
self.timestamp = datetime.datetime.fromtimestamp(int(kwargs.pop('timestamp')), datetime.timezone.utc)
|
||||||
|
|
|
@ -3,22 +3,24 @@
|
||||||
|
|
||||||
import unittest
|
import unittest
|
||||||
|
|
||||||
from pymisp import ExpandedPyMISP, MISPEvent, MISPOrganisation, MISPUser, Distribution, ThreatLevel, Analysis
|
from pymisp import ExpandedPyMISP, MISPEvent, MISPOrganisation, MISPUser, Distribution, ThreatLevel, Analysis, MISPObject
|
||||||
|
from pymisp.tools import make_binary_objects
|
||||||
from datetime import datetime, timedelta, date
|
from datetime import datetime, timedelta, date
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
|
|
||||||
import time
|
import time
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from keys import url, key
|
from keys import url, key
|
||||||
|
verifycert = False
|
||||||
|
travis_run = True
|
||||||
except ImportError as e:
|
except ImportError as e:
|
||||||
print(e)
|
print(e)
|
||||||
url = 'http://localhost:8080'
|
url = 'http://localhost:8080'
|
||||||
key = 'BSip0zVadeFDeolkX2g7MHx8mrlr0uE04hh6CQj0'
|
key = '8h0gHbhS0fv6JUOlTED0AznLXFbf83TYtQrCycqb'
|
||||||
|
verifycert = False
|
||||||
from uuid import uuid4
|
travis_run = False
|
||||||
|
|
||||||
travis_run = True
|
|
||||||
|
|
||||||
|
|
||||||
class TestComprehensive(unittest.TestCase):
|
class TestComprehensive(unittest.TestCase):
|
||||||
|
@ -27,7 +29,7 @@ class TestComprehensive(unittest.TestCase):
|
||||||
def setUpClass(cls):
|
def setUpClass(cls):
|
||||||
cls.maxDiff = None
|
cls.maxDiff = None
|
||||||
# Connect as admin
|
# Connect as admin
|
||||||
cls.admin_misp_connector = ExpandedPyMISP(url, key, debug=False)
|
cls.admin_misp_connector = ExpandedPyMISP(url, key, verifycert, debug=False)
|
||||||
# Creates an org
|
# Creates an org
|
||||||
org = cls.admin_misp_connector.add_organisation(name='Test Org')
|
org = cls.admin_misp_connector.add_organisation(name='Test Org')
|
||||||
cls.test_org = MISPOrganisation()
|
cls.test_org = MISPOrganisation()
|
||||||
|
@ -36,12 +38,12 @@ class TestComprehensive(unittest.TestCase):
|
||||||
usr = cls.admin_misp_connector.add_user(email='testusr@user.local', org_id=cls.test_org.id, role_id=3)
|
usr = cls.admin_misp_connector.add_user(email='testusr@user.local', org_id=cls.test_org.id, role_id=3)
|
||||||
cls.test_usr = MISPUser()
|
cls.test_usr = MISPUser()
|
||||||
cls.test_usr.from_dict(**usr)
|
cls.test_usr.from_dict(**usr)
|
||||||
cls.user_misp_connector = ExpandedPyMISP(url, cls.test_usr.authkey)
|
cls.user_misp_connector = ExpandedPyMISP(url, cls.test_usr.authkey, verifycert, debug=False)
|
||||||
# Creates a publisher
|
# Creates a publisher
|
||||||
pub = cls.admin_misp_connector.add_user(email='testpub@user.local', org_id=cls.test_org.id, role_id=4)
|
pub = cls.admin_misp_connector.add_user(email='testpub@user.local', org_id=cls.test_org.id, role_id=4)
|
||||||
cls.test_pub = MISPUser()
|
cls.test_pub = MISPUser()
|
||||||
cls.test_pub.from_dict(**pub)
|
cls.test_pub.from_dict(**pub)
|
||||||
cls.pub_misp_connector = ExpandedPyMISP(url, cls.test_pub.authkey)
|
cls.pub_misp_connector = ExpandedPyMISP(url, cls.test_pub.authkey, verifycert)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def tearDownClass(cls):
|
def tearDownClass(cls):
|
||||||
|
@ -439,6 +441,51 @@ class TestComprehensive(unittest.TestCase):
|
||||||
self.admin_misp_connector.delete_event(first.id)
|
self.admin_misp_connector.delete_event(first.id)
|
||||||
self.admin_misp_connector.delete_event(second.id)
|
self.admin_misp_connector.delete_event(second.id)
|
||||||
|
|
||||||
|
def test_default_distribution(self):
|
||||||
|
'''The default distributions on the VM are This community only for the events and Inherit from event for attr/obj)'''
|
||||||
|
if travis_run:
|
||||||
|
return
|
||||||
|
first = self.create_simple_event()
|
||||||
|
del first.distribution
|
||||||
|
o = first.add_object(name='file')
|
||||||
|
o.add_attribute('filename', value='foo.exe')
|
||||||
|
try:
|
||||||
|
# Event create
|
||||||
|
first = self.user_misp_connector.add_event(first)
|
||||||
|
self.assertEqual(first.distribution, Distribution.this_community_only.value)
|
||||||
|
self.assertEqual(first.attributes[0].distribution, Distribution.inherit.value)
|
||||||
|
self.assertEqual(first.objects[0].distribution, Distribution.inherit.value)
|
||||||
|
self.assertEqual(first.objects[0].attributes[0].distribution, Distribution.inherit.value)
|
||||||
|
# Event edit
|
||||||
|
first.add_attribute('ip-dst', '12.54.76.43')
|
||||||
|
o = first.add_object(name='file')
|
||||||
|
o.add_attribute('filename', value='foo2.exe')
|
||||||
|
first = self.user_misp_connector.update_event(first)
|
||||||
|
self.assertEqual(first.attributes[1].distribution, Distribution.inherit.value)
|
||||||
|
self.assertEqual(first.objects[1].distribution, Distribution.inherit.value)
|
||||||
|
self.assertEqual(first.objects[1].attributes[0].distribution, Distribution.inherit.value)
|
||||||
|
# Attribute create
|
||||||
|
attribute = self.user_misp_connector.add_named_attribute(first, 'comment', 'bar')
|
||||||
|
# FIXME: Add helper that returns a list of MISPAttribute
|
||||||
|
self.assertEqual(attribute[0]['Attribute']['distribution'], str(Distribution.inherit.value))
|
||||||
|
# Object - add
|
||||||
|
o = MISPObject('file')
|
||||||
|
o.add_attribute('filename', value='blah.exe')
|
||||||
|
new_obj = self.user_misp_connector.add_object(first.id, o.template_uuid, o)
|
||||||
|
# FIXME: Add helper that returns a MISPObject
|
||||||
|
self.assertEqual(new_obj['Object']['distribution'], str(Distribution.inherit.value))
|
||||||
|
self.assertEqual(new_obj['Object']['Attribute'][0]['distribution'], str(Distribution.inherit.value))
|
||||||
|
# Object - edit
|
||||||
|
clean_obj = MISPObject(**new_obj['Object'])
|
||||||
|
clean_obj.from_dict(**new_obj['Object'])
|
||||||
|
clean_obj.add_attribute('filename', value='blah.exe')
|
||||||
|
new_obj = self.user_misp_connector.edit_object(clean_obj)
|
||||||
|
for a in new_obj['Object']['Attribute']:
|
||||||
|
self.assertEqual(a['distribution'], str(Distribution.inherit.value))
|
||||||
|
finally:
|
||||||
|
# Delete event
|
||||||
|
self.admin_misp_connector.delete_event(first.id)
|
||||||
|
|
||||||
def test_simple_event(self):
|
def test_simple_event(self):
|
||||||
'''Search a bunch of parameters:
|
'''Search a bunch of parameters:
|
||||||
* Value not existing
|
* Value not existing
|
||||||
|
@ -517,7 +564,7 @@ class TestComprehensive(unittest.TestCase):
|
||||||
|
|
||||||
# quickfilter
|
# quickfilter
|
||||||
events = self.user_misp_connector.search(timestamp=timeframe,
|
events = self.user_misp_connector.search(timestamp=timeframe,
|
||||||
quickfilter='%bar%', pythonify=True)
|
quickfilter='%foo blah%', pythonify=True)
|
||||||
# FIXME: should return one event
|
# FIXME: should return one event
|
||||||
# print(events)
|
# print(events)
|
||||||
# self.assertEqual(len(events), 1)
|
# self.assertEqual(len(events), 1)
|
||||||
|
@ -709,7 +756,7 @@ class TestComprehensive(unittest.TestCase):
|
||||||
second = self.user_misp_connector.add_event(second)
|
second = self.user_misp_connector.add_event(second)
|
||||||
|
|
||||||
response = self.user_misp_connector.fast_publish(first.id, alert=False)
|
response = self.user_misp_connector.fast_publish(first.id, alert=False)
|
||||||
self.assertEqual(response['errors'][0][1]['message'], 'You do not have permission to use this functionality.')
|
self.assertEqual(response['errors'][1]['message'], 'You do not have permission to use this functionality.')
|
||||||
|
|
||||||
# Default search, attribute with to_ids == True
|
# Default search, attribute with to_ids == True
|
||||||
first.attributes[0].to_ids = True
|
first.attributes[0].to_ids = True
|
||||||
|
@ -835,6 +882,24 @@ class TestComprehensive(unittest.TestCase):
|
||||||
self.admin_misp_connector.enable_tag(tag['id'])
|
self.admin_misp_connector.enable_tag(tag['id'])
|
||||||
# FIXME: returns the tag with ID 1
|
# FIXME: returns the tag with ID 1
|
||||||
|
|
||||||
|
def test_add_event_with_attachment(self):
|
||||||
|
first = self.create_simple_event()
|
||||||
|
try:
|
||||||
|
first = self.user_misp_connector.add_event(first)
|
||||||
|
file_obj, bin_obj, sections = make_binary_objects('tests/viper-test-files/test_files/whoami.exe', standalone=False)
|
||||||
|
first.add_object(file_obj)
|
||||||
|
first.add_object(bin_obj)
|
||||||
|
for s in sections:
|
||||||
|
first.add_object(s)
|
||||||
|
self.assertEqual(len(first.objects[0].references), 1)
|
||||||
|
self.assertEqual(first.objects[0].references[0].relationship_type, 'included-in')
|
||||||
|
first = self.user_misp_connector.update_event(first)
|
||||||
|
self.assertEqual(len(first.objects[0].references), 1)
|
||||||
|
self.assertEqual(first.objects[0].references[0].relationship_type, 'included-in')
|
||||||
|
finally:
|
||||||
|
# Delete event
|
||||||
|
self.admin_misp_connector.delete_event(first.id)
|
||||||
|
|
||||||
def test_taxonomies(self):
|
def test_taxonomies(self):
|
||||||
# Make sure we're up-to-date
|
# Make sure we're up-to-date
|
||||||
self.admin_misp_connector.update_taxonomies()
|
self.admin_misp_connector.update_taxonomies()
|
||||||
|
|
Loading…
Reference in New Issue