MISP
/
PyMISP
mirror of https://github.com/MISP/PyMISP
2
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'master' of github.com:SteveClement/PyMISP

pull/312/head
Steve Clement 2018-12-10 12:55:33 +01:00
parent 54fa1771a7 91fe83a475
commit 211352c74c
10 changed files with 347 additions and 159 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

99
CHANGELOG.txt
View File

@ -2,6 +2,103 @@ Changelog
=========
v2.4.99 (2018-12-06)
--------------------
New
~~~
- Auto generate doc for PyMISPExpanded. [Raphaël Vinot]
- Search_index in ExpandedPyMISP, cleanup, update jupyter. [Raphaël
Vinot]
- Add log search. [Raphaël Vinot]
- Add test for pushing an event to ZMQ. [Raphaël Vinot]
- Change_distribution method. [Raphaël Vinot]
- Add test cases for sightings, cleanup. [Raphaël Vinot]
- [example] Added sighting rest search example. [Sami Mokaddem]
- [sighting] Added support of sighting REST API. [Sami Mokaddem]
- Allow to pass csv to return_format in search. [Raphaël Vinot]
- Page/limit in search. [Raphaël Vinot]
Changes
~~~~~~~
- Bump Changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump misp-objects & describeTypes. [Raphaël Vinot]
- Bump Changelog. [Raphaël Vinot]
- Version bump. [Raphaël Vinot]
- Bump misp-objects. [Raphaël Vinot]
- Add test cases for default distribution levels. [Raphaël Vinot]
- Include proposals in attributes search. [Dawid Czarnecki]
Add includeProposals param to the search method
- Bump misp-objects. [Raphaël Vinot]
- Update readme to document testing. [Raphaël Vinot]
- Fixes & update Jupyter. [Raphaël Vinot]
- [tuto] Update search. [Raphaël Vinot]
- Add a script to load the API key from the file system (training VM)
[Raphaël Vinot]
- Bump misp-objects. [Raphaël Vinot]
- Add print in testlive to debug travis. [Raphaël Vinot]
- Bump objects. [Raphaël Vinot]
Fix
~~~
- Auto generate doc for PyMISPExpanded. [Raphaël Vinot]
- Test failing on travis... [Raphaël Vinot]
- Properly handle errors on event creation/update. [Raphaël Vinot]
- Test case. [Raphaël Vinot]
- Do not run the zmq test on travis. [Raphaël Vinot]
- Type of quick_filter. [Raphaël Vinot]
- Quick_filter was broken. [Raphaël Vinot]
- Properly initialize the config when jupyter runs on the VM. [Raphaël
Vinot]
- Travis run. [Raphaël Vinot]
- Readme update + python3 + pep8. [Christophe Vandeplas]
align python path to readme specifying python3
- Feed-generator gitignore. [Christophe Vandeplas]
- Test cases. [Raphaël Vinot]
Other
~~~~~
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
- Merge pull request #310 from DragonDev1906/master. [Raphaël Vinot]
Added get_object & get_attribute (by ID)
- Dded get_object & get_attribute. [DragonDev1906]
- Merge pull request #307 from garanews/patch-1. [Raphaël Vinot]
fix for last pymisp version
- Fix for last pymisp version. [garanews]
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
- Merge pull request #305 from dawid-
czarnecki/feature/include_proposals. [Raphaël Vinot]
chg: Include proposals in attributes search
- Merge pull request #301 from deralexxx/patch-7. [Raphaël Vinot]
mention virtualenv
- Mention virtualenv. [Alexander J]
mide make sense for people who want to use it with virtualenv
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
- Be more precise with the supported time indicators. [Sascha
Rommelfangen]
- Fixed documentation bug. [Sascha Rommelfangen]
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
- Merge pull request #295 from 3c7/fix/search_index_date. [Raphaël
Vinot]
Fixes date parameters for search_index() function
- Fixes date parameters for search_index() function. [Nils Kuhnert]
- Merge branch 'sightingAPI' [Raphaël Vinot]
- Merge branch 'master' into sightingAPI. [Raphaël Vinot]
- Merge pull request #285 from juju4/devel. [Raphaël Vinot]
align examples on custom usage of misp_verifycert
- Align examples on custom usage of misp_verifycert. [juju4]
v2.4.96 (2018-10-12)
--------------------
@ -22,6 +119,7 @@ New
Changes
~~~~~~~
- Bump changelog. [Raphaël Vinot]
- Bump version. [Raphaël Vinot]
- Bump misp-objects. [Raphaël Vinot]
- Allow to pass a json string to direct_call. [Raphaël Vinot]
@ -32,6 +130,7 @@ Changes
Fix
~~~
- Test cases sample files. [Raphaël Vinot]
- Prevent checking length on a integer. [Sami Mokaddem]
- Direct call & add example. [Raphaël Vinot]
- Disable test for travis, take 2. [Raphaël Vinot]

6
docs/source/modules.rst
View File

@ -14,6 +14,12 @@ PyMISP
.. autoclass:: PyMISP
:members:
PyMISPExpanded (Python 3.6+ only)
---------------------------------
.. autoclass:: PyMISPExpanded
:members:
MISPAbstract
------------

11
examples/add_generic_object.py
View File

@ -20,10 +20,17 @@ if __name__ == '__main__':
args = parser.parse_args()
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)
template = pymisp.get_object_templates_list()
if 'response' in template.keys():
template = template['response']
try:
template_id = [x['ObjectTemplate']['id'] for x in pymisp.get_object_templates_list() if x['ObjectTemplate']['name'] == args.type][0]
template_ids = [x['ObjectTemplate']['id'] for x in template if x['ObjectTemplate']['name'] == args.type]
if len(template_ids) > 0:
template_id = template_ids[0]
else:
raise IndexError
except IndexError:
valid_types = ", ".join([x['ObjectTemplate']['name'] for x in pymisp.get_object_templates_list()])
valid_types = ", ".join([x['ObjectTemplate']['name'] for x in template])
print ("Template for type %s not found! Valid types are: %s" % (args.type, valid_types))
exit()

2
pymisp/__init__.py
View File

@ -1,4 +1,4 @@
__version__ = '2.4.96'
__version__ = '2.4.99'
import logging
import functools
import warnings

28
pymisp/api.py
View File

@ -336,6 +336,24 @@ class PyMISP(object):
response = self._prepare_request('GET', url)
return self._check_response(response)
def get_object(self, obj_id):
"""Get an object
:param obj_id: Object id to get
"""
url = urljoin(self.root_url, 'objects/view/{}'.format(obj_id))
response = self._prepare_request('GET', url)
return self._check_response(response)
def get_attribute(self, att_id):
"""Get an attribute
:param att_id: Attribute id to get
"""
url = urljoin(self.root_url, 'attributes/view/{}'.format(att_id))
response = self._prepare_request('GET', url)
return self._check_response(response)
def add_event(self, event):
"""Add a new event
@ -1148,6 +1166,7 @@ class PyMISP(object):
:param to_ids: return only the attributes with the to_ids flag set
:param deleted: also return the deleted attributes
:param event_timestamp: the timestamp of the last modification of the event (attributes controller only)). Can be a list (from->to)
:param includeProposals: return shadow attributes if True
:param async_callback: The function to run when results are returned
"""
query = {}
@ -1203,6 +1222,7 @@ class PyMISP(object):
query['metadata'] = kwargs.pop('metadata', None)
if controller == 'attributes':
query['event_timestamp'] = kwargs.pop('event_timestamp', None)
query['includeProposals'] = kwargs.pop('includeProposals', None)
# Cleanup
query = {k: v for k, v in query.items() if v is not None}
@ -2262,11 +2282,9 @@ class PyMISP(object):
def get_object_template_id(self, object_uuid):
"""Gets the template ID corresponting the UUID passed as parameter"""
templates = self.get_object_templates_list()
for t in templates:
if t['ObjectTemplate']['uuid'] == object_uuid:
return t['ObjectTemplate']['id']
raise Exception('Unable to find template uuid {} on the MISP instance'.format(object_uuid))
url = urljoin(self.root_url, 'objectTemplates/view/{}'.format(object_uuid))
response = self._prepare_request('GET', url)
return self._check_response(response)
def update_object_templates(self):
url = urljoin(self.root_url, '/objectTemplates/update')

6
pymisp/aping.py
View File

@ -97,6 +97,8 @@ class ExpandedPyMISP(PyMISP):
created_event = super().add_event(event)
if isinstance(created_event, str):
raise NewEventError(f'Unexpected response from server: {created_event}')
elif 'errors' in created_event:
return created_event
e = MISPEvent()
e.load(created_event)
return e
@ -105,6 +107,8 @@ class ExpandedPyMISP(PyMISP):
updated_event = super().update_event(event.uuid, event)
if isinstance(updated_event, str):
raise UpdateEventError(f'Unexpected response from server: {updated_event}')
elif 'errors' in updated_event:
return updated_event
e = MISPEvent()
e.load(updated_event)
return e
@ -113,6 +117,8 @@ class ExpandedPyMISP(PyMISP):
updated_attribute = super().update_attribute(attribute.uuid, attribute)
if isinstance(updated_attribute, str):
raise UpdateAttributeError(f'Unexpected response from server: {updated_attribute}')
elif 'errors' in updated_attribute:
return updated_attribute
a = MISPAttribute()
a.from_dict(**updated_attribute)
return a

288
pymisp/data/describeTypes.json
View File

@ -1,22 +1,22 @@
{
"result": {
"categories": [
"Internal reference",
"Targeting data",
"Antivirus detection",
"Payload delivery",
"Artifacts dropped",
"Payload installation",
"Persistence mechanism",
"Network activity",
"Payload type",
"Attribution",
"External analysis",
"Financial fraud",
"Internal reference",
"Network activity",
"Other",
"Payload delivery",
"Payload installation",
"Payload type",
"Persistence mechanism",
"Person",
"Social network",
"Support Tool",
"Targeting data"
"Social network",
"Person",
"Other"
],
"category_type_mappings": {
"Antivirus detection": [
@ -186,7 +186,9 @@
"attachment",
"comment",
"text",
"x509-fingerprint-md5",
"x509-fingerprint-sha1",
"x509-fingerprint-sha256",
"other",
"hex",
"cookie",
@ -1019,158 +1021,158 @@
}
},
"types": [
"AS",
"aba-rtn",
"attachment",
"authentihash",
"bank-account-nr",
"bic",
"bin",
"boolean",
"bro",
"btc",
"campaign-id",
"campaign-name",
"cc-number",
"comment",
"cookie",
"cortex",
"counter",
"country-of-residence",
"cpe",
"date-of-birth",
"datetime",
"dns-soa-email",
"md5",
"sha1",
"sha256",
"filename",
"pdb",
"filename|md5",
"filename|sha1",
"filename|sha256",
"ip-src",
"ip-dst",
"hostname",
"domain",
"domain|ip",
"email-src",
"email-dst",
"email-subject",
"email-attachment",
"email-body",
"email-dst",
"email-dst-display-name",
"email-header",
"email-message-id",
"email-mime-boundary",
"email-reply-to",
"email-src",
"email-src-display-name",
"email-subject",
"email-thread-index",
"email-x-mailer",
"filename",
"filename|authentihash",
"filename|impfuzzy",
"filename|imphash",
"filename|md5",
"filename|pehash",
"filename|sha1",
"filename|sha224",
"filename|sha256",
"filename|sha384",
"filename|sha512",
"filename|sha512/224",
"filename|sha512/256",
"filename|ssdeep",
"filename|tlsh",
"first-name",
"float",
"frequent-flyer-number",
"gender",
"gene",
"github-organisation",
"github-repository",
"github-username",
"hex",
"hostname",
"hostname|port",
"url",
"http-method",
"iban",
"identity-card-number",
"impfuzzy",
"imphash",
"ip-dst",
"ip-dst|port",
"ip-src",
"ip-src|port",
"issue-date-of-the-visa",
"jabber-id",
"last-name",
"link",
"mac-address",
"mac-eui-64",
"malware-sample",
"malware-type",
"md5",
"middle-name",
"mime-type",
"mobile-application-id",
"mutex",
"named pipe",
"nationality",
"other",
"passenger-name-record-locator-number",
"passport-country",
"passport-expiration",
"passport-number",
"pattern-in-file",
"pattern-in-memory",
"pattern-in-traffic",
"payment-details",
"pdb",
"pehash",
"phone-number",
"place-of-birth",
"place-port-of-clearance",
"place-port-of-onward-foreign-destination",
"place-port-of-original-embarkation",
"port",
"primary-residence",
"prtn",
"redress-number",
"user-agent",
"regkey",
"regkey|value",
"sha1",
"AS",
"snort",
"bro",
"pattern-in-file",
"pattern-in-traffic",
"pattern-in-memory",
"yara",
"stix2-pattern",
"sigma",
"gene",
"mime-type",
"identity-card-number",
"cookie",
"vulnerability",
"attachment",
"malware-sample",
"link",
"comment",
"text",
"hex",
"other",
"named pipe",
"mutex",
"target-user",
"target-email",
"target-machine",
"target-org",
"target-location",
"target-external",
"btc",
"xmr",
"iban",
"bic",
"bank-account-nr",
"aba-rtn",
"bin",
"cc-number",
"prtn",
"phone-number",
"threat-actor",
"campaign-name",
"campaign-id",
"malware-type",
"uri",
"authentihash",
"ssdeep",
"imphash",
"pehash",
"impfuzzy",
"sha224",
"sha256",
"sha384",
"sha512",
"sha512/224",
"sha512/256",
"sigma",
"size-in-bytes",
"snort",
"special-service-request",
"ssdeep",
"stix2-pattern",
"target-email",
"target-external",
"target-location",
"target-machine",
"target-org",
"target-user",
"text",
"threat-actor",
"tlsh",
"travel-details",
"twitter-id",
"uri",
"url",
"user-agent",
"visa-number",
"vulnerability",
"whois-creation-date",
"filename|authentihash",
"filename|ssdeep",
"filename|imphash",
"filename|impfuzzy",
"filename|pehash",
"filename|sha224",
"filename|sha384",
"filename|sha512",
"filename|sha512/224",
"filename|sha512/256",
"filename|tlsh",
"windows-scheduled-task",
"windows-service-name",
"windows-service-displayname",
"whois-registrant-email",
"whois-registrant-phone",
"whois-registrant-name",
"whois-registrant-org",
"whois-registrant-phone",
"whois-registrar",
"windows-scheduled-task",
"windows-service-displayname",
"windows-service-name",
"x509-fingerprint-md5",
"whois-creation-date",
"x509-fingerprint-sha1",
"x509-fingerprint-md5",
"x509-fingerprint-sha256",
"xmr",
"yara"
"dns-soa-email",
"size-in-bytes",
"counter",
"datetime",
"cpe",
"port",
"ip-dst|port",
"ip-src|port",
"hostname|port",
"mac-address",
"mac-eui-64",
"email-dst-display-name",
"email-src-display-name",
"email-header",
"email-reply-to",
"email-x-mailer",
"email-mime-boundary",
"email-thread-index",
"email-message-id",
"github-username",
"github-repository",
"github-organisation",
"jabber-id",
"twitter-id",
"first-name",
"middle-name",
"last-name",
"date-of-birth",
"place-of-birth",
"gender",
"passport-number",
"passport-country",
"passport-expiration",
"redress-number",
"nationality",
"visa-number",
"issue-date-of-the-visa",
"primary-residence",
"country-of-residence",
"special-service-request",
"frequent-flyer-number",
"travel-details",
"payment-details",
"place-port-of-original-embarkation",
"place-port-of-clearance",
"place-port-of-onward-foreign-destination",
"passenger-name-record-locator-number",
"mobile-application-id",
"cortex",
"boolean"
]
}
}
}

2
pymisp/data/misp-objects

@ -1 +1 @@
Subproject commit 7fe77c02affc0abe14cc67fe9f14400e8b72561c
Subproject commit 11a462e79b02428a08b11698d45aa8aa5ab6887d

9
pymisp/mispevent.py
View File

@ -469,8 +469,7 @@ class MISPEvent(AbstractMISP):
'attribute_count' in event.get('Event') and
event.get('Event').get('attribute_count') is None):
event['Event']['attribute_count'] = '0'
e = event.get('Event')
self.from_dict(**e)
self.from_dict(**event['Event'])
if validate:
jsonschema.validate(json.loads(self.to_json()), self.__json_schema)
@ -1017,6 +1016,12 @@ class MISPObject(AbstractMISP):
else:
self._known_template = False
if 'distribution' in kwargs and kwargs['distribution'] is not None:
self.distribution = kwargs.pop('distribution')
self.distribution = int(self.distribution)
if self.distribution not in [0, 1, 2, 3, 4, 5]:
raise NewAttributeError('{} is invalid, the distribution has to be in 0, 1, 2, 3, 4, 5'.format(self.distribution))
if kwargs.get('timestamp'):
if sys.version_info >= (3, 3):
self.timestamp = datetime.datetime.fromtimestamp(int(kwargs.pop('timestamp')), datetime.timezone.utc)

55
tests/testlive_comprehensive.py
View File

@ -3,7 +3,7 @@
import unittest
from pymisp import ExpandedPyMISP, MISPEvent, MISPOrganisation, MISPUser, Distribution, ThreatLevel, Analysis
from pymisp import ExpandedPyMISP, MISPEvent, MISPOrganisation, MISPUser, Distribution, ThreatLevel, Analysis, MISPObject
from datetime import datetime, timedelta, date
from io import BytesIO
@ -11,15 +11,15 @@ import time
try:
from keys import url, key
travis_run = True
except ImportError as e:
print(e)
url = 'http://localhost:8080'
key = 'BSip0zVadeFDeolkX2g7MHx8mrlr0uE04hh6CQj0'
key = 'LBelWqKY9SQyG0huZzAMqiEBl6FODxpgRRXMsZFu'
travis_run = False
from uuid import uuid4
travis_run = True
class TestComprehensive(unittest.TestCase):
@ -439,6 +439,51 @@ class TestComprehensive(unittest.TestCase):
self.admin_misp_connector.delete_event(first.id)
self.admin_misp_connector.delete_event(second.id)
def test_default_distribution(self):
'''The default distributions on the VM are This community only for the events and Inherit from event for attr/obj)'''
if travis_run:
return
first = self.create_simple_event()
del first.distribution
o = first.add_object(name='file')
o.add_attribute('filename', value='foo.exe')
try:
# Event create
first = self.user_misp_connector.add_event(first)
self.assertEqual(first.distribution, Distribution.this_community_only.value)
self.assertEqual(first.attributes[0].distribution, Distribution.inherit.value)
self.assertEqual(first.objects[0].distribution, Distribution.inherit.value)
self.assertEqual(first.objects[0].attributes[0].distribution, Distribution.inherit.value)
# Event edit
first.add_attribute('ip-dst', '12.54.76.43')
o = first.add_object(name='file')
o.add_attribute('filename', value='foo2.exe')
first = self.user_misp_connector.update_event(first)
self.assertEqual(first.attributes[1].distribution, Distribution.inherit.value)
self.assertEqual(first.objects[1].distribution, Distribution.inherit.value)
self.assertEqual(first.objects[1].attributes[0].distribution, Distribution.inherit.value)
# Attribute create
attribute = self.user_misp_connector.add_named_attribute(first, 'comment', 'bar')
# FIXME: Add helper that returns a list of MISPAttribute
self.assertEqual(attribute[0]['Attribute']['distribution'], str(Distribution.inherit.value))
# Object - add
o = MISPObject('file')
o.add_attribute('filename', value='blah.exe')
new_obj = self.user_misp_connector.add_object(first.id, o.template_uuid, o)
# FIXME: Add helper that returns a MISPObject
self.assertEqual(new_obj['Object']['distribution'], str(Distribution.inherit.value))
self.assertEqual(new_obj['Object']['Attribute'][0]['distribution'], str(Distribution.inherit.value))
# Object - edit
clean_obj = MISPObject(**new_obj['Object'])
clean_obj.from_dict(**new_obj['Object'])
clean_obj.add_attribute('filename', value='blah.exe')
new_obj = self.user_misp_connector.edit_object(clean_obj)
for a in new_obj['Object']['Attribute']:
self.assertEqual(a['distribution'], str(Distribution.inherit.value))
finally:
# Delete event
self.admin_misp_connector.delete_event(first.id)
def test_simple_event(self):
'''Search a bunch of parameters:
* Value not existing
@ -517,7 +562,7 @@ class TestComprehensive(unittest.TestCase):
# quickfilter
events = self.user_misp_connector.search(timestamp=timeframe,
quickfilter='%bar%', pythonify=True)
quickfilter='%foo blah%', pythonify=True)
# FIXME: should return one event
# print(events)
# self.assertEqual(len(events), 1)