mirror of https://github.com/MISP/PyMISP
Kory Kyzar
GitHub
commit
9dd04f8664
|
|
@ -20,13 +20,13 @@ install:
|
||||||
- pip install -U nose pip setuptools
|
- pip install -U nose pip setuptools
|
||||||
- pip install coveralls codecov requests-mock
|
- pip install coveralls codecov requests-mock
|
||||||
- pip install git+https://github.com/kbandla/pydeep.git
|
- pip install git+https://github.com/kbandla/pydeep.git
|
||||||
- pip install .[fileobjects,neo,openioc]
|
- pip install .[fileobjects,neo,openioc,virustotal]
|
||||||
- pushd tests
|
- pushd tests
|
||||||
- git clone https://github.com/viper-framework/viper-test-files.git
|
- git clone https://github.com/viper-framework/viper-test-files.git
|
||||||
- popd
|
- popd
|
||||||
|
|
||||||
script:
|
script:
|
||||||
- nosetests --with-coverage --cover-package=pymisp tests/test_offline.py
|
- nosetests --with-coverage --cover-package=pymisp tests/test_*.py
|
||||||
|
|
||||||
after_success:
|
after_success:
|
||||||
- codecov
|
- codecov
|
||||||
|
|
|
||||||
364
CHANGELOG.txt
364
CHANGELOG.txt
|
|
@ -2,6 +2,370 @@ Changelog
|
||||||
=========
|
=========
|
||||||
|
|
||||||
|
|
||||||
|
v2.4.85.1 (2018-01-10)
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
Changes
|
||||||
|
~~~~~~~
|
||||||
|
- Bump version. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Cleanup from last commit. [Raphaël Vinot]
|
||||||
|
- Move MISPTag to Abstract MISP. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Fix tests (new template version) [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Add test for loading existing malware sample from MISP. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
- Multiple changes. [Raphaël Vinot]
|
||||||
|
|
||||||
|
* Fix timestamp dump (properly enforce UTC)
|
||||||
|
* Properly handle proposals
|
||||||
|
* Add many getter/setter
|
||||||
|
* Add dedicated test cases for MISPEvent and other objects
|
||||||
|
- Allow do pass a category in default_attributes_parameters for object.
|
||||||
|
[Raphaël Vinot]
|
||||||
|
|
||||||
|
fix #166
|
||||||
|
- Default for sharing_group_id is 0. [Raphaël Vinot]
|
||||||
|
- Add MISPSighting class. [Raphaël Vinot]
|
||||||
|
- Bump Changelog. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Edited method works as expected, add tests. [Raphaël Vinot]
|
||||||
|
- Forgotten test files in last commit... [Raphaël Vinot]
|
||||||
|
- Disable_correlation from template not properly used. [Raphaël Vinot]
|
||||||
|
- Don't remove the distribution and sharing_group_id from
|
||||||
|
default_attributes_parameters. [Raphaël Vinot]
|
||||||
|
- The sharing_group_id isn't required. [Raphaël Vinot]
|
||||||
|
- Last commit was broken... [Raphaël Vinot]
|
||||||
|
- Properly set Tag to attributes within objects. [Raphaël Vinot]
|
||||||
|
- Add method to add tags to objects. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix #160
|
||||||
|
- Typo in set_sightings. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix #161
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Merge pull request #164 from MISP/refactor. [Raphaël Vinot]
|
||||||
|
|
||||||
|
chg: Multiple changes
|
||||||
|
- Merge pull request #162 from AninaAntonie/patch-1. [Raphaël Vinot]
|
||||||
|
|
||||||
|
fix: set_sightings
|
||||||
|
- Set_sightings. [AninaAntonie]
|
||||||
|
|
||||||
|
Maybe I didn't use it correctly but the method set_sightings didn't work for me. It's working now but I'm not sure whether sending a request for every sighting in the list is the best solution.
|
||||||
|
- Merge pull request #165 from dadokkio/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
_default_attributes_parameters - if set - is a dict
|
||||||
|
- _default_attributes_parameters - if set - is a dict. [Arcuri Davide]
|
||||||
|
|
||||||
|
Manage distribution and sharing_group_id as dict key like the other fields.
|
||||||
|
-- Not sure about default
|
||||||
|
|
||||||
|
|
||||||
|
v2.4.85 (2017-12-22)
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
New
|
||||||
|
~~~
|
||||||
|
- Add last field to get_csv. [Raphaël Vinot]
|
||||||
|
- (hopefully) Cleverer handling of timestamps in the objects. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
|
||||||
|
& some cleanup
|
||||||
|
|
||||||
|
Changes
|
||||||
|
~~~~~~~
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Version bump. [Raphaël Vinot]
|
||||||
|
- Update documentation. [Raphaël Vinot]
|
||||||
|
- Update documentation, cleanup. [Raphaël Vinot]
|
||||||
|
- Bump describeTypes.json. [Raphaël Vinot]
|
||||||
|
- Validate attributes in attributes.setter. [Raphaël Vinot]
|
||||||
|
- Add get_attribute_tag method at MISPEvent level. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Also add a MISPTag class for consistency.
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Bump describeTypes. [Raphaël Vinot]
|
||||||
|
- Add __repr__ methods (fix last commit) [Raphaël Vinot]
|
||||||
|
- Add __repr__ methods. [Raphaël Vinot]
|
||||||
|
- Use new format for filtering. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Bump describeTypes. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Properly use the edited flag. [Raphaël Vinot]
|
||||||
|
- Add setter for Attribute in MISPEvent. [Raphaël Vinot]
|
||||||
|
- Forgotten calls to master class. [Raphaël Vinot]
|
||||||
|
- Properly call datetime.datetime.utcfromtimestamp. [Raphaël Vinot]
|
||||||
|
- Fix typo. [Raphaël Vinot]
|
||||||
|
- Fix python2.7 support. [Raphaël Vinot]
|
||||||
|
- Initialize default class parameters. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix #155
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Merge branch 'cvandeplas-master' [Raphaël Vinot]
|
||||||
|
- Merge branch 'master' of https://github.com/cvandeplas/PyMISP into
|
||||||
|
cvandeplas-master. [Raphaël Vinot]
|
||||||
|
- Merge remote-tracking branch 'MISP/master' [Christophe Vandeplas]
|
||||||
|
- Fix MISPObject missing distribution and sharing_group_id. [Christophe
|
||||||
|
Vandeplas]
|
||||||
|
|
||||||
|
- fix MISPObject missing distribution concept
|
||||||
|
- fix language typo paramaters => parameters
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #156 from cvandeplas/master. [Alexandre Dulaunoy]
|
||||||
|
|
||||||
|
document submodule downloading
|
||||||
|
- Document submodule downloading. [Christophe Vandeplas]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #154 from wagner-certat/inc-meta. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Include documentation and examples in source dist
|
||||||
|
- Include documentation and examples in source dist. [Sebastian Wagner]
|
||||||
|
|
||||||
|
|
||||||
|
v2.4.84 (2017-12-13)
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
New
|
||||||
|
~~~
|
||||||
|
- Add methods to get taxonomy(ies) [Raphaël Vinot]
|
||||||
|
|
||||||
|
Thanks to @truckydev
|
||||||
|
- Add method to get all the events modified in an interval. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
|
||||||
|
Changes
|
||||||
|
~~~~~~~
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Bump Changelog. [Raphaël Vinot]
|
||||||
|
- Bump version. [Raphaël Vinot]
|
||||||
|
- Make the library easier to use. [Raphaël Vinot]
|
||||||
|
- Allow to pass a pseudofile to LIEF. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Update changelog. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Disable pseudofile support in py2, skip tests. [Raphaël Vinot]
|
||||||
|
- Typo in error output text description. [Eric Jaw]
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Merge pull request #151 from MISP/refactor. [Raphaël Vinot]
|
||||||
|
|
||||||
|
chg: Make the library easier to use
|
||||||
|
- Merge pull request #150 from sdrees/first-friendly-contribution-
|
||||||
|
enhance-coverage. [Raphaël Vinot]
|
||||||
|
|
||||||
|
First friendly contribution enhance coverage
|
||||||
|
- Further tests added (for public methods) [Stefan Hagen (Individual)]
|
||||||
|
- Changed asserts from dict usecases to set comparison to workaround non
|
||||||
|
3.6 behavior. [Stefan Hagen (Individual)]
|
||||||
|
- Merge branch 'master' of https://github.com/MISP/PyMISP into first-
|
||||||
|
friendly-contribution-enhance-coverage. [Stefan Hagen (Individual)]
|
||||||
|
- Enhance coverage and fix en passant with focus on api. [Stefan Hagen
|
||||||
|
(Individual)]
|
||||||
|
- Merge branch 'truckydev-get_last_modified_event' [Raphaël Vinot]
|
||||||
|
- Merge branch 'get_last_modified_event' of
|
||||||
|
https://github.com/truckydev/PyMISP into truckydev-
|
||||||
|
get_last_modified_event. [Raphaël Vinot]
|
||||||
|
- - Correction for 'last' param. 'last' gives the latest events that
|
||||||
|
have been published - add get_events_last_modified() this function
|
||||||
|
returns the modified events based on timestamp. [Tristan METAYER]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #149 from naisanza/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
fix: Typo in error output text description
|
||||||
|
|
||||||
|
|
||||||
|
v2.4.83 (2017-12-06)
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
New
|
||||||
|
~~~
|
||||||
|
- Add get CSV method. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Changes
|
||||||
|
~~~~~~~
|
||||||
|
- Allow to pass a proxy to query VT. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Bump version to 2.4.83. [Raphaël Vinot]
|
||||||
|
- Do not get the event from the server before publishing if
|
||||||
|
PyMISP.publish gets an ID. [Raphaël Vinot]
|
||||||
|
- Add live tests for recommended pymisp version and describeTypes up-to-
|
||||||
|
date. [Raphaël Vinot]
|
||||||
|
- Add a way to check if the ACL is up-to-date. [Raphaël Vinot]
|
||||||
|
- Add validators for describeTypes on the live instance. [Raphaël Vinot]
|
||||||
|
- Update PDF link to doc. [Raphaël Vinot]
|
||||||
|
- Add example file to push OpenIOC file to MISP. [Raphaël Vinot]
|
||||||
|
|
||||||
|
chg: Add some imports in the tool's init file
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Change version number to master in the doc. [Raphaël Vinot]
|
||||||
|
- Add new objects: MISPUser and MISPOrganisation. [Raphaël Vinot]
|
||||||
|
- Add a generic MISP object generator. [Raphaël Vinot]
|
||||||
|
- Allow to add multiple attribute of the same type. [Raphaël Vinot]
|
||||||
|
- Add fast publish method. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix #86
|
||||||
|
- Improve documentation. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix #121
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Typo in the tests. [Raphaël Vinot]
|
||||||
|
- Typo in live tests. [Raphaël Vinot]
|
||||||
|
- Bump describeTypes.json. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Add testing
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Merge pull request #147 from StrayLightning/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Check explicitly for a 500 response from the server with no response content
|
||||||
|
- Improve the exception message for a server 500+ response with no
|
||||||
|
response content. [StrayLightning]
|
||||||
|
- Check for zero-length 500 response from the server and produce a
|
||||||
|
suitable error message. [StrayLightning]
|
||||||
|
|
||||||
|
In experimenting with PyMISP I am triggering problems on the server I
|
||||||
|
am using. Occasionally the server will return a 500 response with a
|
||||||
|
message indicating an internal error, but more often than not it returns
|
||||||
|
a 500 response with no contents, and _check_response falls over itself,
|
||||||
|
generating hard-to-fathom exception from the json internals.
|
||||||
|
|
||||||
|
This commit hardens _check_response by detecting zero-length responses
|
||||||
|
and raising a suitable exception.
|
||||||
|
|
||||||
|
Also fix a missing bracket in one of the subsequent exception strings.
|
||||||
|
- Merge pull request #146 from c-goes/lief_integrity_exception. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
|
||||||
|
Make FileObject creation work if lief parsing fails
|
||||||
|
- Make FileObject creation work if lief parsing fails. [c-goes]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #144 from c-goes/objects_delete. [Raphaël Vinot]
|
||||||
|
|
||||||
|
allow deletion of objects and object references
|
||||||
|
- Allow deletion of objects and object references. [c-goes]
|
||||||
|
- Update doc badge links. [Raphaël Vinot]
|
||||||
|
- Merge pull request #143 from 3c7/feature/send_attributes. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
|
||||||
|
Adding multiple named attributes (without proposal) require a single POST request now
|
||||||
|
- Adding multiple named attributes require a single POST request now.
|
||||||
|
[3c7]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #142 from c-goes/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
replaced is_digit() with isdigit()
|
||||||
|
- Fixed typo. [c-goes]
|
||||||
|
- Merge remote-tracking branch 'upstream/master' [c-goes]
|
||||||
|
- Merge pull request #141 from SteveClement/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Remove CIRCL repo references from README.md & fix epydoc
|
||||||
|
- - Remove CIRCL reference from README.md - Updated 2 bad indentations
|
||||||
|
where epydoc was Warning. [Steve Clement]
|
||||||
|
- Merge remote-tracking branch 'upstream/master' [c-goes]
|
||||||
|
- Merge branch 'master' of https://github.com/MISP/PyMISP into
|
||||||
|
messageidtype. [c-goes]
|
||||||
|
- Added default_category for email-message-id. [c-goes]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge branch 'feature/feedgenerator_rework' [iglocska]
|
||||||
|
- Merge branch 'master' of https://github.com/MISP/PyMISP into
|
||||||
|
feature/feedgenerator_rework. [iglocska]
|
||||||
|
- Rework of the feed generator. [iglocska]
|
||||||
|
|
||||||
|
- use objects, attribute tags and object references correctly
|
||||||
|
- generate quickhashlist for fast lookups / future MISP caching mechanism
|
||||||
|
- saner structure (herp-a-derp)
|
||||||
|
|
||||||
|
|
||||||
|
v2.4.82 (2017-11-09)
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
New
|
||||||
|
~~~
|
||||||
|
- Proper debug system. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Make it easy to investigate the json blobs sent to the server.
|
||||||
|
|
||||||
|
Changes
|
||||||
|
~~~~~~~
|
||||||
|
- Bump PyMISP version. [Raphaël Vinot]
|
||||||
|
- Bump CHANGELOG. [Raphaël Vinot]
|
||||||
|
- Bump misp-objects. [Raphaël Vinot]
|
||||||
|
- Update readme for new logging system. [Raphaël Vinot]
|
||||||
|
- Small improvments in the logging system. [Raphaël Vinot]
|
||||||
|
- Properly use python logging module. [Raphaël Vinot]
|
||||||
|
- Update asciidoctor generator. [Raphaël Vinot]
|
||||||
|
- Remove warning if PyMISP is too new. [Raphaël Vinot]
|
||||||
|
- Add simple asciidoc generator for MISP event. [Raphaël Vinot]
|
||||||
|
- Update changelog. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix
|
||||||
|
~~~
|
||||||
|
- Typo loger -> logger. [Raphaël Vinot]
|
||||||
|
- Let load unknown object relations in known templates. [Raphaël Vinot]
|
||||||
|
|
||||||
|
This isn't recommended, but happens very often.
|
||||||
|
- Allow to load non-malware ZIP files in MISP Event. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Prior to his patch, any zip file loaded by MISP Event was unpacked and
|
||||||
|
processed as an excrypted malware from MISP.
|
||||||
|
- Properly pass the distribution when uploading a sample. [Raphaël
|
||||||
|
Vinot]
|
||||||
|
- Properly upload a sample in an existing event. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Fix https://github.com/MISP/PyMISP/issues/123
|
||||||
|
- Properly set the distribution at event level. [Raphaël Vinot]
|
||||||
|
|
||||||
|
fix #120
|
||||||
|
- Properly pop the distribution key. [Raphaël Vinot]
|
||||||
|
- Update dependencies for VT generator. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- Merge pull request #126 from CenturyLinkCIRT/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Added vt_to_misp.py example and VTReportObject
|
||||||
|
- Merge branch 'master' of https://github.com/MISP/PyMISP. [Thomas
|
||||||
|
Gardner]
|
||||||
|
- Fix test suite. [Raphaël Vinot]
|
||||||
|
- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
|
||||||
|
- Merge pull request #122 from LDO-CERT/master. [Raphaël Vinot]
|
||||||
|
|
||||||
|
Created add_generic_object.py
|
||||||
|
- Created add_generic_object.py. [garanews]
|
||||||
|
|
||||||
|
usage: add_generic_object.py [-h] -e EVENT -t TYPE -d DICT
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
python3 add_generic_object.py -e 1683 -t email -d '{"subject":"The Pink Letter", "to":"jon@snow.org"}'
|
||||||
|
python3 add_generic_object.py -e 2343 -t person -d '{"first-name":"Daenerys", "last-name":"Targaryen", "place-of-birth":"Dragonstone"}'
|
||||||
|
python3 add_generic_object.py -e 3596 -t "domain|ip" -d '{"domain":"stormborn.org", "ip":"50.63.202.33"}'
|
||||||
|
- Added vtreportobject and vt_to_misp example. [Thomas Gardner]
|
||||||
|
- Created add_generic_object.py. [garanews]
|
||||||
|
|
||||||
|
usage: add_generic_object.py [-h] -e EVENT -t TYPE -d DICT
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
python3 add_generic_object.py -e 1683 -t email -d '{"subject":"The Pink Letter", "to":"jon@snow.org"}'
|
||||||
|
python3 add_generic_object.py -e 2343 -t person -d '{"first-name":"Daenerys", "last-name":"Targaryen", "place-of-birth":"Dragonstone"}'
|
||||||
|
python3 add_generic_object.py -e 3596 -t "domain|ip" -d '{"domain":"stormborn.org", "ip":"50.63.202.33"}'
|
||||||
|
|
||||||
|
|
||||||
v2.4.81.2 (2017-10-24)
|
v2.4.81.2 (2017-10-24)
|
||||||
----------------------
|
----------------------
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,10 @@
|
||||||
|
graft docs
|
||||||
|
graft examples
|
||||||
|
graft tests
|
||||||
|
include CHANGELOG.txt
|
||||||
|
include LICENSE
|
||||||
include pymisp/data/*.json
|
include pymisp/data/*.json
|
||||||
include pymisp/data/misp-objects/*.json
|
include pymisp/data/misp-objects/*.json
|
||||||
include pymisp/data/misp-objects/objects/*/definition.json
|
include pymisp/data/misp-objects/objects/*/definition.json
|
||||||
include pymisp/data/misp-objects/relationships/definition.json
|
include pymisp/data/misp-objects/relationships/definition.json
|
||||||
|
include README.md
|
||||||
|
|
|
||||||
36
README.md
36
README.md
|
|
@ -1,7 +1,7 @@
|
||||||
README
|
README
|
||||||
======
|
======
|
||||||
|
|
||||||
[](http://pymisp.readthedocs.io/en/master/?badge=master)
|
[](http://pymisp.readthedocs.io/?badge=latest)
|
||||||
[](https://travis-ci.org/MISP/PyMISP)
|
[](https://travis-ci.org/MISP/PyMISP)
|
||||||
[](https://coveralls.io/github/MISP/PyMISP?branch=master)
|
[](https://coveralls.io/github/MISP/PyMISP?branch=master)
|
||||||
|
|
||||||
|
|
@ -24,7 +24,8 @@ pip3 install pymisp
|
||||||
## Install the latest version from repo
|
## Install the latest version from repo
|
||||||
|
|
||||||
```
|
```
|
||||||
git clone https://github.com/CIRCL/PyMISP.git && cd PyMISP
|
git clone https://github.com/MISP/PyMISP.git && cd PyMISP
|
||||||
|
git submodule update --init
|
||||||
pip3 install -I .
|
pip3 install -I .
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -50,14 +51,41 @@ cd examples
|
||||||
python3 last.py -l 10
|
python3 last.py -l 10
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Debugging
|
||||||
|
|
||||||
|
You have two options there:
|
||||||
|
|
||||||
|
1. Pass `debug=True` to `PyMISP` and it will enable logging.DEBUG to stderr on the whole module
|
||||||
|
|
||||||
|
2. Use the python logging module directly:
|
||||||
|
|
||||||
|
```python
|
||||||
|
|
||||||
|
import logging
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
|
# Configure it as you whish, for example, enable DEBUG mode:
|
||||||
|
logger.setLevel(logging.DEBUG)
|
||||||
|
```
|
||||||
|
|
||||||
|
Or if you want to write the debug output to a file instead of stderr:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pymisp
|
||||||
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
logging.basicConfig(level=logging.DEBUG, filename="debug.log", filemode='w', format=pymisp.FORMAT)
|
||||||
|
```
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
[PyMISP API documentation is available](https://media.readthedocs.org/pdf/pymisp/master/pymisp.pdf).
|
[PyMISP API documentation is available](https://media.readthedocs.org/pdf/pymisp/latest/pymisp.pdf).
|
||||||
|
|
||||||
Documentation can be generated with epydoc:
|
Documentation can be generated with epydoc:
|
||||||
|
|
||||||
```
|
```
|
||||||
epydoc --url https://github.com/CIRCL/PyMISP --graph all --name PyMISP --pdf pymisp -o doc
|
epydoc --url https://github.com/MISP/PyMISP --graph all --name PyMISP --pdf pymisp -o doc
|
||||||
```
|
```
|
||||||
|
|
||||||
## Everything is a Mutable Mapping
|
## Everything is a Mutable Mapping
|
||||||
|
|
|
||||||
|
|
@ -17,10 +17,6 @@
|
||||||
# add these directories to sys.path here. If the directory is relative to the
|
# add these directories to sys.path here. If the directory is relative to the
|
||||||
# documentation root, use os.path.abspath to make it absolute, like shown here.
|
# documentation root, use os.path.abspath to make it absolute, like shown here.
|
||||||
#
|
#
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, os.path.abspath('.'))
|
|
||||||
|
|
||||||
from recommonmark.parser import CommonMarkParser
|
from recommonmark.parser import CommonMarkParser
|
||||||
|
|
||||||
# -- General configuration ------------------------------------------------
|
# -- General configuration ------------------------------------------------
|
||||||
|
|
@ -78,9 +74,9 @@ author = 'Raphaël Vinot'
|
||||||
# built documents.
|
# built documents.
|
||||||
#
|
#
|
||||||
# The short X.Y version.
|
# The short X.Y version.
|
||||||
version = '2.4'
|
version = 'master'
|
||||||
# The full version, including alpha/beta/rc tags.
|
# The full version, including alpha/beta/rc tags.
|
||||||
release = '2.4.77'
|
release = 'master'
|
||||||
|
|
||||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||||
# for a list of supported languages.
|
# for a list of supported languages.
|
||||||
|
|
|
||||||
|
|
@ -9,12 +9,11 @@ Welcome to PyMISP's documentation!
|
||||||
Contents:
|
Contents:
|
||||||
|
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:maxdepth: 2
|
:maxdepth: 4
|
||||||
|
|
||||||
README
|
README
|
||||||
modules
|
modules
|
||||||
|
tools
|
||||||
|
|
||||||
|
|
||||||
Indices and tables
|
Indices and tables
|
||||||
==================
|
==================
|
||||||
|
|
@ -22,4 +21,3 @@ Indices and tables
|
||||||
* :ref:`genindex`
|
* :ref:`genindex`
|
||||||
* :ref:`modindex`
|
* :ref:`modindex`
|
||||||
* :ref:`search`
|
* :ref:`search`
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,4 +4,82 @@ pymisp
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:maxdepth: 4
|
:maxdepth: 4
|
||||||
|
|
||||||
pymisp
|
.. automodule:: pymisp
|
||||||
|
:members:
|
||||||
|
|
||||||
|
|
||||||
|
PyMISP
|
||||||
|
------
|
||||||
|
|
||||||
|
.. autoclass:: PyMISP
|
||||||
|
:members:
|
||||||
|
|
||||||
|
MISPAbstract
|
||||||
|
------------
|
||||||
|
|
||||||
|
.. autoclass:: AbstractMISP
|
||||||
|
:members:
|
||||||
|
|
||||||
|
MISPEncode
|
||||||
|
----------
|
||||||
|
|
||||||
|
.. autoclass:: MISPEncode
|
||||||
|
:members:
|
||||||
|
|
||||||
|
MISPEvent
|
||||||
|
---------
|
||||||
|
|
||||||
|
.. autoclass:: MISPEvent
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
MISPAttribute
|
||||||
|
-------------
|
||||||
|
|
||||||
|
.. autoclass:: MISPAttribute
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
MISPObject
|
||||||
|
----------
|
||||||
|
|
||||||
|
.. autoclass:: MISPObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
MISPObjectAttribute
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
.. autoclass:: MISPObjectAttribute
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
MISPObjectReference
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
.. autoclass:: MISPObjectReference
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
MISPTag
|
||||||
|
-------
|
||||||
|
|
||||||
|
.. autoclass:: MISPTag
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
MISPUser
|
||||||
|
--------
|
||||||
|
|
||||||
|
.. autoclass:: MISPUser
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
|
||||||
|
MISPOrganisation
|
||||||
|
----------------
|
||||||
|
|
||||||
|
.. autoclass:: MISPOrganisation
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,22 +0,0 @@
|
||||||
pymisp package
|
|
||||||
==============
|
|
||||||
|
|
||||||
Submodules
|
|
||||||
----------
|
|
||||||
|
|
||||||
pymisp.api module
|
|
||||||
-----------------
|
|
||||||
|
|
||||||
.. automodule:: pymisp.api
|
|
||||||
:members:
|
|
||||||
:undoc-members:
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
|
|
||||||
Module contents
|
|
||||||
---------------
|
|
||||||
|
|
||||||
.. automodule:: pymisp
|
|
||||||
:members:
|
|
||||||
:undoc-members:
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
@ -0,0 +1,69 @@
|
||||||
|
pymisp - Tools
|
||||||
|
==============
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 4
|
||||||
|
|
||||||
|
.. automodule:: pymisp.tools
|
||||||
|
:members:
|
||||||
|
|
||||||
|
File Object
|
||||||
|
-----------
|
||||||
|
|
||||||
|
.. autoclass:: FileObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
ELF Object
|
||||||
|
----------
|
||||||
|
|
||||||
|
.. autoclass:: ELFObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
.. autoclass:: ELFSectionObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
PE Object
|
||||||
|
---------
|
||||||
|
|
||||||
|
.. autoclass:: PEObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
.. autoclass:: PESectionObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
Mach-O Object
|
||||||
|
-------------
|
||||||
|
|
||||||
|
.. autoclass:: MachOObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
.. autoclass:: MachOSectionObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
VT Report Object
|
||||||
|
----------------
|
||||||
|
|
||||||
|
.. autoclass:: VTReportObject
|
||||||
|
:members:
|
||||||
|
:inherited-members:
|
||||||
|
|
||||||
|
STIX
|
||||||
|
----
|
||||||
|
|
||||||
|
.. automodule:: pymisp.tools.stix
|
||||||
|
:members:
|
||||||
|
|
||||||
|
OpenIOC
|
||||||
|
--------
|
||||||
|
|
||||||
|
.. automethod:: pymisp.tools.load_openioc
|
||||||
|
|
||||||
|
.. automethod:: pymisp.tools.load_openioc_file
|
||||||
|
|
||||||
|
|
@ -1,24 +1,22 @@
|
||||||
|
#!/usr/bin/env python
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
import json
|
import json
|
||||||
from pymisp import PyMISP
|
from pymisp import PyMISP
|
||||||
from pymisp.tools.abstractgenerator import AbstractMISPObjectGenerator
|
from pymisp.tools import GenericObjectGenerator
|
||||||
from keys import misp_url, misp_key, misp_verifycert
|
from keys import misp_url, misp_key, misp_verifycert
|
||||||
import argparse
|
import argparse
|
||||||
|
|
||||||
class GenericObject(AbstractMISPObjectGenerator):
|
"""
|
||||||
def __init__(self, type, data_dict):
|
Sample usage:
|
||||||
super(GenericObject, self).__init__(type)
|
./add_generic_object.py -e 5065 -t email -l '[{"to": "undisclosed@ppp.com"}, {"to": "second.to@mail.com"}]'
|
||||||
self.__data = data_dict
|
"""
|
||||||
self.generate_attributes()
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
|
||||||
for key, value in self.__data.items():
|
|
||||||
self.add_attribute(key, value=value)
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
parser = argparse.ArgumentParser(description='Create a MISP Object selectable by type starting from a dictionary')
|
parser = argparse.ArgumentParser(description='Create a MISP Object selectable by type starting from a dictionary')
|
||||||
parser.add_argument("-e", "--event", required=True, help="Event ID to update")
|
parser.add_argument("-e", "--event", required=True, help="Event ID to update")
|
||||||
parser.add_argument("-t", "--type", required=True, help="Type of the generic object")
|
parser.add_argument("-t", "--type", required=True, help="Type of the generic object")
|
||||||
parser.add_argument("-d", "--dict", required=True, help="Dict ")
|
parser.add_argument("-l", "--attr_list", required=True, help="List of attributes")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)
|
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)
|
||||||
|
|
@ -29,5 +27,6 @@ if __name__ == '__main__':
|
||||||
print ("Template for type %s not found! Valid types are: %s" % (args.type, valid_types))
|
print ("Template for type %s not found! Valid types are: %s" % (args.type, valid_types))
|
||||||
exit()
|
exit()
|
||||||
|
|
||||||
misp_object = GenericObject(args.type.replace("|", "-"), json.loads(args.dict))
|
misp_object = GenericObjectGenerator(args.type.replace("|", "-"))
|
||||||
|
misp_object.generate_attributes(json.loads(args.attr_list))
|
||||||
r = pymisp.add_object(args.event, template_id, misp_object)
|
r = pymisp.add_object(args.event, template_id, misp_object)
|
||||||
|
|
|
||||||
|
|
@ -1,55 +1,21 @@
|
||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from datetime import date
|
||||||
|
import importlib
|
||||||
|
|
||||||
from pymisp import MISPEvent
|
from pymisp import MISPEvent
|
||||||
from defang import defang
|
from defang import defang
|
||||||
import argparse
|
|
||||||
from pytaxonomies import Taxonomies
|
from pytaxonomies import Taxonomies
|
||||||
from datetime import date
|
|
||||||
|
|
||||||
headers = """
|
|
||||||
:toc: right
|
|
||||||
:toclevels: 1
|
|
||||||
:toc-title: Daily Report
|
|
||||||
:icons: font
|
|
||||||
:sectanchors:
|
|
||||||
:sectlinks:
|
|
||||||
= Daily report by {org_name}
|
|
||||||
{date}
|
|
||||||
|
|
||||||
:icons: font
|
|
||||||
|
|
||||||
"""
|
|
||||||
|
|
||||||
event_level_tags = """
|
|
||||||
IMPORTANT: This event is classified TLP:{value}.
|
|
||||||
|
|
||||||
{expanded}
|
|
||||||
|
|
||||||
"""
|
|
||||||
|
|
||||||
attributes = """
|
|
||||||
=== Indicator(s) of compromise
|
|
||||||
|
|
||||||
{list_attributes}
|
|
||||||
|
|
||||||
"""
|
|
||||||
|
|
||||||
title = """
|
|
||||||
== ({internal_id}) {title}
|
|
||||||
|
|
||||||
{summary}
|
|
||||||
|
|
||||||
"""
|
|
||||||
|
|
||||||
types_to_attach = ['ip-dst', 'url', 'domain']
|
|
||||||
objects_to_attach = ['domain-ip']
|
|
||||||
|
|
||||||
class ReportGenerator():
|
class ReportGenerator():
|
||||||
|
def __init__(self, profile="daily_report"):
|
||||||
def __init__(self):
|
|
||||||
self.taxonomies = Taxonomies()
|
self.taxonomies = Taxonomies()
|
||||||
self.report = ''
|
self.report = ''
|
||||||
|
profile_name = "profiles.{}".format(profile)
|
||||||
|
self.template = importlib.import_module(name=profile_name)
|
||||||
|
|
||||||
def from_remote(self, event_id):
|
def from_remote(self, event_id):
|
||||||
from pymisp import PyMISP
|
from pymisp import PyMISP
|
||||||
|
|
@ -66,15 +32,16 @@ class ReportGenerator():
|
||||||
def attributes(self):
|
def attributes(self):
|
||||||
if not self.misp_event.attributes:
|
if not self.misp_event.attributes:
|
||||||
return ''
|
return ''
|
||||||
list_attributes = ''
|
list_attributes = []
|
||||||
for attribute in self.misp_event.attributes:
|
for attribute in self.misp_event.attributes:
|
||||||
if attribute.type in types_to_attach:
|
if attribute.type in self.template.types_to_attach:
|
||||||
list_attributes += "\n* {}\n".format(defang(attribute.value))
|
list_attributes.append("* {}".format(defang(attribute.value)))
|
||||||
for obj in self.misp_event.Object:
|
for obj in self.misp_event.Object:
|
||||||
for attribute in obj.Attribute:
|
if obj.name in self.template.objects_to_attach:
|
||||||
if attribute.type in types_to_attach:
|
for attribute in obj.Attribute:
|
||||||
list_attributes += "\n* {}\n".format(defang(attribute.value))
|
if attribute.type in self.template.types_to_attach:
|
||||||
return attributes.format(list_attributes=list_attributes)
|
list_attributes.append("* {}".format(defang(attribute.value)))
|
||||||
|
return self.template.attributes.format(list_attributes="\n".join(list_attributes))
|
||||||
|
|
||||||
def _get_tag_info(self, machinetag):
|
def _get_tag_info(self, machinetag):
|
||||||
return self.taxonomies.revert_machinetag(machinetag)
|
return self.taxonomies.revert_machinetag(machinetag)
|
||||||
|
|
@ -82,7 +49,7 @@ class ReportGenerator():
|
||||||
def report_headers(self):
|
def report_headers(self):
|
||||||
content = {'org_name': 'name',
|
content = {'org_name': 'name',
|
||||||
'date': date.today().isoformat()}
|
'date': date.today().isoformat()}
|
||||||
self.report += headers.format(**content)
|
self.report += self.template.headers.format(**content)
|
||||||
|
|
||||||
def event_level_tags(self):
|
def event_level_tags(self):
|
||||||
if not self.misp_event.Tag:
|
if not self.misp_event.Tag:
|
||||||
|
|
@ -91,7 +58,7 @@ class ReportGenerator():
|
||||||
# Only look for TLP for now
|
# Only look for TLP for now
|
||||||
if tag['name'].startswith('tlp'):
|
if tag['name'].startswith('tlp'):
|
||||||
tax, predicate = self._get_tag_info(tag['name'])
|
tax, predicate = self._get_tag_info(tag['name'])
|
||||||
return event_level_tags.format(value=predicate.predicate.upper(), expanded=predicate.expanded)
|
return self.template.event_level_tags.format(value=predicate.predicate.upper(), expanded=predicate.expanded)
|
||||||
|
|
||||||
def title(self):
|
def title(self):
|
||||||
internal_id = ''
|
internal_id = ''
|
||||||
|
|
@ -106,34 +73,42 @@ class ReportGenerator():
|
||||||
if a.object_relation == 'summary':
|
if a.object_relation == 'summary':
|
||||||
summary = a.value
|
summary = a.value
|
||||||
|
|
||||||
return title.format(internal_id=internal_id, title=self.misp_event.info,
|
return self.template.title.format(internal_id=internal_id, title=self.misp_event.info,
|
||||||
summary=summary)
|
summary=summary)
|
||||||
|
|
||||||
|
|
||||||
def asciidoc(self, lang='en'):
|
def asciidoc(self, lang='en'):
|
||||||
self.report += self.title()
|
self.report += self.title()
|
||||||
self.report += self.event_level_tags()
|
self.report += self.event_level_tags()
|
||||||
self.report += self.attributes()
|
self.report += self.attributes()
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
try:
|
||||||
|
parser = argparse.ArgumentParser(description='Create a human-readable report out of a MISP event')
|
||||||
|
parser.add_argument("--profile", default="daily_report", help="Profile template to use")
|
||||||
|
parser.add_argument("-o", "--output", help="Output file to write to (generally ends in .adoc)")
|
||||||
|
group = parser.add_mutually_exclusive_group(required=True)
|
||||||
|
group.add_argument("-e", "--event", default=[], nargs='+', help="Event ID to get.")
|
||||||
|
group.add_argument("-p", "--path", default=[], nargs='+', help="Path to the JSON dump.")
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(description='Create a human-readable report out of a MISP event')
|
args = parser.parse_args()
|
||||||
group = parser.add_mutually_exclusive_group(required=True)
|
|
||||||
group.add_argument("-e", "--event", default=[], nargs='+', help="Event ID to get.")
|
|
||||||
group.add_argument("-p", "--path", default=[], nargs='+', help="Path to the JSON dump.")
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
report = ReportGenerator(args.profile)
|
||||||
|
report.report_headers()
|
||||||
|
|
||||||
report = ReportGenerator()
|
if args.event:
|
||||||
report.report_headers()
|
for eid in args.event:
|
||||||
|
report.from_remote(eid)
|
||||||
|
report.asciidoc()
|
||||||
|
else:
|
||||||
|
for f in args.path:
|
||||||
|
report.from_file(f)
|
||||||
|
report.asciidoc()
|
||||||
|
|
||||||
if args.event:
|
if args.output:
|
||||||
for eid in args.event:
|
with open(args.output, "w") as ofile:
|
||||||
report.from_remote(eid)
|
ofile.write(report.report)
|
||||||
report.asciidoc()
|
else:
|
||||||
else:
|
print(report.report)
|
||||||
for f in args.path:
|
except ModuleNotFoundError as err:
|
||||||
report.from_file(f)
|
print(err)
|
||||||
report.asciidoc()
|
|
||||||
|
|
||||||
print(report.report)
|
|
||||||
|
|
|
||||||
|
|
@ -4,28 +4,79 @@
|
||||||
import sys
|
import sys
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import hashlib
|
||||||
from pymisp import PyMISP
|
from pymisp import PyMISP
|
||||||
from settings import url, key, ssl, outputdir, filters, valid_attribute_distribution_levels
|
from settings import url, key, ssl, outputdir, filters, valid_attribute_distribution_levels
|
||||||
|
|
||||||
|
objectsFields = {
|
||||||
|
'Attribute': {
|
||||||
|
'uuid',
|
||||||
|
'value',
|
||||||
|
'category',
|
||||||
|
'type',
|
||||||
|
'comment',
|
||||||
|
'data',
|
||||||
|
'timestamp',
|
||||||
|
'to_ids'
|
||||||
|
},
|
||||||
|
'Event': {
|
||||||
|
'uuid',
|
||||||
|
'info',
|
||||||
|
'threat_level_id',
|
||||||
|
'analysis',
|
||||||
|
'timestamp',
|
||||||
|
'publish_timestamp',
|
||||||
|
'published',
|
||||||
|
'date'
|
||||||
|
},
|
||||||
|
'Object': {
|
||||||
|
'name',
|
||||||
|
'meta-category',
|
||||||
|
'description',
|
||||||
|
'template_uuid',
|
||||||
|
'template_version',
|
||||||
|
'uuid',
|
||||||
|
'timestamp',
|
||||||
|
'distribution',
|
||||||
|
'sharing_group_id',
|
||||||
|
'comment'
|
||||||
|
},
|
||||||
|
'ObjectReference': {
|
||||||
|
'uuid',
|
||||||
|
'timestamp',
|
||||||
|
'relationship_type',
|
||||||
|
'comment',
|
||||||
|
'object_uuid',
|
||||||
|
'referenced_uuid'
|
||||||
|
},
|
||||||
|
'Orgc': {
|
||||||
|
'name',
|
||||||
|
'uuid'
|
||||||
|
},
|
||||||
|
'Tag': {
|
||||||
|
'name',
|
||||||
|
'colour',
|
||||||
|
'exportable'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
objectsToSave = {'Orgc': {'fields': ['name', 'uuid'],
|
objectsToSave = {
|
||||||
'multiple': False,
|
'Orgc': {},
|
||||||
},
|
'Tag': {},
|
||||||
'Tag': {'fields': ['name', 'colour', 'exportable'],
|
'Attribute': {
|
||||||
'multiple': True,
|
'Tag': {}
|
||||||
},
|
},
|
||||||
'Attribute': {'fields': ['uuid', 'value', 'category', 'type',
|
'Object': {
|
||||||
'comment', 'data', 'timestamp', 'to_ids'],
|
'Attribute': {
|
||||||
'multiple': True,
|
'Tag': {}
|
||||||
},
|
},
|
||||||
}
|
'ObjectReference': {}
|
||||||
|
}
|
||||||
fieldsToSave = ['uuid', 'info', 'threat_level_id', 'analysis',
|
}
|
||||||
'timestamp', 'publish_timestamp', 'published',
|
|
||||||
'date']
|
|
||||||
|
|
||||||
valid_attribute_distributions = []
|
valid_attribute_distributions = []
|
||||||
|
|
||||||
|
attributeHashes = []
|
||||||
|
|
||||||
def init():
|
def init():
|
||||||
# If we have an old settings.py file then this variable won't exist
|
# If we have an old settings.py file then this variable won't exist
|
||||||
|
|
@ -36,61 +87,65 @@ def init():
|
||||||
valid_attribute_distributions = ['0', '1', '2', '3', '4', '5']
|
valid_attribute_distributions = ['0', '1', '2', '3', '4', '5']
|
||||||
return PyMISP(url, key, ssl)
|
return PyMISP(url, key, ssl)
|
||||||
|
|
||||||
|
def recursiveExtract(container, containerType, leaf, eventUuid):
|
||||||
|
temp = {}
|
||||||
|
if containerType in ['Attribute', 'Object']:
|
||||||
|
if (__blockByDistribution(container)):
|
||||||
|
return False
|
||||||
|
for field in objectsFields[containerType]:
|
||||||
|
if field in container:
|
||||||
|
temp[field] = container[field]
|
||||||
|
if (containerType == 'Attribute'):
|
||||||
|
global attributeHashes
|
||||||
|
if ('|' in container['type'] or container['type'] == 'malware-sample'):
|
||||||
|
split = container['value'].split('|')
|
||||||
|
attributeHashes.append([hashlib.md5(split[0].encode("utf-8")).hexdigest(), eventUuid])
|
||||||
|
attributeHashes.append([hashlib.md5(split[1].encode("utf-8")).hexdigest(), eventUuid])
|
||||||
|
else:
|
||||||
|
attributeHashes.append([hashlib.md5(container['value'].encode("utf-8")).hexdigest(), eventUuid])
|
||||||
|
children = leaf.keys()
|
||||||
|
for childType in children:
|
||||||
|
childContainer = container.get(childType)
|
||||||
|
if (childContainer):
|
||||||
|
if (type(childContainer) is dict):
|
||||||
|
temp[childType] = recursiveExtract(childContainer, childType, leaf[childType], eventUuid)
|
||||||
|
else:
|
||||||
|
temp[childType] = []
|
||||||
|
for element in childContainer:
|
||||||
|
processed = recursiveExtract(element, childType, leaf[childType], eventUuid)
|
||||||
|
if (processed):
|
||||||
|
temp[childType].append(processed)
|
||||||
|
return temp
|
||||||
|
|
||||||
def saveEvent(misp, uuid):
|
def saveEvent(misp, uuid):
|
||||||
|
result = {}
|
||||||
event = misp.get_event(uuid)
|
event = misp.get_event(uuid)
|
||||||
if not event.get('Event'):
|
if not event.get('Event'):
|
||||||
print('Error while fetching event: {}'.format(event['message']))
|
print('Error while fetching event: {}'.format(event['message']))
|
||||||
sys.exit('Could not create file for event ' + uuid + '.')
|
sys.exit('Could not create file for event ' + uuid + '.')
|
||||||
event = __cleanUpEvent(event)
|
event['Event'] = recursiveExtract(event['Event'], 'Event', objectsToSave, event['Event']['uuid'])
|
||||||
event = json.dumps(event)
|
event = json.dumps(event)
|
||||||
eventFile = open(os.path.join(outputdir, uuid + '.json'), 'w')
|
eventFile = open(os.path.join(outputdir, uuid + '.json'), 'w')
|
||||||
eventFile.write(event)
|
eventFile.write(event)
|
||||||
eventFile.close()
|
eventFile.close()
|
||||||
|
|
||||||
|
def __blockByDistribution(element):
|
||||||
def __cleanUpEvent(event):
|
if element['distribution'] not in valid_attribute_distributions:
|
||||||
temp = event
|
|
||||||
event = {'Event': {}}
|
|
||||||
__cleanupEventFields(event, temp)
|
|
||||||
__cleanupEventObjects(event, temp)
|
|
||||||
return event
|
|
||||||
|
|
||||||
|
|
||||||
def __cleanupEventFields(event, temp):
|
|
||||||
for field in fieldsToSave:
|
|
||||||
if field in temp['Event'].keys():
|
|
||||||
event['Event'][field] = temp['Event'][field]
|
|
||||||
return event
|
|
||||||
|
|
||||||
|
|
||||||
def __blockAttributeByDistribution(attribute):
|
|
||||||
if attribute['distribution'] not in valid_attribute_distributions:
|
|
||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def saveHashes():
|
||||||
|
if not attributeHashes:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
hashFile = open(os.path.join(outputdir, 'hashes.csv'), 'w')
|
||||||
|
for element in attributeHashes:
|
||||||
|
hashFile.write('{},{}\n'.format(element[0], element[1]))
|
||||||
|
hashFile.close()
|
||||||
|
except Exception as e:
|
||||||
|
print(e)
|
||||||
|
sys.exit('Could not create the quick hash lookup file.')
|
||||||
|
|
||||||
def __cleanupEventObjects(event, temp):
|
|
||||||
for objectType in objectsToSave.keys():
|
|
||||||
if objectsToSave[objectType]['multiple'] is True:
|
|
||||||
if objectType in temp['Event']:
|
|
||||||
for objectInstance in temp['Event'][objectType]:
|
|
||||||
if objectType is 'Attribute':
|
|
||||||
if __blockAttributeByDistribution(objectInstance):
|
|
||||||
continue
|
|
||||||
tempObject = {}
|
|
||||||
for field in objectsToSave[objectType]['fields']:
|
|
||||||
if field in objectInstance.keys():
|
|
||||||
tempObject[field] = objectInstance[field]
|
|
||||||
if objectType not in event['Event']:
|
|
||||||
event['Event'][objectType] = []
|
|
||||||
event['Event'][objectType].append(tempObject)
|
|
||||||
else:
|
|
||||||
tempObject = {}
|
|
||||||
for field in objectsToSave[objectType]['fields']:
|
|
||||||
tempObject[field] = temp['Event'][objectType][field]
|
|
||||||
event['Event'][objectType] = tempObject
|
|
||||||
return event
|
|
||||||
|
|
||||||
|
|
||||||
def saveManifest(manifest):
|
def saveManifest(manifest):
|
||||||
|
|
@ -138,4 +193,6 @@ if __name__ == '__main__':
|
||||||
print("Event " + str(counter) + "/" + str(total) + " exported.")
|
print("Event " + str(counter) + "/" + str(total) + " exported.")
|
||||||
counter += 1
|
counter += 1
|
||||||
saveManifest(manifest)
|
saveManifest(manifest)
|
||||||
print('Manifest saved. Feed creation completed.')
|
print('Manifest saved.')
|
||||||
|
saveHashes()
|
||||||
|
print('Hashes saved. Feed creation completed.')
|
||||||
|
|
|
||||||
|
|
@ -16,9 +16,9 @@ outputdir = 'output'
|
||||||
# you can use on the event index, such as organisation, tags, etc.
|
# you can use on the event index, such as organisation, tags, etc.
|
||||||
# It uses the same joining and condition rules as the API parameters
|
# It uses the same joining and condition rules as the API parameters
|
||||||
# For example:
|
# For example:
|
||||||
# filters = {'tag':'tlp:white|feed-export|!privint','org':'CIRCL'}
|
# filters = {'tag':'tlp:white|feed-export|!privint','org':'CIRCL', 'published':1}
|
||||||
# the above would generate a feed for all events created by CIRCL, tagged
|
# the above would generate a feed for all published events created by CIRCL,
|
||||||
# tlp:white and/or feed-export but exclude anything tagged privint
|
# tagged tlp:white and/or feed-export but exclude anything tagged privint
|
||||||
filters = {}
|
filters = {}
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -39,7 +39,7 @@ if __name__ == '__main__':
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
if args.output is not None and os.path.exists(args.output):
|
if args.output is not None and os.path.exists(args.output):
|
||||||
print('Output file already exists, abord.')
|
print('Output file already exists, abort.')
|
||||||
exit(0)
|
exit(0)
|
||||||
|
|
||||||
misp = init(misp_url, misp_key)
|
misp = init(misp_url, misp_key)
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
#!/usr/bin/env python
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
from pymisp import PyMISP
|
||||||
|
from keys import misp_url, misp_key, misp_verifycert
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parser = argparse.ArgumentParser(description='Get MISP stuff as CSV.')
|
||||||
|
parser.add_argument("-e", "--event_id", help="Event ID to fetch. Without it, it will fetch the whole database.")
|
||||||
|
parser.add_argument("-a", "--attribute", nargs='+', help="Attribute column names")
|
||||||
|
parser.add_argument("-o", "--object_attribute", nargs='+', help="Object attribute column names")
|
||||||
|
parser.add_argument("-t", "--misp_types", nargs='+', help="MISP types to fetch (ip-src, hostname, ...)")
|
||||||
|
parser.add_argument("-c", "--context", action='store_true', help="Add event level context (tags...)")
|
||||||
|
parser.add_argument("-i", "--ignore", action='store_true', help="Returns the attributes even if the event isn't published, or the attribute doesn't have the to_ids flag")
|
||||||
|
parser.add_argument("-f", "--outfile", help="Output file to write the CSV.")
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
pymisp = PyMISP(misp_url, misp_key, misp_verifycert, debug=True)
|
||||||
|
response = pymisp.get_csv(args.event_id, args.attribute, args.object_attribute, args.misp_types, args.context, args.ignore)
|
||||||
|
|
||||||
|
if args.outfile:
|
||||||
|
with open(args.outfile, 'w') as f:
|
||||||
|
f.write(response)
|
||||||
|
else:
|
||||||
|
print(response)
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
#!/usr/bin/env python
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
from pymisp import PyMISP
|
||||||
|
from keys import misp_url, misp_key, misp_verifycert
|
||||||
|
from pymisp.tools import load_openioc_file
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parser = argparse.ArgumentParser(description='Convert an OpenIOC file to a MISPEvent. Optionnaly send it to MISP.')
|
||||||
|
parser.add_argument("-i", "--input", required=True, help="Input file")
|
||||||
|
group = parser.add_mutually_exclusive_group(required=True)
|
||||||
|
group.add_argument("-o", "--output", help="Output file")
|
||||||
|
group.add_argument("-m", "--misp", action='store_true', help="Create new event on MISP")
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
misp_event = load_openioc_file(args.input)
|
||||||
|
|
||||||
|
if args.misp:
|
||||||
|
pymisp = PyMISP(misp_url, misp_key, misp_verifycert, debug=True)
|
||||||
|
pymisp.add_event(misp_event)
|
||||||
|
else:
|
||||||
|
with open(args.output, 'w') as f:
|
||||||
|
f.write(misp_event.to_json())
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
types_to_attach = ['ip-dst', 'url', 'domain']
|
||||||
|
objects_to_attach = ['domain-ip']
|
||||||
|
|
||||||
|
headers = """
|
||||||
|
:toc: right
|
||||||
|
:toclevels: 1
|
||||||
|
:toc-title: Daily Report
|
||||||
|
:icons: font
|
||||||
|
:sectanchors:
|
||||||
|
:sectlinks:
|
||||||
|
= Daily report by {org_name}
|
||||||
|
{date}
|
||||||
|
|
||||||
|
:icons: font
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
event_level_tags = """
|
||||||
|
IMPORTANT: This event is classified TLP:{value}.
|
||||||
|
|
||||||
|
{expanded}
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
attributes = """
|
||||||
|
=== Indicator(s) of compromise
|
||||||
|
|
||||||
|
{list_attributes}
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
title = """
|
||||||
|
== ({internal_id}) {title}
|
||||||
|
|
||||||
|
{summary}
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
types_to_attach = ['ip-dst', 'url', 'domain', 'md5']
|
||||||
|
objects_to_attach = ['domain-ip', 'file']
|
||||||
|
|
||||||
|
headers = """
|
||||||
|
:toc: right
|
||||||
|
:toclevels: 1
|
||||||
|
:toc-title: Weekly Report
|
||||||
|
:icons: font
|
||||||
|
:sectanchors:
|
||||||
|
:sectlinks:
|
||||||
|
= Weekly report by {org_name}
|
||||||
|
{date}
|
||||||
|
|
||||||
|
:icons: font
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
event_level_tags = """
|
||||||
|
"""
|
||||||
|
|
||||||
|
attributes = """
|
||||||
|
=== Indicator(s) of compromise
|
||||||
|
|
||||||
|
{list_attributes}
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
title = """
|
||||||
|
== ({internal_id}) {title}
|
||||||
|
|
||||||
|
{summary}
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
@ -1,13 +1,40 @@
|
||||||
__version__ = '2.4.81.2'
|
__version__ = '2.4.85.1'
|
||||||
|
import sys
|
||||||
|
import logging
|
||||||
|
import functools
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
FORMAT = "%(levelname)s [%(filename)s:%(lineno)s - %(funcName)s() ] %(message)s"
|
||||||
|
logging.basicConfig(stream=sys.stderr, level=logging.WARNING, format=FORMAT)
|
||||||
|
|
||||||
|
|
||||||
|
def deprecated(func):
|
||||||
|
'''This is a decorator which can be used to mark functions
|
||||||
|
as deprecated. It will result in a warning being emitted
|
||||||
|
when the function is used.'''
|
||||||
|
|
||||||
|
@functools.wraps(func)
|
||||||
|
def new_func(*args, **kwargs):
|
||||||
|
warnings.showwarning(
|
||||||
|
"Call to deprecated function {}.".format(func.__name__),
|
||||||
|
category=DeprecationWarning,
|
||||||
|
filename=func.__code__.co_filename,
|
||||||
|
lineno=func.__code__.co_firstlineno + 1
|
||||||
|
)
|
||||||
|
return func(*args, **kwargs)
|
||||||
|
return new_func
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from .exceptions import PyMISPError, NewEventError, NewAttributeError, MissingDependency, NoURL, NoKey, InvalidMISPObject, UnknownMISPObjectTemplate # noqa
|
from .exceptions import PyMISPError, NewEventError, NewAttributeError, MissingDependency, NoURL, NoKey, InvalidMISPObject, UnknownMISPObjectTemplate, PyMISPInvalidFormat # noqa
|
||||||
from .api import PyMISP # noqa
|
from .api import PyMISP # noqa
|
||||||
from .abstract import AbstractMISP, MISPEncode # noqa
|
from .abstract import AbstractMISP, MISPEncode, MISPTag # noqa
|
||||||
from .mispevent import MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, MISPObject # noqa
|
from .mispevent import MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, MISPObject, MISPUser, MISPOrganisation, MISPSighting # noqa
|
||||||
from .tools import AbstractMISPObjectGenerator # noqa
|
from .tools import AbstractMISPObjectGenerator # noqa
|
||||||
from .tools import Neo4j # noqa
|
from .tools import Neo4j # noqa
|
||||||
from .tools import stix # noqa
|
from .tools import stix # noqa
|
||||||
from .tools import openioc # noqa
|
from .tools import openioc # noqa
|
||||||
except ImportError:
|
logger.debug('pymisp loaded properly')
|
||||||
pass
|
except ImportError as e:
|
||||||
|
logger.warning('Unable to load pymisp properly: {}'.format(e))
|
||||||
|
|
|
||||||
|
|
@ -2,14 +2,36 @@
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
import abc
|
import abc
|
||||||
|
import sys
|
||||||
|
import datetime
|
||||||
import json
|
import json
|
||||||
from json import JSONEncoder
|
from json import JSONEncoder
|
||||||
import collections
|
import collections
|
||||||
import six # Remove that import when discarding python2 support.
|
import six # Remove that import when discarding python2 support.
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from .exceptions import PyMISPInvalidFormat
|
||||||
|
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
if six.PY2:
|
if six.PY2:
|
||||||
import warnings
|
logger.warning("You're using python 2, it is strongly recommended to use python >=3.5")
|
||||||
warnings.warn("You're using python 2, it is strongly recommended to use python >=3.5")
|
|
||||||
|
# This is required because Python 2 is a pain.
|
||||||
|
from datetime import tzinfo, timedelta
|
||||||
|
|
||||||
|
class UTC(tzinfo):
|
||||||
|
"""UTC"""
|
||||||
|
|
||||||
|
def utcoffset(self, dt):
|
||||||
|
return timedelta(0)
|
||||||
|
|
||||||
|
def tzname(self, dt):
|
||||||
|
return "UTC"
|
||||||
|
|
||||||
|
def dst(self, dt):
|
||||||
|
return timedelta(0)
|
||||||
|
|
||||||
|
|
||||||
class MISPEncode(JSONEncoder):
|
class MISPEncode(JSONEncoder):
|
||||||
|
|
@ -25,7 +47,24 @@ class AbstractMISP(collections.MutableMapping):
|
||||||
|
|
||||||
__not_jsonable = []
|
__not_jsonable = []
|
||||||
|
|
||||||
|
def __init__(self, **kwargs):
|
||||||
|
"""Abstract class for all the MISP objects"""
|
||||||
|
super(AbstractMISP, self).__init__()
|
||||||
|
self.__edited = True # As we create a new object, we assume it is edited
|
||||||
|
|
||||||
|
# List of classes having tags
|
||||||
|
from .mispevent import MISPAttribute, MISPEvent
|
||||||
|
self.__has_tags = (MISPAttribute, MISPEvent)
|
||||||
|
if isinstance(self, self.__has_tags):
|
||||||
|
self.Tag = []
|
||||||
|
setattr(AbstractMISP, 'add_tag', AbstractMISP.__add_tag)
|
||||||
|
setattr(AbstractMISP, 'tags', property(AbstractMISP.__get_tags, AbstractMISP.__set_tags))
|
||||||
|
|
||||||
|
@property
|
||||||
def properties(self):
|
def properties(self):
|
||||||
|
"""All the class public properties that will be dumped in the dictionary, and the JSON export.
|
||||||
|
Note: all the properties starting with a `_` (private), or listed in __not_jsonable will be skipped.
|
||||||
|
"""
|
||||||
to_return = []
|
to_return = []
|
||||||
for prop, value in vars(self).items():
|
for prop, value in vars(self).items():
|
||||||
if prop.startswith('_') or prop in self.__not_jsonable:
|
if prop.startswith('_') or prop in self.__not_jsonable:
|
||||||
|
|
@ -34,15 +73,24 @@ class AbstractMISP(collections.MutableMapping):
|
||||||
return to_return
|
return to_return
|
||||||
|
|
||||||
def from_dict(self, **kwargs):
|
def from_dict(self, **kwargs):
|
||||||
|
"""Loading all the parameters as class properties, if they aren't `None`.
|
||||||
|
This method aims to be called when all the properties requiring a special
|
||||||
|
treatment are processed.
|
||||||
|
Note: This method is used when you initialize an object with existing data so by default,
|
||||||
|
the class is flaged as not edited."""
|
||||||
for prop, value in kwargs.items():
|
for prop, value in kwargs.items():
|
||||||
if value is None:
|
if value is None:
|
||||||
continue
|
continue
|
||||||
setattr(self, prop, value)
|
setattr(self, prop, value)
|
||||||
|
# We load an existing dictionary, marking it an not-edited
|
||||||
|
self.__edited = False
|
||||||
|
|
||||||
def update_not_jsonable(self, *args):
|
def update_not_jsonable(self, *args):
|
||||||
|
"""Add entries to the __not_jsonable list"""
|
||||||
self.__not_jsonable += args
|
self.__not_jsonable += args
|
||||||
|
|
||||||
def set_not_jsonable(self, *args):
|
def set_not_jsonable(self, *args):
|
||||||
|
"""Set __not_jsonable to a new list"""
|
||||||
self.__not_jsonable = args
|
self.__not_jsonable = args
|
||||||
|
|
||||||
def from_json(self, json_string):
|
def from_json(self, json_string):
|
||||||
|
|
@ -50,22 +98,42 @@ class AbstractMISP(collections.MutableMapping):
|
||||||
self.from_dict(json.loads(json_string))
|
self.from_dict(json.loads(json_string))
|
||||||
|
|
||||||
def to_dict(self):
|
def to_dict(self):
|
||||||
|
"""Dump the lass to a dictionary.
|
||||||
|
This method automatically removes the timestamp recursively in every object
|
||||||
|
that has been edited is order to let MISP update the event accordingly."""
|
||||||
to_return = {}
|
to_return = {}
|
||||||
for attribute in self.properties():
|
for attribute in self.properties:
|
||||||
val = getattr(self, attribute, None)
|
val = getattr(self, attribute, None)
|
||||||
if val is None:
|
if val is None:
|
||||||
continue
|
continue
|
||||||
|
elif isinstance(val, list) and len(val) == 0:
|
||||||
|
continue
|
||||||
|
if attribute == 'timestamp':
|
||||||
|
if self.edited:
|
||||||
|
# In order to be accepted by MISP, the timestamp of an object
|
||||||
|
# needs to be either newer, or None.
|
||||||
|
# If the current object is marked as edited, the easiest is to
|
||||||
|
# skip the timestamp and let MISP deal with it
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
val = self._datetime_to_timestamp(val)
|
||||||
to_return[attribute] = val
|
to_return[attribute] = val
|
||||||
return to_return
|
return to_return
|
||||||
|
|
||||||
def jsonable(self):
|
def jsonable(self):
|
||||||
|
"""This method is used by the JSON encoder"""
|
||||||
return self.to_dict()
|
return self.to_dict()
|
||||||
|
|
||||||
def to_json(self):
|
def to_json(self):
|
||||||
return json.dumps(self.to_dict(), cls=MISPEncode)
|
"""Dump recursively any class of type MISPAbstract to a json string"""
|
||||||
|
return json.dumps(self, cls=MISPEncode, sort_keys=True, indent=2)
|
||||||
|
|
||||||
def __getitem__(self, key):
|
def __getitem__(self, key):
|
||||||
return getattr(self, key)
|
try:
|
||||||
|
return getattr(self, key)
|
||||||
|
except AttributeError:
|
||||||
|
# Expected by pop and other dict-related methods
|
||||||
|
raise KeyError
|
||||||
|
|
||||||
def __setitem__(self, key, value):
|
def __setitem__(self, key, value):
|
||||||
setattr(self, key, value)
|
setattr(self, key, value)
|
||||||
|
|
@ -78,3 +146,86 @@ class AbstractMISP(collections.MutableMapping):
|
||||||
|
|
||||||
def __len__(self):
|
def __len__(self):
|
||||||
return len(self.to_dict())
|
return len(self.to_dict())
|
||||||
|
|
||||||
|
@property
|
||||||
|
def edited(self):
|
||||||
|
"""Recursively check if an object has been edited and update the flag accordingly
|
||||||
|
to the parent objects"""
|
||||||
|
if self.__edited:
|
||||||
|
return self.__edited
|
||||||
|
for p in self.properties:
|
||||||
|
if self.__edited:
|
||||||
|
break
|
||||||
|
val = getattr(self, p)
|
||||||
|
if isinstance(val, AbstractMISP) and val.edited:
|
||||||
|
self.__edited = True
|
||||||
|
elif isinstance(val, list) and all(isinstance(a, AbstractMISP) for a in val):
|
||||||
|
if any(a.edited for a in val):
|
||||||
|
self.__edited = True
|
||||||
|
return self.__edited
|
||||||
|
|
||||||
|
@edited.setter
|
||||||
|
def edited(self, val):
|
||||||
|
"""Set the edit flag"""
|
||||||
|
if isinstance(val, bool):
|
||||||
|
self.__edited = val
|
||||||
|
else:
|
||||||
|
raise Exception('edited can only be True or False')
|
||||||
|
|
||||||
|
def __setattr__(self, name, value):
|
||||||
|
if name in self.properties:
|
||||||
|
self.__edited = True
|
||||||
|
super(AbstractMISP, self).__setattr__(name, value)
|
||||||
|
|
||||||
|
def _datetime_to_timestamp(self, d):
|
||||||
|
"""Convert a datetime.datetime object to a timestamp (int)"""
|
||||||
|
if isinstance(d, (int, str)) or (sys.version_info < (3, 0) and isinstance(d, unicode)):
|
||||||
|
# Assume we already have a timestamp
|
||||||
|
return d
|
||||||
|
if sys.version_info >= (3, 3):
|
||||||
|
return int(d.timestamp())
|
||||||
|
else:
|
||||||
|
return int((d - datetime.datetime.fromtimestamp(0, UTC())).total_seconds())
|
||||||
|
|
||||||
|
def __add_tag(self, tag=None, **kwargs):
|
||||||
|
"""Add a tag to the attribute (by name or a MISPTag object)"""
|
||||||
|
if isinstance(tag, str):
|
||||||
|
misp_tag = MISPTag()
|
||||||
|
misp_tag.from_dict(name=tag)
|
||||||
|
elif isinstance(tag, MISPTag):
|
||||||
|
misp_tag = tag
|
||||||
|
elif isinstance(tag, dict):
|
||||||
|
misp_tag = MISPTag()
|
||||||
|
misp_tag.from_dict(**tag)
|
||||||
|
elif kwargs:
|
||||||
|
misp_tag = MISPTag()
|
||||||
|
misp_tag.from_dict(**kwargs)
|
||||||
|
else:
|
||||||
|
raise PyMISPInvalidFormat("The tag is in an invalid format (can be either string, MISPTag, or an expanded dict): {}".format(tag))
|
||||||
|
self.Tag.append(misp_tag)
|
||||||
|
self.edited = True
|
||||||
|
|
||||||
|
def __get_tags(self):
|
||||||
|
"""Returns a lost of tags associated to this Attribute"""
|
||||||
|
return self.Tag
|
||||||
|
|
||||||
|
def __set_tags(self, tags):
|
||||||
|
"""Set a list of prepared MISPTag."""
|
||||||
|
if all(isinstance(x, MISPTag) for x in tags):
|
||||||
|
self.Tag = tags
|
||||||
|
else:
|
||||||
|
raise PyMISPInvalidFormat('All the attributes have to be of type MISPTag.')
|
||||||
|
|
||||||
|
|
||||||
|
class MISPTag(AbstractMISP):
|
||||||
|
def __init__(self):
|
||||||
|
super(MISPTag, self).__init__()
|
||||||
|
|
||||||
|
def from_dict(self, name, **kwargs):
|
||||||
|
self.name = name
|
||||||
|
super(MISPTag, self).from_dict(**kwargs)
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
if hasattr(self, 'name'):
|
||||||
|
return '<{self.__class__.__name__}(name={self.name})'.format(self=self)
|
||||||
|
return '<{self.__class__.__name__}(NotInitialized)'.format(self=self)
|
||||||
|
|
|
||||||
813
pymisp/api.py
813
pymisp/api.py
File diff suppressed because it is too large
Load Diff
|
|
@ -121,6 +121,10 @@
|
||||||
"default_category": "Payload installation",
|
"default_category": "Payload installation",
|
||||||
"to_ids": 1
|
"to_ids": 1
|
||||||
},
|
},
|
||||||
|
"stix2-pattern": {
|
||||||
|
"default_category": "Payload installation",
|
||||||
|
"to_ids": 1
|
||||||
|
},
|
||||||
"sigma": {
|
"sigma": {
|
||||||
"default_category": "Payload installation",
|
"default_category": "Payload installation",
|
||||||
"to_ids": 1
|
"to_ids": 1
|
||||||
|
|
@ -361,6 +365,10 @@
|
||||||
"default_category": "Attribution",
|
"default_category": "Attribution",
|
||||||
"to_ids": 0
|
"to_ids": 0
|
||||||
},
|
},
|
||||||
|
"whois-registrant-org": {
|
||||||
|
"default_category": "Attribution",
|
||||||
|
"to_ids": 0
|
||||||
|
},
|
||||||
"whois-registrar": {
|
"whois-registrar": {
|
||||||
"default_category": "Attribution",
|
"default_category": "Attribution",
|
||||||
"to_ids": 0
|
"to_ids": 0
|
||||||
|
|
@ -373,6 +381,14 @@
|
||||||
"default_category": "Network activity",
|
"default_category": "Network activity",
|
||||||
"to_ids": 1
|
"to_ids": 1
|
||||||
},
|
},
|
||||||
|
"x509-fingerprint-md5": {
|
||||||
|
"default_category": "Network activity",
|
||||||
|
"to_ids": 1
|
||||||
|
},
|
||||||
|
"x509-fingerprint-sha256": {
|
||||||
|
"default_category": "Network activity",
|
||||||
|
"to_ids": 1
|
||||||
|
},
|
||||||
"dns-soa-email": {
|
"dns-soa-email": {
|
||||||
"default_category": "Attribution",
|
"default_category": "Attribution",
|
||||||
"to_ids": 0
|
"to_ids": 0
|
||||||
|
|
@ -409,6 +425,14 @@
|
||||||
"default_category": "Network activity",
|
"default_category": "Network activity",
|
||||||
"to_ids": 1
|
"to_ids": 1
|
||||||
},
|
},
|
||||||
|
"mac-address": {
|
||||||
|
"default_category": "Network activity",
|
||||||
|
"to_ids": 0
|
||||||
|
},
|
||||||
|
"mac-eui-64": {
|
||||||
|
"default_category": "Network activity",
|
||||||
|
"to_ids": 0
|
||||||
|
},
|
||||||
"email-dst-display-name": {
|
"email-dst-display-name": {
|
||||||
"default_category": "Payload delivery",
|
"default_category": "Payload delivery",
|
||||||
"to_ids": 0
|
"to_ids": 0
|
||||||
|
|
@ -482,7 +506,7 @@
|
||||||
"to_ids": 0
|
"to_ids": 0
|
||||||
},
|
},
|
||||||
"gender": {
|
"gender": {
|
||||||
"default_category": "",
|
"default_category": "Person",
|
||||||
"to_ids": 0
|
"to_ids": 0
|
||||||
},
|
},
|
||||||
"passport-number": {
|
"passport-number": {
|
||||||
|
|
@ -593,6 +617,7 @@
|
||||||
"pattern-in-traffic",
|
"pattern-in-traffic",
|
||||||
"pattern-in-memory",
|
"pattern-in-memory",
|
||||||
"yara",
|
"yara",
|
||||||
|
"stix2-pattern",
|
||||||
"sigma",
|
"sigma",
|
||||||
"cookie",
|
"cookie",
|
||||||
"vulnerability",
|
"vulnerability",
|
||||||
|
|
@ -653,9 +678,12 @@
|
||||||
"whois-registrant-email",
|
"whois-registrant-email",
|
||||||
"whois-registrant-phone",
|
"whois-registrant-phone",
|
||||||
"whois-registrant-name",
|
"whois-registrant-name",
|
||||||
|
"whois-registrant-org",
|
||||||
"whois-registrar",
|
"whois-registrar",
|
||||||
"whois-creation-date",
|
"whois-creation-date",
|
||||||
"x509-fingerprint-sha1",
|
"x509-fingerprint-sha1",
|
||||||
|
"x509-fingerprint-md5",
|
||||||
|
"x509-fingerprint-sha256",
|
||||||
"dns-soa-email",
|
"dns-soa-email",
|
||||||
"size-in-bytes",
|
"size-in-bytes",
|
||||||
"counter",
|
"counter",
|
||||||
|
|
@ -665,6 +693,8 @@
|
||||||
"ip-dst|port",
|
"ip-dst|port",
|
||||||
"ip-src|port",
|
"ip-src|port",
|
||||||
"hostname|port",
|
"hostname|port",
|
||||||
|
"mac-address",
|
||||||
|
"mac-eui-64",
|
||||||
"email-dst-display-name",
|
"email-dst-display-name",
|
||||||
"email-src-display-name",
|
"email-src-display-name",
|
||||||
"email-header",
|
"email-header",
|
||||||
|
|
@ -777,6 +807,8 @@
|
||||||
"filename|imphash",
|
"filename|imphash",
|
||||||
"filename|impfuzzy",
|
"filename|impfuzzy",
|
||||||
"filename|pehash",
|
"filename|pehash",
|
||||||
|
"mac-address",
|
||||||
|
"mac-eui-64",
|
||||||
"ip-src",
|
"ip-src",
|
||||||
"ip-dst",
|
"ip-dst",
|
||||||
"ip-dst|port",
|
"ip-dst|port",
|
||||||
|
|
@ -793,6 +825,7 @@
|
||||||
"AS",
|
"AS",
|
||||||
"pattern-in-file",
|
"pattern-in-file",
|
||||||
"pattern-in-traffic",
|
"pattern-in-traffic",
|
||||||
|
"stix2-pattern",
|
||||||
"yara",
|
"yara",
|
||||||
"sigma",
|
"sigma",
|
||||||
"attachment",
|
"attachment",
|
||||||
|
|
@ -804,6 +837,8 @@
|
||||||
"hex",
|
"hex",
|
||||||
"vulnerability",
|
"vulnerability",
|
||||||
"x509-fingerprint-sha1",
|
"x509-fingerprint-sha1",
|
||||||
|
"x509-fingerprint-md5",
|
||||||
|
"x509-fingerprint-sha256",
|
||||||
"other",
|
"other",
|
||||||
"hostname|port",
|
"hostname|port",
|
||||||
"email-dst-display-name",
|
"email-dst-display-name",
|
||||||
|
|
@ -850,6 +885,7 @@
|
||||||
"pattern-in-file",
|
"pattern-in-file",
|
||||||
"pattern-in-memory",
|
"pattern-in-memory",
|
||||||
"pdb",
|
"pdb",
|
||||||
|
"stix2-pattern",
|
||||||
"yara",
|
"yara",
|
||||||
"sigma",
|
"sigma",
|
||||||
"attachment",
|
"attachment",
|
||||||
|
|
@ -863,6 +899,8 @@
|
||||||
"text",
|
"text",
|
||||||
"hex",
|
"hex",
|
||||||
"x509-fingerprint-sha1",
|
"x509-fingerprint-sha1",
|
||||||
|
"x509-fingerprint-md5",
|
||||||
|
"x509-fingerprint-sha256",
|
||||||
"other",
|
"other",
|
||||||
"cookie"
|
"cookie"
|
||||||
],
|
],
|
||||||
|
|
@ -899,6 +937,7 @@
|
||||||
"pattern-in-file",
|
"pattern-in-file",
|
||||||
"pattern-in-traffic",
|
"pattern-in-traffic",
|
||||||
"pattern-in-memory",
|
"pattern-in-memory",
|
||||||
|
"stix2-pattern",
|
||||||
"yara",
|
"yara",
|
||||||
"sigma",
|
"sigma",
|
||||||
"vulnerability",
|
"vulnerability",
|
||||||
|
|
@ -909,6 +948,8 @@
|
||||||
"text",
|
"text",
|
||||||
"hex",
|
"hex",
|
||||||
"x509-fingerprint-sha1",
|
"x509-fingerprint-sha1",
|
||||||
|
"x509-fingerprint-md5",
|
||||||
|
"x509-fingerprint-sha256",
|
||||||
"mobile-application-id",
|
"mobile-application-id",
|
||||||
"other"
|
"other"
|
||||||
],
|
],
|
||||||
|
|
@ -930,6 +971,8 @@
|
||||||
"hostname",
|
"hostname",
|
||||||
"domain",
|
"domain",
|
||||||
"domain|ip",
|
"domain|ip",
|
||||||
|
"mac-address",
|
||||||
|
"mac-eui-64",
|
||||||
"email-dst",
|
"email-dst",
|
||||||
"url",
|
"url",
|
||||||
"uri",
|
"uri",
|
||||||
|
|
@ -938,6 +981,7 @@
|
||||||
"AS",
|
"AS",
|
||||||
"snort",
|
"snort",
|
||||||
"pattern-in-file",
|
"pattern-in-file",
|
||||||
|
"stix2-pattern",
|
||||||
"pattern-in-traffic",
|
"pattern-in-traffic",
|
||||||
"attachment",
|
"attachment",
|
||||||
"comment",
|
"comment",
|
||||||
|
|
@ -959,12 +1003,16 @@
|
||||||
"whois-registrant-phone",
|
"whois-registrant-phone",
|
||||||
"whois-registrant-email",
|
"whois-registrant-email",
|
||||||
"whois-registrant-name",
|
"whois-registrant-name",
|
||||||
|
"whois-registrant-org",
|
||||||
"whois-registrar",
|
"whois-registrar",
|
||||||
"whois-creation-date",
|
"whois-creation-date",
|
||||||
"comment",
|
"comment",
|
||||||
"text",
|
"text",
|
||||||
"x509-fingerprint-sha1",
|
"x509-fingerprint-sha1",
|
||||||
"other"
|
"x509-fingerprint-md5",
|
||||||
|
"x509-fingerprint-sha256",
|
||||||
|
"other",
|
||||||
|
"dns-soa-email"
|
||||||
],
|
],
|
||||||
"External analysis": [
|
"External analysis": [
|
||||||
"md5",
|
"md5",
|
||||||
|
|
@ -978,6 +1026,8 @@
|
||||||
"ip-dst",
|
"ip-dst",
|
||||||
"ip-dst|port",
|
"ip-dst|port",
|
||||||
"ip-src|port",
|
"ip-src|port",
|
||||||
|
"mac-address",
|
||||||
|
"mac-eui-64",
|
||||||
"hostname",
|
"hostname",
|
||||||
"domain",
|
"domain",
|
||||||
"domain|ip",
|
"domain|ip",
|
||||||
|
|
@ -997,6 +1047,8 @@
|
||||||
"comment",
|
"comment",
|
||||||
"text",
|
"text",
|
||||||
"x509-fingerprint-sha1",
|
"x509-fingerprint-sha1",
|
||||||
|
"x509-fingerprint-md5",
|
||||||
|
"x509-fingerprint-sha256",
|
||||||
"github-repository",
|
"github-repository",
|
||||||
"other",
|
"other",
|
||||||
"cortex"
|
"cortex"
|
||||||
|
|
|
||||||
|
|
@ -1 +1 @@
|
||||||
Subproject commit bbf3e45649af5af50c98ad90a86916cf75e8c74d
|
Subproject commit 21e58b3ddf1737028b556b93b20d848f86a71cd0
|
||||||
|
|
@ -1,4 +1,3 @@
|
||||||
|
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -40,6 +39,11 @@ class InvalidMISPObject(MISPObjectException):
|
||||||
"""Exception raised when an object doesn't respect the contrains in the definition"""
|
"""Exception raised when an object doesn't respect the contrains in the definition"""
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
class UnknownMISPObjectTemplate(MISPObjectException):
|
class UnknownMISPObjectTemplate(MISPObjectException):
|
||||||
"""Exception raised when the template is unknown"""
|
"""Exception raised when the template is unknown"""
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class PyMISPInvalidFormat(PyMISPError):
|
||||||
|
pass
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -6,3 +6,5 @@ from .elfobject import ELFObject, ELFSectionObject # noqa
|
||||||
from .machoobject import MachOObject, MachOSectionObject # noqa
|
from .machoobject import MachOObject, MachOSectionObject # noqa
|
||||||
from .create_misp_object import make_binary_objects # noqa
|
from .create_misp_object import make_binary_objects # noqa
|
||||||
from .abstractgenerator import AbstractMISPObjectGenerator # noqa
|
from .abstractgenerator import AbstractMISPObjectGenerator # noqa
|
||||||
|
from .genericgenerator import GenericObjectGenerator # noqa
|
||||||
|
from .openioc import load_openioc, load_openioc_file # noqa
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,13 @@
|
||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import six
|
||||||
|
|
||||||
from . import FileObject, PEObject, ELFObject, MachOObject
|
from . import FileObject, PEObject, ELFObject, MachOObject
|
||||||
from ..exceptions import MISPObjectException
|
from ..exceptions import MISPObjectException
|
||||||
import warnings
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import lief
|
import lief
|
||||||
|
|
@ -18,8 +22,8 @@ class FileTypeNotImplemented(MISPObjectException):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
def make_pe_objects(lief_parsed, misp_file):
|
def make_pe_objects(lief_parsed, misp_file, standalone=True, default_attributes_parameters={}):
|
||||||
pe_object = PEObject(parsed=lief_parsed)
|
pe_object = PEObject(parsed=lief_parsed, standalone=standalone, default_attributes_parameters=default_attributes_parameters)
|
||||||
misp_file.add_reference(pe_object.uuid, 'included-in', 'PE indicators')
|
misp_file.add_reference(pe_object.uuid, 'included-in', 'PE indicators')
|
||||||
pe_sections = []
|
pe_sections = []
|
||||||
for s in pe_object.sections:
|
for s in pe_object.sections:
|
||||||
|
|
@ -27,8 +31,8 @@ def make_pe_objects(lief_parsed, misp_file):
|
||||||
return misp_file, pe_object, pe_sections
|
return misp_file, pe_object, pe_sections
|
||||||
|
|
||||||
|
|
||||||
def make_elf_objects(lief_parsed, misp_file):
|
def make_elf_objects(lief_parsed, misp_file, standalone=True, default_attributes_parameters={}):
|
||||||
elf_object = ELFObject(parsed=lief_parsed)
|
elf_object = ELFObject(parsed=lief_parsed, standalone=standalone, default_attributes_parameters=default_attributes_parameters)
|
||||||
misp_file.add_reference(elf_object.uuid, 'included-in', 'ELF indicators')
|
misp_file.add_reference(elf_object.uuid, 'included-in', 'ELF indicators')
|
||||||
elf_sections = []
|
elf_sections = []
|
||||||
for s in elf_object.sections:
|
for s in elf_object.sections:
|
||||||
|
|
@ -36,8 +40,8 @@ def make_elf_objects(lief_parsed, misp_file):
|
||||||
return misp_file, elf_object, elf_sections
|
return misp_file, elf_object, elf_sections
|
||||||
|
|
||||||
|
|
||||||
def make_macho_objects(lief_parsed, misp_file):
|
def make_macho_objects(lief_parsed, misp_file, standalone=True, default_attributes_parameters={}):
|
||||||
macho_object = MachOObject(parsed=lief_parsed)
|
macho_object = MachOObject(parsed=lief_parsed, standalone=standalone, default_attributes_parameters=default_attributes_parameters)
|
||||||
misp_file.add_reference(macho_object.uuid, 'included-in', 'MachO indicators')
|
misp_file.add_reference(macho_object.uuid, 'included-in', 'MachO indicators')
|
||||||
macho_sections = []
|
macho_sections = []
|
||||||
for s in macho_object.sections:
|
for s in macho_object.sections:
|
||||||
|
|
@ -45,27 +49,45 @@ def make_macho_objects(lief_parsed, misp_file):
|
||||||
return misp_file, macho_object, macho_sections
|
return misp_file, macho_object, macho_sections
|
||||||
|
|
||||||
|
|
||||||
def make_binary_objects(filepath=None, pseudofile=None, filename=None):
|
def make_binary_objects(filepath=None, pseudofile=None, filename=None, standalone=True, default_attributes_parameters={}):
|
||||||
misp_file = FileObject(filepath=filepath, pseudofile=pseudofile, filename=filename)
|
misp_file = FileObject(filepath=filepath, pseudofile=pseudofile, filename=filename,
|
||||||
if HAS_LIEF and filepath:
|
standalone=standalone, default_attributes_parameters=default_attributes_parameters)
|
||||||
|
if HAS_LIEF and filepath or (pseudofile and filename):
|
||||||
try:
|
try:
|
||||||
lief_parsed = lief.parse(filepath)
|
if filepath:
|
||||||
|
lief_parsed = lief.parse(filepath=filepath)
|
||||||
|
else:
|
||||||
|
if six.PY2:
|
||||||
|
logger.critical('Pseudofile is not supported in python2. Just update.')
|
||||||
|
lief_parsed = None
|
||||||
|
else:
|
||||||
|
lief_parsed = lief.parse(raw=pseudofile.getvalue(), name=filename)
|
||||||
if isinstance(lief_parsed, lief.PE.Binary):
|
if isinstance(lief_parsed, lief.PE.Binary):
|
||||||
return make_pe_objects(lief_parsed, misp_file)
|
return make_pe_objects(lief_parsed, misp_file, standalone, default_attributes_parameters)
|
||||||
elif isinstance(lief_parsed, lief.ELF.Binary):
|
elif isinstance(lief_parsed, lief.ELF.Binary):
|
||||||
return make_elf_objects(lief_parsed, misp_file)
|
return make_elf_objects(lief_parsed, misp_file, standalone, default_attributes_parameters)
|
||||||
elif isinstance(lief_parsed, lief.MachO.Binary):
|
elif isinstance(lief_parsed, lief.MachO.Binary):
|
||||||
return make_macho_objects(lief_parsed, misp_file)
|
return make_macho_objects(lief_parsed, misp_file, standalone, default_attributes_parameters)
|
||||||
except lief.bad_format as e:
|
except lief.bad_format as e:
|
||||||
warnings.warn('\tBad format: {}'.format(e))
|
logger.warning('Bad format: {}'.format(e))
|
||||||
except lief.bad_file as e:
|
except lief.bad_file as e:
|
||||||
warnings.warn('\tBad file: {}'.format(e))
|
logger.warning('Bad file: {}'.format(e))
|
||||||
|
except lief.conversion_error as e:
|
||||||
|
logger.warning('Conversion file: {}'.format(e))
|
||||||
|
except lief.builder_error as e:
|
||||||
|
logger.warning('Builder file: {}'.format(e))
|
||||||
except lief.parser_error as e:
|
except lief.parser_error as e:
|
||||||
warnings.warn('\tParser error: {}'.format(e))
|
logger.warning('Parser error: {}'.format(e))
|
||||||
except FileTypeNotImplemented as e: # noqa
|
except lief.integrity_error as e:
|
||||||
warnings.warn(e)
|
logger.warning('Integrity error: {}'.format(e))
|
||||||
|
except lief.pe_error as e:
|
||||||
|
logger.warning('PE error: {}'.format(e))
|
||||||
|
except lief.type_error as e:
|
||||||
|
logger.warning('Type error: {}'.format(e))
|
||||||
|
except lief.exception as e:
|
||||||
|
logger.warning('Lief exception: {}'.format(e))
|
||||||
|
except FileTypeNotImplemented as e:
|
||||||
|
logger.warning(e)
|
||||||
if not HAS_LIEF:
|
if not HAS_LIEF:
|
||||||
warnings.warn('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
logger.warning('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
||||||
if not filepath:
|
|
||||||
warnings.warn('LIEF currently requires a filepath and not a pseudo file')
|
|
||||||
return misp_file, None, None
|
return misp_file, None, None
|
||||||
|
|
|
||||||
|
|
@ -5,8 +5,9 @@ from .abstractgenerator import AbstractMISPObjectGenerator
|
||||||
from ..exceptions import InvalidMISPObject
|
from ..exceptions import InvalidMISPObject
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from hashlib import md5, sha1, sha256, sha512
|
from hashlib import md5, sha1, sha256, sha512
|
||||||
import warnings
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import lief
|
import lief
|
||||||
|
|
@ -23,9 +24,9 @@ except ImportError:
|
||||||
|
|
||||||
class ELFObject(AbstractMISPObjectGenerator):
|
class ELFObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, parsed=None, filepath=None, pseudofile=None):
|
def __init__(self, parsed=None, filepath=None, pseudofile=None, standalone=True, **kwargs):
|
||||||
if not HAS_PYDEEP:
|
if not HAS_PYDEEP:
|
||||||
warnings.warn("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
logger.warning("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
||||||
if not HAS_LIEF:
|
if not HAS_LIEF:
|
||||||
raise ImportError('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
raise ImportError('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
||||||
if pseudofile:
|
if pseudofile:
|
||||||
|
|
@ -43,10 +44,8 @@ class ELFObject(AbstractMISPObjectGenerator):
|
||||||
self.__elf = parsed
|
self.__elf = parsed
|
||||||
else:
|
else:
|
||||||
raise InvalidMISPObject('Not a lief.ELF.Binary: {}'.format(type(parsed)))
|
raise InvalidMISPObject('Not a lief.ELF.Binary: {}'.format(type(parsed)))
|
||||||
super(ELFObject, self).__init__('elf')
|
super(ELFObject, self).__init__('elf', standalone=standalone, **kwargs)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable('ObjectReference')
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
# General information
|
# General information
|
||||||
|
|
@ -59,7 +58,7 @@ class ELFObject(AbstractMISPObjectGenerator):
|
||||||
if self.__elf.sections:
|
if self.__elf.sections:
|
||||||
pos = 0
|
pos = 0
|
||||||
for section in self.__elf.sections:
|
for section in self.__elf.sections:
|
||||||
s = ELFSectionObject(section)
|
s = ELFSectionObject(section, self._standalone, default_attributes_parameters=self._default_attributes_parameters)
|
||||||
self.add_reference(s.uuid, 'included-in', 'Section {} of ELF'.format(pos))
|
self.add_reference(s.uuid, 'included-in', 'Section {} of ELF'.format(pos))
|
||||||
pos += 1
|
pos += 1
|
||||||
self.sections.append(s)
|
self.sections.append(s)
|
||||||
|
|
@ -68,15 +67,13 @@ class ELFObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
class ELFSectionObject(AbstractMISPObjectGenerator):
|
class ELFSectionObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, section):
|
def __init__(self, section, standalone=True, **kwargs):
|
||||||
# Python3 way
|
# Python3 way
|
||||||
# super().__init__('pe-section')
|
# super().__init__('pe-section')
|
||||||
super(ELFSectionObject, self).__init__('elf-section')
|
super(ELFSectionObject, self).__init__('elf-section', standalone=standalone, **kwargs)
|
||||||
self.__section = section
|
self.__section = section
|
||||||
self.__data = bytes(self.__section.content)
|
self.__data = bytes(self.__section.content)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable('ObjectReference')
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
self.add_attribute('name', value=self.__section.name)
|
self.add_attribute('name', value=self.__section.name)
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,10 @@ from io import BytesIO
|
||||||
from hashlib import md5, sha1, sha256, sha512
|
from hashlib import md5, sha1, sha256, sha512
|
||||||
import math
|
import math
|
||||||
from collections import Counter
|
from collections import Counter
|
||||||
import warnings
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import pydeep
|
import pydeep
|
||||||
|
|
@ -25,11 +28,11 @@ except ImportError:
|
||||||
|
|
||||||
class FileObject(AbstractMISPObjectGenerator):
|
class FileObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, filepath=None, pseudofile=None, filename=None):
|
def __init__(self, filepath=None, pseudofile=None, filename=None, standalone=True, **kwargs):
|
||||||
if not HAS_PYDEEP:
|
if not HAS_PYDEEP:
|
||||||
warnings.warn("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
logger.warning("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
||||||
if not HAS_MAGIC:
|
if not HAS_MAGIC:
|
||||||
warnings.warn("Please install python-magic: pip install python-magic.")
|
logger.warning("Please install python-magic: pip install python-magic.")
|
||||||
if filename:
|
if filename:
|
||||||
# Useful in case the file is copied with a pre-defined name by a script but we want to keep the original name
|
# Useful in case the file is copied with a pre-defined name by a script but we want to keep the original name
|
||||||
self.__filename = filename
|
self.__filename = filename
|
||||||
|
|
@ -48,11 +51,9 @@ class FileObject(AbstractMISPObjectGenerator):
|
||||||
raise InvalidMISPObject('File buffer (BytesIO) or a path is required.')
|
raise InvalidMISPObject('File buffer (BytesIO) or a path is required.')
|
||||||
# PY3 way:
|
# PY3 way:
|
||||||
# super().__init__('file')
|
# super().__init__('file')
|
||||||
super(FileObject, self).__init__('file')
|
super(FileObject, self).__init__('file', standalone=standalone, **kwargs)
|
||||||
self.__data = self.__pseudofile.getvalue()
|
self.__data = self.__pseudofile.getvalue()
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable('ObjectReference')
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
self.add_attribute('filename', value=self.__filename)
|
self.add_attribute('filename', value=self.__filename)
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
#!/usr/bin/env python
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
from .abstractgenerator import AbstractMISPObjectGenerator
|
||||||
|
|
||||||
|
|
||||||
|
class GenericObjectGenerator(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
|
def generate_attributes(self, attributes):
|
||||||
|
for attribute in attributes:
|
||||||
|
for object_relation, value in attribute.items():
|
||||||
|
if isinstance(value, dict):
|
||||||
|
self.add_attribute(object_relation, **value)
|
||||||
|
else:
|
||||||
|
# In this case, we need a valid template, as all the other parameters will be pre-set.
|
||||||
|
self.add_attribute(object_relation, value=value)
|
||||||
|
|
@ -5,7 +5,9 @@ from ..exceptions import InvalidMISPObject
|
||||||
from .abstractgenerator import AbstractMISPObjectGenerator
|
from .abstractgenerator import AbstractMISPObjectGenerator
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from hashlib import md5, sha1, sha256, sha512
|
from hashlib import md5, sha1, sha256, sha512
|
||||||
import warnings
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
|
@ -23,9 +25,9 @@ except ImportError:
|
||||||
|
|
||||||
class MachOObject(AbstractMISPObjectGenerator):
|
class MachOObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, parsed=None, filepath=None, pseudofile=None):
|
def __init__(self, parsed=None, filepath=None, pseudofile=None, standalone=True, **kwargs):
|
||||||
if not HAS_PYDEEP:
|
if not HAS_PYDEEP:
|
||||||
warnings.warn("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
logger.warning("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
||||||
if not HAS_LIEF:
|
if not HAS_LIEF:
|
||||||
raise ImportError('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
raise ImportError('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
||||||
if pseudofile:
|
if pseudofile:
|
||||||
|
|
@ -45,10 +47,8 @@ class MachOObject(AbstractMISPObjectGenerator):
|
||||||
raise InvalidMISPObject('Not a lief.MachO.Binary: {}'.format(type(parsed)))
|
raise InvalidMISPObject('Not a lief.MachO.Binary: {}'.format(type(parsed)))
|
||||||
# Python3 way
|
# Python3 way
|
||||||
# super().__init__('elf')
|
# super().__init__('elf')
|
||||||
super(MachOObject, self).__init__('macho')
|
super(MachOObject, self).__init__('macho', standalone=standalone, **kwargs)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable(['ObjectReference'])
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
self.add_attribute('type', value=str(self.__macho.header.file_type).split('.')[1])
|
self.add_attribute('type', value=str(self.__macho.header.file_type).split('.')[1])
|
||||||
|
|
@ -61,7 +61,7 @@ class MachOObject(AbstractMISPObjectGenerator):
|
||||||
if self.__macho.sections:
|
if self.__macho.sections:
|
||||||
pos = 0
|
pos = 0
|
||||||
for section in self.__macho.sections:
|
for section in self.__macho.sections:
|
||||||
s = MachOSectionObject(section)
|
s = MachOSectionObject(section, self._standalone, default_attributes_parameters=self._default_attributes_parameters)
|
||||||
self.add_reference(s.uuid, 'included-in', 'Section {} of MachO'.format(pos))
|
self.add_reference(s.uuid, 'included-in', 'Section {} of MachO'.format(pos))
|
||||||
pos += 1
|
pos += 1
|
||||||
self.sections.append(s)
|
self.sections.append(s)
|
||||||
|
|
@ -70,15 +70,13 @@ class MachOObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
class MachOSectionObject(AbstractMISPObjectGenerator):
|
class MachOSectionObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, section):
|
def __init__(self, section, standalone=True, **kwargs):
|
||||||
# Python3 way
|
# Python3 way
|
||||||
# super().__init__('pe-section')
|
# super().__init__('pe-section')
|
||||||
super(MachOSectionObject, self).__init__('macho-section')
|
super(MachOSectionObject, self).__init__('macho-section', standalone=standalone, **kwargs)
|
||||||
self.__section = section
|
self.__section = section
|
||||||
self.__data = bytes(self.__section.content)
|
self.__data = bytes(self.__section.content)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable(['ObjectReference'])
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
self.add_attribute('name', value=self.__section.name)
|
self.add_attribute('name', value=self.__section.name)
|
||||||
|
|
|
||||||
|
|
@ -278,17 +278,3 @@ def set_all_attributes(openioc, misp_event):
|
||||||
misp_event.add_attribute(**attribute_values)
|
misp_event.add_attribute(**attribute_values)
|
||||||
|
|
||||||
return misp_event
|
return misp_event
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
import requests
|
|
||||||
# test file for composite
|
|
||||||
url = 'https://raw.githubusercontent.com/fireeye/iocs/master/BlogPosts/9cee306d-5441-4cd3-932d-f3119752634c.ioc'
|
|
||||||
# ~ url = 'https://raw.githubusercontent.com/MISP/misp-modules/master/tests/openioc.xml'
|
|
||||||
x = requests.get(url)
|
|
||||||
mispEvent = load_openioc(x.text)
|
|
||||||
print(mispEvent)
|
|
||||||
# ~ from pymisp import PyMISP
|
|
||||||
# ~ misp = PyMISP('http://misp.local', 'xxxxx')
|
|
||||||
# ~ r = misp.add_event(mispEvent)
|
|
||||||
# ~ print(r)
|
|
||||||
|
|
|
||||||
|
|
@ -6,8 +6,9 @@ from .abstractgenerator import AbstractMISPObjectGenerator
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from hashlib import md5, sha1, sha256, sha512
|
from hashlib import md5, sha1, sha256, sha512
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
import warnings
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger('pymisp')
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import lief
|
import lief
|
||||||
|
|
@ -24,9 +25,9 @@ except ImportError:
|
||||||
|
|
||||||
class PEObject(AbstractMISPObjectGenerator):
|
class PEObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, parsed=None, filepath=None, pseudofile=None):
|
def __init__(self, parsed=None, filepath=None, pseudofile=None, standalone=True, **kwargs):
|
||||||
if not HAS_PYDEEP:
|
if not HAS_PYDEEP:
|
||||||
warnings.warn("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
logger.warning("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
|
||||||
if not HAS_LIEF:
|
if not HAS_LIEF:
|
||||||
raise ImportError('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
raise ImportError('Please install lief, documentation here: https://github.com/lief-project/LIEF')
|
||||||
if pseudofile:
|
if pseudofile:
|
||||||
|
|
@ -46,10 +47,8 @@ class PEObject(AbstractMISPObjectGenerator):
|
||||||
raise InvalidMISPObject('Not a lief.PE.Binary: {}'.format(type(parsed)))
|
raise InvalidMISPObject('Not a lief.PE.Binary: {}'.format(type(parsed)))
|
||||||
# Python3 way
|
# Python3 way
|
||||||
# super().__init__('pe')
|
# super().__init__('pe')
|
||||||
super(PEObject, self).__init__('pe')
|
super(PEObject, self).__init__('pe', standalone=standalone, **kwargs)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable('ObjectReference')
|
|
||||||
|
|
||||||
def _is_exe(self):
|
def _is_exe(self):
|
||||||
if not self._is_dll() and not self._is_driver():
|
if not self._is_dll() and not self._is_driver():
|
||||||
|
|
@ -105,7 +104,7 @@ class PEObject(AbstractMISPObjectGenerator):
|
||||||
if self.__pe.sections:
|
if self.__pe.sections:
|
||||||
pos = 0
|
pos = 0
|
||||||
for section in self.__pe.sections:
|
for section in self.__pe.sections:
|
||||||
s = PESectionObject(section)
|
s = PESectionObject(section, self._standalone, default_attributes_parameters=self._default_attributes_parameters)
|
||||||
self.add_reference(s.uuid, 'included-in', 'Section {} of PE'.format(pos))
|
self.add_reference(s.uuid, 'included-in', 'Section {} of PE'.format(pos))
|
||||||
if ((self.__pe.entrypoint >= section.virtual_address) and
|
if ((self.__pe.entrypoint >= section.virtual_address) and
|
||||||
(self.__pe.entrypoint < (section.virtual_address + section.virtual_size))):
|
(self.__pe.entrypoint < (section.virtual_address + section.virtual_size))):
|
||||||
|
|
@ -118,15 +117,13 @@ class PEObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
class PESectionObject(AbstractMISPObjectGenerator):
|
class PESectionObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
def __init__(self, section):
|
def __init__(self, section, standalone=True, **kwargs):
|
||||||
# Python3 way
|
# Python3 way
|
||||||
# super().__init__('pe-section')
|
# super().__init__('pe-section')
|
||||||
super(PESectionObject, self).__init__('pe-section')
|
super(PESectionObject, self).__init__('pe-section', standalone=standalone, **kwargs)
|
||||||
self.__section = section
|
self.__section = section
|
||||||
self.__data = bytes(self.__section.content)
|
self.__data = bytes(self.__section.content)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
# Mark as non_jsonable because we need to add them manually
|
|
||||||
self.update_not_jsonable('ObjectReference')
|
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
self.add_attribute('name', value=self.__section.name)
|
self.add_attribute('name', value=self.__section.name)
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,12 @@
|
||||||
import re
|
import re
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
import validators
|
try:
|
||||||
|
import validators
|
||||||
|
has_validators = True
|
||||||
|
except ImportError:
|
||||||
|
has_validators = False
|
||||||
|
|
||||||
|
|
||||||
from .abstractgenerator import AbstractMISPObjectGenerator
|
from .abstractgenerator import AbstractMISPObjectGenerator
|
||||||
from .. import InvalidMISPObject
|
from .. import InvalidMISPObject
|
||||||
|
|
@ -18,20 +23,22 @@ class VTReportObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
:indicator: IOC to search VirusTotal for
|
:indicator: IOC to search VirusTotal for
|
||||||
'''
|
'''
|
||||||
def __init__(self, apikey, indicator):
|
def __init__(self, apikey, indicator, vt_proxies=None, standalone=True, **kwargs):
|
||||||
# PY3 way:
|
# PY3 way:
|
||||||
# super().__init__("virustotal-report")
|
# super().__init__("virustotal-report")
|
||||||
super(VTReportObject, self).__init__("virustotal-report")
|
super(VTReportObject, self).__init__("virustotal-report", standalone=standalone, **kwargs)
|
||||||
indicator = indicator.strip()
|
indicator = indicator.strip()
|
||||||
self._resource_type = self.__validate_resource(indicator)
|
self._resource_type = self.__validate_resource(indicator)
|
||||||
if self._resource_type:
|
if self._resource_type:
|
||||||
|
self._proxies = vt_proxies
|
||||||
self._report = self.__query_virustotal(apikey, indicator)
|
self._report = self.__query_virustotal(apikey, indicator)
|
||||||
self.generate_attributes()
|
self.generate_attributes()
|
||||||
else:
|
else:
|
||||||
error_msg = "A valid indicator is required. (One of type url, md5, sha1, sha256). Received '{}' instead".format(indicator)
|
error_msg = "A valid indicator is required. (One of type url, md5, sha1, sha256). Received '{}' instead".format(indicator)
|
||||||
raise InvalidMISPObject(error_msg)
|
raise InvalidMISPObject(error_msg)
|
||||||
# Mark as non_jsonable because we need to add the references manually after the object(s) have been created
|
|
||||||
self.update_not_jsonable('ObjectReference')
|
def get_report(self):
|
||||||
|
return self._report
|
||||||
|
|
||||||
def generate_attributes(self):
|
def generate_attributes(self):
|
||||||
''' Parse the VirusTotal report for relevant attributes '''
|
''' Parse the VirusTotal report for relevant attributes '''
|
||||||
|
|
@ -48,6 +55,8 @@ class VTReportObject(AbstractMISPObjectGenerator):
|
||||||
|
|
||||||
:ioc: Indicator to search VirusTotal for
|
:ioc: Indicator to search VirusTotal for
|
||||||
'''
|
'''
|
||||||
|
if not has_validators:
|
||||||
|
raise Exception('You need to install validators: pip install validators')
|
||||||
if validators.url(ioc):
|
if validators.url(ioc):
|
||||||
return "url"
|
return "url"
|
||||||
elif re.match(r"\b([a-fA-F0-9]{32}|[a-fA-F0-9]{40}|[a-fA-F0-9]{64})\b", ioc):
|
elif re.match(r"\b([a-fA-F0-9]{32}|[a-fA-F0-9]{40}|[a-fA-F0-9]{64})\b", ioc):
|
||||||
|
|
@ -65,7 +74,10 @@ class VTReportObject(AbstractMISPObjectGenerator):
|
||||||
url = "https://www.virustotal.com/vtapi/v2/{}/report".format(self._resource_type)
|
url = "https://www.virustotal.com/vtapi/v2/{}/report".format(self._resource_type)
|
||||||
params = {"apikey": apikey, "resource": resource}
|
params = {"apikey": apikey, "resource": resource}
|
||||||
# for now assume we're using a public API key - we'll figure out private keys later
|
# for now assume we're using a public API key - we'll figure out private keys later
|
||||||
report = requests.get(url, params=params)
|
if self._proxies:
|
||||||
|
report = requests.get(url, params=params, proxies=self._proxies)
|
||||||
|
else:
|
||||||
|
report = requests.get(url, params=params)
|
||||||
report = report.json()
|
report = report.json()
|
||||||
if report["response_code"] == 1:
|
if report["response_code"] == 1:
|
||||||
return report
|
return report
|
||||||
|
|
|
||||||
3
setup.py
3
setup.py
|
|
@ -30,7 +30,8 @@ setup(
|
||||||
install_requires=['six', 'requests', 'python-dateutil', 'jsonschema', 'setuptools>=36.4'],
|
install_requires=['six', 'requests', 'python-dateutil', 'jsonschema', 'setuptools>=36.4'],
|
||||||
extras_require={'fileobjects': ['lief>=0.8', 'python-magic'],
|
extras_require={'fileobjects': ['lief>=0.8', 'python-magic'],
|
||||||
'neo': ['py2neo'],
|
'neo': ['py2neo'],
|
||||||
'openioc': ['beautifulsoup4']},
|
'openioc': ['beautifulsoup4'],
|
||||||
|
'virustotal': ['validators']},
|
||||||
tests_require=[
|
tests_require=[
|
||||||
'jsonschema',
|
'jsonschema',
|
||||||
'python-dateutil',
|
'python-dateutil',
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,23 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "osint"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar.exe"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "osint"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"deleted": true,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"id": "42",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar.exe"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"category": "Attribution",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "registrar",
|
||||||
|
"to_ids": false,
|
||||||
|
"type": "whois-registrar",
|
||||||
|
"value": "registar.example.com"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"category": "Network activity",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "domain",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "domain",
|
||||||
|
"value": "domain.example.com"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"category": "Network activity",
|
||||||
|
"disable_correlation": true,
|
||||||
|
"object_relation": "nameserver",
|
||||||
|
"to_ids": false,
|
||||||
|
"type": "hostname",
|
||||||
|
"value": "ns1.example.com"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"category": "External analysis",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "nameserver",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "hostname",
|
||||||
|
"value": "ns2.example.com"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "Whois records information for a domain name.",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "network",
|
||||||
|
"name": "whois",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "429faea1-34ff-47af-8a00-7c62d3be5a6a",
|
||||||
|
"template_version": 7,
|
||||||
|
"uuid": "a"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,10 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"published": true,
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,59 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "blah"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": true,
|
||||||
|
"object_relation": "filename",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ObjectReference": [
|
||||||
|
{
|
||||||
|
"comment": "foo",
|
||||||
|
"object_uuid": "a",
|
||||||
|
"referenced_uuid": "b",
|
||||||
|
"relationship_type": "baz"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "File object describing a file with meta-information",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "file",
|
||||||
|
"name": "file",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
|
||||||
|
"template_version": 9,
|
||||||
|
"uuid": "a"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"category": "External analysis",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "url",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "url",
|
||||||
|
"value": "https://www.circl.lu"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "url object describes an url along with its normalized field (like extracted using faup parsing library) and its metadata.",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "network",
|
||||||
|
"name": "url",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "60efb77b-40b5-4c46-871b-ed1ed999fce5",
|
||||||
|
"template_version": 5,
|
||||||
|
"uuid": "b"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,56 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "blah"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": true,
|
||||||
|
"object_relation": "filename",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "File object describing a file with meta-information",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "file",
|
||||||
|
"name": "file",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
|
||||||
|
"template_version": 9,
|
||||||
|
"uuid": "a"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "blah"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": true,
|
||||||
|
"object_relation": "filename",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "baz"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "File object describing a file with meta-information",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "file",
|
||||||
|
"name": "file",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
|
||||||
|
"template_version": 9,
|
||||||
|
"uuid": "b"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "filename",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "osint"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "File object describing a file with meta-information",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "file",
|
||||||
|
"name": "file",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
|
||||||
|
"template_version": 9,
|
||||||
|
"uuid": "a"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Tag": [
|
||||||
|
{
|
||||||
|
"name": "bar"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "baz"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "foo"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,21 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"data": "ewogICJFdmVudCI6IHsKICB9Cn0K",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"encrypt": true,
|
||||||
|
"malware_filename": "bar.exe",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "malware-sample",
|
||||||
|
"value": "bar.exe|7637beddacbeac59d44469b2b120b9e6"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,165 @@
|
||||||
|
{"response":[{
|
||||||
|
"Event": {
|
||||||
|
"id": "6719",
|
||||||
|
"orgc_id": "1",
|
||||||
|
"org_id": "1",
|
||||||
|
"date": "2018-01-04",
|
||||||
|
"threat_level_id": "1",
|
||||||
|
"info": "Test existing malware PyMISP",
|
||||||
|
"published": false,
|
||||||
|
"uuid": "5a4e4fdd-1eb4-4ff3-9e87-43fa950d210f",
|
||||||
|
"attribute_count": "6",
|
||||||
|
"analysis": "0",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"distribution": "0",
|
||||||
|
"proposal_email_lock": false,
|
||||||
|
"locked": false,
|
||||||
|
"publish_timestamp": "0",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"event_creator_email": "raphael.vinot@circl.lu",
|
||||||
|
"Org": {
|
||||||
|
"id": "1",
|
||||||
|
"name": "CIRCL",
|
||||||
|
"uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
|
||||||
|
},
|
||||||
|
"Orgc": {
|
||||||
|
"id": "1",
|
||||||
|
"name": "CIRCL",
|
||||||
|
"uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
|
||||||
|
},
|
||||||
|
"Attribute": [],
|
||||||
|
"ShadowAttribute": [],
|
||||||
|
"RelatedEvent": [],
|
||||||
|
"Galaxy": [],
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"id": "2279",
|
||||||
|
"name": "file",
|
||||||
|
"meta-category": "file",
|
||||||
|
"description": "File object describing a file with meta-information",
|
||||||
|
"template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
|
||||||
|
"template_version": "7",
|
||||||
|
"event_id": "6719",
|
||||||
|
"uuid": "5a4e4ffe-4cb8-48b1-bd5c-48fb950d210f",
|
||||||
|
"timestamp": "1515081726",
|
||||||
|
"distribution": "5",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"comment": "",
|
||||||
|
"deleted": false,
|
||||||
|
"ObjectReference": [],
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"id": "814967",
|
||||||
|
"type": "malware-sample",
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"to_ids": true,
|
||||||
|
"uuid": "5a4e4fff-407c-40ff-9de5-43dc950d210f",
|
||||||
|
"event_id": "6719",
|
||||||
|
"distribution": "5",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"comment": "",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_id": "2279",
|
||||||
|
"object_relation": "malware-sample",
|
||||||
|
"value": "simple.json|7637beddacbeac59d44469b2b120b9e6",
|
||||||
|
"data": "UEsDBAoACQAAAEOAJEyjHboUIQAAABUAAAAgABwANzYzN2JlZGRhY2JlYWM1OWQ0NDQ2OWIyYjEyMGI5ZTZVVAkAA\/5PTlr+T05adXgLAAEEIQAAAAQhAAAATvzonhGOj12MyB1QeGLJ5iZhOjD+zymV4FU2+kjD4oTYUEsHCKMduhQhAAAAFQAAAFBLAwQKAAkAAABDgCRMg45UABcAAAALAAAALQAcADc2MzdiZWRkYWNiZWFjNTlkNDQ0NjliMmIxMjBiOWU2LmZpbGVuYW1lLnR4dFVUCQAD\/k9OWv5PTlp1eAsAAQQhAAAABCEAAADDgZOh6307Bduy829xtRjpivO\/xFI3KVBLBwiDjlQAFwAAAAsAAABQSwECHgMKAAkAAABDgCRMox26FCEAAAAVAAAAIAAYAAAAAAABAAAApIEAAAAANzYzN2JlZGRhY2JlYWM1OWQ0NDQ2OWIyYjEyMGI5ZTZVVAUAA\/5PTlp1eAsAAQQhAAAABCEAAABQSwECHgMKAAkAAABDgCRMg45UABcAAAALAAAALQAYAAAAAAABAAAApIGLAAAANzYzN2JlZGRhY2JlYWM1OWQ0NDQ2OWIyYjEyMGI5ZTYuZmlsZW5hbWUudHh0VVQFAAP+T05adXgLAAEEIQAAAAQhAAAAUEsFBgAAAAACAAIA2QAAABkBAAAAAA==",
|
||||||
|
"ShadowAttribute": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "814968",
|
||||||
|
"type": "filename",
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"to_ids": false,
|
||||||
|
"uuid": "5a4e4fff-9ec0-4822-a405-4e29950d210f",
|
||||||
|
"event_id": "6719",
|
||||||
|
"distribution": "5",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"comment": "",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_id": "2279",
|
||||||
|
"object_relation": "filename",
|
||||||
|
"value": "simple.json",
|
||||||
|
"ShadowAttribute": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "814969",
|
||||||
|
"type": "md5",
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"to_ids": true,
|
||||||
|
"uuid": "5a4e4fff-8000-49f9-8c3e-4598950d210f",
|
||||||
|
"event_id": "6719",
|
||||||
|
"distribution": "5",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"comment": "",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_id": "2279",
|
||||||
|
"object_relation": "md5",
|
||||||
|
"value": "7637beddacbeac59d44469b2b120b9e6",
|
||||||
|
"ShadowAttribute": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "814970",
|
||||||
|
"type": "sha1",
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"to_ids": true,
|
||||||
|
"uuid": "5a4e4fff-dae0-4aa4-81ea-4899950d210f",
|
||||||
|
"event_id": "6719",
|
||||||
|
"distribution": "5",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"comment": "",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_id": "2279",
|
||||||
|
"object_relation": "sha1",
|
||||||
|
"value": "023853a4331db8d67e44553004cf338ec1b7440e",
|
||||||
|
"ShadowAttribute": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "814971",
|
||||||
|
"type": "sha256",
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"to_ids": true,
|
||||||
|
"uuid": "5a4e4fff-03ec-4e88-b5f4-472b950d210f",
|
||||||
|
"event_id": "6719",
|
||||||
|
"distribution": "5",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"comment": "",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_id": "2279",
|
||||||
|
"object_relation": "sha256",
|
||||||
|
"value": "6ae8b0f1c7d6f3238d1fc14038018c3b4704c8cc23dac1c2bfd2c81b5a278eef",
|
||||||
|
"ShadowAttribute": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "814972",
|
||||||
|
"type": "size-in-bytes",
|
||||||
|
"category": "Other",
|
||||||
|
"to_ids": false,
|
||||||
|
"uuid": "5a4e4fff-b6f4-41ba-a6eb-446c950d210f",
|
||||||
|
"event_id": "6719",
|
||||||
|
"distribution": "5",
|
||||||
|
"timestamp": "1515081727",
|
||||||
|
"comment": "",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": true,
|
||||||
|
"object_id": "2279",
|
||||||
|
"object_relation": "size-in-bytes",
|
||||||
|
"value": "21",
|
||||||
|
"ShadowAttribute": []
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}]}
|
||||||
|
|
@ -0,0 +1,40 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"category": "Other",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "member3",
|
||||||
|
"to_ids": false,
|
||||||
|
"type": "text",
|
||||||
|
"value": "foo"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"category": "Other",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"object_relation": "member1",
|
||||||
|
"to_ids": false,
|
||||||
|
"type": "text",
|
||||||
|
"value": "bar"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"description": "TestTemplate.",
|
||||||
|
"distribution": 5,
|
||||||
|
"meta-category": "file",
|
||||||
|
"misp_objects_path_custom": "tests/mispevent_testfiles",
|
||||||
|
"name": "test_object_template",
|
||||||
|
"sharing_group_id": 0,
|
||||||
|
"template_uuid": "4ec55cc6-9e49-4c64-b794-03c25c1a6589",
|
||||||
|
"template_version": 1,
|
||||||
|
"uuid": "a"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"ShadowAttribute": [
|
||||||
|
{
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar.pdf"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "bar.exe"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"ShadowAttribute": [
|
||||||
|
{
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"value": "baz.jpg"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"analysis": "1",
|
||||||
|
"date": "2017-12-31",
|
||||||
|
"distribution": "1",
|
||||||
|
"info": "This is a test",
|
||||||
|
"threat_level_id": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,149 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"ShadowAttribute": [
|
||||||
|
{
|
||||||
|
"Org": {
|
||||||
|
"id": "1",
|
||||||
|
"name": "CIRCL",
|
||||||
|
"uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
|
||||||
|
},
|
||||||
|
"category": "Artifacts dropped",
|
||||||
|
"comment": "",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"event_id": "6676",
|
||||||
|
"event_uuid": "5a4cb19a-f550-437f-bd29-48ed950d210f",
|
||||||
|
"id": "3770",
|
||||||
|
"old_id": "811578",
|
||||||
|
"org_id": "1",
|
||||||
|
"proposal_to_delete": false,
|
||||||
|
"timestamp": "1514975846",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"uuid": "5a4cb1c7-fa84-45fa-8d27-4822950d210f",
|
||||||
|
"value": "blah.exe.jpg"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Artifacts dropped",
|
||||||
|
"comment": "",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"distribution": "5",
|
||||||
|
"event_id": "6676",
|
||||||
|
"id": "811578",
|
||||||
|
"object_id": "0",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"timestamp": "1514975687",
|
||||||
|
"to_ids": false,
|
||||||
|
"type": "filename",
|
||||||
|
"uuid": "5a4cb1c7-fa84-45fa-8d27-4822950d210f",
|
||||||
|
"value": "blah.exe"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"Object": [
|
||||||
|
{
|
||||||
|
"Attribute": [
|
||||||
|
{
|
||||||
|
"ShadowAttribute": [
|
||||||
|
{
|
||||||
|
"Org": {
|
||||||
|
"id": "1",
|
||||||
|
"name": "CIRCL",
|
||||||
|
"uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
|
||||||
|
},
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"comment": "",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"event_id": "6676",
|
||||||
|
"event_uuid": "5a4cb19a-f550-437f-bd29-48ed950d210f",
|
||||||
|
"id": "3771",
|
||||||
|
"old_id": "811579",
|
||||||
|
"org_id": "1",
|
||||||
|
"proposal_to_delete": false,
|
||||||
|
"timestamp": "1514976196",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"uuid": "5a4cb2b8-4748-4c72-96e6-4588950d210f",
|
||||||
|
"value": "baz.png.exe"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"category": "Payload delivery",
|
||||||
|
"comment": "",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": false,
|
||||||
|
"distribution": "5",
|
||||||
|
"event_id": "6676",
|
||||||
|
"id": "811579",
|
||||||
|
"object_id": "2278",
|
||||||
|
"object_relation": "filename",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"timestamp": "1514975928",
|
||||||
|
"to_ids": true,
|
||||||
|
"type": "filename",
|
||||||
|
"uuid": "5a4cb2b8-4748-4c72-96e6-4588950d210f",
|
||||||
|
"value": "baz.png"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"category": "Other",
|
||||||
|
"comment": "",
|
||||||
|
"deleted": false,
|
||||||
|
"disable_correlation": true,
|
||||||
|
"distribution": "5",
|
||||||
|
"event_id": "6676",
|
||||||
|
"id": "811580",
|
||||||
|
"object_id": "2278",
|
||||||
|
"object_relation": "state",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"timestamp": "1514975928",
|
||||||
|
"to_ids": false,
|
||||||
|
"type": "text",
|
||||||
|
"uuid": "5a4cb2b9-92b4-4d3a-82df-4e86950d210f",
|
||||||
|
"value": "Malicious"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"comment": "",
|
||||||
|
"deleted": false,
|
||||||
|
"description": "File object describing a file with meta-information",
|
||||||
|
"distribution": "5",
|
||||||
|
"event_id": "6676",
|
||||||
|
"id": "2278",
|
||||||
|
"meta-category": "file",
|
||||||
|
"name": "file",
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
|
||||||
|
"template_version": "7",
|
||||||
|
"timestamp": "1514975928",
|
||||||
|
"uuid": "5a4cb2b8-7958-4323-852c-4d2a950d210f"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"Org": {
|
||||||
|
"id": "1",
|
||||||
|
"name": "CIRCL",
|
||||||
|
"uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
|
||||||
|
},
|
||||||
|
"Orgc": {
|
||||||
|
"id": "1",
|
||||||
|
"name": "CIRCL",
|
||||||
|
"uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
|
||||||
|
},
|
||||||
|
"analysis": "2",
|
||||||
|
"attribute_count": "3",
|
||||||
|
"date": "2018-01-03",
|
||||||
|
"disable_correlation": false,
|
||||||
|
"distribution": "0",
|
||||||
|
"event_creator_email": "raphael.vinot@circl.lu",
|
||||||
|
"id": "6676",
|
||||||
|
"info": "Test proposals / ShadowAttributes",
|
||||||
|
"locked": false,
|
||||||
|
"org_id": "1",
|
||||||
|
"orgc_id": "1",
|
||||||
|
"proposal_email_lock": true,
|
||||||
|
"publish_timestamp": "0",
|
||||||
|
"published": false,
|
||||||
|
"sharing_group_id": "0",
|
||||||
|
"threat_level_id": "1",
|
||||||
|
"timestamp": "1514975929",
|
||||||
|
"uuid": "5a4cb19a-f550-437f-bd29-48ed950d210f"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
{
|
||||||
|
"timestamp": 11111111,
|
||||||
|
"type": "bar",
|
||||||
|
"value": "1"
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
{
|
||||||
|
"Event": {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"requiredOneOf": [
|
||||||
|
"member1",
|
||||||
|
"member2"
|
||||||
|
],
|
||||||
|
"required": [
|
||||||
|
"member3"
|
||||||
|
],
|
||||||
|
"attributes": {
|
||||||
|
"member1": {
|
||||||
|
"description": "FirstMember",
|
||||||
|
"misp-attribute": "text"
|
||||||
|
},
|
||||||
|
"member2": {
|
||||||
|
"description": "SecondMember",
|
||||||
|
"misp-attribute": "text",
|
||||||
|
"multiple": true
|
||||||
|
},
|
||||||
|
"member3": {
|
||||||
|
"description": "Thirdmember",
|
||||||
|
"misp-attribute": "text"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"version": 1,
|
||||||
|
"description": "TestTemplate.",
|
||||||
|
"meta-category": "file",
|
||||||
|
"uuid": "4ec55cc6-9e49-4c64-b794-03c25c1a6589",
|
||||||
|
"name": "test_object_template"
|
||||||
|
}
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
from pymisp import PyMISP
|
from pymisp import PyMISP, __version__
|
||||||
from keys import url, key
|
from keys import url, key
|
||||||
import time
|
import time
|
||||||
|
|
||||||
|
|
@ -12,6 +12,7 @@ class TestBasic(unittest.TestCase):
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
self.maxDiff = None
|
self.maxDiff = None
|
||||||
self.misp = PyMISP(url, key, True, 'json')
|
self.misp = PyMISP(url, key, True, 'json')
|
||||||
|
self.live_describe_types = self.misp.get_live_describe_types()
|
||||||
|
|
||||||
def _clean_event(self, event):
|
def _clean_event(self, event):
|
||||||
event['Event'].pop('orgc_id', None)
|
event['Event'].pop('orgc_id', None)
|
||||||
|
|
@ -53,7 +54,17 @@ class TestBasic(unittest.TestCase):
|
||||||
def add_hashes(self, eventid):
|
def add_hashes(self, eventid):
|
||||||
r = self.misp.get_event(eventid)
|
r = self.misp.get_event(eventid)
|
||||||
event = r.json()
|
event = r.json()
|
||||||
event = self.misp.add_hashes(event, 'Payload installation', 'dll_installer.dll', '0a209ac0de4ac033f31d6ba9191a8f7a', '1f0ae54ac3f10d533013f74f48849de4e65817a7', '003315b0aea2fcb9f77d29223dd8947d0e6792b3a0227e054be8eb2a11f443d9', 'Fanny modules', False, 2)
|
event = self.misp.add_hashes(event,
|
||||||
|
category='Payload installation',
|
||||||
|
filename='dll_installer.dll',
|
||||||
|
md5='0a209ac0de4ac033f31d6ba9191a8f7a',
|
||||||
|
sha1='1f0ae54ac3f10d533013f74f48849de4e65817a7',
|
||||||
|
sha256='003315b0aea2fcb9f77d29223dd8947d0e6792b3a0227e054be8eb2a11f443d9',
|
||||||
|
ssdeep=None,
|
||||||
|
comment='Fanny modules',
|
||||||
|
to_ids=False,
|
||||||
|
distribution=2,
|
||||||
|
proposal=False)
|
||||||
self._clean_event(event)
|
self._clean_event(event)
|
||||||
to_check = {u'Event': {u'info': u'This is a test', u'locked': False,
|
to_check = {u'Event': {u'info': u'This is a test', u'locked': False,
|
||||||
u'attribute_count': u'3', u'analysis': u'0',
|
u'attribute_count': u'3', u'analysis': u'0',
|
||||||
|
|
@ -253,5 +264,47 @@ class TestBasic(unittest.TestCase):
|
||||||
def test_create_organisation(self):
|
def test_create_organisation(self):
|
||||||
self.add_organisation()
|
self.add_organisation()
|
||||||
|
|
||||||
|
def test_describeTypes_sane_default(self):
|
||||||
|
sane_default = self.live_describe_types['sane_defaults']
|
||||||
|
self.assertEqual(sorted(sane_default.keys()), sorted(self.live_describe_types['types']))
|
||||||
|
|
||||||
|
def test_describeTypes_categories(self):
|
||||||
|
category_type_mappings = self.live_describe_types['category_type_mappings']
|
||||||
|
self.assertEqual(sorted(category_type_mappings.keys()), sorted(self.live_describe_types['categories']))
|
||||||
|
|
||||||
|
def test_describeTypes_types_in_categories(self):
|
||||||
|
category_type_mappings = self.live_describe_types['category_type_mappings']
|
||||||
|
for category, types in category_type_mappings.items():
|
||||||
|
existing_types = [t for t in types if t in self.live_describe_types['types']]
|
||||||
|
self.assertEqual(sorted(existing_types), sorted(types))
|
||||||
|
|
||||||
|
def test_describeTypes_types_have_category(self):
|
||||||
|
category_type_mappings = self.live_describe_types['category_type_mappings']
|
||||||
|
all_types = set()
|
||||||
|
for category, types in category_type_mappings.items():
|
||||||
|
all_types.update(types)
|
||||||
|
self.assertEqual(sorted(list(all_types)), sorted(self.live_describe_types['types']))
|
||||||
|
|
||||||
|
def test_describeTypes_sane_default_valid_category(self):
|
||||||
|
sane_default = self.live_describe_types['sane_defaults']
|
||||||
|
categories = self.live_describe_types['categories']
|
||||||
|
for t, sd in sane_default.items():
|
||||||
|
self.assertTrue(sd['to_ids'] in [0, 1])
|
||||||
|
self.assertTrue(sd['default_category'] in categories)
|
||||||
|
|
||||||
|
def test_describeTypes_uptodate(self):
|
||||||
|
self.assertEqual(self.live_describe_types, self.misp.get_local_describe_types())
|
||||||
|
|
||||||
|
def test_live_acl(self):
|
||||||
|
query_acl = self.misp.get_live_query_acl()
|
||||||
|
self.assertEqual(query_acl['response'], [])
|
||||||
|
|
||||||
|
def test_recommended_pymisp_version(self):
|
||||||
|
response = self.misp.get_recommended_api_version()
|
||||||
|
recommended_version_tup = tuple(int(x) for x in response['version'].split('.'))
|
||||||
|
pymisp_version_tup = tuple(int(x) for x in __version__.split('.'))[:3]
|
||||||
|
self.assertEqual(recommended_version_tup, pymisp_version_tup)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,271 @@
|
||||||
|
#!/usr/bin/env python
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from io import BytesIO
|
||||||
|
|
||||||
|
from pymisp import MISPEvent, MISPSighting, MISPTag
|
||||||
|
from pymisp.exceptions import InvalidMISPObject
|
||||||
|
|
||||||
|
|
||||||
|
class TestMISPEvent(unittest.TestCase):
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.maxDiff = None
|
||||||
|
self.mispevent = MISPEvent()
|
||||||
|
|
||||||
|
def init_event(self):
|
||||||
|
self.mispevent.info = 'This is a test'
|
||||||
|
self.mispevent.distribution = 1
|
||||||
|
self.mispevent.threat_level_id = 1
|
||||||
|
self.mispevent.analysis = 1
|
||||||
|
self.mispevent.set_date("2017-12-31") # test the set date method
|
||||||
|
|
||||||
|
def test_simple(self):
|
||||||
|
with open('tests/mispevent_testfiles/simple.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_event(self):
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.publish()
|
||||||
|
with open('tests/mispevent_testfiles/event.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_loadfile(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/event.json')
|
||||||
|
with open('tests/mispevent_testfiles/event.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_event_tag(self):
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.add_tag('bar')
|
||||||
|
self.mispevent.add_tag(name='baz')
|
||||||
|
new_tag = MISPTag()
|
||||||
|
new_tag.from_dict(name='foo')
|
||||||
|
self.mispevent.add_tag(new_tag)
|
||||||
|
with open('tests/mispevent_testfiles/event_tags.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_attribute(self):
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.add_attribute('filename', 'bar.exe')
|
||||||
|
self.mispevent.add_attribute_tag('osint', 'bar.exe')
|
||||||
|
attr_tags = self.mispevent.get_attribute_tag('bar.exe')
|
||||||
|
self.assertEqual(self.mispevent.attributes[0].tags[0].name, 'osint')
|
||||||
|
self.assertEqual(attr_tags[0].name, 'osint')
|
||||||
|
with open('tests/mispevent_testfiles/attribute.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
# Fake setting an attribute ID for testing
|
||||||
|
self.mispevent.attributes[0].id = 42
|
||||||
|
self.mispevent.delete_attribute(42)
|
||||||
|
with open('tests/mispevent_testfiles/attribute_del.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_object_tag(self):
|
||||||
|
self.mispevent.add_object(name='file', strict=True)
|
||||||
|
self.mispevent.objects[0].add_attribute('filename', value='bar', Tag=[{'name': 'blah'}])
|
||||||
|
self.assertEqual(self.mispevent.objects[0].attributes[0].tags[0].name, 'blah')
|
||||||
|
self.assertTrue(self.mispevent.objects[0].has_attributes_by_relation(['filename']))
|
||||||
|
self.assertEqual(len(self.mispevent.objects[0].get_attributes_by_relation('filename')), 1)
|
||||||
|
self.mispevent.add_object(name='url', strict=True)
|
||||||
|
self.mispevent.objects[1].add_attribute('url', value='https://www.circl.lu')
|
||||||
|
self.mispevent.objects[0].uuid = 'a'
|
||||||
|
self.mispevent.objects[1].uuid = 'b'
|
||||||
|
self.mispevent.objects[0].add_reference('b', 'baz', comment='foo')
|
||||||
|
self.assertEqual(self.mispevent.objects[0].references[0].relationship_type, 'baz')
|
||||||
|
with open('tests/mispevent_testfiles/event_obj_attr_tag.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
@unittest.skip("Not supported on MISP: https://github.com/MISP/MISP/issues/2638 - https://github.com/MISP/PyMISP/issues/168")
|
||||||
|
def test_object_level_tag(self):
|
||||||
|
self.mispevent.add_object(name='file', strict=True)
|
||||||
|
self.mispevent.objects[0].add_attribute('filename', value='bar')
|
||||||
|
self.mispevent.objects[0].add_tag('osint')
|
||||||
|
self.mispevent.objects[0].uuid = 'a'
|
||||||
|
with open('tests/mispevent_testfiles/event_obj_tag.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_malware(self):
|
||||||
|
with open('tests/mispevent_testfiles/simple.json', 'rb') as f:
|
||||||
|
pseudofile = BytesIO(f.read())
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.add_attribute('malware-sample', 'bar.exe', data=pseudofile)
|
||||||
|
attribute = self.mispevent.attributes[0]
|
||||||
|
self.assertEqual(attribute.malware_binary, pseudofile)
|
||||||
|
with open('tests/mispevent_testfiles/malware.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_existing_malware(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/malware_exist.json')
|
||||||
|
with open('tests/mispevent_testfiles/simple.json', 'rb') as f:
|
||||||
|
pseudofile = BytesIO(f.read())
|
||||||
|
self.assertEqual(
|
||||||
|
self.mispevent.objects[0].get_attributes_by_relation('malware-sample')[0].malware_binary.read(),
|
||||||
|
pseudofile.read())
|
||||||
|
|
||||||
|
def test_sighting(self):
|
||||||
|
sighting = MISPSighting()
|
||||||
|
sighting.from_dict(value='1', type='bar', timestamp=11111111)
|
||||||
|
with open('tests/mispevent_testfiles/sighting.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(sighting.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_existing_event(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
with open('tests/mispevent_testfiles/existing_event.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_shadow_attributes_existing(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/shadow.json')
|
||||||
|
with open('tests/mispevent_testfiles/shadow.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_shadow_attributes(self):
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.add_proposal(type='filename', value='baz.jpg')
|
||||||
|
self.mispevent.add_attribute('filename', 'bar.exe')
|
||||||
|
self.mispevent.attributes[0].add_proposal(type='filename', value='bar.pdf')
|
||||||
|
with open('tests/mispevent_testfiles/proposals.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_default_attributes(self):
|
||||||
|
self.mispevent.add_object(name='file', strict=True)
|
||||||
|
self.mispevent.objects[0].add_attribute('filename', value='bar', Tag=[{'name': 'blah'}])
|
||||||
|
self.mispevent.add_object(name='file', strict=False, default_attributes_parameters=self.mispevent.objects[0].attributes[0])
|
||||||
|
self.mispevent.objects[1].add_attribute('filename', value='baz')
|
||||||
|
self.mispevent.objects[0].uuid = 'a'
|
||||||
|
self.mispevent.objects[1].uuid = 'b'
|
||||||
|
with open('tests/mispevent_testfiles/event_obj_def_param.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_obj_default_values(self):
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.add_object(name='whois', strict=True)
|
||||||
|
self.mispevent.objects[0].add_attribute('registrar', value='registar.example.com')
|
||||||
|
self.mispevent.objects[0].add_attribute('domain', value='domain.example.com')
|
||||||
|
self.mispevent.objects[0].add_attribute('nameserver', value='ns1.example.com')
|
||||||
|
self.mispevent.objects[0].add_attribute('nameserver', value='ns2.example.com', disable_correlation=False, to_ids=True, category='External analysis')
|
||||||
|
self.mispevent.objects[0].uuid = 'a'
|
||||||
|
with open('tests/mispevent_testfiles/def_param.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_event_not_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.mispevent.info = 'blah'
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_tag_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
self.mispevent.add_tag('foo')
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_attribute_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.mispevent.attributes[0].value = 'blah'
|
||||||
|
self.assertTrue(self.mispevent.attributes[0].edited)
|
||||||
|
self.assertFalse(self.mispevent.attributes[1].edited)
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_attribute_tag_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
self.mispevent.attributes[0].tags[0].name = 'blah'
|
||||||
|
self.assertTrue(self.mispevent.attributes[0].tags[0].edited)
|
||||||
|
self.assertFalse(self.mispevent.attributes[0].tags[1].edited)
|
||||||
|
self.assertTrue(self.mispevent.attributes[0].edited)
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_attribute_tag_edited_second(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
self.mispevent.attributes[0].add_tag(name='blah')
|
||||||
|
self.assertTrue(self.mispevent.attributes[0].edited)
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_object_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
self.mispevent.objects[0].comment = 'blah'
|
||||||
|
self.assertTrue(self.mispevent.objects[0].edited)
|
||||||
|
self.assertFalse(self.mispevent.objects[1].edited)
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_object_attribute_edited(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
self.mispevent.objects[0].attributes[0].comment = 'blah'
|
||||||
|
self.assertTrue(self.mispevent.objects[0].attributes[0].edited)
|
||||||
|
self.assertTrue(self.mispevent.objects[0].edited)
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
|
||||||
|
def test_event_object_attribute_edited_tag(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
self.assertFalse(self.mispevent.edited)
|
||||||
|
self.mispevent.objects[0].attributes[0].add_tag('blah')
|
||||||
|
self.assertTrue(self.mispevent.objects[0].attributes[0].edited)
|
||||||
|
self.assertTrue(self.mispevent.objects[0].edited)
|
||||||
|
self.assertTrue(self.mispevent.edited)
|
||||||
|
with open('tests/mispevent_testfiles/existing_event_edited.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
def test_obj_by_id(self):
|
||||||
|
self.mispevent.load_file('tests/mispevent_testfiles/existing_event.json')
|
||||||
|
misp_obj = self.mispevent.get_object_by_id(1556)
|
||||||
|
self.assertEqual(misp_obj.uuid, '5a3cd604-e11c-4de5-bbbf-c170950d210f')
|
||||||
|
|
||||||
|
def test_userdefined_object(self):
|
||||||
|
self.init_event()
|
||||||
|
self.mispevent.add_object(name='test_object_template', strict=True, misp_objects_path_custom='tests/mispevent_testfiles')
|
||||||
|
with self.assertRaises(InvalidMISPObject) as e:
|
||||||
|
# Fail on required
|
||||||
|
self.mispevent.to_json()
|
||||||
|
if sys.version_info >= (3, ):
|
||||||
|
self.assertEqual(e.exception.message, '{\'member3\'} are required.')
|
||||||
|
else:
|
||||||
|
# Python2 bullshit
|
||||||
|
self.assertEqual(e.exception.message, 'set([u\'member3\']) are required.')
|
||||||
|
|
||||||
|
self.mispevent.objects[0].add_attribute('member3', value='foo')
|
||||||
|
with self.assertRaises(InvalidMISPObject) as e:
|
||||||
|
# Fail on requiredOneOf
|
||||||
|
self.mispevent.to_json()
|
||||||
|
self.assertEqual(e.exception.message, 'At least one of the following attributes is required: member1, member2')
|
||||||
|
|
||||||
|
self.mispevent.objects[0].add_attribute('member1', value='bar')
|
||||||
|
self.mispevent.objects[0].add_attribute('member1', value='baz')
|
||||||
|
with self.assertRaises(InvalidMISPObject) as e:
|
||||||
|
# member1 is not a multiple
|
||||||
|
self.mispevent.to_json()
|
||||||
|
self.assertEqual(e.exception.message, 'Multiple occurrences of member1 is not allowed')
|
||||||
|
|
||||||
|
self.mispevent.objects[0].attributes = self.mispevent.objects[0].attributes[:2]
|
||||||
|
self.mispevent.objects[0].uuid = 'a'
|
||||||
|
with open('tests/mispevent_testfiles/misp_custom_obj.json', 'r') as f:
|
||||||
|
ref_json = json.load(f)
|
||||||
|
self.assertEqual(self.mispevent.to_json(), json.dumps(ref_json, sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
|
|
@ -5,6 +5,8 @@ import unittest
|
||||||
import requests_mock
|
import requests_mock
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import six
|
||||||
|
from io import BytesIO
|
||||||
|
|
||||||
import pymisp as pm
|
import pymisp as pm
|
||||||
from pymisp import PyMISP
|
from pymisp import PyMISP
|
||||||
|
|
@ -49,6 +51,13 @@ class TestOffline(unittest.TestCase):
|
||||||
m.register_uri('DELETE', self.domain + 'events/3', json={'errors': ['Invalid event'], 'message': 'Invalid event', 'name': 'Invalid event', 'url': '/events/3'})
|
m.register_uri('DELETE', self.domain + 'events/3', json={'errors': ['Invalid event'], 'message': 'Invalid event', 'name': 'Invalid event', 'url': '/events/3'})
|
||||||
m.register_uri('GET', self.domain + 'attributes/delete/2', json={'message': 'Attribute deleted.'})
|
m.register_uri('GET', self.domain + 'attributes/delete/2', json={'message': 'Attribute deleted.'})
|
||||||
m.register_uri('POST', self.domain + 'events/index', json=self.search_index_result)
|
m.register_uri('POST', self.domain + 'events/index', json=self.search_index_result)
|
||||||
|
m.register_uri('POST', self.domain + 'attributes/edit/' + self.key, json={})
|
||||||
|
m.register_uri('GET', self.domain + 'shadow_attributes/view/None', json={})
|
||||||
|
m.register_uri('GET', self.domain + 'shadow_attributes/view/1', json={})
|
||||||
|
m.register_uri('POST', self.domain + 'events/freeTextImport/1', json={})
|
||||||
|
m.register_uri('POST', self.domain + 'attributes/restSearch', json={})
|
||||||
|
m.register_uri('POST', self.domain + 'attributes/downloadSample', json={})
|
||||||
|
m.register_uri('GET', self.domain + 'tags', json={'Tag': 'foo'})
|
||||||
|
|
||||||
def test_getEvent(self, m):
|
def test_getEvent(self, m):
|
||||||
self.initURI(m)
|
self.initURI(m)
|
||||||
|
|
@ -210,9 +219,12 @@ class TestOffline(unittest.TestCase):
|
||||||
p.add_internal_other(evt, 'foobar')
|
p.add_internal_other(evt, 'foobar')
|
||||||
p.add_attachment(evt, "testFile")
|
p.add_attachment(evt, "testFile")
|
||||||
|
|
||||||
def make_objects(self, path):
|
def make_objects(self, path=None, pseudofile=None, filename=None):
|
||||||
to_return = {'objects': [], 'references': []}
|
to_return = {'objects': [], 'references': []}
|
||||||
fo, peo, seos = make_binary_objects(path)
|
if path:
|
||||||
|
fo, peo, seos = make_binary_objects(path)
|
||||||
|
else:
|
||||||
|
fo, peo, seos = make_binary_objects(pseudofile=pseudofile, filename=filename)
|
||||||
|
|
||||||
if seos:
|
if seos:
|
||||||
for s in seos:
|
for s in seos:
|
||||||
|
|
@ -229,8 +241,31 @@ class TestOffline(unittest.TestCase):
|
||||||
to_return['objects'].append(fo)
|
to_return['objects'].append(fo)
|
||||||
if fo.ObjectReference:
|
if fo.ObjectReference:
|
||||||
to_return['references'] += fo.ObjectReference
|
to_return['references'] += fo.ObjectReference
|
||||||
|
|
||||||
|
# Remove UUIDs for comparing the objects.
|
||||||
|
for o in to_return['objects']:
|
||||||
|
o.pop('uuid')
|
||||||
|
for o in to_return['references']:
|
||||||
|
o.pop('referenced_uuid')
|
||||||
|
o.pop('object_uuid')
|
||||||
return json.dumps(to_return, cls=MISPEncode)
|
return json.dumps(to_return, cls=MISPEncode)
|
||||||
|
|
||||||
|
def test_objects_pseudofile(self, m):
|
||||||
|
if six.PY2:
|
||||||
|
return unittest.SkipTest()
|
||||||
|
paths = ['cmd.exe', 'tmux', 'MachO-OSX-x64-ls']
|
||||||
|
try:
|
||||||
|
for path in paths:
|
||||||
|
with open(os.path.join('tests', 'viper-test-files', 'test_files', path), 'rb') as f:
|
||||||
|
pseudo = BytesIO(f.read())
|
||||||
|
json_blob = self.make_objects(pseudofile=pseudo, filename=path)
|
||||||
|
# Compare pseudo file / path
|
||||||
|
filepath_blob = self.make_objects(os.path.join('tests', 'viper-test-files', 'test_files', path))
|
||||||
|
self.assertEqual(json_blob, filepath_blob)
|
||||||
|
except IOError: # Can be replaced with FileNotFoundError when support for python 2 is dropped
|
||||||
|
return unittest.SkipTest()
|
||||||
|
print(json_blob)
|
||||||
|
|
||||||
def test_objects(self, m):
|
def test_objects(self, m):
|
||||||
paths = ['cmd.exe', 'tmux', 'MachO-OSX-x64-ls']
|
paths = ['cmd.exe', 'tmux', 'MachO-OSX-x64-ls']
|
||||||
try:
|
try:
|
||||||
|
|
@ -241,5 +276,125 @@ class TestOffline(unittest.TestCase):
|
||||||
return unittest.SkipTest()
|
return unittest.SkipTest()
|
||||||
print(json_blob)
|
print(json_blob)
|
||||||
|
|
||||||
|
def test_describeTypes_sane_default(self, m):
|
||||||
|
sane_default = self.types['result']['sane_defaults']
|
||||||
|
self.assertEqual(sorted(sane_default.keys()), sorted(self.types['result']['types']))
|
||||||
|
|
||||||
|
def test_describeTypes_categories(self, m):
|
||||||
|
category_type_mappings = self.types['result']['category_type_mappings']
|
||||||
|
self.assertEqual(sorted(category_type_mappings.keys()), sorted(self.types['result']['categories']))
|
||||||
|
|
||||||
|
def test_describeTypes_types_in_categories(self, m):
|
||||||
|
category_type_mappings = self.types['result']['category_type_mappings']
|
||||||
|
for category, types in category_type_mappings.items():
|
||||||
|
existing_types = [t for t in types if t in self.types['result']['types']]
|
||||||
|
self.assertEqual(sorted(existing_types), sorted(types))
|
||||||
|
|
||||||
|
def test_describeTypes_types_have_category(self, m):
|
||||||
|
category_type_mappings = self.types['result']['category_type_mappings']
|
||||||
|
all_types = set()
|
||||||
|
for category, types in category_type_mappings.items():
|
||||||
|
all_types.update(types)
|
||||||
|
self.assertEqual(sorted(list(all_types)), sorted(self.types['result']['types']))
|
||||||
|
|
||||||
|
def test_describeTypes_sane_default_valid_category(self, m):
|
||||||
|
sane_default = self.types['result']['sane_defaults']
|
||||||
|
categories = self.types['result']['categories']
|
||||||
|
for t, sd in sane_default.items():
|
||||||
|
self.assertTrue(sd['to_ids'] in [0, 1])
|
||||||
|
self.assertTrue(sd['default_category'] in categories)
|
||||||
|
|
||||||
|
def test_flatten_error_messages_singular(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
error = pymisp.get(1)
|
||||||
|
response = self.auth_error_msg
|
||||||
|
response['error'] = ['foo', 'bar', 'baz']
|
||||||
|
messages = pymisp.flatten_error_messages(response)
|
||||||
|
self.assertEqual(["foo", "bar", "baz"], messages)
|
||||||
|
|
||||||
|
def test_flatten_error_messages_plural(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
error = pymisp.get(1)
|
||||||
|
response = self.auth_error_msg
|
||||||
|
response['errors'] = {'foo': 42, 'bar': False, 'baz': ['oo', 'ka']}
|
||||||
|
messages = pymisp.flatten_error_messages(response)
|
||||||
|
self.assertEqual(set(['42 (foo)', 'False (bar)', 'oo', 'ka']), set(messages))
|
||||||
|
|
||||||
|
def test_flatten_error_messages_nested(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
error = pymisp.get(1)
|
||||||
|
response = self.auth_error_msg
|
||||||
|
response['errors'] = {
|
||||||
|
'fo': {'o': 42}, 'ba': {'r': True}, 'b': {'a': ['z']}, 'd': {'e': {'e': ['p']}}}
|
||||||
|
messages = pymisp.flatten_error_messages(response)
|
||||||
|
self.assertEqual(set(['Error in o: 42', 'Error in r: True', 'Error in a: z', "Error in e: {'e': ['p']}"]), set(messages))
|
||||||
|
|
||||||
|
def test_test_connection(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertTrue(pymisp.test_connection())
|
||||||
|
|
||||||
|
def test_change_toids(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual({}, pymisp.change_toids(self.key, 1))
|
||||||
|
|
||||||
|
def test_change_toids_invalid(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
try:
|
||||||
|
_ = pymisp.change_toids(self.key, 42)
|
||||||
|
self.assertFalse('Exception required for off domain value')
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def test_proposal_view_default(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual({}, pymisp.proposal_view())
|
||||||
|
|
||||||
|
def test_proposal_view_event_1(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual({}, pymisp.proposal_view(event_id=1))
|
||||||
|
|
||||||
|
def test_proposal_view_event_overdetermined(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertTrue(pymisp.proposal_view(event_id=1, proposal_id=42).get('error') is not None)
|
||||||
|
|
||||||
|
def test_freetext(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual({}, pymisp.freetext(1, 'foo', adhereToWarninglists=True, distribution=42))
|
||||||
|
|
||||||
|
def test_freetext_offdomain(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
try:
|
||||||
|
_ = pymisp.freetext(1, None, adhereToWarninglists='hard')
|
||||||
|
self.assertFalse('Exception required for off domain value')
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def test_get_yara(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual((False, None), pymisp.get_yara(1))
|
||||||
|
|
||||||
|
def test_download_samples(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual((False, None), pymisp.download_samples())
|
||||||
|
|
||||||
|
def test_get_all_tags(self, m):
|
||||||
|
self.initURI(m)
|
||||||
|
pymisp = PyMISP(self.domain, self.key)
|
||||||
|
self.assertEqual({'Tag': 'foo'}, pymisp.get_all_tags())
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue