07137209e2
eml files downloaded from Windows Online security on some Windows 11 systems are automatically encoded in UTF with a byte order mark (BOM) at the front of the file. This will cause the email parser to fail. This is a somewhat isolated problem. It only will affects a small subset of Windows users who download and re-upload eml files. But, this small subset of users is the target user-base for the MISP email module: low expertiese users who wish to quickly share high-value indicators on an ad-hoc basis. While this fix could be tacked onto the MISP email module instead of here, I beleive that this fix is more appropriate in the PyMISP object code. As the "email" object parser this object should be built to parse all manner of emails that it may encounter. This includes common malformations such as this one and, even horrors such as, the .msg format. This commit adds a generically named "attempt_decoding" function which can be expanded to address all manner of sins that are encountered in the future. |
||
|---|---|---|
| docs | ||
| examples | ||
| pymisp | ||
| tests | ||
| travis | ||
| .gitchangelog.rc | ||
| .gitignore | ||
| .gitmodules | ||
| .readthedocs.yml | ||
| .travis.yml | ||
| CHANGELOG.txt | ||
| LICENSE | ||
| MANIFEST.in | ||
| README.md | ||
| poetry.lock | ||
| pyproject.toml | ||
| setup.cfg | ||
| setup.py | ||
README.md
IMPORTANT NOTE: This library will require at least python 3.6 starting the 1st of January 2020. If you have to legacy versions of python, please use PyMISP v2.4.119.1, and consider updating your system(s). Anything released within the last 2 years will do, starting with Ubuntu 18.04.
README
PyMISP - Python Library to access MISP
PyMISP is a Python library to access MISP platforms via their REST API.
PyMISP allows you to fetch events, add or update events/attributes, add or update samples or search for attributes.
Install from pip
It is strongly recommended to use a virtual environment
If you want to know more about virtual environments, python has you covered
Only basic dependencies:
pip3 install pymisp
With optional dependencies:
pip3 install pymisp[fileobjects,openioc,virustotal]
Install the latest version from repo from development purposes
Note: poetry is required
git clone https://github.com/MISP/PyMISP.git && cd PyMISP
git submodule update --init
poetry install -E fileobjects -E openioc -E virustotal -E docs -E pdfexport
Running the tests
poetry run nosetests-3.4 --with-coverage --cover-package=pymisp,tests --cover-tests tests/test_*.py
If you have a MISP instance to test against, you can also run the live ones:
Note: You need to update the key in tests/testlive_comprehensive.py to the automation key of your admin account.
poetry run nosetests-3.4 --with-coverage --cover-package=pymisp,tests --cover-tests tests/testlive_comprehensive.py
Samples and how to use PyMISP
Various examples and samples scripts are in the examples/ directory.
In the examples directory, you will need to change the keys.py.sample to enter your MISP url and API key.
cd examples
cp keys.py.sample keys.py
vim keys.py
The API key of MISP is available in the Automation section of the MISP web interface.
To test if your URL and API keys are correct, you can test with examples/last.py to fetch the events published in the last x amount of time (supported time indicators: days (d), hours (h) and minutes (m)). last.py
cd examples
python3 last.py -l 10h # 10 hours
python3 last.py -l 5d # 5 days
python3 last.py -l 45m # 45 minutes
Debugging
You have two options there:
-
Pass
debug=TruetoPyMISPand it will enable logging.DEBUG to stderr on the whole module -
Use the python logging module directly:
import logging
logger = logging.getLogger('pymisp')
# Configure it as you whish, for example, enable DEBUG mode:
logger.setLevel(logging.DEBUG)
Or if you want to write the debug output to a file instead of stderr:
import pymisp
import logging
logger = logging.getLogger('pymisp')
logging.basicConfig(level=logging.DEBUG, filename="debug.log", filemode='w', format=pymisp.FORMAT)
Test cases
- The content of
mispevent.pyis tested on every commit - The tests cases that require a running MISP instance can be run the following way:
# From poetry
nosetests-3.4 -s --with-coverage --cover-package=pymisp,tests --cover-tests tests/testlive_comprehensive.py:TestComprehensive.[test_name]
Documentation
The documentation is available here.
Jupyter notebook
A series of Jupyter notebooks for PyMISP tutorial are available in the repository.
Everything is a Mutable Mapping
... or at least everything that can be imported/exported from/to a json blob
AbstractMISP is the master class, and inherit collections.MutableMapping which means
the class can be represented as a python dictionary.
The abstraction assumes every property that should not be seen in the dictionary is prepended with a _,
or its name is added to the private list __not_jsonable (accessible through update_not_jsonable and set_not_jsonable.
This master class has helpers that will make it easy to load, and export, to, and from, a json string.
MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, and MISPObject
are subclasses of AbstractMISP, which mean that they can be handled as python dictionaries.
MISP Objects
Creating a new MISP object generator should be done using a pre-defined template and inherit AbstractMISPObjectGenerator.
Your new MISPObject generator need to generate attributes, and add them as class properties using add_attribute.
When the object is sent to MISP, all the class properties will be exported to the JSON export.