Merge remote-tracking branch 'upstream/master'

CHANGELOG.txt

@@ -2,9 +2,62 @@ Changelog
 =========
 
 
+v2.4.93 (2018-07-01)
+--------------------
+
+New
+~~~
+- Add many comments in the jupyter notebook. [Raphaël Vinot]
+- Return the new object in `add_object` [Raphaël Vinot]
+- Add the ability to add Other attributes via the API. [Paul Stark]
+- Tuto for MISPEvent. [Raphaël Vinot]
+- Load Org and Orgc as MISPOrganisation. [Raphaël Vinot]
+
+  Related to #239
+
+Changes
+~~~~~~~
+- Bump changelog & version. [Raphaël Vinot]
+- Moar jupyter. [Raphaël Vinot]
+- Bump misp-objects. [Raphaël Vinot]
+- Add full example. [Raphaël Vinot]
+- Add few more calls. [Raphaël Vinot]
+- Fix sample retrieval from new-style zips. [Xavier Mehrenberger]
+- Bump misp-objects. [Raphaël Vinot]
+- Raise an exception if the response is not in JSON. [Raphaël Vinot]
+
+Other
+~~~~~
+- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
+- Merge pull request #247 from neok0/master. [Raphaël Vinot]
+
+  enabled published search parameter for attributes controler
+- Enabled published search parameter for attributes controler. [Tobias
+  Mainka]
+- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
+- Merge pull request #241 from cipherlock/master. [Raphaël Vinot]
+
+  new:Add the ability to add Other attributes via the API
+- Merge branch 'master' of github.com:MISP/PyMISP. [Raphaël Vinot]
+- Merge pull request #238 from SHSauler/patch-3. [Raphaël Vinot]
+
+  Added download_samples(..., unzip=True)
+- Added unzip-flag. [Steffen Sauler]
+
+  added: download_samples(..., unzip=True)
+- Merge pull request #235 from Lastpixl/master. [Raphaël Vinot]
+
+  chg: fix sample retrieval from new-style zips
+
+
 v2.4.92.1 (2018-06-05)
 ----------------------
 
+Changes
+~~~~~~~
+- Bump version. [Raphaël Vinot]
+- Bump changelog. [Raphaël Vinot]
+
 Fix
 ~~~
 - Index out of range in add_object. [Raphaël Vinot]

docs/tutorial/PyMISP Objects.ipynb

@@ -1,11 +1,79 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Initializing your environment\n",
+    "\n",
+    "## Installation as PyMISP user\n",
+    "\n",
+    "The quick and dirty way:\n",
+    "\n",
+    "```bash\n",
+    "sudo pip3 install pymisp\n",
+    "```\n",
+    "\n",
+    "The clean approach as user:\n",
+    "\n",
+    "```bash\n",
+    "pip3 install --user pymisp\n",
+    "```\n",
+    "\n",
+    "## Installation as PyMISP developer (recommended for this session)\n",
+    "\n",
+    "\n",
+    "```bash\n",
+    "git clone https://github.com/MISP/PyMISP.git\n",
+    "\n",
+    "cd PyMISP\n",
+    "\n",
+    "virtualenv -p python3 pymisp-env\n",
+    "source pymisp-env/bin/activate\n",
+    "\n",
+    "pip install -e . \n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Setting up of jupyter\n",
+    "\n",
+    "**We assume you're in a virtual environment**\n",
+    "\n",
+    "If you want to follow along this workshop on your computer, this is the way to go:\n",
+    "\n",
+    "\n",
+    "```bash\n",
+    "pip install jupyter\n",
+    "cd docs/tutorial\n",
+    "jupyter-notebook\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Using the PyMISP objects\n",
-    "## MISPEvent"
+    "\n",
+    "This page aims to give recommandations about how to efficiently use the `pymisp` library.\n",
+    "\n",
+    "It is strongly recommended (read \"don't do anything else, please\") to use the library this way and never, ever modify the python dictionary you get by loading the json blob you receive from the server.\n",
+    "\n",
+    "This library is made in a way to hide as much as the complexity as possible and we're happy to improve it is there is someting missing."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## MISPEvent\n",
+    "\n",
+    "`MISPEvent` is the main class to use when you want to create/update events on a MISP instance."
    ]
   },
   {
@@ -30,7 +98,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Tag Event"
+    "## Tag Event\n",
+    "\n",
+    "First example of helper aiming to make your life easier."
    ]
   },
   {
@@ -48,7 +118,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Set the Event date"
+    "## Set the Event date\n",
+    "\n",
+    "\n",
+    "The date can be in many different formats. This helper makes sure it normalises it in a way that will be understood by your MISP instance."
    ]
   },
   {
@@ -60,16 +133,19 @@
     "# As text\n",
     "event.set_date('2018-04-13')\n",
     "print('Simple', event.date)\n",
-    "event.set_date('Sat Oct 11 00:13:46 2017')\n",
-    "print('Mess', event.date)\n",
     "\n",
+    "# Some weird text format (anything supported by dateparse will work)\n",
+    "event.set_date('Sat Oct 11 00:13:46 2017')\n",
+    "print('Messy', event.date)\n",
+    "\n",
+    "# datetime.date\n",
     "from datetime import date\n",
     "d = date.today()\n",
     "print(type(d))\n",
     "event.set_date(d)\n",
     "print(event.date)\n",
     "\n",
-    "\n",
+    "# datetime.datetime => MISP expects a day, so the hour will be droped.\n",
     "from datetime import datetime\n",
     "d = datetime.now()\n",
     "print(type(d))\n",
@@ -81,7 +157,20 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Add Attribute to event"
+    "## Add Attribute to event\n",
+    "\n",
+    "More usefull things: adding attributes to an event.\n",
+    "\n",
+    "Attributes have a bunch of parameters you can pass (if you feel like it). If you don't pass them, they'll be automatically set depending on their sane defaults.\n",
+    "\n",
+    "The parameters are the following:\n",
+    "* **type** (required)\n",
+    "* **value** (required)\n",
+    "* **category**: [see default](https://github.com/MISP/PyMISP/blob/master/pymisp/data/describeTypes.json)\n",
+    "* **to_ids**: [see default](https://github.com/MISP/PyMISP/blob/master/pymisp/data/describeTypes.json)\n",
+    "* **distribution**: defaults to inherit from parent (event or object)\n",
+    "* **disable_correlation**: true for a normal attribute, fallback to the value defined in the template object if relevant\n",
+    "* **data**: only for malware-sample or attachment, BytesIO object of the file. If it is a malware, the sample is decrypted in memory"
    ]
   },
   {
@@ -101,7 +190,29 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Modify existing attribute"
+    "## Set parameters (inline)\n",
+    "\n",
+    "This is the was to pass other parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "attribute_second = event.add_attribute('ip-dst', '8.8.8.9', disable_correlation=True)\n",
+    "\n",
+    "print(attribute_second.to_json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Modify existing attribute\n",
+    "\n",
+    "Every parameter can be modified in a pythonic way."
    ]
   },
   {
@@ -115,24 +226,6 @@
     "print(attribute.to_json())"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Set parameters (inline)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "attribute_second = event.add_attribute('ip-dst', '8.8.8.9', disable_correlation=True)  # Minimal parameters\n",
-    "\n",
-    "print(attribute_second.to_json())"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -146,6 +239,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Using the list of attributes in the event\n",
+    "event.attributes[0].add_tag('tlp:green')\n",
+    "\n",
+    "# ... or the variable we got from `add_attribute`\n",
     "attribute_second.add_tag('tlp:amber')\n",
     "\n",
     "print(attribute_second.to_json())"
@@ -155,7 +252,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Soft delete attribute"
+    "## Soft delete attribute\n",
+    "\n",
+    "**Important note**: the default approach to *delete* on MISP is to do a soft delete (meaning the attribue is not displayed on the default view on MISP). The reason we do it this way is that it allows to push *delete* updates to instances we synchronize with.\n",
+    "\n",
+    "The delete method will set the default parameter of the attribute to `True`."
    ]
   },
   {
@@ -172,7 +273,30 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## MISPObject"
+    "## Mark event as published\n",
+    "\n",
+    "Same idea: you can set the published flag from the api"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "event.publish()\n",
+    "print(event.published)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## MISPObject\n",
+    "\n",
+    "Objects in MISP are a way to group attributes together in a way that makes sense. The objects are based on templates that are bundled in the library itself.\n",
+    "\n",
+    "**Note**: you can use your own templates, we will see how later"
    ]
   },
   {
@@ -189,7 +313,7 @@
     "misp_object = MISPObject('domain-ip', standalone=False, default_attributes_parameters=circl_attr)\n",
     "# Notes: \n",
     "# * standalone: this object will be attached to a MISPEvent, so the references will be in the dump\n",
-    "# * default_attributes_parameters: keep parameters from a MISPAttribute (usefull when expanding a existing one) \n",
+    "# * default_attributes_parameters: keep parameters from a MISPAttribute (useful when expanding a existing one) \n",
     "misp_object.comment = 'My Fancy new object'\n",
     "\n",
     "obj_attr = misp_object.add_attribute('domain', value='circl.lu')\n",
@@ -197,7 +321,6 @@
     "misp_object.add_attribute('ip', value='149.13.33.14')\n",
     "misp_object.add_attribute('first-seen', value='2018-04-11')\n",
     "misp_object.add_attribute('last-seen', value='2018-06-11')\n",
-    "misp_object.add_reference(circl_attr.uuid, 'related-to', 'Expanded with passive DNS entry')\n",
     "\n",
     "event.add_object(misp_object)\n",
     "print(event.to_json())\n"
@@ -207,7 +330,48 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Helpers for MISPObjects "
+    "## One-liner to add an object to a MISPEvent\n",
+    "\n",
+    "You can also add the object directly in a misp event this way"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp import MISPObject\n",
+    "\n",
+    "misp_object = event.add_object(name='domain-ip', comment='My Fancy new object, in one line')\n",
+    "\n",
+    "obj_attr = misp_object.add_attribute('domain', value='circl.lu')\n",
+    "obj_attr.add_tag('tlp:green')\n",
+    "misp_object.add_attribute('ip', value='149.13.33.14')\n",
+    "misp_object.add_attribute('first-seen', value='2018-04-11')\n",
+    "misp_object.add_attribute('last-seen', value='2018-06-11')\n",
+    "misp_object.add_reference(circl_attr.uuid, 'related-to', 'Expanded with passive DNS entry')\n",
+    "\n",
+    "print(event.to_json())\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Helpers for MISPObjects \n",
+    "\n",
+    "For some objects, we have helpers in order to make your life easier. The most relevant example is the file object: when you have a file to push on MISP, there are plenty of indicators you can extract at once, and it is pretty simple to automate, so we made it a oneliner.\n",
+    "\n",
+    "**Note**: This requires a few more dependencies to get the full power of the script: \n",
+    "* `lief` to extract indicators out of PE/ELF/MachO files, and soon Android binaries.\n",
+    "* `python-magic` to get the mime type\n",
+    "* `pydeep` to compute the ssdeep of the binary whenever possible\n",
+    "\n",
+    "\n",
+    "```bash\n",
+    "pip install lief python-magic git+https://github.com/kbandla/pydeep.git\n",
+    "```"
    ]
   },
   {
@@ -236,7 +400,101 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Playing with a malware sample"
+    "# Generic helper\n",
+    "\n",
+    "This helper is meant to be used when you alreadu have a script that does the mapping between your own code, and the MISPObject template."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp.tools import GenericObjectGenerator\n",
+    "\n",
+    "attributeAsDict = [{'script': ':(){ :|:& };:', 'comment': 'Forkbomb'},\n",
+    "                   {'language': {'value': 'Bash', 'to_ids': False, 'disable_correlation': True}}, \n",
+    "                   {'filename': {'value': 'forkbomb.sh', 'to_ids': True}},\n",
+    "                   {'state': 'Malicious'}]\n",
+    "\n",
+    "\n",
+    "misp_object = GenericObjectGenerator('script', strict=True)\n",
+    "misp_object.generate_attributes(attributeAsDict)\n",
+    "\n",
+    "print(misp_object.to_json())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp.tools import GenericObjectGenerator\n",
+    "\n",
+    "attributeAsDict = [{'MyCoolAttribute': {'value': 'critical thing', 'type': 'text'}}, \n",
+    "                   {'MyCoolerAttribute': {'value': 'even worse',  'type': 'text'}}]\n",
+    "\n",
+    "\n",
+    "misp_object = GenericObjectGenerator('my-cool-template', strict=True) # This is supposed to fail due to the strict parameter\n",
+    "misp_object.generate_attributes(attributeAsDict)\n",
+    "\n",
+    "print(misp_object.to_json())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp.tools import GenericObjectGenerator\n",
+    "\n",
+    "attributeAsDict = [{'MyCoolAttribute': {'value': 'critical thing', 'type': 'text'}}, \n",
+    "                   {'MyCoolerAttribute': {'value': 'even worse',  'type': 'text'}}]\n",
+    "\n",
+    "\n",
+    "misp_object = GenericObjectGenerator('my-cool-template')\n",
+    "misp_object.generate_attributes(attributeAsDict)\n",
+    "\n",
+    "print(misp_object.to_json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Use locally defined objet templates\n",
+    "\n",
+    "**Important**: The path you pass as parameter for `misp_objects_path_custom` needs to contain a directory equals to the value of the parameter `name` (same structure as the content of the `misp-object` repository)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "user_defined_obj = MISPObject(name='test_object_template', strict=True, misp_objects_path_custom='../../tests/mispevent_testfiles')\n",
+    "\n",
+    "user_defined_obj.add_attribute('member3', value='foo')\n",
+    "user_defined_obj.add_attribute('member1', value='baz')\n",
+    "\n",
+    "print(user_defined_obj.to_json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Playing with a malware sample\n",
+    "\n",
+    "The data you receive out of the JSON dump from a MISP instance is a base64 encoded zip with `infected` as a password. The zip file contains 2 files, one containing the original file name of the uploaded file, and the other one is the binary.\n",
+    "\n",
+    "This is pretty much a pain to use as-is.\n",
+    "\n",
+    "So there is an helper for that!\n"
    ]
   },
   {
@@ -247,15 +505,20 @@
    "source": [
     "sample = file_obj.get_attributes_by_relation('malware-sample')[0]\n",
     "print(sample)\n",
+    "print('File name --->', sample.malware_filename)\n",
     "print(sample.malware_binary)\n",
-    "print(sample.malware_binary.getvalue())"
+    "print('Content of the malware (in bytes) ----->', sample.malware_binary.getvalue())"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Use lief to extract indicators out of binaries"
+    "## Use lief to extract indicators out of binaries\n",
+    "\n",
+    "An other cool helper: one liner to whom you can pass the path to a binary, if it is supported by `lief` (PE/ELF/Mach-o), you get the the file object, a PE, ELF, or Mach-o object, and the relevant sections.\n",
+    "\n",
+    "If it is anything else, it will just generate the the file object.\n"
    ]
   },
   {
@@ -269,6 +532,7 @@
     "file_obj, bin_obj, sections = make_binary_objects(filepath='../../tests/viper-test-files/test_files/whoami.exe', standalone=False)\n",
     "\n",
     "event.add_object(file_obj)\n",
+    "if bin_obj: \n",
     "    event.add_object(bin_obj)\n",
     "    for s in sections:\n",
     "        event.add_object(s)"
@@ -278,7 +542,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## References"
+    "## References\n",
+    "\n",
+    "The references are also set by default by this method."
    ]
   },
   {
@@ -288,53 +554,17 @@
    "outputs": [],
    "source": [
     "print(bin_obj.uuid)\n",
-    "print(bin_obj.references[0].to_json())"
+    "print(bin_obj.references[0].to_json())\n",
+    "print(event.to_json())"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Change creator"
+    "## Dump valid MISP Event ready to push to MISP\n",
-   ]
+    "\n",
-  },
+    "We've been using `to_json` a lot. The thing you should know is that every python MISP objects have this method, and it **always** returns a valid json blob you can send to MISP."
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from pymisp import MISPOrganisation\n",
-    "orgc = MISPOrganisation()\n",
-    "orgc.name = 'bazbaz'\n",
-    "orgc.id = 15\n",
-    "orgc.uuid = '5888a98d-a7e8-4183-94bb-4d19950d210f'\n",
-    "# NOTE: Pushing this object will only work if the user has sync right (if not, the orgc key will be ignored)\n",
-    "event.Orgc = orgc"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Mark event as published"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "event.publish()\n",
-    "print(event.published)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Dump valid MISP Event ready to push to MISP"
    ]
   },
   {
@@ -350,7 +580,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Update an existing MISPEvent"
+    "# Update an existing MISPEvent\n",
+    "\n",
+    "We were creating new events, but you will also want to update an existing one."
    ]
   },
   {
@@ -374,7 +606,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Edit, removes the timestamp when exporting"
+    "## Edit, removes the timestamp when exporting\n",
+    "\n",
+    "If you tried to edit an event manually, and never got the updates on the instance, it is probably because the timestamps weren't updated/removed. Or you removed them all, and adding a single tag was makting every attributes as new.\n",
+    "\n",
+    "PyMISP got you covered."
    ]
   },
   {
@@ -394,6 +630,54 @@
     "## Full example"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp import MISPEvent, MISPObject\n",
+    "from pymisp import PyMISP\n",
+    "\n",
+    "event = MISPEvent()\n",
+    "event.info = 'This is my new MISP event'  # Required\n",
+    "event.distribution = 0  # Optional, defaults to MISP.default_event_distribution in MISP config\n",
+    "event.threat_level_id = 2  # Optional, defaults to MISP.default_event_threat_level in MISP config\n",
+    "event.analysis = 1  # Optional, defaults to 0 (initial analysis)\n",
+    "\n",
+    "mispObject = MISPObject('file')\n",
+    "mispObject.add_attribute('filename', type='filename',\n",
+    "                         value='filename.exe',\n",
+    "                         Tag=[{'name': 'tlp:amber'}])\n",
+    "\n",
+    "event.add_object(mispObject)\n",
+    "\n",
+    "# The URL of the MISP instance to connect to\n",
+    "misp_url = 'http://127.0.0.1:8080'\n",
+    "# Can be found in the MISP web interface under \n",
+    "# http://+MISP_URL+/users/view/me -> Authkey\n",
+    "misp_key = 'yB8DMS8LkfYYpcVX8bN2v7xwDZDMp4bpW0sNqNGj'\n",
+    "# Should PyMISP verify the MISP certificate\n",
+    "misp_verifycert = False\n",
+    "\n",
+    "misp = PyMISP(misp_url, misp_key, misp_verifycert)\n",
+    "res = misp.add_event(event)\n",
+    "existing_event = MISPEvent()\n",
+    "existing_event.load(res)\n",
+    "mispObject = MISPObject('file')\n",
+    "mispObject.add_attribute('filename', type='filename',\n",
+    "                         value='filename2.exe',\n",
+    "                         Tag=[{'name': 'tlp:white'}])\n",
+    "\n",
+    "existing_event.add_object(mispObject)\n",
+    "print(existing_event.to_json())\n",
+    "\n",
+    "res = misp.update(existing_event)\n",
+    "existing_event = MISPEvent()\n",
+    "existing_event.load(res)\n",
+    "print(existing_event.to_json())"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -417,12 +701,12 @@
     "event.add_object(mispObject)\n",
     "\n",
     "# The URL of the MISP instance to connect to\n",
-    "misp_url = 'https://<URL>/'\n",
+    "misp_url = 'http://127.0.0.1:8080'\n",
     "# Can be found in the MISP web interface under \n",
     "# http://+MISP_URL+/users/view/me -> Authkey\n",
-    "misp_key = '<key>'\n",
+    "misp_key = 'yB8DMS8LkfYYpcVX8bN2v7xwDZDMp4bpW0sNqNGj'\n",
     "# Should PyMISP verify the MISP certificate\n",
-    "misp_verifycert = True\n",
+    "misp_verifycert = False\n",
     "\n",
     "misp = PyMISP(misp_url, misp_key, misp_verifycert)\n",
     "res = misp.add_event(event)\n",
@@ -430,6 +714,37 @@
     "existing_event.load(res)\n",
     "print(existing_event.to_json())"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Specific use-cases"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Change creator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp import MISPOrganisation\n",
+    "orgc = MISPOrganisation()\n",
+    "orgc.name = 'bazbaz'\n",
+    "orgc.id = 15\n",
+    "orgc.uuid = '5888a98d-a7e8-4183-94bb-4d19950d210f'\n",
+    "# NOTE: Pushing this object will only work if the user has sync right (if not, the orgc key will be ignored)\n",
+    "event.Orgc = orgc\n",
+    "\n",
+    "print(event.to_json())\n"
+   ]
   }
  ],
  "metadata": {

docs/tutorial/Usage.ipynb

@@ -0,0 +1,468 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Using the API to interact with a remote MISP instance\n",
+    "\n",
+    "You can fetch a VM from here: https://www.circl.lu/misp-images/latest/, or connect to your dev instance.\n",
+    "\n",
+    "This box needs to be run in order to connect to the MISP instance and run the subsequent commands."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pymisp import PyMISP, MISPEvent, MISPAttribute\n",
+    "\n",
+    "# The URL of the MISP instance to connect to\n",
+    "misp_url = 'http://127.0.0.1:8080'\n",
+    "# Can be found in the MISP web interface under \n",
+    "# http://+MISP_URL+/users/view/me -> Authkey\n",
+    "misp_key = 'yB8DMS8LkfYYpcVX8bN2v7xwDZDMp4bpW0sNqNGj'\n",
+    "# Should PyMISP verify the MISP certificate\n",
+    "misp_verifycert = False\n",
+    "\n",
+    "misp = PyMISP(misp_url, misp_key, misp_verifycert)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Get the last events\n",
+    "\n",
+    "There are multiple definition for *last* in MISP.\n",
+    "\n",
+    "## Last *published* events"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "response = misp.search(last='1d')\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.load(event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = misp.search(last=['3d', '2d'])\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.load(event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Last *updated* events"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "from datetime import datetime\n",
+    "\n",
+    "ts = int(datetime.now().timestamp())\n",
+    "\n",
+    "response = misp.search(timestamp=ts-36000)\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.load(event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## On an interval"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp = PyMISP(misp_url, misp_key, misp_verifycert, debug=True)\n",
+    "\n",
+    "ts = int(datetime.now().timestamp())\n",
+    "\n",
+    "response = misp.search(timestamp=[ts-3600, ts])\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.load(event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)\n",
+    "    \n",
+    "misp = PyMISP(misp_url, misp_key, misp_verifycert)  # TODO: remove when fixed"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Get the last attributes\n",
+    "\n",
+    "## Last *published* attributes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = misp.search(controller='attributes', last='1h')\n",
+    "\n",
+    "attributes = []\n",
+    "for attribute in response['response']['Attribute']:\n",
+    "    ma = MISPAttribute()\n",
+    "    ma.from_dict(**attribute)\n",
+    "    attributes.append(ma)\n",
+    "\n",
+    "for a in attributes:\n",
+    "    print(a.event_id, a)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = misp.search(controller='attributes', last=['2h', '1h'])\n",
+    "\n",
+    "attributes = []\n",
+    "for attribute in response['response']['Attribute']:\n",
+    "    ma = MISPAttribute()\n",
+    "    ma.from_dict(**attribute)\n",
+    "    attributes.append(ma)\n",
+    "\n",
+    "for a in attributes:\n",
+    "    print(a)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Last *updated* attributes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "ts = int(datetime.now().timestamp())\n",
+    "\n",
+    "response = misp.search(controller='attributes', timestamp=ts - 36000)\n",
+    "\n",
+    "attributes = []\n",
+    "for attribute in response['response']['Attribute']:\n",
+    "    ma = MISPAttribute()\n",
+    "    ma.from_dict(**attribute)\n",
+    "    attributes.append(ma)\n",
+    "\n",
+    "for a in attributes:\n",
+    "    print(a)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fast search at index event level\n",
+    "\n",
+    "You have multiple ways to search for different values in MISP. Searching in the medadata of the events is very fast and if generally the recommended approach if your query returns lots of events."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = misp.search_index(eventinfo='Cobalt Strike')\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.from_dict(**event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('No attributes are in the event', events[0].attributes)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = misp.search_index(tag='malware_classification:malware-category=\"Ransomware\"')\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.from_dict(**event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "response = misp.search_index(timestamp='1h')\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.from_dict(**event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "events[0].id"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "event = MISPEvent()\n",
+    "event.load(misp.get(events[0].id))\n",
+    "print(event.to_json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Search indicators"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = misp.search(values=['59.157.4.2', 'hotfixmsupload.com'])\n",
+    "\n",
+    "events = []\n",
+    "for event in response['response']:\n",
+    "    me = MISPEvent()\n",
+    "    me.load(event)\n",
+    "    events.append(me)\n",
+    "\n",
+    "for e in events:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Sightings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.sighting(value=e.attributes[3].value)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.sighting_list(e.attributes[3].id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Admin Stuff"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.get_sharing_groups()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## User"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.get_users_list()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.add_user('bar@foo.de', 1, 3)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.get_organisations_list()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.get_roles_list()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.get_feeds_list()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "misp.cache_feeds_all()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

pymisp/__init__.py

@@ -1,4 +1,4 @@
-__version__ = '2.4.92.1'
+__version__ = '2.4.93'
 import logging
 import functools
 import warnings

pymisp/api.py

@@ -1161,12 +1161,12 @@ class PyMISP(object):
         query['enforceWarninglist'] = kwargs.pop('enforceWarninglist', None)
         query['to_ids'] = kwargs.pop('to_ids', None)
         query['deleted'] = kwargs.pop('deleted', None)
+        query['published'] = kwargs.pop('published', None)
 
         if controller == 'events':
             # Event search only:
             query['searchall'] = kwargs.pop('searchall', None)
             query['metadata'] = kwargs.pop('metadata', None)
-            query['published'] = kwargs.pop('published', None)
         if controller == 'attributes':
             query['event_timestamp'] = kwargs.pop('event_timestamp', None)
 

pymisp/data/misp-objects

@@ -1 +1 @@
-Subproject commit ce8472b92e7e74de4e214bb345d2a9f590e11016
+Subproject commit e9fd65cecb028dbc8ed54894c469de7d352469bb

pymisp/mispevent.py

@@ -674,6 +674,7 @@ class MISPEvent(AbstractMISP):
             raise InvalidMISPObject("An object to add to an existing Event needs to be either a MISPObject, or a plain python dictionary")
         self.Object.append(misp_obj)
         self.edited = True
+        return misp_obj
 
     def __repr__(self):
         if hasattr(self, 'info'):
@@ -845,6 +846,8 @@ class MISPObjectAttribute(MISPAttribute):
         if self.to_ids is None:
             # Same for the to_ids flag
             self.to_ids = self._definition.get('to_ids')
+        if not self.type:
+            raise NewAttributeError("The type of the attribute is required. Is the object template missing?")
         super(MISPObjectAttribute, self).from_dict(**dict(self, **kwargs))
 
     def __repr__(self):