{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true, "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true, "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import HTML\n", "\n", "original_print = print\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " if string[0] == '{':\n", " formatter = HtmlFormatter()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, JsonLexer(), formatter)))\n", " else:\n", " original_print(inpt)\n", "\n", "print = json_print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Memory\n", "\n", "The Memory suite consists of [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore), [MemorySource](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemorySource), and [MemorySink](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemorySink). Under the hood, the Memory suite points to an in-memory dictionary. Similarly, the [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore) is a just a wrapper around a paired [MemorySource](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemorySource) and [MemorySink](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemorySink); as there is quite limited uses for just a [MemorySource](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemorySource) or a [MemorySink](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemorySink), it is recommended to always use [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore). The [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore) is intended for retrieving/searching and pushing STIX content to memory. It is important to note that all STIX content in memory is not backed up on the file system (disk), as that functionality is encompassed within the [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore). However, the Memory suite does provide some utility methods for saving and loading STIX content to disk. [MemoryStore.save_to_file()](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore.save_to_file) allows for saving all the STIX content that is in memory to a json file. [MemoryStore.load_from_file()](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore.load_from_file) allows for loading STIX content from a JSON-formatted file. \n", "\n", "\n", "### Memory API\n", "A note on adding and retreiving STIX content to the Memory suite: As mentioned, under the hood the Memory suite is an internal, in-memory dictionary. STIX content that is to be added can be in the following forms: python-stix2 objects, (Python) dictionaries (of valid STIX objects or Bundles), JSON-encoded strings (of valid STIX objects or Bundles), or a (Python) list of any of the previously listed types. [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore) actually stores STIX content either as python-stix2 objects or as (Python) dictionaries, reducing and converting any of the aforementioned types to one of those. Additionally, whatever form the STIX object is stored as, is how it will be returned when retrieved. python-stix2 objects, and json-encoded strings (of STIX content) are stored as python-stix2 objects, while (Python) dictionaries (of STIX objects) are stored as (Python) dictionaries.\n", "\n", "A note on [load_from_file()](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore.load_from_file): For [load_from_file()](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore.load_from_file), STIX content is assumed to be in JSON form within the file, as an individual STIX object or in a Bundle. When the JSON is loaded, the STIX objects are parsed into python-stix2 objects before being stored in the in-memory dictionary.\n", "\n", "A note on [save_to_file()](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore.save_to_file): This method dumps all STIX content that is in the [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore) to the specified file. The file format will be JSON, and the STIX content will be within a STIX Bundle. Note also that the output form will be a JSON STIX Bundle regardless of the form that the individual STIX objects are stored in (i.e. supplied to) the [MemoryStore](../api/datastore/stix2.datastore.memory.rst#stix2.datastore.memory.MemoryStore). \n", "\n", "### Memory Examples\n", "\n", "#### MemoryStore" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "indicator",\n",
       "    "id": "indicator--2f61e4e7-0891-4e09-b79a-66f5e594fec0",\n",
       "    "created": "2017-11-17T17:01:31.590Z",\n",
       "    "modified": "2017-11-17T17:01:31.590Z",\n",
       "    "description": "Crusades C2 implant",\n",
       "    "pattern": "[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']",\n",
       "    "valid_from": "2017-11-17T17:01:31.590939Z",\n",
       "    "labels": [\n",
       "        "malicious-activity"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import MemoryStore, Indicator\n", "\n", "# create default MemoryStore\n", "mem = MemoryStore()\n", "\n", "# insert newly created indicator into memory\n", "ind = Indicator(description=\"Crusades C2 implant\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "mem.add(ind)\n", "\n", "# for visual purposes\n", "print(mem.get(ind.id))\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "indicator",\n",
       "    "id": "indicator--ddb765ba-ff1e-4285-bf33-1f6d08f583d6",\n",
       "    "created": "2017-11-17T17:01:31.799Z",\n",
       "    "modified": "2017-11-17T17:01:31.799Z",\n",
       "    "description": "Crusades stage 2 implant variant",\n",
       "    "pattern": "[file:hashes.'SHA-256' = '31a45e777e4d58b97f4c43e38006f8cd6580ddabc4037905b2fad734712b582c']",\n",
       "    "valid_from": "2017-11-17T17:01:31.799228Z",\n",
       "    "labels": [\n",
       "        "malicious-activity"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import Malware\n", "\n", "# add multiple STIX objects into memory\n", "ind2 = Indicator(description=\"Crusades stage 2 implant\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '70fa62fb218dd9d936ee570dbe531dfa4e7c128ff37e6af7a6a6b2485487e50a']\")\n", "ind3 = Indicator(description=\"Crusades stage 2 implant variant\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '31a45e777e4d58b97f4c43e38006f8cd6580ddabc4037905b2fad734712b582c']\")\n", "mal = Malware(labels=[\"rootkit\"], name= \"Alexios\")\n", "\n", "mem.add([ind2,ind3, mal])\n", "\n", "# for visual purposes\n", "print(mem.get(ind3.id))" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "malware",\n",
       "    "id": "malware--e8170e70-522f-4ec3-aa22-afb55bfad0b0",\n",
       "    "created": "2017-11-17T17:01:31.806Z",\n",
       "    "modified": "2017-11-17T17:01:31.806Z",\n",
       "    "name": "Alexios",\n",
       "    "labels": [\n",
       "        "rootkit"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import Filter\n", "\n", "mal = mem.query([Filter(\"labels\",\"=\", \"rootkit\")])[0]\n", "print(mal)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "report",\n",
       "    "id": "report--2add14d6-bbf3-4308-bb8e-226d314a08e4",\n",
       "    "created": "2017-05-08T18:34:08.042Z",\n",
       "    "modified": "2017-05-08T18:34:08.042Z",\n",
       "    "name": "The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.",\n",
       "    "published": "2017-05-08T10:24:11.011Z",\n",
       "    "object_refs": [\n",
       "        "malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4"\n",
       "    ],\n",
       "    "labels": [\n",
       "        "threat-report"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import Filter\n", "\n", "# add json formatted string to MemoryStore\n", "# Again, would NOT manually create json-formatted string\n", "# but taken as an output form from another source\n", "report = '{\"type\": \"report\",\"id\": \"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\",\"labels\": [\"threat-report\"], \"name\": \"The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.\", \"published\": \"2017-05-08T10:24:11.011Z\", \"object_refs\":[\"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\"], \"created\": \"2017-05-08T18:34:08.042Z\", \"modified\": \"2017-05-08T18:34:08.042Z\"}'\n", "\n", "mem.add(report)\n", "\n", "print(mem.get(\"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### load_from_file() and save_to_file()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "report",\n",
       "    "id": "report--2add14d6-bbf3-4308-bb8e-226d314a08e4",\n",
       "    "created": "2017-05-08T18:34:08.042Z",\n",
       "    "modified": "2017-05-08T18:34:08.042Z",\n",
       "    "name": "The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.",\n",
       "    "published": "2017-05-08T10:24:11.011Z",\n",
       "    "object_refs": [\n",
       "        "malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4"\n",
       "    ],\n",
       "    "labels": [\n",
       "        "threat-report"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem_2 = MemoryStore()\n", "\n", "# save (dump) all STIX content in MemoryStore to json file\n", "mem.save_to_file(\"path_to_target_file.json\")\n", "\n", "# load(add) STIX content from json file into MemoryStore\n", "mem_2.load_from_file(\"path_to_target_file.json\")\n", "\n", "report = mem_2.get(\"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\")\n", "\n", "# for visualpurposes\n", "print(report)" ] } ], "metadata": { "kernelspec": { "display_name": "cti-python-stix2", "language": "python", "name": "cti-python-stix2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 2 }