311 lines
14 KiB
Plaintext
311 lines
14 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"collapsed": true,
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Delete this cell to re-enable tracebacks\n",
|
|
"import sys\n",
|
|
"ipython = get_ipython()\n",
|
|
"\n",
|
|
"def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n",
|
|
" exception_only=False, running_compiled_code=False):\n",
|
|
" etype, value, tb = sys.exc_info()\n",
|
|
" return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n",
|
|
"\n",
|
|
"ipython.showtraceback = hide_traceback"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"collapsed": true,
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# JSON output syntax highlighting\n",
|
|
"from __future__ import print_function\n",
|
|
"from pygments import highlight\n",
|
|
"from pygments.lexers import JsonLexer\n",
|
|
"from pygments.formatters import HtmlFormatter\n",
|
|
"from IPython.display import HTML\n",
|
|
"\n",
|
|
"original_print = print\n",
|
|
"\n",
|
|
"def json_print(inpt):\n",
|
|
" string = str(inpt)\n",
|
|
" if string[0] == '{':\n",
|
|
" formatter = HtmlFormatter()\n",
|
|
" return HTML('<style type=\"text/css\">{}</style>{}'.format(\n",
|
|
" formatter.get_style_defs('.highlight'),\n",
|
|
" highlight(string, JsonLexer(), formatter)))\n",
|
|
" else:\n",
|
|
" original_print(inpt)\n",
|
|
"\n",
|
|
"print = json_print"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Memory\n",
|
|
"\n",
|
|
"The Memory suite consists of [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore), [MemorySource](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemorySource), and [MemorySink](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemorySink). Under the hood, the Memory suite points to an in-memory dictionary. Similarly, the [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore) is a just a wrapper around a paired [MemorySource](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemorySource) and [MemorySink](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemorySink); as there is quite limited uses for just a [MemorySource](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemorySource) or a [MemorySink](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemorySink), it is recommended to always use [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore). The [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore) is intended for retrieving/searching and pushing STIX content to memory. It is important to note that all STIX content in memory is not backed up on the file system (disk), as that functionality is encompassed within the [FileSystemStore](../api/sources/stix2.sources.filesystem.rst#stix2.sources.filesystem.FileSystemStore). However, the Memory suite does provide some utility methods for saving and loading STIX content to disk. [MemoryStore.save_to_file()](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore.save_to_file) allows for saving all the STIX content that is in memory to a json file. [MemoryStore.load_from_file()](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore.load_from_file) allows for loading STIX content from a JSON-formatted file. \n",
|
|
"\n",
|
|
"\n",
|
|
"### Memory API\n",
|
|
"\n",
|
|
"A note on [load_from_file()](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore.load_from_file) and [save_to_file()](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore.save_to_file). These methods both add STIX content to an internal dictionary (maintained by [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore)). STIX content that is to be added can be in the following forms: Python STIX objects, Python dictionaries (of valid STIX objects or Bundles), JSON-encoded strings (of valid STIX objects or Bundles), or a (Python) list of any of the previously listed types. [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore) actually stores STIX content either as python STIX objects or as python dictionaries, reducing and converting any of the aforementioned types to one of those; and whatever form the STIX object is stored as, is how it will be returned as when queried or retrieved. Python STIX objects, and json-encoded strings (of STIX content) are stored as python STIX objects. Python dictionaries (of STIX objects) are stored as Python dictionaries. This is done, as can be efficiently supported, in order to return STIX content in the form it was added to the [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore). Also, for [load_from_file()](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore.load_from_file), STIX content is assumed to be in JSON form within the file, individually or in a Bundle. \n",
|
|
"\n",
|
|
"A note on [save_to_file()](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore.save_to_file). This method dumps all STIX content that is in [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore) to the specified file. The file format will be JSON, and the STIX content will be within a STIX Bundle. Note also that the the output form will be a JSON STIX Bundle regardless of the form that the individual STIX objects are stored (i.e. supplied) to the [MemoryStore](../api/sources/stix2.sources.memory.rst#stix2.sources.memory.MemoryStore). \n",
|
|
"\n",
|
|
"### Memory Examples\n",
|
|
"\n",
|
|
"#### MemoryStore"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"type\": \"indicator\",\n",
|
|
" \"id\": \"indicator--d91ef175-8a82-470a-a610-bbd2ee8a1516\",\n",
|
|
" \"created\": \"2017-09-29T19:52:16.930Z\",\n",
|
|
" \"modified\": \"2017-09-29T19:52:16.930Z\",\n",
|
|
" \"labels\": [\n",
|
|
" \"malicious-activity\"\n",
|
|
" ],\n",
|
|
" \"description\": \"Crusades C2 implant\",\n",
|
|
" \"pattern\": \"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\",\n",
|
|
" \"valid_from\": \"2017-09-29T19:52:16.930909Z\"\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from stix2 import MemoryStore, Indicator\n",
|
|
"\n",
|
|
"# create default MemoryStore\n",
|
|
"mem = MemoryStore()\n",
|
|
"\n",
|
|
"# insert newly created indicator into memory\n",
|
|
"ind = Indicator(description=\"Crusades C2 implant\",\n",
|
|
" labels=[\"malicious-activity\"],\n",
|
|
" pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n",
|
|
"\n",
|
|
"mem.add(ind)\n",
|
|
"\n",
|
|
"# for visual purposes\n",
|
|
"print(mem.get(ind.id))\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"type\": \"indicator\",\n",
|
|
" \"id\": \"indicator--79fdaad7-c461-49bb-ad1d-caa5e9c51c90\",\n",
|
|
" \"created\": \"2017-09-29T19:52:17.021Z\",\n",
|
|
" \"modified\": \"2017-09-29T19:52:17.021Z\",\n",
|
|
" \"labels\": [\n",
|
|
" \"malicious-activity\"\n",
|
|
" ],\n",
|
|
" \"description\": \"Crusades stage 2 implant variant\",\n",
|
|
" \"pattern\": \"[file:hashes.'SHA-256' = '31a45e777e4d58b97f4c43e38006f8cd6580ddabc4037905b2fad734712b582c']\",\n",
|
|
" \"valid_from\": \"2017-09-29T19:52:17.021728Z\"\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from stix2 import Malware\n",
|
|
"\n",
|
|
"# add multiple STIX objects into memory\n",
|
|
"ind2 = Indicator(description=\"Crusades stage 2 implant\",\n",
|
|
" labels=[\"malicious-activity\"],\n",
|
|
" pattern=\"[file:hashes.'SHA-256' = '70fa62fb218dd9d936ee570dbe531dfa4e7c128ff37e6af7a6a6b2485487e50a']\")\n",
|
|
"ind3 = Indicator(description=\"Crusades stage 2 implant variant\",\n",
|
|
" labels=[\"malicious-activity\"],\n",
|
|
" pattern=\"[file:hashes.'SHA-256' = '31a45e777e4d58b97f4c43e38006f8cd6580ddabc4037905b2fad734712b582c']\")\n",
|
|
"mal = Malware(labels=[\"rootkit\"], name= \"Alexios\")\n",
|
|
"\n",
|
|
"mem.add([ind2,ind3, mal])\n",
|
|
"\n",
|
|
"# for visual purposes\n",
|
|
"print(mem.get(ind3.id))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"-----------------------\n",
|
|
"{'name': 'Urban2', 'created': '2017-09-12T13:26:18.023Z', 'labels': ['rootkit'], 'modified': '2017-09-12T13:26:18.023Z', 'type': 'malware', 'id': 'malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4'}\n",
|
|
"-----------------------\n",
|
|
"{\n",
|
|
" \"type\": \"malware\",\n",
|
|
" \"id\": \"malware--2b3dd412-18a5-4e81-8742-4977068eb3eb\",\n",
|
|
" \"created\": \"2017-09-29T19:52:17.028Z\",\n",
|
|
" \"modified\": \"2017-09-29T19:52:17.028Z\",\n",
|
|
" \"name\": \"Alexios\",\n",
|
|
" \"labels\": [\n",
|
|
" \"rootkit\"\n",
|
|
" ]\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from stix2 import Filter\n",
|
|
"\n",
|
|
"# add dictionary (of STIX object) to MemoryStore\n",
|
|
"# (this dict would assumably come from output of another source,\n",
|
|
"# i.e. a loaded json file, NOT manually created as done here for sample purposes)\n",
|
|
"\n",
|
|
"malware = {\n",
|
|
" \"type\": \"malware\",\n",
|
|
" \"id\" : \"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\",\n",
|
|
" \"labels\": [\"rootkit\"],\n",
|
|
" \"name\": \"Urban2\",\n",
|
|
" \"created\": \"2017-09-12T13:26:18.023Z\",\n",
|
|
" \"modified\": \"2017-09-12T13:26:18.023Z\"\n",
|
|
"}\n",
|
|
"\n",
|
|
"mem.add(malware)\n",
|
|
"\n",
|
|
"results = mem.query([Filter(\"labels\",\"=\", \"rootkit\")])\n",
|
|
"for r in results:\n",
|
|
" # note that python STIX objects are pretty-printed\n",
|
|
" # due to some python dunder method magic, but normal\n",
|
|
" # python dictionaries are not by default. Thus the\n",
|
|
" # python STIX objects and python STIX dictionaries\n",
|
|
" # that match the above query can be easily identified visually\n",
|
|
" print(\"-----------------------\")\n",
|
|
" print(r)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"type\": \"report\",\n",
|
|
" \"id\": \"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\",\n",
|
|
" \"created\": \"2017-05-08T18:34:08.042Z\",\n",
|
|
" \"modified\": \"2017-05-08T18:34:08.042Z\",\n",
|
|
" \"name\": \"The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.\",\n",
|
|
" \"published\": \"2017-05-08T10:24:11.011Z\",\n",
|
|
" \"object_refs\": [\n",
|
|
" \"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\"\n",
|
|
" ],\n",
|
|
" \"labels\": [\n",
|
|
" \"threat-report\"\n",
|
|
" ]\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from stix2 import Filter\n",
|
|
"\n",
|
|
"# add json formatted string to MemoryStore\n",
|
|
"# Again, would NOT manual create json-formatted string\n",
|
|
"# but taken as an output form from another source\n",
|
|
"report = '{\"type\": \"report\",\"id\": \"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\",\"labels\": [\"threat-report\"], \"name\": \"The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.\", \"published\": \"2017-05-08T10:24:11.011Z\", \"object_refs\":[\"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\"], \"created\": \"2017-05-08T18:34:08.042Z\", \"modified\": \"2017-05-08T18:34:08.042Z\"}'\n",
|
|
"\n",
|
|
"mem.add(report)\n",
|
|
"\n",
|
|
"print(mem.get(\"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### load_from_file() and save_to_file()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{u'name': u'The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.', u'created': u'2017-05-08T18:34:08.042Z', u'labels': [u'threat-report'], u'modified': u'2017-05-08T18:34:08.042Z', u'object_refs': [u'malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4'], u'published': u'2017-05-08T10:24:11.011Z', u'type': u'report', u'id': u'report--2add14d6-bbf3-4308-bb8e-226d314a08e4'}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"mem_2 = MemoryStore()\n",
|
|
"\n",
|
|
"# save (dump) all STIX content in MemoryStore to json file\n",
|
|
"mem.save_to_file(\"path_to_target_file.json\")\n",
|
|
"\n",
|
|
"# load(add) STIX content from json file into MemoryStore\n",
|
|
"mem_2.load_from_file(\"path_to_target_file.json\")\n",
|
|
"\n",
|
|
"report = mem_2.get(\"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\")\n",
|
|
"\n",
|
|
"# for visualpurposes\n",
|
|
"# Note: Since STIX content was added to MemoryStore as json,\n",
|
|
"# it is maintained as python dictionaries ( as opposed to STIX objects)\n",
|
|
"print(report)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 2",
|
|
"language": "python",
|
|
"name": "python2"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 2
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython2",
|
|
"version": "2.7.12"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|