{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " value.__cause__ = None # suppress chained exceptions\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer, TextLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import display, HTML\n", "from IPython.core.interactiveshell import InteractiveShell\n", "\n", "InteractiveShell.ast_node_interactivity = \"all\"\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " formatter = HtmlFormatter()\n", " if string[0] == '{':\n", " lexer = JsonLexer()\n", " else:\n", " lexer = TextLexer()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, lexer, formatter)))\n", "\n", "globals()['print'] = json_print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# DataStore API\n", "\n", "The ``stix2`` library features an interface for pulling and pushing STIX 2 content. This interface consists of [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin), [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) and [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) constructs: a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) for pulling STIX 2 content, a [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) for pushing STIX 2 content, and a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) for both pulling and pushing.\n", "\n", "The DataStore, [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource), [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) (collectively referred to as the \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then subclassed by real DataStore suites. The ``stix2`` library provides the DataStore suites of [FileSystem](../api/datastore/stix2.datastore.filesystem.rst), [Memory](../api/datastore/stix2.datastore.memory.rst), and [TAXII](../api/datastore/stix2.datastore.taxii.rst). Users are also encouraged to subclass the base classes and create their own custom DataStore suites." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CompositeDataSource\n", "\n", "[CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) is an available controller that can be used as a single interface to a set of defined [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). The purpose of this controller is allow for the grouping of [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) and making `get()`/`query()` calls to a set of DataSources in one API call. [CompositeDataSources](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) can be used to organize/group [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource), federate `get()`/`all_versions()`/`query()` calls, and reduce user code.\n", "\n", "[CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) is just a wrapper around a set of defined [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) (e.g. [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource)) that federates `get()`/`all_versions()`/`query()` calls individually to each of the attached [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) , collects the results from each [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) and returns them.\n", "\n", "Filters can be attached to [CompositeDataSources](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) just as they can be done to [DataStores](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) and [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). When `get()`/`all_versions()`/`query()` calls are made to the [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource), it will pass along any query filters from the call and any of its own filters to the attached [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). In addition, those [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) may have their own attached filters as well. The effect is that all the filters are eventually combined when the `get()`/`all_versions()`/`query()` call is actually executed within a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource). \n", "\n", "A [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) can also be attached to a [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) for multiple layers of grouped [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource).\n", "\n", "\n", "### CompositeDataSource API\n", "\n", "#### CompositeDataSource Examples\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "intrusion-set",\n",
       "    "spec_version": "2.1",\n",
       "    "id": "intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a",\n",
       "    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",\n",
       "    "created": "2017-05-31T21:31:53.197Z",\n",
       "    "modified": "2017-05-31T21:31:53.197Z",\n",
       "    "name": "DragonOK",\n",
       "    "description": "DragonOK is a threat group that has targeted Japanese organizations with phishing emails. Due to overlapping TTPs, including similar custom tools, DragonOK is thought to have a direct or indirect relationship with the threat group Moafee. [[Citation: Operation Quantum Entanglement]][[Citation: Symbiotic APT Groups]] It is known to use a variety of malware, including Sysget/HelloBridge, PlugX, PoisonIvy, FormerFirstRat, NFlog, and NewCT. [[Citation: New DragonOK]]",\n",
       "    "aliases": [\n",
       "        "DragonOK"\n",
       "    ],\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "mitre-attack",\n",
       "            "url": "https://attack.mitre.org/wiki/Group/G0017",\n",
       "            "external_id": "G0017"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Operation Quantum Entanglement",\n",
       "            "description": "Haq, T., Moran, N., Vashisht, S., Scott, M. (2014, September). OPERATION QUANTUM ENTANGLEMENT. Retrieved November 4, 2015.",\n",
       "            "url": "https://www.fireeye.com/content/dam/fireeye-www/global/en/current-threats/pdfs/wp-operation-quantum-entanglement.pdf"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Symbiotic APT Groups",\n",
       "            "description": "Haq, T. (2014, October). An Insight into Symbiotic APT Groups. Retrieved November 4, 2015.",\n",
       "            "url": "https://dl.mandiant.com/EE/library/MIRcon2014/MIRcon%202014%20R&D%20Track%20Insight%20into%20Symbiotic%20APT.pdf"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "New DragonOK",\n",
       "            "description": "Miller-Osborn, J., Grunzweig, J.. (2015, April). Unit 42 Identifies New DragonOK Backdoor Malware Deployed Against Japanese Targets. Retrieved November 4, 2015.",\n",
       "            "url": "http://researchcenter.paloaltonetworks.com/2015/04/unit-42-identifies-new-dragonok-backdoor-malware-deployed-against-japanese-targets/"\n",
       "        }\n",
       "    ],\n",
       "    "object_marking_refs": [\n",
       "        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
{\n",
       "    "type": "indicator",\n",
       "    "spec_version": "2.1",\n",
       "    "id": "indicator--a740531e-63ff-4e49-a9e1-a0a3eed0e3e7",\n",
       "    "created": "2017-11-13T07:00:24.000Z",\n",
       "    "modified": "2017-11-13T07:00:24.000Z",\n",
       "    "name": "Ransomware IP Blocklist",\n",
       "    "description": "IP Blocklist address from abuse.ch",\n",
       "    "indicator_types": [\n",
       "        "malicious-activity",\n",
       "        "Ransomware",\n",
       "        "Botnet",\n",
       "        "C&C"\n",
       "    ],\n",
       "    "pattern": "[ ipv4-addr:value = '91.237.247.24' ]",\n",
       "    "pattern_type": "stix",\n",
       "    "pattern_version": "2.1",\n",
       "    "valid_from": "2017-11-13T07:00:24Z",\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "abuse.ch",\n",
       "            "url": "https://ransomwaretracker.abuse.ch/blocklist/"\n",
       "        }\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from taxii2client import Collection\n", "from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n", "\n", "# create FileSystemStore\n", "fs = FileSystemSource(\"/tmp/stix2_source\")\n", "\n", "# create TAXIICollectionSource\n", "colxn = Collection('http://127.0.0.1:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/', user=\"user1\", password=\"Password1\")\n", "ts = TAXIICollectionSource(colxn)\n", "\n", "# add them both to the CompositeDataSource\n", "cs = CompositeDataSource()\n", "cs.add_data_sources([fs,ts])\n", "\n", "# get an object that is only in the filesystem\n", "intrusion_set = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n", "print(intrusion_set)\n", "\n", "# get an object that is only in the TAXII collection\n", "ind = cs.get('indicator--a740531e-63ff-4e49-a9e1-a0a3eed0e3e7')\n", "print(ind)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Filters\n", "\n", "The ``stix2`` DataStore suites - [FileSystem](../api/datastore/stix2.datastore.filesystem.rst), [Memory](../api/datastore/stix2.datastore.memory.rst), and [TAXII](../api/datastore/stix2.datastore.taxii.rst) - all use the [Filters](../api/datastore/stix2.datastore.filters.rst) module to allow for the querying of STIX content. Filters can be used to explicitly include or exclude results with certain criteria. For example:\n", "\n", "* only trust content from a set of object creators\n", "* exclude content from certain (untrusted) object creators\n", "* only include content with a confidence above a certain threshold (once confidence is added to STIX 2)\n", "* only return content that can be shared with external parties (e.g. only content that has TLP:GREEN markings)\n", "\n", "Filters can be created and supplied with every call to `query()`, and/or attached to a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) so that every future query placed to that [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from [DataStores](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin).\n", "\n", "Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). All properties of STIX 2 objects can be filtered on. In addition, TAXII 2 Filtering parameters for fields can also be used in filters.\n", "\n", "TAXII2 filter fields:\n", "\n", "* added_after\n", "* id\n", "* spec_version\n", "* type\n", "* version\n", "\n", "Supported operators:\n", "\n", "* =\n", "* !=\n", "* in\n", "* ```>```\n", "* < \n", "* ```>=```\n", "* <=\n", "* contains\n", "\n", "Value types of the property values must be one of these (Python) types:\n", "\n", "* bool\n", "* dict\n", "* float\n", "* int\n", "* list\n", "* str\n", "* tuple\n", "\n", "### Filter Examples" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "import sys\n", "from stix2 import Filter\n", "\n", "# create filter for STIX objects that have external references to MITRE ATT&CK framework\n", "f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n", "\n", "# create filter for STIX objects that are not of SDO type Attack-Pattnern\n", "f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n", "\n", "# create filter for STIX objects that have the \"threat-report\" label\n", "f2 = Filter(\"labels\", \"in\", \"threat-report\")\n", "\n", "# create filter for STIX objects that have been modified past the timestamp\n", "f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n", "\n", "# create filter for STIX objects that have been revoked\n", "f4 = Filter(\"revoked\", \"=\", True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For Filters to be applied to a query, they must be either supplied with the query call or attached to a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin), more specifically to a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) whether that [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) is a part of a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) or stands by itself. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n", "\n", "fs = FileSystemStore(\"/tmp/stix2_store\")\n", "fs_source = FileSystemSource(\"/tmp/stix2_source\")\n", "\n", "# attach filter to FileSystemStore\n", "fs.source.filters.add(f)\n", "\n", "# attach multiple filters to FileSystemStore\n", "fs.source.filters.add([f1,f2])\n", "\n", "# can also attach filters to a Source\n", "# attach multiple filters to FileSystemSource\n", "fs_source.filters.add([f3, f4])\n", "\n", "\n", "mem = MemoryStore()\n", "\n", "# As it is impractical to only use MemorySink or MemorySource,\n", "# attach a filter to a MemoryStore\n", "mem.source.filters.add(f)\n", "\n", "# attach multiple filters to a MemoryStore\n", "mem.source.filters.add([f1,f2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note: The `defanged` property is now always included (implicitly) for STIX 2.1 Cyber Observable Objects (SCOs)**\n", "\n", "This is important to remember if you are writing a filter that involves checking the `objects` property of a STIX 2.1 `ObservedData` object. If any of the objects associated with the `objects` property are STIX 2.1 SCOs, then your filter must include the `defanged` property. For an example, refer to `filters[14]` & `filters[15]` in stix2/test/v21/test_datastore_filters.py " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## De-Referencing Relationships\n", "\n", "Given a STIX object, there are several ways to find other STIX objects related to it. To illustrate this, let's first create a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) and add some objects and relationships." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from stix2 import Campaign, Identity, Indicator, Malware, Relationship\n", "\n", "mem = MemoryStore()\n", "cam = Campaign(name='Charge', description='Attack!')\n", "idy = Identity(name='John Doe', identity_class=\"individual\")\n", "ind = Indicator(pattern_type='stix', pattern=\"[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']\")\n", "mal = Malware(name=\"Cryptolocker\", is_family=False, created_by_ref=idy)\n", "rel1 = Relationship(ind, 'indicates', mal,)\n", "rel2 = Relationship(mal, 'targets', idy)\n", "rel3 = Relationship(cam, 'uses', mal)\n", "mem.add([cam, idy, ind, mal, rel1, rel2, rel3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If a STIX object has a `created_by_ref` property, you can use the [creator_of()](../api/stix2.datastore.rst#stix2.datastore.DataSource.creator_of) method to retrieve the [Identity](../api/v21/stix2.v21.sdo.rst#stix2.v21.sdo.Identity) object that created it." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "identity",\n",
       "    "spec_version": "2.1",\n",
       "    "id": "identity--a2628104-e357-44a0-b16f-d5f36c0fd0ec",\n",
       "    "created": "2020-06-26T13:59:21.924055Z",\n",
       "    "modified": "2020-06-26T13:59:21.924055Z",\n",
       "    "name": "John Doe",\n",
       "    "identity_class": "individual"\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(mem.creator_of(mal))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use the [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) method to retrieve all the relationship objects that reference a STIX object." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rels = mem.relationships(mal)\n", "len(rels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can limit it to only specific relationship types:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Relationship(type='relationship', spec_version='2.1', id='relationship--ef837187-773c-41e4-ae86-c66189a832f5', created='2020-06-26T13:59:21.929336Z', modified='2020-06-26T13:59:21.929336Z', relationship_type='indicates', source_ref='indicator--9f10f6f2-b93d-488e-be35-72c3ec1087c3', target_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b')]" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.relationships(mal, relationship_type='indicates')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can limit it to only relationships where the given object is the source:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Relationship(type='relationship', spec_version='2.1', id='relationship--43f5f7a7-8a99-4bbf-8d93-e6f3fd2951a3', created='2020-06-26T13:59:21.937132Z', modified='2020-06-26T13:59:21.937132Z', relationship_type='targets', source_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b', target_ref='identity--a2628104-e357-44a0-b16f-d5f36c0fd0ec')]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.relationships(mal, source_only=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And you can limit it to only relationships where the given object is the target:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Relationship(type='relationship', spec_version='2.1', id='relationship--ef837187-773c-41e4-ae86-c66189a832f5', created='2020-06-26T13:59:21.929336Z', modified='2020-06-26T13:59:21.929336Z', relationship_type='indicates', source_ref='indicator--9f10f6f2-b93d-488e-be35-72c3ec1087c3', target_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b'),\n", " Relationship(type='relationship', spec_version='2.1', id='relationship--596c196f-2f05-4584-b643-2186b327a94f', created='2020-06-26T13:59:21.937354Z', modified='2020-06-26T13:59:21.937354Z', relationship_type='uses', source_ref='campaign--d359f872-7e44-4090-8e08-c5bd10bc5f2d', target_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b')]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.relationships(mal, target_only=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, you can retrieve all STIX objects related to a given STIX object using [related_to()](../api/stix2.datastore.rst#stix2.datastore.DataSource.related_to). This calls [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) but then performs the extra step of getting the objects that these Relationships point to. [related_to()](../api/stix2.datastore.rst#stix2.datastore.DataSource.related_to) takes all the same arguments that [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) does." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Campaign(type='campaign', spec_version='2.1', id='campaign--d359f872-7e44-4090-8e08-c5bd10bc5f2d', created='2020-06-26T13:59:21.923792Z', modified='2020-06-26T13:59:21.923792Z', name='Charge', description='Attack!')]" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.related_to(mal, target_only=True, relationship_type='uses')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.0a6" } }, "nbformat": 4, "nbformat_minor": 2 }