{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true, "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true, "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import HTML\n", "\n", "original_print = print\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " if string[0] == '{':\n", " formatter = HtmlFormatter()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, JsonLexer(), formatter)))\n", " else:\n", " original_print(inpt)\n", "\n", "print = json_print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# DataStore API\n", "\n", "CTI Python STIX2 features a new interface for pulling and pushing STIX2 content. The new interface consists of DataStore, DataSource and DataSink constructs: a DataSource for pulling STIX2 content, a DataSink for pushing STIX2 content, and a DataStore for pulling/pushing.\n", "\n", "The DataStore, DataSource, DataSink (referred to as \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then sublcassed into real DataStore suite(s). CTI Python STIX2 provides for the DataStore suites of **FileSystem**, **Memory**, and **TAXII**. Users are also encrouraged subclassing the base Data suite and creating their own custom DataStore suites." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CompositeDataSource\n", "\n", "**CompositeDataSource** is an available controller that can be used as a single interface to a set of defined DataSources. The purpose of this controller is allow for the grouping of **DataSources** and making get/query calls to a set of DataSources in one API call. **CompositeDataSource** can be used to organize/group **DataSources**, federate get()/all_versions()/query() calls, and reduce user code.\n", "\n", "**CompositeDataSource** is just a wrapper around a set of defined **DataSources** (e.g. FileSystemSource) that federates get()/all_versions()/query() calls individually to each of the attached **DataSources** , collects the results from each **DataSource** and returns them.\n", "\n", "Filters can be attached to **CompositeDataSources** just as they can be done to **DataStores** and **DataSources**. When get()/all_versions()/query() calls are made to the **CompositeDataSource**, it will pass along any query filters from the call and any of its own filters to the attached **DataSources**. To which, those attached **DataSources** may have their own attached filters as well. The effect is that all the filters are eventually combined when the get()/all_versions()/query() call is actually executed within a **DataSource**. \n", "\n", "A **CompositeDataSource** can also be attached to a **CompositeDataSource** for multiple layers of grouped **DataSources**.\n", "\n", "\n", "### CompositeDataSource API\n", "\n", "#### CompositeDataSource Examples\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"type\": \"indicator\",\n", " \"id\": \"indicator--797ae2b5-3f7a-44c5-8ecd-33ba22fdc2b5\",\n", " \"created\": \"2017-10-04T19:27:41.000Z\",\n", " \"modified\": \"2017-10-04T19:27:41.000Z\",\n", " \"labels\": [\n", " \"malicious-activity\"\n", " ],\n", " \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n", " \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n", " \"valid_from\": \"2017-10-04T19:27:41Z\",\n", " \"kill_chain_phases\": [\n", " {\n", " \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n", " \"phase_name\": \"delivery\"\n", " }\n", " ]\n", "}\n", "{\n", " \"type\": \"indicator\",\n", " \"id\": \"indicator--11913f42-2d52-4b9d-842f-94bf06819a66\",\n", " \"created\": \"2017-10-04T19:27:41.000Z\",\n", " \"modified\": \"2017-10-04T19:27:41.000Z\",\n", " \"labels\": [\n", " \"malicious-activity\"\n", " ],\n", " \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n", " \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n", " \"valid_from\": \"2017-10-04T19:27:41Z\",\n", " \"kill_chain_phases\": [\n", " {\n", " \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n", " \"phase_name\": \"delivery\"\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "from taxii2client import Collection\n", "from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n", "\n", "# create FileSystemStore\n", "fs = FileSystemSource(\"/tmp/stix2_data\")\n", "\n", "# create TAXIICollectionSource\n", "colxn = Collection('https://test.freetaxii.com:8000/api1/collections/9cfa669c-ee94-4ece-afd2-f8edac37d8fd/')\n", "ts = TAXIICollectionSource(colxn)\n", "\n", "# add them both to the CompositeDataSource\n", "cs = CompositeDataSource()\n", "cs.add_data_sources([fs, ts])\n", "\n", "# get an object that is only in the filesystem\n", "ta = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n", "print(ta)\n", "\n", "# get an object that is only in the TAXII collection\n", "ind = cs.get('indicator--37a6a5de-a5b9-425a-903a-4ae9cbf1ff3f')\n", "print(ind)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Filters\n", "\n", "The CTI Python STIX2 **DataStore** suites - **FileSystem**, **Memory** and **TAXII** - all use the **Filters** module to allow for the querying of STIX content. The basic functionality is that filters can be created and supplied everytime to calls to **query()**, and/or attached to a **DataStore** so that every future query placed to that **DataStore** is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from **DataStores**.\n", "\n", "Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). Currently, CTI Python STIX2 supports **ONLY** STIX 2 object common properties and TAXII2 Filtering parameters for fields to filter on:\n", "\n", "Fields\n", "\n", "(STIX2 Object Common Properties)\n", "\n", "* created\n", "* created_by_ref\n", "* external_references.source_name\n", "* external_references.description\n", "* external_references.url\n", "* external_references.external_id\n", "* granular_markings.marking_ref\n", "* granular_markings.selectors\n", "* id\n", "* labels\n", "* modified\n", "* object_marking_refs\n", "* revoked\n", "* type\n", "\n", "(TAXII2 filter fields)\n", "\n", "* added_after\n", "* match[id]\n", "* match[type]\n", "* match[version]\n", "\n", "Supported operators on above properties:\n", "\n", "* =\n", "* !=\n", "* in\n", "* >\n", "* < \n", "* ```>=```\n", "* <=\n", "\n", "Value types of the common property values must be one of these (python) types:\n", "\n", "* bool\n", "* dict\n", "* float\n", "* int\n", "* list\n", "* str\n", "* tuple\n", "\n", "### Filter Examples" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import sys\n", "from stix2 import Filter\n", "\n", "# create filter for STIX objects that have external references to MITRE ATT&CK framework\n", "f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n", "\n", "# create filter for STIX objects that are not of SDO type Attack-Pattnern\n", "f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n", "\n", "# create filter for STIX objects that have the \"threat-report\" label\n", "f2 = Filter(\"labels\", \"in\", \"threat-report\")\n", "\n", "# create filter for STIX objects that have been modified past the timestamp\n", "f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n", "\n", "# create filter for STIX objects that have been revoked\n", "f4 = Filter(\"revoked\", \"=\", True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For Filters to be applied to a query, they must be either supplied with the query call or attached a **DataStore**, more specifically to **DataSource** whether that **DataSource** is a part of a **DataStore** or stands by itself. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n", "\n", "fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n", "fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n", "\n", "# attach filter to FileSystemStore\n", "fs.source.filters.add(f)\n", "\n", "# attach multiple filters to FileSystemStore\n", "fs.source.filters.update([f1,f2])\n", "\n", "# can also attach filters to a Source\n", "# attach multiple filters to FileSystemSource\n", "fs_source.filters.update([f3, f4])\n", "\n", "\n", "mem = MemoryStore()\n", "\n", "# As it is impractical to only use MemorySink or MemorySource,\n", "# attach a filter to a MemoryStore\n", "mem.source.filters.add(f)\n", "\n", "# attach multiple filters to a MemoryStore\n", "mem.source.filters.update([f1,f2])" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 2 }