{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# Delete this cell to re-enable tracebacks\n",
    "import sys\n",
    "ipython = get_ipython()\n",
    "\n",
    "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n",
    "                   exception_only=False, running_compiled_code=False):\n",
    "    etype, value, tb = sys.exc_info()\n",
    "    return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n",
    "\n",
    "ipython.showtraceback = hide_traceback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# JSON output syntax highlighting\n",
    "from __future__ import print_function\n",
    "from pygments import highlight\n",
    "from pygments.lexers import JsonLexer\n",
    "from pygments.formatters import HtmlFormatter\n",
    "from IPython.display import HTML\n",
    "\n",
    "original_print = print\n",
    "\n",
    "def json_print(inpt):\n",
    "    string = str(inpt)\n",
    "    if string[0] == '{':\n",
    "        formatter = HtmlFormatter()\n",
    "        return HTML('<style type=\"text/css\">{}</style>{}'.format(\n",
    "                    formatter.get_style_defs('.highlight'),\n",
    "                    highlight(string, JsonLexer(), formatter)))\n",
    "    else:\n",
    "        original_print(inpt)\n",
    "\n",
    "print = json_print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataStore API\n",
    "\n",
    "CTI Python STIX2 features a new interface for pulling and pushing STIX2 content. The new interface consists of [DataStore](../api/stix2.sources.rst#stix2.sources.DataStore), [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource) and [DataSink](../api/stix2.sources.rst#stix2.sources.DataSink) constructs: a [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource) for pulling STIX2 content, a [DataSink](../api/stix2.sources.rst#stix2.sources.DataSink) for pushing STIX2 content, and a [DataStore](../api/stix2.sources.rst#stix2.sources.DataStore) for both pulling and pushing.\n",
    "\n",
    "The DataStore, [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource), [DataSink](../api/stix2.sources.rst#stix2.sources.DataSink) (collectively referred to as the \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then sublcassed by real DataStore suites. CTI Python STIX2 provides the DataStore suites of [FileSystem](../api/sources/stix2.sources.filesystem.rst), [Memory](../api/sources/stix2.sources.memory.rst), and [TAXII](../api/sources/stix2.sources.taxii.rst). Users are also encouraged to subclass the base classes and create their own custom DataStore suites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CompositeDataSource\n",
    "\n",
    "[CompositeDataSource](../api/stix2.sources.rst#stix2.sources.CompositeDataSource) is an available controller that can be used as a single interface to a set of defined [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource). The purpose of this controller is allow for the grouping of [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource) and making `get()`/`query()` calls to a set of DataSources in one API call. [CompositeDataSources](../api/stix2.sources.rst#stix2.sources.CompositeDataSource) can be used to organize/group [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource), federate `get()`/`all_versions()`/`query()` calls, and reduce user code.\n",
    "\n",
    "[CompositeDataSource](../api/stix2.sources.rst#stix2.sources.CompositeDataSource) is just a wrapper around a set of defined [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource) (e.g. [FileSystemSource](../api/sources/stix2.sources.filesystem.rst#stix2.sources.filesystem.FileSystemSource)) that federates `get()`/`all_versions()`/`query()` calls individually to each of the attached [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource) , collects the results from each [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource) and returns them.\n",
    "\n",
    "Filters can be attached to [CompositeDataSources](../api/stix2.sources.rst#stix2.sources.CompositeDataSource) just as they can be done to [DataStores](../api/stix2.sources.rst#stix2.sources.DataStore) and [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource). When `get()`/`all_versions()`/`query()` calls are made to the [CompositeDataSource](../api/stix2.sources.rst#stix2.sources.CompositeDataSource), it will pass along any query filters from the call and any of its own filters to the attached [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource). In addition, those [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource) may have their own attached filters as well. The effect is that all the filters are eventually combined when the `get()`/`all_versions()`/`query()` call is actually executed within a [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource). \n",
    "\n",
    "A [CompositeDataSource](../api/stix2.sources.rst#stix2.sources.CompositeDataSource) can also be attached to a [CompositeDataSource](../api/stix2.sources.rst#stix2.sources.CompositeDataSource) for multiple layers of grouped [DataSources](../api/stix2.sources.rst#stix2.sources.DataSource).\n",
    "\n",
    "\n",
    "### CompositeDataSource API\n",
    "\n",
    "#### CompositeDataSource Examples\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"type\": \"indicator\",\n",
      "    \"id\": \"indicator--797ae2b5-3f7a-44c5-8ecd-33ba22fdc2b5\",\n",
      "    \"created\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"modified\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"labels\": [\n",
      "        \"malicious-activity\"\n",
      "    ],\n",
      "    \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n",
      "    \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n",
      "    \"valid_from\": \"2017-10-04T19:27:41Z\",\n",
      "    \"kill_chain_phases\": [\n",
      "        {\n",
      "            \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n",
      "            \"phase_name\": \"delivery\"\n",
      "        }\n",
      "    ]\n",
      "}\n",
      "{\n",
      "    \"type\": \"indicator\",\n",
      "    \"id\": \"indicator--11913f42-2d52-4b9d-842f-94bf06819a66\",\n",
      "    \"created\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"modified\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"labels\": [\n",
      "        \"malicious-activity\"\n",
      "    ],\n",
      "    \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n",
      "    \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n",
      "    \"valid_from\": \"2017-10-04T19:27:41Z\",\n",
      "    \"kill_chain_phases\": [\n",
      "        {\n",
      "            \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n",
      "            \"phase_name\": \"delivery\"\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from taxii2client import Collection\n",
    "from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n",
    "\n",
    "# create FileSystemStore\n",
    "fs = FileSystemSource(\"/tmp/stix2_data\")\n",
    "\n",
    "# create TAXIICollectionSource\n",
    "colxn = Collection('https://test.freetaxii.com:8000/api1/collections/9cfa669c-ee94-4ece-afd2-f8edac37d8fd/')\n",
    "ts = TAXIICollectionSource(colxn)\n",
    "\n",
    "# add them both to the CompositeDataSource\n",
    "cs = CompositeDataSource()\n",
    "cs.add_data_sources([fs, ts])\n",
    "\n",
    "# get an object that is only in the filesystem\n",
    "ta = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n",
    "print(ta)\n",
    "\n",
    "# get an object that is only in the TAXII collection\n",
    "ind = cs.get('indicator--37a6a5de-a5b9-425a-903a-4ae9cbf1ff3f')\n",
    "print(ind)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filters\n",
    "\n",
    "The CTI Python STIX2 DataStore suites - [FileSystem](../api/sources/stix2.sources.filesystem.rst), [Memory](../api/sources/stix2.sources.memory.rst), and [TAXII](../api/sources/stix2.sources.taxii.rst) - all use the [Filters](../api/sources/stix2.sources.filters.rst) module to allow for the querying of STIX content. The basic functionality is that filters can be created and supplied everytime to calls to `query()`, and/or attached to a [DataStore](../api/stix2.sources.rst#stix2.sources.DataStore) so that every future query placed to that [DataStore](../api/stix2.sources.rst#stix2.sources.DataStore) is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from [DataStores](../api/stix2.sources.rst#stix2.sources.DataStore).\n",
    "\n",
    "Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). All properties of STIX2 objects can be filtered on. In addition, TAXII2 Filtering parameters for fields can also be used in filters.\n",
    "\n",
    "TAXII2 filter fields:\n",
    "\n",
    "* added_after\n",
    "* match[id]\n",
    "* match[type]\n",
    "* match[version]\n",
    "\n",
    "Supported operators:\n",
    "\n",
    "* =\n",
    "* !=\n",
    "* in\n",
    "* >\n",
    "* < \n",
    "* ```>=```\n",
    "* <=\n",
    "\n",
    "Value types of the property values must be one of these (Python) types:\n",
    "\n",
    "* bool\n",
    "* dict\n",
    "* float\n",
    "* int\n",
    "* list\n",
    "* str\n",
    "* tuple\n",
    "\n",
    "### Filter Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from stix2 import Filter\n",
    "\n",
    "# create filter for STIX objects that have external references to MITRE ATT&CK framework\n",
    "f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n",
    "\n",
    "# create filter for STIX objects that are not of SDO type Attack-Pattnern\n",
    "f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n",
    "\n",
    "# create filter for STIX objects that have the \"threat-report\" label\n",
    "f2 = Filter(\"labels\", \"in\", \"threat-report\")\n",
    "\n",
    "# create filter for STIX objects that have been modified past the timestamp\n",
    "f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n",
    "\n",
    "# create filter for STIX objects that have been revoked\n",
    "f4 = Filter(\"revoked\", \"=\", True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Filters to be applied to a query, they must be either supplied with the query call or attached a [DataStore](../api/stix2.sources.rst#stix2.sources.DataStore), more specifically to a [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource) whether that [DataSource](../api/stix2.sources.rst#stix2.sources.DataSource) is a part of a [DataStore](../api/stix2.sources.rst#stix2.sources.DataStore) or stands by itself. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n",
    "\n",
    "fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n",
    "fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n",
    "\n",
    "# attach filter to FileSystemStore\n",
    "fs.source.filters.add(f)\n",
    "\n",
    "# attach multiple filters to FileSystemStore\n",
    "fs.source.filters.update([f1,f2])\n",
    "\n",
    "# can also attach filters to a Source\n",
    "# attach multiple filters to FileSystemSource\n",
    "fs_source.filters.update([f3, f4])\n",
    "\n",
    "\n",
    "mem = MemoryStore()\n",
    "\n",
    "# As it is impractical to only use MemorySink or MemorySource,\n",
    "# attach a filter to a MemoryStore\n",
    "mem.source.filters.add(f)\n",
    "\n",
    "# attach multiple filters to a MemoryStore\n",
    "mem.source.filters.update([f1,f2])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}