cti-python-stix2/docs/guide/datastore.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# Delete this cell to re-enable tracebacks\n",
    "import sys\n",
    "ipython = get_ipython()\n",
    "\n",
    "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n",
    "                   exception_only=False, running_compiled_code=False):\n",
    "    etype, value, tb = sys.exc_info()\n",
    "    return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n",
    "\n",
    "ipython.showtraceback = hide_traceback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# JSON output syntax highlighting\n",
    "from __future__ import print_function\n",
    "from pygments import highlight\n",
    "from pygments.lexers import JsonLexer\n",
    "from pygments.formatters import HtmlFormatter\n",
    "from IPython.display import HTML\n",
    "\n",
    "original_print = print\n",
    "\n",
    "def json_print(inpt):\n",
    "    string = str(inpt)\n",
    "    if string[0] == '{':\n",
    "        formatter = HtmlFormatter()\n",
    "        return HTML('<style type=\"text/css\">{}</style>{}'.format(\n",
    "                    formatter.get_style_defs('.highlight'),\n",
    "                    highlight(string, JsonLexer(), formatter)))\n",
    "    else:\n",
    "        original_print(inpt)\n",
    "\n",
    "print = json_print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataStore API\n",
    "\n",
    "CTI Python STIX2 features a new interface for pulling and pushing STIX2 content. The new interface consists of DataStore, DataSource and DataSink constructs: a DataSource for pulling STIX2 content, a DataSink for pushing STIX2 content, and a DataStore for pulling/pushing.\n",
    "\n",
    "The DataStore, DataSource, DataSink (referred to as \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then sublcassed into real DataStore suite(s). CTI Python STIX2 provides for the DataStore suites of **FileSystem**, **Memory**, and **TAXII**. Users are also encrouraged subclassing the base Data suite and creating their own custom DataStore suites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CompositeDataSource\n",
    "\n",
    "**CompositeDataSource** is an available controller that can be used as a single interface to a set of defined DataSources. The purpose of this controller is allow for the grouping of **DataSources** and making get/query calls to a set of DataSources in one API call. **CompositeDataSource** can be used to organize/group **DataSources**, federate get()/all_versions()/query() calls, and reduce user code.\n",
    "\n",
    "**CompositeDataSource** is just a wrapper around a set of defined **DataSources** (e.g. FileSystemSource) that federates get()/all_versions()/query() calls individually to each of the attached **DataSources** , collects the results from each **DataSource** and returns them.\n",
    "\n",
    "Filters can be attached to **CompositeDataSources** just as they can be done to **DataStores** and **DataSources**. When get()/all_versions()/query() calls are made to the **CompositeDataSource**, it will pass along any query filters from the call and any of its own filters to the attached **DataSources**. To which, those attached **DataSources** may have their own attached filters as well. The effect is that all the filters are eventually combined when the get()/all_versions()/query() call is actually executed within a **DataSource**. \n",
    "\n",
    "A **CompositeDataSource** can also be attached to a **CompositeDataSource** for multiple layers of grouped **DataSources**.\n",
    "\n",
    "\n",
    "### CompositeDataSource API\n",
    "\n",
    "#### CompositeDataSource Examples\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"type\": \"indicator\",\n",
      "    \"id\": \"indicator--797ae2b5-3f7a-44c5-8ecd-33ba22fdc2b5\",\n",
      "    \"created\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"modified\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"labels\": [\n",
      "        \"malicious-activity\"\n",
      "    ],\n",
      "    \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n",
      "    \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n",
      "    \"valid_from\": \"2017-10-04T19:27:41Z\",\n",
      "    \"kill_chain_phases\": [\n",
      "        {\n",
      "            \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n",
      "            \"phase_name\": \"delivery\"\n",
      "        }\n",
      "    ]\n",
      "}\n",
      "{\n",
      "    \"type\": \"indicator\",\n",
      "    \"id\": \"indicator--11913f42-2d52-4b9d-842f-94bf06819a66\",\n",
      "    \"created\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"modified\": \"2017-10-04T19:27:41.000Z\",\n",
      "    \"labels\": [\n",
      "        \"malicious-activity\"\n",
      "    ],\n",
      "    \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n",
      "    \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n",
      "    \"valid_from\": \"2017-10-04T19:27:41Z\",\n",
      "    \"kill_chain_phases\": [\n",
      "        {\n",
      "            \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n",
      "            \"phase_name\": \"delivery\"\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from taxii2client import Collection\n",
    "from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n",
    "\n",
    "# create FileSystemStore\n",
    "fs = FileSystemSource(\"/tmp/stix2_data\")\n",
    "\n",
    "# create TAXIICollectionSource\n",
    "colxn = Collection('https://test.freetaxii.com:8000/api1/collections/9cfa669c-ee94-4ece-afd2-f8edac37d8fd/')\n",
    "ts = TAXIICollectionSource(colxn)\n",
    "\n",
    "# add them both to the CompositeDataSource\n",
    "cs = CompositeDataSource()\n",
    "cs.add_data_sources([fs, ts])\n",
    "\n",
    "# get an object that is only in the filesystem\n",
    "ta = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n",
    "print(ta)\n",
    "\n",
    "# get an object that is only in the TAXII collection\n",
    "ind = cs.get('indicator--37a6a5de-a5b9-425a-903a-4ae9cbf1ff3f')\n",
    "print(ind)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filters\n",
    "\n",
    "The CTI Python STIX2 **DataStore** suites - **FileSystem**, **Memory** and **TAXII** - all use the **Filters** module to allow for the querying of STIX content. The basic functionality is that filters can be created and supplied everytime to calls to **query()**, and/or attached to a **DataStore** so that every future query placed to that **DataStore** is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from **DataStores**.\n",
    "\n",
    "Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). Currently, CTI Python STIX2 supports **ONLY** STIX 2 object common properties and TAXII2 Filtering parameters for fields to filter on:\n",
    "\n",
    "Fields\n",
    "\n",
    "(STIX2 Object Common Properties)\n",
    "\n",
    "* created\n",
    "* created_by_ref\n",
    "* external_references.source_name\n",
    "* external_references.description\n",
    "* external_references.url\n",
    "* external_references.external_id\n",
    "* granular_markings.marking_ref\n",
    "* granular_markings.selectors\n",
    "* id\n",
    "* labels\n",
    "* modified\n",
    "* object_marking_refs\n",
    "* revoked\n",
    "* type\n",
    "\n",
    "(TAXII2 filter fields)\n",
    "\n",
    "* added_after\n",
    "* match[id]\n",
    "* match[type]\n",
    "* match[version]\n",
    "\n",
    "Supported operators on above properties:\n",
    "\n",
    "* =\n",
    "* !=\n",
    "* in\n",
    "* >\n",
    "* < \n",
    "* ```>=```\n",
    "* <=\n",
    "\n",
    "Value types of the common property values must be one of these (python) types:\n",
    "\n",
    "* bool\n",
    "* dict\n",
    "* float\n",
    "* int\n",
    "* list\n",
    "* str\n",
    "* tuple\n",
    "\n",
    "### Filter Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from stix2 import Filter\n",
    "\n",
    "# create filter for STIX objects that have external references to MITRE ATT&CK framework\n",
    "f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n",
    "\n",
    "# create filter for STIX objects that are not of SDO type Attack-Pattnern\n",
    "f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n",
    "\n",
    "# create filter for STIX objects that have the \"threat-report\" label\n",
    "f2 = Filter(\"labels\", \"in\", \"threat-report\")\n",
    "\n",
    "# create filter for STIX objects that have been modified past the timestamp\n",
    "f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n",
    "\n",
    "# create filter for STIX objects that have been revoked\n",
    "f4 = Filter(\"revoked\", \"=\", True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Filters to be applied to a query, they must be either supplied with the query call or attached a **DataStore**, more specifically to **DataSource** whether that **DataSource** is a part of a **DataStore** or stands by itself. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n",
    "\n",
    "fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n",
    "fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n",
    "\n",
    "# attach filter to FileSystemStore\n",
    "fs.source.filters.add(f)\n",
    "\n",
    "# attach multiple filters to FileSystemStore\n",
    "fs.source.filters.update([f1,f2])\n",
    "\n",
    "# can also attach filters to a Source\n",
    "# attach multiple filters to FileSystemSource\n",
    "fs_source.filters.update([f3, f4])\n",
    "\n",
    "\n",
    "mem = MemoryStore()\n",
    "\n",
    "# As it is impractical to only use MemorySink or MemorySource,\n",
    "# attach a filter to a MemoryStore\n",
    "mem.source.filters.add(f)\n",
    "\n",
    "# attach multiple filters to a MemoryStore\n",
    "mem.source.filters.update([f1,f2])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}