308 lines
11 KiB
Plaintext
308 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"collapsed": true,
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Delete this cell to re-enable tracebacks\n",
|
|
"import sys\n",
|
|
"ipython = get_ipython()\n",
|
|
"\n",
|
|
"def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n",
|
|
" exception_only=False, running_compiled_code=False):\n",
|
|
" etype, value, tb = sys.exc_info()\n",
|
|
" return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n",
|
|
"\n",
|
|
"ipython.showtraceback = hide_traceback"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"collapsed": true,
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# JSON output syntax highlighting\n",
|
|
"from __future__ import print_function\n",
|
|
"from pygments import highlight\n",
|
|
"from pygments.lexers import JsonLexer\n",
|
|
"from pygments.formatters import HtmlFormatter\n",
|
|
"from IPython.display import HTML\n",
|
|
"\n",
|
|
"original_print = print\n",
|
|
"\n",
|
|
"def json_print(inpt):\n",
|
|
" string = str(inpt)\n",
|
|
" if string[0] == '{':\n",
|
|
" formatter = HtmlFormatter()\n",
|
|
" return HTML('<style type=\"text/css\">{}</style>{}'.format(\n",
|
|
" formatter.get_style_defs('.highlight'),\n",
|
|
" highlight(string, JsonLexer(), formatter)))\n",
|
|
" else:\n",
|
|
" original_print(inpt)\n",
|
|
"\n",
|
|
"print = json_print"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# DataStore API\n",
|
|
"\n",
|
|
"CTI Python STIX2 features a new interface for pulling and pushing STIX2 content. The new interface consists of DataStore, DataSource and DataSink constructs: a DataSource for pulling STIX2 content, a DataSink for pushing STIX2 content, and a DataStore for pulling/pushing.\n",
|
|
"\n",
|
|
"The DataStore, DataSource, DataSink (referred to as \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then sublcassed into real DataStore suite(s). CTI Python STIX2 provides for the DataStore suites of **FileSystem**, **Memory**, and **TAXII**. Users are also encrouraged subclassing the base Data suite and creating their own custom DataStore suites."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## CompositeDataSource\n",
|
|
"\n",
|
|
"**CompositeDataSource** is an available controller that can be used as a single interface to a set of defined DataSources. The purpose of this controller is allow for the grouping of **DataSources** and making get/query calls to a set of DataSources in one API call. **CompositeDataSource** can be used to organize/group **DataSources**, federate get()/all_versions()/query() calls, and reduce user code.\n",
|
|
"\n",
|
|
"**CompositeDataSource** is just a wrapper around a set of defined **DataSources** (e.g. FileSystemSource) that federates get()/all_versions()/query() calls individually to each of the attached **DataSources** , collects the results from each **DataSource** and returns them.\n",
|
|
"\n",
|
|
"Filters can be attached to **CompositeDataSources** just as they can be done to **DataStores** and **DataSources**. When get()/all_versions()/query() calls are made to the **CompositeDataSource**, it will pass along any query filters from the call and any of its own filters to the attached **DataSources**. To which, those attached **DataSources** may have their own attached filters as well. The effect is that all the filters are eventually combined when the get()/all_versions()/query() call is actually executed within a **DataSource**. \n",
|
|
"\n",
|
|
"A **CompositeDataSource** can also be attached to a **CompositeDataSource** for multiple layers of grouped **DataSources**.\n",
|
|
"\n",
|
|
"\n",
|
|
"### CompositeDataSource API\n",
|
|
"\n",
|
|
"#### CompositeDataSource Examples\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"type\": \"indicator\",\n",
|
|
" \"id\": \"indicator--797ae2b5-3f7a-44c5-8ecd-33ba22fdc2b5\",\n",
|
|
" \"created\": \"2017-10-04T19:27:41.000Z\",\n",
|
|
" \"modified\": \"2017-10-04T19:27:41.000Z\",\n",
|
|
" \"labels\": [\n",
|
|
" \"malicious-activity\"\n",
|
|
" ],\n",
|
|
" \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n",
|
|
" \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n",
|
|
" \"valid_from\": \"2017-10-04T19:27:41Z\",\n",
|
|
" \"kill_chain_phases\": [\n",
|
|
" {\n",
|
|
" \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n",
|
|
" \"phase_name\": \"delivery\"\n",
|
|
" }\n",
|
|
" ]\n",
|
|
"}\n",
|
|
"{\n",
|
|
" \"type\": \"indicator\",\n",
|
|
" \"id\": \"indicator--11913f42-2d52-4b9d-842f-94bf06819a66\",\n",
|
|
" \"created\": \"2017-10-04T19:27:41.000Z\",\n",
|
|
" \"modified\": \"2017-10-04T19:27:41.000Z\",\n",
|
|
" \"labels\": [\n",
|
|
" \"malicious-activity\"\n",
|
|
" ],\n",
|
|
" \"name\": \"Emerging Threats - Block Rules - Compromised IPs\",\n",
|
|
" \"pattern\": \"[ ipv4-addr:value = '98.138.19.88' ]\",\n",
|
|
" \"valid_from\": \"2017-10-04T19:27:41Z\",\n",
|
|
" \"kill_chain_phases\": [\n",
|
|
" {\n",
|
|
" \"kill_chain_name\": \"lockheed-martin-cyber-kill-chain\",\n",
|
|
" \"phase_name\": \"delivery\"\n",
|
|
" }\n",
|
|
" ]\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from taxii2client import Collection\n",
|
|
"from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n",
|
|
"\n",
|
|
"# create FileSystemStore\n",
|
|
"fs = FileSystemSource(\"/tmp/stix2_data\")\n",
|
|
"\n",
|
|
"# create TAXIICollectionSource\n",
|
|
"colxn = Collection('https://test.freetaxii.com:8000/api1/collections/9cfa669c-ee94-4ece-afd2-f8edac37d8fd/')\n",
|
|
"ts = TAXIICollectionSource(colxn)\n",
|
|
"\n",
|
|
"# add them both to the CompositeDataSource\n",
|
|
"cs = CompositeDataSource()\n",
|
|
"cs.add_data_sources([fs, ts])\n",
|
|
"\n",
|
|
"# get an object that is only in the filesystem\n",
|
|
"ta = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n",
|
|
"print(ta)\n",
|
|
"\n",
|
|
"# get an object that is only in the TAXII collection\n",
|
|
"ind = cs.get('indicator--37a6a5de-a5b9-425a-903a-4ae9cbf1ff3f')\n",
|
|
"print(ind)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Filters\n",
|
|
"\n",
|
|
"The CTI Python STIX2 **DataStore** suites - **FileSystem**, **Memory** and **TAXII** - all use the **Filters** module to allow for the querying of STIX content. The basic functionality is that filters can be created and supplied everytime to calls to **query()**, and/or attached to a **DataStore** so that every future query placed to that **DataStore** is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from **DataStores**.\n",
|
|
"\n",
|
|
"Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). Currently, CTI Python STIX2 supports **ONLY** STIX 2 object common properties and TAXII2 Filtering parameters for fields to filter on:\n",
|
|
"\n",
|
|
"Fields\n",
|
|
"\n",
|
|
"(STIX2 Object Common Properties)\n",
|
|
"\n",
|
|
"* created\n",
|
|
"* created_by_ref\n",
|
|
"* external_references.source_name\n",
|
|
"* external_references.description\n",
|
|
"* external_references.url\n",
|
|
"* external_references.external_id\n",
|
|
"* granular_markings.marking_ref\n",
|
|
"* granular_markings.selectors\n",
|
|
"* id\n",
|
|
"* labels\n",
|
|
"* modified\n",
|
|
"* object_marking_refs\n",
|
|
"* revoked\n",
|
|
"* type\n",
|
|
"\n",
|
|
"(TAXII2 filter fields)\n",
|
|
"\n",
|
|
"* added_after\n",
|
|
"* match[id]\n",
|
|
"* match[type]\n",
|
|
"* match[version]\n",
|
|
"\n",
|
|
"Supported operators on above properties:\n",
|
|
"\n",
|
|
"* =\n",
|
|
"* !=\n",
|
|
"* in\n",
|
|
"* >\n",
|
|
"* < \n",
|
|
"* ```>=```\n",
|
|
"* <=\n",
|
|
"\n",
|
|
"Value types of the common property values must be one of these (python) types:\n",
|
|
"\n",
|
|
"* bool\n",
|
|
"* dict\n",
|
|
"* float\n",
|
|
"* int\n",
|
|
"* list\n",
|
|
"* str\n",
|
|
"* tuple\n",
|
|
"\n",
|
|
"### Filter Examples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"from stix2 import Filter\n",
|
|
"\n",
|
|
"# create filter for STIX objects that have external references to MITRE ATT&CK framework\n",
|
|
"f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n",
|
|
"\n",
|
|
"# create filter for STIX objects that are not of SDO type Attack-Pattnern\n",
|
|
"f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n",
|
|
"\n",
|
|
"# create filter for STIX objects that have the \"threat-report\" label\n",
|
|
"f2 = Filter(\"labels\", \"in\", \"threat-report\")\n",
|
|
"\n",
|
|
"# create filter for STIX objects that have been modified past the timestamp\n",
|
|
"f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n",
|
|
"\n",
|
|
"# create filter for STIX objects that have been revoked\n",
|
|
"f4 = Filter(\"revoked\", \"=\", True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"For Filters to be applied to a query, they must be either supplied with the query call or attached a **DataStore**, more specifically to **DataSource** whether that **DataSource** is a part of a **DataStore** or stands by itself. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n",
|
|
"\n",
|
|
"fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n",
|
|
"fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n",
|
|
"\n",
|
|
"# attach filter to FileSystemStore\n",
|
|
"fs.source.filters.add(f)\n",
|
|
"\n",
|
|
"# attach multiple filters to FileSystemStore\n",
|
|
"fs.source.filters.update([f1,f2])\n",
|
|
"\n",
|
|
"# can also attach filters to a Source\n",
|
|
"# attach multiple filters to FileSystemSource\n",
|
|
"fs_source.filters.update([f3, f4])\n",
|
|
"\n",
|
|
"\n",
|
|
"mem = MemoryStore()\n",
|
|
"\n",
|
|
"# As it is impractical to only use MemorySink or MemorySource,\n",
|
|
"# attach a filter to a MemoryStore\n",
|
|
"mem.source.filters.add(f)\n",
|
|
"\n",
|
|
"# attach multiple filters to a MemoryStore\n",
|
|
"mem.source.filters.update([f1,f2])"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.5.2"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|