MISP
/
cti-python-stix2
mirror of https://github.com/MISP/cti-python-stix2
2
0
Fork 0
Code Issues Releases Wiki Activity
cti-python-stix2/docs/guide/datastore.ipynb

759 lines
45 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Delete this cell to re-enable tracebacks\n",
"import sys\n",
"ipython = get_ipython()\n",
"\n",
"def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n",
" exception_only=False, running_compiled_code=False):\n",
" etype, value, tb = sys.exc_info()\n",
" value.__cause__ = None # suppress chained exceptions\n",
" return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n",
"\n",
"ipython.showtraceback = hide_traceback"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# JSON output syntax highlighting\n",
"from __future__ import print_function\n",
"from pygments import highlight\n",
"from pygments.lexers import JsonLexer, TextLexer\n",
"from pygments.formatters import HtmlFormatter\n",
"from IPython.display import display, HTML\n",
"from IPython.core.interactiveshell import InteractiveShell\n",
"\n",
"InteractiveShell.ast_node_interactivity = \"all\"\n",
"\n",
"def json_print(inpt):\n",
" string = str(inpt)\n",
" formatter = HtmlFormatter()\n",
" if string[0] == '{':\n",
" lexer = JsonLexer()\n",
" else:\n",
" lexer = TextLexer()\n",
" return HTML('<style type=\"text/css\">{}</style>{}'.format(\n",
" formatter.get_style_defs('.highlight'),\n",
" highlight(string, lexer, formatter)))\n",
"\n",
"globals()['print'] = json_print"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DataStore API\n",
"\n",
"The ``stix2`` library features an interface for pulling and pushing STIX 2 content. This interface consists of [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin), [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) and [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) constructs: a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) for pulling STIX 2 content, a [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) for pushing STIX 2 content, and a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) for both pulling and pushing.\n",
"\n",
"The DataStore, [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource), [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) (collectively referred to as the \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then subclassed by real DataStore suites. The ``stix2`` library provides the DataStore suites of [FileSystem](../api/datastore/stix2.datastore.filesystem.rst), [Memory](../api/datastore/stix2.datastore.memory.rst), and [TAXII](../api/datastore/stix2.datastore.taxii.rst). Users are also encouraged to subclass the base classes and create their own custom DataStore suites."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CompositeDataSource\n",
"\n",
"[CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) is an available controller that can be used as a single interface to a set of defined [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). The purpose of this controller is allow for the grouping of [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) and making `get()`/`query()` calls to a set of DataSources in one API call. [CompositeDataSources](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) can be used to organize/group [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource), federate `get()`/`all_versions()`/`query()` calls, and reduce user code.\n",
"\n",
"[CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) is just a wrapper around a set of defined [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) (e.g. [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource)) that federates `get()`/`all_versions()`/`query()` calls individually to each of the attached [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) , collects the results from each [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) and returns them.\n",
"\n",
"Filters can be attached to [CompositeDataSources](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) just as they can be done to [DataStores](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) and [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). When `get()`/`all_versions()`/`query()` calls are made to the [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource), it will pass along any query filters from the call and any of its own filters to the attached [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). In addition, those [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) may have their own attached filters as well. The effect is that all the filters are eventually combined when the `get()`/`all_versions()`/`query()` call is actually executed within a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource). \n",
"\n",
"A [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) can also be attached to a [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) for multiple layers of grouped [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource).\n",
"\n",
"\n",
"### CompositeDataSource API\n",
"\n",
"#### CompositeDataSource Examples\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
".highlight { background: #f8f8f8; }\n",
".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
".highlight .o { color: #666666 } /* Operator */\n",
".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
".highlight .ge { font-style: italic } /* Generic.Emph */\n",
".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
".highlight .go { color: #888888 } /* Generic.Output */\n",
".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
".highlight .m { color: #666666 } /* Literal.Number */\n",
".highlight .s { color: #BA2121 } /* Literal.String */\n",
".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
".highlight .nb { color: #008000 } /* Name.Builtin */\n",
".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
".highlight .no { color: #880000 } /* Name.Constant */\n",
".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
".highlight .nf { color: #0000FF } /* Name.Function */\n",
".highlight .nl { color: #A0A000 } /* Name.Label */\n",
".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
".highlight .nv { color: #19177C } /* Name.Variable */\n",
".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;type&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;intrusion-set&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;spec_version&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2.1&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;id&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;created_by_ref&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;created&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2017-05-31T21:31:53.197Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;modified&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2017-05-31T21:31:53.197Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;DragonOK&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;description&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;DragonOK is a threat group that has targeted Japanese organizations with phishing emails. Due to overlapping TTPs, including similar custom tools, DragonOK is thought to have a direct or indirect relationship with the threat group Moafee. [[Citation: Operation Quantum Entanglement]][[Citation: Symbiotic APT Groups]] It is known to use a variety of malware, including Sysget/HelloBridge, PlugX, PoisonIvy, FormerFirstRat, NFlog, and NewCT. [[Citation: New DragonOK]]&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;aliases&quot;</span><span class=\"p\">:</span> <span class=\"p\">[</span>\n",
" <span class=\"s2\">&quot;DragonOK&quot;</span>\n",
" <span class=\"p\">],</span>\n",
" <span class=\"nt\">&quot;external_references&quot;</span><span class=\"p\">:</span> <span class=\"p\">[</span>\n",
" <span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;source_name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;mitre-attack&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;url&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;https://attack.mitre.org/wiki/Group/G0017&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;external_id&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;G0017&quot;</span>\n",
" <span class=\"p\">},</span>\n",
" <span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;source_name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;Operation Quantum Entanglement&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;description&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;Haq, T., Moran, N., Vashisht, S., Scott, M. (2014, September). OPERATION QUANTUM ENTANGLEMENT. Retrieved November 4, 2015.&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;url&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;https://www.fireeye.com/content/dam/fireeye-www/global/en/current-threats/pdfs/wp-operation-quantum-entanglement.pdf&quot;</span>\n",
" <span class=\"p\">},</span>\n",
" <span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;source_name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;Symbiotic APT Groups&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;description&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;Haq, T. (2014, October). An Insight into Symbiotic APT Groups. Retrieved November 4, 2015.&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;url&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;https://dl.mandiant.com/EE/library/MIRcon2014/MIRcon%202014%20R&amp;D%20Track%20Insight%20into%20Symbiotic%20APT.pdf&quot;</span>\n",
" <span class=\"p\">},</span>\n",
" <span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;source_name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;New DragonOK&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;description&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;Miller-Osborn, J., Grunzweig, J.. (2015, April). Unit 42 Identifies New DragonOK Backdoor Malware Deployed Against Japanese Targets. Retrieved November 4, 2015.&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;url&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;http://researchcenter.paloaltonetworks.com/2015/04/unit-42-identifies-new-dragonok-backdoor-malware-deployed-against-japanese-targets/&quot;</span>\n",
" <span class=\"p\">}</span>\n",
" <span class=\"p\">],</span>\n",
" <span class=\"nt\">&quot;object_marking_refs&quot;</span><span class=\"p\">:</span> <span class=\"p\">[</span>\n",
" <span class=\"s2\">&quot;marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168&quot;</span>\n",
" <span class=\"p\">]</span>\n",
"<span class=\"p\">}</span>\n",
"</pre></div>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"text/html": [
"<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
".highlight { background: #f8f8f8; }\n",
".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
".highlight .o { color: #666666 } /* Operator */\n",
".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
".highlight .ge { font-style: italic } /* Generic.Emph */\n",
".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
".highlight .go { color: #888888 } /* Generic.Output */\n",
".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
".highlight .m { color: #666666 } /* Literal.Number */\n",
".highlight .s { color: #BA2121 } /* Literal.String */\n",
".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
".highlight .nb { color: #008000 } /* Name.Builtin */\n",
".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
".highlight .no { color: #880000 } /* Name.Constant */\n",
".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
".highlight .nf { color: #0000FF } /* Name.Function */\n",
".highlight .nl { color: #A0A000 } /* Name.Label */\n",
".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
".highlight .nv { color: #19177C } /* Name.Variable */\n",
".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;type&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;indicator&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;spec_version&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2.1&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;id&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;indicator--a740531e-63ff-4e49-a9e1-a0a3eed0e3e7&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;created&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2017-11-13T07:00:24.000Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;modified&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2017-11-13T07:00:24.000Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;Ransomware IP Blocklist&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;description&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;IP Blocklist address from abuse.ch&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;indicator_types&quot;</span><span class=\"p\">:</span> <span class=\"p\">[</span>\n",
" <span class=\"s2\">&quot;malicious-activity&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"s2\">&quot;Ransomware&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"s2\">&quot;Botnet&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"s2\">&quot;C&amp;C&quot;</span>\n",
" <span class=\"p\">],</span>\n",
" <span class=\"nt\">&quot;pattern&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;[ ipv4-addr:value = &#39;91.237.247.24&#39; ]&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;pattern_type&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;stix&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;pattern_version&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2.1&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;valid_from&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2017-11-13T07:00:24Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;external_references&quot;</span><span class=\"p\">:</span> <span class=\"p\">[</span>\n",
" <span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;source_name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;abuse.ch&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;url&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;https://ransomwaretracker.abuse.ch/blocklist/&quot;</span>\n",
" <span class=\"p\">}</span>\n",
" <span class=\"p\">]</span>\n",
"<span class=\"p\">}</span>\n",
"</pre></div>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from taxii2client import Collection\n",
"from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n",
"\n",
"# create FileSystemStore\n",
"fs = FileSystemSource(\"/tmp/stix2_source\")\n",
"\n",
"# create TAXIICollectionSource\n",
"colxn = Collection('http://127.0.0.1:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/', user=\"user1\", password=\"Password1\")\n",
"ts = TAXIICollectionSource(colxn)\n",
"\n",
"# add them both to the CompositeDataSource\n",
"cs = CompositeDataSource()\n",
"cs.add_data_sources([fs,ts])\n",
"\n",
"# get an object that is only in the filesystem\n",
"intrusion_set = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n",
"print(intrusion_set.serialize(pretty=True))\n",
"\n",
"# get an object that is only in the TAXII collection\n",
"ind = cs.get('indicator--a740531e-63ff-4e49-a9e1-a0a3eed0e3e7')\n",
"print(ind.serialize(pretty=True))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Filters\n",
"\n",
"The ``stix2`` DataStore suites - [FileSystem](../api/datastore/stix2.datastore.filesystem.rst), [Memory](../api/datastore/stix2.datastore.memory.rst), and [TAXII](../api/datastore/stix2.datastore.taxii.rst) - all use the [Filters](../api/datastore/stix2.datastore.filters.rst) module to allow for the querying of STIX content. Filters can be used to explicitly include or exclude results with certain criteria. For example:\n",
"\n",
"* only trust content from a set of object creators\n",
"* exclude content from certain (untrusted) object creators\n",
"* only include content with a confidence above a certain threshold (once confidence is added to STIX 2)\n",
"* only return content that can be shared with external parties (e.g. only content that has TLP:GREEN markings)\n",
"\n",
"Filters can be created and supplied with every call to `query()`, and/or attached to a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) so that every future query placed to that [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from [DataStores](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin).\n",
"\n",
"Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). All properties of STIX 2 objects can be filtered on. In addition, TAXII 2 Filtering parameters for fields can also be used in filters.\n",
"\n",
"TAXII2 filter fields:\n",
"\n",
"* added_after\n",
"* id\n",
"* spec_version\n",
"* type\n",
"* version\n",
"\n",
"Supported operators:\n",
"\n",
"* =\n",
"* !=\n",
"* in\n",
"* ```>```\n",
"* < \n",
"* ```>=```\n",
"* <=\n",
"* contains\n",
"\n",
"Value types of the property values must be one of these (Python) types:\n",
"\n",
"* bool\n",
"* dict\n",
"* float\n",
"* int\n",
"* list\n",
"* str\n",
"* tuple\n",
"\n",
"### Filter Examples"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"from stix2 import Filter\n",
"\n",
"# create filter for STIX objects that have external references to MITRE ATT&CK framework\n",
"f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n",
"\n",
"# create filter for STIX objects that are not of SDO type Attack-Pattnern\n",
"f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n",
"\n",
"# create filter for STIX objects that have the \"threat-report\" label\n",
"f2 = Filter(\"labels\", \"in\", \"threat-report\")\n",
"\n",
"# create filter for STIX objects that have been modified past the timestamp\n",
"f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n",
"\n",
"# create filter for STIX objects that have been revoked\n",
"f4 = Filter(\"revoked\", \"=\", True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For Filters to be applied to a query, they must be either supplied with the query call or attached to a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin), more specifically to a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) whether that [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) is a part of a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) or stands by itself. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n",
"\n",
"fs = FileSystemStore(\"/tmp/stix2_store\")\n",
"fs_source = FileSystemSource(\"/tmp/stix2_source\")\n",
"\n",
"# attach filter to FileSystemStore\n",
"fs.source.filters.add(f)\n",
"\n",
"# attach multiple filters to FileSystemStore\n",
"fs.source.filters.add([f1,f2])\n",
"\n",
"# can also attach filters to a Source\n",
"# attach multiple filters to FileSystemSource\n",
"fs_source.filters.add([f3, f4])\n",
"\n",
"\n",
"mem = MemoryStore()\n",
"\n",
"# As it is impractical to only use MemorySink or MemorySource,\n",
"# attach a filter to a MemoryStore\n",
"mem.source.filters.add(f)\n",
"\n",
"# attach multiple filters to a MemoryStore\n",
"mem.source.filters.add([f1,f2])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note: The `defanged` property is now always included (implicitly) for STIX 2.1 Cyber Observable Objects (SCOs)**\n",
"\n",
"This is important to remember if you are writing a filter that involves checking the `objects` property of a STIX 2.1 `ObservedData` object. If any of the objects associated with the `objects` property are STIX 2.1 SCOs, then your filter must include the `defanged` property. For an example, refer to `filters[14]` & `filters[15]` in stix2/test/v21/test_datastore_filters.py "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## De-Referencing Relationships\n",
"\n",
"Given a STIX object, there are several ways to find other STIX objects related to it. To illustrate this, let's first create a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) and add some objects and relationships."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"from stix2 import Campaign, Identity, Indicator, Malware, Relationship\n",
"\n",
"mem = MemoryStore()\n",
"cam = Campaign(name='Charge', description='Attack!')\n",
"idy = Identity(name='John Doe', identity_class=\"individual\")\n",
"ind = Indicator(pattern_type='stix', pattern=\"[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']\")\n",
"mal = Malware(name=\"Cryptolocker\", is_family=False, created_by_ref=idy)\n",
"rel1 = Relationship(ind, 'indicates', mal,)\n",
"rel2 = Relationship(mal, 'targets', idy)\n",
"rel3 = Relationship(cam, 'uses', mal)\n",
"mem.add([cam, idy, ind, mal, rel1, rel2, rel3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If a STIX object has a `created_by_ref` property, you can use the [creator_of()](../api/stix2.datastore.rst#stix2.datastore.DataSource.creator_of) method to retrieve the [Identity](../api/v21/stix2.v21.sdo.rst#stix2.v21.sdo.Identity) object that created it."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<style type=\"text/css\">.highlight .hll { background-color: #ffffcc }\n",
".highlight { background: #f8f8f8; }\n",
".highlight .c { color: #408080; font-style: italic } /* Comment */\n",
".highlight .err { border: 1px solid #FF0000 } /* Error */\n",
".highlight .k { color: #008000; font-weight: bold } /* Keyword */\n",
".highlight .o { color: #666666 } /* Operator */\n",
".highlight .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
".highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
".highlight .cp { color: #BC7A00 } /* Comment.Preproc */\n",
".highlight .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
".highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
".highlight .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
".highlight .gd { color: #A00000 } /* Generic.Deleted */\n",
".highlight .ge { font-style: italic } /* Generic.Emph */\n",
".highlight .gr { color: #FF0000 } /* Generic.Error */\n",
".highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
".highlight .gi { color: #00A000 } /* Generic.Inserted */\n",
".highlight .go { color: #888888 } /* Generic.Output */\n",
".highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
".highlight .gs { font-weight: bold } /* Generic.Strong */\n",
".highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
".highlight .gt { color: #0044DD } /* Generic.Traceback */\n",
".highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
".highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
".highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
".highlight .kp { color: #008000 } /* Keyword.Pseudo */\n",
".highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
".highlight .kt { color: #B00040 } /* Keyword.Type */\n",
".highlight .m { color: #666666 } /* Literal.Number */\n",
".highlight .s { color: #BA2121 } /* Literal.String */\n",
".highlight .na { color: #7D9029 } /* Name.Attribute */\n",
".highlight .nb { color: #008000 } /* Name.Builtin */\n",
".highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
".highlight .no { color: #880000 } /* Name.Constant */\n",
".highlight .nd { color: #AA22FF } /* Name.Decorator */\n",
".highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
".highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
".highlight .nf { color: #0000FF } /* Name.Function */\n",
".highlight .nl { color: #A0A000 } /* Name.Label */\n",
".highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
".highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
".highlight .nv { color: #19177C } /* Name.Variable */\n",
".highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
".highlight .w { color: #bbbbbb } /* Text.Whitespace */\n",
".highlight .mb { color: #666666 } /* Literal.Number.Bin */\n",
".highlight .mf { color: #666666 } /* Literal.Number.Float */\n",
".highlight .mh { color: #666666 } /* Literal.Number.Hex */\n",
".highlight .mi { color: #666666 } /* Literal.Number.Integer */\n",
".highlight .mo { color: #666666 } /* Literal.Number.Oct */\n",
".highlight .sa { color: #BA2121 } /* Literal.String.Affix */\n",
".highlight .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
".highlight .sc { color: #BA2121 } /* Literal.String.Char */\n",
".highlight .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
".highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
".highlight .s2 { color: #BA2121 } /* Literal.String.Double */\n",
".highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
".highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
".highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
".highlight .sx { color: #008000 } /* Literal.String.Other */\n",
".highlight .sr { color: #BB6688 } /* Literal.String.Regex */\n",
".highlight .s1 { color: #BA2121 } /* Literal.String.Single */\n",
".highlight .ss { color: #19177C } /* Literal.String.Symbol */\n",
".highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
".highlight .fm { color: #0000FF } /* Name.Function.Magic */\n",
".highlight .vc { color: #19177C } /* Name.Variable.Class */\n",
".highlight .vg { color: #19177C } /* Name.Variable.Global */\n",
".highlight .vi { color: #19177C } /* Name.Variable.Instance */\n",
".highlight .vm { color: #19177C } /* Name.Variable.Magic */\n",
".highlight .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"p\">{</span>\n",
" <span class=\"nt\">&quot;type&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;identity&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;spec_version&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2.1&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;id&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;identity--a2628104-e357-44a0-b16f-d5f36c0fd0ec&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;created&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2020-06-26T13:59:21.924055Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;modified&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;2020-06-26T13:59:21.924055Z&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;name&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;John Doe&quot;</span><span class=\"p\">,</span>\n",
" <span class=\"nt\">&quot;identity_class&quot;</span><span class=\"p\">:</span> <span class=\"s2\">&quot;individual&quot;</span>\n",
"<span class=\"p\">}</span>\n",
"</pre></div>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print(mem.creator_of(mal).serialize(pretty=True))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use the [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) method to retrieve all the relationship objects that reference a STIX object."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rels = mem.relationships(mal)\n",
"len(rels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can limit it to only specific relationship types:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Relationship(type='relationship', spec_version='2.1', id='relationship--ef837187-773c-41e4-ae86-c66189a832f5', created='2020-06-26T13:59:21.929336Z', modified='2020-06-26T13:59:21.929336Z', relationship_type='indicates', source_ref='indicator--9f10f6f2-b93d-488e-be35-72c3ec1087c3', target_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b')]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mem.relationships(mal, relationship_type='indicates')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can limit it to only relationships where the given object is the source:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Relationship(type='relationship', spec_version='2.1', id='relationship--43f5f7a7-8a99-4bbf-8d93-e6f3fd2951a3', created='2020-06-26T13:59:21.937132Z', modified='2020-06-26T13:59:21.937132Z', relationship_type='targets', source_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b', target_ref='identity--a2628104-e357-44a0-b16f-d5f36c0fd0ec')]"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mem.relationships(mal, source_only=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And you can limit it to only relationships where the given object is the target:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Relationship(type='relationship', spec_version='2.1', id='relationship--ef837187-773c-41e4-ae86-c66189a832f5', created='2020-06-26T13:59:21.929336Z', modified='2020-06-26T13:59:21.929336Z', relationship_type='indicates', source_ref='indicator--9f10f6f2-b93d-488e-be35-72c3ec1087c3', target_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b'),\n",
" Relationship(type='relationship', spec_version='2.1', id='relationship--596c196f-2f05-4584-b643-2186b327a94f', created='2020-06-26T13:59:21.937354Z', modified='2020-06-26T13:59:21.937354Z', relationship_type='uses', source_ref='campaign--d359f872-7e44-4090-8e08-c5bd10bc5f2d', target_ref='malware--315597db-2a74-4a29-8e54-38572e1ac07b')]"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mem.relationships(mal, target_only=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, you can retrieve all STIX objects related to a given STIX object using [related_to()](../api/stix2.datastore.rst#stix2.datastore.DataSource.related_to). This calls [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) but then performs the extra step of getting the objects that these Relationships point to. [related_to()](../api/stix2.datastore.rst#stix2.datastore.DataSource.related_to) takes all the same arguments that [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) does."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Campaign(type='campaign', spec_version='2.1', id='campaign--d359f872-7e44-4090-8e08-c5bd10bc5f2d', created='2020-06-26T13:59:21.923792Z', modified='2020-06-26T13:59:21.923792Z', name='Charge', description='Attack!')]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mem.related_to(mal, target_only=True, relationship_type='uses')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.0a6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink