{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer, TextLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import display, HTML\n", "from IPython.core.interactiveshell import InteractiveShell\n", "\n", "InteractiveShell.ast_node_interactivity = \"all\"\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " formatter = HtmlFormatter()\n", " if string[0] == '{':\n", " lexer = JsonLexer()\n", " else:\n", " lexer = TextLexer()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, lexer, formatter)))\n", "\n", "globals()['print'] = json_print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Custom STIX Content" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Custom Properties\n", "\n", "Attempting to create a STIX object with properties not defined by the specification will result in an error. Try creating an ``Identity`` object with a custom ``x_foo`` property:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "ename": "ExtraPropertiesError", "evalue": "Unexpected properties for Identity: (x_foo).", "output_type": "error", "traceback": [ "\u001b[0;31mExtraPropertiesError\u001b[0m\u001b[0;31m:\u001b[0m Unexpected properties for Identity: (x_foo).\n" ] } ], "source": [ "from stix2 import Identity\n", "\n", "Identity(name=\"John Smith\",\n", " identity_class=\"individual\",\n", " x_foo=\"bar\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To create a STIX object with one or more custom properties, pass them in as a dictionary parameter called ``custom_properties``:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "identity",\n",
       "    "id": "identity--e7fd0fe0-25ff-4fcb-abe5-b6254a9d1a22",\n",
       "    "created": "2019-07-25T18:18:18.241Z",\n",
       "    "modified": "2019-07-25T18:18:18.241Z",\n",
       "    "name": "John Smith",\n",
       "    "identity_class": "individual",\n",
       "    "x_foo": "bar"\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "identity = Identity(name=\"John Smith\",\n", " identity_class=\"individual\",\n", " custom_properties={\n", " \"x_foo\": \"bar\"\n", " })\n", "print(identity)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively, setting ``allow_custom`` to ``True`` will allow custom properties without requiring a ``custom_properties`` dictionary." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "identity",\n",
       "    "id": "identity--033b5f42-c94f-488f-8efa-2b6a167f0d6f",\n",
       "    "created": "2019-07-25T18:18:21.352Z",\n",
       "    "modified": "2019-07-25T18:18:21.352Z",\n",
       "    "name": "John Smith",\n",
       "    "identity_class": "individual",\n",
       "    "x_foo": "bar"\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "identity2 = Identity(name=\"John Smith\",\n", " identity_class=\"individual\",\n", " x_foo=\"bar\",\n", " allow_custom=True)\n", "print(identity2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Likewise, when parsing STIX content with custom properties, pass ``allow_custom=True`` to [parse()](../api/stix2.core.rst#stix2.core.parse):" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
bar\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import parse\n", "\n", "input_string = \"\"\"{\n", " \"type\": \"identity\",\n", " \"id\": \"identity--311b2d2d-f010-4473-83ec-1edf84858f4c\",\n", " \"created\": \"2015-12-21T19:59:11Z\",\n", " \"modified\": \"2015-12-21T19:59:11Z\",\n", " \"name\": \"John Smith\",\n", " \"identity_class\": \"individual\",\n", " \"x_foo\": \"bar\"\n", "}\"\"\"\n", "identity3 = parse(input_string, allow_custom=True)\n", "print(identity3.x_foo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To remove a custom properties, use `new_version()` and set it to `None`." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "identity",\n",
       "    "id": "identity--311b2d2d-f010-4473-83ec-1edf84858f4c",\n",
       "    "created": "2015-12-21T19:59:11.000Z",\n",
       "    "modified": "2019-07-25T18:18:25.927Z",\n",
       "    "name": "John Smith",\n",
       "    "identity_class": "individual"\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "identity4 = identity3.new_version(x_foo=None)\n", "print(identity4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Custom STIX Object Types\n", "\n", "To create a custom STIX object type, define a class with the @[CustomObject](../api/v20/stix2.v20.sdo.rst#stix2.v20.sdo.CustomObject) decorator. It takes the type name and a list of property tuples, each tuple consisting of the property name and a property instance. Any special validation of the properties can be added by supplying an ``__init__`` function.\n", "\n", "Let's say zoo animals have become a serious cyber threat and we want to model them in STIX using a custom object type. Let's use a ``species`` property to store the kind of animal, and make that property required. We also want a property to store the class of animal, such as \"mammal\" or \"bird\" but only want to allow specific values in it. We can add some logic to validate this property in ``__init__``." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from stix2 import CustomObject, properties\n", "\n", "@CustomObject('x-animal', [\n", " ('species', properties.StringProperty(required=True)),\n", " ('animal_class', properties.StringProperty()),\n", "])\n", "class Animal(object):\n", " def __init__(self, animal_class=None, **kwargs):\n", " if animal_class and animal_class not in ['mammal', 'bird', 'fish', 'reptile']:\n", " raise ValueError(\"'%s' is not a recognized class of animal.\" % animal_class)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can create an instance of our custom ``Animal`` type." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "x-animal",\n",
       "    "id": "x-animal--b1e4fe7f-7985-451d-855c-6ba5c265b22a",\n",
       "    "created": "2018-04-05T18:38:19.790Z",\n",
       "    "modified": "2018-04-05T18:38:19.790Z",\n",
       "    "species": "lion",\n",
       "    "animal_class": "mammal"\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "animal = Animal(species=\"lion\",\n", " animal_class=\"mammal\")\n", "print(animal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Trying to create an ``Animal`` instance with an ``animal_class`` that's not in the list will result in an error:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "'alien' is not a recognized class of animal.", "output_type": "error", "traceback": [ "\u001b[0;31mValueError\u001b[0m\u001b[0;31m:\u001b[0m 'alien' is not a recognized class of animal.\n" ] } ], "source": [ "Animal(species=\"xenomorph\",\n", " animal_class=\"alien\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Parsing custom object types that you have already defined is simple and no different from parsing any other STIX object." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
shark\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input_string2 = \"\"\"{\n", " \"type\": \"x-animal\",\n", " \"id\": \"x-animal--941f1471-6815-456b-89b8-7051ddf13e4b\",\n", " \"created\": \"2015-12-21T19:59:11Z\",\n", " \"modified\": \"2015-12-21T19:59:11Z\",\n", " \"species\": \"shark\",\n", " \"animal_class\": \"fish\"\n", "}\"\"\"\n", "animal2 = parse(input_string2)\n", "print(animal2.species)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, parsing custom object types which you have not defined will result in an error:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "ename": "ParseError", "evalue": "Can't parse unknown object type 'x-foobar'! For custom types, use the CustomObject decorator.", "output_type": "error", "traceback": [ "\u001b[0;31mParseError\u001b[0m\u001b[0;31m:\u001b[0m Can't parse unknown object type 'x-foobar'! For custom types, use the CustomObject decorator.\n" ] } ], "source": [ "input_string3 = \"\"\"{\n", " \"type\": \"x-foobar\",\n", " \"id\": \"x-foobar--d362beb5-a04e-4e6b-a030-b6935122c3f9\",\n", " \"created\": \"2015-12-21T19:59:11Z\",\n", " \"modified\": \"2015-12-21T19:59:11Z\",\n", " \"bar\": 1,\n", " \"baz\": \"frob\"\n", "}\"\"\"\n", "parse(input_string3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Custom Cyber Observable Types\n", "\n", "Similar to custom STIX object types, use a decorator to create [custom Cyber Observable](../api/v20/stix2.v20.observables.rst#stix2.v20.observables.CustomObservable) types. Just as before, ``__init__()`` can hold additional validation, but it is not necessary." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "x-new-observable",\n",
       "    "a_property": "something",\n",
       "    "property_2": 10\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import CustomObservable\n", "\n", "@CustomObservable('x-new-observable', [\n", " ('a_property', properties.StringProperty(required=True)),\n", " ('property_2', properties.IntegerProperty()),\n", "])\n", "class NewObservable():\n", " pass\n", "\n", "new_observable = NewObservable(a_property=\"something\",\n", " property_2=10)\n", "print(new_observable)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "Likewise, after the custom Cyber Observable type has been defined, it can be parsed." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
foobaz\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
5\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import ObservedData\n", "\n", "input_string4 = \"\"\"{\n", " \"type\": \"observed-data\",\n", " \"id\": \"observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf\",\n", " \"created_by_ref\": \"identity--f431f809-377b-45e0-aa1c-6a4751cae5ff\",\n", " \"created\": \"2016-04-06T19:58:16.000Z\",\n", " \"modified\": \"2016-04-06T19:58:16.000Z\",\n", " \"first_observed\": \"2015-12-21T19:00:00Z\",\n", " \"last_observed\": \"2015-12-21T19:00:00Z\",\n", " \"number_observed\": 50,\n", " \"objects\": {\n", " \"0\": {\n", " \"type\": \"x-new-observable\",\n", " \"a_property\": \"foobaz\",\n", " \"property_2\": 5\n", " }\n", " }\n", "}\"\"\"\n", "obs_data = parse(input_string4)\n", "print(obs_data.objects[\"0\"].a_property)\n", "print(obs_data.objects[\"0\"].property_2)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### Custom Cyber Observable Extensions\n", "\n", "Finally, custom extensions to existing Cyber Observable types can also be created. Just use the @[CustomExtension](../api/v20/stix2.v20.observables.rst#stix2.v20.observables.CustomExtension) decorator. Note that you must provide the Cyber Observable class to which the extension applies. Again, any extra validation of the properties can be implemented by providing an ``__init__()`` but it is not required. Let's say we want to make an extension to the ``File`` Cyber Observable Object:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "property1": "something",\n",
       "    "property2": 10\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import File, CustomExtension\n", "\n", "@CustomExtension(File, 'x-new-ext', [\n", " ('property1', properties.StringProperty(required=True)),\n", " ('property2', properties.IntegerProperty()),\n", "])\n", "class NewExtension():\n", " pass\n", "\n", "new_ext = NewExtension(property1=\"something\",\n", " property2=10)\n", "print(new_ext)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the custom Cyber Observable extension has been defined, it can be parsed." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
bla\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
50\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input_string5 = \"\"\"{\n", " \"type\": \"observed-data\",\n", " \"id\": \"observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf\",\n", " \"created_by_ref\": \"identity--f431f809-377b-45e0-aa1c-6a4751cae5ff\",\n", " \"created\": \"2016-04-06T19:58:16.000Z\",\n", " \"modified\": \"2016-04-06T19:58:16.000Z\",\n", " \"first_observed\": \"2015-12-21T19:00:00Z\",\n", " \"last_observed\": \"2015-12-21T19:00:00Z\",\n", " \"number_observed\": 50,\n", " \"objects\": {\n", " \"0\": {\n", " \"type\": \"file\",\n", " \"name\": \"foo.bar\",\n", " \"hashes\": {\n", " \"SHA-256\": \"35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f\"\n", " },\n", " \"extensions\": {\n", " \"x-new-ext\": {\n", " \"property1\": \"bla\",\n", " \"property2\": 50\n", " }\n", " }\n", " }\n", " }\n", "}\"\"\"\n", "obs_data2 = parse(input_string5)\n", "print(obs_data2.objects[\"0\"].extensions[\"x-new-ext\"].property1)\n", "print(obs_data2.objects[\"0\"].extensions[\"x-new-ext\"].property2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Deterministic IDs for Cyber Observables\n", "### Deterministic IDs\n", "STIX 2.1 Cyber Observables (SCOs) have an ID property since SCOs are now top-level objects. However, SCOs have deterministic IDs, meaning that the ID of a SCO is based on the values of its properties. Thus, if multiple cyber observables of the same type have the same values for their ID contributing properties (explained below), then these SCOs will have the same ID; the SCOs' ID is deterministic because the ID will not change if the values of the ID contributing properties do not change. \n", "\n", "A UUIDv5 is generated for the deterministic IDs, using the namespace `\"00abedb4-aa42-466c-9c01-fed23315a9b7\"`.\n", "\n", "In the case where a SCO does not have any defined ID contributing properties, or in the case where all of the values for the ID contributing properties are not specified, then the SCO will be assigned a randomly-generated UUIDv4.\n", "\n", "### ID Contributing Properties\n", "So, what are ID contributing properties? \n", "Every SCO has multiple defined properties, but the values of only some of those properties will contribute to the determination of the SCO's ID. \n", "\n", "A SCO's ID contributing properties may contain a combination of required properties and optional properties. And it is possible for all of the ID contributing properties to be optional, which means the corresponding SCO could be created without values for any of those properties, which would lead the SCO to have a randomly-generated UUIDv4 ID.\n", "\n", "\n", "We will demonstrate deterministic IDs by creating four `EmailAddress` SCOs. The `EmailAddress` SCO has one ID contributing property: `value`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
email-addr--8fab0ad0-03c0-5cba-bf7b-c2bd41fb73a7\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
email-addr--8fab0ad0-03c0-5cba-bf7b-c2bd41fb73a7\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
email-addr--8fab0ad0-03c0-5cba-bf7b-c2bd41fb73a7\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
email-addr--d3742ef4-9452-5935-bc42-e8c35a119757\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2.v21 import EmailAddress\n", "\n", "email_addr_1 = EmailAddress(value=\"johnnydoe@matching.com\", display_name=\"Johnny Doe\")\n", "print (email_addr_1.id)\n", "\n", "email_addr_2 = EmailAddress(value=\"johnnydoe@matching.com\", display_name=\"Johnny Doe\")\n", "print (email_addr_2.id)\n", "\n", "email_addr_3 = EmailAddress(value=\"johnnydoe@matching.com\", display_name=\"Johnathon Doe\")\n", "print (email_addr_3.id)\n", "\n", "email_addr_4 = EmailAddress(value=\"johnnydoe@notmatching.com\", display_name=\"Johnny Doe\")\n", "print (email_addr_4.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that the ID for the first three `EmailAddress` objects is the same while the ID for the fourth is different. This is because the first three objects all have the same value for the ID contributing property. And this is despite having a different value for the `display_name` property, since it is not an ID contributing property for the `EmailAddress` SCO.\n", "\n", "Also note that the fourth object has a different ID despite having the same `display_name` as the first two objects; the value for the fourth object's ID contributing property is different." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 2 }