MISP
/
cti-python-stix2
mirror of https://github.com/MISP/cti-python-stix2
2
0
Fork 0
Code Issues Releases Wiki Activity

guide docs for DataStore/Source/Sink, Memory, FileSystem, TAXIICollection

stix2.1
= 2017-09-24 18:14:15 -04:00
parent 4450eb545b
commit e2a7b89975
4 changed files with 1139 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

221
docs/guide/DataStore, CompositeDataSource, Filters.ipynb Normal file
View File

@ -0,0 +1,221 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DataStore API\n",
"\n",
"CTI Python STIX2 features a new interface for pulling and pushing STIX2 content. The new interface consists of DataStore, DataSource and DataSink constructs: a DataSource for pulling STIX2 content, a DataSink for pushing STIX2 content, and a DataStore for pulling/pushing.\n",
"\n",
"### DataSource API (snapshot)\n",
"* **get()** - search/retrieve most current STIX SDO/SRO via its ID\n",
"* **all_versions()** - search/retrieve all versions of STIX SDO/SRO via its id\n",
"* **query()** - search/retrieve STIX SDO/SRO(s) via search filters\n",
"* **add_filters()** - attach filter(s) to the DataSource\n",
" \n",
"### DataSink API (snapshot)\n",
"* **add()** - add a set of STIX SDO/SRO to a target endpoint.\n",
" \n",
"### DataStore API (snapshot) \n",
"\n",
"(super set of the DataSource and DataSink)\n",
"* **get()**\n",
"* **all_versions()**\n",
"* **query()**\n",
"* **add_filters()**\n",
"* **add()**\n",
"\n",
"The DataStore, DataSource, DataSink (referred to as \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then sublcassed into real DataStore suite(s). CTI Python STIX2 provides for the DataStore suites of **FileSystem**, **Memory**, and **TAXII**. Users are also encrouraged subclassing the base Data suite and creating their own custom DataStore suites."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CompositeDataSource\n",
"\n",
"**CompositeDataSource** is an available controller that can be used as a single interface to a set of defined DataSources. The purpose of this controller is allow for the grouping of **DataSources** and making get/query calls to a set of DataSources in one API call. **CompositeDataSource** can be used to organize/group **DataSources**, federate get()/all_versions()/query() calls, and reduce user code.\n",
"\n",
"**CompositeDataSource** is just a wrapper around a set of defined **DataSources** (e.g. FileSystemSource) that federates get()/all_versions()/query() calls individually to each of the attached **DataSources** , collects the results from each **DataSource** and returns them.\n",
"\n",
"Filters can be attached to **CompositeDataSources** just as they can be done to **DataStores** and **DataSources**. When get()/all_versions()/query() calls are made to the **CompositeDataSource**, it will pass along any query filters from the call and any of its own filters to the attached **DataSources**. To which, those attached **DataSources** may have their own attached filters as well. The effect is that all the filters are eventually combined when the get()/all_versions()/query() call is actually executed within a **DataSource**. \n",
"\n",
"A **CompositeDataSource** can also be attached to a **CompositeDataSource** for multiple layers of grouped **DataSources**.\n",
"\n",
"\n",
"### CompositeDataSource API (snapshot)\n",
"\n",
"* **get()** - search/retrieve from CompositeDataSource most current STIX object via its ID\n",
"* **all_versions()** - search/retrieve from CompositeDataSource all versions of STIX object via its ID\n",
"* **query()** - search/retrieve from CompositeDataSource STIX object(s) via search filters\n",
"* **add_data_source()** - Attach a DataSource to the CompositeDataSource\n",
"* **remove_data_source()** - Remove DataSource from the CompositeDataSource\n",
"* **add_filters()** - attach filter(s) to the CompositeDataSource\n",
"\n",
"### CompositeDataSource Examples\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"sys.path.append(\"/home/michael/oasis-python-stix2/cti-python-stix2/\")\n",
"sys.path\n",
"\n",
"from stix2 import CompositeDataSource, FileSystemStore\n",
"\n",
"\n",
"# TO BE COMPLETED, WAITING ON TAXIICollection,\n",
"# want to put a TAXIICollection in here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Filters\n",
"\n",
"The CTI Python STIX2 **DataStore** suites, to include **FileSystem**, **Memory** and **TAXII** all use the **Filters** module to allow for the querying of STIX content. The basic functionality is that filters can be created and supplied everytime to calls to **query()**, and/or attached to a **DataStore** so that every future query placed to that **DataStore** is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from **DataStores**.\n",
"\n",
"Filters are very simple, as they consist of a STIX object common property name, comparison operator and an object property value (i.e. value to compare to). Currently, CTI Python STIX2 supports **ONLY** STIX 2 object common properties:\n",
"\n",
"* created\n",
"* created_by_ref\n",
"* external_references.source_name\n",
"* external_references.description\n",
"* external_references.url\n",
"* external_references.external_id\n",
"* granular_markings.marking_ref\n",
"* granular_markings.selectors\n",
"* id\n",
"* labels\n",
"* modified\n",
"* object_marking_refs\n",
"* revoked\n",
"* type\n",
"\n",
"Supported operators on these properties:\n",
"\n",
"* =\n",
"* !=\n",
"* in\n",
"* >\n",
"* < \n",
"* ```>=```\n",
"* <=\n",
"\n",
"To align with STIX 2 object common properties, the value types of the common property values must be one of these (python) types:\n",
"\n",
"* bool\n",
"* dict\n",
"* float\n",
"* int\n",
"* list\n",
"* str\n",
"* tuple\n",
"\n",
"### Filter Examples"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"sys.path.append(\"/home/michael/oasis-python-stix2/cti-python-stix2/\")\n",
"sys.path\n",
"from stix2 import Filter\n",
"\n",
"# create filter for STIX objects that have external references to MITRE ATT&CK framework\n",
"f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n",
"\n",
"# create filter for STIX objects that are not of SDO type Attack-Pattnern\n",
"f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n",
"\n",
"# create filter for STIX objects that have the \"threat-report\" label\n",
"f2 = Filter(\"labels\", \"in\", \"threat-report\")\n",
"\n",
"# create filter for STIX objects that have been modified past the timestamp\n",
"f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n",
"\n",
"# create filter for STIX objects that have been revoked\n",
"f4 = Filter(\"revoked\", \"=\", True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For Filters to be applied to a query, they must be either supplied with the query call or attached a **DataStore**, more specifically to **DataSource** whether that **DataSource** is a part of a **DataStore** or stands by itself. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n",
"\n",
"fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n",
"fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n",
"\n",
"# attach filter to FileSystemStore\n",
"fs.add_filters(f)\n",
"\n",
"# attach multiple filters to FileSystemStore\n",
"fs.add_filters([f1,f2])\n",
"\n",
"# can also attach filters to a Source\n",
"# attach multiple filters to FileSystemSource\n",
"fs_source.add_filters([f3, f4])\n",
"\n",
"\n",
"mem = MemoryStore()\n",
"\n",
"# As it is impractical to only use MemorySink or MemorySource,\n",
"# attach a filter to a MemoryStore\n",
"mem.add_filters(f)\n",
"\n",
"# attach multiple filters to a MemoryStore\n",
"mem.add_filters([f1,f2])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "cti-python-stix2",
"language": "python",
"name": "cti-python-stix2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

521
docs/guide/FileSystem.ipynb Normal file
View File

@ -0,0 +1,521 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## FileSystem \n",
"\n",
"The FileSystem suite contains **FileSystemStore **, **FileSystemSource** and **FileSystemSink**. Under the hood, all FileSystem objects point to a file directory (on disk) that contains STIX2 content. \n",
"\n",
"The directory and file structure of the intended STIX2 content should be:\n",
"\n",
"```\n",
"stix2_content/\n",
" /STIX2 Domain Object type\n",
" STIX2 Domain Object\n",
" STIX2 Domain Object\n",
" .\n",
" .\n",
" .\n",
" /STIX2 Domain Object type\n",
" STIX2 Domain Object\n",
" STIX2 Domain Object\n",
" .\n",
" .\n",
" .\n",
" .\n",
" .\n",
" .\n",
" /STIX2 Domain Object type\n",
"```\n",
"\n",
"Essentially a master STIX2 content directory where each subdirectory aligns to a STIX2 domain object type (i.e. \"attack-pattern\", \"campaign\", \"malware\" etc..). Within each STIX2 domain object subdirectory are json files that are STIX2 domain objects of the specified type. The name of the json files correspond to the ID of the STIX2 domain object found within that file. A real example of the FileSystem directory structure:\n",
"\n",
"```\n",
"stix2_content/\n",
" /attack-pattern\n",
" attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6.json\n",
" attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22.json\n",
" attack-pattern--1b7ba276-eedc-4951-a762-0ceea2c030ec.json\n",
" /campaign\n",
" /course-of-action\n",
" course-of-action--2a8de25c-f743-4348-b101-3ee33ab5871b.json\n",
" course-of-action--2c3ce852-06a2-40ee-8fe6-086f6402a739.json\n",
" /identity\n",
" identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5.json\n",
" /indicator\n",
" /intrusion-set\n",
" /malware\n",
" malware--1d808f62-cf63-4063-9727-ff6132514c22.json\n",
" malware--2eb9b131-d333-4a48-9eb4-d8dec46c19ee.json\n",
" /observed-data\n",
" /report\n",
" /threat-actor\n",
" /vulnerability\n",
"```\n",
"\n",
"**FileSystemStore** is intended for use cases where STIX2 content is retrieved and pushed to the same file directory. As **FileSystemStore** is just a wrapper around a paired **FileSystemSource** and **FileSystemSink** that point the same file directory.\n",
"\n",
"Use cases where STIX2 content will only be retrieved or pushed, then a **FileSystemSource** and **FileSystemSink** can be used individually. Or for the use case where STIX2 content will be retrieved from one distinct file directory and pushed to another.\n",
"\n",
"### FileSystem API\n",
"\n",
"\n",
"### FileSystemSource (snapshot)\n",
"* **get()** - search/retrieve most current STIX SDO/SRO via its ID\n",
"* **all_versions()** - search/retrieve all versions of STIX SDO/SRO via its id\n",
"* **query()** - search/retrieve STIX SDO/SRO(s) via search filters\n",
"* **add_filters()** - attach filter(s) to the FileSystemSource\n",
" \n",
"### FileSystemSink (snapshot)\n",
"* **add()** - add STIX SDO/SRO to a target file system endpoint.\n",
" \n",
"### FileSystemStore (snapshot)\n",
"\n",
"(super set of the FileSystemSource and FileSystemSink)\n",
"* **get()**\n",
"* **all_versions()**\n",
"* **query()**\n",
"* **add_filters()**\n",
"* **add()**\n",
"\n",
"A note on **get()**, **all_versions()**, and **query()**. The format of the STIX2 content targeted by the FileSystem suite is json files. When STIX2 content (in json) is retrieved by the **FileSystemStore** from disk, the content will attempt to be parsed into full-featured python STIX2 objects and returned as such. \n",
"\n",
"A note on **add()**. When STIX content is added (pushed) to the file system, the STIX content can be supplied in the following forms: python STIX objects, python dicts (of valid STIX objects or Bundles), json-encoded strings (of valid STIX objects or Bundles), or a (python)list of any of the previously listed types. Any of the previous STIX content forms will be converted to a STIX json object (in a STIX Bundle) and written to disk. \n",
"\n",
"### FileSystem Examples\n",
"\n",
"#### FileSystemStore\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--00c3bfcb-99bd-4767-8c03-b08f585f5c8a\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:33:19.746Z\",\n",
" \"modified\": \"2017-05-31T21:33:19.746Z\",\n",
" \"name\": \"PowerDuke\",\n",
" \"description\": \"PowerDuke is a backdoor that was used by APT29 in 2016. It has primarily been delivered through Microsoft Word or Excel attachments containing malicious macros.[[Citation: Volexity PowerDuke November 2016]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0139\",\n",
" \"external_id\": \"S0139\"\n",
" },\n",
" {\n",
" \"source_name\": \"Volexity PowerDuke November 2016\",\n",
" \"description\": \"Adair, S.. (2016, November 9). PowerDuke: Widespread Post-Election Spear Phishing Campaigns Targeting Think Tanks and NGOs. Retrieved January 11, 2017.\",\n",
" \"url\": \"https://www.volexity.com/blog/2016/11/09/powerduke-post-election-spear-phishing-campaigns-targeting-think-tanks-and-ngos/\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"from stix2 import FileSystemStore\n",
"\n",
"\"\"\"\n",
"Working with the FileSystemStore, where STIX content can be retrieved and pushed to a file system.\n",
"\"\"\"\n",
"\n",
"# create FileSystemStore\n",
"fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n",
"\n",
"# retrieve STIX2 content from FileSystemStore\n",
"ap = fs.get(\"attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6\")\n",
"mal = fs.get(\"malware--00c3bfcb-99bd-4767-8c03-b08f585f5c8a\")\n",
"\n",
"# for visual purposes\n",
"print(mal)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from stix2 import ThreatActor, Indicator\n",
"\n",
"# create new STIX threat-actor\n",
"ta = ThreatActor(name=\"Adjective Bear\",\n",
" labels=[\"nation-state\"],\n",
" sophistication=\"innovator\",\n",
" resource_level=\"government\",\n",
" goals=[\n",
" \"compromising media outlets\",\n",
" \"water-hole attacks geared towards political, military targets\",\n",
" \"intelligence collection\"\n",
" ])\n",
"\n",
"# create new indicators\n",
"ind = Indicator(description=\"Crusades C2 implant\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n",
"\n",
"ind1 = Indicator(description=\"Crusades C2 implant 2\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '64c7e05e40a59511743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n",
"\n",
"# add STIX object (threat-actor) to FileSystemStore\n",
"fs.add(ta)\n",
"\n",
"# can also add multiple STIX objects to FileSystemStore in one call\n",
"fs.add([ind, ind1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### FileSystemSource - (if STIX content is only to be retrieved from FileSystem)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"attack-pattern\",\n",
" \"id\": \"attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:30:54.176Z\",\n",
" \"modified\": \"2017-05-31T21:30:54.176Z\",\n",
" \"name\": \"Indicator Removal from Tools\",\n",
" \"description\": \"If a malicious...command-line parameters, Process monitoring\",\n",
" \"kill_chain_phases\": [\n",
" {\n",
" \"kill_chain_name\": \"mitre-attack\",\n",
" \"phase_name\": \"defense-evasion\"\n",
" }\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Technique/T1066\",\n",
" \"external_id\": \"T1066\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"from stix2 import FileSystemSource\n",
"\"\"\"\n",
"Working with FileSystemSource for retrieveing STIX content.\n",
"\"\"\"\n",
"\n",
"# create FileSystemSource\n",
"fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n",
"\n",
"# retrieve STIX 2 objects\n",
"ap = fs_source.get(\"attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6\")\n",
"\n",
"# for visual purposes\n",
"print(ap)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--0f862b01-99da-47cc-9bdb-db4a86a95bb1\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:32:54.772Z\",\n",
" \"modified\": \"2017-05-31T21:32:54.772Z\",\n",
" \"name\": \"Emissary\",\n",
" \"description\": \"Emissary is a Trojan that has been used by Lotus Blossom. It shares code with Elise, with both Trojans being part of a malware group referred to as LStudio.[[Citation: Lotus Blossom Dec 2015]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0082\",\n",
" \"external_id\": \"S0082\"\n",
" },\n",
" {\n",
" \"source_name\": \"Lotus Blossom Dec 2015\",\n",
" \"description\": \"Falcone, R. and Miller-Osborn, J.. (2015, December 18). Attack on French Diplomat Linked to Operation Lotus Blossom. Retrieved February 15, 2016.\",\n",
" \"url\": \"http://researchcenter.paloaltonetworks.com/2015/12/attack-on-french-diplomat-linked-to-operation-lotus-blossom/\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n",
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--2a6f4c7b-e690-4cc7-ab6b-1f821fb6b80b\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:32:33.348Z\",\n",
" \"modified\": \"2017-05-31T21:32:33.348Z\",\n",
" \"name\": \"LOWBALL\",\n",
" \"description\": \"LOWBALL is malware used by admin@338. It was used in August 2015 in email messages targeting Hong Kong-based media organizations.[[Citation: FireEye admin@338]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0042\",\n",
" \"external_id\": \"S0042\"\n",
" },\n",
" {\n",
" \"source_name\": \"FireEye admin@338\",\n",
" \"description\": \"FireEye Threat Intelligence. (2015, December 1). China-based Cyber Threat Group Uses Dropbox for Malware Communications and Targets Hong Kong Media Outlets. Retrieved December 4, 2015.\",\n",
" \"url\": \"https://www.fireeye.com/blog/threat-research/2015/11/china-based-threat.html\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n",
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--00c3bfcb-99bd-4767-8c03-b08f585f5c8a\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:33:19.746Z\",\n",
" \"modified\": \"2017-05-31T21:33:19.746Z\",\n",
" \"name\": \"PowerDuke\",\n",
" \"description\": \"PowerDuke is a backdoor that was used by APT29 in 2016. It has primarily been delivered through Microsoft Word or Excel attachments containing malicious macros.[[Citation: Volexity PowerDuke November 2016]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0139\",\n",
" \"external_id\": \"S0139\"\n",
" },\n",
" {\n",
" \"source_name\": \"Volexity PowerDuke November 2016\",\n",
" \"description\": \"Adair, S.. (2016, November 9). PowerDuke: Widespread Post-Election Spear Phishing Campaigns Targeting Think Tanks and NGOs. Retrieved January 11, 2017.\",\n",
" \"url\": \"https://www.volexity.com/blog/2016/11/09/powerduke-post-election-spear-phishing-campaigns-targeting-think-tanks-and-ngos/\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n",
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--0db09158-6e48-4e7c-8ce7-2b10b9c0c039\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:32:55.126Z\",\n",
" \"modified\": \"2017-05-31T21:32:55.126Z\",\n",
" \"name\": \"Misdat\",\n",
" \"description\": \"Misdat is a backdoor that was used by Dust Storm from 2010 to 2011.[[Citation: Cylance Dust Storm]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0083\",\n",
" \"external_id\": \"S0083\"\n",
" },\n",
" {\n",
" \"source_name\": \"Cylance Dust Storm\",\n",
" \"description\": \"Gross, J. (2016, February 23). Operation Dust Storm. Retrieved February 25, 2016.\",\n",
" \"url\": \"https://www.cylance.com/hubfs/2015%20cylance%20website/assets/operation-dust-storm/Op%20Dust%20Storm%20Report.pdf?t=1456259131512\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n",
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--1d808f62-cf63-4063-9727-ff6132514c22\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:33:06.433Z\",\n",
" \"modified\": \"2017-05-31T21:33:06.433Z\",\n",
" \"name\": \"WEBC2\",\n",
" \"description\": \"WEBC2 is a backdoor used by APT1 to retrieve a Web page from a predetermined C2 server.[[Citation: Mandiant APT1 Appendix]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0109\",\n",
" \"external_id\": \"S0109\"\n",
" },\n",
" {\n",
" \"source_name\": \"Mandiant APT1 Appendix\",\n",
" \"description\": \"Mandiant. (n.d.). Appendix C (Digital) - The Malware Arsenal. Retrieved July 18, 2016.\",\n",
" \"url\": \"https://www.fireeye.com/content/dam/fireeye-www/services/pdfs/mandiant-apt1-report-appendix.zip\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"from stix2 import Filter\n",
"\n",
"# create filter for type=malware\n",
"query = [Filter(\"type\", \"=\", \"malware\")]\n",
"\n",
"# query on the filter\n",
"mals = fs_source.query(query)\n",
"\n",
"for mal in mals:\n",
" print(mal)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--00c3bfcb-99bd-4767-8c03-b08f585f5c8a\",\n",
" \"created_by_ref\": \"identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\",\n",
" \"created\": \"2017-05-31T21:33:19.746Z\",\n",
" \"modified\": \"2017-05-31T21:33:19.746Z\",\n",
" \"name\": \"PowerDuke\",\n",
" \"description\": \"PowerDuke is a backdoor that was used by APT29 in 2016. It has primarily been delivered through Microsoft Word or Excel attachments containing malicious macros.[[Citation: Volexity PowerDuke November 2016]]\",\n",
" \"labels\": [\n",
" \"malware\"\n",
" ],\n",
" \"external_references\": [\n",
" {\n",
" \"source_name\": \"mitre-attack\",\n",
" \"url\": \"https://attack.mitre.org/wiki/Software/S0139\",\n",
" \"external_id\": \"S0139\"\n",
" },\n",
" {\n",
" \"source_name\": \"Volexity PowerDuke November 2016\",\n",
" \"description\": \"Adair, S.. (2016, November 9). PowerDuke: Widespread Post-Election Spear Phishing Campaigns Targeting Think Tanks and NGOs. Retrieved January 11, 2017.\",\n",
" \"url\": \"https://www.volexity.com/blog/2016/11/09/powerduke-post-election-spear-phishing-campaigns-targeting-think-tanks-and-ngos/\"\n",
" }\n",
" ],\n",
" \"object_marking_refs\": [\n",
" \"marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"# add more filters to the query\n",
"query.append(Filter(\"modified\", \">\" , \"2017-05-31T21:33:10.772474Z\"))\n",
"\n",
"mals = fs_source.query(query)\n",
"\n",
"# for visual purposes\n",
"for mal in mals:\n",
" print(mal)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### FileSystemSink - (if STIX content is only to be pushed to FileSystem)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from stix2 import FileSystemSink, Campaign\n",
"\"\"\"\n",
"Working with FileSystemSink for pushing STIX content.\n",
"\"\"\"\n",
"# create FileSystemSink\n",
"fs_sink = FileSystemSink(\"/home/michael/Desktop/sample_stix2_data\")\n",
"\n",
"# create STIX objects and add to sink\n",
"camp = Campaign(name=\"The Crusades\",\n",
" objective=\"Infiltrating Israeli, Iranian and Palestinian digital infrastructure and government systems.\",\n",
" aliases=[\"Desert Moon\"])\n",
"\n",
"ind = Indicator(description=\"Crusades C2 implant\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n",
"\n",
"ind1 = Indicator(description=\"Crusades C2 implant\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n",
"\n",
"# add Campaign object to FileSystemSink\n",
"fs_sink.add(camp)\n",
"\n",
"# can also add STIX objects to FileSystemSink in on call\n",
"fs_sink.add([ind, ind1])"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "cti-python-stix2",
"language": "python",
"name": "cti-python-stix2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

280
docs/guide/Memory.ipynb Normal file
View File

@ -0,0 +1,280 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Memory\n",
"\n",
"The Memory suite consists of **MemoryStore**, **MemorySource**, and **MemorySink**. Under the hood, the Memory suite points to an in-memory dictionary. Similarly, the **MemoryStore** is a just a wrapper around a paired **MemorySource** and **MemorySink**; as there is quite limited uses for just a **MemorySource** or a **MemorySink**, it is recommended to always use **MemoryStore**. The **MemoryStore** is intended for retrieving/searching and pushing STIX content to memory. It is important to note that all STIX content in memory is not backed up on the file system (disk), as that functionality is ecompassed within the **FileSystemStore**. However, the Memory suite does provide some utility methods for saving and loading STIX content to disk. **MemoryStore.save_to_file()** allows for saving all the STIX content that is in memory to a json file. **MemoryStore.load_from_file()** allows for loading STIX content from a json-formatted file. \n",
"\n",
"\n",
"### Memory API\n",
"\n",
"### MemorySource (snapshot)\n",
"* **get()** - search/retrieve from memory most current STIX SDO/SRO via its ID\n",
"* **all_versions()** - search/retrieve from memory all versions of STIX SDO/SRO via its id\n",
"* **query()** - search/retrieve from memory STIX SDO/SRO(s) via search filters\n",
"* **load_from_file()** - load STIX content into memory from file\n",
"* **add_filters()** - attach filter(s) to the MemorySource\n",
" \n",
"### MemorySink (snapshot)\n",
"* **add()** - add a set of STIX SDO/SRO(s) to memory\n",
"* **save_to_file()** - save STIX content that is in memory to file\n",
" \n",
"### MemoryStore (snapshot)\n",
"\n",
"(super set of the MemorySource and MemorySink)\n",
"* **get()**\n",
"* **all_versions()**\n",
"* **query()**\n",
"* **load_from_file()**\n",
"* **save_to_file()**\n",
"* **add_filters()**\n",
"* **add()**\n",
"\n",
"A note on **load_from_file()** and **save()**. These methods both add STIX content to an internal dictionary (maintained by MemoryStore). STIX content that is to be added can be in the following forms: python STIX objects, python dicts (of valid STIX objects or Bundles), json-encoded strings (of valid STIX objects or Bundles), or a (python)list of any of the previously listed types. **MemoryStore** actually stores STIX content either as python STIX objects or as python dictionaries, reducing and converting any of the aforementioned types to one of those; and whatever form the STIX object is stored as , is what it will be returned as when queried or retrieved. Python STIX objects, and json-encoded strings (of STIX content) are stored as python STIX objects. Python dicts (of STIX objects) are stored as python dictionaries. This is done, as can be efficiently supported, in order to return STIX content in the form it was added to the **MemoryStore**. Also, for **load_from_file()**, STIX content is assumed to be in json form within the file, individually or in a Bundle. \n",
"\n",
"A note on **save_to_file()**. This method dumps all STIX content that is in MemoryStore to the specified file. The file format will be json, and the STIX content will be within a STIX Bundle. ntoe, the the output form will be a json STIX Bundle regardless of the form that the individual STIX objects are stored(i.e. supplied) to the MemoryStore. \n",
"\n",
"### Memory Examples\n",
"\n",
"#### MemoryStore"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"indicator\",\n",
" \"id\": \"indicator--340e9e90-80f7-4587-a1cb-172af51356e3\",\n",
" \"created\": \"2017-09-24T19:32:32.257Z\",\n",
" \"modified\": \"2017-09-24T19:32:32.257Z\",\n",
" \"labels\": [\n",
" \"malicious-activity\"\n",
" ],\n",
" \"description\": \"Crusades C2 implant\",\n",
" \"pattern\": \"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\",\n",
" \"valid_from\": \"2017-09-24T19:32:32.257186Z\"\n",
"}\n"
]
}
],
"source": [
"from stix2 import MemoryStore, Indicator\n",
"\n",
"# create default MemoryStore\n",
"mem = MemoryStore()\n",
"\n",
"# insert newly created indicator into memory\n",
"ind = Indicator(description=\"Crusades C2 implant\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n",
"\n",
"mem.add(ind)\n",
"\n",
"# for visual purposes\n",
"print(mem.get(ind.id))\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"indicator\",\n",
" \"id\": \"indicator--8f6df0be-6ada-4e2f-b711-886f355b3628\",\n",
" \"created\": \"2017-09-24T19:32:32.359Z\",\n",
" \"modified\": \"2017-09-24T19:32:32.359Z\",\n",
" \"labels\": [\n",
" \"malicious-activity\"\n",
" ],\n",
" \"description\": \"Crusades stage 2 implant variant\",\n",
" \"pattern\": \"[file:hashes.'SHA-256' = '31a45e777e4d58b97f4c43e38006f8cd6580ddabc4037905b2fad734712b582c']\",\n",
" \"valid_from\": \"2017-09-24T19:32:32.359374Z\"\n",
"}\n"
]
}
],
"source": [
"from stix2 import Malware\n",
"\n",
"# add multiple STIX objects into memory\n",
"ind2 = Indicator(description=\"Crusades stage 2 implant\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '70fa62fb218dd9d936ee570dbe531dfa4e7c128ff37e6af7a6a6b2485487e50a']\")\n",
"ind3 = Indicator(description=\"Crusades stage 2 implant variant\",\n",
" labels=[\"malicious-activity\"],\n",
" pattern=\"[file:hashes.'SHA-256' = '31a45e777e4d58b97f4c43e38006f8cd6580ddabc4037905b2fad734712b582c']\")\n",
"mal = Malware(labels=[\"rootkit\"], name= \"Alexios\")\n",
"\n",
"mem.add([ind2,ind3, mal])\n",
"\n",
"# for visual purposes\n",
"print(mem.get(ind3.id))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-----------------------\n",
"{\n",
" \"type\": \"malware\",\n",
" \"id\": \"malware--b14b7ae7-5738-46db-bf8d-aec4ddd7c38c\",\n",
" \"created\": \"2017-09-24T19:32:32.367Z\",\n",
" \"modified\": \"2017-09-24T19:32:32.367Z\",\n",
" \"name\": \"Alexios\",\n",
" \"labels\": [\n",
" \"rootkit\"\n",
" ]\n",
"}\n",
"-----------------------\n",
"{'name': 'Urban2', 'created': '2017-09-12T13:26:18.023Z', 'labels': ['rootkit'], 'modified': '2017-09-12T13:26:18.023Z', 'type': 'malware', 'id': 'malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4'}\n"
]
}
],
"source": [
"from stix2 import Filter\n",
"\n",
"# add dictionary (of STIX object) to MemoryStore\n",
"# (this dict would assumably come from output of another source,\n",
"# i.e. a loaded json file, NOT manually created as done here for sample purposes)\n",
"\n",
"malware = {\n",
" \"type\": \"malware\",\n",
" \"id\" : \"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\",\n",
" \"labels\": [\"rootkit\"],\n",
" \"name\": \"Urban2\",\n",
" \"created\": \"2017-09-12T13:26:18.023Z\",\n",
" \"modified\": \"2017-09-12T13:26:18.023Z\"\n",
"}\n",
"\n",
"mem.add(malware)\n",
"\n",
"results = mem.query([Filter(\"labels\",\"=\", \"rootkit\")])\n",
"for r in results:\n",
" # note that python STIX objects are pretty-printed\n",
" # due to some python dunder method magic, but normal\n",
" # python dictionaries are not by default. Thus the\n",
" # python STIX objects and python STIX dictionaries\n",
" # that match the above query can be easily identified visually\n",
" print(\"-----------------------\")\n",
" print(r)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"type\": \"report\",\n",
" \"id\": \"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\",\n",
" \"created\": \"2017-05-08T18:34:08.042Z\",\n",
" \"modified\": \"2017-05-08T18:34:08.042Z\",\n",
" \"name\": \"The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.\",\n",
" \"published\": \"2017-05-08T10:24:11.011Z\",\n",
" \"object_refs\": [\n",
" \"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\"\n",
" ],\n",
" \"labels\": [\n",
" \"threat-report\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"from stix2 import Filter\n",
"\n",
"# add json formatted string to MemoryStore\n",
"# Again, would NOT manual create json-formatted string\n",
"# but taken as an output form from another source\n",
"report = '{\"type\": \"report\",\"id\": \"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\",\"labels\": [\"threat-report\"], \"name\": \"The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.\", \"published\": \"2017-05-08T10:24:11.011Z\", \"object_refs\":[\"malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4\"], \"created\": \"2017-05-08T18:34:08.042Z\", \"modified\": \"2017-05-08T18:34:08.042Z\"}'\n",
"\n",
"mem.add(report)\n",
"\n",
"print(mem.get(\"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### load_from_file() and save_to_file()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{u'name': u'The Crusades: Looking into the relentless infiltration of Israels digital infrastructure.', u'created': u'2017-05-08T18:34:08.042Z', u'labels': [u'threat-report'], u'modified': u'2017-05-08T18:34:08.042Z', u'object_refs': [u'malware--2daa14d6-cbf3-4308-bb8e-226d324a08e4'], u'published': u'2017-05-08T10:24:11.011Z', u'type': u'report', u'id': u'report--2add14d6-bbf3-4308-bb8e-226d314a08e4'}\n"
]
}
],
"source": [
"mem_2 = MemoryStore()\n",
"\n",
"# save (dump) all STIX content in MemoryStore to json file\n",
"mem.save_to_file(\"path_to_target_file.json\")\n",
"\n",
"# load(add) STIX content from json file into MemoryStore\n",
"mem_2.load_from_file(\"path_to_target_file.json\")\n",
"\n",
"report = mem_2.get(\"report--2add14d6-bbf3-4308-bb8e-226d314a08e4\")\n",
"\n",
"# for visualpurposes\n",
"# Note: Since STIX content was added to MemoryStore as json,\n",
"# it is maintained as python dictionaries ( as opposed to STIX objects)\n",
"print(report)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "cti-python-stix2",
"language": "python",
"name": "cti-python-stix2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

117
docs/guide/TAXIICollection.ipynb Normal file
View File

@ -0,0 +1,117 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## TAXIICollection\n",
"\n",
"The TAXIICollection suite contains **TAXIICollectionStore**, **TAXIICollectionSource**, and **TAXIICollectionSink**. **TAXIICollectionStore** for pushing and retrieving STIX content to local/remote TAXII Collection(s). **TAXIICollectionSource** for retrieving STIX content to local/remote TAXII Collection(s). **TAXIICollectionSink** for pushing STIX content to local/remote TAXII Collection(s). Each of the interfaces is designed to be binded to a Collection from the taxii2client library (taxii2client.Collection), where all **TAXIICollection** API calls will be executed through that Collection instance.\n",
"\n",
"A note on TAXII2 searching/filtering of STIX content. TAXII2 server implementations natively support searching on the STIX2 object properties: id, type and version; API requests made to TAXII2 can contain filter arguments for those 3 properties. However, the **TAXIICollection** suite supports searching on all STIX2 common object properties (see **Filters** documentation for full listing). This works simply by augmenting the filtering that is done remotely at the TAXII2 server instance. **TAXIICollection** will seperate any supplied queries into TAXII supported filters and non-supported filters. During a **TAXIICollection** API call, TAXII2 supported filters get inserted into the TAXII2 server request (to be evaluated at the server). The rest of the filters are kept locally and then applied to the STIX2 content that is returned from the TAXII2 server, before being returned from the **TAXIICollection** API call. \n",
"\n",
"### TAXIICollection API\n",
"\n",
"### TAXIICollectionSource\n",
"* **get()** - search/retrieve most current STIX SDO/SRO via its ID\n",
"* **all_versions()** - search/retrieve all versions of STIX SDO/SRO via its id\n",
"* **query()** - search/retrieve STIX SDO/SRO(s) via search filters\n",
"* **add_filters()** - attach filter(s) to the TAXIICollectionSource\n",
"\n",
"### TAXIICollectionSink\n",
"* **add()** - add a set of STIX SDO/SRO to a TAXII Collection endpoint.\n",
"\n",
"### TAXIICollectionStore\n",
"\n",
"(super set of TAXIICollectionSource and TAXIICollectionSink)\n",
"* **get()**\n",
"* **all_versions()**\n",
"* **query()**\n",
"* **add()**\n",
"* **add_filters()**\n",
"\n",
"### TAXIICollection Examples\n",
"\n",
"#### TAXIICollectionSource"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from stix2 import TAXIICollectionSource\n",
"from taxii2client import Collection\n",
"\n",
"# establish TAXII 2 Collection instance\n",
"collection = Collection(\"do we have a TAXII 2 instance anywhere? or Vargass\")\n",
"\n",
"tc_source = TAXIICollectionSource(collection)\n",
"\n",
"# NEED TO BE COMPLETED\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"#### TAXIICollectionSink"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# NEED TO BE COMPLETED"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### TAXIICollectionStore"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# NEED TO BE COMPLETED"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}