MISP
/
PyMISP
mirror of https://github.com/MISP/PyMISP
2
0
Fork 0
Code Issues Releases Wiki Activity
PyMISP/docs/tutorial/a.7-rest-api-extensive-rest...

1615 lines
90 KiB
Plaintext
Raw Normal View History

new: [doc] added the Jupyter notebook used in a.7-rest-api-extensive-restsearch
2023-02-06 18:18:12 +01:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Extracting data from MISP using PyMISP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Recovering the API KEY"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Go to `Global Actions` then `My Profile`\n",
"- Access the `/users/view/me` URL"
]
},
{
"cell_type": "code",
"execution_count": 491,
"metadata": {},
"outputs": [],
"source": [
"from pymisp import PyMISP\n",
"import urllib3\n",
"urllib3.disable_warnings()\n",
"\n",
"misp_url = 'https://localhost:8443/'\n",
"misp_key = 'GqfuZo444EFlylND0XaKZsEXgWgkPgguUZ6KVRuq'\n",
"# Should PyMISP verify the MISP certificate\n",
"misp_verifycert = False\n",
"\n",
"misp = PyMISP(misp_url, misp_key, misp_verifycert)"
]
},
{
"cell_type": "code",
"execution_count": 492,
"metadata": {},
"outputs": [],
"source": [
"import datetime\n",
"from pprint import pprint\n",
"import base64\n",
"import subprocess"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Retreiving an Event"
]
},
{
"cell_type": "code",
"execution_count": 493,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<MISPEvent(info=Test PUSH filtering type)\n",
"<class 'dict'>\n"
]
}
],
"source": [
"r1 = misp.get_event('7907c4a9-a15c-4c60-a1b4-1d214cf8cf41', pythonify=True)\n",
"print(r1)\n",
"r2 = misp.get_event(2, pythonify=False)\n",
"print(type(r2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Searching the Event index"
]
},
{
"cell_type": "code",
"execution_count": 494,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"7907c4a9-a15c-4c60-a1b4-1d214cf8cf41\n"
]
}
],
"source": [
"r = misp.search_index(pythonify=True)\n",
"print(r[1].uuid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Only published Events"
]
},
{
"cell_type": "code",
"execution_count": 495,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[<MISPEvent(info=Test 43214324), <MISPEvent(info=Test enrichment via WF), <MISPEvent(info=Big event), <MISPEvent(info=Small event)]\n"
]
}
],
"source": [
"r = misp.search_index(published=True, pythonify=True)\n",
"print(r)\n",
"# print(r[0].to_dict())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Playing with time"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Multiple type of timestamps for Events**\n",
"- `timestamp`: Timestamp of the **last modification** of the Event or its content (include Attributes, Objects, Tags, ...)\n",
"- `published_timestamp`: Timestamp of the **last publication** of the Event\n",
"- To generate report, you usually want to use `publish_timestamp`\n",
"\n",
"**Multiple type of dates for Events**\n",
"- `date_from`: Only events having a more recent date will be returned\n",
"- `date_to`: Only events having an older date will be returned\n",
"- Both can be used at once to specify a time window\n"
]
},
{
"cell_type": "code",
"execution_count": 496,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[<MISPEvent(info=Test 43214324), <MISPEvent(info=Test enrichment via WF)]\n"
]
}
],
"source": [
"# Using string literal\n",
"sinceLastMonth = '30d'\n",
"# Using Python's datetime\n",
"sinceLastMonth = datetime.date.today() - datetime.timedelta(days=30)\n",
"\n",
"r = misp.search_index(published=True, publish_timestamp=sinceLastMonth, pythonify=True)\n",
"print(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Data returned\n",
"- Searching the index will only returns high-level information about the Event and its attached context\n",
"\n",
"- Can be useful for:\n",
" - Statistics about number of created Event\n",
" - Statistics about Organisation creating Event over time\n",
" - Statistics about distribution level usage\n",
"- And, **If Event correctly contextualized**\n",
" - Statistics about **type of incident**\n",
" - Adversary tactics and techniques with **MITRE ATT&CK** usage\n",
" - Malware familly"
]
},
{
"cell_type": "code",
"execution_count": 497,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Event properties\n",
"['uuid', 'info', 'distribution', 'threat_level_id', 'analysis', 'published', 'date', 'id', 'orgc_id', 'org_id', 'timestamp', 'publish_timestamp', 'sighting_timestamp', 'sharing_group_id', 'Org', 'Orgc', 'attribute_count', 'proposal_email_lock', 'locked', 'disable_correlation', 'extends_uuid', 'GalaxyCluster', 'EventTag']\n",
"\n",
" # Event Tags (12)\n",
"{'Tag': {'colour': '#326300',\n",
" 'id': '29',\n",
" 'is_galaxy': False,\n",
" 'name': 'circl:incident-classification=\"phishing\"'},\n",
" 'event_id': '18',\n",
" 'id': '69',\n",
" 'local': False,\n",
" 'relationship_type': '',\n",
" 'tag_id': '29'}\n",
"\n",
" # Event Clusters (11)\n"
]
}
],
"source": [
"event = r[0].to_dict()\n",
"event_properties = event.keys()\n",
"print('# Event properties')\n",
"print(list(event_properties))\n",
"\n",
"print('\\n # Event Tags ({0})'.format(len(event['EventTag'])))\n",
"pprint(event['EventTag'][0])\n",
"\n",
"print('\\n # Event Clusters ({0})'.format(len(event['GalaxyCluster'])))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Useful parameters\n",
"\n",
"- `attribute` (Optional[str]) *Filter events on attribute's value*\n",
"- `published` (Optional[bool])\n",
"- `hasproposal` (Optional[bool])\n",
"- `eventid` (Optional[str, int])\n",
"- `tags` (Optional[str, List[str]])\n",
"- `date_from` (Optional[datetime, date, int, str, float, None])\n",
"- `date_to` (Optional[datetime, date, int, str, float, None])\n",
"- `eventinfo` (Optional[str])\n",
"- `threatlevel` (Optional[str, int])\n",
"- `analysis` (Optional[str, int])\n",
"- `distribution` (Optional[str, int])\n",
"- `sharinggroup` (Optional[str, int])\n",
"- `org` (Optional[str, List[[str, int]])\n",
"- `timestamp` (Optional[datetime, date, int, str, float, None, List[[datetime, date, int, str, float, None], [datetime, date, int, str, float, None]]])\n",
" - timestamp=(datetime.today() - timedelta(days=1))\n",
" - timestamp=['14d', '7d']\n",
" - timestamp=int(datetime.today().timestamp())\n",
"- `publish_timestamp` (Optional[datetime, date, int, str, float, None, List[[datetime, date, int, str, float, None], [datetime, date, int, str, float, None]]])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Retreiving data with RestSearch\n",
"\n",
"The `RestSearch` endpoint can be used on multiple scopes. It has more filtering parameters and is generally flexible.\n",
"\n",
"Supported scopes (also called Controllers): `events`, `attributes`, `objects`\n",
"\n",
"### `/events/restSearch` VS `/attributes/restSearch`\n",
"\n",
"- Both endpoints support most of the parameter\n",
"- They differs in the data returned\n",
" - `/events/restSearch` returns the whole Event with its child elements (Attributes, Objects, Proposals, ..)\n",
" - `/attributes/restSearch` returns all attributes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Getting only metadata: Do not include child elements (such as Attributes, ...)"
]
},
{
"cell_type": "code",
"execution_count": 498,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[<MISPEvent(info=Test), <MISPEvent(info=Test PUSH filtering type), <MISPEvent(info=Decaying example), <MISPEvent(info=Test tag filtering), <MISPEvent(info=Should not be pulled), <MISPEvent(info=Event report example), <MISPEvent(info=Wireshark test event), <MISPEvent(info=Test 4), <MISPEvent(info=Test 43214324), <MISPEvent(info=Test btc), <MISPEvent(info=Analysis of a Flubot malware captured by a honeypot), <MISPEvent(info=Test enrichment via WF), <MISPEvent(info=Test TLP replace), <MISPEvent(info=Test event for MM), <MISPEvent(info=Big event), <MISPEvent(info=test 77), <MISPEvent(info=Small event), <MISPEvent(info=Small event), <MISPEvent(info=Infection via spear-phishing email), <MISPEvent(info=test event TLPs), <MISPEvent(info=test), <MISPEvent(info=Test event ip|port -> ip-port), <MISPEvent(info=test bgp), <MISPEvent(info=test tlp:white and clear), <MISPEvent(info=Test)]\n"
]
}
],
"source": [
"r = misp.search(controller='events', metadata=True, pythonify=True)\n",
"print(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Searching Attributes with RestSearch"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Searching for values"
]
},
{
"cell_type": "code",
"execution_count": 499,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Simple value: [<MISPAttribute(type=ip-src, value=8.8.8.8), <MISPAttribute(type=ip-dst, value=8.8.8.8), <MISPAttribute(type=ip-dst|port, value=8.8.8.8|443)]\n",
"List of values: [<MISPAttribute(type=ip-src, value=5.4.2.1), <MISPAttribute(type=ip-src, value=8.8.8.8), <MISPAttribute(type=ip-dst, value=8.8.8.8), <MISPAttribute(type=ip-dst|port, value=8.8.8.8|443)]\n",
"Wildcard: [<MISPAttribute(type=url, value=https://www.github.com/stricaud)]\n"
]
}
],
"source": [
"r1 = misp.search(controller='attributes', value='8.8.8.8', pythonify=True)\n",
"print('Simple value:', r1)\n",
"\n",
"r2 = misp.search(controller='attributes', value=['8.8.8.8', '5.4.2.1'], pythonify=True)\n",
"print('List of values:', r2)\n",
"\n",
"r3 = misp.search(controller='attributes', value=['https://www.github.com/%'], pythonify=True)\n",
"print('Wildcard:', r3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Searching for types"
]
},
{
"cell_type": "code",
"execution_count": 500,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[<MISPAttribute(type=first-name, value=Sam), <MISPAttribute(type=first-name, value=NETRICSA), <MISPAttribute(type=first-name, value=Mental), <MISPAttribute(type=first-name, value=Andrew)]\n",
"[<MISPAttribute(type=attachment, value=SeriousSam.png), <MISPAttribute(type=attachment, value=mental.png), <MISPAttribute(type=attachment, value=EDF.png), <MISPAttribute(type=attachment, value=malicious.exe), <MISPAttribute(type=attachment, value=malicious.exe), <MISPAttribute(type=attachment, value=original.jpeg), <MISPAttribute(type=attachment, value=payload-1-8), <MISPAttribute(type=attachment, value=drawing.svg), <MISPAttribute(type=attachment, value=drawing.png), <MISPAttribute(type=attachment, value=Screenshot from 2021-10-19 16-31-56.png), <MISPAttribute(type=malware-sample, value=sample.apk|eff61f1bf7b14d261d5b421208d1bf68), <MISPAttribute(type=malware-sample, value=malware.exe|70f3bc193dfa56b78f3e6e4f800f701f)]\n"
]
}
],
"source": [
"r1 = misp.search(controller='attributes', type_attribute='first-name', pythonify=True)\n",
"print(r1)\n",
"\n",
"r2 = misp.search(controller='attributes', type_attribute=['malware-sample', 'attachment'], pythonify=True)\n",
"print(r2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Searching for tags"
]
},
{
"cell_type": "code",
"execution_count": 501,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Simple tag: 9\n",
"\tFirst Attribute [<MISPTag(name=tlp:red)>]\n",
"List of tags: 18\n",
"\tThird Attribute [<MISPTag(name=PAP:RED)>, <MISPTag(name=adversary:infrastructure-type=\"exploit-distribution-point\")>]\n"
]
}
],
"source": [
"r1 = misp.search(controller='attributes', tags='tlp:red', pythonify=True)\n",
"print('Simple tag:', len(r1))\n",
"print('\\tFirst Attribute', r1[0].Tag)\n",
"\n",
"r2 = misp.search(controller='attributes', tags=['PAP:RED', 'tlp:red'], pythonify=True)\n",
"print('List of tags:', len(r2))\n",
"print('\\tThird Attribute', r2[2].Tag)"
]
},
{
"cell_type": "code",
"execution_count": 502,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Wildcard: 22\n",
"\tTags of all Attributes: [[], [], [], [], [], [], [], [], [], [], [], [], [<MISPTag(name=tlp:red)>], [], [], [], [], [], [], [], [], [<MISPTag(name=test_foo)>]]\n",
"\n",
"Open question: Why do we have Attributes despite them not having the correct tag attached?\n",
"\n"
]
}
],
"source": [
"r3 = misp.search(controller='attributes', tags=['misp-galaxy:target-information=%'], pythonify=True)\n",
"print('Wildcard:', len(r3))\n",
"print('\\tTags of all Attributes:', [attr.Tag for attr in r3])\n",
"print()\n",
"print(base64.b64decode('T3BlbiBxdWVzdGlvbjogV2h5IGRvIHdlIGhhdmUgQXR0cmlidXRlcyBkZXNwaXRlIHRoZW0gbm90IGhhdmluZyB0aGUgY29ycmVjdCB0YWcgYXR0YWNoZWQ/Cg==').decode())"
]
},
{
"cell_type": "code",
"execution_count": 503,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"All unique Event tags: {'misp-galaxy:target-information=\"Canada\"', 'misp-galaxy:target-information=\"China\"', 'misp-galaxy:target-information=\"Germany\"', 'misp-galaxy:target-information=\"Luxembourg\"'}\n"
]
}
],
"source": [
"allEventTags = [\n",
" [tag.name for tag in misp.get_event(attr.event_id, pythonify=True).Tag if tag.name.startswith('misp-galaxy:target-information=')]\n",
" for attr in r3\n",
"]\n",
"allUniqueEventTag = set()\n",
"for tags in allEventTags:\n",
" for tag in tags:\n",
" allUniqueEventTag.add(tag)\n",
"print('All unique Event tags:', allUniqueEventTag)"
]
},
{
"cell_type": "code",
"execution_count": 504,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Negation: 17\n",
"All unique Event tags: {'misp-galaxy:target-information=\"Canada\"', 'misp-galaxy:target-information=\"China\"', 'misp-galaxy:target-information=\"Germany\"'}\n"
]
}
],
"source": [
"r4 = misp.search(\n",
" controller='attributes',\n",
" tags=['misp-galaxy:target-information=%', '!misp-galaxy:target-information=\"Luxembourg\"'],\n",
" pythonify=True)\n",
"print('Negation:', len(r4))\n",
"\n",
"\n",
"# Showing unique Event tags\n",
"allEventTags = [\n",
" [tag.name for tag in misp.get_event(attr.event_id, pythonify=True).Tag if tag.name.startswith('misp-galaxy:target-information=')]\n",
" for attr in r4\n",
"]\n",
"allUniqueEventTag = set()\n",
"for tags in allEventTags:\n",
" for tag in tags:\n",
" allUniqueEventTag.add(tag)\n",
"print('All unique Event tags:', allUniqueEventTag)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Want to also have the Event tags included**?"
]
},
{
"cell_type": "code",
"execution_count": 505,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tags of first attibute: []\n",
"Tags of first attibute: ['tlp:white', 'osint:lifetime=\"perpetual\"', 'osint:certainty=\"50\"', 'workflow:state=\"draft\"', 'misp-galaxy:threat-actor=\"APT 29\"', 'smo:sync', 'misp-galaxy:target-information=\"Canada\"', 'misp-galaxy:target-information=\"China\"', 'misp-galaxy:sector=\"Defense\"', 'misp-galaxy:sector=\"Infrastructure\"', 'misp-galaxy:malpedia=\"Kobalos\"', 'misp-galaxy:mitre-attack-pattern=\"SSH - T1021.004\"', 'misp-galaxy:mitre-attack-pattern=\"Software - T1592.002\"']\n"
]
}
],
"source": [
"r5 = misp.search(\n",
" controller='attributes',\n",
" tags='misp-galaxy:target-information=%',\n",
" pythonify=True)\n",
"print('Tags of first attibute:', [tag.name for tag in r5[0].Tag])\n",
"\n",
"r6 = misp.search(\n",
" controller='attributes',\n",
" tags='misp-galaxy:target-information=%',\n",
" includeEventTags=True,\n",
" pythonify=True)\n",
"print('Tags of first attibute:', [tag.name for tag in r6[0].Tag])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Complex query**"
]
},
{
"cell_type": "code",
"execution_count": 506,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Or: 1056\n",
"[['tlp:amber'], ['tlp:amber'], ['tlp:amber'], ['tlp:amber'], ['tlp:amber']]\n",
"\n",
"And: 5\n",
"[['adversary:infrastructure-type=\"c2\"', 'tlp:amber'],\n",
" ['adversary:infrastructure-type=\"c2\"', 'tlp:amber'],\n",
" ['adversary:infrastructure-type=\"c2\"', 'tlp:amber'],\n",
" ['adversary:infrastructure-type=\"c2\"', 'tlp:amber'],\n",
" ['adversary:infrastructure-type=\"c2\"', 'tlp:amber']]\n"
]
}
],
"source": [
"complex_query = misp.build_complex_query(or_parameters=['tlp:amber', 'adversary:infrastructure-type=\"c2\"'])\n",
"r7 = misp.search(\n",
" controller='attributes',\n",
" tags=complex_query,\n",
" includeEventTags=True,\n",
" pythonify=True)\n",
"print('Or:', len(r7))\n",
"pprint([\n",
" [tag.name for tag in attr.Tag if (tag.name == 'tlp:amber' or tag.name == 'adversary:infrastructure-type=\"c2\"')] for attr in r7[:5]\n",
"])\n",
"print()\n",
"\n",
"complex_query = misp.build_complex_query(and_parameters=['tlp:amber', 'adversary:infrastructure-type=\"c2\"'])\n",
"r8 = misp.search(\n",
" controller='attributes',\n",
" tags=complex_query,\n",
" includeEventTags=True,\n",
" pythonify=True)\n",
"print('And:', len(r8))\n",
"pprint([\n",
" [tag.name for tag in attr.Tag if (tag.name == 'tlp:amber' or tag.name == 'adversary:infrastructure-type=\"c2\"')] for attr in r8\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Searching on GalaxyCluster metadata"
]
},
{
"cell_type": "code",
"execution_count": 507,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Events: 2\n",
"[['misp-galaxy:target-information=\"Canada\"',\n",
" 'misp-galaxy:target-information=\"China\"'],\n",
" ['misp-galaxy:target-information=\"Luxembourg\"']]\n"
]
}
],
"source": [
"body = {\n",
" 'galaxy.member-of': 'NATO',\n",
" 'galaxy.official-languages': 'French',\n",
"}\n",
"\n",
"events = misp.direct_call('/events/restSearch', body)\n",
"print('Events: ', len(events))\n",
"pprint([\n",
" [tag['name'] for tag in event['Event']['Tag'] if tag['name'].startswith('misp-galaxy:target-information')] for event in events\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- **Note 1**: The `galaxy.*` instructions are not supported by PyMISP\n",
"- **Note 2**: Each `galaxy.*` instructions are **AND**ed and are applied for the same cluster\n",
" - Cannot combine from different clusters\n",
" - Combining `Galaxy.official-languages` and `Galaxy.synonyms` would likely gives no result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Searching on creator Organisation metadata"
]
},
{
"cell_type": "code",
"execution_count": 508,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Organisation nationality: {'admin_org': '', 'CIRCL': '', 'ORGNAME': '', 'Training': 'Luxembourg'}\n",
"Events: 4\n",
"Org for each Event: ['Training', 'Training', 'Training', 'Training']\n"
]
}
],
"source": [
"all_orgs = misp.organisations()\n",
"print('Organisation nationality:', {org['Organisation']['name']: org['Organisation']['nationality'] for org in all_orgs})\n",
"\n",
"body = {\n",
" 'org.nationality': ['Luxembourg'],\n",
" 'org.sector': ['financial'],\n",
"}\n",
"\n",
"events = misp.direct_call('/events/restSearch', body)\n",
"print('Events: ', len(events))\n",
"print('Org for each Event:', [event['Event']['Orgc']['name'] for event in events])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- **Note 1**: The `org.*` instructions are not supported by PyMISP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### ReturnFormat"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**CSV**"
]
},
{
"cell_type": "code",
"execution_count": 509,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"uuid,event_id,category,type,value,comment,to_ids,date,object_relation,attribute_tag,object_uuid,object_name,object_meta_category\n",
"\"724d5417-41e6-40a5-b368-bdfbe652302a\",2,\"Network activity\",\"ip-dst\",\"4.3.2.1\",\"Hello all!\",0,1639127173,\"\",\"\",\"\",\"\",\"\"\n",
"\"ba8e1a5a-6bb6-4ae5-9872-0a01b6b05cad\",2,\"Network activity\",\"ip-dst\",\"5.3.1.2\",\"\",1,1639060465,\"ip\",\"\",\"\",\"\",\"\"\n",
"\"8c16cf20-d5bd-4ed3-b243-98c00c16e591\",2,\"Network activity\",\"ip-dst\",\"23.1.4.2\",\"\",1,1639126626,\"ip\",\"\",\"\",\"\",\"\"\n",
"\"25a7bbb0-31f6-4525-94c0-89af86030201\",16,\"Network activity\",\"ip-dst\",\"127.0.0.1\",\"\",1,1645191487,\"ip-dst\",\"\",\"\",\"\",\"\"\n",
"\"f3eb2f37-d08d-4dbb-be0c-346ac508693f\",16,\"Network activity\",\"ip-dst\",\"127.0.0.1\",\"\",1,1645191487,\"ip-dst\",\"\",\"\",\"\",\"\"\n",
"\"f0a002d8-38a5-40f9-9a62-7e975cc8f987\",16,\"Network activity\",\"ip-dst\",\"127.0.0.1\",\"\",1,1645191487,\"ip-dst\",\"\",\"\",\"\",\"\"\n",
"\"61bfb8e3-20e3-4f37-905d-9d4e14f2564a\",20,\"Network activity\",\"ip-dst\",\"8.231.77.176\",\"\",1,1665471239,\"ip\",\"PAP:RED,adversary:infrastructure-type=\"\"exploit-distribution-point\"\"\",\"\",\"\",\"\"\n",
"\"1ac08260-a5d6-4bee-bdcd-1525685ea07d\",20,\"Network activity\",\"ip-dst\",\"226.140.183.77\",\"\",1,1665471204,\"ip\",\"PAP:RED,adversary:infrastructure-type=\"\"c2\"\"\",\"\",\"\",\"\"\n",
"\"78ce291d-241b-4162-8d6b-6a85964a31b8\",20,\"Network activity\",\"ip-dst\",\"2efe:65b4:7533:4f5f:1081:995:ff87:348f\",\"\",1,1665471204,\"ip\",\"PAP:RED,adversary:infrastructure-type=\"\"c2\"\"\",\"\",\"\",\"\"\n",
"\"b760f7a7-0d96-4b47-86b2-d5524cd2eff0\",26,\"Network activity\",\"ip-dst\",\"8.8.8.8\",\"\",1,1663321650,\"ip\",\"\",\"\",\"\",\"\"\n",
"\"9023deba-1ba0-4ab3-a0bf-64a2d5c90520\",29,\"Network activity\",\"ip-dst\",\"81.177.170.166\",\"\",1,1665472920,\"ip\",\"adversary:infrastructure-type=\"\"c2\"\",misp-galaxy:mitre-attack-pattern=\"\"Botnet - T1583.005\"\"\",\"\",\"\",\"\"\n",
"\"c9d681ad-4087-4847-8f93-aef2e54452f2\",42,\"Network activity\",\"ip-dst\",\"2.2.2.2\",\"\",0,1671095982,\"ip\",\"\",\"\",\"\",\"\"\n",
"\"60950f6a-b3bf-4a0a-b901-43308e2f761a\",2,\"Network activity\",\"ip-src\",\"1.2.3.4\",\"\",0,1639060409,\"\",\"\",\"\",\"\",\"\"\n",
"\"f2a6eb8c-7a3e-4524-8036-1b90cb18fe75\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.0\",\"today\",1,1622184577,\"\",\"\",\"\",\"\",\"\"\n",
"\"93bc9e55-20e9-4be1-b3e5-057e56a3b82e\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.1\",\"today - 1 days\",1,1622184577,\"\",\"\",\"\",\"\",\"\"\n",
"\"f7771a53-fbdf-4980-822d-9a2339ce9076\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.2\",\"today - 2 days\",1,1622184577,\"\",\"\",\"\",\"\",\"\"\n",
"\"4972022a-26fd-4270-b614-506a9c951be6\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.3\",\"today - 3 days\",1,1622184578,\"\",\"admiralty-scale:information-credibility=\"\"1\"\",admiralty-scale:source-reliability=\"\"a\"\"\",\"\",\"\",\"\"\n",
"\"c661cd4b-0474-48eb-b4ed-eb02f6b569ea\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.4\",\"today - 4 days\",1,1622184578,\"\",\"\",\"\",\"\",\"\"\n",
"\"42f68239-a794-492c-8fed-7520677824b0\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.5\",\"today - 5 days\",1,1622184578,\"\",\"\",\"\",\"\",\"\"\n",
"\"d6404ba7-c847-49b8-8748-3029ce62e2b0\",7,\"Payload delivery\",\"ip-src\",\"149.23.54.6\",\"today - 6 days\",1,1622184578,\"\",\"\",\"\",\"\",\"\"\n",
"\"f04de340-ec63-471e-b5a2-66c3fe0676b6\",9,\"Network activity\",\"ip-src\",\"5.4.2.1\",\"\",0,1650956697,\"\",\"misp-galaxy:mitre-course-of-action=\"\"Access Token Manipulation Mitigation - T1134\"\"\",\"\",\"\",\"\"\n",
"\"7bb5432f-3d67-4d59-8a43-04e57e0dcc3f\",16,\"Network activity\",\"ip-src\",\"127.0.0.1\",\"\",1,1645191487,\"ip-src\",\"\",\"\",\"\",\"\"\n",
"\"b663b3b3-92af-41bf-a18f-8582bd0983b1\",16,\"Network activity\",\"ip-src\",\"127.0.0.1\",\"\",1,1645191487,\"ip-src\",\"\",\"\",\"\",\"\"\n",
"\"0ee4a946-d826-4884-aa28-e1b9da8cbbcb\",16,\"Network activity\",\"ip-src\",\"127.0.0.1\",\"\",1,1645191487,\"ip-src\",\"\",\"\",\"\",\"\"\n",
"\"1f4b0f6b-6cf9-47bf-acd4-f15b33e7d588\",21,\"Network activity\",\"ip-src\",\"185.194.93.14\",\"Attribute #281 enriched by dns.\",0,1668077578,\"\",\"\",\"\",\"\",\"\"\n",
"\"9f7f2d28-bcc8-466e-847f-3cf2a1ec4070\",21,\"Network activity\",\"ip-src\",\"31.22.121.122\",\"Attribute #291 enriched by dns.\",0,1663922175,\"\",\"\",\"\",\"\",\"\"\n",
"\"8153e053-c7c3-4a34-ae1c-b5cd3c80ba06\",22,\"Network activity\",\"ip-src\",\"8.231.77.176\",\"\",0,1659602097,\"\",\"\",\"\",\"\",\"\"\n",
"\"a57f70a2-70dd-4ea4-b879-fbcd03d465df\",24,\"Network activity\",\"ip-src\",\"8.231.77.176\",\"\",0,1662025545,\"\",\"another:tag\",\"\",\"\",\"\"\n",
"\"af044e10-5549-4018-bc6b-162cde1a1016\",21,\"Network activity\",\"ip-src\",\"8.231.77.176\",\"\",0,1661517935,\"\",\"\",\"\",\"\",\"\"\n",
"\"fbb12142-0f82-4430-b0bc-2b1f9e26af67\",23,\"Network activity\",\"ip-src\",\"8.231.77.176\",\"\",0,1661518277,\"\",\"\",\"\",\"\",\"\"\n",
"\"a783c55f-ac52-44b4-8be1-74d52bc2c4c3\",17,\"Network activity\",\"ip-src\",\"8.231.77.176\",\"\",0,1661517997,\"\",\"\",\"\",\"\",\"\"\n",
"\"90f6fd39-a426-43b3-9157-0c48bf0710fb\",22,\"Network activity\",\"ip-src\",\"31.22.121.122\",\"\",0,1661762437,\"\",\"\",\"\",\"\",\"\"\n",
"\"bc0a1ba5-d337-42b3-81fe-9d4b75a17bec\",26,\"Network activity\",\"ip-src\",\"185.194.93.14\",\"\",0,1663137408,\"\",\"\",\"\",\"\",\"\"\n",
"\"93931645-c86c-4dcf-aa4e-591edab44c4e\",26,\"Network activity\",\"ip-src\",\"8.8.8.8\",\"\",1,1663320641,\"\",\"\",\"\",\"\",\"\"\n",
"\n",
"\n"
]
}
],
"source": [
"r1 = misp.search(\n",
" controller='attributes',\n",
" type_attribute=['ip-src', 'ip-dst'],\n",
" return_format='csv')\n",
"print(r1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Aggregated context** with `context-markdown`, `context` and `attack`"
]
},
{
"cell_type": "code",
"execution_count": 510,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Aggregated context data\n",
"## Tags and Taxonomies\n",
"#### admiralty-scale\n",
"*The Admiralty Scale or Ranking (also called the NATO System) is used to rank the reliability of a source and the credibility of an information. Reference based on FM 2-22.3 (FM 34-52) HUMAN INTELLIGENCE COLLECTOR OPERATIONS and NATO documents.*\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #0eb100; color: white\">admiralty-scale:information-credibility=&quot;1&quot;</span></span>\n",
"\n",
" - **information-credibility**: Information Credibility\n",
" - **1**: Confirmed by other sources\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #0fc000; color: white\">admiralty-scale:information-credibility=&quot;2&quot;</span></span>\n",
"\n",
" - **information-credibility**: Information Credibility\n",
" - **2**: Probably true\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #054300; color: white\">admiralty-scale:source-reliability=&quot;a&quot;</span></span>\n",
"\n",
" - **source-reliability**: Source Reliability\n",
" - **a**: Completely reliable\n",
"#### economical-impact\n",
"*Economical impact is a taxonomy to describe the financial impact as positive or negative gain to the tagged information (e.g. data exfiltration loss, a positive gain for an adversary).*\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #038e00; color: white\">economical-impact:loss=&quot;less-than-1B-euro&quot;</span></span>\n",
"\n",
" - **loss**: Loss\n",
" - **less-than-1B-euro**: Less than 1 billion EUR\n",
"#### osint\n",
"*Open Source Intelligence - Classification (MISP taxonomies)*\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #0087e8; color: white\">osint:certainty=&quot;50&quot;</span></span>\n",
"\n",
" - **certainty**: Certainty of the elements mentioned in this Open Source Intelligence\n",
" - **50**: Chances about even (probability equals 0.50 - 50%)\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #0071c3; color: white\">osint:lifetime=&quot;perpetual&quot;</span></span>\n",
"\n",
" - **lifetime**: Lifetime of the information as Open Source Intelligence\n",
" - **perpetual**: Perpetual\n",
"#### tlp\n",
"*The Traffic Light Protocol - or short: TLP - was designed with the objective to create a favorable classification scheme for sharing sensitive information while keeping the control over its distribution at the same time.*\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #CC0033; color: white\">tlp:red</span></span>\n",
"\n",
" - **red**: (TLP:RED) Information exclusively and directly given to (a group of) individual recipients. Sharing outside is not legitimate.\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #ffffff; color: black\">tlp:white</span></span>\n",
"\n",
" - **white**: (TLP:WHITE) Information can be shared publicly in accordance with the law.\n",
"#### workflow\n",
"*Workflow support language is a common language to support intelligence analysts to perform their analysis on data and information.*\n",
"- <span class=\"tag-container\"><span class=\"tag\" style=\"background-color: #f70085; color: white\">workflow:state=&quot;draft&quot;</span></span>\n",
"\n",
" - **state**: State\n",
" - **draft**: Draft means the information tagged can be released as a preliminary version or outline\n",
"## Galaxy Clusters\n",
"#### <i class=\"fas fa-map\"></i> Misinformation Pattern\n",
"*AM!TT Tactic*\n",
"- *[Adapt existing narratives](https://localhost:8443/galaxy_clusters/view/2712)*\n",
"Adapting existing narratives to current operational goals is the tactical sweet-spot for an effective misinformation campaign. Leveraging existing narratives is not only more effective, it requires substantially less resourcing, as the promotion of new master narratives operates on a much larger sca...\n",
"#### <i class=\"fas fa-shield\"></i> Malpedia\n",
"*Malware galaxy based on Malpedia archive.*\n",
"- *[Kobalos](https://localhost:8443/galaxy_clusters/view/4530)*\n",
"\n",
"#### <i class=\"fas fa-map\"></i> Attack Pattern\n",
"*ATT&amp;CK Tactic*\n",
"- *[SSH - T1021.004](https://localhost:8443/galaxy_clusters/view/9691)*\n",
"Adversaries may use [Valid Accounts](https://attack.mitre.org/techniques/T1078) to log into remote machines using Secure Shell (SSH). The adversary may then perform actions as the logged-on user.\n",
"\n",
"SSH is a protocol that allows authorized users to open remote shells on other computers. Many Linux and...\n",
"- *[Software - T1592.002](https://localhost:8443/galaxy_clusters/view/9721)*\n",
"Adversaries may gather information about the victim&#039;s host software that can be used during targeting. Information about installed software may include a variety of details such as types and versions on specific hosts, as well as the presence of additional components that might be indicative of...\n",
"#### <i class=\"fas fa-link\"></i> Course of Action\n",
"*ATT&amp;CK Mitigation*\n",
"- *[Access Token Manipulation Mitigation - T1134](https://localhost:8443/galaxy_clusters/view/8213)*\n",
"Access tokens are an integral part of the security system within Windows and cannot be turned off. However, an attacker must already have administrator level access on the local system to make full use of this technique; be sure to restrict users and accounts to the least privileges they require to ...\n",
"#### <i class=\"fas fa-industry\"></i> Sector\n",
"*Activity sectors*\n",
"- *[Defense](https://localhost:8443/galaxy_clusters/view/2762)*\n",
"\n",
"- *[Infrastructure](https://localhost:8443/galaxy_clusters/view/2780)*\n",
"\n",
"#### <i class=\"fas fa-bullseye\"></i> Target Information\n",
"*Description of targets of threat actors.*\n",
"- *[Canada](https://localhost:8443/galaxy_clusters/view/1994)*\n",
"\n",
"- *[China](https://localhost:8443/galaxy_clusters/view/2000)*\n",
"\n",
"#### <i class=\"fas fa-user-secret\"></i> Threat Actor\n",
"*Threat actors are characteristics of malicious actors (or adversaries) representing a cyber attack threat including presumed intent and historically observed behaviour.*\n",
"- *[APT 29](https://localhost:8443/galaxy_clusters/view/7251)*\n",
"A 2015 report by F-Secure describe APT29 as: &#039;The Dukes are a well-resourced, highly dedicated and organized cyberespionage group that we believe has been working for the Russian Federation since at least 2008 to collect intelligence in support of foreign and security policy decision-making. Th...\n"
]
}
],
"source": [
"# Get the context of Events that were created by organisations from the financial sector\n",
"\n",
"body = {\n",
" 'returnFormat': 'context-markdown',\n",
" 'org.sector': ['financial'],\n",
"}\n",
"\n",
"r2 = misp.direct_call('/events/restSearch', body)\n",
"print(r2)"
]
},
{
"cell_type": "code",
"execution_count": 511,
"metadata": {},
"outputs": [],
"source": [
"# Get the context of Events that had the threat actor APT-29 attached\n",
"\n",
"body = {\n",
" 'returnFormat': 'context',\n",
" 'tags': ['misp-galaxy:threat-actor=\\\"APT 29\\\"'],\n",
" 'staticHtml': 1, # If you want a JS-free HTML\n",
"}\n",
"\n",
"r2 = misp.direct_call('/events/restSearch', body)\n",
"with open('/tmp/attackOutput.html', 'w') as f:\n",
" f.write(r2)\n",
" # subprocess.run(['google-chrome', '--incognito', '/tmp/attackOutput.html'])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Be carefull with the amount of data you ask, use `pagination` if needed\n",
"\n",
"- `limit`: Specify the amount of data to be returned\n",
"- `page`: Specify the start of the rolling window. Is **not** zero-indexed\n",
"\n",
"If the size of the returned data is larger than the memory enveloppe you might get a different behavior based on your MISP setting:\n",
"- Nothing returned. Allowed memeory by PHP process exausted\n",
"- Data returned but slow. MISP will concatenante the returned data in a temporary file on disk\n",
" - This behavior is only applicable for `/*/restSearch` endpoints"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"r1 = misp.search(controller='attributes', pythonify=True)\n",
"print('Amount of Attributes', len(r1))\n",
"\n",
"r2 = misp.search(\n",
" controller='attributes',\n",
" page=1,\n",
" limit=5,\n",
" pythonify=True)\n",
"print('Amount of paginated Attributes', len(r2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Searching for Sightings"
]
},
{
"cell_type": "code",
"execution_count": 513,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[{'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1441',\n",
" 'date_sighting': '1670924035',\n",
" 'event_id': '40',\n",
" 'id': '12',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '65bd7539-29eb-46eb-bf7b-4c02473062c7',\n",
" 'value': '398324'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1441',\n",
" 'date_sighting': '1670924430',\n",
" 'event_id': '40',\n",
" 'id': '13',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '10857410-0033-4457-8a1d-c8331ee55d72',\n",
" 'value': '398324'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1441',\n",
" 'date_sighting': '1670924454',\n",
" 'event_id': '40',\n",
" 'id': '14',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '1',\n",
" 'uuid': '1639fe60-0458-40f3-961b-7dc14eee9a7b',\n",
" 'value': '398324'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1441',\n",
" 'date_sighting': '1670924455',\n",
" 'event_id': '40',\n",
" 'id': '15',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '1',\n",
" 'uuid': 'ee54ec70-3597-4455-bce9-c889202d533e',\n",
" 'value': '398324'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1441',\n",
" 'date_sighting': '1670924456',\n",
" 'event_id': '40',\n",
" 'id': '16',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '1',\n",
" 'uuid': '2c1cf4d1-a6ce-474b-8878-0251ee2b6bc5',\n",
" 'value': '398324'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1448',\n",
" 'date_sighting': '1671027299',\n",
" 'event_id': '41',\n",
" 'id': '17',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '39dff1d2-7082-48a9-8d30-ce29d412879b',\n",
" 'value': 'testtest'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1448',\n",
" 'date_sighting': '1671027301',\n",
" 'event_id': '41',\n",
" 'id': '18',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '84a8e7d0-715b-453f-8cdb-07db0c208185',\n",
" 'value': 'testtest'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '77',\n",
" 'date_sighting': '1671027307',\n",
" 'event_id': '9',\n",
" 'id': '19',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '264e4a25-e072-46e5-8460-b8df72e3115c',\n",
" 'value': '5.4.2.1'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '77',\n",
" 'date_sighting': '1671027308',\n",
" 'event_id': '9',\n",
" 'id': '20',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': 'b9f15aeb-54ea-44e5-90b8-22a418b973df',\n",
" 'value': '5.4.2.1'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '243',\n",
" 'date_sighting': '1671027309',\n",
" 'event_id': '9',\n",
" 'id': '21',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '4ef355f8-1cd3-476c-bccf-90a23b4eebfe',\n",
" 'value': 'test'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1342',\n",
" 'date_sighting': '1671029412',\n",
" 'event_id': '29',\n",
" 'id': '22',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': 'f0e76bec-2e04-4e88-a976-df831257c856',\n",
" 'value': 'malware.exe|70f3bc193dfa56b78f3e6e4f800f701f'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1342',\n",
" 'date_sighting': '1671029413',\n",
" 'event_id': '29',\n",
" 'id': '23',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': '803bb696-ae86-4a04-9793-5f54a45c99b7',\n",
" 'value': 'malware.exe|70f3bc193dfa56b78f3e6e4f800f701f'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1342',\n",
" 'date_sighting': '1671029414',\n",
" 'event_id': '29',\n",
" 'id': '24',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': 'fd8c4c0f-ebbb-4294-ade1-57493f1edc9a',\n",
" 'value': 'malware.exe|70f3bc193dfa56b78f3e6e4f800f701f'}},\n",
" {'Sighting': {'Organisation': {'id': '1',\n",
" 'name': 'ORGNAME',\n",
" 'uuid': 'c5de83b4-36ba-49d6-9530-2a315caeece6'},\n",
" 'attribute_id': '1441',\n",
" 'date_sighting': '1671030274',\n",
" 'event_id': '40',\n",
" 'id': '25',\n",
" 'org_id': '1',\n",
" 'source': '',\n",
" 'type': '0',\n",
" 'uuid': 'c84dd497-ad48-4b82-8203-6135a9a924fc',\n",
" 'value': '398324'}}]\n"
]
}
],
"source": [
"body = {\n",
" 'last': '7d'\n",
"}\n",
"\n",
"sightings = misp.direct_call('/sightings/restSearch', body)\n",
"pprint(sightings)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plotting data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Sightings over time"
]
},
{
"cell_type": "code",
"execution_count": 512,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 514,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>attribute_id</th>\n",
" <th>event_id</th>\n",
" <th>org_id</th>\n",
" <th>date_sighting</th>\n",
" <th>uuid</th>\n",
" <th>source</th>\n",
" <th>type</th>\n",
" <th>value</th>\n",
" <th>Organisation</th>\n",
" <th>one</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>12</td>\n",
" <td>1441</td>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" <td>2022-12-13 09:33:55</td>\n",
" <td>65bd7539-29eb-46eb-bf7b-4c02473062c7</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>398324</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>13</td>\n",
" <td>1441</td>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" <td>2022-12-13 09:40:30</td>\n",
" <td>10857410-0033-4457-8a1d-c8331ee55d72</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>398324</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>14</td>\n",
" <td>1441</td>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" <td>2022-12-13 09:40:54</td>\n",
" <td>1639fe60-0458-40f3-961b-7dc14eee9a7b</td>\n",
" <td></td>\n",
" <td>1</td>\n",
" <td>398324</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>15</td>\n",
" <td>1441</td>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" <td>2022-12-13 09:40:55</td>\n",
" <td>ee54ec70-3597-4455-bce9-c889202d533e</td>\n",
" <td></td>\n",
" <td>1</td>\n",
" <td>398324</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>16</td>\n",
" <td>1441</td>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" <td>2022-12-13 09:40:56</td>\n",
" <td>2c1cf4d1-a6ce-474b-8878-0251ee2b6bc5</td>\n",
" <td></td>\n",
" <td>1</td>\n",
" <td>398324</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>17</td>\n",
" <td>1448</td>\n",
" <td>41</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:14:59</td>\n",
" <td>39dff1d2-7082-48a9-8d30-ce29d412879b</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>testtest</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>18</td>\n",
" <td>1448</td>\n",
" <td>41</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:15:01</td>\n",
" <td>84a8e7d0-715b-453f-8cdb-07db0c208185</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>testtest</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>19</td>\n",
" <td>77</td>\n",
" <td>9</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:15:07</td>\n",
" <td>264e4a25-e072-46e5-8460-b8df72e3115c</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>5.4.2.1</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>20</td>\n",
" <td>77</td>\n",
" <td>9</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:15:08</td>\n",
" <td>b9f15aeb-54ea-44e5-90b8-22a418b973df</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>5.4.2.1</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>21</td>\n",
" <td>243</td>\n",
" <td>9</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:15:09</td>\n",
" <td>4ef355f8-1cd3-476c-bccf-90a23b4eebfe</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>test</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>22</td>\n",
" <td>1342</td>\n",
" <td>29</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:50:12</td>\n",
" <td>f0e76bec-2e04-4e88-a976-df831257c856</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>malware.exe|70f3bc193dfa56b78f3e6e4f800f701f</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>23</td>\n",
" <td>1342</td>\n",
" <td>29</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:50:13</td>\n",
" <td>803bb696-ae86-4a04-9793-5f54a45c99b7</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>malware.exe|70f3bc193dfa56b78f3e6e4f800f701f</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>24</td>\n",
" <td>1342</td>\n",
" <td>29</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 14:50:14</td>\n",
" <td>fd8c4c0f-ebbb-4294-ade1-57493f1edc9a</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>malware.exe|70f3bc193dfa56b78f3e6e4f800f701f</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>25</td>\n",
" <td>1441</td>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" <td>2022-12-14 15:04:34</td>\n",
" <td>c84dd497-ad48-4b82-8203-6135a9a924fc</td>\n",
" <td></td>\n",
" <td>0</td>\n",
" <td>398324</td>\n",
" <td>{'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id attribute_id event_id org_id date_sighting \\\n",
"0 12 1441 40 1 2022-12-13 09:33:55 \n",
"1 13 1441 40 1 2022-12-13 09:40:30 \n",
"2 14 1441 40 1 2022-12-13 09:40:54 \n",
"3 15 1441 40 1 2022-12-13 09:40:55 \n",
"4 16 1441 40 1 2022-12-13 09:40:56 \n",
"5 17 1448 41 1 2022-12-14 14:14:59 \n",
"6 18 1448 41 1 2022-12-14 14:15:01 \n",
"7 19 77 9 1 2022-12-14 14:15:07 \n",
"8 20 77 9 1 2022-12-14 14:15:08 \n",
"9 21 243 9 1 2022-12-14 14:15:09 \n",
"10 22 1342 29 1 2022-12-14 14:50:12 \n",
"11 23 1342 29 1 2022-12-14 14:50:13 \n",
"12 24 1342 29 1 2022-12-14 14:50:14 \n",
"13 25 1441 40 1 2022-12-14 15:04:34 \n",
"\n",
" uuid source type \\\n",
"0 65bd7539-29eb-46eb-bf7b-4c02473062c7 0 \n",
"1 10857410-0033-4457-8a1d-c8331ee55d72 0 \n",
"2 1639fe60-0458-40f3-961b-7dc14eee9a7b 1 \n",
"3 ee54ec70-3597-4455-bce9-c889202d533e 1 \n",
"4 2c1cf4d1-a6ce-474b-8878-0251ee2b6bc5 1 \n",
"5 39dff1d2-7082-48a9-8d30-ce29d412879b 0 \n",
"6 84a8e7d0-715b-453f-8cdb-07db0c208185 0 \n",
"7 264e4a25-e072-46e5-8460-b8df72e3115c 0 \n",
"8 b9f15aeb-54ea-44e5-90b8-22a418b973df 0 \n",
"9 4ef355f8-1cd3-476c-bccf-90a23b4eebfe 0 \n",
"10 f0e76bec-2e04-4e88-a976-df831257c856 0 \n",
"11 803bb696-ae86-4a04-9793-5f54a45c99b7 0 \n",
"12 fd8c4c0f-ebbb-4294-ade1-57493f1edc9a 0 \n",
"13 c84dd497-ad48-4b82-8203-6135a9a924fc 0 \n",
"\n",
" value \\\n",
"0 398324 \n",
"1 398324 \n",
"2 398324 \n",
"3 398324 \n",
"4 398324 \n",
"5 testtest \n",
"6 testtest \n",
"7 5.4.2.1 \n",
"8 5.4.2.1 \n",
"9 test \n",
"10 malware.exe|70f3bc193dfa56b78f3e6e4f800f701f \n",
"11 malware.exe|70f3bc193dfa56b78f3e6e4f800f701f \n",
"12 malware.exe|70f3bc193dfa56b78f3e6e4f800f701f \n",
"13 398324 \n",
"\n",
" Organisation one \n",
"0 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"1 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"2 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"3 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"4 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"5 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"6 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"7 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"8 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"9 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"10 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"11 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"12 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 \n",
"13 {'id': '1', 'uuid': 'c5de83b4-36ba-49d6-9530-2... 1 "
]
},
"execution_count": 514,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Converting our data to Panda DataFrame\n",
"sighting_rearranged = [sighting['Sighting'] for sighting in sightings]\n",
"df = pd.DataFrame.from_dict(sighting_rearranged)\n",
"df[\"date_sighting\"] = pd.to_datetime(df[\"date_sighting\"], unit='s')\n",
"df['one'] = 1\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print('Min and Max:', df['date_sighting'].min(), df['date_sighting'].max())\n",
"print('Time delta:', df['date_sighting'].max() - df['date_sighting'].min())\n",
"print('Unique Event IDs:', df.event_id.unique())"
]
},
{
"cell_type": "code",
"execution_count": 515,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1441 6\n",
"1342 3\n",
"1448 2\n",
"77 2\n",
"243 1\n",
"Name: attribute_id, dtype: int64\n"
]
},
{
"data": {
"text/plain": [
"<AxesSubplot: >"
]
},
"execution_count": 515,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGyCAYAAAC4Io22AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAhQUlEQVR4nO3de1TUdf7H8dcAMmICKt6DVdrKBG/lLTLdLNNF87RtlrmWZrtdNS+UJZqarYa15SHbMu3mntZba2tlpXbZzDpo3kItjxomiGLaYjKKOQLz+f3hz1lJUQc/wzDD83HOnNMM32He9mGYJ9/5zozDGGMEAABgQVigBwAAAKGDsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1EVV9gx6PRwUFBYqOjpbD4ajqmwcAAJVgjNHhw4fVvHlzhYVVvF+iysOioKBACQkJVX2zAADAgvz8fMXHx1f49SoPi+joaEknBouJianqmwcAAJXgcrmUkJDgfRyvSJWHxcmnP2JiYggLAACCzLkOY+DgTQAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArPE5LPbu3as777xTcXFxioqKUtu2bbV+/Xp/zAYAAIKMT58V8vPPP6tbt27q2bOnli1bpkaNGun7779X/fr1/TUfAAAIIj6FxTPPPKOEhAS9+eab3ssSExOtDwUAAIKTT0+FvP/+++rUqZNuu+02NW7cWFdeeaVeffXVs17H7XbL5XKVOwEAgNDk0x6LH374QbNmzVJaWprGjx+vdevWaeTIkYqMjNTQoUPPeJ2MjAxNmTLFyrDnq+W4D6v09vwld3q/QI8AAIBPHMYYc74bR0ZGqlOnTsrKyvJeNnLkSK1bt06rV68+43Xcbrfcbrf3vMvlUkJCgoqKihQTE3MBo1eMsAAAwC6Xy6XY2NhzPn779FRIs2bNlJSUVO6y1q1ba/fu3RVex+l0KiYmptwJAACEJp/Colu3btq+fXu5y3bs2KEWLVpYHQoAAAQnn8JizJgxWrNmjZ5++mnl5ORo/vz5mjNnjoYPH+6v+QAAQBDxKSw6d+6sJUuWaMGCBWrTpo3++te/KjMzU4MHD/bXfAAAIIj49KoQSbrpppt00003+WMWAAAQ5PisEAAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGt8Cosnn3xSDoej3OmKK67w12wAACDIRPh6heTkZH366af/+wYRPn8LAAAQonyugoiICDVt2tQfswAAgCDn8zEW33//vZo3b65LLrlEgwcP1u7du8+6vdvtlsvlKncCAAChyaew6Nq1q+bOnavly5dr1qxZ2rVrl7p3767Dhw9XeJ2MjAzFxsZ6TwkJCRc8NAAAqJ4cxhhT2SsfOnRILVq00IwZM/TnP//5jNu43W653W7veZfLpYSEBBUVFSkmJqayN31WLcd96JfvW9Vyp/cL9AgAAEg68fgdGxt7zsfvCzrysl69err88suVk5NT4TZOp1NOp/NCbgYAAASJC3ofiyNHjmjnzp1q1qyZrXkAAEAQ8yksHn30UX3xxRfKzc1VVlaWbrnlFoWHh2vQoEH+mg8AAAQRn54K2bNnjwYNGqTCwkI1atRI1157rdasWaNGjRr5az4AABBEfAqLhQsX+msOAAAQAvisEAAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhzQWExffp0ORwOjR492tI4AAAgmFU6LNatW6fZs2erXbt2NucBAABBrFJhceTIEQ0ePFivvvqq6tevb3smAAAQpCoVFsOHD1e/fv3Uq1cv2/MAAIAgFuHrFRYuXKiNGzdq3bp157W92+2W2+32nne5XL7eJAAACBI+7bHIz8/XqFGjNG/ePNWuXfu8rpORkaHY2FjvKSEhoVKDAgCA6s9hjDHnu/G7776rW265ReHh4d7LysrK5HA4FBYWJrfbXe5r0pn3WCQkJKioqEgxMTEW/gmnaznuQ79836qWO71foEcAAEDSicfv2NjYcz5++/RUyA033KAtW7aUu2zYsGG64oor9Pjjj58WFZLkdDrldDp9uRkAABCkfAqL6OhotWnTptxlF110keLi4k67HAAA1Dy88yYAALDG51eF/NrKlSstjAEAAEIBeywAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACs8SksZs2apXbt2ikmJkYxMTFKSUnRsmXL/DUbAAAIMj6FRXx8vKZPn64NGzZo/fr1uv7663XzzTfru+++89d8AAAgiET4snH//v3LnZ82bZpmzZqlNWvWKDk52epgAAAg+PgUFqcqKyvTv/71LxUXFyslJaXC7dxut9xut/e8y+Wq7E0CAIBqzuew2LJli1JSUnTs2DHVrVtXS5YsUVJSUoXbZ2RkaMqUKRc0JIJby3EfBnqEC5Y7vV+gRwCAoODzq0JatWql7Oxsff3113rwwQc1dOhQbd26tcLt09PTVVRU5D3l5+df0MAAAKD68nmPRWRkpC699FJJUseOHbVu3Tq98MILmj179hm3dzqdcjqdFzYlAAAIChf8PhYej6fcMRQAAKDm8mmPRXp6ulJTU/Wb3/xGhw8f1vz587Vy5UqtWLHCX/MBAIAg4lNYHDhwQEOGDNG+ffsUGxurdu3aacWKFbrxxhv9NR8AAAgiPoXF66+/7q85AABACOCzQgAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKzxKSwyMjLUuXNnRUdHq3HjxvrDH/6g7du
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Grouping by Attribute value\n",
"value_count = df['attribute_id'].value_counts()\n",
"print(value_count)\n",
"value_count.plot(kind='bar', rot=45)"
]
},
{
"cell_type": "code",
"execution_count": 516,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2 9\n",
"1 5\n",
"Name: date_sighting, dtype: int64\n"
]
},
{
"data": {
"text/plain": [
"<AxesSubplot: >"
]
},
"execution_count": 516,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGdCAYAAABO2DpVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAQG0lEQVR4nO3dX2jVdR/A8c9UOlnMPWrNFFdKBOafytKihCiKIizyJgoMzCCiZmZC5C4shtQSQgYl9geqXaTWjRU9ZIgwRUoqrciLNKmnRmEaxWYLTuL2XDw02KOrfvNzth19veB3cb7+fvw+QafefH/neGp6e3t7AwAgwajhHgAAOH0ICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgzZihvmFPT0/8+OOPUVtbGzU1NUN9ewBgEHp7e+Po0aMxZcqUGDVq4H2JIQ+LH3/8MRoaGob6tgBAgo6Ojpg6deqAfz7kYVFbWxsR/xts3LhxQ317AGAQurq6oqGhoe//4wMZ8rD48/HHuHHjhAUAVJm/+xiDD28CAGmEBQCQRlgAAGmEBQCQRlgAAGmEBQCQRlgAAGmEBQCQRlgAAGmEBQCQRlgAAGmEBQCQRlgAAGmEBQCQZsh/Nv1MNm3Vv4d7BIbQf55dONwjAAw5OxYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkKRQWx48fj9WrV8f06dNj7NixcfHFF8eaNWuit7e3UvMBAFVkTJGT165dGxs2bIi2traYNWtWfPrpp7F06dKoq6uL5cuXV2pGAKBKFAqLDz/8MO68885YuHBhRERMmzYtNm3aFB9//HFFhgMAqkuhRyHXXXddbN++PQ4cOBAREV988UXs2rUrbrvttgGvKZfL0dXV1e8AAE5PhXYsVq1aFV1dXTFjxowYPXp0HD9+PJ5++ulYvHjxgNe0tLREc3PzKQ8KAIx8hXYs3nrrrXjjjTdi48aNsXfv3mhra4vnnnsu2traBrymqakpOjs7+46Ojo5THhoAGJkK7Vg8/vjjsWrVqrjnnnsiImLOnDnx3XffRUtLSyxZsuSk15RKpSiVSqc+KQAw4hXasfj9999j1Kj+l4wePTp6enpShwIAqlOhHYs77rgjnn766bjwwgtj1qxZ8dlnn8W6devi/vvvr9R8AEAVKRQWzz//fKxevToefvjhOHz4cEyZMiUefPDBePLJJys1HwBQRQqFRW1tbbS2tkZra2uFxgEAqpnfCgEA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0ggLACCNsAAA0hQOix9++CHuvffemDhxYowdOzbmzJkTn376aSVmAwCqzJgiJ//666+xYMGCuPHGG+P999+P888/P77++usYP358peYDAKpIobBYu3ZtNDQ0xGuvvda3Nn369PShAIDqVOhRyLvvvhvz5s2Lu+66K+rr62Pu3LnxyiuvVGo2AKDKFAqLb775JjZs2BCXXHJJfPDBB/HQQw/F8uXLo62tbcBryuVydHV19TsAgNNToUchPT09MW/evHjmmWciImLu3Lmxb9++ePHFF2PJkiUnvaalpSWam5tPfVIAYMQrtGMxefLkmDlzZr+1Sy+9NL7//vsBr2lqaorOzs6+o6OjY3CTAgAjXqEdiwULFsT+/fv7rR04cCAuuuiiAa8plUpRKpUGNx0AUFUK7Vg89thjsXv37njmmWfi4MGDsXHjxnj55ZejsbGxUvMBAFWkUFjMnz8/tmzZEps2bYrZs2fHmjVrorW1NRYvXlyp+QCAKlLoUUhExO233x633357JWYBAKqc3woBANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANIICwAgjbAAANKMGe4BAE4H01b9e7hHYAj959mFwz3CiGXHAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDTCAgBIIywAgDSnFBbPPvts1NTUxIoVK5LGAQCq2aDD4pNPPomXXnopLrvsssx5AIAqNqiw+O2332Lx4sXxyiuvxPjx47NnAgCq1KDCorGxMRYuXBg333zz355bLpejq6ur3wEAnJ7GFL1g8+bNsXfv3vjkk0/+0fktLS3R3NxceDAAoPoU2rHo6OiIRx99NN544404++yz/9E1TU1N0dnZ2Xd0dHQMalAAYOQrtGOxZ8+eOHz4cFx55ZV9a8ePH4+dO3fGCy+8EOVyOUaPHt3vmlKpFKVSKWdaAGBEKxQWN910U3z55Zf91pYuXRozZsyIJ5544oSoAADOLIXCora2NmbPnt1v7dxzz42JEyeesA4AnHn8zZsAQJrC3wr5f+3t7QljAACnAzsWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAEAaYQEApBEWAECaQmHR0tIS8+fPj9ra2qivr49FixbF/v37KzUbAFBlCoXFjh07orGxMXbv3h3btm2LY8eOxS233BLd3d2Vmg8AqCJjipy8devWfq9ff/31qK+vjz179sT111+fOhgAUH0KhcX/6+zsjIiICRMmDHhOuVyOcrnc97qrq+tUbgkAjGCD/vBmT09PrFixIhYsWBCzZ88e8LyWlpaoq6vrOxoaGgZ7SwBghBt0WDQ2Nsa+ffti8+bNf3leU1NTdHZ29h0dHR2DvSUAMMIN6lHIsmXL4r333oudO3fG1KlT//LcUqkUpVJpUMMBANWlUFj09vbGI488Elu2bIn29vaYPn16peYCAKpQobBobGyMjRs3xjvvvBO1tbVx6NChiIioq6uLsWPHVmRAAKB6FPqMxYYNG6KzszNuuOGGmDx5ct/x5ptvVmo+AKCKFH4UAgAwEL8VAgCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERYAQBphAQCkERY
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Grouping by weekday (0-indexed)\n",
"amount_per_weekday = df['date_sighting'].dt.weekday.value_counts()\n",
"print(amount_per_weekday)\n",
"amount_per_weekday.plot(kind='bar', rot=0)"
]
},
{
"cell_type": "code",
"execution_count": 517,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"date_sighting\n",
"9 5\n",
"14 8\n",
"15 1\n",
"Name: one, dtype: int64\n"
]
},
{
"data": {
"text/plain": [
"<AxesSubplot: xlabel='date_sighting'>"
]
},
"execution_count": 517,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGxCAYAAAA+tv8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAeUklEQVR4nO3dfZBV9XnA8ee6KxeU3VUQBHQRRAsCASMYRnyJVhS2hNFOmraWmAXUUYsaJBjZcTQySBaaxMFm7NJkUkB8QROrJhpDhQnYaIiAwUhSUSzKqiS0FHcBk9Xu3v6RceOW17v8lt0Ln8/MmfGce849z05O9Ms5d7mZXC6XCwCABI5p7wEAgCOHsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSKD/cJm5qa4r333ouSkpLIZDKH+/QAQCvkcrnYuXNn9OnTJ445Zt/3JQ57WLz33ntRXl5+uE8LACRQW1sbp5566j5fP+xhUVJSEhF/HKy0tPRwnx4AaIX6+vooLy9v/u/4vhz2sPj48UdpaamwAIACc6CPMfjwJgCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSySssGhsb484774z+/ftHly5dYsCAATF79uzI5XJtNR8AUEDy+q6QefPmRU1NTSxevDiGDBkSa9eujcmTJ0dZWVnccsstbTUjAFAg8gqLF198Ma644ooYP358RET069cvHnnkkXjppZfaZDgAoLDk9Shk9OjRsWLFinj99dcjIuKVV16Jn/3sZ1FRUdEmwwEAhSWvOxYzZ86M+vr6GDRoUBQVFUVjY2PMmTMnJk6cuM9jGhoaoqGhoXm9vr6+9dMCAB1aXmHx2GOPxUMPPRQPP/xwDBkyJNavXx/Tpk2LPn36RGVl5V6Pqa6ujlmzZiUZFo5m/WY+094jHDHemju+vUeAI1Yml8evdJSXl8fMmTNj6tSpzdvuueeeePDBB+O1117b6zF7u2NRXl4edXV1UVpaegijw9FFWKQjLCB/9fX1UVZWdsD/fud1x+KDDz6IY45p+bGMoqKiaGpq2ucx2Ww2stlsPqcBAApUXmExYcKEmDNnTvTt2zeGDBkSv/zlL+Pee++NKVOmtNV8AEABySssvv3tb8edd94Zf//3fx/btm2LPn36xPXXXx933XVXW80HABSQvMKipKQk5s+fH/Pnz2+jcQCAQua7QgCAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGTyCot+/fpFJpPZY5k6dWpbzQcAFJDifHZes2ZNNDY2Nq9v2LAhLrvssvjCF76QfDAAoPDkFRY9evRosT537twYMGBAfPazn006FABQmFr9GYsPP/wwHnzwwZgyZUpkMpmUMwEABSqvOxaf9OSTT8b7778fkyZN2u9+DQ0N0dDQ0LxeX1/f2lMCAB1cq+9YfO9734uKioro06fPfverrq6OsrKy5qW8vLy1pwQAOrhWhcXbb78dy5cvj2uvvfaA+1ZVVUVdXV3zUltb25pTAgAFoFWPQhYuXBg9e/aM8ePHH3DfbDYb2Wy2NacBAApM3ncsmpqaYuHChVFZWRnFxa3+iAYAcATKOyyWL18eW7ZsiSlTprTFPABAAcv7lsPll18euVyuLWYBAAqc7woBAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSyTss3n333fjiF78Y3bt3jy5dusSnPvWpWLt2bVvMBgAUmOJ8dt6xY0ecf/75cckll8Szzz4bPXr0iDfeeCNOPPHEtpoPACggeYXFvHnzory8PBYuXNi8rX///smHAgAKU16PQn74wx/GyJEj4wtf+EL07NkzPv3pT8d3v/vdtpoNACgweYXFf/7nf0ZNTU2ceeaZsWzZsrjxxhvjlltuicWLF+/zmIaGhqivr2+xAABHprwehTQ1NcXIkSPj61//ekREfPrTn44NGzbEggULorKycq/HVFdXx6xZsw59UgCgw8vrjkXv3r1j8ODBLbadddZZsWXLln0eU1VVFXV1dc1LbW1t6yYFADq8vO5YnH/++bFx48YW215//fU47bTT9nlMNpuNbDbbuukAgIKS1x2LW2+9NVavXh1f//rXY9OmTfHwww/Hd77znZg6dWpbzQcAFJC8wuLcc8+NJ554Ih555JEYOnRozJ49O+bPnx8TJ05sq/kAgAKS16OQiIjPfe5z8bnPfa4tZgEACpzvCgEAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJLJKyzuvvvuyGQyLZZBgwa11WwAQIEpzveAIUOGxPLly//0BsV5vwUAcITKuwqKi4ujV69ebTELAFDg8v6MxRtvvBF9+vSJ008/PSZOnBhbtmxpi7kAgAKU1x2LUaNGxaJFi2LgwIGxdevWmDVrVlx44YWxYcOGKCkp2esxDQ0N0dDQ0LxeX19/aBMDAB1WXmFRUVHR/M/Dhg2LUaNGxWmnnRaPPfZYXHPNNXs9prq6OmbNmnVoU7aDfjOfae8RjhhvzR3f3iMAcJgc0q+bnnDCCfFnf/ZnsWnTpn3uU1VVFXV1dc1LbW3toZwSAOjADiksdu3aFW+++Wb07t17n/tks9koLS1tsQAAR6a8wmLGjBmxatWqeOutt+LFF1+Mv/zLv4yioqK46qqr2mo+AKCA5PUZi3feeSeuuuqq2L59e/To0SMuuOCCWL16dfTo0aOt5gMACkheYbF06dK2mgMAOAL4rhAAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACCZQwqLuXPnRiaTiWnTpiUaBwAoZK0OizVr1sQ///M/x7Bhw1LOAwAUsFaFxa5du2LixInx3e9+N0488cTUMwEABapVYTF16tQYP358jBkzJvU8AEABK873gKVLl8bLL78ca9asOaj9GxoaoqGhoXm9vr4+31MCAAUirzsWtbW18eUvfzkeeuih6Ny580EdU11dHWVlZc1LeXl5qwYFADq+vMJi3bp1sW3btjjnnHOiuLg4iouLY9WqVfGP//iPUVxcHI2NjXscU1VVFXV1dc1LbW1tsuEBgI4lr0chl156abz66qsttk2ePDkGDRoUt99+exQVFe1xTDabjWw2e2hTAgAFIa+wKCkpiaFDh7bYdvzxx0f37t332A4AHH38zZsAQDJ5/1bI/7dy5coEYwAARwJ3LACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGSEBQCQjLAAAJIRFgBAMsICAEhGWAAAyQgLACAZYQEAJCMsAIBkhAUAkIywAACSERYAQDLCAgBIRlgAAMkICwAgGWEBACQjLACAZIQFAJCMsAAAkhEWAEAywgIASEZYAADJCAsAIBlhAQAkIywAgGTyCouampoYNmxYlJaWRml
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"amount_per_weekday_for_each_attribute = df.groupby([df['date_sighting'].dt.hour])['one'].sum()\n",
"print(amount_per_weekday_for_each_attribute)\n",
"amount_per_weekday_for_each_attribute.plot(kind='bar', rot=0)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.10 ('venv': venv)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "99e19f785595e5572f3a0434505ffd496bc893a60c3b4501be593ee9ddcf6bde"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}