MISP
/
misp-training
mirror of https://github.com/MISP/misp-training
2
0
Fork 0
Code Issues Releases Wiki Activity

new: [a.2] pymisp slides added

improvedChecklist
Alexandre Dulaunoy 2018-12-29 21:43:09 +01:00
parent 8e2041c98d
commit d8c22a9589
No known key found for this signature in database
GPG Key ID: 09E2CD4944E6CBCD
8 changed files with 483 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
a.2-pymisp/awareness.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 93 KiB

365
a.2-pymisp/content.tex Executable file
View File

@ -0,0 +1,365 @@
% DO NOT COMPILE THIS FILE DIRECTLY!
% This is included by the other .tex files.
\colorlet{punct}{red!60!black}
\definecolor{background}{HTML}{EEEEEE}
\definecolor{delim}{RGB}{20,105,176}
\colorlet{numb}{magenta!60!black}
\lstdefinelanguage{json}{
basicstyle=\ttfamily\footnotesize,
numbers=left,
numberstyle=\ttfamily\footnotesize,
stepnumber=1,
numbersep=8pt,
showstringspaces=false,
breaklines=true,
frame=lines,
backgroundcolor=\color{background},
literate=
*{0}{{{\color{numb}0}}}{1}
{1}{{{\color{numb}1}}}{1}
{2}{{{\color{numb}2}}}{1}
{3}{{{\color{numb}3}}}{1}
{4}{{{\color{numb}4}}}{1}
{5}{{{\color{numb}5}}}{1}
{6}{{{\color{numb}6}}}{1}
{7}{{{\color{numb}7}}}{1}
{8}{{{\color{numb}8}}}{1}
{9}{{{\color{numb}9}}}{1}
{:}{{{\color{punct}{:}}}}{1}
{,}{{{\color{punct}{,}}}}{1}
{\{}{{{\color{delim}{\{}}}}{1}
{\}}{{{\color{delim}{\}}}}}{1}
{[}{{{\color{delim}{[}}}}{1}
{]}{{{\color{delim}{]}}}}{1},
}
\begin{frame}[t,plain]
\titlepage
\end{frame}
\begin{frame}
\frametitle{Context}
\begin{itemize}
\item MISP is a large project
\item Your production environment is even more complex
\item 3rd party services are even worse
\item Querying MISP via CURL is doable, but get's painful fast
\item Talking to MySQL directly can be dangerous
\item POST a JSON blob, receive a JSON blob. You can do it manually(-ish)
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Big picture}
\begin{itemize}
\item Core goal: providing stable access to APIs, respect access control
\item Simplifying handling \& automation of indicators in 3rd party tools
\item Hiding complexity of the JSON blobs
\item Providing pre-cooked examples for commonly used operations
\item Helping integration with existing infrastructure
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Common queries: Recent changes on a timeframe}
There are 4 main cases here:
\begin{itemize}
\item Metadata of the events that have been modified
\begin{itemize}
\item {\bf search\_index} $\Rightarrow$ timestamp (1h, 1d, 7d, ...), returns list of all the modified events
\end{itemize}
\item Full events (metadata + attributes)
\begin{itemize}
\item {\bf search} $\Rightarrow$ timestamp (1h, 1d, 7d, ...)
\end{itemize}
\item Modified attributes
\begin{itemize}
\item {\bf search} $\Rightarrow$ controller = attributes and timestamp (1h, 1d, 7d, ...)
\end{itemize}
\item Other use case: get last {\bf published} events by using the last parameter in the {\bf search} method.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Common queries: Search things}
There are 3 main cases here:
\begin{itemize}
\item Easy, but slow: full text search with {\bf search\_all}
\item Faster: use the {\bf search} method and search by tag, type, enforce the warning lists, with(-out) attachments, dates interval, ...
\item Get malware samples (if available on the instance).
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Common queries: create things}
There are 3 main cases here:
\begin{itemize}
\item Add Event, edit its metadata
\item Add attributes or objects to event
\item (un)Tag event or attribute (soon object)
\item Edit Attributes medatada
\item Upload malware sample (and automatically expand it)
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Administrative tasks}
Assyming you have the right to do it on the instance.
\begin{itemize}
\item Managing users
\item Managing organisations
\item Managing sync servers
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{Other Capabilities}
\begin{itemize}
\item Upload/download samples
\item {\bf Proposals}: add, edit, accept, discard
\item {\bf Sightings}: Get, set, update
\item Export {\bf statistics}
\item Manage {\bf feeds}
\item Get MISP server version, recommended PyMISP version
\item And more, look at the api file
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{MISPEvent - Usecase}
\begin{adjustbox}{width=\textwidth,height=3cm,keepaspectratio}
\begin{lstlisting}[language=Python,firstnumber=1]
from pymisp import MISPEvent, EncodeUpdate
# Create a new event with default values
event = MISPEvent()
# Load an existing JSON dump (optional)
event.load_file('Path/to/event.json')
event.info = 'My cool event' # Duh.
# Add an attribute of type ip-dst
event.add_attribute('ip-dst', '8.8.8.8')
# Mark an attribute as deleted (From 2.4.60)
event.delete_attribute('<Attribute UUID>')
# Dump as json
event_as_jsondump = json.dumps(event, cls=EncodeUpdate)
\end{lstlisting}
\end{adjustbox}
\end{frame}
\begin{frame}
\frametitle{Basics}
\begin{itemize}
\item Python 3.5+ is recommended
\item PyMISP is always inline with current version (pip3 install pymisp)
\item Dev version: pip3 install git+https://github.com/MISP/PyMISP.git
\item Get your auth key from: \url{https://misppriv.circl.lu/events/automation}
\begin{itemize}
\item Not available: you don't have "Auth key access" role. Contact your instance admin.
\end{itemize}
\item Source available here: git clone \url{https://github.com/MISP/PyMISP.git}
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{Examples}
\begin{itemize}
\item {\bf PyMISP needs to be installed (duh)}
\item Usage:
\begin{itemize}
\item Create examples/keys.py with the following content
\begin{adjustbox}{width=\textwidth,height=.9cm,keepaspectratio}
\begin{lstlisting}[language=python,firstnumber=1]
misp_url = "https://url-to-your-misp"
misp_key = "<API_KEY>"
misp_verifycert = True
\end{lstlisting}
\end{adjustbox}
\end{itemize}
\item Proxy support:
\begin{adjustbox}{width=\textwidth,height=.9cm,keepaspectratio}
\begin{lstlisting}[language=python,firstnumber=1]
proxies = {
'http': 'http://127.0.0.1:8123',
'https': 'http://127.0.0.1:8123',
}
PyMISP(misp_url, misp_key, misp_verifycert, proxies=proxies)
\end{lstlisting}
\end{adjustbox}
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{Examples}
\begin{itemize}
\item Lots of ideas on how to use the API
\item You may also want to look at the tests directory
\item All the examples use argparse. Help usage is available: {\bf script.py -h}
\begin{itemize}
\item {\bf add\_file\_object.py}: Attach a file (PE/ELF/Mach-O) object to an event
\item {\bf upload.py}: Upload a malware sample (use advanced expansion is available on the server)
\item {\bf last.py}: Returns all the most recent events (on a timeframe)
\item {\bf add\_named\_attribute.py}: Add attribute to an event
\item {\bf sighting.py}: Update sightings on an attribute
\item {\bf stats.py}: Returns the stats of a MISP instance
\item {\bf \{add,edit,create\}\_user.py} : Add, Edit, Create a user on MISP
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{Usage}
\begin{itemize}
\item Basic example
\end{itemize}
\begin{adjustbox}{width=\textwidth,height=2cm,keepaspectratio}
\begin{lstlisting}[language=python,firstnumber=1]
from pymisp import PyMISP
api = PyMISP(url, apikey, verifycert=True, debug=False, proxies=None)
response = api.<function>
if response['error']:
# <something went wrong>
else:
# <do something with the output>
\end{lstlisting}
\end{adjustbox}
\end{frame}
\begin{frame}[fragile]
\frametitle{Concept behind AbstractMISP}
\begin{itemize}
\item JSON blobs are python dictionaries
\item ... Accessing content can be a pain
\item {\bf AbstractMISP inherits collections.MutableMapping}, they are all dictionaries!
\item ... Has helpers to load, dump, and edit JSON blobs
\item {\bf Important}: All the public attributes (not starting with a \_) defined in a class are dumped to JSON
\item {\bf Tags}: Events and Attributes have tags, soon Objects. Tag handling is defined in this class.
\item {\bf edited}: When pushing a full MISPEvent, only the objects without a timestamp,
or with a newer timestamp will be updated. This method recursively finds updated events,
and removes the timestamp key from the object.
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{MISPEvent, MISPAttribute, MISPObject, MISPSighting...}
\begin{itemize}
\item {\bf Pythonic} representation of MISP elements
\item {\bf Easy manipulation}
\begin{itemize}
\item Load an existing event
\item Update te metadata, add attributes, objects, tags, mark an attribute as deleted, ...
\item Set relations between objects
\item Load and add attachments or malware samples as pseudo files
\end{itemize}
\item {\bf Dump} to JSON
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{MISPEvent - Main entrypoints}
\begin{itemize}
\item load\_file(event\_path)
\item load(json\_event)
\item add\_attribute(type, value, **kwargs)
\item add\_object(obj=None, **kwargs)
\item add\_attribute\_tag(tag, attribute\_identifier)
\item get\_attribute\_tag(attribute\_identifier)
\item add\_tag(tag=None, **kwargs)
\item objects[], attributes[], tags[]
\item edited, all other paramaters of the MISPEvent element (info, date, ...)
\item to\_json()
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{MISPObject - Main entrypoints}
\begin{itemize}
\item add\_attribute(object\_relation, **value)
\item add\_reference(referenced\_uuid, relationship\_type, comment=None, **kwargs)
\item has\_attributes\_by\_relation(list\_of\_relations)
\item get\_attributes\_by\_relation(object\_relation)
\item attributes[], relations[]
\item edited, all other paramaters of the MISPObject element (name, comment, ...)
\item to\_json()
\item Can be validated against their template
\item Can have default parameters applied to all attributes (i.e. distribution, category, ...)
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{MISPAttribute - Main entrypoints}
\begin{itemize}
\item add\_tag(tag=None, **kwargs)
\item delete()
\item malware\_binary (if relevant)
\item tags[]
\item edited, all other paramaters of the MISPObject element (value, comment, ...)
\item to\_json()
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{PyMISP - Tools}
\begin{itemize}
\item Libraries requiring specfic 3rd party dependencies
\item Callable via PyMISP for specific usecases
\item Curently implemented:
\begin{itemize}
\item {\bf OpenIOC} to MISP Event
\item MISP to {\bf Neo4J}
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{PyMISP - Default objects generators }
\begin{itemize}
\item File - PE/ELF/MachO - Sections
\item VirusTotal
\item Generic object generator
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{PyMISP - Logging / Debugging}
\begin{itemize}
\item debug=True passed to the constructor enable debug to stdout
\item Configurable using the standard logging module
\item Show everything send to the server and received by the client
\end{itemize}
\begin{adjustbox}{width=\textwidth,height=3cm,keepaspectratio}
\begin{lstlisting}[language=Python,firstnumber=1]
import pymisp
import logging
logger = logging.getLogger('pymisp')
logger.setLevel(logging.DEBUG) # enable debug to stdout
logging.basicConfig(level=logging.DEBUG, # Enable debug to file
filename="debug.log",
filemode='w',
format=pymisp.FORMAT)
\end{lstlisting}
\end{adjustbox}
\end{frame}
\begin{frame}[t,fragile] {Q\&A}
\includegraphics[scale=0.5]{misplogo.pdf}
\begin{itemize}
\item \url{https://github.com/MISP/PyMISP}
\item \url{https://github.com/MISP/}
\item \url{https://pymisp.readthedocs.io/}
\item We welcome new functionalities and pull requests.
\end{itemize}
\end{frame}

BIN
a.2-pymisp/logo-circl.pdf Normal file
View File

Binary file not shown.

BIN
a.2-pymisp/misp.pdf Normal file
View File

Binary file not shown.

BIN
a.2-pymisp/misplogo.pdf Normal file
View File

Binary file not shown.

89
a.2-pymisp/notes.md Normal file
View File

@ -0,0 +1,89 @@
MISP is a huge beast hat oes lots of things you really don't want to mess with. But our approach is to make your life as easy as possible to interconnect it with 3rd party services, within your own organisation, and to the outside world.
We already talked about syncing, connecting it to TheHive, and sending HTTP requests to it to get information out of the platform to push it into your SIEM for example.
It is nice and all, but at the moment you want to use something else than extracting attributes in CSV, or updating the sightings, it get's tricky to do it with bash and curl. And please, please do not connect directly to the database: it bypasses all ACLs, and if you start editing the datasets, you will most probably break it: there is a lot of data-massaging done when the data enters the system through the web interface, bypassing it will end up in a disaster, and we won't be able to help you.
Before going into the PyMISP internals, let's discuss a bit the API itself.
The format used to push and recieve data to and from a MISP instance is JSON. In the past, there was an XML interface too, but this one has been deprecated years ago. If you're still using it, you should know isn't supported anymore, and hasn't been in the last 2 years, at least.
To query the interface, you send a JSON blob, and it will answer with an other JSON blob. In Python, it is easy, you can load a json blob and get a dictionary, and dump it back to json. It is relatively simple to do it manually for simple requests such as adding a tag, getting the most recently updated attributes, but it get's pretty trycky when you want to run complex search queries, or create new events.
This is the reason we developped PyMISP.
PyMISP has two main interfaces: the first one is directly interacting with a remote MISP instance. Keep in mine that it follows the same access control rules as the ones defined for the account on the web interface.
In order to use the API, you need to get the automation key associated to your account -> URL
If you don't have one, it means your user doesn't have the "Auth key access", you will need to get in touch with your org admin, or the administrator of the platform.
With the key, you can access the same data you can see on the web interface.
# RO
The most common requests we see are the following:
* Get the most recent updates on a timeframe
* Lookup attributes or campaigns
## Recent changes
There are 3 main cases here:
* Metadata of the events that have been modified
* search_index -> timestamp (1h, 1d, 7d, ...)
returns list of all the modified events
* Full events (metadata + attributes)
* search -> timestamp (1h, 1d, 7d, ...)
* Modified attributes
* search -> controller = attributes & timestamp (1h, 1d, 7d, ...)
if you want to get all the attributes of a modified event: controller = attributes & timestamp (1h, 1d, 7d, ...)
Other use case: get last **published** events by using the last parameter
## Search
* Easy, but slow: full text search with `search_all`
* search by tag, type, to_ids flag set enforce the warning lists, with attachments, date interval, by organisation
* get malware samples
# Write
add event /attributes/tag/objects, ...
Upload malware sample (with or without the expansion)
(overview, no creation yet)
# Admin tasks
Assuming you have the proper access, you can also do plenty of administrative tasks through the API, such as managing users, organisations, and sync servers.
# Offline creation of a MISPEvent (usage)
# Examples in the example directory
# Internals of the module itself (dev)
Organisation of the project, muttable types
--------------------------------------------
In order to make your life easier, we developped a python modules that will help you to programmatically interact with a MISP instance: PyMISP.
PyMISP has default settings for plenty of comon use cases which simplify the interactions with MISP as much as possible.
PyMISP is the recommended way to programmatically connect MISP to your internal systems, in Python.
It uses the API, which means it is

28
a.2-pymisp/slide.tex Normal file
View File

@ -0,0 +1,28 @@
\documentclass{beamer}
\usetheme[numbering=progressbar]{focus}
\definecolor{main}{RGB}{47, 161, 219}
\definecolor{textcolor}{RGB}{128, 128, 128}
\definecolor{background}{RGB}{240, 247, 255}
\usepackage[utf8]{inputenc}
\usepackage{tikz}
\usepackage{listings}
\usepackage{adjustbox}
\usetikzlibrary{positioning}
\usetikzlibrary{shapes,arrows}
%\usepackage[T1]{fontenc}
%\usepackage[scaled]{beramono}
\author{\small{\input{../includes/authors.txt}}}
\title{Deep-dive into PyMISP}
\subtitle{MISP - Threat Sharing}
\institute{\href{http://www.misp-project.org/}{http://www.misp-project.org/} \\ Twitter: \emph{\href{https://twitter.com/mispproject}{@MISPProject}}}
\titlegraphic{\includegraphics[scale=0.85]{misp.pdf}}
\date{\input{../includes/location.txt}}
\begin{document}
\include{content}
\end{document}

2
build.sh
View File

@ -1,7 +1,7 @@
#!/bin/bash
#
slidedecks=("0-misp-introduction-to-information-sharing" "1-misp-usage" "1.2-misp-integration" "1.1-misp-viper-integration" "1.2.1-misp-integration-mail2misp" "2-misp-administration" "3-misp-taxonomy-tagging" "3.1-misp-modules" "3.2-misp-galaxy" "3.3-misp-object-template" "6.0-misp-dashboard" "a.0-contributing" "a.1-devintro")
slidedecks=("0-misp-introduction-to-information-sharing" "1-misp-usage" "1.2-misp-integration" "1.1-misp-viper-integration" "1.2.1-misp-integration-mail2misp" "2-misp-administration" "3-misp-taxonomy-tagging" "3.1-misp-modules" "3.2-misp-galaxy" "3.3-misp-object-template" "6.0-misp-dashboard" "a.0-contributing" "a.1-devintro" "a.2-pymisp")
mkdir output
export TEXINPUTS=::`pwd`/themes/
echo ${TEXINPUTS}