misp-website/_posts/2020-09-30-How-To-Make-A-MISP-Import-Script.md

---
title: Create an import script for MISP , step-by-step tutorial
layout: post
authors:
- Pauline Bourmeau
date: 2020-09-30
tags: ["MISP", "Import Script", "MISP", "Threat Intelligence"]
categories: []
featured: /assets/images/misp/blog/misp-python.png
---

# Create an import script for MISP in Python, step-by-step tutorial

## Script description
### Example add_github_user.py

Here the goal is to push to MISP information gathered on Github. The script [add_github_user.py](https://github.com/MISP/PyMISP/blob/main/examples/add_github_user.py) will be used as an example.

An **import script**, it's different from a MISP module.

An import script is run from a terminal to push data into a MISP, but a MISP module runs into a MISP instance. For more information about MISP modules, see [here](https://misp.github.io/misp-modules/)

This script will add data from GitHub as object to an existing MISP event.

**Tips:** in early phase or for testing purpose, it's better to start with an import script to test the code and results. If it's conclusive you can code a MISP module.


### PyMISP install

PyMISP is a Python library to access MISP with REST API. Install the [PyMISP](https://github.com/MISP/PyMISP) library on your machine.

To do so, clone the [repository](https://github.com/MISP/PyMISP) and install the library:

~~~bash
git clone https://github.com/MISP/PyMISP
cd PyMISP
pip3 install .
~~~

### Fork

If you wish to contribute and push your code to the MISP community, you'll need to fork the PyMISP library.

### Meaningful naming

Where to put the files and what are their names?

The script will be in **[PyMISP/examples](https://github.com/MISP/PyMISP/tree/main/examples)** and has the name of the imported service. A short key word is a good practice to describe the script action:

~~~
PyMISP/examples/add_github_user.py
~~~

**tips:** before naming your script, consult the list of available scripts into the PyMISP/examples folder, in order to describe accurately what your import module does.

### Keys management

For all scripts , you need to have a single key file **keys.py**. The file is not to be committed and contains the credentials for connecting to the MISP instance: API key, URL, SSH validations.

- You create a file keys.py into the dir PyMISP/examples.

References to this file will be later set in the script:

~~~python
from keys import misp_url, misp_key, misp_verifycert
~~~


### Required libraries

- PyMISP
- request
- sys
- argparse

### Usage block

We will go through the Python code and explain step-by-step each part.

It will be used by pyDoc, starts with """ and ends with """. You will here describe the usage for the script.

~~~~python
"""
usage: add_github_user.py [-h] -e EVENT [-f] -u USERNAME
Fetch GitHub user details and add it in object in MISP
optional arguments:
-h, --help            show this help message and exit
-e EVENT, --event EVENT
		      Event ID to update
-f, --force-template-update
-u USERNAME, --username USERNAME
		      GitHub username to add
"""
~~~~

### Argument parser

The script needs to grab the event-id (-e), a force to update the template from the repository (-f), and here a user account (-u).

~~~~python
parser = argparse.ArgumentParser(description='Fetch GitHub user details and add it in object in MISP')
parser.add_argument("-e", "--event", required=True, help="Event ID to update")
parser.add_argument("-f", "--force-template-update", required=False, action="store_true")
parser.add_argument("-u", "--username", required=True, help="GitHub username to add")
args = parser.parse_args()
~~~~

### Request

NOTE: If the user doesn't exist, the script is interrupted.

~~~~python
r = requests.get("https://api.github.com/users/{}".format(args.username))
if r.status_code != 200:
		sys.exit("HTTP return is {} and not 200 as expected".format(r.status_code))
if args.force_template_update:
		print("Updating MISP Object templates...")
		update_objects()
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)
~~~~

The PyMISP connection is instantiated with following parameters coming from the file keys.py : *misp_url, misp_keys and misp_verifycert*.

### Object creation

We create a MISP object, the name is the object template name:

~~~~python
misp_object = MISPObject(name="github-user")
~~~~

**tips:** you can find the object template name is the [MISP object template repository](https://github.com/MISP/misp-objects).

**contributing:** if an object template is missing, you can propose your own.

### User data

User data from the GitHub API are transformed into a JSON in order to be parsed by the following commands:

~~~~python
github_user = r.json()
~~~~

Following HTTP GET requests to obtain the following and followers sections of the GitHub user.

~~~~python
rfollowers = requests.get(github_user['followers_url'])
followers = rfollowers.json()
rfollowing = requests.get("https://api.github.com/users/{}/following".format(args.username))
followings = rfollowing.json()
~~~~

Same things for the SSH public keys attached to the GitHub user:

~~~~python
rkeys = requests.get("https://api.github.com/users/{}/keys".format(args.username))
keys = rkeys.json()
~~~~

Then, filling of the object from the github_user json object:

~~~~python
misp_object.add_attributes("follower", *[follower['login'] for follower in followers])
misp_object.add_attributes("following", *[following['login'] for following in followings])
misp_object.add_attributes("ssh-public-key", *[sshkey['key'] for sshkey in keys])
misp_object.add_attribute('bio', github_user['bio'])
misp_object.add_attribute('link', github_user['html_url'])
misp_object.add_attribute('user-fullname', github_user['name'])
misp_object.add_attribute('username', github_user['login'])
misp_object.add_attribute('twitter_username', github_user['twitter_username'])
misp_object.add_attribute('location', github_user['location'])
misp_object.add_attribute('company', github_user['company'])
misp_object.add_attribute('public_gists', github_user['public_gists'])
misp_object.add_attribute('public_repos', github_user['public_repos'])
misp_object.add_attribute('blog', github_user['blog'])
misp_object.add_attribute('node_id', github_user['node_id'])
~~~~

Finally, you add the new object created, then pushed to the specified event id, on your MISP instance:

~~~~python
retcode = pymisp.add_object(args.event, misp_object)
~~~~

Now it's your turn, to do a MISP import script.