History

Alexandre Dulaunoy 46850c89e2 automation: feed fetching updated		2017-08-24 11:05:21 +02:00
..
README.md	automation: feed fetching updated	2017-08-24 11:05:21 +02:00

README.md

Automation API

Automation functionality is designed to automatically generate signatures for intrusion detection systems. To enable signature generation for a given attribute, Signature field of this attribute must be set to Yes. Note that not all attribute types are applicable for signature generation, currently we only support NIDS signature generation for IP, domains, host names, user agents etc., and hash list generation for MD5/SHA1 values of file artifacts. Support for more attribute types is planned. To to make this functionality available for automated tools an authentication key is used. This makes it easier for your tools to access the data without further form-based-authentication.

Automation URL

The documentation will include a default MISP url in the examples. Don't forget to replace it with your MISP url.

Default MISP url in the documentation:

https://<misp url>/

Automation key

The authentication of the automation is performed via a secure key available in the MISP UI interface. Make sure you keep that key secret as it gives access to the entire database! The API key is available in the event actions menu under automation.

Since version 2.2 the usage of the authentication key in the url is deprecated. Instead, pass the auth key in an Authorization header in the request. The legacy option of having the auth key in the url is temporarily still supported but not recommended.

The authorization is performed by using the following header:

Authorization: YOUR API KEY

Accept and Content-Type headers

When performing your request, depending on the type of request, you might need to explicitly specify in what content type you want to get your results. This is done by setting one of the below Accept headers:

Accept: application/json
Accept: application/xml

When submitting data in a POST, PUT or DELETE operation you also need to specify in what content-type you encoded the payload. This is done by setting one of the below Content-Type headers:

Content-Type: application/json
Content-Type: application/xml

XML Export

An automatic export of all events and attributes (except file attachments) is available under a custom XML format.

You can configure your tools to automatically download the following file:

https://<misp url>/events/xml/download

If you only want to fetch a specific event append the eventid number:

https://<misp url>/events/xml/download/1

You can post an XML or JSON object containing additional parameters in the JSON query format or XML query format. Query parameters provide a way to filter the output to specific parameters.

JSON query format

The URL is appended with json:

https://<misp url>/events/xml/download.json

The query parameters can be the following:

{"request": {"eventid":["!51","!62"],"withAttachment":false,"tags":["APT1","!OSINT"],"from":false,"to":"2015-02-15"}}

XML query format

The URL is path is:

https://<misp url>/events/xml/download

The query parameters can be the following:

<request><eventid>!51</eventid><eventid>!62</eventid><withAttachment>false</withAttachment><tags>APT1</tags><tags>!OSINT</tags><from>false</from><to>2015-02-15</to></request>

XML download and URL parameters

The XML download also accepts two additional the following optional parameters in the url:

https://<misp url>/events/xml/download/[eventid]/[withattachments]/[tags]/[from]/[to]/[last]

eventid: Restrict the download to a single event
withattachments: A boolean field that determines whether attachments should be encoded and a second parameter that controls the eligible tags.
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/events/xml/download/false/true/tag1&&tag2&&!tag3

from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

The keywords false or null should be used for optional empty parameters in the URL. Also check out the User Guide to read about the REST API.

CSV export

An automatic export of attributes is available as CSV. Only attributes that are flagged "to_ids" will get exported.

You can configure your tools to automatically download the following file:

https://<misp url>/events/csv/download

You can specify additional flags for CSV exports as follows:

POST to:

https://<misp url>/events/csv/download

Headers:

Authorization: <your auth key>
Content-type: application/json

Body:

{"parameter1":"value1", "parameter2":1, "parameter3":["value3", "value4", "!value5"]}

eventid: Restrict the download to a single event
ignore: Setting this flag to true will include attributes that are not marked "to_ids".
tags: Simply add a list of tags that should be included or negated (by prepending the tag name with a "!"). Any event with a negated tag will be ignored, even if an included tag is matching. An example is included further down.
category: The attribute category, any valid MISP attribute category is accepted.
type: The attribute type, any valid MISP attribute type is accepted.
includeContext: Include the event data with each attribute.
from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

For example, to only download a csv generated of the "domain" type and the "Network activity" category attributes all events except for the one and further restricting it to events that are tagged "tag1" or "tag2" but not "tag3", only allowing attributes that are IDS flagged use the following syntax:

POST to:

https://<misp url>/events/csv/download

Headers:

Authorization: <your auth key>
Content-type: application/json

Body:

{"tags":["tag1", "tag2", "!tag3"], "category":"Network activity", "type": "domain"}

Alternatively you can fall back to the deprecated syntax of passing parameters in a GET request via the URL, however this is discouraged:

https://<misp url>/events/csv/download/[eventid]/[ignore]/[tags]/[category]/[type]/[includeContext]/[from]/[to]/[last]

If you use the deprecated URL parameter method, keep in mind that the keywords false or null should be used for optional empty parameters. To export the attributes of all events that are of the type "domain", use the following syntax:

https://<misp url>/events/csv/download/false/false/false/false/domain

NIDS rules export

Automatic export of all network related attributes is available under the Snort or Suricata rule format. Only published events and attributes marked as IDS Signature are exported.

You can configure your tools to automatically download the following file:

https://<misp url>/events/nids/suricata/download
https://<misp url>/events/nids/snort/download

The full API syntax is as follows:

https://<misp url>/events/nids/[format]/download/[eventid]/[frame]/[tags]/[from]/[to]/[last]

format: The export format, can be "suricata" or "snort"
eventid: Restrict the download to a single event
frame: Some commented out explanation framing the data. The reason to disable this would be if you would like to concatenate a list of exports from various select events in order to avoid unnecessary duplication of the comments.
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/events/nids/snort/download/false/false/tag1&&tag2&&!tag3

from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 6d or 12h or 30m). This filter will use the published timestamp of the event.

The keywords false or null should be used for optional empty parameters in the URL.

An example for a Suricata export for all events excluding those tagged tag1, without all of the commented information at the start of the file would look like this:

https://<misp url>/events/nids/suricata/download/null/true/!tag1

Administration is able to maintain a white-list containing host, domain name and IP numbers to exclude from the NIDS export.

Hash - HIDS database export

Automatic export of MD5/SHA1 checksums contained in file-related attributes. This list can be used to feed forensic software when searching for suspicious files. Only published events and attributes marked as IDS Signature are exported.

You can configure your tools to automatically download all the MD5 hashes from MISP:

https://<misp url>/events/hids/md5/download

Or the SHA1 hashes:

https://<misp url>/events/hids/sha1/download

The API's full format is as follow:

https://<misp url>/events/hids/[format]/download/[tags]/[from]/[to]/[last]

format: The export format, can be "md5" or "sha1"
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/events/hids/md5/download/tag1&&tag2&&!tag3

from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

The keywords false or null should be used for optional empty parameters in the URL.

For example, to only show sha1 values from events tagged tag1, use:

https://<misp url>/events/hids/sha1/download/tag1

STIX export

You can export MISP events in MITRE's STIX format (to read more about STIX). The STIX XML export is currently very slow and can lead to timeouts with larger events or collections of events. The STIX JSON return format does not suffer from this issue.

Usage of the API:

https://<misp url>/events/stix/download

Search parameters can be passed to the function via url parameters or by POSTing an xml or json object (depending on the return type). The following parameters can be passed to the STIX export tool: id, withAttachments, tags. Both id and tags can use the && (and) and ! (not) operators to build queries. Using the url parameters, the syntax is as follows:

https://<misp url>/events/stix/download/[id]/[withAttachments]/[tags]/[from]/[to]/[last]

id: The event's ID
withAttachments: Encode attachments where applicable
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).

For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/events/stix/download/false/true/tag1&&tag2&&!tag3

from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

You can post an XML or JSON object containing additional parameters in the following formats.

If you use JSON query objects:

https://<misp url>/events/stix/download.json

{"request": {"id":["!51","!62"],"withAttachment":false,"tags":["APT1","!OSINT"],"from":false,"to":"2015-02-15"}}

If you use XML query objects:

https://<misp url>/events/stix/download

<request><id>!51</id><id>!62</id><withAttachment>false</withAttachment><tags>APT1</tags><tags>!OSINT</tags><from>false</from><to>2015-02-15</to></request>

Various ways to narrow down the search results of the STIX export

For example, to retrieve all events tagged "APT1" but excluding events tagged "OSINT" and excluding events #51 and #62 without any attachments:

https://<misp url>/events/stix/download/!51&&!62/false/APT1&&!OSINT/2015-02-15

To export the same events using a POST request use:

https://<misp url>/events/stix/download.json

Together with this JSON object in the POST message:

{"request": {"id":["!51","!62"],"tags":["APT1","!OSINT"],"from":"2015-02-15"}}

XML is automatically assumed when using the STIX export:

https://<misp url>/events/stix/download

The same search could be accomplished using the following POSTed XML object (note that ampersands need to be escaped, or alternatively separate id and tag elements can be used):

<request><id>!51</id><id>!62</id><tags>APT1</tags><tags>!OSINT</tags><from>2015-02-15</from></request>

RPZ export

You can export RPZ zone files for DNS level firewall by using the RPZ export functionality of MISP. The file generated will include all of the IDS flagged domain, hostname and IP-src/IP-dst attribute values that you have access to.

It is possible to further restrict the exported values using the following filters:

tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search when passed through the url. Use semicolons instead (the search will automatically search for colons instead).
id: The event's ID
from: Events with the date set to a date after the one specified in the from field (format: 2015-02-03)
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-03)

MISP will inject header values into the zone file as well as define the action taken for each of the values that can all be overwritten. By default these values are either the default values shipped with the application, or ones that are overwritten by your site administrator. The values are as follows:

Value name	Default value
RPZ_policy	DROP
RPZ_walled_garden	127.0.0.1
RPZ_serial	$date00
RPZ_refresh	2h
RPZ_retry	30m
RPZ_expiry	30d
RPZ_minimum_ttl	1h
RPZ_ttl	1w
RPZ_ns	localhost.
RPZ_email	root.localhost

To override the above values, either use the url parameters as described below:

https://<misp url>/attributes/rpz/download/[tags]/[eventId]/[from]/[to]/[policy]/[walled_garden]/[ns]/[email]/[serial]/[refresh]/[retry]/[expiry]/[minim
um_ttl]/[ttl]

Or POST an XML or JSON object with the above listed options:

<request><tags>OSINT&&!OUTDATED</tags><policy>walled-garden</policy><walled_garden>teamliquid.net</walled_garden><refresh>5h</refresh></request>

{"request": {"tags": ["OSINT", "!OUTDATED"], "policy": "walled-garden", "walled_garden": "teamliquid.net", "refresh": "5h"}

Text export

An export of all attributes of a specific type to a plain text file. By default only published and IDS flagged attributes are exported.

You can configure your tools to automatically download the following files:

https://<misp url>/attributes/text/download/md5
https://<misp url>/attributes/text/download/sha1
https://<misp url>/attributes/text/download/sha256
https://<misp url>/attributes/text/download/filename
https://<misp url>/attributes/text/download/filename|md5
https://<misp url>/attributes/text/download/filename|sha1
https://<misp url>/attributes/text/download/filename|sha256
https://<misp url>/attributes/text/download/ip-src
https://<misp url>/attributes/text/download/ip-dst
https://<misp url>/attributes/text/download/hostname
https://<misp url>/attributes/text/download/domain
https://<misp url>/attributes/text/download/email-src
https://<misp url>/attributes/text/download/email-dst
https://<misp url>/attributes/text/download/email-subject
https://<misp url>/attributes/text/download/email-attachment
https://<misp url>/attributes/text/download/url
https://<misp url>/attributes/text/download/http-method
https://<misp url>/attributes/text/download/user-agent
https://<misp url>/attributes/text/download/regkey
https://<misp url>/attributes/text/download/regkey|value
https://<misp url>/attributes/text/download/AS
https://<misp url>/attributes/text/download/snort
https://<misp url>/attributes/text/download/pattern-in-file
https://<misp url>/attributes/text/download/pattern-in-traffic
https://<misp url>/attributes/text/download/pattern-in-memory
https://<misp url>/attributes/text/download/yara
https://<misp url>/attributes/text/download/vulnerability
https://<misp url>/attributes/text/download/attachment
https://<misp url>/attributes/text/download/malware-sample
https://<misp url>/attributes/text/download/link
https://<misp url>/attributes/text/download/comment
https://<misp url>/attributes/text/download/text
https://<misp url>/attributes/text/download/other
https://<misp url>/attributes/text/download/named pipe
https://<misp url>/attributes/text/download/mutex
https://<misp url>/attributes/text/download/target-user
https://<misp url>/attributes/text/download/target-email
https://<misp url>/attributes/text/download/target-machine
https://<misp url>/attributes/text/download/target-org
https://<misp url>/attributes/text/download/target-location
https://<misp url>/attributes/text/download/target-external

To restrict the results by tags, use the usual syntax. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). To get ip-src values from events tagged tag1 but not tag2 use:

https://<misp url>/attributes/text/download/ip-src/tag1&&

It is possible to restrict the text exports on additional flags. The first allows the user to restrict based on event ID, whilst the second is a boolean switch allowing non IDS flagged attributes to be exported. Additionally, choosing "all" in the type field will return all eligible attributes.

https://<misp url>/attributes/text/download/[type]/[tags]/[event_id]/[allowNonIDS]/[from]/[to]/[last]

type: The attribute type, any valid MISP attribute type is accepted.
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).
allowNonIDS: Include attributes that would normally be excluded due to the IDS flag not being set or due to being whitelisted
from: Set the lowest "date" field value that should be included in the export (format YYYY-MM-DD)
to: Set the highest "date" field value that should be included in the export (format YYYY-MM-DD)
last: Set the timeframe of the export based on the "timestamp" value. The parameter uses a time + metric notation (valid examples: "2w", "60m", "24h")

For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/attributes/text/download/all/tag1&&tag2&&!tag3

event_id: Restrict the results to the given event IDs.
allowNonIDS: Allow attributes to be exported that are not marked as "to_ids".
from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

The keywords false or null should be used for optional empty parameters in the URL.

For example, to retrieve all attributes for event #5, including non IDS marked attributes too, use the following line:

https://<misp url>/attributes/text/download/all/null/5/true

RESTful searches with XML result export

It is possible to search the database for attributes based on a list of criteria.

To return an event with all of its attributes, relations, shadowAttributes, use the following syntax:

https://<misp url>/events/restSearch/download/[value]/[type]/[category]/[org]/[tag]/[quickfilter]/[from]/[to]/[last]/[eventid]/[withAttachments]/[metadata]/[uuid]

value: Search for the given value in the attributes' value field.
type: The attribute type, any valid MISP attribute type is accepted.
category: The attribute category, any valid MISP attribute category is accepted.
org: Search by the creator organisation by supplying the organisation idenfitier.
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).

For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/events/restSearch/download/null/null/null/null/tag1&&tag2&&!tag3

quickfilter: Enabling this (by passing "1" as the argument) will make the search ignore all of the other arguments, except for the auth key and value. MISP will return an xml / json (depending on the header sent) of all events that have a sub-string match on value in the event info, event orgc, or any of the attribute value1 / value2 fields, or in the attribute comment.
from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.
eventid: The events that should be included / excluded from the search
withAttachments: Include the attachments/encrypted samples in the export
metadata: Only fetch the event metadata (event data, tags, relations) and skip the attributes

The keywords false or null should be used for optional empty parameters in the URL.

For example, to find any event with the term "red october" mentioned, use the following syntax (the example is shown as a POST request instead of a GET, which is highly recommended):

POST to:

https://<misp url>/events/restSearch/download

POST message payload (XML):

   <request><value>red october</value><searchall>1</searchall><eventid>!15</eventid></request>

POST message payload (JSON):

{"request": {"value":"red october","searchall":1,"eventid":"!15"}}

To just return a list of attributes, use the following syntax:

value: Search for the given value in the attributes' value field.
type: The attribute type, any valid MISP attribute type is accepted.
category: The attribute category, any valid MISP attribute category is accepted.
org: Search by the creator organisation by supplying the organisation identifier.
tags: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).
from: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last: Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.
eventid: The events that should be included / excluded from the search.
uuid: The returned events must include an attribute with the given UUID, or alternatively the event's UUID must match the value(s) passed.

The keywords false or null should be used for optional empty parameters in the URL.

https://<misp url>/attributes/restSearch/download/[value]/[type]/[category]/[org]/[tag]/[from]/[to]/[last]/[eventid]/[withattachments]/[uuid]

Value, type, category and org are optional. It is possible to search for several terms in each category by joining them with the '&&' operator. It is also possible to negate a term with the '!' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, in order to search for all attributes created by your organisation that contain 192.168 or 127.0 but not 0.1 and are of the type ip-src, excluding the events that were tagged tag1 use the following syntax:

https://<misp url>/attributes/restSearch/download/192.168&&127.0&&!0.1/ip-src/false/CIRCL/!tag1

You can also use search for IP addresses using CIDR. Make sure that you use '|' (pipe) instead of '/' (slashes). Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). See below for an example:

https://<misp url>/attributes/restSearch/download/192.168.1.1|16/ip-src/null/CIRCL

Export attributes of event with specified type as XML

If you want to export all attributes of a pre-defined type that belong to an event, use the following syntax:

https://<misp url>/attributes/returnAttributes/download/[id]/[type]/[sigOnly]

sigOnly is an optional flag that will block all attributes from being exported that don't have the IDS flag turned on. It is possible to search for several types with the '&&' operator and to exclude values with the '!' operator. For example, to get all IDS signature attributes of type md5 and sha256, but not filename|md5 and filename|sha256 from event 25, use the following:

https://<misp url>/attributes/returnAttributes/download/25/md5&&sha256&&!filename/true

Filtering event metadata

As described in the REST section, it is possible to retrieve a list of events along with their metadata by sending a GET request to the /events API. However, this API in particular is a bit more versatile. You can pass search parameters along to search among the events on various fields and retrieve a list of matching events (along with their metadata). Use the following URL:

https://<misp url>/events/index

POST a JSON object with the desired lookup fields and values to receive a JSON back. An example for a valid lookup:

Authorization: <your API key>
Accept: application/json
Content-type: application/json

Body:

{"searchinfo":"Locky", "searchpublished":1, "searchdistribution":0}

The list of valid parameters:

searchpublished:: Filters on published or unpulished events [0,1] - negatable
searchinfo:: Filters on strings found in the event info - negatable
searchtag:: Filters on attached tag names - negatable
searcheventid:: Filters on specific event IDs - negatable
searchthreatlevel:: Filters on a given event threat level [1,2,3,4] - negatable
searchdistribution:: Filters on the distribution level [0,1,2,3] - negatable
searchanalysis:: Filters on the given analysis phase of the event [0,1,2,3] - negatable
searchattribute:: Filters on a contained attribute value - negatable
searchorg:: Filters on the creator organisation - negatable
searchemail:: Filters on the creator user's email address (admin only) - negatable
searchDatefrom:: Filters on the date, anything newer than the given date in YYYY-MM-DD format is taken - non-negatable
searchDateuntil:: Filters on the date, anything older than the given date in YYYY-MM-DD format is taken - non-negatable

Download attachment or malware sample

If you know the attribute ID of a malware-sample or an attachment, you can download it with the following syntax:

https://<misp url>/attributes/downloadAttachment/download/[Attribute_id]

Download malware sample by hash

You can also download samples by knowing its MD5 hash. Simply pass the hash along as a JSON/XML object or in the URL (with the URL having overruling the passed objects) to receive a JSON/XML object back with the zipped sample base64 encoded along with some contextual information.

You can also use this API to get all samples from events that contain the passed hash. For this functionality, just pass the "allSamples" flag along. Note that if you are getting all samples from matching events, you can use all supported hash types (md5, sha1, sha256) for the lookup.

You can also get all the samples from an event with a given event ID, by passing along the eventID parameter. Make sure that either an event ID or a hash is passed along, otherwise an error message will be returned. Also, if no hash is set, the allSamples flag will get set automatically.

https:///attributes/downloadSample/[hash]/[allSamples]/[eventID]

POST message payload (XML):

<request><hash>7c12772809c1c0c3deda6103b10fdfa0</hash><allSamples>1</allSamples><eventID>13</eventID</request>

POST message payload (json):

{"request": {"hash": "7c12772809c1c0c3deda6103b10fdfa0", "allSamples": 1, "eventID": 13}}

A description of all the parameters in the passed object:

hash: A hash in MD5 format. If allSamples is set, this can be any one of the following: md5, sha1, sha256.
allSamples: If set, it will return all samples from events that have a match for the hash provided above.
eventID: If set, it will only fetch data from the given event ID.

Upload malware samples using the "Upload Sample" API

https://<misp url>/events/upload_sample/[Event_id]

This API will allow you to populate an event that you have modify rights to with malware samples (and all related hashes). Alternatively, if you do not supply an event ID, it will create a new event for you.

The files have to be base64 encoded and POSTed as explained below. All samples will be zipped and password protected (with the password being "infected"). The hashes of the original file will be captured as additional attributes.

The event ID is optional. MISP will accept either a JSON or an XML object posted to the above URL.

The general structure of the expected objects is as follows:

{"request": {"files": [{"filename": filename1, "data": base64encodedfile1}, {"filename": filename2, "data": base64encodedfile2}],
   "optional_parameter1", "optional_parameter2", "optional_parameter3"}}

JSON:

{"request":{"files": [{"filename": "test1.txt", "data": "dGVzdA=="}, {"filename": "test2.txt", "data": "dGVzdDI="}], "distribution": 1, "info" : "test", "event_id": 15}}

XML:

<request><files><filename>test3.txt</filename><data>dGVzdA==</data></files><files><filename>test4.txt</filename><data>dGVzdDI=</data></files><info>test</info><distribution>1</distribution><event_id>15</event_id></request>

The following optional parameters are expected:

event_id: The Event's ID is optional. It can be either supplied via the URL or the POSTed object, but the URL has priority if both are provided. Not supplying an event ID will cause MISP to create a single new event for all of the POSTed malware samples. You can define the default settings for the event, otherwise a set of default settings will be used.
distribution: The distribution setting used for the attributes and for the newly created event, if relevant. [0-3]
to_ids: You can flag all attributes created during the transaction to be marked as "to_ids" or not.
category: The category that will be assigned to the uploaded samples. Valid options are: Payload delivery, Artifacts dropped, Payload Installation, External Analysis.
info: Used to populate the event info field if no event ID supplied. Alternatively, if not set, MISP will simply generate a message showing that it's a malware sample collection generated on the given day.
analysis: The analysis level of the newly created event, if applicable. [0-2] threat_level_id: The threat level ID of the newly created event, if applicatble. [0-3]
comment: This will populate the comment field of any attribute created using this API.

Add or remove tags from events

You can add or remove an existing tag from an event in the following way:

https://<misp url>/events/addTag
https://<misp url>/events/removeTag

Just POST a JSON object in the following format (to the appropriate API depending on whether you want to add or delete a tag from an event):

{"request": {"Event": {"id": "228", "tag": "8"}}}

Where "tag" is the ID of the tag. You can also use the name of the tag the following way (has to be an exact match):

{"request": {"Event": {"id": "228", "tag": "OSINT"}}}

Proposals API

You can interact with the proposals via the API directly since version 2.3.148.

HTTP	URL	Explanation	Expected Payload	Response
GET	/shadow_attributes/view/[proposal_id]	View a proposal	N/A	ShadowAttribute object
POST	/shadow_attributes/add/[event_id]	Propose a new attribute to an event	ShadowAttribute object	ShadowAttribute object
POST	/shadow_attributes/edit/[attribute_id]	Propose an edit to an attribute	ShadowAttribute object	ShadowAttribute object
POST	/shadow_attributes/accept/[proposal_id]	Accept a proposal	N/A	Message
POST	/shadow_attributes/discard/[proposal_id]	Discard a proposal	N/A	Message

When posting a shadow attribute object, use the following format

JSON:

{"request": {"ShadowAttribute": {"value": "5.5.5.5", "to_ids": false, "type": "ip-dst", "category": "Network activity"}}}

XML:

<request><ShadowAttribute><value>5.5.5.5</value><to_ids>0</to_ids><type>ip-src</type><category>Network activity</category></ShadowAttribute></request>

None of the above fields are mandatory, but at least one of them has to be provided.

MISP allows sharing groups to be retrieved via the API.

https://<misp url>/sharing_groups/index.json

Based on the API key used, the list of visible sharing groups will be returned in a JSON file. The JSON includes the organization parts of a given sharing group along with the associated server.

Enable, disable and fetching feeds via the API

The MISP feeds can be enabled via the API.

A feed can be enabled by POSTing on the following url (feed_id is the id of the feed):

/feeds/enable/feed_id

A feed can be disabled by POSTing on the following url (feed_id is the id of the feed):

/feeds/disable/feed_id

All feeds can fetch via the API:

/feeds/cacheFeeds/all

or you can replace all by the feed format to fetch like misp or freetext. all can be replaced with the id value of the feed to fetch a specific feed.

This API can be also used to download feeds at regular interval via cronjobs or alike.

Sightings API

MISP allows Sightings data to be conveyed in several ways.

The most basic way is to POST a blank message to the Sightings API with the attribute ID or attribute UUID. This will create a sightings entry with the creation of the entry as the timestamp for the organisation of the authenticated user.

https://<misp url>/sightings/add/[attribute_id]
https://<misp url>/sightings/add/[attribute_uuid]

Alternatively, it is possible to POST a JSON object and gain additional granularity. The following fields are recognised by the API:

id: The attribute's ID
uuid: The attribute's UUID
value: Will create a sighting for any attribute with the given value or for composite attributes, for the value matching any element of the attribute value
values: Expects a list, MISP will create sightings for any attribute matching any of the given values or for composite attributes, for any of the values matching any element of the attribute value
timestamp: Unix timestamp of the sighting, overrides the current time

Some examples:

To create a sighting for attribute #9001:

{"id":"9001"}

To create a sighting for any attribute with the value being teamliquid.net or 173.231.136.216 with the time of sighting being :

{"values":["teamliquid.net", "173.231.136.216"], "timestamp":1460558710}

It is also possible to POST a STIX indicator with sighting data to the following URL (keep in mind that the content type has to be XML):

https://<misp url>/sightings/add/stix

MISP will use the sighting's related observables to gather all values and create sightings for each attribute that matches any of the values. If no related observables are provided in the Sighting object, then MISP will fall back to the Indicator itself and use its observables' values to create the sightings. The time of the sighting is the current time, unless the timestamp attribute is set on the Sightings object, in which case that is taken.

An example STIX sightings document:

<stix:STIX_Package
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:stix="http://stix.mitre.org/stix-1"
    xmlns:indicator="http://stix.mitre.org/Indicator-2"
    xmlns:stixCommon="http://stix.mitre.org/common-1"
    xmlns:cybox="http://cybox.mitre.org/cybox-2"
    xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
    xmlns:DomainNameObj="http://cybox.mitre.org/objects#DomainNameObject-1"
    xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
    xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
    xmlns:example="http://example.com/"
    xsi:schemaLocation="
    http://stix.mitre.org/stix-1 ../stix_core.xsd
    http://stix.mitre.org/Indicator-2 ../indicator.xsd
    http://cybox.mitre.org/objects#DomainNameObject-1 http://cybox.mitre.org/XMLSchema/objects/Domain_Name/1.0/Domain_Name_Object.xsd
    http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
    http://cybox.mitre.org/default_vocabularies-2 ../cybox/cybox_default_vocabularies.xsd
    http://stix.mitre.org/default_vocabularies-1 ../stix_default_vocabularies.xsd
    http://cybox.mitre.org/objects#AddressObject-2 ../cybox/objects/Address_Object.xsd"
    id="example:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
    timestamp="2014-05-08T09:00:00.000000Z"
    version="1.1.1"
    >
    <stix:STIX_Header>
        <stix:Title>Example watchlist that contains IP information.</stix:Title>
        <stix:Package_Intent xsi:type="stixVocabs:PackageIntentVocab-1.0">Indicators - Watchlist</stix:Package_Intent>
    </stix:STIX_Header>
    <stix:Indicators>
        <stix:Indicator xsi:type="indicator:IndicatorType" id="example:Indicator-2e20c5b2-56fa-46cd-9662-8f199c69d2c9" timestamp="2014-05-08T09:00:00.000000Z">
          <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">Domain Watchlist</indicator:Type>
          <indicator:Observable id="example:Observable-87c9a5bb-d005-4b3e-8081-99f720fad62b">
              <cybox:Object id="example:Object-12c760ba-cd2c-4f5d-a37d-18212eac7928">
                  <cybox:Properties xsi:type="DomainNameObj:DomainNameObjectType" type="FQDN">
                      <DomainNameObj:Value condition="Equals" apply_condition="ANY">malicious1.example.com##comma##malicious2.example.com##comma##malicious3.example.com</DomainNameObj:Value>
                  </cybox:Properties>
              </cybox:Object>
          </indicator:Observable>
          <indicator:Sightings>
            <indicator:Sighting timestamp="2014-05-08T09:00:00.000000Z">
              <indicator:Source>
                <stixCommon:Identity>
                  <stixCommon:Name>FooBar Inc.</stixCommon:Name>
                </stixCommon:Identity>
              </indicator:Source>
              <indicator:Related_Observables>
                <indicator:Related_Observable>
                  <stixCommon:Observable id="example:Observable-45b3acdf-1888-4bcc-89a9-6d9f8116fede">
                    <cybox:Object id="example:Object-a3d36250-42fa-4653-9172-87b87598390c">
                      <cybox:Properties xsi:type="DomainNameObj:DomainNameObjectType" type="FQDN">
                        <DomainNameObj:Value>malicious2.example.com</DomainNameObj:Value>
                      </cybox:Properties>
                    </cybox:Object>
                  </stixCommon:Observable>
                </indicator:Related_Observable>
              </indicator:Related_Observables>
            </indicator:Sighting>
          </indicator:Sightings>
        </stix:Indicator>
    </stix:Indicators>
</stix:STIX_Package>

POSTing this as the message's body to MISP will sight any attributes visible to the user with he value "malicious2.example.com". For composite types, a match on a component will also trigger a sighting (so for example for attributes of type domain|ip a domain match would be sufficient).

If no Related observables are set in the Sighting itself, MISP will fall back to the observable directly contained in the indicator. So in the following example:

<stix:STIX_Package
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:stix="http://stix.mitre.org/stix-1"
    xmlns:indicator="http://stix.mitre.org/Indicator-2"
    xmlns:stixCommon="http://stix.mitre.org/common-1"
    xmlns:cybox="http://cybox.mitre.org/cybox-2"
    xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
    xmlns:DomainNameObj="http://cybox.mitre.org/objects#DomainNameObject-1"
    xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
    xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
    xmlns:example="http://example.com/"
    xsi:schemaLocation="
    http://stix.mitre.org/stix-1 ../stix_core.xsd
    http://stix.mitre.org/Indicator-2 ../indicator.xsd
    http://cybox.mitre.org/objects#DomainNameObject-1 http://cybox.mitre.org/XMLSchema/objects/Domain_Name/1.0/Domain_Name_Object.xsd
    http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
    http://cybox.mitre.org/default_vocabularies-2 ../cybox/cybox_default_vocabularies.xsd
    http://stix.mitre.org/default_vocabularies-1 ../stix_default_vocabularies.xsd
    http://cybox.mitre.org/objects#AddressObject-2 ../cybox/objects/Address_Object.xsd"
    id="example:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
    timestamp="2014-05-08T09:00:00.000000Z"
    version="1.1.1"
    >
    <stix:STIX_Header>
        <stix:Title>Example watchlist that contains IP information.</stix:Title>
        <stix:Package_Intent xsi:type="stixVocabs:PackageIntentVocab-1.0">Indicators - Watchlist</stix:Package_Intent>
    </stix:STIX_Header>
    <stix:Indicators>
        <stix:Indicator xsi:type="indicator:IndicatorType" id="example:Indicator-2e20c5b2-56fa-46cd-9662-8f199c69d2c9" timestamp="2014-05-08T09:00:00.000000Z">
          <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">Domain Watchlist</indicator:Type>
          <indicator:Observable id="example:Observable-87c9a5bb-d005-4b3e-8081-99f720fad62b">
              <cybox:Object id="example:Object-12c760ba-cd2c-4f5d-a37d-18212eac7928">
                  <cybox:Properties xsi:type="DomainNameObj:DomainNameObjectType" type="FQDN">
                      <DomainNameObj:Value condition="Equals" apply_condition="ANY">malicious1.example.com##comma##malicious2.example.com##comma##malicious3.example.com</DomainNameObj:Value>
                  </cybox:Properties>
              </cybox:Object>
          </indicator:Observable>
          <indicator:Sightings>
            <indicator:Sighting timestamp="2014-05-08T09:00:00.000000Z">
              <indicator:Source>
                <stixCommon:Identity>
                  <stixCommon:Name>FooBar Inc.</stixCommon:Name>
                </stixCommon:Identity>
              </indicator:Source>
            </indicator:Sighting>
          </indicator:Sightings>
        </stix:Indicator>
    </stix:Indicators>
</stix:STIX_Package>

MISP would create sightings for attributes matching any of the following: malicious1.example.com, malicious2.example.com, malicious3.example.com

Describe types API

MISP can procedurally describe all attribute types and attribute categories it currently supports including the category - type mappings. To access this information simply send a GET request to:

https://<misp url>/attributes/describeTypes

Depending on the headers passed the returned data will be a JSON object or an XML, with 3 main sections: types, categories, category_type_mappings.

Attribute statistics API

If you are interested in the attribute type or attribute category data distribution on your instance, MISP offers an API that will create an aggregates list. To access the API, simple sent a GET request to:

https://<misp url>/attributes/attributeStatistics/[context]/[percentage]

Where the following parameters can be set:

Context: Set whether you are interested in the type or category statistics of your instance. This parameter can be either set to "type" or "category", with type being the default setting if the parameter is not set.
Percentage: An optional field, if set, it will return the results in percentages instead of the count.

The results are always returned as JSON.

Sample output of the types in percentages from CIRCL's MISP instance:

{
    "AS": "0.015%",
    "attachment": "0.177%",
    "btc": "0.005%",
    "campaign-name": "0.005%",
    "comment": "1.47%",
    "domain": "15.992%",
    "domain|ip": "0.005%",
    "email-attachment": "0.207%",
    "email-dst": "0.121%",
    "email-src": "0.192%",
    "email-subject": "0.146%",
    "filename": "3.698%",
    "filename|md5": "0.349%",
    "filename|sha1": "0.894%",
    "filename|sha256": "0.652%",
    "hostname": "17.558%",
    "http-method": "0.045%",
    "ip-dst": "7.087%",
    "ip-src": "2.707%",
    "link": "5.748%",
    "malware-sample": "0.702%",
    "malware-type": "0.005%",
    "md5": "21.064%",
    "mutex": "0.278%",
    "named pipe": "0.03%",
    "other": "1.495%",
    "pattern-in-file": "0.192%",
    "pattern-in-memory": "0.303%",
    "pattern-in-traffic": "0.051%",
    "regkey": "0.126%",
    "regkey|value": "0.187%",
    "sha1": "8.921%",
    "sha256": "5.597%",
    "snort": "0.045%",
    "target-machine": "0.248%",
    "target-org": "0.01%",
    "target-user": "0.106%",
    "text": "0.934%",
    "threat-actor": "0.005%",
    "url": "2.258%",
    "user-agent": "0.081%",
    "vulnerability": "0.182%",
    "whois-registrant-email": "0.01%",
    "x509-fingerprint-sha1": "0.01%",
    "yara": "0.086%"
}

Additional statistics

Additional statistics are available as JSON which are the statistics also usable via the user interface. A ".json" can be appended to the following URLs:

https:///users/statistics/tags.json
https:///users/statistics.json
https:///users/statistics/attributehistogram.json
https:///users/statistics/orgs.json

An example output of https:///users/statistics.json:

{
    "stats": {
        "event_count": 5233,
        "event_count_month": 21,
        "attribute_count": 645498,
        "attribute_count_month": 723,
        "correlation_count": 207152,
        "proposal_count": 48944,
        "user_count": 1073,
        "org_count": 587,
        "thread_count": 191,
        "thread_count_month": 0,
        "post_count": 337,
        "post_count_month": 0
    }
}

User management

MISP allows administrators to create and manage users via its REST API

The API is available in JSON format so make sure you use the following headers:

Authorization: [Your auth key]
Content-type: application/json
Accept: application/json

To fetch all users send a GET request to:

https://<misp url>/admin/users

To view a user simply send a GET request to the following url:

https://<misp url>/admin/users/view/[user id]

To create a new user, send a POST request to:

https://<misp url>/admin/users/add

Sample input:

{
    "email":"andras.iklody@circl.lu",
    "org\_id":1,
    "role\_id":1
}

To view the mandatory and optional fields, use a GET request on the above URL.

Sample output:

{
    "name": "\/admin\/users\/add API description",
    "description": "POST a User object in JSON format to this API to create a new user.",
    "mandatory_fields": [
        "email",
        "org_id",
        "role_id"
    ],
    "optional_fields": [
        "password",
        "external_auth_required",
        "external_auth_key",
        "enable_password",
        "nids_sid",
        "server_id",
        "gpgkey",
        "certif_public",
        "autoalert",
        "contactalert",
        "disabled",
        "change_pw",
        "termsaccepted",
        "newsread"
    ],
    "url": "\/admin\/users\/add"
}

To edit an existing user send a POST request to:

https://<misp url>/admin/users/edit/[user id]

Only the fields POSTed will be updated, the rest is left intact. To view all possible parameters, simply send a GET request to the above URL.

You can also delete users by POSTing to the below URL, but keep in mind that disabling users (by setting the disabled flag via an edit) is always prefered to keep user associations to events intact.

https://<misp url>/admin/users/delete/[user id]

Organisation management

MISP allows administrators to create and manage organisations via its REST API

The API is available in JSON format so make sure you use the following headers:

Authorization: [Your auth key]
Content-type: application/json
Accept: application/json

To fetch all organisations send a GET request to:

https://<misp url>/organisations

To view an individual organisation, send a get request to:

https://<misp url>/organisations/view/id

The management of users happens via three apis:

https://<misp url>/admin/organisations/add
https://<misp url>/admin/organisations/edit/[org id]
https://<misp url>/admin/organisations/delete/[org id]

To delete an organisation simply send a POST or DELETE request to the above URL.

For creating or modifying an organisation, simply POST a JSON containing the relevant fields to the appropriate API. The only mandatory field is the organisation name, with a host of optional parameters

An example for a simple organisation object:

{
    "name": "Blizzard",
    "nationality": "US"
}

Not setting a field will assume the default settings for the given field in the case of a new organisation whilst it would retain the existing setting for existing organisations. The above example would create the following object in MISP:

{
    "Organisation": {
        "id": "1108",
        "name": "Blizzard",
        "alias": "",
        "anonymise": false,
        "date_created": "2017-01-22 17:32:29",
        "date_modified": "2017-01-22 17:32:29",
        "description": "",
        "type": "",
        "nationality": "US",
        "sector": "",
        "created_by": "1",
        "uuid": "5884de9d-04f0-4d7d-bf15-0b88c0a83865",
        "contacts": "",
        "local": true,
        "landingpage": ""
    }
}

To query the add or edit APIs for the valid parameters, simply send a GET requests to either API. The result currently looks like this (which might change when new fields are added):

{
    "name": "\/admin\/organisations\/add API description",
    "description": "POST an Organisation object in JSON format to this API to create a new organsiation.",
    "mandatory_fields": [
        "name"
    ],
    "optional_fields": [
        "anonymise",
        "description",
        "type",
        "nationality",
        "sector",
        "uuid",
        "contacts",
        "local"
    ],
    "url": "\/admin\/organisations\/add"
}

Discussion API

If you would like to fetch a discussion thread including all of its posts, simply send a GET request to:

https://<misp url>/threads/view/<thread id>

Using the following headers:

Authorization: [Your auth key]
Content-type: application/json
Accept: application/json

To get all posts related to an event simply send a GET request to:

https://<misp url>/threads/viewEvent/<event id>

Automation using PyMISP

PyMISP is a Python library to access MISP platforms via their REST API.

PyMISP allows you to fetch events, add or update events/attributes, add or update samples or search for attributes.

PyMISP is available including a documentation with various examples.

README.md

Automation API

Automation URL

Automation key

Accept and Content-Type headers

XML Export

JSON query format

XML query format

XML download and URL parameters

CSV export

NIDS rules export

Hash - HIDS database export

STIX export

Various ways to narrow down the search results of the STIX export

RPZ export

Text export

RESTful searches with XML result export

Export attributes of event with specified type as XML

Filtering event metadata

Download attachment or malware sample

Download malware sample by hash

Upload malware samples using the "Upload Sample" API

Add or remove tags from events

Proposals API

Sharing groups

Enable, disable and fetching feeds via the API

Sightings API

Describe types API

Attribute statistics API

Additional statistics

User management

Organisation management

Discussion API

Automation using PyMISP