misp-rfc/misp-query-format/raw.md.txt

449 lines
15 KiB
Plaintext
Raw Normal View History

Network Working Group A. Dulaunoy
Internet-Draft A. Iklody
Intended status: Informational CIRCL
Expires: April 11, 2019 October 8, 2018
MISP query format
draft-dulaunoy-misp-core-format
Abstract
This document describes the MISP query format used to search MISP
(Malware Information and threat Sharing Platform) [MISP-P] threat
intelligence instances. MISP query format is a simple format used to
query MISP instances over a REST (Representational State Transfer )
interface. The query format includes the JSON format to describe the
query and the minimal API access to perform the query. The JSON
format includes the overall structure along with the semantic
associated for each respective key. The goal of the format is to
query MISP threat intelligence instances can feed and integrate with
network security devices (such as firewall, network intrusion
detection system, routers, SIEMs), endpoint security devices or
monitoring devices.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 11, 2019.
Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
Dulaunoy & Iklody Expires April 11, 2019 [Page 1]
Internet-Draft MISP query format October 2018
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Conventions and Terminology . . . . . . . . . . . . . . . 3
2. Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2. query format criteria . . . . . . . . . . . . . . . . . . 3
2.2.1. returnFormat . . . . . . . . . . . . . . . . . . . . 3
2.2.2. limit . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.3. page . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.4. value . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.5. type . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.6. category . . . . . . . . . . . . . . . . . . . . . . 5
2.2.7. org . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.8. tags . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.9. quickfilter . . . . . . . . . . . . . . . . . . . . . 5
2.2.10. from . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.11. to . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.12. last . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.13. eventid . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.14. withAttachments . . . . . . . . . . . . . . . . . . . 6
2.2.15. uuid . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.16. publish_timestamp . . . . . . . . . . . . . . . . . . 6
2.2.17. timestamp . . . . . . . . . . . . . . . . . . . . . . 7
2.2.18. published . . . . . . . . . . . . . . . . . . . . . . 7
2.2.19. enforceWarninglist . . . . . . . . . . . . . . . . . 7
2.2.20. to_ids . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.21. deleted . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.22. includeEventUuid . . . . . . . . . . . . . . . . . . 7
2.2.23. event_timestamp . . . . . . . . . . . . . . . . . . . 7
2.2.24. sgReferenceOnly . . . . . . . . . . . . . . . . . . . 7
2.2.25. eventinfo . . . . . . . . . . . . . . . . . . . . . . 7
2.2.26. searchall . . . . . . . . . . . . . . . . . . . . . . 7
2.2.27. requested_attributes . . . . . . . . . . . . . . . . 7
2.2.28. includeContext . . . . . . . . . . . . . . . . . . . 7
3. Security Considerations . . . . . . . . . . . . . . . . . . . 7
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7
5. References . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.1. Normative References . . . . . . . . . . . . . . . . . . 8
5.2. Informative References . . . . . . . . . . . . . . . . . 8
Dulaunoy & Iklody Expires April 11, 2019 [Page 2]
Internet-Draft MISP query format October 2018
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8
1. Introduction
Sharing threat information became a fundamental requirements in the
Internet, security and intelligence community at large. Threat
information can include indicators of compromise, malicious file
indicators, financial fraud indicators or even detailed information
about a threat actor. MISP [MISP-P] started as an open source
project in late 2011 and the MISP format started to be widely used as
an exchange format within the community in the past years. The core
format is described in an Internet-Draft as misp-core-format [MISP-C]
and contain the standard MISP JSON format used for threat
intelligence.
The aim of this document is to describe the specification of the MISP
query format and how the query can be perform against a REST
interface.
1.1. Conventions and Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
2. Format
2.1. Overview
The MISP query format is in the JSON [RFC8259] format.
2.2. query format criteria
2.2.1. returnFormat
returnFormat MUST be present. returnFormat sets the type of output
format. MISP allows multiple format (depending of the
configuration):
Dulaunoy & Iklody Expires April 11, 2019 [Page 3]
Internet-Draft MISP query format October 2018
+----------+-------------------------------------------------+
| value | Description |
+----------+-------------------------------------------------+
| json | MISP JSON core format as described in [MISP-C] |
| xml | MISP XML format |
| openioc | OpenIOC format |
| suricata | Suricata NIDS format |
| snort | Snort NIDS format |
| csv | CSV format |
| rpz | Response policy zone format |
| text | Raw value list format |
| cache | MISP cache format (hashed values of attributes) |
+----------+-------------------------------------------------+
2.2.2. limit
limit MAY be present. If present, the page parameter MUST also be
supplied. limit sets the number of returned elements when paginating,
depending on the scope of the request (x number of attributes or x
number of events) as converted into the output format.
2.2.3. page
page MAY be present. If present, the page parameter MUST also be
supplied. page generates the offset for the pagination and will
return a result set consisting of a slice of the query results
starting with offset (limit * page) + 1 and ending with (limit *
(page+1)).
2.2.4. value
value MAY be present. If set, the returned data set will be filtered
on the attribute value field. value MUST be a string or a sub-string,
the latter of which starts with, ends with or is encapsulated in
wildcard (\%) characters.
2.2.5. type
type MAY be present. If set, the returned data set will be filtered
on the attribute type field. type MUST be a string or a sub-string,
the latter of which starts with, ends with or is encapsulated in
wildcard (\%) characters. The list of valid attribute types is
described in the MISP core format [MISP-C] in the attribute type
section.
Dulaunoy & Iklody Expires April 11, 2019 [Page 4]
Internet-Draft MISP query format October 2018
2.2.6. category
category MAY be present. If set, the returned data set will be
filtered on the attribute category field. category MUST be a string
or a sub-string, the latter of which starts with, ends with or is
encapsulated in wildcard (\%) characters. The list of valid
categories is described in the MISP core format [MISP-C] in the
attribute type section.
A sample query to lookup for the last 30 days of indicators in the
"Financial fraud" category and output in CSV format:
{
"returnFormat": "csv",
"last": "30d",
"category": "Financial fraud"
}
2.2.7. org
org MAY be present. If set, the returned data set will be filtered
by the organisation identifier (local ID of the instance). org MUST
be the identifier of the organisation in a string format.
2.2.8. tags
tags MAY be present. If set, the returned data set will be filtered
by tags. tags MUST be a string or a sub-string, the latter of which
starts with, ends with or is encapsulated in wildcard (\%)
characters.
{
"returnFormat": "cache",
"limit": "100",
"tags": ["tlp:red", "%private%"]
}
2.2.9. quickfilter
2.2.10. from
from MAY be present. If set, the returned data set will be filtered
from a starting date. from MUST be a string represented in the format
year-month-date.
Dulaunoy & Iklody Expires April 11, 2019 [Page 5]
Internet-Draft MISP query format October 2018
{
"returnFormat": "json",
"limit": "100",
"tags": ["tlp:amber"],
"from": "2018-09-02",
"to": "2018-10-01"
}
2.2.11. to
to MAY be present. If set, the returned data set will be filtered
until the specified date. from MUST be a string represented in the
format year-month-date.
2.2.12. last
last MAY be present. If set, the returned data set will be filtered
in the number of days, hours or minutes defined (such as 5d, 12h or
30m). last MUST be a string represented in the format expressing
days, hours or minutes.
2.2.13. eventid
eventid MAY be present. If set, the returned data set will be
filtered to a specific event. eventid MUST be a string representing
the event id as an integer.
{
"returnFormat": "json",
"eventid": 1
}
2.2.14. withAttachments
withAttachments MAY be present. If set to True (1), the returned
data set will include the attachment(s) matching the query.
withAttachments MUST be an integer set as 1 (True) to include the
attachment(s). If not, the attachment(s) won't be included in the
results.
2.2.15. uuid
2.2.16. publish_timestamp
Dulaunoy & Iklody Expires April 11, 2019 [Page 6]
Internet-Draft MISP query format October 2018
2.2.17. timestamp
2.2.18. published
2.2.19. enforceWarninglist
2.2.20. to_ids
2.2.21. deleted
2.2.22. includeEventUuid
2.2.23. event_timestamp
2.2.24. sgReferenceOnly
2.2.25. eventinfo
2.2.26. searchall
2.2.27. requested_attributes
2.2.28. includeContext
3. Security Considerations
MISP threat intelligence instances might contain sensitive or
confidential information. Adequate access control and encryption
measures shall be implemented to ensure the confidentiality of the
threat intelligence.
Adversaries might include malicious content in MISP queries.
Implementation MUST consider the input of malicious inputs beside the
standard threat information that might already include malicious
intended inputs.
4. Acknowledgements
The authors wish to thank all the MISP community who are supporting
the creation of open standards in threat intelligence sharing. A
special thank to all the committees which triggered us to come with
better and flexible format.
5. References
Dulaunoy & Iklody Expires April 11, 2019 [Page 7]
Internet-Draft MISP query format October 2018
5.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>.
5.2. Informative References
[MISP-C] MISP, "MISP core format", <https://tools.ietf.org/html/
draft-dulaunoy-misp-core-format>.
[MISP-P] MISP, "MISP Project - Malware Information Sharing Platform
and Threat Sharing", <https://github.com/MISP>.
Authors' Addresses
Alexandre Dulaunoy
Computer Incident Response Center Luxembourg
16, bd d'Avranches
Luxembourg L-1160
Luxembourg
Phone: +352 247 88444
Email: alexandre.dulaunoy@circl.lu
Andras Iklody
Computer Incident Response Center Luxembourg
16, bd d'Avranches
Luxembourg L-1160
Luxembourg
Phone: +352 247 88444
Email: andras.iklody@circl.lu
Dulaunoy & Iklody Expires April 11, 2019 [Page 8]