mirror of https://github.com/MISP/misp-rfc
393 lines
11 KiB
Plaintext
393 lines
11 KiB
Plaintext
|
||
|
||
|
||
|
||
Network Working Group S. Tricaud
|
||
Internet-Draft Devo Inc.
|
||
Intended status: Informational November 3, 2019
|
||
Expires: May 6, 2020
|
||
|
||
|
||
SightingDB query format
|
||
draft-tricaud-sightingdb-format
|
||
|
||
Abstract
|
||
|
||
This document describes the format used by SightingDB to give
|
||
automated context to a given Attribute by counting occurrences and
|
||
tracking times of observability. SightingDB was designed to provide
|
||
to MISP a Scalable and Fast way to store and retrieve Attributes.
|
||
|
||
Status of This Memo
|
||
|
||
This Internet-Draft is submitted in full conformance with the
|
||
provisions of BCP 78 and BCP 79.
|
||
|
||
Internet-Drafts are working documents of the Internet Engineering
|
||
Task Force (IETF). Note that other groups may also distribute
|
||
working documents as Internet-Drafts. The list of current Internet-
|
||
Drafts is at https://datatracker.ietf.org/drafts/current/.
|
||
|
||
Internet-Drafts are draft documents valid for a maximum of six months
|
||
and may be updated, replaced, or obsoleted by other documents at any
|
||
time. It is inappropriate to use Internet-Drafts as reference
|
||
material or to cite them other than as "work in progress."
|
||
|
||
This Internet-Draft will expire on May 6, 2020.
|
||
|
||
Copyright Notice
|
||
|
||
Copyright (c) 2019 IETF Trust and the persons identified as the
|
||
document authors. All rights reserved.
|
||
|
||
This document is subject to BCP 78 and the IETF Trust's Legal
|
||
Provisions Relating to IETF Documents
|
||
(https://trustee.ietf.org/license-info) in effect on the date of
|
||
publication of this document. Please review these documents
|
||
carefully, as they describe your rights and restrictions with respect
|
||
to this document. Code Components extracted from this document must
|
||
include Simplified BSD License text as described in Section 4.e of
|
||
the Trust Legal Provisions and are provided without warranty as
|
||
described in the Simplified BSD License.
|
||
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 1]
|
||
|
||
Internet-Draft SightingDB query format November 2019
|
||
|
||
|
||
Table of Contents
|
||
|
||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
|
||
1.1. Conventions and Terminology . . . . . . . . . . . . . . . 2
|
||
2. Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
|
||
2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 2
|
||
2.1.1. Attribute Storage . . . . . . . . . . . . . . . . . . 2
|
||
2.1.2. Namespace . . . . . . . . . . . . . . . . . . . . . . 3
|
||
2.1.3. Attribute fields . . . . . . . . . . . . . . . . . . 3
|
||
2.2. SightingDB Format - One Attribute . . . . . . . . . . . . 4
|
||
2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5
|
||
2.3.1. Configuring the value format for a Namespace . . . . 5
|
||
2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5
|
||
2.4.1. Response . . . . . . . . . . . . . . . . . . . . . . 6
|
||
3. Security Considerations . . . . . . . . . . . . . . . . . . . 6
|
||
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6
|
||
5. Normative References . . . . . . . . . . . . . . . . . . . . 6
|
||
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6
|
||
|
||
1. Introduction
|
||
|
||
Adding context to any Attribute is the key that makes it useful.
|
||
While there exist numerous ways of doing it, SightingDB does it by
|
||
just counting. Whenever somebody retrieves an Attribute, this
|
||
counting is provided, allowing anyone to understand whenever
|
||
something was observed few or many times.
|
||
|
||
1.1. Conventions and Terminology
|
||
|
||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
|
||
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
|
||
document are to be interpreted as described in RFC 2119 [RFC2119].
|
||
|
||
2. Format
|
||
|
||
2.1. Overview
|
||
|
||
The SightingDB format is in JSON [RFC8259] format and used to query a
|
||
SightingDB compatible connector. In SightingDB, a Sighting Object is
|
||
composed of a single JSON object. This object contains the following
|
||
fields: value, first_seen, last_seen, count, tags, ttl and manifold.
|
||
|
||
2.1.1. Attribute Storage
|
||
|
||
The fields described previously describe an Attribute and all the
|
||
required characteristics. However they are stored in a Namespace. A
|
||
Namespace is similar to a path in a file-system where the same file
|
||
can be stored in multiple places.
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 2]
|
||
|
||
Internet-Draft SightingDB query format November 2019
|
||
|
||
|
||
2.1.2. Namespace
|
||
|
||
A Namespace with multiple levels MUST be separated with the slash '/'
|
||
character. There is no specification on how they are structured,
|
||
since it depends on the use cases.
|
||
|
||
A Namespace starting with the underscore '_' character means it is
|
||
private and internal to SightingDB. There are all reserved for the
|
||
engine and MUST NOT be used.
|
||
|
||
Reserved namespaces are:
|
||
|
||
_expired/: Which contains all the attributes that expired, preserving
|
||
the origin namespace
|
||
|
||
_shadow/: When a value is searched and does not exists, it is stored
|
||
there
|
||
|
||
_stats: Statistics
|
||
|
||
_config: Configuration
|
||
|
||
_all: All the Attributes in one place, used to retrieve the
|
||
'manifold' property.
|
||
|
||
The Attribute Key MUST always be the last part of the Namespace.
|
||
|
||
2.1.2.1. Sample Namespaces
|
||
|
||
/Organization1/service/ipv4: Store values for ipv4 keys in
|
||
/Organization1/service
|
||
|
||
/everything/domain: Store domains in /everything
|
||
|
||
2.1.3. Attribute fields
|
||
|
||
2.1.3.1. value
|
||
|
||
The attribute value, used to store and retrieve information about an
|
||
attribute. Note that value is not returned back in the JSON object,
|
||
since it is queried, it is known. The Value is described in a
|
||
section below, as it is very specific and can be either "as is", a
|
||
hash, encoded in base64 or any other convenient mechanism.
|
||
|
||
The value implementation MUST offer at least: 1) Raw value 2) Base64
|
||
URL Encoded 3) SHA256 Hash
|
||
|
||
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 3]
|
||
|
||
Internet-Draft SightingDB query format November 2019
|
||
|
||
|
||
2.1.3.2. first_seen
|
||
|
||
Time in UTC of the first time this value was captured
|
||
|
||
2.1.3.3. last_seen
|
||
|
||
Time in UTC of the last time this value was captured
|
||
|
||
2.1.3.4. count
|
||
|
||
How many time this value was written
|
||
|
||
2.1.3.5. tags
|
||
|
||
Tags follow how they are defined in MISP using the MISP Taxonomy.
|
||
Each Tag is separated with the ';' character.
|
||
|
||
2.1.3.6. ttl
|
||
|
||
Time To Live, represents the expiration in seconds since the time the
|
||
Attribute was created. Once it has expired, it moves in the private
|
||
Namespace _expired.
|
||
|
||
When an Attribute has this field set to 0, it means it is not set to
|
||
expired. This is the default behavior.
|
||
|
||
When an Attribute has this field set to a number greater than 0, the
|
||
expiration status is computed only at retrieval time.
|
||
|
||
2.1.3.7. manifold
|
||
|
||
When a given Attribute Value is stored in different namespaces, the
|
||
manifold field keeps track of them so it returns in how many
|
||
different places this attributes exists. This is a simple counter.
|
||
|
||
2.2. SightingDB Format - One Attribute
|
||
|
||
{
|
||
"value":"127.0.0.1",
|
||
"first_seen":1530394819,
|
||
"last_seen":1572933618,
|
||
"count":578391,
|
||
"tags":"",
|
||
"ttl":0,
|
||
"manifold": 17
|
||
}
|
||
|
||
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 4]
|
||
|
||
Internet-Draft SightingDB query format November 2019
|
||
|
||
|
||
2.3. Value
|
||
|
||
The value submitted can be in multiple format according to the use-
|
||
case. Any implementation MUST offer three alternatives:
|
||
|
||
1. Raw value: where nothing is encoded and the value is stored AS
|
||
IS, such as show in the example above with the One Attribute in
|
||
JSON.
|
||
|
||
2. SHA256: which prevents from seeing content (see Security
|
||
Considerations), has a fixed size and is convenient for most
|
||
requirements
|
||
|
||
3. Base64 URL: Where the specification of Base64 is followed, except
|
||
the characters conflicting with an URL argument are replaced
|
||
|
||
The value is configured as part of the Namespace. The private
|
||
"_config" Namespace prefix stores this value storage mechanism.
|
||
|
||
2.3.1. Configuring the value format for a Namespace
|
||
|
||
If one has the Namespace "/Organization1/BU1/ip" and want to store
|
||
those IP addresses in SHA256, it will be configured like this: The
|
||
Namespace is kept but prefixed by "_config" and has a json object
|
||
about value format set. "/_config/Organization1/BU1/ip"
|
||
|
||
{
|
||
"value_format":"SHA256"
|
||
}
|
||
|
||
Where "value_format" is either: "SHA256", "RAW" or "BASE64URL".
|
||
|
||
2.4. Bulk
|
||
|
||
When data must be sent and received in large amounts, it is
|
||
preferable to embed in JSON all the objects at once. As such, for
|
||
reading and writing, the format is the following:
|
||
|
||
{
|
||
"items": [
|
||
{ "/your/namespace": "127.0.0.1" },
|
||
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" }
|
||
]
|
||
}
|
||
|
||
Which will either store or retrieve the wanted data.
|
||
|
||
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 5]
|
||
|
||
Internet-Draft SightingDB query format November 2019
|
||
|
||
|
||
2.4.1. Response
|
||
|
||
The response when retrieving sightings also has the list of items, in
|
||
order, one per line of the results:
|
||
|
||
{
|
||
"items": [
|
||
{ "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 },
|
||
{ "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 }
|
||
]
|
||
}
|
||
|
||
3. Security Considerations
|
||
|
||
While this document solely focuses on the format, the reference
|
||
implementation is SightingDB. The authentication, the data access is
|
||
not handled by SightingDB. It is possible a value can leak if the
|
||
access is too permissive.
|
||
|
||
Even a Hashed value can be discovered, as re-hashing known values
|
||
would match.
|
||
|
||
4. Acknowledgements
|
||
|
||
The author wish to thank all the MISP community who are supporting
|
||
the creation of open standards in threat intelligence sharing. As
|
||
well as amazing feedback gathered during the MISP Summit 2019 in
|
||
Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody.
|
||
|
||
5. Normative References
|
||
|
||
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
|
||
Requirement Levels", BCP 14, RFC 2119,
|
||
DOI 10.17487/RFC2119, March 1997,
|
||
<https://www.rfc-editor.org/info/rfc2119>.
|
||
|
||
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
|
||
Interchange Format", STD 90, RFC 8259,
|
||
DOI 10.17487/RFC8259, December 2017,
|
||
<https://www.rfc-editor.org/info/rfc8259>.
|
||
|
||
Author's Address
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 6]
|
||
|
||
Internet-Draft SightingDB query format November 2019
|
||
|
||
|
||
Sebastien Tricaud
|
||
Devo Inc.
|
||
150 Cambridgepark Drive
|
||
Cambridge, MA 02140
|
||
USA
|
||
|
||
Phone: +1 866-221-2254
|
||
Email: sebastien.tricaud@devo.com
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
Tricaud Expires May 6, 2020 [Page 7]
|