MISP
/
misp-rfc
mirror of https://github.com/MISP/misp-rfc
2
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'master' of github.com:MISP/misp-rfc

pull/36/head
Alexandre Dulaunoy 2020-05-27 10:06:04 +02:00
parent 32bb51a185 47433897a3
commit 3917ee1979
No known key found for this signature in database
GPG Key ID: 09E2CD4944E6CBCD
2 changed files with 102 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
sightingdb-format/raw.md
View File

@ -6,7 +6,7 @@ docName = "draft-tricaud-sightingdb-format"
ipr= "trust200902" ipr= "trust200902"
area = "Security" area = "Security"
date = 2019-11-03T00:00:00Z date = 2020-04-13T00:00:00Z
[[author]] [[author]]
initials="S." initials="S."
@ -49,7 +49,7 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119].
## Overview ## Overview
The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl and manifold. The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl and consensus.
### Attribute Storage ### Attribute Storage
@ -67,11 +67,9 @@ _expired/<namespace>: Which contains all the attributes that expired, preserving
_shadow/<namespace>: When a value is searched and does not exists, it is stored there _shadow/<namespace>: When a value is searched and does not exists, it is stored there
_stats: Statistics
_config: Configuration _config: Configuration
_all: All the Attributes in one place, used to retrieve the 'manifold' property. _all: All the Attributes in one place, used to retrieve the 'consensus' property.
The Attribute Key MUST always be the last part of the Namespace. The Attribute Key MUST always be the last part of the Namespace.
@ -113,9 +111,9 @@ When an Attribute has this field set to 0, it means it is not set to expired. Th
When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time. When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time.
#### manifold #### consensus
When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter. When a given Attribute Value is stored in different namespaces, the consensus field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter.
## SightingDB Format - One Attribute ## SightingDB Format - One Attribute
@ -127,7 +125,7 @@ When a given Attribute Value is stored in different namespaces, the manifold fie
"count":578391, "count":578391,
"tags":"", "tags":"",
"ttl":0, "ttl":0,
"manifold": 17 "consensus": 17
} }
~~~~ ~~~~
@ -159,14 +157,36 @@ Where "value_format" is either: "SHA256", "RAW" or "BASE64URL".
## Bulk ## Bulk
When data must be sent and received in large amounts, it is preferable to embed in JSON all the objects at once. As such, for reading When data must be sent and received in large amounts, it is preferable to embed in JSON all the objects at once. As such, for reading and writing, the format is the following:
and writing, the format is the following:
~~~~
{
"items": [
{ "<namespace>": "<value>" },
{ "<namespace>": "<value>", "timestamp": <epoch> }
]
}
~~~~
Where:
namespace: is the wanted namespace where to store the value
value: the value one want to track
timestamp: **OPTIONAL** epoch timestamp to set the value at.
The timestamp is how one can use SightingDB and use old datasets where the first seen and last seen is not relative to "right now".
### Request
A Proper request with two items is made like this:
~~~~ ~~~~
{ {
"items": [ "items": [
{ "/your/namespace": "127.0.0.1" }, { "/your/namespace": "127.0.0.1" },
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" } { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db", "timestamp": 1586825229 }
] ]
} }
~~~~ ~~~~
@ -179,8 +199,8 @@ The response when retrieving sightings also has the list of items, in order, one
~~~~ ~~~~
{ {
"items": [ "items": [
{ "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 }, {"value": "Octave_Hergebel", "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "consensus": 1},
{ "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 } {"value": "127.0.0.1", "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "consensus": 3}
] ]
} }
~~~~ ~~~~

138
sightingdb-format/raw.md.txt
View File

@ -4,8 +4,8 @@
Network Working Group S. Tricaud Network Working Group S. Tricaud
Internet-Draft Devo Inc. Internet-Draft Devo Inc.
Intended status: Informational November 3, 2019 Intended status: Informational April 13, 2020
Expires: May 6, 2020 Expires: October 15, 2020
SightingDB query format SightingDB query format
@ -33,11 +33,11 @@ Status of This Memo
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 6, 2020. This Internet-Draft will expire on October 15, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
@ -53,9 +53,9 @@ Copyright Notice
Tricaud Expires May 6, 2020 [Page 1] Tricaud Expires October 15, 2020 [Page 1]
Internet-Draft SightingDB query format November 2019 Internet-Draft SightingDB query format April 2020
Table of Contents Table of Contents
@ -71,11 +71,12 @@ Table of Contents
2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.1. Configuring the value format for a Namespace . . . . 5 2.3.1. Configuring the value format for a Namespace . . . . 5
2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4.1. Response . . . . . . . . . . . . . . . . . . . . . . 6 2.4.1. Request . . . . . . . . . . . . . . . . . . . . . . . 6
2.4.2. Response . . . . . . . . . . . . . . . . . . . . . . 6
3. Security Considerations . . . . . . . . . . . . . . . . . . . 6 3. Security Considerations . . . . . . . . . . . . . . . . . . . 6
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6
5. Normative References . . . . . . . . . . . . . . . . . . . . 6 5. Normative References . . . . . . . . . . . . . . . . . . . . 7
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 7
1. Introduction 1. Introduction
@ -98,22 +99,24 @@ Table of Contents
The SightingDB format is in JSON [RFC8259] format and used to query a The SightingDB format is in JSON [RFC8259] format and used to query a
SightingDB compatible connector. In SightingDB, a Sighting Object is SightingDB compatible connector. In SightingDB, a Sighting Object is
composed of a single JSON object. This object contains the following composed of a single JSON object. This object contains the following
fields: value, first_seen, last_seen, count, tags, ttl and manifold. fields: value, first_seen, last_seen, count, tags, ttl and consensus.
2.1.1. Attribute Storage 2.1.1. Attribute Storage
The fields described previously describe an Attribute and all the The fields described previously describe an Attribute and all the
required characteristics. However they are stored in a Namespace. A required characteristics. However they are stored in a Namespace. A
Tricaud Expires October 15, 2020 [Page 2]
Internet-Draft SightingDB query format April 2020
Namespace is similar to a path in a file-system where the same file Namespace is similar to a path in a file-system where the same file
can be stored in multiple places. can be stored in multiple places.
Tricaud Expires May 6, 2020 [Page 2]
Internet-Draft SightingDB query format November 2019
2.1.2. Namespace 2.1.2. Namespace
A Namespace with multiple levels MUST be separated with the slash '/' A Namespace with multiple levels MUST be separated with the slash '/'
@ -132,12 +135,10 @@ Internet-Draft SightingDB query format November 2019
_shadow/: When a value is searched and does not exists, it is stored _shadow/: When a value is searched and does not exists, it is stored
there there
_stats: Statistics
_config: Configuration _config: Configuration
_all: All the Attributes in one place, used to retrieve the _all: All the Attributes in one place, used to retrieve the
'manifold' property. 'consensus' property.
The Attribute Key MUST always be the last part of the Namespace. The Attribute Key MUST always be the last part of the Namespace.
@ -164,10 +165,9 @@ Internet-Draft SightingDB query format November 2019
Tricaud Expires October 15, 2020 [Page 3]
Tricaud Expires May 6, 2020 [Page 3] Internet-Draft SightingDB query format April 2020
Internet-Draft SightingDB query format November 2019
2.1.3.2. first_seen 2.1.3.2. first_seen
@ -199,10 +199,10 @@ Internet-Draft SightingDB query format November 2019
When an Attribute has this field set to a number greater than 0, the When an Attribute has this field set to a number greater than 0, the
expiration status is computed only at retrieval time. expiration status is computed only at retrieval time.
2.1.3.7. manifold 2.1.3.7. consensus
When a given Attribute Value is stored in different namespaces, the When a given Attribute Value is stored in different namespaces, the
manifold field keeps track of them so it returns in how many consensus field keeps track of them so it returns in how many
different places this attributes exists. This is a simple counter. different places this attributes exists. This is a simple counter.
2.2. SightingDB Format - One Attribute 2.2. SightingDB Format - One Attribute
@ -214,16 +214,16 @@ Internet-Draft SightingDB query format November 2019
"count":578391, "count":578391,
"tags":"", "tags":"",
"ttl":0, "ttl":0,
"manifold": 17 "consensus": 17
} }
Tricaud Expires May 6, 2020 [Page 4] Tricaud Expires October 15, 2020 [Page 4]
Internet-Draft SightingDB query format November 2019 Internet-Draft SightingDB query format April 2020
2.3. Value 2.3. Value
@ -264,33 +264,53 @@ Internet-Draft SightingDB query format November 2019
preferable to embed in JSON all the objects at once. As such, for preferable to embed in JSON all the objects at once. As such, for
reading and writing, the format is the following: reading and writing, the format is the following:
{
"items": [
{ "<namespace>": "<value>" },
{ "<namespace>": "<value>", "timestamp": <epoch> }
]
}
Where:
namespace: is the wanted namespace where to store the value
Tricaud Expires October 15, 2020 [Page 5]
Internet-Draft SightingDB query format April 2020
value: the value one want to track
timestamp: OPTIONAL epoch timestamp to set the value at.
The timestamp is how one can use SightingDB and use old datasets
where the first seen and last seen is not relative to "right now".
2.4.1. Request
A Proper request with two items is made like this:
{ {
"items": [ "items": [
{ "/your/namespace": "127.0.0.1" }, { "/your/namespace": "127.0.0.1" },
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" } { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db", "timestamp": 1586825229 }
] ]
} }
Which will either store or retrieve the wanted data. Which will either store or retrieve the wanted data.
2.4.2. Response
Tricaud Expires May 6, 2020 [Page 5]
Internet-Draft SightingDB query format November 2019
2.4.1. Response
The response when retrieving sightings also has the list of items, in The response when retrieving sightings also has the list of items, in
order, one per line of the results: order, one per line of the results:
{ {
"items": [ "items": [
{ "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 }, {"value": "Octave_Hergebel", "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "consensus": 1},
{ "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 } {"value": "127.0.0.1", "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "consensus": 3}
] ]
} }
@ -311,6 +331,13 @@ Internet-Draft SightingDB query format November 2019
well as amazing feedback gathered during the MISP Summit 2019 in well as amazing feedback gathered during the MISP Summit 2019 in
Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody. Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody.
Tricaud Expires October 15, 2020 [Page 6]
Internet-Draft SightingDB query format April 2020
5. Normative References 5. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
@ -325,19 +352,6 @@ Internet-Draft SightingDB query format November 2019
Author's Address Author's Address
Tricaud Expires May 6, 2020 [Page 6]
Internet-Draft SightingDB query format November 2019
Sebastien Tricaud Sebastien Tricaud
Devo Inc. Devo Inc.
150 Cambridgepark Drive 150 Cambridgepark Drive
@ -375,18 +389,4 @@ Internet-Draft SightingDB query format November 2019
Tricaud Expires October 15, 2020 [Page 7]
Tricaud Expires May 6, 2020 [Page 7]