MISP
/
misp-rfc
mirror of https://github.com/MISP/misp-rfc
2
0
Fork 0
Code Issues Releases Wiki Activity

Merge pull request #35 from stricaud/master 

Improvements with implementation
pull/36/head
Alexandre Dulaunoy 2020-04-15 11:23:31 +02:00 committed by GitHub
parent 4505f65523 6f5fb3b5ae
commit 47433897a3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 102 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
sightingdb-format/raw.md
View File

@ -6,7 +6,7 @@ docName = "draft-tricaud-sightingdb-format"
ipr= "trust200902"
area = "Security"
date = 2019-11-03T00:00:00Z
date = 2020-04-13T00:00:00Z
[[author]]
initials="S."
@ -49,7 +49,7 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119].
## Overview
The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl and manifold.
The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl and consensus.
### Attribute Storage
@ -67,11 +67,9 @@ _expired/<namespace>: Which contains all the attributes that expired, preserving
_shadow/<namespace>: When a value is searched and does not exists, it is stored there
_stats: Statistics
_config: Configuration
_all: All the Attributes in one place, used to retrieve the 'manifold' property.
_all: All the Attributes in one place, used to retrieve the 'consensus' property.
The Attribute Key MUST always be the last part of the Namespace.
@ -113,9 +111,9 @@ When an Attribute has this field set to 0, it means it is not set to expired. Th
When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time.
#### manifold
#### consensus
When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter.
When a given Attribute Value is stored in different namespaces, the consensus field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter.
## SightingDB Format - One Attribute
@ -127,7 +125,7 @@ When a given Attribute Value is stored in different namespaces, the manifold fie
"count":578391,
"tags":"",
"ttl":0,
"manifold": 17
"consensus": 17
}
~~~~
@ -159,14 +157,36 @@ Where "value_format" is either: "SHA256", "RAW" or "BASE64URL".
## Bulk
When data must be sent and received in large amounts, it is preferable to embed in JSON all the objects at once. As such, for reading
and writing, the format is the following:
When data must be sent and received in large amounts, it is preferable to embed in JSON all the objects at once. As such, for reading and writing, the format is the following:
~~~~
{
"items": [
{ "<namespace>": "<value>" },
{ "<namespace>": "<value>", "timestamp": <epoch> }
]
}
~~~~
Where:
namespace: is the wanted namespace where to store the value
value: the value one want to track
timestamp: **OPTIONAL** epoch timestamp to set the value at.
The timestamp is how one can use SightingDB and use old datasets where the first seen and last seen is not relative to "right now".
### Request
A Proper request with two items is made like this:
~~~~
{
"items": [
{ "/your/namespace": "127.0.0.1" },
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" }
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db", "timestamp": 1586825229 }
]
}
~~~~
@ -179,8 +199,8 @@ The response when retrieving sightings also has the list of items, in order, one
~~~~
{
"items": [
{ "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 },
{ "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 }
{"value": "Octave_Hergebel", "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "consensus": 1},
{"value": "127.0.0.1", "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "consensus": 3}
]
}
~~~~

138
sightingdb-format/raw.md.txt
View File

@ -4,8 +4,8 @@
Network Working Group S. Tricaud
Internet-Draft Devo Inc.
Intended status: Informational November 3, 2019
Expires: May 6, 2020
Intended status: Informational April 13, 2020
Expires: October 15, 2020
SightingDB query format
@ -33,11 +33,11 @@ Status of This Memo
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 6, 2020.
This Internet-Draft will expire on October 15, 2020.
Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
@ -53,9 +53,9 @@ Copyright Notice
Tricaud Expires May 6, 2020 [Page 1]
Tricaud Expires October 15, 2020 [Page 1]
Internet-Draft SightingDB query format November 2019
Internet-Draft SightingDB query format April 2020
Table of Contents
@ -71,11 +71,12 @@ Table of Contents
2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.1. Configuring the value format for a Namespace . . . . 5
2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4.1. Response . . . . . . . . . . . . . . . . . . . . . . 6
2.4.1. Request . . . . . . . . . . . . . . . . . . . . . . . 6
2.4.2. Response . . . . . . . . . . . . . . . . . . . . . . 6
3. Security Considerations . . . . . . . . . . . . . . . . . . . 6
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6
5. Normative References . . . . . . . . . . . . . . . . . . . . 6
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6
5. Normative References . . . . . . . . . . . . . . . . . . . . 7
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 7
1. Introduction
@ -98,22 +99,24 @@ Table of Contents
The SightingDB format is in JSON [RFC8259] format and used to query a
SightingDB compatible connector. In SightingDB, a Sighting Object is
composed of a single JSON object. This object contains the following
fields: value, first_seen, last_seen, count, tags, ttl and manifold.
fields: value, first_seen, last_seen, count, tags, ttl and consensus.
2.1.1. Attribute Storage
The fields described previously describe an Attribute and all the
required characteristics. However they are stored in a Namespace. A
Tricaud Expires October 15, 2020 [Page 2]
Internet-Draft SightingDB query format April 2020
Namespace is similar to a path in a file-system where the same file
can be stored in multiple places.
Tricaud Expires May 6, 2020 [Page 2]
Internet-Draft SightingDB query format November 2019
2.1.2. Namespace
A Namespace with multiple levels MUST be separated with the slash '/'
@ -132,12 +135,10 @@ Internet-Draft SightingDB query format November 2019
_shadow/: When a value is searched and does not exists, it is stored
there
_stats: Statistics
_config: Configuration
_all: All the Attributes in one place, used to retrieve the
'manifold' property.
'consensus' property.
The Attribute Key MUST always be the last part of the Namespace.
@ -164,10 +165,9 @@ Internet-Draft SightingDB query format November 2019
Tricaud Expires May 6, 2020 [Page 3]
Tricaud Expires October 15, 2020 [Page 3]
Internet-Draft SightingDB query format November 2019
Internet-Draft SightingDB query format April 2020
2.1.3.2. first_seen
@ -199,10 +199,10 @@ Internet-Draft SightingDB query format November 2019
When an Attribute has this field set to a number greater than 0, the
expiration status is computed only at retrieval time.
2.1.3.7. manifold
2.1.3.7. consensus
When a given Attribute Value is stored in different namespaces, the
manifold field keeps track of them so it returns in how many
consensus field keeps track of them so it returns in how many
different places this attributes exists. This is a simple counter.
2.2. SightingDB Format - One Attribute
@ -214,16 +214,16 @@ Internet-Draft SightingDB query format November 2019
"count":578391,
"tags":"",
"ttl":0,
"manifold": 17
"consensus": 17
}
Tricaud Expires May 6, 2020 [Page 4]
Tricaud Expires October 15, 2020 [Page 4]
Internet-Draft SightingDB query format November 2019
Internet-Draft SightingDB query format April 2020
2.3. Value
@ -264,33 +264,53 @@ Internet-Draft SightingDB query format November 2019
preferable to embed in JSON all the objects at once. As such, for
reading and writing, the format is the following:
{
"items": [
{ "<namespace>": "<value>" },
{ "<namespace>": "<value>", "timestamp": <epoch> }
]
}
Where:
namespace: is the wanted namespace where to store the value
Tricaud Expires October 15, 2020 [Page 5]
Internet-Draft SightingDB query format April 2020
value: the value one want to track
timestamp: OPTIONAL epoch timestamp to set the value at.
The timestamp is how one can use SightingDB and use old datasets
where the first seen and last seen is not relative to "right now".
2.4.1. Request
A Proper request with two items is made like this:
{
"items": [
{ "/your/namespace": "127.0.0.1" },
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" }
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db", "timestamp": 1586825229 }
]
}
Which will either store or retrieve the wanted data.
Tricaud Expires May 6, 2020 [Page 5]
Internet-Draft SightingDB query format November 2019
2.4.1. Response
2.4.2. Response
The response when retrieving sightings also has the list of items, in
order, one per line of the results:
{
"items": [
{ "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 },
{ "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 }
{"value": "Octave_Hergebel", "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "consensus": 1},
{"value": "127.0.0.1", "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "consensus": 3}
]
}
@ -311,6 +331,13 @@ Internet-Draft SightingDB query format November 2019
well as amazing feedback gathered during the MISP Summit 2019 in
Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody.
Tricaud Expires October 15, 2020 [Page 6]
Internet-Draft SightingDB query format April 2020
5. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
@ -325,19 +352,6 @@ Internet-Draft SightingDB query format November 2019
Author's Address
Tricaud Expires May 6, 2020 [Page 6]
Internet-Draft SightingDB query format November 2019
Sebastien Tricaud
Devo Inc.
150 Cambridgepark Drive
@ -375,18 +389,4 @@ Internet-Draft SightingDB query format November 2019
Tricaud Expires May 6, 2020 [Page 7]
Tricaud Expires October 15, 2020 [Page 7]