From 9e98429bcd3459b78f4e3850bbcd6db0e70370a7 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Wed, 6 Nov 2019 22:59:52 -0800 Subject: [PATCH 1/3] Remove frequency definition as: 1) It is far from clear 2) The algorithm used to create that number must be written 3) Confusion must not happen in an RFC --- sightingdb-format/raw.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index b8814b4..b43844b 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -49,7 +49,7 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. ## Overview -The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl, frequency and manifold. +The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl and manifold. ### Attribute Storage @@ -62,10 +62,15 @@ A Namespace with multiple levels **MUST** be separated with the slash '/' charac A Namespace starting with the underscore '_' character means it is private and internal to SightingDB. There are all reserved for the engine and **MUST** NOT be used. Reserved namespaces are: + _expired/: Which contains all the attributes that expired, preserving the origin namespace + _shadow/: When a value is searched and does not exists, it is stored there + _stats: Statistics + _config: Configuration + _all: All the Attributes in one place, used to retrieve the 'manifold' property. The Attribute Key MUST always be the last part of the Namespace. @@ -73,6 +78,7 @@ The Attribute Key MUST always be the last part of the Namespace. #### Sample Namespaces /Organization1/service/ipv4: Store values for ipv4 keys in /Organization1/service + /everything/domain: Store domains in /everything ### Attribute fields @@ -107,10 +113,6 @@ When an Attribute has this field set to 0, it means it is not set to expired. Th When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time. -#### frequency - -Frequency is the number of time an Attribute is seen in average per day. As this field can introduced latence, its implementation is **OPTIONAL**. - #### manifold When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter. @@ -125,7 +127,6 @@ When a given Attribute Value is stored in different namespaces, the manifold fie "count":578391, "tags":"", "ttl":0, - "frequency":1185, "manifold": 17 } ~~~~ From 6187769b4d4b8a3b3b0ca3f33fd7b4549f230265 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Wed, 6 Nov 2019 23:11:15 -0800 Subject: [PATCH 2/3] Added bulk format --- sightingdb-format/raw.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index b43844b..305ad23 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -131,17 +131,19 @@ When a given Attribute Value is stored in different namespaces, the manifold fie } ~~~~ -# Value +## Value The value submitted can be in multiple format according to the use-case. Any implementation **MUST** offer three alternatives: 1) Raw value: where nothing is encoded and the value is stored AS IS, such as show in the example above with the One Attribute in JSON. + 2) SHA256: which prevents from seeing content (see Security Considerations), has a fixed size and is convenient for most requirements + 3) Base64 URL: Where the specification of Base64 is followed, except the characters conflicting with an URL argument are replaced The value is configured as part of the Namespace. The private "_config" Namespace prefix stores this value storage mechanism. -## Configuring the value format for a Namespace +### Configuring the value format for a Namespace If one has the Namespace "/Organization1/BU1/ip" and want to store those IP addresses in SHA256, it will be configured like this: The Namespace is kept but prefixed by "_config" and has a json object about value format set. @@ -155,6 +157,34 @@ The Namespace is kept but prefixed by "_config" and has a json object about valu Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". +## Bulk + +When data must be sent and received in large amounts, it is preferable to embed in JSON all the objects at once. As such, for reading +and writing, the format is the following: + +~~~~ +{ + "items": [ + { "/your/namespace": "127.0.0.1" }, + { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" } + ] +} +~~~~ + +Which will either store or retrieve the wanted data. + +### Response + +The response when retrieving sightings also has the list of items, in order, one per line of the results: +~~~~ +{ + "items": [ + { "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 }, + { "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 } + ] +} +~~~~ + # Security Considerations While this document solely focuses on the format, the reference implementation is SightingDB. The authentication, the data access is not handled by SightingDB. From 51c1230cd2cbd1c6cdf1532cfa4ab3df97418508 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Wed, 6 Nov 2019 23:13:23 -0800 Subject: [PATCH 3/3] Change the generated document --- sightingdb-format/raw.md.txt | 150 ++++++++++++++++++++++++----------- 1 file changed, 103 insertions(+), 47 deletions(-) diff --git a/sightingdb-format/raw.md.txt b/sightingdb-format/raw.md.txt index 4f90776..5d88547 100644 --- a/sightingdb-format/raw.md.txt +++ b/sightingdb-format/raw.md.txt @@ -68,11 +68,13 @@ Table of Contents 2.1.2. Namespace . . . . . . . . . . . . . . . . . . . . . . 3 2.1.3. Attribute fields . . . . . . . . . . . . . . . . . . 3 2.2. SightingDB Format - One Attribute . . . . . . . . . . . . 4 - 3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 - 3.1. Configuring the value format for a Namespace . . . . . . 5 - 4. Security Considerations . . . . . . . . . . . . . . . . . . . 5 - 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 - 6. Normative References . . . . . . . . . . . . . . . . . . . . 6 + 2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.3.1. Configuring the value format for a Namespace . . . . 5 + 2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.4.1. Response . . . . . . . . . . . . . . . . . . . . . . 6 + 3. Security Considerations . . . . . . . . . . . . . . . . . . . 6 + 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6 + 5. Normative References . . . . . . . . . . . . . . . . . . . . 6 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6 1. Introduction @@ -96,8 +98,7 @@ Table of Contents The SightingDB format is in JSON [RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following - fields: value, first_seen, last_seen, count, tags, ttl, frequency and - manifold. + fields: value, first_seen, last_seen, count, tags, ttl and manifold. 2.1.1. Attribute Storage @@ -108,7 +109,6 @@ Table of Contents - Tricaud Expires May 6, 2020 [Page 2] Internet-Draft SightingDB query format November 2019 @@ -124,19 +124,29 @@ Internet-Draft SightingDB query format November 2019 private and internal to SightingDB. There are all reserved for the engine and MUST NOT be used. - Reserved namespaces are: _expired/: Which contains all the attributes - that expired, preserving the origin namespace _shadow/: When a value - is searched and does not exists, it is stored there _stats: - Statistics _config: Configuration _all: All the Attributes in one - place, used to retrieve the 'manifold' property. + Reserved namespaces are: + + _expired/: Which contains all the attributes that expired, preserving + the origin namespace + + _shadow/: When a value is searched and does not exists, it is stored + there + + _stats: Statistics + + _config: Configuration + + _all: All the Attributes in one place, used to retrieve the + 'manifold' property. The Attribute Key MUST always be the last part of the Namespace. 2.1.2.1. Sample Namespaces /Organization1/service/ipv4: Store values for ipv4 keys in - /Organization1/service /everything/domain: Store domains in - /everything + /Organization1/service + + /everything/domain: Store domains in /everything 2.1.3. Attribute fields @@ -151,16 +161,6 @@ Internet-Draft SightingDB query format November 2019 The value implementation MUST offer at least: 1) Raw value 2) Base64 URL Encoded 3) SHA256 Hash -2.1.3.2. first_seen - - Time in UTC of the first time this value was captured - -2.1.3.3. last_seen - - Time in UTC of the last time this value was captured - - - @@ -170,6 +170,14 @@ Tricaud Expires May 6, 2020 [Page 3] Internet-Draft SightingDB query format November 2019 +2.1.3.2. first_seen + + Time in UTC of the first time this value was captured + +2.1.3.3. last_seen + + Time in UTC of the last time this value was captured + 2.1.3.4. count How many time this value was written @@ -191,13 +199,7 @@ Internet-Draft SightingDB query format November 2019 When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time. -2.1.3.7. frequency - - Frequency is the number of time an Attribute is seen in average per - day. As this field can introduced latence, its implementation is - OPTIONAL. - -2.1.3.8. manifold +2.1.3.7. manifold When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many @@ -212,7 +214,6 @@ Internet-Draft SightingDB query format November 2019 "count":578391, "tags":"", "ttl":0, - "frequency":1185, "manifold": 17 } @@ -220,13 +221,12 @@ Internet-Draft SightingDB query format November 2019 - Tricaud Expires May 6, 2020 [Page 4] Internet-Draft SightingDB query format November 2019 -3. Value +2.3. Value The value submitted can be in multiple format according to the use- case. Any implementation MUST offer three alternatives: @@ -245,7 +245,7 @@ Internet-Draft SightingDB query format November 2019 The value is configured as part of the Namespace. The private "_config" Namespace prefix stores this value storage mechanism. -3.1. Configuring the value format for a Namespace +2.3.1. Configuring the value format for a Namespace If one has the Namespace "/Organization1/BU1/ip" and want to store those IP addresses in SHA256, it will be configured like this: The @@ -258,7 +258,43 @@ Internet-Draft SightingDB query format November 2019 Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". -4. Security Considerations +2.4. Bulk + + When data must be sent and received in large amounts, it is + preferable to embed in JSON all the objects at once. As such, for + reading and writing, the format is the following: + +{ + "items": [ + { "/your/namespace": "127.0.0.1" }, + { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" } + ] +} + + Which will either store or retrieve the wanted data. + + + + + +Tricaud Expires May 6, 2020 [Page 5] + +Internet-Draft SightingDB query format November 2019 + + +2.4.1. Response + + The response when retrieving sightings also has the list of items, in + order, one per line of the results: + +{ + "items": [ + { "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 }, + { "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 } + ] +} + +3. Security Considerations While this document solely focuses on the format, the reference implementation is SightingDB. The authentication, the data access is @@ -268,21 +304,14 @@ Internet-Draft SightingDB query format November 2019 Even a Hashed value can be discovered, as re-hashing known values would match. -5. Acknowledgements +4. Acknowledgements The author wish to thank all the MISP community who are supporting the creation of open standards in threat intelligence sharing. As well as amazing feedback gathered during the MISP Summit 2019 in Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody. - - -Tricaud Expires May 6, 2020 [Page 5] - -Internet-Draft SightingDB query format November 2019 - - -6. Normative References +5. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, @@ -296,6 +325,19 @@ Internet-Draft SightingDB query format November 2019 Author's Address + + + + + + + + +Tricaud Expires May 6, 2020 [Page 6] + +Internet-Draft SightingDB query format November 2019 + + Sebastien Tricaud Devo Inc. 150 Cambridgepark Drive @@ -333,4 +375,18 @@ Author's Address -Tricaud Expires May 6, 2020 [Page 6] + + + + + + + + + + + + + + +Tricaud Expires May 6, 2020 [Page 7]