From c49f77fed994b0b01811bdbed82e4ba1a5d45155 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Wed, 6 Nov 2019 11:31:34 +0100 Subject: [PATCH] chg: [sightingDB] ascii output added --- sightingdb-format/raw.md.txt | 336 +++++++++++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 sightingdb-format/raw.md.txt diff --git a/sightingdb-format/raw.md.txt b/sightingdb-format/raw.md.txt new file mode 100644 index 0000000..4f90776 --- /dev/null +++ b/sightingdb-format/raw.md.txt @@ -0,0 +1,336 @@ + + + + +Network Working Group S. Tricaud +Internet-Draft Devo Inc. +Intended status: Informational November 3, 2019 +Expires: May 6, 2020 + + + SightingDB query format + draft-tricaud-sightingdb-format + +Abstract + + This document describes the format used by SightingDB to give + automated context to a given Attribute by counting occurrences and + tracking times of observability. SightingDB was designed to provide + to MISP a Scalable and Fast way to store and retrieve Attributes. + +Status of This Memo + + This Internet-Draft is submitted in full conformance with the + provisions of BCP 78 and BCP 79. + + Internet-Drafts are working documents of the Internet Engineering + Task Force (IETF). Note that other groups may also distribute + working documents as Internet-Drafts. The list of current Internet- + Drafts is at https://datatracker.ietf.org/drafts/current/. + + Internet-Drafts are draft documents valid for a maximum of six months + and may be updated, replaced, or obsoleted by other documents at any + time. It is inappropriate to use Internet-Drafts as reference + material or to cite them other than as "work in progress." + + This Internet-Draft will expire on May 6, 2020. + +Copyright Notice + + Copyright (c) 2019 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (https://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + + + + +Tricaud Expires May 6, 2020 [Page 1] + +Internet-Draft SightingDB query format November 2019 + + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 + 1.1. Conventions and Terminology . . . . . . . . . . . . . . . 2 + 2. Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 + 2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 2 + 2.1.1. Attribute Storage . . . . . . . . . . . . . . . . . . 2 + 2.1.2. Namespace . . . . . . . . . . . . . . . . . . . . . . 3 + 2.1.3. Attribute fields . . . . . . . . . . . . . . . . . . 3 + 2.2. SightingDB Format - One Attribute . . . . . . . . . . . . 4 + 3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 3.1. Configuring the value format for a Namespace . . . . . . 5 + 4. Security Considerations . . . . . . . . . . . . . . . . . . . 5 + 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 + 6. Normative References . . . . . . . . . . . . . . . . . . . . 6 + Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6 + +1. Introduction + + Adding context to any Attribute is the key that makes it useful. + While there exist numerous ways of doing it, SightingDB does it by + just counting. Whenever somebody retrieves an Attribute, this + counting is provided, allowing anyone to understand whenever + something was observed few or many times. + +1.1. Conventions and Terminology + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119 [RFC2119]. + +2. Format + +2.1. Overview + + The SightingDB format is in JSON [RFC8259] format and used to query a + SightingDB compatible connector. In SightingDB, a Sighting Object is + composed of a single JSON object. This object contains the following + fields: value, first_seen, last_seen, count, tags, ttl, frequency and + manifold. + +2.1.1. Attribute Storage + + The fields described previously describe an Attribute and all the + required characteristics. However they are stored in a Namespace. A + Namespace is similar to a path in a file-system where the same file + can be stored in multiple places. + + + + +Tricaud Expires May 6, 2020 [Page 2] + +Internet-Draft SightingDB query format November 2019 + + +2.1.2. Namespace + + A Namespace with multiple levels MUST be separated with the slash '/' + character. There is no specification on how they are structured, + since it depends on the use cases. + + A Namespace starting with the underscore '_' character means it is + private and internal to SightingDB. There are all reserved for the + engine and MUST NOT be used. + + Reserved namespaces are: _expired/: Which contains all the attributes + that expired, preserving the origin namespace _shadow/: When a value + is searched and does not exists, it is stored there _stats: + Statistics _config: Configuration _all: All the Attributes in one + place, used to retrieve the 'manifold' property. + + The Attribute Key MUST always be the last part of the Namespace. + +2.1.2.1. Sample Namespaces + + /Organization1/service/ipv4: Store values for ipv4 keys in + /Organization1/service /everything/domain: Store domains in + /everything + +2.1.3. Attribute fields + +2.1.3.1. value + + The attribute value, used to store and retrieve information about an + attribute. Note that value is not returned back in the JSON object, + since it is queried, it is known. The Value is described in a + section below, as it is very specific and can be either "as is", a + hash, encoded in base64 or any other convenient mechanism. + + The value implementation MUST offer at least: 1) Raw value 2) Base64 + URL Encoded 3) SHA256 Hash + +2.1.3.2. first_seen + + Time in UTC of the first time this value was captured + +2.1.3.3. last_seen + + Time in UTC of the last time this value was captured + + + + + + + +Tricaud Expires May 6, 2020 [Page 3] + +Internet-Draft SightingDB query format November 2019 + + +2.1.3.4. count + + How many time this value was written + +2.1.3.5. tags + + Tags follow how they are defined in MISP using the MISP Taxonomy. + Each Tag is separated with the ';' character. + +2.1.3.6. ttl + + Time To Live, represents the expiration in seconds since the time the + Attribute was created. Once it has expired, it moves in the private + Namespace _expired. + + When an Attribute has this field set to 0, it means it is not set to + expired. This is the default behavior. + + When an Attribute has this field set to a number greater than 0, the + expiration status is computed only at retrieval time. + +2.1.3.7. frequency + + Frequency is the number of time an Attribute is seen in average per + day. As this field can introduced latence, its implementation is + OPTIONAL. + +2.1.3.8. manifold + + When a given Attribute Value is stored in different namespaces, the + manifold field keeps track of them so it returns in how many + different places this attributes exists. This is a simple counter. + +2.2. SightingDB Format - One Attribute + + { + "value":"127.0.0.1", + "first_seen":1530394819, + "last_seen":1572933618, + "count":578391, + "tags":"", + "ttl":0, + "frequency":1185, + "manifold": 17 + } + + + + + + +Tricaud Expires May 6, 2020 [Page 4] + +Internet-Draft SightingDB query format November 2019 + + +3. Value + + The value submitted can be in multiple format according to the use- + case. Any implementation MUST offer three alternatives: + + 1. Raw value: where nothing is encoded and the value is stored AS + IS, such as show in the example above with the One Attribute in + JSON. + + 2. SHA256: which prevents from seeing content (see Security + Considerations), has a fixed size and is convenient for most + requirements + + 3. Base64 URL: Where the specification of Base64 is followed, except + the characters conflicting with an URL argument are replaced + + The value is configured as part of the Namespace. The private + "_config" Namespace prefix stores this value storage mechanism. + +3.1. Configuring the value format for a Namespace + + If one has the Namespace "/Organization1/BU1/ip" and want to store + those IP addresses in SHA256, it will be configured like this: The + Namespace is kept but prefixed by "_config" and has a json object + about value format set. "/_config/Organization1/BU1/ip" + + { + "value_format":"SHA256" + } + + Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". + +4. Security Considerations + + While this document solely focuses on the format, the reference + implementation is SightingDB. The authentication, the data access is + not handled by SightingDB. It is possible a value can leak if the + access is too permissive. + + Even a Hashed value can be discovered, as re-hashing known values + would match. + +5. Acknowledgements + + The author wish to thank all the MISP community who are supporting + the creation of open standards in threat intelligence sharing. As + well as amazing feedback gathered during the MISP Summit 2019 in + Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody. + + + +Tricaud Expires May 6, 2020 [Page 5] + +Internet-Draft SightingDB query format November 2019 + + +6. Normative References + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, + DOI 10.17487/RFC2119, March 1997, + . + + [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", STD 90, RFC 8259, + DOI 10.17487/RFC8259, December 2017, + . + +Author's Address + + Sebastien Tricaud + Devo Inc. + 150 Cambridgepark Drive + Cambridge, MA 02140 + USA + + Phone: +1 866-221-2254 + Email: sebastien.tricaud@devo.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Tricaud Expires May 6, 2020 [Page 6]