misp-standard.org/rfc/sightingdb-format.txt

Network Working Group                                         S. Tricaud
Internet-Draft                                                 Devo Inc.
Expires: October 15, 2020                                 April 13, 2020


                        SightingDB query format

Abstract

   This document describes the format used by SightingDB to give
   automated context to a given Attribute by counting occurrences and
   tracking times of observability.  SightingDB was designed to provide
   to MISP a Scalable and Fast way to store and retrieve Attributes.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on October 15, 2020.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.






Tricaud                 Expires October 15, 2020                [Page 1]

Internet-Draft           SightingDB query format              April 2020


Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Conventions and Terminology . . . . . . . . . . . . . . .   2
   2.  Format  . . . . . . . . . . . . . . . . . . . . . . . . . . .   2
     2.1.  Overview  . . . . . . . . . . . . . . . . . . . . . . . .   2
       2.1.1.  Attribute Storage . . . . . . . . . . . . . . . . . .   2
       2.1.2.  Namespace . . . . . . . . . . . . . . . . . . . . . .   3
       2.1.3.  Attribute fields  . . . . . . . . . . . . . . . . . .   3
     2.2.  SightingDB Format - One Attribute . . . . . . . . . . . .   4
     2.3.  Value . . . . . . . . . . . . . . . . . . . . . . . . . .   5
       2.3.1.  Configuring the value format for a Namespace  . . . .   5
     2.4.  Bulk  . . . . . . . . . . . . . . . . . . . . . . . . . .   5
       2.4.1.  Request . . . . . . . . . . . . . . . . . . . . . . .   6
       2.4.2.  Response  . . . . . . . . . . . . . . . . . . . . . .   6
   3.  Security Considerations . . . . . . . . . . . . . . . . . . .   6
   4.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   6
   5.  Normative References  . . . . . . . . . . . . . . . . . . . .   7
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .   7

1.  Introduction

   Adding context to any Attribute is the key that makes it useful.
   While there exist numerous ways of doing it, SightingDB does it by
   just counting.  Whenever somebody retrieves an Attribute, this
   counting is provided, allowing anyone to understand whenever
   something was observed few or many times.

1.1.  Conventions and Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.  Format

2.1.  Overview

   The SightingDB format is in JSON [RFC8259] format and used to query a
   SightingDB compatible connector.  In SightingDB, a Sighting Object is
   composed of a single JSON object.  This object contains the following
   fields: value, first_seen, last_seen, count, tags, ttl and consensus.

2.1.1.  Attribute Storage

   The fields described previously describe an Attribute and all the
   required characteristics.  However they are stored in a Namespace.  A




Tricaud                 Expires October 15, 2020                [Page 2]

Internet-Draft           SightingDB query format              April 2020


   Namespace is similar to a path in a file-system where the same file
   can be stored in multiple places.

2.1.2.  Namespace

   A Namespace with multiple levels MUST be separated with the slash '/'
   character.  There is no specification on how they are structured,
   since it depends on the use cases.

   A Namespace starting with the underscore '_' character means it is
   private and internal to SightingDB.  There are all reserved for the
   engine and MUST NOT be used.

   Reserved namespaces are:

   _expired/<namespace>: Which contains all the attributes that expired,
   preserving the origin namespace

   _shadow/<namespace>: When a value is searched and does not exists, it
   is stored there

   _config: Configuration

   _all: All the Attributes in one place, used to retrieve the
   'consensus' property.

   The Attribute Key MUST always be the last part of the Namespace.

2.1.2.1.  Sample Namespaces

   /Organization1/service/ipv4: Store values for ipv4 keys in
   /Organization1/service

   /everything/domain: Store domains in /everything

2.1.3.  Attribute fields

2.1.3.1.  value

   The attribute value, used to store and retrieve information about an
   attribute.  Note that value is not returned back in the JSON object,
   since it is queried, it is known.  The Value is described in a
   section below, as it is very specific and can be either "as is", a
   hash, encoded in base64 or any other convenient mechanism.

   The value implementation MUST offer at least: 1) Raw value 2) Base64
   URL Encoded 3) SHA256 Hash




Tricaud                 Expires October 15, 2020                [Page 3]

Internet-Draft           SightingDB query format              April 2020


2.1.3.2.  first_seen

   Time in UTC of the first time this value was captured

2.1.3.3.  last_seen

   Time in UTC of the last time this value was captured

2.1.3.4.  count

   How many time this value was written

2.1.3.5.  tags

   Tags follow how they are defined in MISP using the MISP Taxonomy.
   Each Tag is separated with the ';' character.

2.1.3.6.  ttl

   Time To Live, represents the expiration in seconds since the time the
   Attribute was created.  Once it has expired, it moves in the private
   Namespace _expired.

   When an Attribute has this field set to 0, it means it is not set to
   expired.  This is the default behavior.

   When an Attribute has this field set to a number greater than 0, the
   expiration status is computed only at retrieval time.

2.1.3.7.  consensus

   When a given Attribute Value is stored in different namespaces, the
   consensus field keeps track of them so it returns in how many
   different places this attributes exists.  This is a simple counter.

2.2.  SightingDB Format - One Attribute

   {
     "value":"127.0.0.1",
     "first_seen":1530394819,
     "last_seen":1572933618,
     "count":578391,
     "tags":"",
     "ttl":0,
     "consensus": 17
   }





Tricaud                 Expires October 15, 2020                [Page 4]

Internet-Draft           SightingDB query format              April 2020


2.3.  Value

   The value submitted can be in multiple format according to the use-
   case.  Any implementation MUST offer three alternatives:

   1) Raw value: where nothing is encoded and the value is stored AS IS,
   such as show in the example above with the One Attribute in JSON.

   2) SHA256: which prevents from seeing content (see Security
   Considerations), has a fixed size and is convenient for most
   requirements

   3) Base64 URL: Where the specification of Base64 is followed, except
   the characters conflicting with an URL argument are replaced

   The value is configured as part of the Namespace.  The private
   "_config" Namespace prefix stores this value storage mechanism.

2.3.1.  Configuring the value format for a Namespace

   If one has the Namespace "/Organization1/BU1/ip" and want to store
   those IP addresses in SHA256, it will be configured like this: The
   Namespace is kept but prefixed by "_config" and has a json object
   about value format set.  "/_config/Organization1/BU1/ip"

   {
     "value_format":"SHA256"
   }

   Where "value_format" is either: "SHA256", "RAW" or "BASE64URL".

2.4.  Bulk

   When data must be sent and received in large amounts, it is
   preferable to embed in JSON all the objects at once.  As such, for
   reading and writing, the format is the following:

   {
     "items": [
       { "<namespace>": "<value>" },
       { "<namespace>": "<value>", "timestamp": <epoch> }
     ]
   }

   Where:

   namespace: is the wanted namespace where to store the value




Tricaud                 Expires October 15, 2020                [Page 5]

Internet-Draft           SightingDB query format              April 2020


   value: the value one want to track

   timestamp: OPTIONAL epoch timestamp to set the value at.

   The timestamp is how one can use SightingDB and use old datasets
   where the first seen and last seen is not relative to "right now".

2.4.1.  Request

   A Proper request with two items is made like this:

{
  "items": [
    { "/your/namespace": "127.0.0.1" },
    { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db", "timestamp": 1586825229 }
  ]
}

   Which will either store or retrieve the wanted data.

2.4.2.  Response

   The response when retrieving sightings also has the list of items, in
   order, one per line of the results:

{
  "items": [
    {"value": "Octave_Hergebel", "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "consensus": 1},
    {"value": "127.0.0.1", "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "consensus": 3}
  ]
}

3.  Security Considerations

   While this document solely focuses on the format, the reference
   implementation is SightingDB.  The authentication, the data access is
   not handled by SightingDB.  It is possible a value can leak if the
   access is too permissive.

   Even a Hashed value can be discovered, as re-hashing known values
   would match.

4.  Acknowledgements

   The author wish to thank all the MISP community who are supporting
   the creation of open standards in threat intelligence sharing.  As
   well as amazing feedback gathered during the MISP Summit 2019 in
   Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody.



Tricaud                 Expires October 15, 2020                [Page 6]

Internet-Draft           SightingDB query format              April 2020


5.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8259]  Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
              Interchange Format", STD 90, RFC 8259,
              DOI 10.17487/RFC8259, December 2017,
              <https://www.rfc-editor.org/info/rfc8259>.

Author's Address

   Sebastien Tricaud
   Devo Inc.
   150 Cambridgepark Drive
   Cambridge, MA  02140
   USA

   Phone: +1 866-221-2254
   Email: sebastien.tricaud@devo.com





























Tricaud                 Expires October 15, 2020                [Page 7]