2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Network Working Group S. Tricaud
|
|
|
|
|
Internet-Draft Devo Inc.
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Intended status: Informational April 13, 2020
|
|
|
|
|
Expires: October 15, 2020
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
SightingDB query format
|
|
|
|
|
draft-tricaud-sightingdb-format
|
|
|
|
|
|
|
|
|
|
Abstract
|
|
|
|
|
|
|
|
|
|
This document describes the format used by SightingDB to give
|
|
|
|
|
automated context to a given Attribute by counting occurrences and
|
|
|
|
|
tracking times of observability. SightingDB was designed to provide
|
|
|
|
|
to MISP a Scalable and Fast way to store and retrieve Attributes.
|
|
|
|
|
|
|
|
|
|
Status of This Memo
|
|
|
|
|
|
|
|
|
|
This Internet-Draft is submitted in full conformance with the
|
|
|
|
|
provisions of BCP 78 and BCP 79.
|
|
|
|
|
|
|
|
|
|
Internet-Drafts are working documents of the Internet Engineering
|
|
|
|
|
Task Force (IETF). Note that other groups may also distribute
|
|
|
|
|
working documents as Internet-Drafts. The list of current Internet-
|
|
|
|
|
Drafts is at https://datatracker.ietf.org/drafts/current/.
|
|
|
|
|
|
|
|
|
|
Internet-Drafts are draft documents valid for a maximum of six months
|
|
|
|
|
and may be updated, replaced, or obsoleted by other documents at any
|
|
|
|
|
time. It is inappropriate to use Internet-Drafts as reference
|
|
|
|
|
material or to cite them other than as "work in progress."
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
This Internet-Draft will expire on October 15, 2020.
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
Copyright Notice
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Copyright (c) 2020 IETF Trust and the persons identified as the
|
2019-11-06 11:31:34 +01:00
|
|
|
|
document authors. All rights reserved.
|
|
|
|
|
|
|
|
|
|
This document is subject to BCP 78 and the IETF Trust's Legal
|
|
|
|
|
Provisions Relating to IETF Documents
|
|
|
|
|
(https://trustee.ietf.org/license-info) in effect on the date of
|
|
|
|
|
publication of this document. Please review these documents
|
|
|
|
|
carefully, as they describe your rights and restrictions with respect
|
|
|
|
|
to this document. Code Components extracted from this document must
|
|
|
|
|
include Simplified BSD License text as described in Section 4.e of
|
|
|
|
|
the Trust Legal Provisions and are provided without warranty as
|
|
|
|
|
described in the Simplified BSD License.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Tricaud Expires October 15, 2020 [Page 1]
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Internet-Draft SightingDB query format April 2020
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table of Contents
|
|
|
|
|
|
|
|
|
|
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
|
|
|
|
|
1.1. Conventions and Terminology . . . . . . . . . . . . . . . 2
|
|
|
|
|
2. Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
|
|
|
|
|
2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 2
|
|
|
|
|
2.1.1. Attribute Storage . . . . . . . . . . . . . . . . . . 2
|
|
|
|
|
2.1.2. Namespace . . . . . . . . . . . . . . . . . . . . . . 3
|
|
|
|
|
2.1.3. Attribute fields . . . . . . . . . . . . . . . . . . 3
|
|
|
|
|
2.2. SightingDB Format - One Attribute . . . . . . . . . . . . 4
|
2019-11-07 08:13:23 +01:00
|
|
|
|
2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5
|
|
|
|
|
2.3.1. Configuring the value format for a Namespace . . . . 5
|
|
|
|
|
2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5
|
2020-04-14 03:05:31 +02:00
|
|
|
|
2.4.1. Request . . . . . . . . . . . . . . . . . . . . . . . 6
|
|
|
|
|
2.4.2. Response . . . . . . . . . . . . . . . . . . . . . . 6
|
2019-11-07 08:13:23 +01:00
|
|
|
|
3. Security Considerations . . . . . . . . . . . . . . . . . . . 6
|
|
|
|
|
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6
|
2020-04-14 03:05:31 +02:00
|
|
|
|
5. Normative References . . . . . . . . . . . . . . . . . . . . 7
|
|
|
|
|
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 7
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
1. Introduction
|
|
|
|
|
|
|
|
|
|
Adding context to any Attribute is the key that makes it useful.
|
|
|
|
|
While there exist numerous ways of doing it, SightingDB does it by
|
|
|
|
|
just counting. Whenever somebody retrieves an Attribute, this
|
|
|
|
|
counting is provided, allowing anyone to understand whenever
|
|
|
|
|
something was observed few or many times.
|
|
|
|
|
|
|
|
|
|
1.1. Conventions and Terminology
|
|
|
|
|
|
|
|
|
|
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
|
|
|
|
|
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
|
|
|
|
|
document are to be interpreted as described in RFC 2119 [RFC2119].
|
|
|
|
|
|
|
|
|
|
2. Format
|
|
|
|
|
|
|
|
|
|
2.1. Overview
|
|
|
|
|
|
|
|
|
|
The SightingDB format is in JSON [RFC8259] format and used to query a
|
|
|
|
|
SightingDB compatible connector. In SightingDB, a Sighting Object is
|
|
|
|
|
composed of a single JSON object. This object contains the following
|
2020-04-14 03:05:31 +02:00
|
|
|
|
fields: value, first_seen, last_seen, count, tags, ttl and consensus.
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
2.1.1. Attribute Storage
|
|
|
|
|
|
|
|
|
|
The fields described previously describe an Attribute and all the
|
|
|
|
|
required characteristics. However they are stored in a Namespace. A
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
|
|
|
|
|
Tricaud Expires October 15, 2020 [Page 2]
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Internet-Draft SightingDB query format April 2020
|
|
|
|
|
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Namespace is similar to a path in a file-system where the same file
|
|
|
|
|
can be stored in multiple places.
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
2.1.2. Namespace
|
|
|
|
|
|
|
|
|
|
A Namespace with multiple levels MUST be separated with the slash '/'
|
|
|
|
|
character. There is no specification on how they are structured,
|
|
|
|
|
since it depends on the use cases.
|
|
|
|
|
|
|
|
|
|
A Namespace starting with the underscore '_' character means it is
|
|
|
|
|
private and internal to SightingDB. There are all reserved for the
|
|
|
|
|
engine and MUST NOT be used.
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
Reserved namespaces are:
|
|
|
|
|
|
|
|
|
|
_expired/: Which contains all the attributes that expired, preserving
|
|
|
|
|
the origin namespace
|
|
|
|
|
|
|
|
|
|
_shadow/: When a value is searched and does not exists, it is stored
|
|
|
|
|
there
|
|
|
|
|
|
|
|
|
|
_config: Configuration
|
|
|
|
|
|
|
|
|
|
_all: All the Attributes in one place, used to retrieve the
|
2020-04-14 03:05:31 +02:00
|
|
|
|
'consensus' property.
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
The Attribute Key MUST always be the last part of the Namespace.
|
|
|
|
|
|
|
|
|
|
2.1.2.1. Sample Namespaces
|
|
|
|
|
|
|
|
|
|
/Organization1/service/ipv4: Store values for ipv4 keys in
|
2019-11-07 08:13:23 +01:00
|
|
|
|
/Organization1/service
|
|
|
|
|
|
|
|
|
|
/everything/domain: Store domains in /everything
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
2.1.3. Attribute fields
|
|
|
|
|
|
|
|
|
|
2.1.3.1. value
|
|
|
|
|
|
|
|
|
|
The attribute value, used to store and retrieve information about an
|
|
|
|
|
attribute. Note that value is not returned back in the JSON object,
|
|
|
|
|
since it is queried, it is known. The Value is described in a
|
|
|
|
|
section below, as it is very specific and can be either "as is", a
|
|
|
|
|
hash, encoded in base64 or any other convenient mechanism.
|
|
|
|
|
|
|
|
|
|
The value implementation MUST offer at least: 1) Raw value 2) Base64
|
|
|
|
|
URL Encoded 3) SHA256 Hash
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Tricaud Expires October 15, 2020 [Page 3]
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Internet-Draft SightingDB query format April 2020
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
2.1.3.2. first_seen
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
Time in UTC of the first time this value was captured
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
2.1.3.3. last_seen
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
Time in UTC of the last time this value was captured
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
2.1.3.4. count
|
|
|
|
|
|
|
|
|
|
How many time this value was written
|
|
|
|
|
|
|
|
|
|
2.1.3.5. tags
|
|
|
|
|
|
|
|
|
|
Tags follow how they are defined in MISP using the MISP Taxonomy.
|
|
|
|
|
Each Tag is separated with the ';' character.
|
|
|
|
|
|
|
|
|
|
2.1.3.6. ttl
|
|
|
|
|
|
|
|
|
|
Time To Live, represents the expiration in seconds since the time the
|
|
|
|
|
Attribute was created. Once it has expired, it moves in the private
|
|
|
|
|
Namespace _expired.
|
|
|
|
|
|
|
|
|
|
When an Attribute has this field set to 0, it means it is not set to
|
|
|
|
|
expired. This is the default behavior.
|
|
|
|
|
|
|
|
|
|
When an Attribute has this field set to a number greater than 0, the
|
|
|
|
|
expiration status is computed only at retrieval time.
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
2.1.3.7. consensus
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
When a given Attribute Value is stored in different namespaces, the
|
2020-04-14 03:05:31 +02:00
|
|
|
|
consensus field keeps track of them so it returns in how many
|
2019-11-06 11:31:34 +01:00
|
|
|
|
different places this attributes exists. This is a simple counter.
|
|
|
|
|
|
|
|
|
|
2.2. SightingDB Format - One Attribute
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
"value":"127.0.0.1",
|
|
|
|
|
"first_seen":1530394819,
|
|
|
|
|
"last_seen":1572933618,
|
|
|
|
|
"count":578391,
|
|
|
|
|
"tags":"",
|
|
|
|
|
"ttl":0,
|
2020-04-14 03:05:31 +02:00
|
|
|
|
"consensus": 17
|
2019-11-06 11:31:34 +01:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Tricaud Expires October 15, 2020 [Page 4]
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Internet-Draft SightingDB query format April 2020
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
2.3. Value
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
The value submitted can be in multiple format according to the use-
|
|
|
|
|
case. Any implementation MUST offer three alternatives:
|
|
|
|
|
|
|
|
|
|
1. Raw value: where nothing is encoded and the value is stored AS
|
|
|
|
|
IS, such as show in the example above with the One Attribute in
|
|
|
|
|
JSON.
|
|
|
|
|
|
|
|
|
|
2. SHA256: which prevents from seeing content (see Security
|
|
|
|
|
Considerations), has a fixed size and is convenient for most
|
|
|
|
|
requirements
|
|
|
|
|
|
|
|
|
|
3. Base64 URL: Where the specification of Base64 is followed, except
|
|
|
|
|
the characters conflicting with an URL argument are replaced
|
|
|
|
|
|
|
|
|
|
The value is configured as part of the Namespace. The private
|
|
|
|
|
"_config" Namespace prefix stores this value storage mechanism.
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
2.3.1. Configuring the value format for a Namespace
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
If one has the Namespace "/Organization1/BU1/ip" and want to store
|
|
|
|
|
those IP addresses in SHA256, it will be configured like this: The
|
|
|
|
|
Namespace is kept but prefixed by "_config" and has a json object
|
|
|
|
|
about value format set. "/_config/Organization1/BU1/ip"
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
"value_format":"SHA256"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Where "value_format" is either: "SHA256", "RAW" or "BASE64URL".
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
2.4. Bulk
|
|
|
|
|
|
|
|
|
|
When data must be sent and received in large amounts, it is
|
|
|
|
|
preferable to embed in JSON all the objects at once. As such, for
|
|
|
|
|
reading and writing, the format is the following:
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
{
|
|
|
|
|
"items": [
|
|
|
|
|
{ "<namespace>": "<value>" },
|
|
|
|
|
{ "<namespace>": "<value>", "timestamp": <epoch> }
|
|
|
|
|
]
|
|
|
|
|
}
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Where:
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
namespace: is the wanted namespace where to store the value
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Tricaud Expires October 15, 2020 [Page 5]
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Internet-Draft SightingDB query format April 2020
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
value: the value one want to track
|
|
|
|
|
|
|
|
|
|
timestamp: OPTIONAL epoch timestamp to set the value at.
|
|
|
|
|
|
|
|
|
|
The timestamp is how one can use SightingDB and use old datasets
|
|
|
|
|
where the first seen and last seen is not relative to "right now".
|
|
|
|
|
|
|
|
|
|
2.4.1. Request
|
|
|
|
|
|
|
|
|
|
A Proper request with two items is made like this:
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
"items": [
|
|
|
|
|
{ "/your/namespace": "127.0.0.1" },
|
|
|
|
|
{ "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db", "timestamp": 1586825229 }
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Which will either store or retrieve the wanted data.
|
|
|
|
|
|
|
|
|
|
2.4.2. Response
|
2019-11-07 08:13:23 +01:00
|
|
|
|
|
|
|
|
|
The response when retrieving sightings also has the list of items, in
|
|
|
|
|
order, one per line of the results:
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
"items": [
|
2020-04-14 03:05:31 +02:00
|
|
|
|
{"value": "Octave_Hergebel", "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "consensus": 1},
|
|
|
|
|
{"value": "127.0.0.1", "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "consensus": 3}
|
2019-11-07 08:13:23 +01:00
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
3. Security Considerations
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
While this document solely focuses on the format, the reference
|
|
|
|
|
implementation is SightingDB. The authentication, the data access is
|
|
|
|
|
not handled by SightingDB. It is possible a value can leak if the
|
|
|
|
|
access is too permissive.
|
|
|
|
|
|
|
|
|
|
Even a Hashed value can be discovered, as re-hashing known values
|
|
|
|
|
would match.
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
4. Acknowledgements
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
The author wish to thank all the MISP community who are supporting
|
|
|
|
|
the creation of open standards in threat intelligence sharing. As
|
|
|
|
|
well as amazing feedback gathered during the MISP Summit 2019 in
|
|
|
|
|
Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody.
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Tricaud Expires October 15, 2020 [Page 6]
|
|
|
|
|
|
|
|
|
|
Internet-Draft SightingDB query format April 2020
|
|
|
|
|
|
|
|
|
|
|
2019-11-07 08:13:23 +01:00
|
|
|
|
5. Normative References
|
2019-11-06 11:31:34 +01:00
|
|
|
|
|
|
|
|
|
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
|
|
|
|
|
Requirement Levels", BCP 14, RFC 2119,
|
|
|
|
|
DOI 10.17487/RFC2119, March 1997,
|
|
|
|
|
<https://www.rfc-editor.org/info/rfc2119>.
|
|
|
|
|
|
|
|
|
|
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
|
|
|
|
|
Interchange Format", STD 90, RFC 8259,
|
|
|
|
|
DOI 10.17487/RFC8259, December 2017,
|
|
|
|
|
<https://www.rfc-editor.org/info/rfc8259>.
|
|
|
|
|
|
|
|
|
|
Author's Address
|
|
|
|
|
|
|
|
|
|
Sebastien Tricaud
|
|
|
|
|
Devo Inc.
|
|
|
|
|
150 Cambridgepark Drive
|
|
|
|
|
Cambridge, MA 02140
|
|
|
|
|
USA
|
|
|
|
|
|
|
|
|
|
Phone: +1 866-221-2254
|
|
|
|
|
Email: sebastien.tricaud@devo.com
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-04-14 03:05:31 +02:00
|
|
|
|
Tricaud Expires October 15, 2020 [Page 7]
|