From e2e78f5fc29d858897d04743568c157ee4f5049f Mon Sep 17 00:00:00 2001 From: mokaddem Date: Thu, 20 Jun 2019 09:36:37 +0200 Subject: [PATCH 01/28] chg: Added first_seen/last_seen sections --- misp-core-format/raw.md | 52 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 48 insertions(+), 4 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index c2efc68..975e8cb 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -279,7 +279,9 @@ A MISP document **MUST** at least includes category-type-value triplet described "value": "Hello world", "SharingGroup": [], "ShadowAttribute": [], - "RelatedAttribute": [] + "RelatedAttribute": [], + "first_seen": null, + "last_seen": null } ~~~~ @@ -450,6 +452,18 @@ value represents the payload of an attribute. The format of the value is depende value is represented by a JSON string. value **MUST** be present. +#### first_seen + +first_seen represents a reference time when the attribute was first seen. first_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. + +first_seen is represented as a JSON string. first_seen **SHALL** be present. + +#### last_seen + +last_seen represents a reference time when the attribute was last seen. last_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. + +last_seen is represented as a JSON string. last_seen **SHALL** be present. + ## ShadowAttribute ShadowAttributes are 3rd party created attributes that either propose to add new information to an event or modify existing information. They are not meant to be actionable until the event creator accepts them - at which point they will be converted into attributes or modify an existing attribute. @@ -477,7 +491,9 @@ They are similar in structure to Attributes but additionally carry a reference t "id": "1", "name": "MISP", "uuid": "568cce5a-0c80-412b-8fdf-1ffac0a83869" - } + }, + "first_seen": null, + "last_seen": null } ~~~~ @@ -620,6 +636,18 @@ the sample **MUST** be encrypted using a password protected zip archive, with th data is represented by a JSON string in base64 encoding. data **MUST** be set for shadow attributes of type malware-sample and attachment. +#### first_seen + +first_seen represents a reference time when the attribute was first seen. first_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. + +first_seen is represented as a JSON string. first_seen **SHALL** be present. + +#### last_seen + +last_seen represents a reference time when the attribute was last seen. last_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. + +last_seen is represented as a JSON string. last_seen **SHALL** be present. + ### Org An Org object is composed of an uuid, name and id. @@ -693,8 +721,12 @@ A MISP document containing an Object **MUST** contain a name, a meta-category, a "object_id": "588", "object_relation": "filename", "value": "StarCraft.exe", - "ShadowAttribute": [] - } + "ShadowAttribute": [], + "first_seen": null, + "last_seen": null + }, + "first_seen": null, + "last_seen": null ] } ~~~~~ @@ -802,6 +834,18 @@ Attribute is an array of attributes that describe the object with data. Each attribute in an object **MUST** contain the parent event's ID in the event_id field and the parent object's ID in the object_id field. +#### first_seen + +first_seen represents a reference time when the object was first seen. first_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. + +first_seen is represented as a JSON string. first_seen **SHALL** be present. + +#### last_seen + +last_seen represents a reference time when the object was last seen. last_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. + +last_seen is represented as a JSON string. last_seen **SHALL** be present. + ## Object References Object References serve as a logical link between an Object and another referenced Object or Attribute. The relationship is categorised by an enumerated value from a fixed vocabulary. From f65efaec395e4e72d79f3a28f5ddc17ea125872a Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 16:24:35 +0200 Subject: [PATCH 02/28] chg: [object-template] release 03 of the Internet-Draft (list of default templates added) --- misp-object-template-format/raw.md | 132 +++++- misp-object-template-format/raw.md.txt | 542 ++++++++++++++++++++++--- 2 files changed, 626 insertions(+), 48 deletions(-) diff --git a/misp-object-template-format/raw.md b/misp-object-template-format/raw.md index f66943f..1e08e20 100755 --- a/misp-object-template-format/raw.md +++ b/misp-object-template-format/raw.md @@ -313,7 +313,137 @@ format is represented by a JSON list containing a list of formats that the relat The MISP object template directory is publicly available [@?MISP-O] in a git repository. The repository contains an objects directory, which contains a directory per object type, containing a file named definition.json which contains the definition of the object template in the above described format. -A relationships directory is also included, containing a definition.json file which contains a list of MISP object relation definitions. There are more than 90 existing templates object documented in [@?MISP-O-DOC]. +A relationships directory is also included, containing a definition.json file which contains a list of MISP object relation definitions. There are more than 125 existing templates object documented in [@?MISP-O-DOC]. + +## Existing and public MISP object templates + +- tsk-chats - An Object Template to gather information from evidential or interesting exchange of messages identified during a digital forensic investigation. +- tsk-web-bookmark - An Object Template to add evidential bookmarks identified during a digital forensic investigation. +- tsk-web-cookie - An TSK-Autopsy Object Template to represent cookies identified during a forensic investigation. +- tsk-web-downloads - An Object Template to add web-downloads. +- tsk-web-history - An Object Template to share web history information. +- tsk-web-search-query - An Object Template to share web search query information. +- ail-leak - An information leak as defined by the AIL Analysis Information Leak framework. +- ais-info - Automated Indicator Sharing (AIS) Information Source Markings. +- android-permission - A set of android permissions - one or more permission(s) which can be linked to other objects (e.g. malware, app). +- annotation - An annotation object allowing analysts to add annotations, comments, executive summary to a MISP event, objects or attributes. +- anonymisation - Anonymisation object describing an anonymisation technique used to encode MISP attribute values. Reference: https://www.caida.org/tools/taxonomy/anonymization.xml. +- asn - Autonomous system object describing an autonomous system which can include one or more network operators management an entity (e.g. ISP) along with their routing policy, routing prefixes or alike. +- authenticode-signerinfo - Authenticode Signer Info. +- av-signature - Antivirus detection signature. +- bank-account - An object describing bank account information based on account description from goAML 4.0. +- bgp-hijack - Object encapsulating BGP Hijack description as specified, for example, by bgpstream.com. +- cap-alert - Common Alerting Protocol Version (CAP) alert object. +- cap-info - Common Alerting Protocol Version (CAP) info object. +- cap-resource - Common Alerting Protocol Version (CAP) resource object. +- coin-address - An address used in a cryptocurrency. +- cookie - An HTTP cookie (web cookie, browser cookie) is a small piece of data that a server sends to the user's web browser. The browser may store it and send it back with the next request to the same server. Typically, it's used to tell if two requests came from the same browser — keeping a user logged-in, for example. It remembers stateful information for the stateless HTTP protocol. (as defined by the Mozilla foundation. +- cortex - Cortex object describing a complete cortex analysis. Observables would be attribute with a relationship from this object. +- cortex-taxonomy - Cortex object describing an Cortex Taxonomy (or mini report). +- course-of-action - An object describing a specific measure taken to prevent or respond to an attack. +- cowrie - Cowrie honeypot object template. +- credential - Credential describes one or more credential(s) including password(s), api key(s) or decryption key(s). +- credit-card - A payment card like credit card, debit card or any similar cards which can be used for financial transactions. +- ddos - DDoS object describes a current DDoS activity from a specific or/and to a specific target. Type of DDoS can be attached to the object as a taxonomy. +- device - An object to define a device. +- diameter-attack - Attack as seen on diameter authentication against a GSM, UMTS or LTE network. +- domain-ip - A domain and IP address seen as a tuple in a specific time frame. +- elf - Object describing a Executable and Linkable Format. +- elf-section - Object describing a section of an Executable and Linkable Format. +- email - Email object describing an email with meta-information. +- exploit-poc - Exploit-poc object describing a proof of concept or exploit of a vulnerability. This object has often a relationship with a vulnerability object. +- facial-composite - An object which describes a facial composite. +- fail2ban - Fail2ban event. +- file - File object describing a file with meta-information. +- forensic-case - An object template to describe a digital forensic case. +- forensic-evidence - An object template to describe a digital forensic evidence. +- geolocation - An object to describe a geographic location. +- gtp-attack - GTP attack object as seen on a GSM, UMTS or LTE network. +- http-request - A single HTTP request header. +- ilr-impact - Institut Luxembourgeois de Regulation - Impact. +- ilr-notification-incident - Institut Luxembourgeois de Regulation - Notification d'incident. +- internal-reference - Internal reference. +- interpol-notice - An object which describes a Interpol notice. +- ip-api-address - IP Address information. Useful if you are pulling your ip information from ip-api.com. +- ip-port - An IP address (or domain or hostname) and a port seen as a tuple (or as a triple) in a specific time frame. +- irc - An IRC object to describe an IRC server and the associated channels. +- ja3 - JA3 is a new technique for creating SSL client fingerprints that are easy to produce and can be easily shared for threat intelligence. Fingerprints are composed of Client Hello packet; SSL Version, Accepted Ciphers, List of Extensions, Elliptic Curves, and Elliptic Curve Formats. https://github.com/salesforce/ja3. +- legal-entity - An object to describe a legal entity. +- lnk - LNK object describing a Windows LNK binary file (aka Windows shortcut). +- macho - Object describing a file in Mach-O format. +- macho-section - Object describing a section of a file in Mach-O format. +- mactime-timeline-analysis - Mactime template, used in forensic investigations to describe the timeline of a file activity. +- malware-config - Malware configuration recovered or extracted from a malicious binary. +- microblog - Microblog post like a Twitter tweet or a post on a Facebook wall. +- mutex - Object to describe mutual exclusion locks (mutex) as seen in memory or computer program. +- netflow - Netflow object describes an network object based on the Netflowv5/v9 minimal definition. +- network-connection - A local or remote network connection. +- network-socket - Network socket object describes a local or remote network connections based on the socket data structure. +- misc - An object which describes an organization. +- original-imported-file - Object describing the original file used to import data in MISP. +- passive-dns - Passive DNS records as expressed in draft-dulaunoy-dnsop-passive-dns-cof-01. +- paste - Paste or similar post from a website allowing to share privately or publicly posts. +- pcap-metadata - Network packet capture metadata. +- pe - Object describing a Portable Executable. +- pe-section - Object describing a section of a Portable Executable. +- person - An object which describes a person or an identity. +- phishing - Phishing template to describe a phishing website and its analysis. +- phishing-kit - Object to describe a phishing-kit. +- phone - A phone or mobile phone object which describe a phone. +- process - Object describing a system process. +- python-etvx-event-log - Event log object template to share information of the activities conducted on a system. . +- r2graphity - Indicators extracted from files using radare2 and graphml. +- regexp - An object describing a regular expression (regex or regexp). The object can be linked via a relationship to other attributes or objects to describe how it can be represented as a regular expression. +- registry-key - Registry key object describing a Windows registry key with value and last-modified timestamp. +- regripper-NTUser - Regripper Object template designed to present user specific configuration details extracted from the NTUSER.dat hive. +- regripper-sam-hive-single-user - Regripper Object template designed to present user profile details extracted from the SAM hive. +- regripper-sam-hive-user-group - Regripper Object template designed to present group profile details extracted from the SAM hive. +- regripper-software-hive-BHO - Regripper Object template designed to gather information of the browser helper objects installed on the system. +- regripper-software-hive-appInit-DLLS - Regripper Object template designed to gather information of the DLL files installed on the system. +- regripper-software-hive-application-paths - Regripper Object template designed to gather information of the application paths. +- regripper-software-hive-applications-installed - Regripper Object template designed to gather information of the applications installed on the system. +- regripper-software-hive-command-shell - Regripper Object template designed to gather information of the shell commands executed on the system. +- regripper-software-hive-windows-general-info - Regripper Object template designed to gather general windows information extracted from the software-hive. +- regripper-software-hive-software-run - Regripper Object template designed to gather information of the applications set to run on the system. +- regripper-software-hive-userprofile-winlogon - Regripper Object template designed to gather user profile information when the user logs onto the system, gathered from the software hive. +- regripper-system-hive-firewall-configuration - Regripper Object template designed to present firewall configuration information extracted from the system-hive. +- regripper-system-hive-general-configuration - Regripper Object template designed to present general system properties extracted from the system-hive. +- regripper-system-hive-network-information. - Regripper object template designed to gather network information from the system-hive. +- regripper-system-hive-services-drivers - Regripper Object template designed to gather information regarding the services/drivers from the system-hive. +- report - Metadata used to generate an executive level report. +- research-scanner - Information related to known scanning activity (e.g. from research projects). +- rogue-dns - Rogue DNS as defined by CERT.br. +- rtir - RTIR - Request Tracker for Incident Response. +- sandbox-report - Sandbox report. +- sb-signature - Sandbox detection signature. +- script - Object describing a computer program written to be run in a special run-time environment. The script or shell script can be used for malicious activities but also as support tools for threat analysts. +- shell-commands - Object describing a series of shell commands executed. This object can be linked with malicious files in order to describe a specific execution of shell commands. +- short-message-service - Short Message Service (SMS) object template describing one or more SMS message. Restriction of the initial format 3GPP 23.038 GSM character set doesn't apply. +- shortened-link - Shortened link and its redirect target. +- splunk - Splunk / Splunk ES object. +- ss7-attack - SS7 object of an attack seen on a GSM, UMTS or LTE network via SS7 logging. +- ssh-authorized-keys - An object to store ssh authorized keys file. +- stix2-pattern - An object describing a STIX pattern. The object can be linked via a relationship to other attributes or objects to describe how it can be represented as a STIX pattern. +- suricata - An object describing one or more Suricata rule(s) along with version and contextual information. +- target-system - Description about an targeted system, this could potentially be a compromissed internal system. +- threatgrid-report - ThreatGrid report. +- timecode - Timecode object to describe a start of video sequence (e.g. CCTV evidence) and the end of the video sequence. +- timesketch-timeline - A timesketch timeline object based on mandatory field in timesketch to describe a log entry. +- timesketch_message - A timesketch message entry. +- timestamp - A generic timestamp object to represent time including first time and last time seen. Relationship will then define the kind of time relationship. +- tor-hiddenservice - Tor hidden service (onion service) object. +- tor-node - Tor node (which protects your privacy on the internet by hiding the connection between users Internet address and the services used by the users) description which are part of the Tor network at a time. +- tracking-id - Analytics and tracking ID such as used in Google Analytics or other analytic platform. +- transaction - An object to describe a financial transaction. +- url - url object describes an url along with its normalized field (like extracted using faup parsing library) and its metadata. +- vehicle - Vehicle object template to describe a vehicle information and registration. +- victim - Victim object describes the target of an attack or abuse. +- virustotal-report - VirusTotal report. +- vulnerability - Vulnerability object describing a common vulnerability enumeration which can describe published, unpublished, under review or embargo vulnerability for software, equipments or hardware. +- whois - Whois records information for a domain name or an IP address. +- x509 - x509 object describing a X.509 certificate. +- yabin - yabin.py generates Yara rules from function prologs, for matching and hunting binaries. ref: https://github.com/AlienVault-OTX/yabin. +- yara - An object describing a YARA rule along with its version. # Acknowledgements diff --git a/misp-object-template-format/raw.md.txt b/misp-object-template-format/raw.md.txt index c9aed45..9e8aa98 100755 --- a/misp-object-template-format/raw.md.txt +++ b/misp-object-template-format/raw.md.txt @@ -27,7 +27,7 @@ Status of This Memo Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- - Drafts is at http://datatracker.ietf.org/drafts/current/. + Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any @@ -43,7 +43,7 @@ Copyright Notice This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents - (http://trustee.ietf.org/license-info) in effect on the date of + (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must @@ -69,11 +69,12 @@ Table of Contents 2.1.3. Sample Object Template object . . . . . . . . . . . . 6 2.1.4. Object Relationships . . . . . . . . . . . . . . . . 9 3. Directory . . . . . . . . . . . . . . . . . . . . . . . . . . 10 - 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 - 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 - 5.1. Normative References . . . . . . . . . . . . . . . . . . 10 - 5.2. Informative References . . . . . . . . . . . . . . . . . 10 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 + 3.1. Existing and public MISP object templates . . . . . . . . 10 + 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 18 + 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 + 5.1. Normative References . . . . . . . . . . . . . . . . . . 18 + 5.2. Informative References . . . . . . . . . . . . . . . . . 18 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 1. Introduction @@ -108,7 +109,6 @@ Table of Contents - Dulaunoy & Iklody Expires October 12, 2018 [Page 2] Internet-Draft MISP object template format April 2018 @@ -123,13 +123,15 @@ Internet-Draft MISP object template format April 2018 MISP object templates themselves consist of a name (MUST), a meta- category (MUST) and a description (SHOULD). They are identified by a - uuid (MUST) and a version (MUST). The list of requirements when it - comes to the contained MISP object template elements is defined in - the requirements field (OPTIONAL). + uuid (MUST) and a version (MUST). For any updates or transfer of the + same object reference. UUID version 4 is RECOMMENDED when assigning + it to a new object reference. The list of requirements when it comes + to the contained MISP object template elements is defined in the + requirements field (OPTIONAL). - MISP object template elements consist of an object_relation (MUST) a - type (MUST) an object_template_id (SHOULD) a ui_priority (SHOULD) a - list of categories (MAY), a list of sane_default values (MAY) or a + MISP object template elements consist of an object_relation (MUST), a + type (MUST), an object_template_id (SHOULD), a ui_priority (SHOULD), + a list of categories (MAY), a list of sane_default values (MAY) or a values_list (MAY). 2.1. Overview @@ -157,10 +159,8 @@ Internet-Draft MISP object template format April 2018 be created based on the given template. The requiredOneOf field MAY be present. -2.1.1.3. required - required is represented as a JSON list and contains a list of - attribute relationships of which all must be present in the object to + @@ -170,6 +170,10 @@ Dulaunoy & Iklody Expires October 12, 2018 [Page 3] Internet-Draft MISP object template format April 2018 +2.1.1.3. required + + required is represented as a JSON list and contains a list of + attribute relationships of which all must be present in the object to be created based on the given template. The required field MAY be present. @@ -195,7 +199,7 @@ Internet-Draft MISP object template format April 2018 list of options but can be created on the fly. meta-category is represented as a JSON string. meta-category MUST be - present + present. 2.1.1.7. name @@ -212,11 +216,7 @@ Internet-Draft MISP object template format April 2018 attributes is represented as a JSON list. attributes MUST be present. -2.1.2.1. description - description is represented as a JSON string and contains the - description of the given attribute in the context of the object with - the given relationship. The description field MUST be present. @@ -226,6 +226,12 @@ Dulaunoy & Iklody Expires October 12, 2018 [Page 4] Internet-Draft MISP object template format April 2018 +2.1.2.1. description + + description is represented as a JSON string and contains the + description of the given attribute in the context of the object with + the given relationship. The description field MUST be present. + 2.1.2.2. ui-priority ui-priority is represented by a numeric values in JSON string format @@ -268,12 +274,6 @@ Internet-Draft MISP object template format April 2018 The multiple field MAY be present. -2.1.2.7. sane_default - - sane_default is represented by a JSON list containing one or several - recommended/sane values for an attribute. sane_default is mutually - exclusive with values_list. - @@ -282,6 +282,12 @@ Dulaunoy & Iklody Expires October 12, 2018 [Page 5] Internet-Draft MISP object template format April 2018 +2.1.2.7. sane_default + + sane_default is represented by a JSON list containing one or several + recommended/sane values for an attribute. sane_default is mutually + exclusive with values_list. + The sane_default field MAY be present. 2.1.2.8. values_list @@ -313,12 +319,6 @@ Internet-Draft MISP object template format April 2018 - - - - - - @@ -522,7 +522,453 @@ Internet-Draft MISP object template format April 2018 A relationships directory is also included, containing a definition.json file which contains a list of MISP object relation - definitions + definitions. There are more than 125 existing templates object + documented in [MISP-O-DOC]. + +3.1. Existing and public MISP object templates + + o tsk-chats - An Object Template to gather information from + evidential or interesting exchange of messages identified during a + digital forensic investigation. + + o tsk-web-bookmark - An Object Template to add evidential bookmarks + identified during a digital forensic investigation. + + o tsk-web-cookie - An TSK-Autopsy Object Template to represent + cookies identified during a forensic investigation. + + o tsk-web-downloads - An Object Template to add web-downloads. + + o tsk-web-history - An Object Template to share web history + information. + + o tsk-web-search-query - An Object Template to share web search + query information. + + o ail-leak - An information leak as defined by the AIL Analysis + Information Leak framework. + + o ais-info - Automated Indicator Sharing (AIS) Information Source + Markings. + + o android-permission - A set of android permissions - one or more + permission(s) which can be linked to other objects (e.g. malware, + app). + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 10] + +Internet-Draft MISP object template format April 2018 + + + o annotation - An annotation object allowing analysts to add + annotations, comments, executive summary to a MISP event, objects + or attributes. + + o anonymisation - Anonymisation object describing an anonymisation + technique used to encode MISP attribute values. Reference: + . + + o asn - Autonomous system object describing an autonomous system + which can include one or more network operators management an + entity (e.g. ISP) along with their routing policy, routing + prefixes or alike. + + o authenticode-signerinfo - Authenticode Signer Info. + + o av-signature - Antivirus detection signature. + + o bank-account - An object describing bank account information based + on account description from goAML 4.0. + + o bgp-hijack - Object encapsulating BGP Hijack description as + specified, for example, by bgpstream.com. + + o cap-alert - Common Alerting Protocol Version (CAP) alert object. + + o cap-info - Common Alerting Protocol Version (CAP) info object. + + o cap-resource - Common Alerting Protocol Version (CAP) resource + object. + + o coin-address - An address used in a cryptocurrency. + + o cookie - An HTTP cookie (web cookie, browser cookie) is a small + piece of data that a server sends to the user's web browser. The + browser may store it and send it back with the next request to the + same server. Typically, it's used to tell if two requests came + from the same browser -- keeping a user logged-in, for example. + It remembers stateful information for the stateless HTTP protocol. + (as defined by the Mozilla foundation. + + o cortex - Cortex object describing a complete cortex analysis. + Observables would be attribute with a relationship from this + object. + + o cortex-taxonomy - Cortex object describing an Cortex Taxonomy (or + mini report). + + + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 11] + +Internet-Draft MISP object template format April 2018 + + + o course-of-action - An object describing a specific measure taken + to prevent or respond to an attack. + + o cowrie - Cowrie honeypot object template. + + o credential - Credential describes one or more credential(s) + including password(s), api key(s) or decryption key(s). + + o credit-card - A payment card like credit card, debit card or any + similar cards which can be used for financial transactions. + + o ddos - DDoS object describes a current DDoS activity from a + specific or/and to a specific target. Type of DDoS can be + attached to the object as a taxonomy. + + o device - An object to define a device. + + o diameter-attack - Attack as seen on diameter authentication + against a GSM, UMTS or LTE network. + + o domain-ip - A domain and IP address seen as a tuple in a specific + time frame. + + o elf - Object describing a Executable and Linkable Format. + + o elf-section - Object describing a section of an Executable and + Linkable Format. + + o email - Email object describing an email with meta-information. + + o exploit-poc - Exploit-poc object describing a proof of concept or + exploit of a vulnerability. This object has often a relationship + with a vulnerability object. + + o facial-composite - An object which describes a facial composite. + + o fail2ban - Fail2ban event. + + o file - File object describing a file with meta-information. + + o forensic-case - An object template to describe a digital forensic + case. + + o forensic-evidence - An object template to describe a digital + forensic evidence. + + o geolocation - An object to describe a geographic location. + + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 12] + +Internet-Draft MISP object template format April 2018 + + + o gtp-attack - GTP attack object as seen on a GSM, UMTS or LTE + network. + + o http-request - A single HTTP request header. + + o ilr-impact - Institut Luxembourgeois de Regulation - Impact. + + o ilr-notification-incident - Institut Luxembourgeois de Regulation + - Notification d'incident. + + o internal-reference - Internal reference. + + o interpol-notice - An object which describes a Interpol notice. + + o ip-api-address - IP Address information. Useful if you are + pulling your ip information from ip-api.com. + + o ip-port - An IP address (or domain or hostname) and a port seen as + a tuple (or as a triple) in a specific time frame. + + o irc - An IRC object to describe an IRC server and the associated + channels. + + o ja3 - JA3 is a new technique for creating SSL client fingerprints + that are easy to produce and can be easily shared for threat + intelligence. Fingerprints are composed of Client Hello packet; + SSL Version, Accepted Ciphers, List of Extensions, Elliptic + Curves, and Elliptic Curve Formats. + . + + o legal-entity - An object to describe a legal entity. + + o lnk - LNK object describing a Windows LNK binary file (aka Windows + shortcut). + + o macho - Object describing a file in Mach-O format. + + o macho-section - Object describing a section of a file in Mach-O + format. + + o mactime-timeline-analysis - Mactime template, used in forensic + investigations to describe the timeline of a file activity. + + o malware-config - Malware configuration recovered or extracted from + a malicious binary. + + o microblog - Microblog post like a Twitter tweet or a post on a + Facebook wall. + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 13] + +Internet-Draft MISP object template format April 2018 + + + o mutex - Object to describe mutual exclusion locks (mutex) as seen + in memory or computer program. + + o netflow - Netflow object describes an network object based on the + Netflowv5/v9 minimal definition. + + o network-connection - A local or remote network connection. + + o network-socket - Network socket object describes a local or remote + network connections based on the socket data structure. + + o misc - An object which describes an organization. + + o original-imported-file - Object describing the original file used + to import data in MISP. + + o passive-dns - Passive DNS records as expressed in draft-dulaunoy- + dnsop-passive-dns-cof-01. + + o paste - Paste or similar post from a website allowing to share + privately or publicly posts. + + o pcap-metadata - Network packet capture metadata. + + o pe - Object describing a Portable Executable. + + o pe-section - Object describing a section of a Portable Executable. + + o person - An object which describes a person or an identity. + + o phishing - Phishing template to describe a phishing website and + its analysis. + + o phishing-kit - Object to describe a phishing-kit. + + o phone - A phone or mobile phone object which describe a phone. + + o process - Object describing a system process. + + o python-etvx-event-log - Event log object template to share + information of the activities conducted on a system. . + + o r2graphity - Indicators extracted from files using radare2 and + graphml. + + o regexp - An object describing a regular expression (regex or + regexp). The object can be linked via a relationship to other + + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 14] + +Internet-Draft MISP object template format April 2018 + + + attributes or objects to describe how it can be represented as a + regular expression. + + o registry-key - Registry key object describing a Windows registry + key with value and last-modified timestamp. + + o regripper-NTUser - Regripper Object template designed to present + user specific configuration details extracted from the NTUSER.dat + hive. + + o regripper-sam-hive-single-user - Regripper Object template + designed to present user profile details extracted from the SAM + hive. + + o regripper-sam-hive-user-group - Regripper Object template designed + to present group profile details extracted from the SAM hive. + + o regripper-software-hive-BHO - Regripper Object template designed + to gather information of the browser helper objects installed on + the system. + + o regripper-software-hive-appInit-DLLS - Regripper Object template + designed to gather information of the DLL files installed on the + system. + + o regripper-software-hive-application-paths - Regripper Object + template designed to gather information of the application paths. + + o regripper-software-hive-applications-installed - Regripper Object + template designed to gather information of the applications + installed on the system. + + o regripper-software-hive-command-shell - Regripper Object template + designed to gather information of the shell commands executed on + the system. + + o regripper-software-hive-windows-general-info - Regripper Object + template designed to gather general windows information extracted + from the software-hive. + + o regripper-software-hive-software-run - Regripper Object template + designed to gather information of the applications set to run on + the system. + + o regripper-software-hive-userprofile-winlogon - Regripper Object + template designed to gather user profile information when the user + logs onto the system, gathered from the software hive. + + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 15] + +Internet-Draft MISP object template format April 2018 + + + o regripper-system-hive-firewall-configuration - Regripper Object + template designed to present firewall configuration information + extracted from the system-hive. + + o regripper-system-hive-general-configuration - Regripper Object + template designed to present general system properties extracted + from the system-hive. + + o regripper-system-hive-network-information. - Regripper object + template designed to gather network information from the system- + hive. + + o regripper-system-hive-services-drivers - Regripper Object template + designed to gather information regarding the services/drivers from + the system-hive. + + o report - Metadata used to generate an executive level report. + + o research-scanner - Information related to known scanning activity + (e.g. from research projects). + + o rogue-dns - Rogue DNS as defined by CERT.br. + + o rtir - RTIR - Request Tracker for Incident Response. + + o sandbox-report - Sandbox report. + + o sb-signature - Sandbox detection signature. + + o script - Object describing a computer program written to be run in + a special run-time environment. The script or shell script can be + used for malicious activities but also as support tools for threat + analysts. + + o shell-commands - Object describing a series of shell commands + executed. This object can be linked with malicious files in order + to describe a specific execution of shell commands. + + o short-message-service - Short Message Service (SMS) object + template describing one or more SMS message. Restriction of the + initial format 3GPP 23.038 GSM character set doesn't apply. + + o shortened-link - Shortened link and its redirect target. + + o splunk - Splunk / Splunk ES object. + + o ss7-attack - SS7 object of an attack seen on a GSM, UMTS or LTE + network via SS7 logging. + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 16] + +Internet-Draft MISP object template format April 2018 + + + o ssh-authorized-keys - An object to store ssh authorized keys file. + + o stix2-pattern - An object describing a STIX pattern. The object + can be linked via a relationship to other attributes or objects to + describe how it can be represented as a STIX pattern. + + o suricata - An object describing one or more Suricata rule(s) along + with version and contextual information. + + o target-system - Description about an targeted system, this could + potentially be a compromissed internal system. + + o threatgrid-report - ThreatGrid report. + + o timecode - Timecode object to describe a start of video sequence + (e.g. CCTV evidence) and the end of the video sequence. + + o timesketch-timeline - A timesketch timeline object based on + mandatory field in timesketch to describe a log entry. + + o timesketch_message - A timesketch message entry. + + o timestamp - A generic timestamp object to represent time including + first time and last time seen. Relationship will then define the + kind of time relationship. + + o tor-hiddenservice - Tor hidden service (onion service) object. + + o tor-node - Tor node (which protects your privacy on the internet + by hiding the connection between users Internet address and the + services used by the users) description which are part of the Tor + network at a time. + + o tracking-id - Analytics and tracking ID such as used in Google + Analytics or other analytic platform. + + o transaction - An object to describe a financial transaction. + + o url - url object describes an url along with its normalized field + (like extracted using faup parsing library) and its metadata. + + o vehicle - Vehicle object template to describe a vehicle + information and registration. + + o victim - Victim object describes the target of an attack or abuse. + + o virustotal-report - VirusTotal report. + + + + +Dulaunoy & Iklody Expires October 12, 2018 [Page 17] + +Internet-Draft MISP object template format April 2018 + + + o vulnerability - Vulnerability object describing a common + vulnerability enumeration which can describe published, + unpublished, under review or embargo vulnerability for software, + equipments or hardware. + + o whois - Whois records information for a domain name or an IP + address. + + o x509 - x509 object describing a X.509 certificate. + + o yabin - yabin.py generates Yara rules from function prologs, for + matching and hunting binaries. ref: . + + o yara - An object describing a YARA rule along with its version. 4. Acknowledgements @@ -535,29 +981,31 @@ Internet-Draft MISP object template format April 2018 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, - DOI 10.17487/RFC2119, March 1997, . + DOI 10.17487/RFC2119, March 1997, + . [RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally Unique IDentifier (UUID) URN Namespace", RFC 4122, - DOI 10.17487/RFC4122, July 2005, . + DOI 10.17487/RFC4122, July 2005, + . [RFC4627] Crockford, D., "The application/json Media Type for JavaScript Object Notation (JSON)", RFC 4627, - DOI 10.17487/RFC4627, July 2006, . + DOI 10.17487/RFC4627, July 2006, + . 5.2. Informative References - [MISP-O] MISP, , "MISP Objects - shared and common object - templates", . + [MISP-O] MISP, "MISP Objects - shared and common object templates", + . + + [MISP-O-DOC] + "MISP objects directory", 2018, + . - - -Dulaunoy & Iklody Expires October 12, 2018 [Page 10] +Dulaunoy & Iklody Expires October 12, 2018 [Page 18] Internet-Draft MISP object template format April 2018 @@ -613,4 +1061,4 @@ Authors' Addresses -Dulaunoy & Iklody Expires October 12, 2018 [Page 11] +Dulaunoy & Iklody Expires October 12, 2018 [Page 19] From ecd217c1385a7bed5f254cd5f95e465e1a483eaf Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 16:34:59 +0200 Subject: [PATCH 03/28] chg: [doc] latest version of the object released --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 897a3cc..f60b32f 100755 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ All the formats can be freely reused by everyone. * [misp-core-format](misp-core-format/raw.md.txt) ([markdown source](misp-core-format/raw.md)) which describes the core JSON format of MISP. Current Internet-Draft: [07](https://tools.ietf.org/html/draft-dulaunoy-misp-core-format) * [misp-taxonomy-format](misp-taxonomy-format/raw.md.txt) ([markdown source](misp-taxonomy-format/raw.md)) which describes the taxonomy JSON format of MISP. Current Internet-Draft: [07](https://tools.ietf.org/html/draft-dulaunoy-misp-taxonomy-format) * [misp-galaxy-format](misp-galaxy-format/raw.md.txt) which describes the [galaxy](https://github.com/MISP/misp-galaxy) template format used to expand the threat actor modelling of MISP. Current Internet-Draft: [06](https://datatracker.ietf.org/doc/draft-dulaunoy-misp-galaxy-format/) -* [misp-object-template-format](misp-object-template-format/raw.md.txt) which describes the [object](https://github.com/MISP/misp-objects) template format to add combinedand composite object to the MISP core format. Current Internet-Draft: [01](https://datatracker.ietf.org/doc/draft-dulaunoy-misp-object-template-format/) +* [misp-object-template-format](misp-object-template-format/raw.md.txt) which describes the [object](https://github.com/MISP/misp-objects) template format to add combinedand composite object to the MISP core format. Current Internet-Draft: [03](https://datatracker.ietf.org/doc/draft-dulaunoy-misp-object-template-format/) ## MISP Format in design phase and implemented in at least one software prototype From 8885fa2f494cf502cc9af3fa6a593a261e26e7fd Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 17:16:21 +0200 Subject: [PATCH 04/28] chg: [misp-core] JSON reference is now RFC 8259 - Comment from Carsten Bormann --- misp-core-format/raw.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index c2efc68..4e5aba7 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -64,7 +64,7 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. ## Overview -The MISP core format is in the JSON [@!RFC4627] format. In MISP, an event is composed of a single JSON object. +The MISP core format is in the JSON [@!RFC8259] format. In MISP, an event is composed of a single JSON object. A capitalized key (like Event, Org) represent a data model and a non-capitalised key is just an attribute. This nomenclature can support an implementation to represent the MISP format in another data structure. From a11090c9beb22f7207f5a6c53a4ca0526b25cfb1 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 17:18:56 +0200 Subject: [PATCH 05/28] chg: [misp-galaxy-format] JSON reference is now RFC 8259 - Comment from Carsten Bormann --- misp-galaxy-format/raw.md | 4 +- misp-galaxy-format/raw.md.txt | 158 +++++++++++++++++++++++----------- 2 files changed, 109 insertions(+), 53 deletions(-) diff --git a/misp-galaxy-format/raw.md b/misp-galaxy-format/raw.md index 8f745e4..21d71e6 100644 --- a/misp-galaxy-format/raw.md +++ b/misp-galaxy-format/raw.md @@ -74,11 +74,11 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. A cluster is composed of a value (**MUST**), a description (**OPTIONAL**) and metadata (**OPTIONAL**). -Clusters are represented as a JSON [@!RFC4627] dictionary. +Clusters are represented as a JSON [@!RFC8259] dictionary. ## Overview -The MISP galaxy format uses the JSON [@!RFC4627] format. Each galaxy is represented as a JSON object with meta information including the following fields: name, uuid, description, version, type, authors, source, values, category. +The MISP galaxy format uses the JSON [@!RFC8259] format. Each galaxy is represented as a JSON object with meta information including the following fields: name, uuid, description, version, type, authors, source, values, category. name defines the name of the galaxy. The name is represented as a string and **MUST** be present. The uuid represents the Universally Unique IDentifier (UUID) [@!RFC4122] of the object reference. The uuid **MUST** be preserved. For any updates or transfer of the same object reference. UUID version 4 is **RECOMMENDED** when assigning it to a new object reference and **MUST** be present. The description is represented as a string and **MUST** be present. The uuid is represented as a string and **MUST** be present. The version is represented as a decimal and **MUST** be present. The type is represented as a string and **MUST** be present and **MUST** match the name of the galaxy file. The source is represented as a string and **MUST** be present. Authors are represented as an array containing one or more authors and **MUST** be present. The category is represented as a string and **MUST** be present and describes the overall category of the galaxy such as tool or actor. diff --git a/misp-galaxy-format/raw.md.txt b/misp-galaxy-format/raw.md.txt index 593a2c3..c4e4d31 100755 --- a/misp-galaxy-format/raw.md.txt +++ b/misp-galaxy-format/raw.md.txt @@ -72,14 +72,14 @@ Table of Contents 2.2. values . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3. related . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.4. meta . . . . . . . . . . . . . . . . . . . . . . . . . . 4 - 3. JSON Schema . . . . . . . . . . . . . . . . . . . . . . . . . 8 + 3. JSON Schema . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1. MISP galaxy format - galaxy . . . . . . . . . . . . . . . 9 - 3.2. MISP galaxy format - clusters . . . . . . . . . . . . . . 9 - 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13 - 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 - 5.1. Normative References . . . . . . . . . . . . . . . . . . 13 - 5.2. Informative References . . . . . . . . . . . . . . . . . 13 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14 + 3.2. MISP galaxy format - clusters . . . . . . . . . . . . . . 10 + 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 14 + 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 14 + 5.1. Normative References . . . . . . . . . . . . . . . . . . 14 + 5.2. Informative References . . . . . . . . . . . . . . . . . 14 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15 1. Introduction @@ -119,11 +119,11 @@ Internet-Draft MISP galaxy format September 2018 A cluster is composed of a value (MUST), a description (OPTIONAL) and metadata (OPTIONAL). - Clusters are represented as a JSON [RFC4627] dictionary. + Clusters are represented as a JSON [RFC8259] dictionary. 2.1. Overview - The MISP galaxy format uses the JSON [RFC4627] format. Each galaxy + The MISP galaxy format uses the JSON [RFC8259] format. Each galaxy is represented as a JSON object with meta information including the following fields: name, uuid, description, version, type, authors, source, values, category. @@ -195,7 +195,8 @@ Internet-Draft MISP galaxy format September 2018 filenames, ransomnotes-refs, suspected-victims, suspected-state- sponsor, type-of-incident, target-category, cfr-suspected-victims, cfr-suspected-state-sponsor, cfr-type-of-incident, cfr-target- - category, attribution-confidence wherever applicable. + category, attribution-confidence, payment-method, price wherever + applicable. refs, synonyms SHALL be used to give further informations. refs is represented as an array containing one or more strings and SHALL be @@ -217,7 +218,6 @@ Internet-Draft MISP galaxy format September 2018 give further information in preventive-measure galaxy. complexity is represented by an enumerated value from a fixed vocabulary and SHALL be present. effectiveness is represented by an enumerated value from - a fixed vocabulary and SHALL be present. impact is represented by an @@ -226,6 +226,7 @@ Dulaunoy, et al. Expires March 24, 2019 [Page 4] Internet-Draft MISP galaxy format September 2018 + a fixed vocabulary and SHALL be present. impact is represented by an enumerated value from a fixed vocabulary and SHALL be present. possible_issues is represented as a string and SHOULD be present. @@ -274,7 +275,6 @@ Internet-Draft MISP galaxy format September 2018 - Dulaunoy, et al. Expires March 24, 2019 [Page 5] @@ -303,14 +303,16 @@ Internet-Draft MISP galaxy format September 2018 } encryption, extensions, ransomnotes, ransomnotes-filenames, - ransomnotes-refs MAY be used to give further information in - ransomware galaxy. encryption is represented as a string and SHALL be - present. extensions is represented as an array containing one or more - strings and SHALL be present. ransomnotes is represented as an array - containing one or more strings ans SHALL be present. ransomnotes- - filenames is represented as an array containing one or more strings - ans SHALL be present. ransomnotes-refs is represented as an array - containing one or more strings ans SHALL be present. + ransomnotes-refs, payment-method, price MAY be used to give further + information in ransomware galaxy. encryption is represented as a + string and SHALL be present. extensions is represented as an array + containing one or more strings and SHALL be present. ransomnotes is + represented as an array containing one or more strings ans SHALL be + present. ransomnotes-filenames is represented as an array containing + one or more strings ans SHALL be present. ransomnotes-refs is + represented as an array containing one or more strings ans SHALL be + present. payment-method is represented as a string and SHALL be + present. price is represented as a string and SHALL be present. Example use of the encryption, extensions, ransomnotes fields in the ransomware galaxy: @@ -331,8 +333,6 @@ Internet-Draft MISP galaxy format September 2018 - - Dulaunoy, et al. Expires March 24, 2019 [Page 6] Internet-Draft MISP galaxy format September 2018 @@ -356,11 +356,44 @@ Internet-Draft MISP galaxy format September 2018 "value": "Ryuk ransomware" } + Example use of the payment-method, price fields in the ransomware + galaxy: + +{ + "description": "This is most likely to affect English speaking users, since the note is written in English. English is understood worldwide, thus anyone can be harmed. The hacker spread the virus using email spam, fake updates, and harmful attachments. All your files are compromised including music, MS Office, Open Office, pictures, videos, shared online files etc..", + "meta": { + "date": "March 2017", + "encryption": "AES-128", + "extensions": [ + ".enc" + ], + "payment-method": "Bitcoin", + "price": "0.1", + "ransomnotes": [ + "Blocked Your computer has been blocked All your files are encrypted. To access your PC, you need to send to Bitcoin at the address below loading Step 1: Go to xxxxs : //wvw.coinbase.com/ siqnup Step 2: Create an account and follow the instructions Step 3: Go to the \"Buy Bitcoins\" section and then buy Bitcoin Step 4: Go to the \"Send\" section, enter the address above and the amount (0.1 Bitcoin) Step 5: Click on the button below to verify the payment, your files will be decrypted and the virus will disappear 'Check' If you try to bypass the lock, all files will be published on the Internet, as well as your login for all sites." + ], + "refs": [ + "https://id-ransomware.blogspot.co.il/2017/03/cryptomeister-ransomware.html" + ] + }, + "uuid": "4c76c845-c5eb-472c-93a1-4178f86c319b", + "value": "CryptoMeister Ransomware" +} + source-uuid, target-uuid SHALL be used to describe relationships. source-uuid and target-uuid represent the Universally Unique IDentifier (UUID) [RFC4122] of the value reference. source-uuid and target-uuid MUST be preserved. + + + + +Dulaunoy, et al. Expires March 24, 2019 [Page 7] + +Internet-Draft MISP galaxy format September 2018 + + Example use of the source-uuid, target-uuid fields in the mitre- enterprise-attack-relationship galaxy: @@ -387,17 +420,36 @@ Internet-Draft MISP galaxy format September 2018 exhaustive list of possible values for cfr-target-category includes "Private sector", "Government", "Civil society", "Military". - - -Dulaunoy, et al. Expires March 24, 2019 [Page 7] - -Internet-Draft MISP galaxy format September 2018 - - Example use of the cfr-suspected-victims, cfr-suspected-state- sponsor, cfr-type-of-incident, cfr-target-category fields in the threat-actor galaxy: + + + + + + + + + + + + + + + + + + + + + +Dulaunoy, et al. Expires March 24, 2019 [Page 8] + +Internet-Draft MISP galaxy format September 2018 + + { "meta": { "country": "CN", @@ -441,17 +493,19 @@ Internet-Draft MISP galaxy format September 2018 formats. The main format is the MISP galaxy format used for the clusters. +3.1. MISP galaxy format - galaxy -Dulaunoy, et al. Expires March 24, 2019 [Page 8] + + + +Dulaunoy, et al. Expires March 24, 2019 [Page 9] Internet-Draft MISP galaxy format September 2018 -3.1. MISP galaxy format - galaxy - { "$schema": "http://json-schema.org/schema#", "title": "Validator for misp-galaxies - Galaxies", @@ -498,16 +552,16 @@ Internet-Draft MISP galaxy format September 2018 { "$schema": "http://json-schema.org/schema#", "title": "Validator for misp-galaxies - Clusters", + "id": "https://www.github.com/MISP/misp-galaxies/schema_clusters.json", + "type": "object", -Dulaunoy, et al. Expires March 24, 2019 [Page 9] +Dulaunoy, et al. Expires March 24, 2019 [Page 10] Internet-Draft MISP galaxy format September 2018 - "id": "https://www.github.com/MISP/misp-galaxies/schema_clusters.json", - "type": "object", "additionalProperties": false, "properties": { "description": { @@ -554,16 +608,16 @@ Internet-Draft MISP galaxy format September 2018 "type": "object" }, "properties": { + "dest-uuid": { + "type": "string" -Dulaunoy, et al. Expires March 24, 2019 [Page 10] +Dulaunoy, et al. Expires March 24, 2019 [Page 11] Internet-Draft MISP galaxy format September 2018 - "dest-uuid": { - "type": "string" }, "type": { "type": "string" @@ -610,16 +664,16 @@ Internet-Draft MISP galaxy format September 2018 "type": "string" }, "refs": { + "type": "array", + "uniqueItems": true, -Dulaunoy, et al. Expires March 24, 2019 [Page 11] +Dulaunoy, et al. Expires March 24, 2019 [Page 12] Internet-Draft MISP galaxy format September 2018 - "type": "array", - "uniqueItems": true, "items": { "type": "string" } @@ -666,16 +720,16 @@ Internet-Draft MISP galaxy format September 2018 "type": "array", "uniqueItems": true, "items": { + "type": "string" + } -Dulaunoy, et al. Expires March 24, 2019 [Page 12] +Dulaunoy, et al. Expires March 24, 2019 [Page 13] Internet-Draft MISP galaxy format September 2018 - "type": "string" - } } }, "required": [ @@ -710,10 +764,10 @@ Internet-Draft MISP galaxy format September 2018 DOI 10.17487/RFC4122, July 2005, . - [RFC4627] Crockford, D., "The application/json Media Type for - JavaScript Object Notation (JSON)", RFC 4627, - DOI 10.17487/RFC4627, July 2006, - . + [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", STD 90, RFC 8259, + DOI 10.17487/RFC8259, December 2017, + . 5.2. Informative References @@ -725,7 +779,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 13] + + +Dulaunoy, et al. Expires March 24, 2019 [Page 14] Internet-Draft MISP galaxy format September 2018 @@ -781,7 +837,7 @@ Authors' Addresses -Dulaunoy, et al. Expires March 24, 2019 [Page 14] +Dulaunoy, et al. Expires March 24, 2019 [Page 15] Internet-Draft MISP galaxy format September 2018 @@ -837,4 +893,4 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 15] +Dulaunoy, et al. Expires March 24, 2019 [Page 16] From 56ee9b01a56b99b4e952190c46039682a02f4893 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 17:20:09 +0200 Subject: [PATCH 06/28] chg: [misp-object-template] JSON reference is now RFC 8259 - Comment from Carsten Bormann --- misp-object-template-format/raw.md | 2 +- misp-object-template-format/raw.md.txt | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/misp-object-template-format/raw.md b/misp-object-template-format/raw.md index 1e08e20..cccab4c 100755 --- a/misp-object-template-format/raw.md +++ b/misp-object-template-format/raw.md @@ -67,7 +67,7 @@ MISP object template elements consist of an object\_relation (**MUST**), a type ## Overview -The MISP object template format uses the JSON [@!RFC4627] format. Each template is represented as a JSON object with meta information including the following fields: uuid, requiredOneOf, description, version, meta-category, name. +The MISP object template format uses the JSON [@!RFC8259] format. Each template is represented as a JSON object with meta information including the following fields: uuid, requiredOneOf, description, version, meta-category, name. ### Object Template diff --git a/misp-object-template-format/raw.md.txt b/misp-object-template-format/raw.md.txt index 9e8aa98..fc22dd1 100755 --- a/misp-object-template-format/raw.md.txt +++ b/misp-object-template-format/raw.md.txt @@ -136,7 +136,7 @@ Internet-Draft MISP object template format April 2018 2.1. Overview - The MISP object template format uses the JSON [RFC4627] format. Each + The MISP object template format uses the JSON [RFC8259] format. Each template is represented as a JSON object with meta information including the following fields: uuid, requiredOneOf, description, version, meta-category, name. @@ -989,10 +989,10 @@ Internet-Draft MISP object template format April 2018 DOI 10.17487/RFC4122, July 2005, . - [RFC4627] Crockford, D., "The application/json Media Type for - JavaScript Object Notation (JSON)", RFC 4627, - DOI 10.17487/RFC4627, July 2006, - . + [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", STD 90, RFC 8259, + DOI 10.17487/RFC8259, December 2017, + . 5.2. Informative References From d3d9f8a3c8fd1973cba18540a6c6a9a941b610dd Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 17:21:15 +0200 Subject: [PATCH 07/28] chg: [misp-taxonomy-format] JSON reference is now RFC 8259 - Comment from Carsten Bormann --- misp-taxonomy-format/raw.md | 2 +- misp-taxonomy-format/raw.md.txt | 220 ++++++++++++++++++++------------ 2 files changed, 139 insertions(+), 83 deletions(-) diff --git a/misp-taxonomy-format/raw.md b/misp-taxonomy-format/raw.md index 458654c..d71a069 100755 --- a/misp-taxonomy-format/raw.md +++ b/misp-taxonomy-format/raw.md @@ -82,7 +82,7 @@ to describe machine tag (aka triple tag) vocabularies. ## Overview -The MISP taxonomy format uses the JSON [@!RFC4627] format. Each namespace is represented as a JSON object with meta information including the following fields: namespace, description, version, type. +The MISP taxonomy format uses the JSON [@!RFC8259] format. Each namespace is represented as a JSON object with meta information including the following fields: namespace, description, version, type. namespace defines the overall namespace of the machine tag. The namespace is represented as a string and **MUST** be present. The description is represented as a string and **MUST** be present. A version is represented as a unsigned integer **MUST** be present. A type defines where a specific taxonomy is applicable and a type can be applicable at event, user or org level. The type is represented as an array containing one or more type and **SHOULD** be present. If a type is not mentioned, by default, the taxonomy is applicable at event level only. An exclusive boolean property **MAY** be present and defines at namespace level if the predicates are mutually exclusive. diff --git a/misp-taxonomy-format/raw.md.txt b/misp-taxonomy-format/raw.md.txt index 36f735a..49a7be7 100644 --- a/misp-taxonomy-format/raw.md.txt +++ b/misp-taxonomy-format/raw.md.txt @@ -79,13 +79,13 @@ Table of Contents 4.1. Admiralty Scale Taxonomy . . . . . . . . . . . . . . . . 7 4.2. Open Source Intelligence - Classification . . . . . . . . 9 4.3. Available taxonomies in the public directory . . . . . . 11 - 5. JSON Schema . . . . . . . . . . . . . . . . . . . . . . . . . 19 - 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 22 - 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 - 7.1. Normative References . . . . . . . . . . . . . . . . . . 22 - 7.2. Informative References . . . . . . . . . . . . . . . . . 22 + 5. JSON Schema . . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 23 + 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 + 7.1. Normative References . . . . . . . . . . . . . . . . . . 23 + 7.2. Informative References . . . . . . . . . . . . . . . . . 23 7.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 23 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 1. Introduction @@ -145,7 +145,7 @@ Internet-Draft MISP taxonomy format November 2017 2.1. Overview - The MISP taxonomy format uses the JSON [RFC4627] format. Each + The MISP taxonomy format uses the JSON [RFC8259] format. Each namespace is represented as a JSON object with meta information including the following fields: namespace, description, version, type. @@ -153,7 +153,7 @@ Internet-Draft MISP taxonomy format November 2017 namespace defines the overall namespace of the machine tag. The namespace is represented as a string and MUST be present. The description is represented as a string and MUST be present. A - version is represented as a decimal and MUST be present. A type + version is represented as a unsigned integer MUST be present. A type defines where a specific taxonomy is applicable and a type can be applicable at event, user or org level. The type is represented as an array containing one or more type and SHOULD be present. If a @@ -683,11 +683,22 @@ Internet-Draft MISP taxonomy format November 2017 to support analysts to perform their analysis to get crowdsourced support when using threat intelligence sharing platform like MISP. + common-taxonomy: + The Common Taxonomy for Law Enforcement and The National Network + of CSIRTs bridges the gap between the CSIRTs and international Law + Enforcement communities by adding a legislative framework to + facilitate the harmonisation of incident reporting to competent + authorities, the development of useful statistics and sharing + information within the entire cybercrime ecosystem. + copine-scale: The COPINE Scale is a rating system created in Ireland and used in the United Kingdom to categorise the severity of images of child sex abuse. + cryptocurrency-threat: + Threats targetting cryptocurrency, based on CipherTrace report. + csirt_case_classification: FIRST CSIRT Case Classification. @@ -701,7 +712,24 @@ Internet-Draft MISP taxonomy format November 2017 of cyber adversaries. + data-classification: + Data classification for data potentially at risk of exfiltration + based on table 2.1 of Solving Cyber Risk book. + + dcso-sharing: + DCSO Sharing Taxonomy to classify certain types of MISP events + using the DCSO Event Guide + ddos: + + + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 13] + +Internet-Draft MISP taxonomy format November 2017 + + Distributed Denial of Service - or short: DDoS - taxonomy supports the description of Denial of Service attacks and especially the types they belong too. @@ -723,16 +751,13 @@ Internet-Draft MISP taxonomy format November 2017 ISM (Information Security Marking Metadata) V13 as described by DNI.gov (Director of National Intelligence - US). - - -Dulaunoy & Iklody Expires June 2, 2018 [Page 13] - -Internet-Draft MISP taxonomy format November 2017 - - domain-abuse: Taxonomy to tag domain names used for cybercrime. + drugs: + A taxonomy based on the superclass and class of drugs, based on + + economical-impact: Economical impact is a taxonomy to describe the financial impact as positive or negative gain to the tagged information. @@ -753,6 +778,14 @@ Internet-Draft MISP taxonomy format November 2017 (6.2.(a)) and JP 2-0, Joint Intelligence. eu-marketop-and-publicadmin: + + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 14] + +Internet-Draft MISP taxonomy format November 2017 + + Market operators and public administrations that must comply to some notifications requirements under EU NIS directive. @@ -764,7 +797,9 @@ Internet-Draft MISP taxonomy format November 2017 designated by a EU security classification, the unauthorised disclosure of which could cause varying degrees of prejudice to the interests of the European Union or of one or more of the - Member States as described in CELEX 32013D0488 + Member States as described in COUNCIL DECISION of 23 September + 2013 on the security rules for protecting EU classified + information europol-event: EUROPOL type of events taxonomy. @@ -778,19 +813,11 @@ Internet-Draft MISP taxonomy format November 2017 uncertainty. event-classification: - - - -Dulaunoy & Iklody Expires June 2, 2018 [Page 14] - -Internet-Draft MISP taxonomy format November 2017 - - Event Classification. exercise: Exercise is a taxonomy to describe if the information is part of - one or more cyber or crisis exercise + one or more cyber or crisis exercise. false-positive: This taxonomy aims to ballpark the expected amount of false @@ -799,7 +826,22 @@ Internet-Draft MISP taxonomy format November 2017 file-type: List of known file types. + flesch-reading-ease: + Flesch Reading Ease is a revised system for determining the + comprehension difficulty of written material. The scoring of the + flesh score can have a maximum of 121.22 and there is no limit on + how low a score can be (negative score are valid). + fpf: + + + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 15] + +Internet-Draft MISP taxonomy format November 2017 + + The Future of Privacy Forum (FPF) visual guide to practical de- identification [1] taxonomy is used to evaluate the degree of identifiability of personal data and the types of pseudonymous @@ -833,15 +875,6 @@ Internet-Draft MISP taxonomy format November 2017 Christian Seifert, Ian Welch, Peter Komisarczuk, 'Taxonomy of Honeypots', Technical Report CS-TR-06/12, VICTORIA UNIVERSITY OF WELLINGTON, School of Mathematical and Computing Sciences, June - - - - -Dulaunoy & Iklody Expires June 2, 2018 [Page 15] - -Internet-Draft MISP taxonomy format November 2017 - - 2006, @@ -858,10 +891,20 @@ Internet-Draft MISP taxonomy format November 2017 taxonomy is inspired from NASA Incident Response and Management Handbook. + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 16] + +Internet-Draft MISP taxonomy format November 2017 + + infoleak: A taxonomy describing information leaks and especially information classified as being potentially leaked. + information-security-data-source: + Taxonomy to classify the information security data sources + information-security-indicators: Information security indicators have been standardized by the ETSI Industrial Specification Group (ISG) ISI. These indicators @@ -890,14 +933,6 @@ Internet-Draft MISP taxonomy format November 2017 Malware Capabilities based on MAEC 5.0 maec-malware-obfuscation-methods: - - - -Dulaunoy & Iklody Expires June 2, 2018 [Page 16] - -Internet-Draft MISP taxonomy format November 2017 - - Obfuscation methods used by malware based on MAEC 5.0 malware_classification: @@ -910,6 +945,15 @@ Internet-Draft MISP taxonomy format November 2017 MONARC threat taxonomy. ms-caro-malware: + + + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 17] + +Internet-Draft MISP taxonomy format November 2017 + + Malware Type and Platform classification based on Microsoft's implementation of the Computer Antivirus Research Organization (CARO) Naming Scheme and Malware Terminology. @@ -946,14 +990,6 @@ Internet-Draft MISP taxonomy format November 2017 to help provide a common lexicon when discussing incidents. This priority assignment drives NCCIC urgency, pre-approved incident response offerings, reporting requirements, and recommendations - - - -Dulaunoy & Iklody Expires June 2, 2018 [Page 17] - -Internet-Draft MISP taxonomy format November 2017 - - for leadership escalation. Generally, incident priority distribution should follow a similar pattern to the graph below. Based on . - [RFC4627] Crockford, D., "The application/json Media Type for - JavaScript Object Notation (JSON)", RFC 4627, - DOI 10.17487/RFC4627, July 2006, - . + [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", STD 90, RFC 8259, + DOI 10.17487/RFC8259, December 2017, + . 7.2. Informative References @@ -1223,22 +1276,20 @@ Internet-Draft MISP taxonomy format November 2017 [MISP-T] MISP, "MISP Taxonomies - shared and common vocabularies of tags", . - - - - - - -Dulaunoy & Iklody Expires June 2, 2018 [Page 22] - -Internet-Draft MISP taxonomy format November 2017 - - 7.3. URIs [1] https://fpf.org/2016/04/25/a-visual-guide-to-practical-data-de- identification/ + + + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 23] + +Internet-Draft MISP taxonomy format November 2017 + + Authors' Addresses Alexandre Dulaunoy @@ -1285,4 +1336,9 @@ Authors' Addresses -Dulaunoy & Iklody Expires June 2, 2018 [Page 23] + + + + + +Dulaunoy & Iklody Expires June 2, 2018 [Page 24] From 77efda923c39d4757567c1ecad308519834c7565 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 17:22:06 +0200 Subject: [PATCH 08/28] chg: [misp-query-format] JSON reference is now RFC 8259 - Comment from Carsten Bormann --- misp-query-format/raw.md | 2 +- misp-query-format/raw.md.txt | 332 +++++++++++++++++++++++------------ 2 files changed, 223 insertions(+), 111 deletions(-) diff --git a/misp-query-format/raw.md b/misp-query-format/raw.md index 29acbe5..df3e651 100755 --- a/misp-query-format/raw.md +++ b/misp-query-format/raw.md @@ -65,7 +65,7 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. ## Overview -The MISP query format is in the JSON [@!RFC4627] format. +The MISP query format is in the JSON [@!RFC8259] format. ## query format criteria diff --git a/misp-query-format/raw.md.txt b/misp-query-format/raw.md.txt index 1ad03a4..043fc9a 100644 --- a/misp-query-format/raw.md.txt +++ b/misp-query-format/raw.md.txt @@ -68,23 +68,53 @@ Internet-Draft MISP query format October 2018 Table of Contents - 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Conventions and Terminology . . . . . . . . . . . . . . . 3 2. Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2. query format criteria . . . . . . . . . . . . . . . . . . 3 2.2.1. returnFormat . . . . . . . . . . . . . . . . . . . . 3 - 2.2.2. limit . . . . . . . . . . . . . . . . . . . . . . . . 3 - 2.2.3. page . . . . . . . . . . . . . . . . . . . . . . . . 3 + 2.2.2. limit . . . . . . . . . . . . . . . . . . . . . . . . 4 + 2.2.3. page . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2.4. value . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2.5. type . . . . . . . . . . . . . . . . . . . . . . . . 4 - 2.2.6. category . . . . . . . . . . . . . . . . . . . . . . 4 - 3. Security Considerations . . . . . . . . . . . . . . . . . . . 4 - 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 - 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 5 - 5.1. Normative References . . . . . . . . . . . . . . . . . . 5 - 5.2. Informative References . . . . . . . . . . . . . . . . . 5 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.2.6. category . . . . . . . . . . . . . . . . . . . . . . 5 + 2.2.7. org . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.2.8. tags . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.2.9. quickfilter . . . . . . . . . . . . . . . . . . . . . 5 + 2.2.10. from . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.2.11. to . . . . . . . . . . . . . . . . . . . . . . . . . 6 + 2.2.12. last . . . . . . . . . . . . . . . . . . . . . . . . 6 + 2.2.13. eventid . . . . . . . . . . . . . . . . . . . . . . . 6 + 2.2.14. withAttachments . . . . . . . . . . . . . . . . . . . 6 + 2.2.15. uuid . . . . . . . . . . . . . . . . . . . . . . . . 6 + 2.2.16. publish_timestamp . . . . . . . . . . . . . . . . . . 6 + 2.2.17. timestamp . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2.18. published . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2.19. enforceWarninglist . . . . . . . . . . . . . . . . . 7 + 2.2.20. to_ids . . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2.21. deleted . . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2.22. includeEventUuid . . . . . . . . . . . . . . . . . . 7 + 2.2.23. event_timestamp . . . . . . . . . . . . . . . . . . . 7 + 2.2.24. sgReferenceOnly . . . . . . . . . . . . . . . . . . . 7 + 2.2.25. eventinfo . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2.26. searchall . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2.27. requested_attributes . . . . . . . . . . . . . . . . 7 + 2.2.28. includeContext . . . . . . . . . . . . . . . . . . . 7 + 3. Security Considerations . . . . . . . . . . . . . . . . . . . 7 + 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 + 5. References . . . . . . . . . . . . . . . . . . . . . . . . . 7 + 5.1. Normative References . . . . . . . . . . . . . . . . . . 8 + 5.2. Informative References . . . . . . . . . . . . . . . . . 8 + + + +Dulaunoy & Iklody Expires April 11, 2019 [Page 2] + +Internet-Draft MISP query format October 2018 + + + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8 1. Introduction @@ -103,17 +133,6 @@ Table of Contents query format and how the query can be perform against a REST interface. - - - - - - -Dulaunoy & Iklody Expires April 11, 2019 [Page 2] - -Internet-Draft MISP query format October 2018 - - 1.1. Conventions and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", @@ -124,7 +143,7 @@ Internet-Draft MISP query format October 2018 2.1. Overview - The MISP query format is in the JSON [RFC4627] format. + The MISP query format is in the JSON [RFC8259] format. 2.2. query format criteria @@ -134,18 +153,36 @@ Internet-Draft MISP query format October 2018 format. MISP allows multiple format (depending of the configuration): - +----------+------------------------------------------------+ - | value | Description | - +----------+------------------------------------------------+ - | json | MISP JSON core format as described in [MISP-C] | - | xml | MISP XML format | - | openioc | OpenIOC format | - | suricata | Suricata NIDS format | - | snort | Snort NIDS format | - | csv | CSV format | - | rpz | Response policy zone format | - | text | Raw value list format | - +----------+------------------------------------------------+ + + + + + + + + + + + + +Dulaunoy & Iklody Expires April 11, 2019 [Page 3] + +Internet-Draft MISP query format October 2018 + + + +----------+-------------------------------------------------+ + | value | Description | + +----------+-------------------------------------------------+ + | json | MISP JSON core format as described in [MISP-C] | + | xml | MISP XML format | + | openioc | OpenIOC format | + | suricata | Suricata NIDS format | + | snort | Snort NIDS format | + | csv | CSV format | + | rpz | Response policy zone format | + | text | Raw value list format | + | cache | MISP cache format (hashed values of attributes) | + +----------+-------------------------------------------------+ 2.2.2. limit @@ -162,35 +199,38 @@ Internet-Draft MISP query format October 2018 starting with offset (limit * page) + 1 and ending with (limit * (page+1)). - - - -Dulaunoy & Iklody Expires April 11, 2019 [Page 3] - -Internet-Draft MISP query format October 2018 - - 2.2.4. value value MAY be present. If set, the returned data set will be filtered - on the attribute value field. value MAY be a string or a sub-string, - the latter of which start with, ends with or is encapsulated in + on the attribute value field. value MUST be a string or a sub-string, + the latter of which starts with, ends with or is encapsulated in wildcard (\%) characters. 2.2.5. type type MAY be present. If set, the returned data set will be filtered - on the attribute type field. type MAY be a string or a sub-string, - the latter of which start with, ends with or is encapsulated in + on the attribute type field. type MUST be a string or a sub-string, + the latter of which starts with, ends with or is encapsulated in wildcard (\%) characters. The list of valid attribute types is described in the MISP core format [MISP-C] in the attribute type section. + + + + + + +Dulaunoy & Iklody Expires April 11, 2019 [Page 4] + +Internet-Draft MISP query format October 2018 + + 2.2.6. category category MAY be present. If set, the returned data set will be - filtered on the attribute category field. category MAY be a string or - a sub-string, the latter of which start with, ends with or is + filtered on the attribute category field. category MUST be a string + or a sub-string, the latter of which starts with, ends with or is encapsulated in wildcard (\%) characters. The list of valid categories is described in the MISP core format [MISP-C] in the attribute type section. @@ -204,6 +244,124 @@ Internet-Draft MISP query format October 2018 "category": "Financial fraud" } +2.2.7. org + + org MAY be present. If set, the returned data set will be filtered + by the organisation identifier (local ID of the instance). org MUST + be the identifier of the organisation in a string format. + +2.2.8. tags + + tags MAY be present. If set, the returned data set will be filtered + by tags. tags MUST be a string or a sub-string, the latter of which + starts with, ends with or is encapsulated in wildcard (\%) + characters. + + { + "returnFormat": "cache", + "limit": "100", + "tags": ["tlp:red", "%private%"] + } + +2.2.9. quickfilter + +2.2.10. from + + from MAY be present. If set, the returned data set will be filtered + from a starting date. from MUST be a string represented in the format + year-month-date. + + + + + + + +Dulaunoy & Iklody Expires April 11, 2019 [Page 5] + +Internet-Draft MISP query format October 2018 + + + { + "returnFormat": "json", + "limit": "100", + "tags": ["tlp:amber"], + "from": "2018-09-02", + "to": "2018-10-01" + } + +2.2.11. to + + to MAY be present. If set, the returned data set will be filtered + until the specified date. from MUST be a string represented in the + format year-month-date. + +2.2.12. last + + last MAY be present. If set, the returned data set will be filtered + in the number of days, hours or minutes defined (such as 5d, 12h or + 30m). last MUST be a string represented in the format expressing + days, hours or minutes. + +2.2.13. eventid + + eventid MAY be present. If set, the returned data set will be + filtered to a specific event. eventid MUST be a string representing + the event id as an integer. + + { + "returnFormat": "json", + "eventid": 1 + } + +2.2.14. withAttachments + + withAttachments MAY be present. If set to True (1), the returned + data set will include the attachment(s) matching the query. + withAttachments MUST be an integer set as 1 (True) to include the + attachment(s). If not, the attachment(s) won't be included in the + results. + +2.2.15. uuid + +2.2.16. publish_timestamp + + + + + + + + +Dulaunoy & Iklody Expires April 11, 2019 [Page 6] + +Internet-Draft MISP query format October 2018 + + +2.2.17. timestamp + +2.2.18. published + +2.2.19. enforceWarninglist + +2.2.20. to_ids + +2.2.21. deleted + +2.2.22. includeEventUuid + +2.2.23. event_timestamp + +2.2.24. sgReferenceOnly + +2.2.25. eventinfo + +2.2.26. searchall + +2.2.27. requested_attributes + +2.2.28. includeContext + 3. Security Considerations MISP threat intelligence instances might contain sensitive or @@ -216,16 +374,6 @@ Internet-Draft MISP query format October 2018 standard threat information that might already include malicious intended inputs. - - - - - -Dulaunoy & Iklody Expires April 11, 2019 [Page 4] - -Internet-Draft MISP query format October 2018 - - 4. Acknowledgements The authors wish to thank all the MISP community who are supporting @@ -235,6 +383,17 @@ Internet-Draft MISP query format October 2018 5. References + + + + + + +Dulaunoy & Iklody Expires April 11, 2019 [Page 7] + +Internet-Draft MISP query format October 2018 + + 5.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate @@ -242,10 +401,10 @@ Internet-Draft MISP query format October 2018 DOI 10.17487/RFC2119, March 1997, . - [RFC4627] Crockford, D., "The application/json Media Type for - JavaScript Object Notation (JSON)", RFC 4627, - DOI 10.17487/RFC4627, July 2006, - . + [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", STD 90, RFC 8259, + DOI 10.17487/RFC8259, December 2017, + . 5.2. Informative References @@ -267,21 +426,6 @@ Authors' Addresses Email: alexandre.dulaunoy@circl.lu - - - - - - - - - - -Dulaunoy & Iklody Expires April 11, 2019 [Page 5] - -Internet-Draft MISP query format October 2018 - - Andras Iklody Computer Incident Response Center Luxembourg 16, bd d'Avranches @@ -301,36 +445,4 @@ Internet-Draft MISP query format October 2018 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Dulaunoy & Iklody Expires April 11, 2019 [Page 6] +Dulaunoy & Iklody Expires April 11, 2019 [Page 8] From d847ea8701add63cda3fc12205bd6f8137fd4012 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Sun, 23 Jun 2019 17:23:06 +0200 Subject: [PATCH 09/28] chg: [misp-warninglist] JSON reference is now RFC 8259 - Comment from Carsten Bormann --- misp-warninglist-format/raw.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/misp-warninglist-format/raw.md b/misp-warninglist-format/raw.md index 9683398..1bbd8d2 100644 --- a/misp-warninglist-format/raw.md +++ b/misp-warninglist-format/raw.md @@ -72,11 +72,11 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. # Format -Warninglists are represented as a JSON [@!RFC4627] dictionary. +Warninglists are represented as a JSON [@!RFC8259] dictionary. ## Overview -The MISP warninglist format uses the JSON [@!RFC4627] format. Each warninglist is represented as a JSON object with meta information including the following fields: name, description, version, type, matching_attributes, list. +The MISP warninglist format uses the JSON [@!RFC8259] format. Each warninglist is represented as a JSON object with meta information including the following fields: name, description, version, type, matching_attributes, list. name defines the name of the warninglist. The name is represented as a string and **MUST** be present. The description is represented as a string and **MUST** be present. The version is represented as a decimal and **MUST** be present. matching_attributes is represented as an array containing one or more values and is **RECOMMENDED**. type is represented as a string from an non exaustive list and **MUST** be present. From cd6174e3acd4263f579187395bda2ab0243f292e Mon Sep 17 00:00:00 2001 From: mokaddem Date: Mon, 24 Jun 2019 10:42:19 +0200 Subject: [PATCH 10/28] chg: precision and example about the ISO 8601 datetime for fs/ls --- misp-core-format/raw.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index 975e8cb..6c2afb3 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -280,7 +280,7 @@ A MISP document **MUST** at least includes category-type-value triplet described "SharingGroup": [], "ShadowAttribute": [], "RelatedAttribute": [], - "first_seen": null, + "first_seen": "2019-06-02T22:14:28.711954+00:00", "last_seen": null } ~~~~ @@ -454,13 +454,13 @@ value is represented by a JSON string. value **MUST** be present. #### first_seen -first_seen represents a reference time when the attribute was first seen. first_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. +first_seen represents a reference time when the attribute was first seen. first_seen is expressed as an ISO 8601 datetime up to the micro-second and supporting time zone. first_seen is represented as a JSON string. first_seen **SHALL** be present. #### last_seen -last_seen represents a reference time when the attribute was last seen. last_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. +last_seen represents a reference time when the attribute was last seen. last_seen is expressed as an ISO 8601 datetime up to the micro-second and supporting time zone. last_seen is represented as a JSON string. last_seen **SHALL** be present. @@ -492,7 +492,7 @@ They are similar in structure to Attributes but additionally carry a reference t "name": "MISP", "uuid": "568cce5a-0c80-412b-8fdf-1ffac0a83869" }, - "first_seen": null, + "first_seen": "2019-06-02T22:14:28.711954+00:00", "last_seen": null } ~~~~ @@ -638,13 +638,13 @@ data is represented by a JSON string in base64 encoding. data **MUST** be set fo #### first_seen -first_seen represents a reference time when the attribute was first seen. first_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. +first_seen represents a reference time when the attribute was first seen. first_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. first_seen is represented as a JSON string. first_seen **SHALL** be present. #### last_seen -last_seen represents a reference time when the attribute was last seen. last_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. +last_seen represents a reference time when the attribute was last seen. last_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. last_seen is represented as a JSON string. last_seen **SHALL** be present. @@ -725,7 +725,7 @@ A MISP document containing an Object **MUST** contain a name, a meta-category, a "first_seen": null, "last_seen": null }, - "first_seen": null, + "first_seen": "2019-06-02T22:14:28.711954+00:00", "last_seen": null ] } @@ -836,13 +836,13 @@ Each attribute in an object **MUST** contain the parent event's ID in the event_ #### first_seen -first_seen represents a reference time when the object was first seen. first_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. +first_seen represents a reference time when the object was first seen. first_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. first_seen is represented as a JSON string. first_seen **SHALL** be present. #### last_seen -last_seen represents a reference time when the object was last seen. last_seen is expressed in micro-seconds (decimal) since 1st of January 1970 (Unix timestamp). The time zone **MUST** be UTC. +last_seen represents a reference time when the object was last seen. last_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. last_seen is represented as a JSON string. last_seen **SHALL** be present. From b46942e0a23ad6387f97e4f95f92329cd4f01d98 Mon Sep 17 00:00:00 2001 From: mokaddem Date: Mon, 24 Jun 2019 16:05:11 +0200 Subject: [PATCH 11/28] chg: *-seen rephrasing --- misp-core-format/raw.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index 6c2afb3..42bee80 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -454,15 +454,15 @@ value is represented by a JSON string. value **MUST** be present. #### first_seen -first_seen represents a reference time when the attribute was first seen. first_seen is expressed as an ISO 8601 datetime up to the micro-second and supporting time zone. +first_seen represents a reference time when the attribute was first seen. first_seen is expressed as an ISO 8601 datetime up to the micro-second with time zone support. -first_seen is represented as a JSON string. first_seen **SHALL** be present. +first_seen is represented as a JSON string. first_seen **MAY** be present. #### last_seen -last_seen represents a reference time when the attribute was last seen. last_seen is expressed as an ISO 8601 datetime up to the micro-second and supporting time zone. +last_seen represents a reference time when the attribute was last seen. last_seen is expressed as an ISO 8601 datetime up to the micro-second with time zone support. -last_seen is represented as a JSON string. last_seen **SHALL** be present. +last_seen is represented as a JSON string. last_seen **MAY** be present. ## ShadowAttribute @@ -638,15 +638,15 @@ data is represented by a JSON string in base64 encoding. data **MUST** be set fo #### first_seen -first_seen represents a reference time when the attribute was first seen. first_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. +first_seen represents a reference time when the attribute was first seen. first_seen as an ISO 8601 datetime up to the micro-second with time zone support. -first_seen is represented as a JSON string. first_seen **SHALL** be present. +first_seen is represented as a JSON string. first_seen **MAY** be present. #### last_seen -last_seen represents a reference time when the attribute was last seen. last_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. +last_seen represents a reference time when the attribute was last seen. last_seen as an ISO 8601 datetime up to the micro-second with time zone support. -last_seen is represented as a JSON string. last_seen **SHALL** be present. +last_seen is represented as a JSON string. last_seen **MAY** be present. ### Org @@ -838,13 +838,13 @@ Each attribute in an object **MUST** contain the parent event's ID in the event_ first_seen represents a reference time when the object was first seen. first_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. -first_seen is represented as a JSON string. first_seen **SHALL** be present. +first_seen is represented as a JSON string. first_seen **MAY** be present. #### last_seen last_seen represents a reference time when the object was last seen. last_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. -last_seen is represented as a JSON string. last_seen **SHALL** be present. +last_seen is represented as a JSON string. last_seen **MAY** be present. ## Object References From 60d1b1dad82c53d96be32c6ab220d10d8ae69649 Mon Sep 17 00:00:00 2001 From: mokaddem Date: Mon, 24 Jun 2019 16:06:39 +0200 Subject: [PATCH 12/28] chg: *-seen rephrasing 2 --- misp-core-format/raw.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index 42bee80..6c4f268 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -836,13 +836,13 @@ Each attribute in an object **MUST** contain the parent event's ID in the event_ #### first_seen -first_seen represents a reference time when the object was first seen. first_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. +first_seen represents a reference time when the object was first seen. first_seen as an ISO 8601 datetime up to the micro-second with time zone support. first_seen is represented as a JSON string. first_seen **MAY** be present. #### last_seen -last_seen represents a reference time when the object was last seen. last_seen as an ISO 8601 datetime up to the micro-second and supporting time zone. +last_seen represents a reference time when the object was last seen. last_seen as an ISO 8601 datetime up to the micro-second with time zone support. last_seen is represented as a JSON string. last_seen **MAY** be present. From 515467efa2955ddebaa9f6e7f7ac2958e8a72cfb Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Tue, 16 Jul 2019 07:26:50 +0200 Subject: [PATCH 13/28] chg: [misp-galaxy-format] updated to the latest version of mmark format --- misp-galaxy-format/raw.md | 103 +++++++++++++++++++------------------- 1 file changed, 52 insertions(+), 51 deletions(-) diff --git a/misp-galaxy-format/raw.md b/misp-galaxy-format/raw.md index 21d71e6..c2a09f6 100644 --- a/misp-galaxy-format/raw.md +++ b/misp-galaxy-format/raw.md @@ -1,55 +1,56 @@ -% Title = "MISP galaxy format" -% abbrev = "MISP galaxy format" -% category = "info" -% docName = "draft-dulaunoy-misp-galaxy-format" -% ipr= "trust200902" -% area = "Security" -% -% date = 2018-09-20T00:00:00Z -% -% [[author]] -% initials="A." -% surname="Dulaunoy" -% fullname="Alexandre Dulaunoy" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "alexandre.dulaunoy@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = "16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" -% [[author]] -% initials="A." -% surname="Iklody" -% fullname="Andras Iklody" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "andras.iklody@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = " 16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" -% [[author]] -% initials="D." -% surname="Servili" -% fullname="Deborah Servili" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "deborah.servili@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = " 16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" +%%% +Title = "MISP galaxy format" +abbrev = "MISP galaxy format" +category = "info" +docName = "draft-dulaunoy-misp-galaxy-format" +ipr= "trust200902" +area = "Security" +date = 2018-09-20T00:00:00Z + +[[author]] +initials="A." +surname="Dulaunoy" +fullname="Alexandre Dulaunoy" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "alexandre.dulaunoy@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = "16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +[[author]] +initials="A." +surname="Iklody" +fullname="Andras Iklody" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "andras.iklody@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = " 16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +[[author]] +initials="D." +surname="Servili" +fullname="Deborah Servili" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "deborah.servili@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = " 16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +%%% .# Abstract From c7db81bf6367df95a6172feb2dde17563a40e796 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Tue, 16 Jul 2019 07:27:48 +0200 Subject: [PATCH 14/28] chg: [core] updated to the latest version of mmark format --- misp-core-format/raw.md | 185 ++++++++++++++++++++-------------------- 1 file changed, 94 insertions(+), 91 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index 629dee7..6011021 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -1,40 +1,42 @@ -% Title = "MISP core format" -% abbrev = "MISP core format" -% category = "info" -% docName = "draft-dulaunoy-misp-core-format" -% ipr= "trust200902" -% area = "Security" -% -% date = 2018-08-08T00:00:00Z -% -% [[author]] -% initials="A." -% surname="Dulaunoy" -% fullname="Alexandre Dulaunoy" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "alexandre.dulaunoy@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = "16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1160" -% country = "Luxembourg" -% [[author]] -% initials="A." -% surname="Iklody" -% fullname="Andras Iklody" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "andras.iklody@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = "16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1160" -% country = "Luxembourg" +%%% +Title = "MISP core format" +abbrev = "MISP core format" +category = "info" +docName = "draft-dulaunoy-misp-core-format" +ipr= "trust200902" +area = "Security" + +date = 2018-08-08T00:00:00Z + +[[author]] +initials="A." +surname="Dulaunoy" +fullname="Alexandre Dulaunoy" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "alexandre.dulaunoy@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = "16, bd d'Avranches" + city = "Luxembourg" + code = "L-1160" + country = "Luxembourg" +[[author]] +initials="A." +surname="Iklody" +fullname="Andras Iklody" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "andras.iklody@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = "16, bd d'Avranches" + city = "Luxembourg" + code = "L-1160" + country = "Luxembourg" +%%% .# Abstract @@ -105,7 +107,7 @@ of the event. info **SHOULD** NOT be bigger than 256 characters and **SHOULD** N info is represented as a JSON string. info **MUST** be present. -#### threat_level_id +#### threat\_level\_id threat_level_id represents the threat level. @@ -154,13 +156,13 @@ timestamp represents a reference time when the event, or one of the attributes w timestamp is represented as a JSON string. timestamp **MUST** be present. -#### publish_timestamp +#### publish\_timestamp publish_timestamp represents a reference time when the event was published on the instance. published_timestamp is expressed in seconds (decimal) since 1st of January 1970 (Unix timestamp). At each publication of an event, publish_timestamp **MUST** be updated. The time zone **MUST** be UTC. If the published_timestamp is present and the published flag is set to false, the publish_timestamp represents the previous publication timestamp. If the event was never published, the published_timestamp **MUST** be set to 0. publish_timestamp is represented as a JSON string. publish_timestamp **MUST** be present. -#### org_id +#### org\_id org_id represents a human-readable identifier referencing an Org object of the organisation which generated the event. A human-readable identifier **MUST** be represented as an unsigned integer. @@ -169,7 +171,7 @@ The org_id **MUST** be updated when the event is generated by a new instance. org_id is represented as a JSON string. org_id **MUST** be present. -#### orgc_id +#### orgc\_id orgc_id represents a human-readable identifier referencing an Orgc object of the organisation which created the event. @@ -177,7 +179,7 @@ The orgc_id and Org object **MUST** be preserved for any updates or transfer of orgc_id is represented as a JSON string. orgc_id **MUST** be present. -#### attribute_count +#### attribute\_count attribute_count represents the number of attributes in the event. attribute_count is expressed in decimal. @@ -204,7 +206,7 @@ distribution is represented by a JSON string. distribution **MUST** be present a 4 : Sharing Group -#### sharing_group_id +#### sharing\_group\_id sharing\_group\_id represents a human-readable identifier referencing a Sharing Group object that defines the distribution of the event, if distribution level "4" is set. A human-readable identifier **MUST** be represented as an unsigned integer. @@ -307,52 +309,52 @@ type represents the means through which an attribute tries to describe the inten type is represented as a JSON string. type **MUST** be present and it **MUST** be a valid selection for the chosen category. The list of valid category-type combinations is as follows: -**Antivirus detection** +Antivirus detection : link, comment, text, hex, attachment, other, anonymised -**Artifacts dropped** +Artifacts dropped : md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, regkey, regkey|value, pattern-in-file, pattern-in-memory, pdb, stix2-pattern, yara, sigma, attachment, malware-sample, named pipe, mutex, windows-scheduled-task, windows-service-name, windows-service-displayname, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, other, cookie, gene, mime-type, anonymised -**Attribution** +Attribution : threat-actor, campaign-name, campaign-id, whois-registrant-phone, whois-registrant-email, whois-registrant-name, whois-registrant-org, whois-registrar, whois-creation-date, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, other, dns-soa-email, anonymised -**External analysis** +External analysis : md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised -**Financial fraud** +Financial fraud : btc, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised -**Internal reference** +Internal reference : text, link, comment, other, hex, anonymised -**Network activity** +Network activity : ip-src, ip-dst, ip-dst|port, ip-src|port, port, hostname, domain, domain|ip, mac-address, mac-eui-64, email-dst, url, uri, user-agent, http-method, AS, snort, pattern-in-file, stix2-pattern, pattern-in-traffic, attachment, comment, text, x509-fingerprint-md5, x509-fingerprint-sha1, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hex, cookie, hostname|port, bro, zeek, anonymised -**Other** +Other : comment, text, other, size-in-bytes, counter, datetime, cpe, port, float, hex, phone-number, boolean, anonymised -**Payload delivery** +Payload delivery : md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, mac-address, mac-eui-64, ip-src, ip-dst, ip-dst|port, ip-src|port, hostname, domain, email-src, email-dst, email-subject, email-attachment, email-body, url, user-agent, AS, pattern-in-file, pattern-in-traffic, stix2-pattern, yara, sigma, mime-type, attachment, malware-sample, link, malware-type, comment, text, hex, vulnerability, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hostname|port, email-dst-display-name, email-src-display-name, email-header, email-reply-to, email-x-mailer, email-mime-boundary, email-thread-index, email-message-id, mobile-application-id, whois-registrant-email, anonymised -**Payload installation** +Payload installation : md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, pattern-in-file, pattern-in-traffic, pattern-in-memory, stix2-pattern, yara, sigma, vulnerability, attachment, malware-sample, malware-type, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, mobile-application-id, other, mime-type, anonymised -**Payload type** +Payload type : comment, text, other, anonymised -**Persistence mechanism** +Persistence mechanism : filename, regkey, regkey|value, comment, text, other, hex, anonymised -**Person** +Person : first-name, middle-name, last-name, date-of-birth, place-of-birth, gender, passport-number, passport-country, passport-expiration, redress-number, nationality, visa-number, issue-date-of-the-visa, primary-residence, country-of-residence, special-service-request, frequent-flyer-number, travel-details, payment-details, place-port-of-original-embarkation, place-port-of-clearance, place-port-of-onward-foreign-destination, passenger-name-record-locator-number, comment, text, other, phone-number, identity-card-number, anonymised -**Social network** +Social network : github-username, github-repository, github-organisation, jabber-id, twitter-id, email-src, email-dst, comment, text, other, whois-registrant-email, anonymised -**Support Tool** +Support Tool : link, text, attachment, comment, other, hex, anonymised -**Targeting data** +Targeting data : target-user, target-email, target-machine, target-org, target-location, target-external, comment, anonymised Attributes are based on the usage within their different communities. Attributes can be extended on a regular basis and this reference document is updated accordingly. @@ -414,7 +416,7 @@ comment is a contextual comment field. comment is represented by a JSON string. comment **MAY** be present. -#### sharing_group_id +#### sharing\_group\_id sharing\_group\_id represents a human-readable identifier referencing a Sharing Group object that defines the distribution of the attribute, if distribution level "4" is set. A human-readable identifier **MUST** be represented as an unsigned integer. @@ -517,52 +519,52 @@ type represents the means through which an attribute tries to describe the inten type is represented as a JSON string. type **MUST** be present and it **MUST** be a valid selection for the chosen category. The list of valid category-type combinations is as follows: -**Antivirus detection** +Antivirus detection : link, comment, text, hex, attachment, other, anonymised -**Artifacts dropped** +Artifacts dropped : md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, regkey, regkey|value, pattern-in-file, pattern-in-memory, pdb, stix2-pattern, yara, sigma, attachment, malware-sample, named pipe, mutex, windows-scheduled-task, windows-service-name, windows-service-displayname, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, other, cookie, gene, mime-type, anonymised -**Attribution** +Attribution : threat-actor, campaign-name, campaign-id, whois-registrant-phone, whois-registrant-email, whois-registrant-name, whois-registrant-org, whois-registrar, whois-creation-date, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, other, dns-soa-email, anonymised -**External analysis** +External analysis : md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised -**Financial fraud** +Financial fraud : btc, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised -**Internal reference** +Internal reference : text, link, comment, other, hex, anonymised -**Network activity** +Network activity : ip-src, ip-dst, ip-dst|port, ip-src|port, port, hostname, domain, domain|ip, mac-address, mac-eui-64, email-dst, url, uri, user-agent, http-method, AS, snort, pattern-in-file, stix2-pattern, pattern-in-traffic, attachment, comment, text, x509-fingerprint-md5, x509-fingerprint-sha1, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hex, cookie, hostname|port, bro, zeek, anonymised -**Other** +Other : comment, text, other, size-in-bytes, counter, datetime, cpe, port, float, hex, phone-number, boolean, anonymised -**Payload delivery** +Payload delivery : md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, mac-address, mac-eui-64, ip-src, ip-dst, ip-dst|port, ip-src|port, hostname, domain, email-src, email-dst, email-subject, email-attachment, email-body, url, user-agent, AS, pattern-in-file, pattern-in-traffic, stix2-pattern, yara, sigma, mime-type, attachment, malware-sample, link, malware-type, comment, text, hex, vulnerability, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hostname|port, email-dst-display-name, email-src-display-name, email-header, email-reply-to, email-x-mailer, email-mime-boundary, email-thread-index, email-message-id, mobile-application-id, whois-registrant-email, anonymised -**Payload installation** +Payload installation : md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, pattern-in-file, pattern-in-traffic, pattern-in-memory, stix2-pattern, yara, sigma, vulnerability, attachment, malware-sample, malware-type, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, mobile-application-id, other, mime-type, anonymised -**Payload type** +Payload type : comment, text, other, anonymised -**Persistence mechanism** +Persistence mechanism : filename, regkey, regkey|value, comment, text, other, hex, anonymised -**Person** +Person : first-name, middle-name, last-name, date-of-birth, place-of-birth, gender, passport-number, passport-country, passport-expiration, redress-number, nationality, visa-number, issue-date-of-the-visa, primary-residence, country-of-residence, special-service-request, frequent-flyer-number, travel-details, payment-details, place-port-of-original-embarkation, place-port-of-clearance, place-port-of-onward-foreign-destination, passenger-name-record-locator-number, comment, text, other, phone-number, identity-card-number, anonymised -**Social network** +Social network : github-username, github-repository, github-organisation, jabber-id, twitter-id, email-src, email-dst, comment, text, other, whois-registrant-email, anonymised -**Support Tool** +Support Tool : link, text, attachment, comment, other, hex, anonymised -**Targeting data** +Targeting data : target-user, target-email, target-machine, target-org, target-location, target-external, comment, anonymised Attributes are based on the usage within their different communities. Attributes can be extended on a regular basis and this reference document is updated accordingly. @@ -686,9 +688,10 @@ The schema used is described by the template_uuid and template_version fields. A MISP document containing an Object **MUST** contain a name, a meta-category, a description, a template_uuid and a template_version as described in the "Object Attributes" section. -### Sample Object object +### Sample Object -~~~~~ +{#fig-sample-object} +~~~ "Object": { "id": "588", "name": "file", @@ -729,7 +732,7 @@ A MISP document containing an Object **MUST** contain a name, a meta-category, a "last_seen": null ] } -~~~~~ +~~~ ### Object Attributes @@ -764,19 +767,19 @@ description is a human-readable description of the given object type, as derived description is represented as a JSON string. id **SHALL** be present. -#### template_uuid +#### template\_uuid uuid represents the Universally Unique IDentifier (UUID) [@!RFC4122] of the template used to create the object. The uuid **MUST** be preserved to preserve the object's association with the correct template used for creation. UUID version 4 is **RECOMMENDED** when assigning it to a new object. -#### template_version +#### template\_version template_version represents a numeric incrementing version of the template used to create the object. It is used to associate the object to the correct version of the template and together with the template_uuid forms an association to the correct template type and version. version is represented as a JSON string. version **MUST** be present. -#### event_id +#### event\_id event_id represents the human-readable identifier of the event that the object belongs to on a specific MISP instance. A human-readable identifier **MUST** be represented as an unsigned integer. @@ -810,7 +813,7 @@ distribution is represented by a JSON string. distribution **MUST** be present a 4 : Sharing Group -#### sharing_group_id +#### sharing\_group\_id sharing\_group\_id represents a human-readable identifier referencing a Sharing Group object that defines the distribution of the object, if distribution level "4" is set. A human-readable identifier **MUST** be represented as an unsigned integer. @@ -834,13 +837,13 @@ Attribute is an array of attributes that describe the object with data. Each attribute in an object **MUST** contain the parent event's ID in the event_id field and the parent object's ID in the object_id field. -#### first_seen +#### first\_seen first_seen represents a reference time when the object was first seen. first_seen as an ISO 8601 datetime up to the micro-second with time zone support. first_seen is represented as a JSON string. first_seen **MAY** be present. -#### last_seen +#### last\_seen last_seen represents a reference time when the object was last seen. last_seen as an ISO 8601 datetime up to the micro-second with time zone support. @@ -850,9 +853,9 @@ last_seen is represented as a JSON string. last_seen **MAY** be present. Object References serve as a logical link between an Object and another referenced Object or Attribute. The relationship is categorised by an enumerated value from a fixed vocabulary. -The relationship_type is recommended to be taken from the MISP object relationship list [[@?MISP-R]] is **RECOMMENDED** to ensure a coherent naming of the tags +The relationship\_type is recommended to be taken from the MISP object relationship list [[@?MISP-R]] is **RECOMMENDED** to ensure a coherent naming of the tags -All Object References **MUST** contain an object_uuid, a referenced_uuid and a relationship type. +All Object References **MUST** contain an object\_uuid, a referenced\_uuid and a relationship type. ### Sample ObjectReference object @@ -936,14 +939,14 @@ deleted represents a setting that allows object references to be revoked. Revoke deleted is represented by a JSON boolean. deleted **MUST** be present. -#### object_uuid +#### object\_uuid -object_uuid represents the Universally Unique IDentifier (UUID) [@!RFC4122] of the object that the given object reference belongs to. The object_uuid **MUST** be preserved +object\_uuid represents the Universally Unique IDentifier (UUID) [@!RFC4122] of the object that the given object reference belongs to. The object\_uuid **MUST** be preserved to preserve the object reference's association with the object. -#### referenced_uuid +#### referenced\_uuid -referenced_uuid represents the Universally Unique IDentifier (UUID) [@!RFC4122] of the object or attribute that is being referenced by the object reference. The referenced_uuid **MUST** be preserved +referenced\_uuid represents the Universally Unique IDentifier (UUID) [@!RFC4122] of the object or attribute that is being referenced by the object reference. The referenced\_uuid **MUST** be preserved to preserve the object reference's association with the object or attribute. ## Tag From b40dd4b7ffcebd36c9532d9f1b7bc0c7c0b8b1ad Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Tue, 16 Jul 2019 07:33:06 +0200 Subject: [PATCH 15/28] chg: [misp-object-template-format] updated to the latest version of mmark2 --- misp-object-template-format/raw.md | 76 +++++++++++++++--------------- 1 file changed, 39 insertions(+), 37 deletions(-) diff --git a/misp-object-template-format/raw.md b/misp-object-template-format/raw.md index cccab4c..925948f 100755 --- a/misp-object-template-format/raw.md +++ b/misp-object-template-format/raw.md @@ -1,40 +1,42 @@ -% Title = "MISP object template format" -% abbrev = "MISP object template format" -% category = "info" -% docName = "draft-dulaunoy-misp-object-template-format" -% ipr= "trust200902" -% area = "Security" -% -% date = 2018-04-10T00:00:00Z -% -% [[author]] -% initials="A." -% surname="Dulaunoy" -% fullname="Alexandre Dulaunoy" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "alexandre.dulaunoy@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = "16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" -% [[author]] -% initials="A." -% surname="Iklody" -% fullname="Andras Iklody" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "andras.iklody@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = " 16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" +%%% +Title = "MISP object template format" +abbrev = "MISP object template format" +category = "info" +docName = "draft-dulaunoy-misp-object-template-format" +ipr= "trust200902" +area = "Security" + +date = 2018-04-10T00:00:00Z + +[[author]] +initials="A." +surname="Dulaunoy" +fullname="Alexandre Dulaunoy" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "alexandre.dulaunoy@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = "16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +[[author]] +initials="A." +surname="Iklody" +fullname="Andras Iklody" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "andras.iklody@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = " 16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +%%% .# Abstract From bd1bda98a6b21385750a3720ff0181b21ae26262 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Tue, 16 Jul 2019 07:35:30 +0200 Subject: [PATCH 16/28] chg: [taxonomy-format] updated to the latest version of mmark2 --- misp-taxonomy-format/raw.md | 76 +++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 37 deletions(-) diff --git a/misp-taxonomy-format/raw.md b/misp-taxonomy-format/raw.md index d71a069..d5d9c7c 100755 --- a/misp-taxonomy-format/raw.md +++ b/misp-taxonomy-format/raw.md @@ -1,40 +1,42 @@ -% Title = "MISP taxonomy format" -% abbrev = "MISP taxonomy format" -% category = "info" -% docName = "draft-dulaunoy-misp-taxonomy-format" -% ipr= "trust200902" -% area = "Security" -% -% date = 2017-11-29T00:00:00Z -% -% [[author]] -% initials="A." -% surname="Dulaunoy" -% fullname="Alexandre Dulaunoy" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "alexandre.dulaunoy@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = "16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" -% [[author]] -% initials="A." -% surname="Iklody" -% fullname="Andras Iklody" -% abbrev="CIRCL" -% organization = "Computer Incident Response Center Luxembourg" -% [author.address] -% email = "andras.iklody@circl.lu" -% phone = "+352 247 88444" -% [author.address.postal] -% street = " 16, bd d'Avranches" -% city = "Luxembourg" -% code = "L-1611" -% country = "Luxembourg" +%%% +Title = "MISP taxonomy format" +abbrev = "MISP taxonomy format" +category = "info" +docName = "draft-dulaunoy-misp-taxonomy-format" +ipr= "trust200902" +area = "Security" + +date = 2017-11-29T00:00:00Z + +[[author]] +initials="A." +surname="Dulaunoy" +fullname="Alexandre Dulaunoy" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "alexandre.dulaunoy@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = "16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +[[author]] +initials="A." +surname="Iklody" +fullname="Andras Iklody" +abbrev="CIRCL" +organization = "Computer Incident Response Center Luxembourg" + [author.address] + email = "andras.iklody@circl.lu" + phone = "+352 247 88444" + [author.address.postal] + street = " 16, bd d'Avranches" + city = "Luxembourg" + code = "L-1611" + country = "Luxembourg" +%%% .# Abstract From 77c44154b8426b32b71d036424ad1aa0e36ed584 Mon Sep 17 00:00:00 2001 From: Christophe Vandeplas Date: Thu, 8 Aug 2019 12:14:43 +0200 Subject: [PATCH 17/28] chg: [misp-core-format] updated to the latest version of type/categories --- misp-core-format/raw.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index 6011021..b8a60be 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -319,7 +319,7 @@ Attribution : threat-actor, campaign-name, campaign-id, whois-registrant-phone, whois-registrant-email, whois-registrant-name, whois-registrant-org, whois-registrar, whois-creation-date, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, other, dns-soa-email, anonymised External analysis -: md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised +: md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, weakness, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised, community-id Financial fraud : btc, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised @@ -328,16 +328,16 @@ Internal reference : text, link, comment, other, hex, anonymised Network activity -: ip-src, ip-dst, ip-dst|port, ip-src|port, port, hostname, domain, domain|ip, mac-address, mac-eui-64, email-dst, url, uri, user-agent, http-method, AS, snort, pattern-in-file, stix2-pattern, pattern-in-traffic, attachment, comment, text, x509-fingerprint-md5, x509-fingerprint-sha1, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hex, cookie, hostname|port, bro, zeek, anonymised +: ip-src, ip-dst, ip-dst|port, ip-src|port, port, hostname, domain, domain|ip, mac-address, mac-eui-64, email-dst, url, uri, user-agent, http-method, AS, snort, pattern-in-file, stix2-pattern, pattern-in-traffic, attachment, comment, text, x509-fingerprint-md5, x509-fingerprint-sha1, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hex, cookie, hostname|port, bro, zeek, anonymised, community-id, email-subject Other : comment, text, other, size-in-bytes, counter, datetime, cpe, port, float, hex, phone-number, boolean, anonymised Payload delivery -: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, mac-address, mac-eui-64, ip-src, ip-dst, ip-dst|port, ip-src|port, hostname, domain, email-src, email-dst, email-subject, email-attachment, email-body, url, user-agent, AS, pattern-in-file, pattern-in-traffic, stix2-pattern, yara, sigma, mime-type, attachment, malware-sample, link, malware-type, comment, text, hex, vulnerability, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hostname|port, email-dst-display-name, email-src-display-name, email-header, email-reply-to, email-x-mailer, email-mime-boundary, email-thread-index, email-message-id, mobile-application-id, whois-registrant-email, anonymised +: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, mac-address, mac-eui-64, ip-src, ip-dst, ip-dst|port, ip-src|port, hostname, domain, email-src, email-dst, email-subject, email-attachment, email-body, url, user-agent, AS, pattern-in-file, pattern-in-traffic, stix2-pattern, yara, sigma, mime-type, attachment, malware-sample, link, malware-type, comment, text, hex, vulnerability, weakness, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hostname|port, email-dst-display-name, email-src-display-name, email-header, email-reply-to, email-x-mailer, email-mime-boundary, email-thread-index, email-message-id, mobile-application-id, whois-registrant-email, anonymised Payload installation -: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, pattern-in-file, pattern-in-traffic, pattern-in-memory, stix2-pattern, yara, sigma, vulnerability, attachment, malware-sample, malware-type, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, mobile-application-id, other, mime-type, anonymised +: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, pattern-in-file, pattern-in-traffic, pattern-in-memory, stix2-pattern, yara, sigma, vulnerability, weakness, attachment, malware-sample, malware-type, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, mobile-application-id, other, mime-type, anonymised Payload type : comment, text, other, anonymised @@ -529,7 +529,7 @@ Attribution : threat-actor, campaign-name, campaign-id, whois-registrant-phone, whois-registrant-email, whois-registrant-name, whois-registrant-org, whois-registrar, whois-creation-date, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, other, dns-soa-email, anonymised External analysis -: md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised +: md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, weakness, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised, community-id Financial fraud : btc, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised @@ -538,16 +538,16 @@ Internal reference : text, link, comment, other, hex, anonymised Network activity -: ip-src, ip-dst, ip-dst|port, ip-src|port, port, hostname, domain, domain|ip, mac-address, mac-eui-64, email-dst, url, uri, user-agent, http-method, AS, snort, pattern-in-file, stix2-pattern, pattern-in-traffic, attachment, comment, text, x509-fingerprint-md5, x509-fingerprint-sha1, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hex, cookie, hostname|port, bro, zeek, anonymised +: ip-src, ip-dst, ip-dst|port, ip-src|port, port, hostname, domain, domain|ip, mac-address, mac-eui-64, email-dst, url, uri, user-agent, http-method, AS, snort, pattern-in-file, stix2-pattern, pattern-in-traffic, attachment, comment, text, x509-fingerprint-md5, x509-fingerprint-sha1, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hex, cookie, hostname|port, bro, zeek, anonymised, community-id, email-subject Other : comment, text, other, size-in-bytes, counter, datetime, cpe, port, float, hex, phone-number, boolean, anonymised Payload delivery -: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, mac-address, mac-eui-64, ip-src, ip-dst, ip-dst|port, ip-src|port, hostname, domain, email-src, email-dst, email-subject, email-attachment, email-body, url, user-agent, AS, pattern-in-file, pattern-in-traffic, stix2-pattern, yara, sigma, mime-type, attachment, malware-sample, link, malware-type, comment, text, hex, vulnerability, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hostname|port, email-dst-display-name, email-src-display-name, email-header, email-reply-to, email-x-mailer, email-mime-boundary, email-thread-index, email-message-id, mobile-application-id, whois-registrant-email, anonymised +: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, mac-address, mac-eui-64, ip-src, ip-dst, ip-dst|port, ip-src|port, hostname, domain, email-src, email-dst, email-subject, email-attachment, email-body, url, user-agent, AS, pattern-in-file, pattern-in-traffic, stix2-pattern, yara, sigma, mime-type, attachment, malware-sample, link, malware-type, comment, text, hex, vulnerability, weakness, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, other, hostname|port, email-dst-display-name, email-src-display-name, email-header, email-reply-to, email-x-mailer, email-mime-boundary, email-thread-index, email-message-id, mobile-application-id, whois-registrant-email, anonymised Payload installation -: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, pattern-in-file, pattern-in-traffic, pattern-in-memory, stix2-pattern, yara, sigma, vulnerability, attachment, malware-sample, malware-type, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, mobile-application-id, other, mime-type, anonymised +: md5, sha1, sha224, sha256, sha384, sha512, sha512/224, sha512/256, ssdeep, imphash, impfuzzy, authentihash, pehash, tlsh, cdhash, filename, filename|md5, filename|sha1, filename|sha224, filename|sha256, filename|sha384, filename|sha512, filename|sha512/224, filename|sha512/256, filename|authentihash, filename|ssdeep, filename|tlsh, filename|imphash, filename|impfuzzy, filename|pehash, pattern-in-file, pattern-in-traffic, pattern-in-memory, stix2-pattern, yara, sigma, vulnerability, weakness, attachment, malware-sample, malware-type, comment, text, hex, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, mobile-application-id, other, mime-type, anonymised Payload type : comment, text, other, anonymised From 5e9e9dc9703501b59c33876224e9aa646d235d3d Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Tue, 1 Oct 2019 20:10:36 +0200 Subject: [PATCH 18/28] chg: [types] updated --- misp-core-format/raw.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/misp-core-format/raw.md b/misp-core-format/raw.md index b8a60be..9a749f0 100755 --- a/misp-core-format/raw.md +++ b/misp-core-format/raw.md @@ -322,7 +322,7 @@ External analysis : md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, weakness, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised, community-id Financial fraud -: btc, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised +: btc, dash, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised Internal reference : text, link, comment, other, hex, anonymised @@ -532,7 +532,7 @@ External analysis : md5, sha1, sha256, filename, filename|md5, filename|sha1, filename|sha256, ip-src, ip-dst, ip-dst|port, ip-src|port, mac-address, mac-eui-64, hostname, domain, domain|ip, url, user-agent, regkey, regkey|value, AS, snort, bro, zeek, pattern-in-file, pattern-in-traffic, pattern-in-memory, vulnerability, weakness, attachment, malware-sample, link, comment, text, x509-fingerprint-sha1, x509-fingerprint-md5, x509-fingerprint-sha256, ja3-fingerprint-md5, hassh-md5, hasshserver-md5, github-repository, other, cortex, anonymised, community-id Financial fraud -: btc, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised +: btc, dash, xmr, iban, bic, bank-account-nr, aba-rtn, bin, cc-number, prtn, phone-number, comment, text, other, hex, anonymised Internal reference : text, link, comment, other, hex, anonymised From b8a6eed77b251b2731db038ee88cd4c529769e7c Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Fri, 4 Oct 2019 13:57:53 +0200 Subject: [PATCH 19/28] chg: [misp-galaxy] clarification regarding the meta field usage --- misp-galaxy-format/raw.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/misp-galaxy-format/raw.md b/misp-galaxy-format/raw.md index c2a09f6..816d94e 100644 --- a/misp-galaxy-format/raw.md +++ b/misp-galaxy-format/raw.md @@ -6,7 +6,7 @@ docName = "draft-dulaunoy-misp-galaxy-format" ipr= "trust200902" area = "Security" -date = 2018-09-20T00:00:00Z +date = 2019-10-04T00:00:00Z [[author]] initials="A." @@ -105,7 +105,7 @@ Related contains a list of JSON key value pairs which describe the related value ## meta -Meta contains a list of custom defined JSON key value pairs. Users **SHOULD** reuse commonly used keys such as complexity, effectiveness, country, possible_issues, colour, motive, impact, refs, synonyms, status, date, encryption, extensions, ransomnotes, ransomnotes-filenames, ransomnotes-refs, suspected-victims, suspected-state-sponsor, type-of-incident, target-category, cfr-suspected-victims, cfr-suspected-state-sponsor, cfr-type-of-incident, cfr-target-category, attribution-confidence, payment-method, price wherever applicable. +Meta contains a list of custom defined JSON key value pairs. Users **SHOULD** reuse commonly used keys such as complexity, effectiveness, country, possible_issues, colour, motive, impact, refs, synonyms, status, date, encryption, extensions, ransomnotes, ransomnotes-filenames, ransomnotes-refs, suspected-victims, suspected-state-sponsor, type-of-incident, target-category, cfr-suspected-victims, cfr-suspected-state-sponsor, cfr-type-of-incident, cfr-target-category, attribution-confidence, payment-method, price wherever applicable. Additional meta field **MAY** be added without the need to be referenced or registered in advance. refs, synonyms **SHALL** be used to give further informations. refs is represented as an array containing one or more strings and **SHALL** be present. synonyms is represented as an array containing one or more strings and **SHALL** be present. From fea0f543e3e0f0bdd578908719c94c46d5192d04 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Fri, 4 Oct 2019 13:58:40 +0200 Subject: [PATCH 20/28] chg: [galaxy] txt output updated --- misp-galaxy-format/raw.md.txt | 76 +++++++++++++++++------------------ 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/misp-galaxy-format/raw.md.txt b/misp-galaxy-format/raw.md.txt index c4e4d31..41e2858 100755 --- a/misp-galaxy-format/raw.md.txt +++ b/misp-galaxy-format/raw.md.txt @@ -5,8 +5,8 @@ Network Working Group A. Dulaunoy Internet-Draft A. Iklody Intended status: Informational D. Servili -Expires: March 24, 2019 CIRCL - September 20, 2018 +Expires: April 6, 2020 CIRCL + October 4, 2019 MISP galaxy format @@ -38,11 +38,11 @@ Status of This Memo time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on March 24, 2019. + This Internet-Draft will expire on April 6, 2020. Copyright Notice - Copyright (c) 2018 IETF Trust and the persons identified as the + Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal @@ -53,9 +53,9 @@ Copyright Notice -Dulaunoy, et al. Expires March 24, 2019 [Page 1] +Dulaunoy, et al. Expires April 6, 2020 [Page 1] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 to this document. Code Components extracted from this document must @@ -109,9 +109,9 @@ Table of Contents -Dulaunoy, et al. Expires March 24, 2019 [Page 2] +Dulaunoy, et al. Expires April 6, 2020 [Page 2] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 2. Format @@ -165,9 +165,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 3] +Dulaunoy, et al. Expires April 6, 2020 [Page 3] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 dest-uuid represents the target UUID which encompasses a relation of @@ -196,7 +196,8 @@ Internet-Draft MISP galaxy format September 2018 sponsor, type-of-incident, target-category, cfr-suspected-victims, cfr-suspected-state-sponsor, cfr-type-of-incident, cfr-target- category, attribution-confidence, payment-method, price wherever - applicable. + applicable. Additional meta field MAY be added without the need to + be referenced or registered in advance. refs, synonyms SHALL be used to give further informations. refs is represented as an array containing one or more strings and SHALL be @@ -217,15 +218,15 @@ Internet-Draft MISP galaxy format September 2018 complexity, effectiveness, impact, possible_issues MAY be used to give further information in preventive-measure galaxy. complexity is represented by an enumerated value from a fixed vocabulary and SHALL - be present. effectiveness is represented by an enumerated value from -Dulaunoy, et al. Expires March 24, 2019 [Page 4] +Dulaunoy, et al. Expires April 6, 2020 [Page 4] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 + be present. effectiveness is represented by an enumerated value from a fixed vocabulary and SHALL be present. impact is represented by an enumerated value from a fixed vocabulary and SHALL be present. possible_issues is represented as a string and SHOULD be present. @@ -276,10 +277,9 @@ Internet-Draft MISP galaxy format September 2018 - -Dulaunoy, et al. Expires March 24, 2019 [Page 5] +Dulaunoy, et al. Expires April 6, 2020 [Page 5] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 { @@ -333,9 +333,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 6] +Dulaunoy, et al. Expires April 6, 2020 [Page 6] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 { @@ -389,9 +389,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 7] +Dulaunoy, et al. Expires April 6, 2020 [Page 7] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 Example use of the source-uuid, target-uuid fields in the mitre- @@ -445,9 +445,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 8] +Dulaunoy, et al. Expires April 6, 2020 [Page 8] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 { @@ -501,9 +501,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 9] +Dulaunoy, et al. Expires April 6, 2020 [Page 9] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 { @@ -557,9 +557,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 10] +Dulaunoy, et al. Expires April 6, 2020 [Page 10] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 "additionalProperties": false, @@ -613,9 +613,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 11] +Dulaunoy, et al. Expires April 6, 2020 [Page 11] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 }, @@ -669,9 +669,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 12] +Dulaunoy, et al. Expires April 6, 2020 [Page 12] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 "items": { @@ -725,9 +725,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 13] +Dulaunoy, et al. Expires April 6, 2020 [Page 13] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 } @@ -781,9 +781,9 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 14] +Dulaunoy, et al. Expires April 6, 2020 [Page 14] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 [JSON-SCHEMA] @@ -837,9 +837,9 @@ Authors' Addresses -Dulaunoy, et al. Expires March 24, 2019 [Page 15] +Dulaunoy, et al. Expires April 6, 2020 [Page 15] -Internet-Draft MISP galaxy format September 2018 +Internet-Draft MISP galaxy format October 2019 Deborah Servili @@ -893,4 +893,4 @@ Internet-Draft MISP galaxy format September 2018 -Dulaunoy, et al. Expires March 24, 2019 [Page 16] +Dulaunoy, et al. Expires April 6, 2020 [Page 16] From 44b150ff3edb1d381901acb9f0c089194363fc2a Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Sun, 3 Nov 2019 22:11:56 -0800 Subject: [PATCH 21/28] Adding first skeleton for the SightingDB format --- sightingdb-format/Makefile | 8 +++++ sightingdb-format/raw.md | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100644 sightingdb-format/Makefile create mode 100755 sightingdb-format/raw.md diff --git a/sightingdb-format/Makefile b/sightingdb-format/Makefile new file mode 100644 index 0000000..73b05d7 --- /dev/null +++ b/sightingdb-format/Makefile @@ -0,0 +1,8 @@ +MMARK:=mmark -xml2 -page + +docs = $(wildcard *.md) + +all: $(docs) + $(MMARK) $< > $<.xml + xml2rfc --text $<.xml + xml2rfc --html $<.xml diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md new file mode 100755 index 0000000..06823e1 --- /dev/null +++ b/sightingdb-format/raw.md @@ -0,0 +1,61 @@ +%%% +Title = "SightingDB format" +abbrev = "SightingDB format" +category = "info" +docName = "draft-tricaud-sightingdb-format" +ipr= "trust200902" +area = "Security" + +date = 2019-03-03T00:00:00Z + +[[author]] +initials="S." +surname="Tricaud" +fullname="Sebastien Tricaud" +abbrev="Devo Inc." +organization = "Devo Inc." + [author.address] + email = "sebastien.tricaud@devo.com" + phone = "+1 866-221-2254" + [author.address.postal] + street = "150 Cambridgepark Drive" + city = "Cambridge, MA" + code = "02140" + country = "USA" +%%% + +.# Abstract + +This document describes the format used by SightingDB to give automated context to a given Attribute +by counting occurences and tracking times of observability. +SightingDB was designed to provide to MISP a Scalable and Fast way to store and retrive Attributes. + +{mainmatter} + +# Introduction + +Adding context to any Attribute is the key that makes it useful. While there exist numerous ways of doing it, +SightingDB does it by just counting. +Whenever somebody retrieves an Attribute, this counting is provided, allowing anyone to understand wether something +was observed few or many times. + +## Conventions and Terminology + +The key words "**MUST**", "**MUST NOT**", "**REQUIRED**", "**SHALL**", "**SHALL NOT**", +"**SHOULD**", "**SHOULD NOT**", "**RECOMMENDED**", "**MAY**", and "**OPTIONAL**" in this +document are to be interpreted as described in RFC 2119 [@!RFC2119]. + +# Format + +## Overview + +The SightingDB format is in the JSON [@!RFC8259] format. In SightingDB, a Sighting Object is composed of a single JSON object. + +# Acknowledgements + +The author wish to thank all the MISP community who are supporting the creation +of open standards in threat intelligence sharing. As well as amazing feedback gathered +during the MISP Summit 2019 in Luxembourg, in particular with Alexandre Dulaunoy and +Andras Iklody. + +{backmatter} From ac9006e9ef31771dfda798dabf956fc292512f87 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Mon, 4 Nov 2019 22:26:23 -0800 Subject: [PATCH 22/28] Added description for each field used by SightingDB --- sightingdb-format/raw.md | 81 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 79 insertions(+), 2 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index 06823e1..a4ec0d2 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -6,7 +6,7 @@ docName = "draft-tricaud-sightingdb-format" ipr= "trust200902" area = "Security" -date = 2019-03-03T00:00:00Z +date = 2019-11-03T00:00:00Z [[author]] initials="S." @@ -49,7 +49,84 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. ## Overview -The SightingDB format is in the JSON [@!RFC8259] format. In SightingDB, a Sighting Object is composed of a single JSON object. +The SightingDB format is in the JSON [@!RFC8259] format. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl, frequency and manifold. + +### Attribute Storage + +The fields described previously describe an Attribute and all the required characteristics. However they are stored in a Namespace. A Namespace is similar to a path in a filesystem where the same file can be stored in multiple places. + +### Namespace + +A Namespace with multiple levels MUST be separated with the slash '/' character. There is no specification on how they are structured, since it depends on the use cases. + +A Namespace starting with the underscore '_' character means it is private and internal to SightingDB. There are all reserved for the engine and MUST NOT be used. + +Reserved namespaces are: +_expired/: Which contains all the attributes that expired, preserving the origin namespace +_shadow/: When a value is searched and does not exists, it is stored there +_stats: Statistics +_config: Configuration +_all: All the Attributes in one place, used to retrieve the 'manifold' property. + +The Attribute Key MUST always be the last part of the Namespace. + +#### Sample Namespaces + +/Organization1/service/ipv4: Store values for ipv4 keys in /Organization1/service +/everything/domain: Store domains in /everything + +### Attribute fields + +#### value + +The attribute value, used to store and retrieve information about an attribute. Note that value is not returned back in the JSON object, since it is queried, it is known. + +#### first_seen + +Time in UTC of the first time this value was captured + +#### last_seen + +Time in UTC of the last time this value was captured + +#### count + +How many time this value was written + +#### tags + +Tags follow how they are defined in MISP using the MISP Taxonomy. Each Tag is separated with the ';' character. + +#### ttl + +Time To Live, represents the expiration in seconds since the time the Attribute was created. Once it has expired, it moves in the private Namespace _expired. + +When an Attribute has this field set to 0, it means it is not set to expired. This is the default behavior. + +When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time. + +#### frequency + +Frequency is the number of time an Attribute is seen in average per day. As this field can introduced latence, its implementation is OPTIONAL. + +#### manifold + +When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter. + +## SightingDB Format - One Attribute + +~~~~ +{ + "value":"127.0.0.1", + "first_seen":1530394819, + "last_seen":1572933618, + "count":578391, + "tags":"", + "ttl":0, + "frequency":1185, + "manifold": 17 +} +~~~~ # Acknowledgements From cf03cb308d906ed76636105a8d8c614b7d50036b Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Tue, 5 Nov 2019 17:29:47 -0800 Subject: [PATCH 23/28] * More details on the value format * Adding the security considerations --- sightingdb-format/raw.md | 41 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index a4ec0d2..f1c5c29 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -57,9 +57,9 @@ The fields described previously describe an Attribute and all the required chara ### Namespace -A Namespace with multiple levels MUST be separated with the slash '/' character. There is no specification on how they are structured, since it depends on the use cases. +A Namespace with multiple levels **MUST** be separated with the slash '/' character. There is no specification on how they are structured, since it depends on the use cases. -A Namespace starting with the underscore '_' character means it is private and internal to SightingDB. There are all reserved for the engine and MUST NOT be used. +A Namespace starting with the underscore '_' character means it is private and internal to SightingDB. There are all reserved for the engine and **MUST** NOT be used. Reserved namespaces are: _expired/: Which contains all the attributes that expired, preserving the origin namespace @@ -79,7 +79,9 @@ The Attribute Key MUST always be the last part of the Namespace. #### value -The attribute value, used to store and retrieve information about an attribute. Note that value is not returned back in the JSON object, since it is queried, it is known. +The attribute value, used to store and retrieve information about an attribute. Note that value is not returned back in the JSON object, since it is queried, it is known. The Value is described in a section below, as it is very specific and can be either "as is", a hash, encoded in base64 or any other convenient mechanism. + +The value implementation **MUST** offer at least: 1) Raw value 2) Base64 URL Encoded 3) SHA256 Hash #### first_seen @@ -107,7 +109,7 @@ When an Attribute has this field set to a number greater than 0, the expiration #### frequency -Frequency is the number of time an Attribute is seen in average per day. As this field can introduced latence, its implementation is OPTIONAL. +Frequency is the number of time an Attribute is seen in average per day. As this field can introduced latence, its implementation is **OPTIONAL**. #### manifold @@ -128,6 +130,37 @@ When a given Attribute Value is stored in different namespaces, the manifold fie } ~~~~ +# Value + +The value submitted can be in multiple format according to the use-case. Any implementation **MUST** offer three alternatives: + +1) Raw value: where nothing is encoded and the value is stored AS IS, such as show in the example above with the One Attribute in JSON. +2) SHA256: which prevents from seeing content (see Security Considerations), has a fixed size and is convenient for most requirements +3) Base64 URL: Where the specification of Base64 is followed, except the characters conflicting with an URL argument are replaced + +The value is configured as part of the Namespace. The private "_config" Namespace prefix stores this value storage mechanism. + +## Configuring the value format for a Namespace + +If one has the Namespace "/Organization1/BU1/ip" and want to store those IP addresses in SHA256, it will be configured like this: +The Namespace is kept but prefixed by "_config" and has a json object about value format set. +"/_config/Organization1/BU1/ip" + +~~~~ +{ + "value_format":"SHA256" +} +~~~~ + +Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". + +# Security Considerations + +While this document solely focuses on the format, the reference implementation is SightingDB. The authentication, the data access is not handled by SightingDB. +It is possible a value can leak if the access is too permissive. + +Even a Hashed value can be discovered, as re-hashing known values would match. + # Acknowledgements The author wish to thank all the MISP community who are supporting the creation From 0e2a6130ba26d001a404a787b13ec364e31f8730 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Wed, 6 Nov 2019 11:29:48 +0100 Subject: [PATCH 24/28] chg: [sightingDB] query clarification --- sightingdb-format/raw.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index f1c5c29..b8814b4 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -1,6 +1,6 @@ %%% -Title = "SightingDB format" -abbrev = "SightingDB format" +Title = "SightingDB query format" +abbrev = "SightingDB query format" category = "info" docName = "draft-tricaud-sightingdb-format" ipr= "trust200902" @@ -27,8 +27,8 @@ organization = "Devo Inc." .# Abstract This document describes the format used by SightingDB to give automated context to a given Attribute -by counting occurences and tracking times of observability. -SightingDB was designed to provide to MISP a Scalable and Fast way to store and retrive Attributes. +by counting occurrences and tracking times of observability. +SightingDB was designed to provide to MISP a Scalable and Fast way to store and retrieve Attributes. {mainmatter} @@ -36,7 +36,7 @@ SightingDB was designed to provide to MISP a Scalable and Fast way to store and Adding context to any Attribute is the key that makes it useful. While there exist numerous ways of doing it, SightingDB does it by just counting. -Whenever somebody retrieves an Attribute, this counting is provided, allowing anyone to understand wether something +Whenever somebody retrieves an Attribute, this counting is provided, allowing anyone to understand whenever something was observed few or many times. ## Conventions and Terminology @@ -49,11 +49,11 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. ## Overview -The SightingDB format is in the JSON [@!RFC8259] format. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl, frequency and manifold. +The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl, frequency and manifold. ### Attribute Storage -The fields described previously describe an Attribute and all the required characteristics. However they are stored in a Namespace. A Namespace is similar to a path in a filesystem where the same file can be stored in multiple places. +The fields described previously describe an Attribute and all the required characteristics. However they are stored in a Namespace. A Namespace is similar to a path in a file-system where the same file can be stored in multiple places. ### Namespace From c49f77fed994b0b01811bdbed82e4ba1a5d45155 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Wed, 6 Nov 2019 11:31:34 +0100 Subject: [PATCH 25/28] chg: [sightingDB] ascii output added --- sightingdb-format/raw.md.txt | 336 +++++++++++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 sightingdb-format/raw.md.txt diff --git a/sightingdb-format/raw.md.txt b/sightingdb-format/raw.md.txt new file mode 100644 index 0000000..4f90776 --- /dev/null +++ b/sightingdb-format/raw.md.txt @@ -0,0 +1,336 @@ + + + + +Network Working Group S. Tricaud +Internet-Draft Devo Inc. +Intended status: Informational November 3, 2019 +Expires: May 6, 2020 + + + SightingDB query format + draft-tricaud-sightingdb-format + +Abstract + + This document describes the format used by SightingDB to give + automated context to a given Attribute by counting occurrences and + tracking times of observability. SightingDB was designed to provide + to MISP a Scalable and Fast way to store and retrieve Attributes. + +Status of This Memo + + This Internet-Draft is submitted in full conformance with the + provisions of BCP 78 and BCP 79. + + Internet-Drafts are working documents of the Internet Engineering + Task Force (IETF). Note that other groups may also distribute + working documents as Internet-Drafts. The list of current Internet- + Drafts is at https://datatracker.ietf.org/drafts/current/. + + Internet-Drafts are draft documents valid for a maximum of six months + and may be updated, replaced, or obsoleted by other documents at any + time. It is inappropriate to use Internet-Drafts as reference + material or to cite them other than as "work in progress." + + This Internet-Draft will expire on May 6, 2020. + +Copyright Notice + + Copyright (c) 2019 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (https://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + + + + +Tricaud Expires May 6, 2020 [Page 1] + +Internet-Draft SightingDB query format November 2019 + + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 + 1.1. Conventions and Terminology . . . . . . . . . . . . . . . 2 + 2. Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 + 2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 2 + 2.1.1. Attribute Storage . . . . . . . . . . . . . . . . . . 2 + 2.1.2. Namespace . . . . . . . . . . . . . . . . . . . . . . 3 + 2.1.3. Attribute fields . . . . . . . . . . . . . . . . . . 3 + 2.2. SightingDB Format - One Attribute . . . . . . . . . . . . 4 + 3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 3.1. Configuring the value format for a Namespace . . . . . . 5 + 4. Security Considerations . . . . . . . . . . . . . . . . . . . 5 + 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 + 6. Normative References . . . . . . . . . . . . . . . . . . . . 6 + Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6 + +1. Introduction + + Adding context to any Attribute is the key that makes it useful. + While there exist numerous ways of doing it, SightingDB does it by + just counting. Whenever somebody retrieves an Attribute, this + counting is provided, allowing anyone to understand whenever + something was observed few or many times. + +1.1. Conventions and Terminology + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119 [RFC2119]. + +2. Format + +2.1. Overview + + The SightingDB format is in JSON [RFC8259] format and used to query a + SightingDB compatible connector. In SightingDB, a Sighting Object is + composed of a single JSON object. This object contains the following + fields: value, first_seen, last_seen, count, tags, ttl, frequency and + manifold. + +2.1.1. Attribute Storage + + The fields described previously describe an Attribute and all the + required characteristics. However they are stored in a Namespace. A + Namespace is similar to a path in a file-system where the same file + can be stored in multiple places. + + + + +Tricaud Expires May 6, 2020 [Page 2] + +Internet-Draft SightingDB query format November 2019 + + +2.1.2. Namespace + + A Namespace with multiple levels MUST be separated with the slash '/' + character. There is no specification on how they are structured, + since it depends on the use cases. + + A Namespace starting with the underscore '_' character means it is + private and internal to SightingDB. There are all reserved for the + engine and MUST NOT be used. + + Reserved namespaces are: _expired/: Which contains all the attributes + that expired, preserving the origin namespace _shadow/: When a value + is searched and does not exists, it is stored there _stats: + Statistics _config: Configuration _all: All the Attributes in one + place, used to retrieve the 'manifold' property. + + The Attribute Key MUST always be the last part of the Namespace. + +2.1.2.1. Sample Namespaces + + /Organization1/service/ipv4: Store values for ipv4 keys in + /Organization1/service /everything/domain: Store domains in + /everything + +2.1.3. Attribute fields + +2.1.3.1. value + + The attribute value, used to store and retrieve information about an + attribute. Note that value is not returned back in the JSON object, + since it is queried, it is known. The Value is described in a + section below, as it is very specific and can be either "as is", a + hash, encoded in base64 or any other convenient mechanism. + + The value implementation MUST offer at least: 1) Raw value 2) Base64 + URL Encoded 3) SHA256 Hash + +2.1.3.2. first_seen + + Time in UTC of the first time this value was captured + +2.1.3.3. last_seen + + Time in UTC of the last time this value was captured + + + + + + + +Tricaud Expires May 6, 2020 [Page 3] + +Internet-Draft SightingDB query format November 2019 + + +2.1.3.4. count + + How many time this value was written + +2.1.3.5. tags + + Tags follow how they are defined in MISP using the MISP Taxonomy. + Each Tag is separated with the ';' character. + +2.1.3.6. ttl + + Time To Live, represents the expiration in seconds since the time the + Attribute was created. Once it has expired, it moves in the private + Namespace _expired. + + When an Attribute has this field set to 0, it means it is not set to + expired. This is the default behavior. + + When an Attribute has this field set to a number greater than 0, the + expiration status is computed only at retrieval time. + +2.1.3.7. frequency + + Frequency is the number of time an Attribute is seen in average per + day. As this field can introduced latence, its implementation is + OPTIONAL. + +2.1.3.8. manifold + + When a given Attribute Value is stored in different namespaces, the + manifold field keeps track of them so it returns in how many + different places this attributes exists. This is a simple counter. + +2.2. SightingDB Format - One Attribute + + { + "value":"127.0.0.1", + "first_seen":1530394819, + "last_seen":1572933618, + "count":578391, + "tags":"", + "ttl":0, + "frequency":1185, + "manifold": 17 + } + + + + + + +Tricaud Expires May 6, 2020 [Page 4] + +Internet-Draft SightingDB query format November 2019 + + +3. Value + + The value submitted can be in multiple format according to the use- + case. Any implementation MUST offer three alternatives: + + 1. Raw value: where nothing is encoded and the value is stored AS + IS, such as show in the example above with the One Attribute in + JSON. + + 2. SHA256: which prevents from seeing content (see Security + Considerations), has a fixed size and is convenient for most + requirements + + 3. Base64 URL: Where the specification of Base64 is followed, except + the characters conflicting with an URL argument are replaced + + The value is configured as part of the Namespace. The private + "_config" Namespace prefix stores this value storage mechanism. + +3.1. Configuring the value format for a Namespace + + If one has the Namespace "/Organization1/BU1/ip" and want to store + those IP addresses in SHA256, it will be configured like this: The + Namespace is kept but prefixed by "_config" and has a json object + about value format set. "/_config/Organization1/BU1/ip" + + { + "value_format":"SHA256" + } + + Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". + +4. Security Considerations + + While this document solely focuses on the format, the reference + implementation is SightingDB. The authentication, the data access is + not handled by SightingDB. It is possible a value can leak if the + access is too permissive. + + Even a Hashed value can be discovered, as re-hashing known values + would match. + +5. Acknowledgements + + The author wish to thank all the MISP community who are supporting + the creation of open standards in threat intelligence sharing. As + well as amazing feedback gathered during the MISP Summit 2019 in + Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody. + + + +Tricaud Expires May 6, 2020 [Page 5] + +Internet-Draft SightingDB query format November 2019 + + +6. Normative References + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, + DOI 10.17487/RFC2119, March 1997, + . + + [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", STD 90, RFC 8259, + DOI 10.17487/RFC8259, December 2017, + . + +Author's Address + + Sebastien Tricaud + Devo Inc. + 150 Cambridgepark Drive + Cambridge, MA 02140 + USA + + Phone: +1 866-221-2254 + Email: sebastien.tricaud@devo.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Tricaud Expires May 6, 2020 [Page 6] From 9e98429bcd3459b78f4e3850bbcd6db0e70370a7 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Wed, 6 Nov 2019 22:59:52 -0800 Subject: [PATCH 26/28] Remove frequency definition as: 1) It is far from clear 2) The algorithm used to create that number must be written 3) Confusion must not happen in an RFC --- sightingdb-format/raw.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index b8814b4..b43844b 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -49,7 +49,7 @@ document are to be interpreted as described in RFC 2119 [@!RFC2119]. ## Overview -The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl, frequency and manifold. +The SightingDB format is in JSON [@!RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following fields: value, first_seen, last_seen, count, tags, ttl and manifold. ### Attribute Storage @@ -62,10 +62,15 @@ A Namespace with multiple levels **MUST** be separated with the slash '/' charac A Namespace starting with the underscore '_' character means it is private and internal to SightingDB. There are all reserved for the engine and **MUST** NOT be used. Reserved namespaces are: + _expired/: Which contains all the attributes that expired, preserving the origin namespace + _shadow/: When a value is searched and does not exists, it is stored there + _stats: Statistics + _config: Configuration + _all: All the Attributes in one place, used to retrieve the 'manifold' property. The Attribute Key MUST always be the last part of the Namespace. @@ -73,6 +78,7 @@ The Attribute Key MUST always be the last part of the Namespace. #### Sample Namespaces /Organization1/service/ipv4: Store values for ipv4 keys in /Organization1/service + /everything/domain: Store domains in /everything ### Attribute fields @@ -107,10 +113,6 @@ When an Attribute has this field set to 0, it means it is not set to expired. Th When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time. -#### frequency - -Frequency is the number of time an Attribute is seen in average per day. As this field can introduced latence, its implementation is **OPTIONAL**. - #### manifold When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many different places this attributes exists. This is a simple counter. @@ -125,7 +127,6 @@ When a given Attribute Value is stored in different namespaces, the manifold fie "count":578391, "tags":"", "ttl":0, - "frequency":1185, "manifold": 17 } ~~~~ From 6187769b4d4b8a3b3b0ca3f33fd7b4549f230265 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Wed, 6 Nov 2019 23:11:15 -0800 Subject: [PATCH 27/28] Added bulk format --- sightingdb-format/raw.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/sightingdb-format/raw.md b/sightingdb-format/raw.md index b43844b..305ad23 100755 --- a/sightingdb-format/raw.md +++ b/sightingdb-format/raw.md @@ -131,17 +131,19 @@ When a given Attribute Value is stored in different namespaces, the manifold fie } ~~~~ -# Value +## Value The value submitted can be in multiple format according to the use-case. Any implementation **MUST** offer three alternatives: 1) Raw value: where nothing is encoded and the value is stored AS IS, such as show in the example above with the One Attribute in JSON. + 2) SHA256: which prevents from seeing content (see Security Considerations), has a fixed size and is convenient for most requirements + 3) Base64 URL: Where the specification of Base64 is followed, except the characters conflicting with an URL argument are replaced The value is configured as part of the Namespace. The private "_config" Namespace prefix stores this value storage mechanism. -## Configuring the value format for a Namespace +### Configuring the value format for a Namespace If one has the Namespace "/Organization1/BU1/ip" and want to store those IP addresses in SHA256, it will be configured like this: The Namespace is kept but prefixed by "_config" and has a json object about value format set. @@ -155,6 +157,34 @@ The Namespace is kept but prefixed by "_config" and has a json object about valu Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". +## Bulk + +When data must be sent and received in large amounts, it is preferable to embed in JSON all the objects at once. As such, for reading +and writing, the format is the following: + +~~~~ +{ + "items": [ + { "/your/namespace": "127.0.0.1" }, + { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" } + ] +} +~~~~ + +Which will either store or retrieve the wanted data. + +### Response + +The response when retrieving sightings also has the list of items, in order, one per line of the results: +~~~~ +{ + "items": [ + { "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 }, + { "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 } + ] +} +~~~~ + # Security Considerations While this document solely focuses on the format, the reference implementation is SightingDB. The authentication, the data access is not handled by SightingDB. From 51c1230cd2cbd1c6cdf1532cfa4ab3df97418508 Mon Sep 17 00:00:00 2001 From: Sebastien Tricaud Date: Wed, 6 Nov 2019 23:13:23 -0800 Subject: [PATCH 28/28] Change the generated document --- sightingdb-format/raw.md.txt | 150 ++++++++++++++++++++++++----------- 1 file changed, 103 insertions(+), 47 deletions(-) diff --git a/sightingdb-format/raw.md.txt b/sightingdb-format/raw.md.txt index 4f90776..5d88547 100644 --- a/sightingdb-format/raw.md.txt +++ b/sightingdb-format/raw.md.txt @@ -68,11 +68,13 @@ Table of Contents 2.1.2. Namespace . . . . . . . . . . . . . . . . . . . . . . 3 2.1.3. Attribute fields . . . . . . . . . . . . . . . . . . 3 2.2. SightingDB Format - One Attribute . . . . . . . . . . . . 4 - 3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 - 3.1. Configuring the value format for a Namespace . . . . . . 5 - 4. Security Considerations . . . . . . . . . . . . . . . . . . . 5 - 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 - 6. Normative References . . . . . . . . . . . . . . . . . . . . 6 + 2.3. Value . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.3.1. Configuring the value format for a Namespace . . . . 5 + 2.4. Bulk . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2.4.1. Response . . . . . . . . . . . . . . . . . . . . . . 6 + 3. Security Considerations . . . . . . . . . . . . . . . . . . . 6 + 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 6 + 5. Normative References . . . . . . . . . . . . . . . . . . . . 6 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 6 1. Introduction @@ -96,8 +98,7 @@ Table of Contents The SightingDB format is in JSON [RFC8259] format and used to query a SightingDB compatible connector. In SightingDB, a Sighting Object is composed of a single JSON object. This object contains the following - fields: value, first_seen, last_seen, count, tags, ttl, frequency and - manifold. + fields: value, first_seen, last_seen, count, tags, ttl and manifold. 2.1.1. Attribute Storage @@ -108,7 +109,6 @@ Table of Contents - Tricaud Expires May 6, 2020 [Page 2] Internet-Draft SightingDB query format November 2019 @@ -124,19 +124,29 @@ Internet-Draft SightingDB query format November 2019 private and internal to SightingDB. There are all reserved for the engine and MUST NOT be used. - Reserved namespaces are: _expired/: Which contains all the attributes - that expired, preserving the origin namespace _shadow/: When a value - is searched and does not exists, it is stored there _stats: - Statistics _config: Configuration _all: All the Attributes in one - place, used to retrieve the 'manifold' property. + Reserved namespaces are: + + _expired/: Which contains all the attributes that expired, preserving + the origin namespace + + _shadow/: When a value is searched and does not exists, it is stored + there + + _stats: Statistics + + _config: Configuration + + _all: All the Attributes in one place, used to retrieve the + 'manifold' property. The Attribute Key MUST always be the last part of the Namespace. 2.1.2.1. Sample Namespaces /Organization1/service/ipv4: Store values for ipv4 keys in - /Organization1/service /everything/domain: Store domains in - /everything + /Organization1/service + + /everything/domain: Store domains in /everything 2.1.3. Attribute fields @@ -151,16 +161,6 @@ Internet-Draft SightingDB query format November 2019 The value implementation MUST offer at least: 1) Raw value 2) Base64 URL Encoded 3) SHA256 Hash -2.1.3.2. first_seen - - Time in UTC of the first time this value was captured - -2.1.3.3. last_seen - - Time in UTC of the last time this value was captured - - - @@ -170,6 +170,14 @@ Tricaud Expires May 6, 2020 [Page 3] Internet-Draft SightingDB query format November 2019 +2.1.3.2. first_seen + + Time in UTC of the first time this value was captured + +2.1.3.3. last_seen + + Time in UTC of the last time this value was captured + 2.1.3.4. count How many time this value was written @@ -191,13 +199,7 @@ Internet-Draft SightingDB query format November 2019 When an Attribute has this field set to a number greater than 0, the expiration status is computed only at retrieval time. -2.1.3.7. frequency - - Frequency is the number of time an Attribute is seen in average per - day. As this field can introduced latence, its implementation is - OPTIONAL. - -2.1.3.8. manifold +2.1.3.7. manifold When a given Attribute Value is stored in different namespaces, the manifold field keeps track of them so it returns in how many @@ -212,7 +214,6 @@ Internet-Draft SightingDB query format November 2019 "count":578391, "tags":"", "ttl":0, - "frequency":1185, "manifold": 17 } @@ -220,13 +221,12 @@ Internet-Draft SightingDB query format November 2019 - Tricaud Expires May 6, 2020 [Page 4] Internet-Draft SightingDB query format November 2019 -3. Value +2.3. Value The value submitted can be in multiple format according to the use- case. Any implementation MUST offer three alternatives: @@ -245,7 +245,7 @@ Internet-Draft SightingDB query format November 2019 The value is configured as part of the Namespace. The private "_config" Namespace prefix stores this value storage mechanism. -3.1. Configuring the value format for a Namespace +2.3.1. Configuring the value format for a Namespace If one has the Namespace "/Organization1/BU1/ip" and want to store those IP addresses in SHA256, it will be configured like this: The @@ -258,7 +258,43 @@ Internet-Draft SightingDB query format November 2019 Where "value_format" is either: "SHA256", "RAW" or "BASE64URL". -4. Security Considerations +2.4. Bulk + + When data must be sent and received in large amounts, it is + preferable to embed in JSON all the objects at once. As such, for + reading and writing, the format is the following: + +{ + "items": [ + { "/your/namespace": "127.0.0.1" }, + { "/your/other/namespace": "110812f67fa1e1f0117f6f3d70241c1a42a7b07711a93c2477cc516d9042f9db" } + ] +} + + Which will either store or retrieve the wanted data. + + + + + +Tricaud Expires May 6, 2020 [Page 5] + +Internet-Draft SightingDB query format November 2019 + + +2.4.1. Response + + The response when retrieving sightings also has the list of items, in + order, one per line of the results: + +{ + "items": [ + { "first_seen":1530337182, "last_seen":1573110615, "count":93021, "tags":"", "ttl":0, "manifold": 1 }, + { "first_seen":1562930418, "last_seen":1573110404, "count":1020492, "tags":"", "ttl":8912, "manifold": 3 } + ] +} + +3. Security Considerations While this document solely focuses on the format, the reference implementation is SightingDB. The authentication, the data access is @@ -268,21 +304,14 @@ Internet-Draft SightingDB query format November 2019 Even a Hashed value can be discovered, as re-hashing known values would match. -5. Acknowledgements +4. Acknowledgements The author wish to thank all the MISP community who are supporting the creation of open standards in threat intelligence sharing. As well as amazing feedback gathered during the MISP Summit 2019 in Luxembourg, in particular with Alexandre Dulaunoy and Andras Iklody. - - -Tricaud Expires May 6, 2020 [Page 5] - -Internet-Draft SightingDB query format November 2019 - - -6. Normative References +5. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, @@ -296,6 +325,19 @@ Internet-Draft SightingDB query format November 2019 Author's Address + + + + + + + + +Tricaud Expires May 6, 2020 [Page 6] + +Internet-Draft SightingDB query format November 2019 + + Sebastien Tricaud Devo Inc. 150 Cambridgepark Drive @@ -333,4 +375,18 @@ Author's Address -Tricaud Expires May 6, 2020 [Page 6] + + + + + + + + + + + + + + +Tricaud Expires May 6, 2020 [Page 7]