From 63c75bcb6e73362c47e47e7901b1149906962660 Mon Sep 17 00:00:00 2001 From: Alexandre Dulaunoy Date: Tue, 10 Sep 2019 14:54:00 +0200 Subject: [PATCH] chg: [blog] decaying indicator blog post updated --- _posts/2019-09-12-Decaying-Of-Indicators.md | 56 ++++++++++++++------- 1 file changed, 38 insertions(+), 18 deletions(-) diff --git a/_posts/2019-09-12-Decaying-Of-Indicators.md b/_posts/2019-09-12-Decaying-Of-Indicators.md index cbe2459..5e44e39 100644 --- a/_posts/2019-09-12-Decaying-Of-Indicators.md +++ b/_posts/2019-09-12-Decaying-Of-Indicators.md @@ -1,13 +1,21 @@ -# Problem statement -MISP being a P2P system, various users and organisations are sharing data, sometimes without even knowing each others. While having access to a lot of information is extremelly benificial for all parties, it, however, also induces challenges to deal with. +--- +title: Decaying of Indicators - MISP improved model to expire indicators based on custom models +layout: post +featured: /assets/images/misp/blog/community-view.png +--- -In this blogpost, we will mainly discuss about information **quality** and **freshness**, other issues like **trust**, **use-cases**, **interests**, etc. are partially taken into account but will not be presented. Nevertheless, these concepts are examined in this [paper](https://arxiv.org/abs/1902.03914) along with a detailed explanation of the solution we've choosen to tackle these issues. +# An improved and flexible model to expire indicators + +MISP being a distributed system, various users and organisations are sharing data, sometimes without even knowing each others. While having access to a lot of information is extremely beneficial for all parties, it, however, also induces challenges to deal with. + +In this blog post, we will mainly discuss about information **quality** and **freshness**, other issues like **trust**, **use-cases**, **interests**, etc. are partially taken into account but will not be presented. Nevertheless, these concepts are examined in this [paper](https://arxiv.org/abs/1902.03914) along with a detailed explanation of the solution we have chosen to tackle these issues. Our main objective is to provide users a **simple yet customizable system** to automatically (or manually) mark an *Indicator Of Compromise* (or more generic, an *Attribute*) as **expired**. Before getting started to show how the model presented in the paper is implemented in MISP, we first need to have a look at some concepts needed to better understand how components are working and tied together. # The (potentially) annoying bits of theory + The solution currently supported in MISP is based on two components: ``base_score`` and ``score``. The idea is to have an initial fixed value called ``base_score`` taking into account the **quality** of an indicator; and a time-dependant ``score``, which decreases the more time passes. A simplified version would be something like this: @@ -21,7 +29,8 @@ Where ``P`` is composed of ``parameters``: **⚠** **It should be noted that everytime a [*Sightings*](https://www.circl.lu/doc/misp/sightings/) is added to an *Attribute*, the ``score`` is refresh to the ``base_score`` and a new decay is initiated from that point.** # Polynomial Decaying Model built-in in MISP -We still have to see how the ``base_score`` is actually computed. In the built-in version of the *Decaying Model* in MISP, the ``base_score`` is computed from the *Taxonomies* and some weigths. Weights are a mean to prioritize extracted ``numerical_values`` from *Taxonomies* over others. + +We still have to see how the ``base_score`` is actually computed. In the built-in version of the *Decaying Model* in MISP, the ``base_score`` is computed from the *Taxonomies* and some weight. Weights are a mean to prioritize extracted ``numerical_values`` from *Taxonomies* over others. To give the intuition of how the ``base_score`` computation works, let's look at two examples. In these examples, the two *Taxonomies* used are [*phishing*](https://github.com/MISP/misp-taxonomies/blob/master/phishing/machinetag.json) and [*admiralty-scale*](https://github.com/MISP/misp-taxonomies/blob/master/admiralty-scale/machinetag.json). Both of them contain *Tags* that have a ``numerical_value`` associated to them: @@ -34,7 +43,8 @@ So, if an *Attribute* only have one *Tag* attached, let's say ``admiralty-scale: base_score = 100 ``` -Weights come into action when multiple *Tags* are attached to an *Attribute*. To make things a bit easier, let's suppose that both *Taxonomies* should have the same importance in regards to the *Attribute*'s score. Thus, the total weigth (100) will be shared, assigning both *Taxonomy* a weight of 50. +Weights come into action when multiple *Tags* are attached to an *Attribute*. To make things a bit easier, let's suppose that both *Taxonomies* should have the same importance in regards to the *Attribute*'s score. Thus, the total weight (100) will be shared, assigning both *Taxonomy* a weight of 50. + ``` admiralty-scale = 50 phishing = 50 @@ -49,16 +59,19 @@ Thus, the ``base_score`` of this *Attribute* will be ``87.50``. # Short tutorial -Now that we've seen the basic concepts, let's have a look at how MISP implents these components. For these examples, we are using the default [phishing model](https://github.com/MISP/misp-decaying-models/blob/master/models/phishing-model.json) model on a **test** *Event*. + +Now that we've seen the basic concepts, let's have a look at how MISP implements these components. For these examples, we are using the default [phishing model](https://github.com/MISP/misp-decaying-models/blob/master/models/phishing-model.json) model on a **test** *Event*. ## Practical integration in MISP ### Endpoint: ``events/view`` + At the *Event* level, a new filtering button has been added to attach the real-time computed ``score`` of any *Attributes* that has been mapped to a *Model*. -Decaying Model index +Decaying Model index ### Endpoint: ``attribute/restSearch`` + The ``attribute/restSearch`` endpoint has been updated and now supports four new parameters to filter out expired *Attributes* or play with the different available models. - ``includeDecayScore`` **[bool]**: Attach the real-time computed ``score`` of the *Attribute* along with *Model(s)* informations - ``excludeDecayed`` **[bool]**: Filter out all expired IOC @@ -83,29 +96,35 @@ Example ``` ## Default and Custom Models + In MISP, Some *Decaying Models* called **Default Models** will be supplied by default. Similarly to *Taxonomies*, *Galaxies* or *misp-objects*, *Decaying Models* will have their [own repository](https://github.com/MISP/misp-decaying-models) and will have the possibility to be updated directly from the UI via a single click. **Default Models** are available to everyone, meaning that they can been viewed and customized by any users having a presence on the MISP instance. **Custom Models** are user-defined models that are shared to other users. However, if desired, they can be hidden by turning off the sharing flag, similarly to the *Tag Collection* feature. ## Decaying Fine Tuning Tool: Setting parameters and mapping model to *Attribute* types -When creating a new *Decaying Model*, setting a parameters and viewing its impact should be as easy and straighforward as possible. To do so, few widgets are shipped with the latest version of MISP. + +When creating a new *Decaying Model*, setting a parameters and viewing its impact should be as easy and straight forward as possible. To do so, few widgets are shipped with the latest version of MISP. ### Customizing lifetime and decay speed parameters - -### Setting the ``base_score``: Customizing Taxonomies' weigth - -### Viewing scores and Simulating the model - + +### Viewing scores and simulating the model + + # Developer perspective: Creating a model using a different algorithm + The Built-in Polynomial *Decaying Model* implemented in MISP allows any user to customize various components to achieve fine-grained decay behaviors. Still, it is possible that our model doesn't encompass your specific use-case. Thanks to the implemented architecture, any other formulas or algorithms can be added and used in a straightforward way. Steps to create a new decay algorithm: @@ -136,6 +155,7 @@ class Polynomial extends DecayingModelBase ``` # Outcomes + Evaluating **quality** and **freshness** of IOCs is a problem commonly found in Threat Intelligence Platforms. We tried to solve it using a simple yet customizable system. Upon release, MISP will be shipped with few models that could fit most use-cases. Still, we are eagerly waiting for contributions, fine-tunings or feedbacks from users. This would opens up plenty of opportunities includings improved *Models*' precision, parameters tweaking or even integration of machine learning as a new *Model* algorithm. @@ -144,4 +164,4 @@ Furthermore, we are not done yet! There are already improvements cooking in the - Integration of ``False Positive`` and ``Expiration`` *Sightings* - Formula tweakings to provide better control on how to reset the ``base_score`` once a *Sighting* is created - Per-user Taxonomies' ``numerical_value`` overrides -- Weights on *Tag*'s predicate level \ No newline at end of file +- Weights on *Tag*'s predicate level