diff --git a/_posts/2019-09-12-Decaying-Of-Indicators.md b/_posts/2019-09-12-Decaying-Of-Indicators.md index 7c6f6a1..f36808e 100644 --- a/_posts/2019-09-12-Decaying-Of-Indicators.md +++ b/_posts/2019-09-12-Decaying-Of-Indicators.md @@ -6,33 +6,33 @@ featured: /assets/images/misp/blog/decay.png # An improved and flexible model to expire indicators -This blog post introduces the new functionality of MISP 2.4.116 which allows users and organisations to easily expire information depending on their objectives and targets. +This blog post aims to give a thorough introduction of a new functionality added in MISP 2.4.116, allowing users and organisations to easily expire information depending on their personalised objectives and targets. -MISP being a distributed system, various users and organisations are sharing data, sometimes without even knowing each others. While having access to a lot of information is extremely beneficial for all parties, it, however, also induces challenges to deal with. +MISP, being a distributed system, enables the sharing of data between various users and organisations, often resulting in the parties involved in an exchange not even knowing one another. Whilst having access to a large trove of information is extremely beneficial for all parties involved, however, it can also introduce a whole new set of challenges to deal with. -In this blog post, we will mainly discuss about information **quality** and **freshness**, other issues like **trust**, **use-cases**, **interests**, etc. are partially taken into account but will not be presented. Nevertheless, these concepts are examined in this [paper](https://arxiv.org/abs/1902.03914) along with a detailed explanation of the solution we have chosen to tackle these issues. +In this blog post, we will mainly touch on information **quality** and **freshness**, along with other issues such as **trust**, **use-cases** and **interests**. The latter group is partially taken into account, but will not be presented. Nevertheless, these concepts are examined more thoroughly in a [paper](https://arxiv.org/abs/1902.03914) we published that has served as our basis for the implementation, along with a detailed explanation of the solution we have chosen to tackle these issues. -Our main objective is to provide users a **simple yet customizable system** to automatically (or manually) mark an *Indicator Of Compromise* (or more generic, an *Attribute*) as **expired**. +Our main objective is to provide users with a **simple yet customisable system** to automatically (or in some cases manually) mark an *Indicator Of Compromise* (or more generically, an *Attribute*) as **expired**. -Before getting started to show how the model presented in the paper is implemented in MISP, we first need to have a look at some concepts needed to better understand how components are working and tied together. +Before getting started with showing how the model presented in the paper is implemented in MISP, we first need to have a look at some of the basic concepts required to better understand how the various components are working together. -# The (potentially) annoying bits of theory +# The (potentially) annoying bits of the theory -The solution currently supported in MISP is based on two components: ``base_score`` and ``score``. The idea is to have an initial fixed value called ``base_score`` taking into account the **quality** of an indicator; and a time-dependant ``score``, which decreases the more time passes. +The solution currently supported in MISP is based on two components: The ``base_score`` and the ``score``. The idea is to have an initial fixed value called ``base_score`` taking into account the **quality** of an indicator and a time-dependant ``score``, which decreases with the passage of time. -A simplified version would be something like this: +A simplified version would look something like this: ``` score = base_score * P ``` -Where ``P`` is composed of ``parameters``: -- ``lifetime``: The lifetime of the IOC or the time at which the score of the *Attribute*'s score will be 0 -- ``decay_speed``: The speed at which the decay happens or the speed at which an *Attribute* will loose score +Where ``P`` is composed of the following ``parameters``: +- ``lifetime``: The lifetime of the IOC or the point in time at which the score of the *Attribute*'s score will reach 0 +- ``decay_speed``: The speed at which the decay happens or the speed at which an *Attribute* will lose score -**⚠** **It should be noted that everytime a [*Sightings*](https://www.circl.lu/doc/misp/sightings/) is added to an *Attribute*, the ``score`` is refresh to the ``base_score`` and a new decay is initiated from that point.** +**⚠** **It should be noted that everytime a [*Sightings*](https://www.circl.lu/doc/misp/sightings/) is added to an *Attribute*, the ``score`` is refreshed back to the value of the original ``base_score`` and a new decay is initiated from that point forward.** # Polynomial Decaying Model built-in in MISP -We still have to see how the ``base_score`` is actually computed. In the built-in version of the *Decaying Model* in MISP, the ``base_score`` is computed from the *Taxonomies* and some weight. Weights are a mean to prioritize extracted ``numerical_values`` from *Taxonomies* over others. +We still have to see how the ``base_score`` is actually computed. In the current implementation of the *Decaying Model* in MISP, the ``base_score`` is computed from the *Taxonomies* along with the various attached weights. Weights are a means to prioritise extracted ``numerical_values`` from *Taxonomies* over others. To give the intuition of how the ``base_score`` computation works, let's look at two examples. In these examples, the two *Taxonomies* used are [*phishing*](https://github.com/MISP/misp-taxonomies/blob/master/phishing/machinetag.json) and [*admiralty-scale*](https://github.com/MISP/misp-taxonomies/blob/master/admiralty-scale/machinetag.json). Both of them contain *Tags* that have a ``numerical_value`` associated to them: @@ -40,12 +40,12 @@ To give the intuition of how the ``base_score`` computation works, let's look at - admiraly-scale:source-reliability = Not usually reliable, ``numerical_value = 25`` - phishing:psychological-acceptability = high, ``numerical_value = 75`` -So, if an *Attribute* only have one *Tag* attached, let's say ``admiralty-scale:source-reliability="Completely reliable"``, the ``base_score`` would be: +So, if an *Attribute* only has a single *Tag* attached, for example ``admiralty-scale:source-reliability="Completely reliable"``, the ``base_score`` would be: ``` base_score = 100 ``` -Weights come into action when multiple *Tags* are attached to an *Attribute*. To make things a bit easier, let's suppose that both *Taxonomies* should have the same importance in regards to the *Attribute*'s score. Thus, the total weight (100) will be shared, assigning both *Taxonomy* a weight of 50. +Weights come into play when multiple *Tags* are attached to an *Attribute*. As a simplification, let's suppose that both *Taxonomies* should have the same importance in regards to the *Attribute*'s score. Thus, the total weight (100) will be shared, assigning both *Taxonomies* a weight of 50. ``` admiralty-scale = 50 @@ -64,21 +64,21 @@ Thus, the ``base_score`` of this *Attribute* will be ``87.50``. Now that we've seen the basic concepts, let's have a look at how MISP implements these components. For these examples, we are using the default [phishing model](https://github.com/MISP/misp-decaying-models/blob/master/models/phishing-model.json) model on a **test** *Event*. -## Practical integration in MISP +## Integration in MISP ### Endpoint: ``events/view`` -At the *Event* level, a new filtering button has been added to attach the real-time computed ``score`` of any *Attributes* that has been mapped to a *Model*. +At the *Event* level, a new filter button has been added, which attaches the real-time computed ``score`` to all *Attributes* that have been mapped to a *Model*. Decaying Model index ### Endpoint: ``attribute/restSearch`` -The ``attribute/restSearch`` endpoint has been updated and now supports four new parameters to filter out expired *Attributes* or play with the different available models. -- ``includeDecayScore`` **[bool]**: Attach the real-time computed ``score`` of the *Attribute* along with *Model(s)* informations -- ``excludeDecayed`` **[bool]**: Filter out all expired IOC -- ``decayingModel`` **[list]**: List of *Model(s)*, which will be attached to the *Attribute* -- ``modelOverrides`` **[dict]**: JSON that can be used to on-the-fly modify *Model(s)* parameters +The ``attribute/restSearch`` endpoint has been updated and now supports four new parameters to help with filtering out expired *Attributes* or to simply play with the different available models. +- ``includeDecayScore`` **[bool]**: Attach the real-time computed ``score`` of the *Attribute* along with the associated *Model* information +- ``excludeDecayed`` **[bool]**: Filter out all expired IOCs +- ``decayingModel`` **[list]**: List of *Model(s)* that will be attached to the given *Attributes* +- ``modelOverrides`` **[dict]**: JSON that can be used to modify *Model* parameters on-the-fly Example ``` @@ -99,21 +99,21 @@ Example ## Default and Custom Models -In MISP, Some *Decaying Models* called **Default Models** will be supplied by default. Similarly to *Taxonomies*, *Galaxies* or *misp-objects*, *Decaying Models* will have their [own repository](https://github.com/MISP/misp-decaying-models) and will have the possibility to be updated directly from the UI via a single click. **Default Models** are available to everyone, meaning that they can been viewed and customized by any users having a presence on the MISP instance. +In MISP, Some *Decaying Models*, called **Default Models**, will be supplied by default. Similarly to *Taxonomies*, *Galaxies* and *misp-objects*, *Decaying Models* will have their [own repository](https://github.com/MISP/misp-decaying-models) along with the possibility to be updated directly from both the API and the UI via a single click. **Default Models** are available to everyone, meaning that they can been viewed and customised by any user having a presence on the MISP instance. -**Custom Models** are user-defined models that are shared to other users. However, if desired, they can be hidden by turning off the sharing flag, similarly to the *Tag Collection* feature. +**Custom Models** are user-defined models that are shared with other users. However, if desired, they can be kept private to one's own organisation by turning off the sharing flag, similarly to how the *Tag Collection* feature handles the same concept. ## Decaying Fine Tuning Tool: Setting parameters and mapping model to *Attribute* types -When creating a new *Decaying Model*, setting a parameters and viewing its impact should be as easy and straight forward as possible. To do so, few widgets are shipped with the latest version of MISP. +When creating a new *Decaying Model*, modifying its parameters and viewing the outcome of the any changes should be as easy and straight forward as possible. In order to do so, several widgets are included with the latest version of MISP. -### Customizing lifetime and decay speed parameters +### Customising the lifetime and the decay speed parameters -### Setting the ``base_score``: Customizing Taxonomies' weight +### Setting the ``base_score``: Customising Taxonomies' weights