MISP
/
misp-website
mirror of https://github.com/MISP/misp-website
2
0
Fork 0
Code Issues Releases Wiki Activity

Update 2019-09-12-Decaying-Of-Indicators.md

pull/15/head
Andras Iklody 2019-09-13 09:49:45 +02:00 committed by GitHub
parent 65a18d9a17
commit 2e09902448
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 37 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

74
_posts/2019-09-12-Decaying-Of-Indicators.md
View File

@ -6,33 +6,33 @@ featured: /assets/images/misp/blog/decay.png
# An improved and flexible model to expire indicators
This blog post introduces the new functionality of MISP 2.4.116 which allows users and organisations to easily expire information depending on their objectives and targets.
This blog post aims to give a thorough introduction of a new functionality added in MISP 2.4.116, allowing users and organisations to easily expire information depending on their personalised objectives and targets.
MISP being a distributed system, various users and organisations are sharing data, sometimes without even knowing each others. While having access to a lot of information is extremely beneficial for all parties, it, however, also induces challenges to deal with.
MISP, being a distributed system, enables the sharing of data between various users and organisations, often resulting in the parties involved in an exchange not even knowing one another. Whilst having access to a large trove of information is extremely beneficial for all parties involved, however, it can also introduce a whole new set of challenges to deal with.
In this blog post, we will mainly discuss about information **quality** and **freshness**, other issues like **trust**, **use-cases**, **interests**, etc. are partially taken into account but will not be presented. Nevertheless, these concepts are examined in this [paper](https://arxiv.org/abs/1902.03914) along with a detailed explanation of the solution we have chosen to tackle these issues.
In this blog post, we will mainly touch on information **quality** and **freshness**, along with other issues such as **trust**, **use-cases** and **interests**. The latter group is partially taken into account, but will not be presented. Nevertheless, these concepts are examined more thoroughly in a [paper](https://arxiv.org/abs/1902.03914) we published that has served as our basis for the implementation, along with a detailed explanation of the solution we have chosen to tackle these issues.
Our main objective is to provide users a **simple yet customizable system** to automatically (or manually) mark an *Indicator Of Compromise* (or more generic, an *Attribute*) as **expired**.
Our main objective is to provide users with a **simple yet customisable system** to automatically (or in some cases manually) mark an *Indicator Of Compromise* (or more generically, an *Attribute*) as **expired**.
Before getting started to show how the model presented in the paper is implemented in MISP, we first need to have a look at some concepts needed to better understand how components are working and tied together.
Before getting started with showing how the model presented in the paper is implemented in MISP, we first need to have a look at some of the basic concepts required to better understand how the various components are working together.
# The (potentially) annoying bits of theory
# The (potentially) annoying bits of the theory
The solution currently supported in MISP is based on two components: ``base_score`` and ``score``. The idea is to have an initial fixed value called ``base_score`` taking into account the **quality** of an indicator; and a time-dependant ``score``, which decreases the more time passes.
The solution currently supported in MISP is based on two components: The ``base_score`` and the ``score``. The idea is to have an initial fixed value called ``base_score`` taking into account the **quality** of an indicator and a time-dependant ``score``, which decreases with the passage of time.
A simplified version would be something like this:
A simplified version would look something like this:
```
score = base_score * P
```
Where ``P`` is composed of ``parameters``:
- ``lifetime``: The lifetime of the IOC or the time at which the score of the *Attribute*'s score will be 0
- ``decay_speed``: The speed at which the decay happens or the speed at which an *Attribute* will loose score
Where ``P`` is composed of the following ``parameters``:
- ``lifetime``: The lifetime of the IOC or the point in time at which the score of the *Attribute*'s score will reach 0
- ``decay_speed``: The speed at which the decay happens or the speed at which an *Attribute* will lose score
**⚠** **It should be noted that everytime a [*Sightings*](https://www.circl.lu/doc/misp/sightings/) is added to an *Attribute*, the ``score`` is refresh to the ``base_score`` and a new decay is initiated from that point.**
**⚠** **It should be noted that everytime a [*Sightings*](https://www.circl.lu/doc/misp/sightings/) is added to an *Attribute*, the ``score`` is refreshed back to the value of the original ``base_score`` and a new decay is initiated from that point forward.**
# Polynomial Decaying Model built-in in MISP
We still have to see how the ``base_score`` is actually computed. In the built-in version of the *Decaying Model* in MISP, the ``base_score`` is computed from the *Taxonomies* and some weight. Weights are a mean to prioritize extracted ``numerical_values`` from *Taxonomies* over others.
We still have to see how the ``base_score`` is actually computed. In the current implementation of the *Decaying Model* in MISP, the ``base_score`` is computed from the *Taxonomies* along with the various attached weights. Weights are a means to prioritise extracted ``numerical_values`` from *Taxonomies* over others.
To give the intuition of how the ``base_score`` computation works, let's look at two examples. In these examples, the two *Taxonomies* used are
[*phishing*](https://github.com/MISP/misp-taxonomies/blob/master/phishing/machinetag.json) and [*admiralty-scale*](https://github.com/MISP/misp-taxonomies/blob/master/admiralty-scale/machinetag.json). Both of them contain *Tags* that have a ``numerical_value`` associated to them:
@ -40,12 +40,12 @@ To give the intuition of how the ``base_score`` computation works, let's look at
- <img src="/assets/images/misp/blog/decaying//tag-as-D.png" alt="admiraly-scale:source-reliability = Not usually reliable" width="300"/>, ``numerical_value = 25``
- <img src="/assets/images/misp/blog/decaying//tag-p-H.png" alt="phishing:psychological-acceptability = high" width="250"/>, ``numerical_value = 75``
So, if an *Attribute* only have one *Tag* attached, let's say ``admiralty-scale:source-reliability="Completely reliable"``, the ``base_score`` would be:
So, if an *Attribute* only has a single *Tag* attached, for example ``admiralty-scale:source-reliability="Completely reliable"``, the ``base_score`` would be:
```
base_score = 100
```
Weights come into action when multiple *Tags* are attached to an *Attribute*. To make things a bit easier, let's suppose that both *Taxonomies* should have the same importance in regards to the *Attribute*'s score. Thus, the total weight (100) will be shared, assigning both *Taxonomy* a weight of 50.
Weights come into play when multiple *Tags* are attached to an *Attribute*. As a simplification, let's suppose that both *Taxonomies* should have the same importance in regards to the *Attribute*'s score. Thus, the total weight (100) will be shared, assigning both *Taxonomies* a weight of 50.
```
admiralty-scale = 50
@ -64,21 +64,21 @@ Thus, the ``base_score`` of this *Attribute* will be ``87.50``.
Now that we've seen the basic concepts, let's have a look at how MISP implements these components. For these examples, we are using the default [phishing model](https://github.com/MISP/misp-decaying-models/blob/master/models/phishing-model.json) model on a **test** *Event*.
## Practical integration in MISP
## Integration in MISP
### Endpoint: ``events/view``
At the *Event* level, a new filtering button has been added to attach the real-time computed ``score`` of any *Attributes* that has been mapped to a *Model*.
At the *Event* level, a new filter button has been added, which attaches the real-time computed ``score`` to all *Attributes* that have been mapped to a *Model*.
<img src="/assets/images/misp/blog/decaying/dm-event.png" alt="Decaying Model index" width="700"/>
### Endpoint: ``attribute/restSearch``
The ``attribute/restSearch`` endpoint has been updated and now supports four new parameters to filter out expired *Attributes* or play with the different available models.
- ``includeDecayScore`` **[bool]**: Attach the real-time computed ``score`` of the *Attribute* along with *Model(s)* informations
- ``excludeDecayed`` **[bool]**: Filter out all expired IOC
- ``decayingModel`` **[list]**: List of *Model(s)*, which will be attached to the *Attribute*
- ``modelOverrides`` **[dict]**: JSON that can be used to on-the-fly modify *Model(s)* parameters
The ``attribute/restSearch`` endpoint has been updated and now supports four new parameters to help with filtering out expired *Attributes* or to simply play with the different available models.
- ``includeDecayScore`` **[bool]**: Attach the real-time computed ``score`` of the *Attribute* along with the associated *Model* information
- ``excludeDecayed`` **[bool]**: Filter out all expired IOCs
- ``decayingModel`` **[list]**: List of *Model(s)* that will be attached to the given *Attributes*
- ``modelOverrides`` **[dict]**: JSON that can be used to modify *Model* parameters on-the-fly
Example
```
@ -99,21 +99,21 @@ Example
## Default and Custom Models
In MISP, Some *Decaying Models* called **Default Models** will be supplied by default. Similarly to *Taxonomies*, *Galaxies* or *misp-objects*, *Decaying Models* will have their [own repository](https://github.com/MISP/misp-decaying-models) and will have the possibility to be updated directly from the UI via a single click. **Default Models** are available to everyone, meaning that they can been viewed and customized by any users having a presence on the MISP instance.
In MISP, Some *Decaying Models*, called **Default Models**, will be supplied by default. Similarly to *Taxonomies*, *Galaxies* and *misp-objects*, *Decaying Models* will have their [own repository](https://github.com/MISP/misp-decaying-models) along with the possibility to be updated directly from both the API and the UI via a single click. **Default Models** are available to everyone, meaning that they can been viewed and customised by any user having a presence on the MISP instance.
**Custom Models** are user-defined models that are shared to other users. However, if desired, they can be hidden by turning off the sharing flag, similarly to the *Tag Collection* feature.
**Custom Models** are user-defined models that are shared with other users. However, if desired, they can be kept private to one's own organisation by turning off the sharing flag, similarly to how the *Tag Collection* feature handles the same concept.
## Decaying Fine Tuning Tool: Setting parameters and mapping model to *Attribute* types
When creating a new *Decaying Model*, setting a parameters and viewing its impact should be as easy and straight forward as possible. To do so, few widgets are shipped with the latest version of MISP.
When creating a new *Decaying Model*, modifying its parameters and viewing the outcome of the any changes should be as easy and straight forward as possible. In order to do so, several widgets are included with the latest version of MISP.
### Customizing lifetime and decay speed parameters
### Customising the lifetime and the decay speed parameters
<video src="/assets/images/misp/blog/decaying/dm-tool.mp4" title="Decaying Model Fine Tuning Tool - Parameters" width="800" height="450" controls autoplay loop>
Your browser does not support the video tag.
</video>
### Setting the ``base_score``: Customizing Taxonomies' weight
### Setting the ``base_score``: Customising Taxonomies' weights
<video src="/assets/images/misp/blog/decaying/dm-bs.mp4" type="video/mp4" title="Decaying Model Fine Tuning Tool - Base score" width="800" height="450" controls autoplay loop>
Your browser does not support the video tag.
@ -127,12 +127,12 @@ When creating a new *Decaying Model*, setting a parameters and viewing its impac
# Developer perspective: Creating a model using a different algorithm
The Built-in Polynomial *Decaying Model* implemented in MISP allows any user to customize various components to achieve fine-grained decay behaviors. Still, it is possible that our model doesn't encompass your specific use-case. Thanks to the implemented architecture, any other formulas or algorithms can be added and used in a straightforward way.
The Built-in Polynomial *Decaying Model* implemented in MISP allows users to customise various of its components in order to achieve the desired decay behaviors in a fine-grained manner. Nevertheless, even with the flexibility offered by the system, it is absolutely possible that our model doesn't encompass your specific use-case's needs. Thanks to the implemented architecture, any other formulae or algorithms can be added and used in a straightforward way.
Steps to create a new decay algorithm:
- Create a new file ``$filename`` in ``app/Model/DecayingModelsFormulas/``
- Extend the **Base** class ``DecayingModelBase``
- Implement the two functions ``computeScore`` and ``isDecayed`` with you own formula/algorithm
- Extend the **Base** class as defined in ``DecayingModelBase``
- Implement the two mandatory functions ``computeScore`` and ``isDecayed`` using your own formula/algorithm
- Create a *Model* and set the ``formula`` field to ``$filename``
```
@ -157,14 +157,14 @@ class Polynomial extends DecayingModelBase
?>
```
# Outcomes
# Outcome
Evaluating **quality** and **freshness** of IOCs is a problem commonly found in Threat Intelligence Platforms. We tried to solve it using a simple yet customizable system.
Evaluating the **quality** and **freshness** of IOCs is a problem commonly found in Threat Intelligence Platforms. We have tried to solve it using a simple yet customisable system.
Upon release, MISP will be shipped with few models that could fit most use-cases. Still, we are eagerly waiting for contributions, fine-tunings or feedbacks from users. This would opens up plenty of opportunities including improved *Models*' precision, parameters tweaking or even integration of machine learning as a new *Model* algorithm.
Event though the small set of models included in the upcoming MISP release should fit the most common use-cases, we are eagerly awaiting any contributions, fine-tunings or feedback from users. We consider the current implementation to be a foundation, upon which we want to gradually build using both our own findings and the feedback of the community. These would opens us up to plenty of opportunities, including the inclusion of new *Models*, improvements to the precision of existing *Models*', tweaking of the various parameters and even the integration of machine learning algorithms as new *Models*.
Furthermore, we are not done yet! There are already improvements cooking in the MISP-Project oven,
Furthermore, we are not done yet! There are already a bunch of improvements cooking at the MISP-Project kitchen,
- Integration of ``False Positive`` and ``Expiration`` *Sightings*
- Formula tweaking to provide better control on how to reset the ``base_score`` once a *Sighting* is created
- Per-user Taxonomies' ``numerical_value`` overrides
- Weights on *Tag*'s predicate level
- Tweaks to the formula in order to provide better controls in regards to reseting the ``base_score`` once a *Sighting* has been added
- Per-user overrides for the Taxonomies' ``numerical_values``
- Weights on a *Tag*'s predicate level