Merge branch 'main' of github.com:MISP/misp-book into main

CONTRIBUTING.md

@@ -0,0 +1,78 @@
+# Contributing to MISP Book
+
+Most of MISP's documentation pages are stored as plain text files in this dedicated misp-book repository. 
+MISP Book is generic documentation for MISP including usage, API documentation, best practices, and specific configuration settings.
+The installation guides are held in the [INSTALL](https://github.com/MISP/MISP/tree/2.4/INSTALL) and [download](https://www.misp-project.org/download/) sections of the core and website repositories respectively.
+By cloning and regularly pulling from this repo, users can maintain their own up-to-date offline copy of MISP documentation rather than relying solely on the web.
+
+MISP book is generated in HTML, PDF, epub, and mobi using [Honkit](https://github.com/honkit/honkit), which is a framework to write documentation in Markdown format. 
+The documentation is regularly published on [misp-project.org](https://www.misp-project.org/) and [circl.lu website](https://www.circl.lu/doc/misp/).
+
+## Documentation guidelines
+
+The documentation is a community effort. Volunteers work hard trying to keep everything accurate and comprehensive. 
+If you notice a problem or some way it can be improved, please edit the documentation.
+
+All pull requests against MISP must pass review before being merged. 
+This process is designed to ensure that contributed text is accurate and non-malicious. 
+Once a pull request passes review, the documentation maintainer then verifies that the pull request is mechanically sound (no merge conflicts or broken links). 
+If so, the documentation maintainer then merges the pull request. 
+
+If you have a question about something you read in the documentation, you may ask into the [Gitter Support channel](https://gitter.im/MISP/Support) or open an issue for it. 
+If you see that something in the documentation should be fixed or improved, please [contribute the change](#how-to-make-a-contributution) yourself. 
+
+## How to make a contribution
+
+Editing the documentation is easy, so if you see that a change should be made, please contribute it!
+
+A few notes before we get started:
+
+- Every documentation change will be reviewed before it’s accepted. This allows us to maintain quality control and protect our users.
+- If your contribution would take a lot of time, please [file an issue](https://github.com/MISP/misp-book/issues) for it first so that we can make sure we’re on the same page before significant work begins. This ensures you do not spend time and effort on a contribution that we can’t accept. 
+
+Thanks to GitHub’s interface, you can edit MISP's documentation even if you don’t know git at all. 
+All you need to do to contribute is to [fork and clone](https://guides.github.com/activities/forking/) the [misp-book repo](https://github.com/MISP/misp-book/), make your changes, then [submit a pull request](https://help.github.com/articles/using-pull-requests/)). 
+You can also [open issues](https://github.com/MISP/misp-book/issues) if you find any errors or propose changes.
+
+[TravisCI](https://travis-ci.org/MISP) (used for automatic integration) is included in misp-book and the book generation is tested at each commit. 
+When you make a pull request, verify if your changes affect the result of the tests. 
+If the checks fail, review the output at Travis and make fixes where possible. 
+
+### For native English speakers
+
+Most MISP developers are not native English speakers so you're more than welcome to correct or improve our English. 
+For this, you can either submit a pull request or use another way that suits you better to share your improvements with us.
+
+
+## Organizational guidelines
+
+- Do not duplicate documentation. Duplicating documentation is almost always a bad idea. There are many reasons for this. The main one is that almost all documentation has to be updated at some point. When similar documentation appears in more than one place, it is very easy for it to get updated in one place but not the others (perhaps because the person updating it doesn’t realize it’s in more than one place). Such traps are often more harmful than if the documentation never existed in the first place. The solution is to link to existing documentation rather than duplicating it. There are some exceptions to this policy (e.g., information that is certain not to change for a very long time), but they are rare.
+- We maintain only one set of documentation for MISP. We do not maintain a different set of documentation for each version of MISP. Our single set of MISP documentation is updated on a continual, rolling basis. Our first priority is to document all current, stable releases of MISP. Our second priority is to document the next, upcoming release (if any) that is currently in the beta or release candidate stage.
+- Instructions on how to install MISP are held in [INSTALL](https://github.com/MISP/MISP/tree/2.4/INSTALL) and [download](https://www.misp-project.org/download/) sections of the core and website repositories respectively. 
+
+## Style guidelines
+
+- Familiarize yourself with the terms defined in the [glossary](https://www.circl.lu/doc/misp/GLOSSARY.html). Use these terms consistently and accurately throughout your writing.
+- The language in this book is American English. All the screenshots and examples are in English.
+
+## Markdown conventions
+
+All the documentation is written in Markdown for maximum accessibility. When making contributions, please try to observe the following style conventions:
+
+- Use spaces instead of tabs.
+- To enable offline browsing, always use relative (rather than absolute) links, e.g., misp/automation/ instead of https://www.circl.lu/doc/misp/automation/. Examples of exceptions:
+    - URLs that appear inside code blocks (e.g., in comments and code snippets)
+    - Files like README.md and CONTRIBUTING.md
+- Insert a newline at, and only at, the end of each sentence. This practice results in one sentence per line, which is most appropriate for sources that consist primarily of natural language text. It results in the most useful diffs and facilitates translation into other languages while mostly preserving source readability.
+- If appropriate, make numerals in numbered lists match between Markdown source and HTML output. If a user is required to read the Markdown source directly, this will make it easier to follow, e.g., numbered steps in a set of instructions.
+- Use Atx-style headings: `# h1`, `## h2`, `### h3`, etc.
+- When writing code blocks, use syntax highlighting where possible and use [...] for anything omitted.
+([This](https://daringfireball.net/projects/markdown/) is a great source for learning about Markdown.)
+
+## Git conventions
+
+Please try to write good commit messages, according to the [instructions in our wiki](https://github.com/MISP/MISP/wiki/CommitMessageBestPractices).
+
+## Translate MISP Book
+
+You can help us make MISP Book available in your language. See [MISP and Internationalization (i18n)](https://www.circl.lu/doc/misp/translation/).

GLOSSARY.md

@@ -51,7 +51,7 @@ MISP can now extend an event (starting from version 2.4.90). This allows users t
 
 ## MISP feeds
 MISP includes a set of public OSINT feeds in its default configuration. The feeds can be used as a source of correlations for all of your events and attributes without the need to import them directly into your system. The MISP feed system allows for fast correlation but also a for quick comparisons of the feeds against one another.
-To get started with MISP we advise to enable the CIRCL OSINT feed withing your MISP instance. This feed is generated with the PyMISP [feed-generator](https://github.com/CIRCL/PyMISP/tree/master/examples/feed-generator).
+To get started with MISP we advise to enable the CIRCL OSINT feed within your MISP instance. This feed is generated with the PyMISP [feed-generator](https://github.com/CIRCL/PyMISP/tree/master/examples/feed-generator).
 [More](http://www.misp-project.org/feeds/)
 
 ## MISP format

README.md

@@ -25,7 +25,7 @@ and many other contributors especially the ones during the [MISP hackathons](htt
 
 ## Contributing
 
-We welcome contributions to the MISP book. If you want to contribute, fork the [misp-book](https://github.com/MISP/misp-book) repository and pull a request with your changes. You can also [open issues](https://github.com/MISP/misp-book/issues) if you find any errors or propose changes.
+We welcome contributions to the MISP book. If you want to contribute, see our [contributing guide](/CONTRIBUTING.md) 
 
 <div class="pagebreak"></div>
 
@@ -46,5 +46,3 @@ The MISP user guide is dual-licensed under [GNU Affero General Public License ve
 * Copyright \(C\) 2018 Camille Schneider
 * Copyright \(C\) 2018-2020 Steve Clement
 * Copyright \(C\) 2021 Jeroen Pinoy
-
-

USAGE.md

@@ -212,7 +212,7 @@ canvas needs to be compiled and needs the following dependencies:
 xcode-select --install
 # If you have homebrew not installed yet:
 ## /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
-# For the more adventureous you can install a cask of calibre which gives you access to *ebook-convert*
+# For the more adventurous you can install a cask of calibre which gives you access to *ebook-convert*
 ## brew cask install calibre
 brew install pkg-config cairo pango libpng jpeg giflib
 ```

administration/README.md

@@ -766,7 +766,7 @@ Warning: Scheduled tasks come with a lot of caveats and little in regards of cus
 """
 
 The task scheduler is a sub-par component to enable minimal functionality in terms of automating certain MISP tasks.
-If you have a dedicated and concious MISP Site Admin she can keep an eye on the Scheduler to make sure everything runs smoothly.
+If you have a dedicated and conscious MISP Site Admin she can keep an eye on the Scheduler to make sure everything runs smoothly.
 
 For better performance please use a real scheduler like your systems' crontab.
 As a rule of thumb: If you can click on it, MISP can automate it.

appendices/README.md

@@ -665,4 +665,4 @@ Because LDAP and MISP users are paired by e-mail address, it is possible to migr
 * When a user is disabled in LDAP and also in MISP and then enabled in LDAP, it will be enabled in MISP for next login just when `updateUser` is set to `true`.
 * Currently it is not possible to log in with both LDAP and local (MISP) accounts.
 * Admins can change users email address. But when `updateUser` is set to true, when the user will log in again, the e-mail address will be updated from LDAP.
-* `Security.require_password_confirmation` setting currently doesnt work with LDAP authentication. But on the other hand, since user cannot change e-mail address and password, this setting is not important.
+* `Security.require_password_confirmation` setting currently does not work with LDAP authentication. But on the other hand, since user cannot change e-mail address and password, this setting is not important.

automation/README.md

@@ -84,7 +84,7 @@ curl --header "Authorization: YOUR API KEY " --header "Accept: application/json"
 
 ## Search
 
-It is possible to search in the database for a list of attributes or events based on a list of criterias.
+It is possible to search in the database for a list of attributes or events based on a list of criteria.
 
 To return attributes or events in a desired format, use the following URL and header settings:
 
@@ -139,19 +139,27 @@ Find below a non exhaustive list of parameters that can be used to filter data i
 - **withAttachments**: If set, encodes the attachments / zipped malware samples as base64 in the data field within each attribute
 - **metadata**: Only the metadata (event, tags, relations) is returned, attributes and proposals are omitted.
 - **uuid**: Restrict the results by uuid.
-- **publish_timestamp**: Restrict the results by the timestamp of the last publishing of the event. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **publish_timestamp**: Restrict the results by the timestamp of the last publishing of the event. The input can be a timsetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
-- **last**: (Deprecated synonym for publish_timestamp) Restrict the results by the timestamp of the last publishing of the event. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **last**: (Deprecated synonym for publish_timestamp) Restrict the results by the timestamp of the last publishing of the event. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
-- **timestamp**: Restrict the results by the timestamp (last edit). Any event with a timestamp newer than the given timestamp will be returned. In case you are dealing with /attributes as scope, the attribute's timestamp will be used for the lookup. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **timestamp**: Restrict the results by the timestamp (last edit). Any event with a timestamp newer than the given timestamp will be returned. In case you are dealing with /attributes as scope, the attribute's timestamp will be used for the lookup. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
 - **published**: Set whether published or unpublished events should be returned. Do not set the parameter if you want both.
 - **enforceWarninglist**: Remove any attributes from the result that would cause a hit on a warninglist entry.
-- **to_ids**: By default (0) all attributes are returned that match the other filter parameters, irregardless of their to_ids setting. To restrict the returned data set to to_ids only attributes set this parameter to 1. You can only use the special "exclude" setting to only return attributes that have the to_ids flag disabled.
+- **to_ids**: By default (0) all attributes are returned that match the other filter parameters, regardless of their to_ids setting. To restrict the returned data set to to_ids only attributes set this parameter to 1. You can only use the special "exclude" setting to only return attributes that have the to_ids flag disabled.
-- **deleted**: If this parameter is set to 1, it will return soft-deleted attributes along with active ones. By using "only" as a parameter it will limit the returned data set to soft-deleted data only.
+- **deleted**: Default value 0. If set to 1, only deleted attributes will be returned. If set to [0,1] , both deleted and non-deleted attributes wil be returned.
 - **includeEventUuid**: Instead of just including the event ID, also include the event UUID in each of the attributes.
-- **event_timestamp**: Only return attributes from events that have received a modification after the given timestamp. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **event_timestamp**: Only return attributes from events that have received a modification after the given timestamp. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
 - **sgReferenceOnly**: If this flag is set, sharing group objects will not be included, instead only the sharing group ID is set.
 - **eventinfo**: Filter on the event's info field.
 - **searchall**: Search for a full or a substring (delimited by % for substrings) in the event info, event tags, attribute tags, attribute values or attribute comment fields.
 - **attackGalaxy**: Select the ATT&CK matrix like galaxy to use when using returnFormat = attack. Defaults to the Mitre ATT&CK library via mitre-attack-pattern.
+- **order**: Only available for /events/restSearch. Sorts the results, can be useful in combination with limit. The accepted values to sort on are "Event.[any_field_in_the_events_table] [asc|desc]". You can also sort on multiple columns, see example below.
+~~~
+{
+    "returnFormat": "json",
+    "limit": "3",
+    "order": "Event.attribute_count desc, Event.date desc"
+}
+~~~
 
 ## Events management
 
@@ -844,6 +852,251 @@ https://<misp url>/attributes/describeTypes
 
 Depending on the headers passed the returned data will be a JSON object or an XML, with 3 main sections: types, categories, category\_type\_mappings.
 
+### POST /attributes/restSearch
+
+Do not use this function with GET!
+
+#### Parameters
+
+- **returnFormat**: The format to return data in. Allowed formats:
+  - **attack-sightings**: Returns ATTA&CK Sightings in json format for
+  attributes with mitre-attack-pattern galaxies attached. For further details on the ATT&CK Sightings, please visit the related [MITRE website page](https://attack.mitre.org/resources/sightings/).
+  - **cache**: Hashes the attributes and returns them as txt. A hashing algorithm can be chosen by also adding the hash_type parameter. Supported hashing algorithms can be found on the [PHP website](https://www.php.net/manual/en/function.hash-algos.php]).
+  - **count**: Returns the attribute count as txt.
+  - **csv**
+  - **hashes**: Returns hash attributes in txt format. For composite attributes, only the hash part is returned. 
+  - **json**
+  - **netfilter**: Returns netfilter rules for IPs. Action can be set with the  netfilter_action parameter. The default action is DROP.
+  - **opendata**: Please refer to the related MISP project [blog post](https://www.misp-project.org/2020/07/30/publishing-open-data-using-MISP.html).
+  - **openioc**
+  - **rpz**
+  - **snort**
+  - **suricata**
+  - **text**: Returns only the attribute values in text format.
+  - **xml**
+  - **yara**:
+  - **yara-json**
+- **value**: Search for the given value in the attributes' value field.
+- **type**: The attribute type, any valid MISP attribute type is accepted.
+- **category**: The attribute category, any valid MISP attribute category is accepted.
+- **org**: Search by the creator organisation by supplying the organisation identifier.
+- **tags**: Include or exclude attributes with certain tags. See example below. It is strongly recommended to specifically exclude the tags you want to avoid, even if the tags should be exclusive, for example tlp:red and tlp:green.
+~~~~json
+{
+    "returnFormat": "json",
+    "tags": {
+        "NOT": [
+            "tlp:red"
+        ],
+        "OR": [
+            "tlp:green"
+        ]
+    }
+}
+~~~~
+- **from**: Will return attributes from events with the date set to a date after the one specified in the from field (format: 2015-02-15).
+- **to**: Will return attributes with the date set to a date before the one specified in the to field (format: 2015-02-15).
+- **last**: ***Deprecated!!!*** (synonym for publish_timestamp) Restrict the results by the timestamp of the last publication of the event. Any attribute with a last publication timestamp newer than the given timestamp will be returned. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **eventid**: The events that should be included / excluded from the search.
+- **withAttachments**: If set, encodes the attachments / zipped malware samples as base64 in the data field within each attribute
+- **uuid**: Restrict the results by uuid.
+- **publish_timestamp**: Restrict the results by the timestamp of the last publication of the event. Any attribute with a last publication timestamp newer than the given timestamp will be returned. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **published**: Set whether published or unpublished events should be returned. Do not set the parameter if you want both.
+- **timestamp**: ***Deprecated!!!*** (synonym for attribute_timestamp) Restrict the results by the timestamp (last edit). Any attribute with a timestamp newer than the given timestamp will be returned. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **enforceWarninglist**: Remove any attributes from the result that would cause a hit on a warninglist entry.
+- **to_ids**: By default (0) all attributes are returned that match the other filter parameters, regardless of their to_ids setting. To restrict the returned data set to to_ids only attributes set this parameter to 1. You can only use the special "exclude" setting to only return attributes that have the to_ids flag disabled.
+- **deleted**: Default value 0. If set to 1, only deleted attributes will be returned. If set to [0,1] , both deleted and non-deleted attributes wil be returned.
+- **includeEventUuid**: Instead of just including the event ID, also include the event UUID in each of the attributes.
+- **event_timestamp**: Only return attributes from events that have received a modification after the given timestamp. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **threat_level_id**: Only return attributes of events with the given threat level id(s). 1 = High, 2 = Medium, 3=Low, 4 = Undefined. See example below.
+```
+{
+    "returnFormat": "json",
+    "threat_level_id": [1,2]
+}
+```
+- **includeEventTags**: If set to 1, the event tags of the event the attribute belongs to will be added to the attribute.
+- **limit**: Limit the number of results returned, for example 10 attributes.
+- **page**: If a limit is set, sets the page to be returned. page 3, limit 100 will return records 201->300).
+- **requested_attributes**: Only for CSV export. Choose the fields you want in the csv output. Available fields are (*non-exhaustive list, more fields can be available depending on the values of other parameters*):
+  - uuid
+  - event_id
+  - category
+  - type
+  - value
+  - comment
+  - to_ids
+  - date
+  - object_relation
+  - attribute_tag
+  - object_uuid
+  - object_name
+  - object_meta_category
+  - event_info. Only available if includeContext parameter is set to 1.
+  - event_member_org. Only available if includeContext parameter is set to 1.
+  - event_source_org. Only available if includeContext parameter is set to 1.
+  - event_distribution. Only available if includeContext parameter is set to 1.
+  - event_threat_level_id. Only available if includeContext parameter is set to 1.
+  - event_analysis. Only available if includeContext parameter is set to 1.
+  - event_date. Only available if includeContext parameter is set to 1.
+  - event_tag. Only available if includeContext parameter is set to 1.
+  - event_timestamp. Only available if includeContext parameter is set to 1.
+- **includeContext**: Adds  extra event level context to the output. For each attribute more details are added to the Event object in the output. Please note that this significantly bloats the output data. Example below.
+```
+"Event": {
+          "id": "31",
+          "orgc_id": "1",
+          "org_id": "1",
+          "date": "2021-03-11",
+          "threat_level_id": "1",
+          "info": "Correlation 2",
+          "published": true,
+          "uuid": "0bfe7bf3-f793-4761-8450-8b30ca9d9964",
+          "analysis": "0",
+          "timestamp": "1616972381",
+          "distribution": "1",
+          "publish_timestamp": "1616972392",
+          "sharing_group_id": "0",
+          "extends_uuid": "",
+          "Tag": [],
+          "Orgc": {
+            "id": "1",
+            "name": "SHARINGORG",
+            "uuid": "26867ddf-5a9b-4af0-b552-e4020a913b95",
+            "local": true
+          }
+        }
+```
+- **headerless**: Only for CSV export. The CSV created when this setting is set to true will not contain the header row.
+- **includeWarninglistHits**: Adds a warnings block to an attribute if it has warninglist hits. See example below.
+```
+"warnings": [
+          {
+            "match": "10.0.0.0/8",
+            "value": "10.0.0.1",
+            "warninglist_name": "List of RFC 5735 CIDR blocks",
+            "warninglist_id": "46"
+          },
+          {
+            "match": "10.0.0.0/8",
+            "value": "10.0.0.1",
+            "warninglist_name": "List of RFC 1918 CIDR blocks",
+            "warninglist_id": "44"
+          }
+        ]
+```
+- **object_relation**: Search on the object_relation field of attributes. You can search for 'malware-sample' attributes of file objects for example. Searching for multiple values at the same time is possible as well.
+```
+{
+    "returnFormat": "json",
+    "object_relation": ["malware-sample", "institution-name"]
+}
+```
+- **includeSightings**: Adds a list of sightings for attributes that have sightings. See example below.
+```
+"Sighting": [
+          {
+            "id": "1",
+            "attribute_id": "29",
+            "event_id": "31",
+            "org_id": "1",
+            "date_sighting": "1617017091",
+            "uuid": "48d21518-6b2a-4615-8c4e-91fbe4f08fe7",
+            "source": "",
+            "type": "0",
+            "attribute_uuid": "b3c25257-7f47-41af-a29b-89188e583b5c",
+            "Organisation": {
+              "id": "1",
+              "uuid": "26867ddf-5a9b-4af0-b552-e4020a913b95",
+              "name": "SHARINGORG"
+            }
+          }
+        ]
+```
+- **includeCorrelations**: Adds a list of correlated attributes for attributes that have correlations. See example below.
+```
+"RelatedAttribute": [
+          {
+            "id": "31",
+            "event_id": "30",
+            "object_id": "0",
+            "object_relation": null,
+            "category": "Network activity",
+            "type": "ip-dst",
+            "uuid": "f3b54c94-89ff-4fcf-9f47-52f70c6540b8",
+            "timestamp": "1616961683",
+            "distribution": "5",
+            "sharing_group_id": "0",
+            "to_ids": false,
+            "comment": "",
+            "value": "10.0.0.1",
+            "Event": {
+              "id": "30",
+              "uuid": "8cca9f2f-9281-49fd-9b30-e16a8dbf6855",
+              "threat_level_id": "1",
+              "analysis": "0",
+              "info": "Correlation 1",
+              "extends_uuid": "",
+              "distribution": "1",
+              "sharing_group_id": "0",
+              "published": false,
+              "date": "2021-03-11",
+              "orgc_id": "1",
+              "org_id": "1"
+            }
+          }
+        ]
+```
+- **includeDecayScore**: If set to 1, decay score information will be included for attributes that are affected by decaying. See example below. Note that includeEventTags will be set to 1 automatically if includeDecayScore is true.
+```
+"decay_score": [
+          {
+            "score": 77.40239901751683,
+            "base_score": 80,
+            "decayed": false,
+            "DecayingModel": {
+              "id": "2",
+              "name": "NIDS Simple Decaying Model"
+            }
+          }
+        ],
+```
+- **decayingModel**: Allows you to set the decaying model(s) to use to calculate the decay score. You can use a model that is not enabled. The value should be set to the id of the model. If this value is not set, a decay score entry will be added for all enabled decaying models that apply to the attribute type.
+- **excludeDecayed**: Filter out all expired IOCs. Note that includeDecayScore will be set to 1 automatically if excludeDecayed is true.
+- **modelOverrides**: JSON that can be used to modify Model parameters on-the-fly. Example can be found beow.
+```
+{
+    "type": "ip-src",
+    "tags": ["tlp:%","phishing:%"],
+    "includeDecayScore": 1,
+    "excludeDecayed": 1,
+    "modelOverrides": {
+        "threshold": 30
+    }
+    "decayingModel": [84, 12],
+}
+```
+- **includeFullModel**: If set to 1, includes the full decaying model details instead of just the id and name.
+- **score**: Overrides the model threshold value with the one you set. This means attributes for which the decay score calculated for all relevant models is lower than this value, will be considered decayed.
+- **attribute_timestamp**: Restrict the results by the timestamp (last edit). Any attribute with a timestamp newer than the given timestamp will be returned. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **first_seen**: Restrict the results by the first_seen timestamp of the attribute. Any attribute with a first_seen timestamp newer than the given timestamp will be returned. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **last_seen**: Restrict the results by the last_seen timestamp of the attribute. Any attribute with a first_seen timestamp newer than the given timestamp will be returned. The input can be a timestamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]).
+- **searchall**: Search for a full or a substring (delimited by % for substrings) in the attribute tags, attribute values or attribute comment fields.
+
+#### Example
+~~~~
+curl \
+ -d '{"returnFormat":"json","value":"foobar"}' \
+ -H "Authorization: YOUR API KEY" \
+ -H "Accept: application/json" \
+ -H "Content-type: application/json" \
+ -X POST https://192.168.0.220/attributes/restSearch
+~~~~
+
+~~~~json
+{"response": {"Attribute": [{"id":"44","event_id":"30","object_id":"0","object_relation":null,"category":"Other","type":"comment","to_ids":false,"uuid":"7a5d856c-048a-4dbd-8e6d-41d1790c5ad0","timestamp":"1617056037","distribution":"5","sharing_group_id":"0","comment":"","deleted":false,"disable_correlation":false,"first_seen":null,"last_seen":null,"value":"foobar","Event":{"org_id":"1","distribution":"1","id":"30","info":"Correlation 1","orgc_id":"1","uuid":"8cca9f2f-9281-49fd-9b30-e16a8dbf6855"}}]}}
+~~~~
+
 ## Objects management
 ### POST /objects/delete/[object_id]/[hard_delete]
 #### Description
@@ -977,7 +1230,7 @@ Only the fields POSTed will be updated, the rest is left intact. To view all pos
 
 ### POST admin/users/delete/
 
-You can also delete users by POSTing to the below URL, but keep in mind that disabling users (by setting the disabled flag via an edit) is always prefered to keep user associations to events intact.
+You can also delete users by POSTing to the below URL, but keep in mind that disabling users (by setting the disabled flag via an edit) is always preferred to keep user associations to events intact.
 
 #### Parameters
 
@@ -1564,35 +1817,6 @@ For example, to retrieve all attributes for event #5, including non IDS marked a
 https://<misp url>/attributes/text/download/all/null/5/true
 ~~~~
 
-## RESTful searches with JSON result
-
-It is possible to search the database for attributes based on a list of criteria
-
-To return an event with all of its attributes, relations, shadowAttributes, use the following syntax:
-
-~~~~
-https://<misp url>/attributes/restSearch/json/[value]/[type]/[category]/[org]/[tag]/[quickfilter]/[from]/[to]/[last]/[eventid]/[withAttachments]/[metadata]/[uuid]
-~~~~
-    
- If you include "includeEventUuid":1" in the json request, it will give you the event_uuid as a result as well.
-
-Be careful if you GET the /attributes/restSearch/json/ without an value, it will return all attributes.
-
-### POST /attributes/restSearch
-
-Do not use that function with GET!
-
-#### Example
-~~~~
-curl -X POST -k -H 'Accept: application/json' -H 'Authorization: API Key' -H 'Content-Type: application/json' -i 'https://URL/attributes/restSearch' --data '{"value":"foobar"}'
-~~~~
-
-~~~~json
-{
-    "response": []
-}
-~~~~
-
 ## RESTful searches with XML result export
 
 It is possible to search the database for attributes based on a list of criteria.

categories-and-types/README.md

@@ -119,6 +119,7 @@
 |passport-country| | | | | | |
 |passport-expiration| | | | | | |
 |passport-number| | | | | | |
+|pattern-filename| | | | | | |
 |pattern-in-file| | X | | X | | |
 |pattern-in-memory| | X | | X | | |
 |pattern-in-traffic| | | | X | | |
@@ -309,6 +310,7 @@
 |passport-country| | | | | | |
 |passport-expiration| | | | | | |
 |passport-number| | | | | | |
+|pattern-filename| | | | | | |
 |pattern-in-file| X | | X | X | | |
 |pattern-in-memory| | | | X | | |
 |pattern-in-traffic| X | | X | X | | |
@@ -499,6 +501,7 @@
 |passport-country| X | | | |
 |passport-expiration| X | | | |
 |passport-number| X | | | |
+|pattern-filename| | | | |
 |pattern-in-file| | | | |
 |pattern-in-memory| | | | |
 |pattern-in-traffic| | | | |
@@ -709,6 +712,7 @@
 *   **passport-country**: The country in which the passport was issued
 *   **passport-expiration**: The expiration date of a passport
 *   **passport-number**: The passport number of a natural person
+*   **pattern-filename**: A pattern in the name of a file
 *   **pattern-in-file**: Pattern in file that identifies the malware
 *   **pattern-in-memory**: Pattern in memory dump that identifies the malware
 *   **pattern-in-traffic**: Pattern in network traffic that identifies the malware

connectors/README.md

@@ -159,7 +159,7 @@ Configure a sync user.
 ### Verify Cert
 This gives you the option to choose if python should validate the certificate of the misp instance. (This allows ease within testing environments) 
 
-`misp_verifycert = False` IT IS RECOMENDED TO USE A VALID SSL CERT IN PRODUCTION AND CHANGE THIS TO TRUE
+`misp_verifycert = False` IT IS RECOMMENDED TO USE A VALID SSL CERT IN PRODUCTION AND CHANGE THIS TO TRUE
 
 ## Instructions on Reading TiIndicators That Have Been Pushed
 In the command line, run `python3 script.py -r`

create-event-report/README.md

@@ -18,7 +18,7 @@ Then we get the add event form.
 
 Let's fill it with the data we already have:
 * Date: Here we will put the date of the report, so 2016-11-14
-* Distribution: Depending on the event, we might want it to be more or less spread accross the MISP instances. For this one, since it is a public report, there is no reason to limit the diffusion so "All communities".
+* Distribution: Depending on the event, we might want it to be more or less spread across the MISP instances. For this one, since it is a public report, there is no reason to limit the diffusion so "All communities".
 * Threat Level: Self explainatory. Since the ransomware in the report is not using a huge exploit, we can use low, or undefined as we don't really know. we'll go for the latter since it can be edited. 
 * Analysis: Give the current stage of the analysis. Since the report is published, we can assume that the analysis is completed.
 * Event Info: The event's info is in fact the name or title of the event, so it seems legit to put the title of the report here as well. Since it is public information, we also prefix it with "OSINT".
@@ -113,7 +113,7 @@ We only have the network indicators left, and as said before, we will let MISP d
 
 ![type recognition fail](figures/surprise.png)
 
-Oh well, that was unexpected. In fact, it is not that surprising regarding the format of the tor address that look more like a filename than like a url but it is still a problem, since we can't change the type nor the category to a more consistant one. This is indeed one of the limitation of freetext import. To solve this issue, we will use a simple trick: we will add a slash at the end of the tor address so it won't be confused for a filename. 
+Oh well, that was unexpected. In fact, it is not that surprising regarding the format of the tor address that look more like a filename than like a url but it is still a problem, since we can't change the type nor the category to a more consistent one. This is indeed one of the limitation of freetext import. To solve this issue, we will use a simple trick: we will add a slash at the end of the tor address so it won't be confused for a filename. 
 
 ![freetext import network](figures/free_network2.png)
 

faq/README.md

@@ -34,7 +34,7 @@ for different monitoring tools:
 - Using [Cacti](https://www.cacti.net/), a blog post with the [instruction](https://www.misp-project.org/2020/08/22/MISP-Monitoring-with-Cacti.html) is available.
 - Using [Munin](http://munin-monitoring.org/), [misp-monitor](https://github.com/SteveClement/misp-monitor) for instructions.
 - Using [Nagios](https://www.nagios.org/), [Monitoring MISP with Nagios](https://blog.rootshell.be/2020/08/25/monitoring-misp-with-nagios/)
-- Using [OpenNMS](https://www.opennms.com/), a blog post with the [instructions](https://www.misp-project.org/2020/08/18/MISP-Monitoring-with-OpenNMS.html) is availabe.
+- Using [OpenNMS](https://www.opennms.com/), a blog post with the [instructions](https://www.misp-project.org/2020/08/18/MISP-Monitoring-with-OpenNMS.html) is available.
 - [Live monitoring of MISP usage](https://github.com/MISP/misp-monitoring) via the httpd logs. 
 
 ***
@@ -129,7 +129,7 @@ Source: [Getting started with MISP](http://www.vanimpe.eu/2015/05/31/getting-sta
 MISP can be made more appealing to the eye by adding some graphics.
 
 As Org.- or Site-admin navigate to *Administration* -> *List organisations* and edit the corresponding organization.
-Withing this editor you will be able to update the logo.
+Within this editor you will be able to update the logo.
 
 Other ways to achieve this, would be:
     
@@ -269,7 +269,7 @@ You can customize this for your own needs. There are also pre-made boostrap them
 
 Before making any changes, confirm the version of boostrap currenlty used by running `head -5 /var/www/MISP/app/webroot/css/bootstrap.css`. You can find themes on sites like [Bootswatch](https://bootswatch.com/2/). 
 
-To replace the current theme with a theme you found on bootsplash, run: `wget https://bootswatch.com/2/readable/bootstrap.css   -O /var/www/MISP/app/webroot/css/bootstrap.css` , replacing the URL as needed. 
+To replace the current theme with a theme you found on bootsplash, run: `sudo -u www-data wget https://bootswatch.com/2/readable/bootstrap.css -O /var/www/MISP/app/webroot/css/bootstrap.css` , replacing the URL as needed.
 
 Some bootswatch themes applied on MISP: 
 * https://i.imgur.com/usONTLk.png
@@ -627,7 +627,7 @@ OR if you were foolish enough to not install in a Python virtualenv:
 sudo -u www-data misp-modules -l 127.0.0.1 -s &
 ```
 
-> [warning] Running misp-modules like this will certainly kill it once you quit the session. Make sure it is in your **/etc/rc.local** or some ther init script that gets run on boot.
+> [warning] Running misp-modules like this will certainly kill it once you quit the session. Make sure it is in your **/etc/rc.local** or some other init script that gets run on boot.
 
 ## Uninstalling MISP
 
@@ -1025,7 +1025,7 @@ sudo sudo systemctl restart apache2
 
 ### What are the required steps after a MISP installation to have a properly running instance?
 
-- First login with the installation credentials and change the password immediatly (especially if your instance is publicly accessible)
+- First login with the installation credentials and change the password immediately (especially if your instance is publicly accessible)
 - Set the base_url to the hostname of your machine (apache virtualhost name)
 - Create a new organisation which will be the host organisation running the MISP instance
 - Set the new organisation in `MISP.host_org_id` to replace the default one

galaxy/README.md

@@ -74,7 +74,7 @@ The __/galaxies__ file contains metatdatas and galaxy structure.
 The __/clusters__ file contains actual data.
 
 
-#### The galaxy managment GUI
+#### The galaxy management GUI
 
 ![GalaxyManagment](./figures/GalaxyManagmentGui.png)
 

user-stories/README.md

@@ -4,7 +4,7 @@
 |-|-|
 | As a lead threat intelligence analyst, I want to lead a team focused on hunting down threats so that I can prevent attacks against ICT infrastructures and organizations | <ul> <li>Monitor what teams are up to in real-time using the Live Dashboard </li></ul>|
 | As a threat analyst, I want to research, analyze and reverse engineer malware so that I can know how to counter it | <ul> <li> Attach and download files and malware samples from events</li> <li>Search for hashes/IPs/domains/URLs from malware events, or add malware samples hashes to an event</li> <li>Analyse observables and malware collected during an incident (e.g. domain name, IP addresses etc.) by checking whether observables are IoCs or false positives using ‘correlation graph’ and ‘expansion modules’.</li> <li> Enrich malware events by querying data sources external to MISP using modules</li>  <li>Perform dynamic malware analysis correlations</li> <li> Submit events with malware samples to analysis tools (e.g VirusTotal, VMRay) for further analysis, and then extend MISP with malware analysis results</li> </ul> |
-| As a lead threat intelligence analyst, I want to convert threat data into actionable threat intelligence so that I can improve security posture. | <ul> <li>Import data from external sources</li> <li>Add feeds</li> <li>Contextualise events and attributes using tags, taxonomies and galaxies</li></li> |
+| As a lead threat intelligence analyst, I want to convert threat data into actionable threat intelligence so that I can improve security posture. | <ul> <li>Import data from external sources</li> <li>Add feeds</li> <li>Contextualise events and attributes using tags, taxonomies and galaxies</li></ul> |
 | As a threat analyst, I want to exchange threat information with third parties so that we can gain shared situational awareness | <ul> <li>Setup different models of distribution on MISP instance</li> <li>Sync events and attributes between instances</li> <li>Use filtering functionalities to meet an organisation's sharing policy</li> <li>Share information, pentest information, malware samples, vulnerabilities internally and externally</li> <li>Use feature/achievements widget adding gamification to the information sharing</li> </ul> |
 | As a threat analyst, I want to monitor threats and access live data so that I can manage threats before they cause major damage | <ul>  <li>Import lists of indicators and check if the IOCs are present in feeds.</li> <li>Monitor statistics and sightings using widgets</li> <li>Show live data and stats from one or more MISP instances via the Dashboard</li> <li>Process information in real-time when it's updated, created, or published by instances by integrating with ZMQ</li> <li>Use sightings to notify an instance about activities related to an indicator</li> </ul> |
 | As a threat analyst, I want to aggregate and compare indicators from various sources so that I can connect the dots between various threats | <ul><li>Join communities and subscribe to the feeds</li> <li>Add events and assign events to specific feeds</li> <li>Correlate indicators using MISP's automated correlation engine</li> <li> Use the overlap feed analysis available in MISP</li> <li>Link events and attributes using the correlation graph</li> <li>Analyse and gain more information on attributes using modules</li> <li>Link events with malware, threat actors etc using galaxies (e.g ATT&CK)</li></ul> |

using-the-system/README.md

@@ -68,7 +68,7 @@ Sharing groups in MISP are a more granular way to create re-usable distribution
 
 The most general use-cases for sharing groups are creating re-usable topical subgroups in MISP that share events or for ad-hoc sharing scenarios (such as several organisations involved in a specific incident wanting to work together). Generally sharing groups add a level of complexity for the users involved as well as a performance overhead on the data marked with it.
 
-As a best-practice recommendation, using traditional distribution methods is prefered unless they cannot cover the given use-case. Also, whilst sharing groups can be assigned to both events and attributes, it is highly recommended to use the special "inherit" distribution setting on attributes whenever the attribute's sharing group would match the event's.
+As a best-practice recommendation, using traditional distribution methods is preferred unless they cannot cover the given use-case. Also, whilst sharing groups can be assigned to both events and attributes, it is highly recommended to use the special "inherit" distribution setting on attributes whenever the attribute's sharing group would match the event's.
 
 Sharing groups consist of the following elements, each of which has its own page in the sharing group creator/editor tool (accessed via the Global actions -> List Sharing Groups and Add Sharing Group functionalities):
 
@@ -105,7 +105,7 @@ For users trying to populate an event, after clicking on the populate from templ
 ![Choose the most appropriate template for your event.](figures/template_choice.png)
 
 Once you have chosen a template, you'll be presented with the actual form contained within. Make sure you fill out as many fields as possible with the mandatory fields - marked by a star in a bracket such as this: (*) - are filled out.
-Templates are devided into sections, with each section having a title and a description in addition to a series of fields. Each field can be an attribute or a file attachment field. An attribute field has the following components:
+Templates are divided into sections, with each section having a title and a description in addition to a series of fields. Each field can be an attribute or a file attachment field. An attribute field has the following components:
 
 ![MISP will generate attributes based on the field's settings and the data that you provide.](figures/template_field.png)
 
@@ -467,7 +467,7 @@ If you ever need to change the data about the linked servers or remove any conne
 
 ![Apart from editing / deleting the link to the remote server, you can issue a push all or pull all command from here.](figures/list_servers.png)
 
-*   **Editing the connection to the:** By clicking edit a view, [that is identical to the new instance view](#setting-up-a-connection-to-another-server), is loaded, with all the current information of the instance pre-entered.
+*   **Editing the connection to the instance:** By clicking edit a view, [that is identical to the new instance view](#setting-up-a-connection-to-another-server), is loaded, with all the current information of the instance pre-entered.
 *   **Deleting the connection to the instance:** Clicking the delete button will delete the link to the instance.
 *   **Push all:** By clicking this button, all events that are eligible to be pushed on the instance you are on will start to be pushed to the remote instance. Events and attributes that exist on the far end will be updated.
 *   **Pull all:** By clicking this button, all events that are set to be pull-able or full access on the remote server will be copied to this instance. Existing events will not be updated.
@@ -481,12 +481,12 @@ The platform is also [RESTfull](http://en.wikipedia.org/wiki/Representational_st
 Use any HTTP compliant library to perform requests.
 You can choose which format you would like to use as input/output for the REST calls by specifying the Accept and Content-Type headers.
 
-The following headers are required if you wish to recieve / push XML data:
+The following headers are required if you wish to receive / push XML data:
 **Authorization**: _your authorisation key_
 **Accept**: _application/xml_
 **Content-Type**: _application/xml_
 
-The following headers are required if you wish to recieve / push JSON data:
+The following headers are required if you wish to receive / push JSON data:
 **Authorization**: _your authorisation key_
 **Accept**: _application/json_
 **Content-Type**: _application/json_
@@ -658,7 +658,7 @@ Content-Type: application/xml
 </response>
 ```
 
-The respone from requesting an invalid page
+The response from requesting an invalid page
 
 ```xml
 <?xml version = "1.0" encoding = "UTF-8"?>

warninglists/README.md

@@ -1,24 +1,49 @@
+<!-- toc -->
 # MISP warninglists
 MISP warninglists are lists of well-known indicators that can be associated to potential false positives, errors or mistakes.
 There is a Python module available to work with warninglists in a Pythonic way called [PyMISPWarningLists](https://github.com/MISP/PyMISPWarningLists).
 [MISP warninglists GitHub Repo](https://github.com/MISP/misp-warninglists)
 
-## MISP warning lists: The dilemma of false-positive
+## MISP warning lists introduction: The dilemma of false-positive
 
-- False-positive is a common issue in threat intelligence sharing.
+- False-positives are a common issue in threat intelligence sharing.
 
 - It’s often a contextual issue:
-   - false-positive might be different per community of users sharing
+   - False-positives might be different per community of users sharing
 information.
+   - Organizations might have their own view on false-positives.
 
-   - organization might have their own view on false-positive.
+## Usage
+By default MISP will only trigger hits for warninglists if the attribute IDS flag is set. This behaviour can be changed by setting the MISP config parameter MISP.warning_for_all to true.
 
-- Based on the success of the MISP taxonomy model, we build misp-warninglists. They are lists of well-known indicators that can be
+When an attribute matches a warninglist entry, an info/warning box is displayed at the event and attribute level, as can be seen in the screenshot below.
-associated to potential false positives, errors or mistakes. They are Simple JSON files.
+![Screenshot of event with one attribute that has warning list hits](./figures/warninglist-warnings.png)
 
-![MISP warning lists](./figures/MISPwarninglist.png)
+Individual warninglists can be enabled or disabled at instance level using the warninglists index page. Examples of default warning lists are known public DNS resolvers, multicast IP addresses, hashes for empty values, rfc1918, TLDs or known google domains.
+![Screenshot of the warninglists index page](./figures/MISPwarninglist.png)
 
-The warning lists are integrated in MISP to display an info/warning box at the event and attribute level. This can be enabled at MISP instance level. Default warning lists can be enabled or disabled like known public
+The warning lists can be expanded or added in JSON locally or via
-resolver, multicast IP addresses, hashes for empty values, rfc1918, TLDs or known google domains. The warning lists can be expanded or added in JSON locally or via
+pull requests (https://github.com/MISP/misp-warninglists). Warning lists can also be used for critical or core infrastructure or personally identifiable information.
-pull requests (https://github.com/MISP/misp-warninglists). Warning lists can be also used for critical or core infrastructure
+
-warning, personally identifiable information...
+### Warninglists and data export
+The enforceWarninglist parameter of MISP restSearch can be used to exclude attributes that have a warninglist hit from the export. For more information on the MISP API, please refer to the [Automation and MISP API chapter](../sharing/).
+
+### Check individual values for warning list hits
+It is also possible to do a lookup for a specific value in the warninglists. This functionality is accessible by using the top menu "Input Filters" > "List Warninglists" and then using the link in the left side menu bar (or by browsing directly to [misp_base_url]/warninglists/checkValue). Only enabled warninglists will be searched.
+
+![Screenshot of example search in warninglists using check value, for value 192.168.0.1. The result shows two hits, one for RFC 5735 CIDR blocka nd one for RFC 1918 CIDR blocks](./figures/warninglist-checkValue.png)
+### Updating warninglists
+An update of the warninglists can be triggered via the GUI using the "Update Warninglists" button in the side menu bar when viewing any of the relevant warninglists pages, for example the index page.
+
+Alternatively, it is also possible to trigger an update using a CLI command.
+~~~
+MISP/app/Console/cake Admin updateWarningLists
+~~~
+
+If you are updating an existing warninglist, make sure you incremented the version number before triggering the update on the MISP. You can also contribute to the existing warninglists by forking [MISP warninglists GitHub Repo](https://github.com/MISP/misp-warninglists), making changes and then creating a pull request.
+### Creating a custom warninglist
+1. Create a new directory for your warninglist in /var/www/MISP/app/files/warninglists.
+2. Add a file called list.json to the newly created directory and add the content you want. You can use any of the existing warning lists in https://github.com/MISP/misp-warninglists as reference.
+3. Trigger an update of the warning lists on the instance to load in your new warning list.
+
+Example use cases are a list of domain names owned by you or your organisation or employee email addresses.