Merge remote-tracking branch 'origin/master'

.travis.yml

@@ -8,6 +8,7 @@ sudo: required
 
 install:
     - sudo apt-get -y install libgif-dev calibre npm
+    - sudo npm config set strict-ssl false
     - sudo npm cache clean -f
     - sudo npm install -g n
     - sudo n stable
@@ -18,6 +19,7 @@ install:
     - npm install gitbook-plugin-github
     - npm install gitbook-plugin-toc
     - npm install gitbook-plugin-anchors
+    - npm install gitbook-plugin-image-class
 
 script:
     - gitbook install

GLOSSARY.md

@@ -1,6 +1,6 @@
 ## MISP Glossary
 This glossary is meant as a quick lookup document in case of any need of clarification of any threat sharing, threat-intel lingo.
-Be careful when adding terms to the glossary. Adding a generic term like: *MISP* will prevent terms like *MISP noticelist* to be addded. As a matter of definition please use the singular of for any terms.
+Be careful when adding terms to the glossary. Adding a generic term like: *MISP* will prevent terms like *MISP noticelist* to be addded. As a matter of definition please use the singular for any terms.
 In case you use any CCBYSA licensed content, or other pieces that are subject to licensing, make sure to add it as a by-line at the end of the mention.
 
 ## API
@@ -36,7 +36,8 @@ Attributes in MISP can be network indicators (e.g. IP address), system indicator
 ◦ A type (e.g. MD5, url) is how an attribute is described.
 ◦ An attribute is always in a category (e.g. Payload delivery) which puts it in a context.
 • A category is what describes an attribute.
-◦ An IDS flag on an attribute allows to determine if an attribute can
+◦ An IDS flag on an attribute allows to determine if an attribute can be automated (such as being exported as an IDS ruleset or used for detection). If the IDS flag is not present, the attribute
+can be useful for contextualisation only.
 
 ## MISP Event
 MISP events are encapsulations for contextually linked information
@@ -78,6 +79,9 @@ MISP objects are used in MISP (starting from version 2.4.80) system and can be u
 or GnuPG instance key is the GnuPG (Gnu Privacy Guard) key used by the MISP instance and which is only used to sign notification.
 The GnuPG key used in the MISP instance must **not** be used anywhere else and should not be valuable.
 
+## MISP Sightings
+Basically, sighting is a system allowing people to react on attributes on an event. It was originally designed to provide an easy method for user to tell when they see a given attribute, giving it more credibility.
+
 ## MISP Taxonomies
 [Taxonomy](https://en.wikipedia.org/wiki/Taxonomy_(general)) is the practice and science of classification. The word is also used as a count noun: a taxonomy, or taxonomic scheme, is a particular classification. The word finds its roots in the Greek language τάξις, taxis (meaning 'order', 'arrangement') and νόμος, nomos ('law' or 'science').
 Taxonomies that can be used in MISP (2.4) and other information sharing tool and expressed in Machine Tags (Triple Tags). A machine tag is composed of a namespace (MUST), a predicate (MUST) and an (OPTIONAL) value. Machine tags are often called triple tag due to their format.

README.md

@@ -29,6 +29,10 @@ We welcome contributions to the MISP book. If you want to contribute, fork the [
 
 <div class="pagebreak"></div>
 
+## Format
+
+MISP book is available in [HTML](https://www.circl.lu/doc/misp/), [PDF](https://www.circl.lu/doc/misp/book.pdf), [ePub](https://www.circl.lu/doc/misp/book.epub) and [Kindle mobi format](https://www.circl.lu/doc/misp/book.mobi).
+
 ## License
 
 The MISP user guide is dual-licensed under [GNU Affero General Public License version 3](http://www.gnu.org/licenses/agpl-3.0.html) and [CC-BY-SA 4.0 international](https://creativecommons.org/licenses/by-sa/4.0/).
@@ -40,6 +44,7 @@ The MISP user guide is dual-licensed under [GNU Affero General Public License ve
 * Copyright \(C\) 2015-2018 Alexandre Dulaunoy
 * Copyright \(C\) 2014-2018 CIRCL - Computer Incident Response Center Luxembourg
 * Copyright \(C\) 2018 Camille Schneider
+* Copyright \(C\) 2018 Steve Clement
 
 
 

SUMMARY.md

@@ -2,6 +2,7 @@
 
 * [Book Convention](book-convention/README.md)
 * [Quick Start](quick-start/README.md)
+* [Requirements](requirements/README.md)
 * [Get Your Instance](get-your-instance/README.md)
 * [General Layout](general-layout/README.md)
 * [General Concepts](general-concepts/README.md)
@@ -24,4 +25,6 @@
 * [Synchronisation/Sharing](sharing/README.md)
 * [ZeroMQ - MISP publish-subscribe](misp-zmq/README.md)
 * [Translations - i18n & l10n](translation/README.md)
+* [FAQ](faq/README.md)
+* [Dev FAQ](dev-faq/README.md)
 * [Appendices](appendices/README.md)

USAGE.md

@@ -1,22 +1,137 @@
-
 Install notes
 =============
 
-:warning: Make sure to be in the *misp-book* repository directory for the *npm magic*.
-Also: The *npm* plugin *autocover* is broken. It pulls an incompatible *canvas* module version. Thus patched repository used (forked from original)
+:warning: Make sure to be in the *misp-book* repository directory for the *npm magic*.<br />
+_Also_: The *npm* plugin *autocover* is broken. It pulls an incompatible *canvas* module version. Thus patched repository used (forked from original)<br />
+_Finally_: You will get a few errors on Ubuntu 18.04 which you can ignore for now. In the rather near future we need to think about an alternative as gitbook glides towards obsoletion and security risk.
 
-Tested on: *Ubuntu 16.04.4 LTS*
+Reason for concern:
+```
+npm WARN deprecated sprintf@0.1.5: The sprintf package is deprecated in favor of sprintf-js.
+npm WARN deprecated tough-cookie@2.2.2: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
+npm WARN deprecated cryptiles@2.0.5: This version is no longer maintained. Please upgrade to the latest version.
+npm WARN deprecated boom@2.10.1: This version is no longer maintained. Please upgrade to the latest version.
+npm WARN deprecated hoek@2.16.3: This version is no longer maintained. Please upgrade to the latest version.
+npm WARN saveError ENOENT: no such file or directory, open '/home/steve/Desktop/code/MISP_Main/misp-book/package.json'
+npm WARN enoent ENOENT: no such file or directory, open '/home/steve/Desktop/code/MISP_Main/misp-book/package.json'
+npm WARN misp-book No description
+npm WARN misp-book No repository field.
+npm WARN misp-book No README data
+npm WARN misp-book No license field.
+npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@0.3.8 (node_modules/fsevents):
+npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@0.3.8: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
+
++ gitbook-plugin-alerts@0.2.0
++ gitbook-plugin-advanced-emoji@0.2.2
++ gitbook-plugin-gist@1.0.0
++ gitbook-plugin-sitemap@1.2.0
++ gitbook-plugin-github@3.0.0
++ gitbook-plugin-toc@0.0.2
++ gitbook-plugin-anchors@0.7.1
++ gitbook-plugin-search@2.2.1
++ gitbook-plugin-codesnippet@1.2.0
++ gitbook-plugin-last-modified@1.0.0
++ gitbook-plugin-image-class@1.0.5
++ gitbook@2.6.9
++ gitbook-plugin-autocover@2.0.1
+updated 13 packages, moved 9 packages and audited 4906 packages in 5.316s
+found 368 vulnerabilities (48 low, 250 moderate, 62 high, 8 critical)
+  run `npm audit fix` to fix them, or `npm audit` for details
+#weHaveBeenWarned
+```
+
+
+Tested on: *Ubuntu 18.04 LTS* *Debian 9.5/sid/testing*
+[Terminal Recording of npm install lines on Ubuntu 18.04](https://asciinema.org/a/84JZMuGu2QlFH59q6mK8jbdQS)
 
 ```bash
-curl -sL https://deb.nodesource.com/setup_9.x | sudo -E bash -
+git clone git@github.com:MISP/misp-book.git
+cd misp-book
+curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
 sudo apt-get install -y nodejs
 sudo apt-get install -y build-essential
-sudo apt install -y npm pkg-config libcairo2-dev
-npm install gitbook git+https://github.com/SteveClement/plugin-autocover.git gitbook-plugin-github gitbook-plugin-toc gitbook-plugin-anchors gitbook-plugin-image-class
+sudo apt install -y pkg-config libcairo2-dev libgif-dev libjpeg-dev
+npm install gitbook git+https://github.com/SteveClement/plugin-autocover.git gitbook-plugin-github gitbook-plugin-toc gitbook-plugin-anchors gitbook-plugin-alerts gitbook-plugin-search gitbook-plugin-gist gitbook-plugin-advanced-emoji gitbook-plugin-sitemap gitbook-plugin-codesnippet gitbook-plugin-image-class gitbook-plugin-last-modified gitbook-plugin-fontsettings
+
 sudo npm install gitbook-cli -g
 gitbook install
 ```
 
+Plugins
+=======
+
+The following plugins are installed and this is how they impact the build.
+
+
++ gitbook-plugin-codesnippet@1.2.0
+
+  You can include any code snippet from a file like follows.
+  ```
+  {% codesnippet "/pymisp/last.py", language="python" %}{% endcodesnippet %}
+  ```
+
+
++ gitbook-plugin-advanced-emoji@0.2.2
+
+  This plainly enables converting ```:smile:``` to :smile:
+  Consult the [Emoji Cheat Sheeet](https://www.webpagefx.com/tools/emoji-cheat-sheet/) for a reference.
+
+
++ gitbook-plugin-alerts@0.2.0
+
+  Info, warning, danger, success blockquotes for your gitbook.
+
+  Warning styling
+  ```
+  > **[warning] For warning**
+  >
+  > Use this for warning messages.
+  ```
+
+  Danger styling
+  ```
+  > **[danger] For danger**
+  >
+  > Use this for danger messages.
+  ```
+
+  Success styling
+  ```
+  > **[success] For info**
+  >
+  > Use this for success messages.
+  ```
+
+
++ gitbook-plugin-gist@1.0.0
+
+  With this plugin you can include GitHub Gists](https://gist.github.com)
+  ```
+  {% gist id="https://gist.github.com/SteveClement/1ba901612a97d63938ed5cf32c2100ee" %}{% endgist %}
+  
+  {% gist id="SteveClement/1ba901612a97d63938ed" %}{% endgist %}
+  
+  {% gist id="SteveClement/1ba901612a97d63938ed",file="README.md" %}{% endgist %}
+  
+  {% gist id="SteveClement/1ba901612a97d63938ed",hideFooter=true %}{% endgist %}
+  ```
+
+
++ gitbook-plugin-sitemap@1.2.0
+
+  With this we include a **sitemape.xml** in the webroot directory.
+
+
++ gitbook-plugin-last-modified@1.0.0
+
+  You will see a **Last modified: Thu Nov 01 2018 16:29:37 GMT+0100 (CET)** string on top of all the generated pages.
+
+
++ gitbook-plugin-image-class@1.0.5
+
+  The [image-class](https://www.npmjs.com/package/gitbook-plugin-image-class) plugin allows you more flexibility in including images.
+  See the above link for examples.
+
 Usage
 =====
 
@@ -101,7 +216,7 @@ xcode-select --install
 brew install pkg-config cairo pango libpng jpeg giflib
 ```
 
-:warning: Make sure to be in the *misp-book* repository directory for the npm magic.
+:warning: Make sure to be in the *misp-book* repository directory for the npm magic to work correctly.
 
 Installing gitbook and all dependencies wants to look like this (Tested on *macOS 11.13.3*):
 
@@ -111,6 +226,19 @@ npm install gitbook-cli -g
 gitbook install
 ```
 
+gitbook rebuild on change
+=========================
+
+By default gitbook has the '--watch' option enabled.
+If this is broken, use the included 'serve.sh' and the 'inoticoming' package.
+Start 'gitbook serve &' and run the following command line:
+
+```
+inoticoming --foreground . --suffix .md bash serve.sh \;
+```
+
+If any md changes, the gitbook process is killed and restarted.
+
 npm salvage
 ===========
 

administration/README.md

@@ -7,19 +7,19 @@
 * [Roles](#roles)
 * [Tools](#tools)
 * [Server Settings](#server-settings)
-* Jobs
-* Scheduled Tasks
+* [Jobs](#jobs)
+* [Scheduled Tasks](#scheduled-tasks)
 
 
 > [warning] This page is under modification for updating the content. Current status:
 
-- [x] Users
-- [x] Organisations
-- [x] Roles
-- [x] Tools
-- [ ] Server Settings
-- [ ] Jobs
-- [ ] Scheduled Tasks
+- [x] Users - Reviewed/Updated on: ?
+- [x] Organisations - Reviewed/Updated on: ?
+- [x] Roles - Reviewed/Updated on: ?
+- [x] Tools - Reviewed/Updated on: ?
+- [ ] Server Settings - Reviewed/Updated on: ?
+- [ ] Jobs aka. Background processing - Reviewed/Updated on: ?
+- [ ] Scheduled Tasks aka. Background processing - Reviewed/Updated on: ?
 
 - - -
 
@@ -45,7 +45,7 @@ To add a new user, click on the Add User button in the administration menu to th
 *   **Fetch GnuPG key:** Fetch GnuPG public key.
 *   **Receive alerts when events are published:** This option will subscribe the new user to automatically generated e-mails whenever an event is published.
 *   **Receive alerts from "contact reporter" requests:** This option will subscribe the new user to e-mails that are generated when another user tries to get in touch with an event's reporting organisation that matches that of the new user.
-*   **Disable this user account:** Tick it if you want to disable this user account.
+*   **Disable this user account:** Tick it if you want to disable this user account. (preferred to removing an account)
 
 #### Listing all users:
 
@@ -72,7 +72,7 @@ To list all current users of the system, just click on List Users under the admi
 		*   **Change Password:** Setting this flag will require the user to change password after the next login.
 		*   **Reset Auth Key:** Use this link for generate a new AuthKey.
 ![Edit user.](figures/edit_users.png)
-	*    **Delete the user:** If you want to delete a user.
+	*    **Delete the user:** If you want to delete a user. (Note: disabling is the preferred method)
 ![delete user.](figures/delete_user.png)
 	*    **Display the user:** Display all user's information.<br />
 ![display user.](figures/display_user.png)
@@ -329,13 +329,22 @@ If enabled, MISP can delegate a lot of the time intensive tasks to the backgroun
 #### Command Line Tools for the Background Workers
 
 The background workers are powered by [CakeResque](https://github.com/kamisama/Cake-Resque), so all of the CakeResque commands work.
-To start all of the workers needed by MISP go to your `/var/www/MISP/app/Console/worker` (assuming a standard installation path) and execute start.sh.
-To interact with the workers, here is a list of useful commands. Go to your `/var/www/MISP/app/Console` (assuming a standard installation path) and execute one of the following commands as a parameter to `./cake CakeResque.CakeResque` (for example: `./cake CakeResque.CakeResque tail`):
+To start all of the workers needed by MISP go to your `/var/www/MISP/app/Console/worker` (assuming a standard installation path) and execute `start.sh`.
+To interact with the workers, here is a list of useful commands. Go to your `/var/www/MISP/app/Console` (assuming a standard installation path) and execute one of the following commands as a parameter to `./cake CakeResque` (for example: `./cake CakeResque tail`):
 
-*   **tail**: tail the various log files that CakeResque creates, just choose the one from the list that you are interested in.
-*   **cleanup**: terminate the job that a worker is working on with immediate effect. You will be presented with a choice of workers to choose from when executing this command.
-*   **clear**: Clear the queue of a worker immediately.
-*   **stats**: Display some statistics about your workers including the count of successful and failed jobs.
+*   **start**:           Start a new worker.
+*   **startscheduler**:  Start a new scheduler worker.
+*   **stop**:            Stop a worker.
+*   **pause**:           Pause a worker.
+*   **resume**:          Resume a paused worker.
+*   **cleanup**:         Terminate the job that a worker is working on with immediate effect. You will be presented with a choice of workers to choose from when executing this command.
+*   **restart**:         Stop all Resque workers, and start a new one.
+*   **clear**:           Clear all jobs inside a queue
+*   **reset**:           Reset CakeResque internal worker's saved status
+*   **stats**:           Display some statistics about your workers including the count of successful and failed jobs.
+*   **tail**:            Tail the various (workers) log files that CakeResque creates, just choose the one from the list that you are interested in.
+*   **track**:           Track a job status.
+*   **load**:            Load a set of predefined workers.
 
 The other commands should not be required, instead of starting / stopping or restarting workers use the supplied start.sh (it stops all workers and starts them all up again). For further instructions on how to use the console commands for the workers, visit the [CakeResque list of commands](http://cakeresque.kamisama.me/commands#cleanup).
 
@@ -585,3 +594,33 @@ An example of error message:
 ```
 Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'Task.job_id' in 'field list'
 ```
+
+### Jobs
+
+The Jobs tab gives you an overview on any currently running jobs or jobs that were previously completed and their status.
+
+![Running Jobs](figures/jobs-running.png)
+
+Typically this is one of the places you would turn to even some background process might not complete as expected to get an indication on any issues related to user initiated Jobs.
+
+For ease of use, you can filter the Jobs by 'All', 'Default', 'Email', 'Cache'
+##### Todo: Explain differences Default, Email, Cache
+
+You can also purge the entries, either only by completed status or purge all.
+This is not automated and needs to be done manually.
+
+### Scheduled Tasks
+
+Straight from the UI:
+
+"""
+Here you can schedule pre-defined tasks that will be executed every x hours. You can alter the date and time of the next scheduled execution and the frequency at which it will be repeated (expressed in hours). If you set the frequency to 0 then the task will not be repeated. To change and of the above mentioned settings just click on the appropriate field and hit update all when you are done editing the scheduled tasks.
+
+Warning: Scheduled tasks come with a lot of caveats and little in regards of customisations / granularity. You can instead simply create cron jobs out of the console commands as described here: Automating certain console tasks
+"""
+
+The task scheduler is a sub-par component to enable minimal functionality in terms of automating certain MISP tasks.
+If you have a dedicated and concious MISP Site Admin she can keep an eye on the Scheduler to make sure everything runs smoothly.
+
+For better performance please use a real scheduler like your systems' crontab.
+As a rule of thumb: If you can click on it, MISP can automate it.

appendices/README.md

@@ -1,3 +1,7 @@
+# Summary
+
+<!-- toc -->
+
 # Appendix A: External Authentication
 
 #### The external authentication mechanism described
@@ -220,3 +224,86 @@ https://<misp url>/servers/queryACL/findMissingFunctionNames
 
 Functions that have not been tied into the new ACL yet show up here. These functions will (until added to the ACL) only be accessible to site admins.
 
+# Appendix C: Official MISP developments
+
+This section lists the projects that can be found on the main [MISP GitHub](https://github.com/MISP/repositories) page
+ e know of but not officially support and rely on their respective maintainers to keep up to date to the MISP 2.4 developments.
+
+
+| Project | Description | Status |
+| -- | -- | -- |
+| [misp-objects](https://github.com/MISP/misp-objects) | Definition, description and relationship types of MISP objects | Core to MISP, frequently updated and tested |
+
+<!--
+| []() |  | Core to MISP, frequently updated and tested |
+| []() |  | Core to MISP, frequently updated and tested |
+| []() |  | Core to MISP, frequently updated and tested |
+-->
+
+# Appendix D: Third-party development
+
+This section lists some projects we know of but not officially support and rely on their respective maintainers to keep up to date to the MISP 2.4 developments.
+
+| Project | Description | Status |
+| -- | -- | -- |
+| [MISP-STIX-ESM](https://github.com/mohlcyber/MISP-STIX-ESM) | Exports MISP events to STIX and ingest into McAfee ESM | Not tested by MISP core team |
+| [Docker MISP](https://github.com/harvard-itsecurity/docker-misp) | Automated Docker MISP container | Not tested by MISP core team |
+| [misp42splunk](https://github.com/remg427/misp42splunk) | A Splunk app to use MISP in background and combine with TheHive | Not tested by MISP core team |
+| [getmispioc](https://github.com/xme/splunk/tree/master/getmispioc) | getiocmisp is a Splunk custom search command that helps to extract IOCs from a MISP instance. | Not tested by MISP core team |
+| [OTX MISP](https://github.com/gcrahay/otx_misp) | Imports Alienvault OTX pulses to a MISP instance | Not tested by MISP core team |
+| [BTG](https://github.com/conix-security/BTG) | BTG's purpose is to make fast and efficient search on IOC | Not tested by MISP core team |
+| [MISP OSINT Collection](https://github.com/adulau/misp-osint-collection) | Collection of best practices to add OSINT into MISP and/or MISP communities | Not tested by MISP core team |
+| [IBM XFE module](https://github.com/johestephan/XFE) | Various IBM X-Force Exchange modules | Not tested by MISP core team |
+| [MISP dockerized](https://github.com/DCSO/MISP-dockerized-misp-modules) | MISP dockerized is a project designed to provide an easy-to-use and easy-to-install'out of the box' MISP instance that includes everything you need to run MISP with minimal host-side requirements. | Not tested by MISP core team |
+| [MISP dockerized modules](https://github.com/DCSO/MISP-dockerized-misp-modules) | MISP-modules for MISP dockerized | Not tested by MISP core team |
+| [FireMISP](https://github.com/deralexxx/FireMISP) | FireEye Alert json files to MISP Malware information sharing plattform (Alpha) | Not tested by MISP core team |
+| [MISP Chrome Plugin](https://github.com/deralexxx/misp-chrome-plugin) | MISP Chrome plugin for adding and looking up indicators | Not tested by MISP core team |
+| [PySight2MISP](https://github.com/deralexxx/PySight2MISP) | PySight2MISP is a project that can be run to be used as glue between iSight intel API and MISP API | Not tested by MISP core team |
+| [tie2misp](https://github.com/DCSO/tie2misp) | Import DCSO TIE IOCs as MISP events | Not tested by MISP core team |
+| [security onion MISP](https://github.com/weslambert/securityonion-misp) | Grab NIDS rules and Bro Intel generated from a MISP instance and use them in Security Onion | Not tested by MISP core team |
+| [virustream](https://github.com/ntddk/virustream) | A script to track malware IOCs with OSINT on Twitter. | Not tested by MISP core team |
+| [LAC CSV Import](https://github.com/LAC-Japan/MISP-CSVImport) | Register MISP events based on information described in files such as CSV and TSV. | Not tested by MISP core team |
+| [The Hive](https://github.com/TheHive-Project/TheHive) | TheHive: a Scalable, Open Source and Free Security Incident Response Platform | Strong links between core team members, tested and known working |
+| [puppet-misp](https://github.com/voxpupuli/puppet-misp) | This module installs and configures MISP - [puppet forge site](https://forge.puppet.com/puppet/misp) | Not tested by MISP core team |
+| [Ansible MISP](https://github.com/StamusNetworks/ansible-misp) | Ansible playbook to install Malware Information Sharing Platform (MISP) | **unmaintained** |
+| [ansible MISP](https://github.com/juju4/ansible-MISP) | ansible role to setup MISP | Not tested by MISP core team |
+| [OpenDXL ATD MISP](https://github.com/mohlcyber/OpenDXL-ATD-MISP) | Automated threat intelligence collection with McAfee ATD, OpenDXL and MISP | Not tested by MISP core team |
+| [IMAP Proxy](https://github.com/CIRCL/IMAP-Proxy) | Modular IMAP proxy (including PyCIRCLeanMail and MISP forward modules) | Not tested by MISP core team |
+| [AutoMISP](https://github.com/da667/AutoMISP) | automate your MISP installs - This shell script is designed to automatically install [MISP](https://github.com/MISP/MISP) and the [misp-modules](https://github.com/MISP/misp-modules) extension on either Ubuntu 16.04, or 18.04. | Not tested by MISP core team |
+| [Palo Alto Networks report_to_misp](https://github.com/PaloAltoNetworks/report_to_misp) | Parse a report and import the events into MISP | Not tested by MISP core team |
+| [Palo Alto Networks minemeld-misp](https://github.com/PaloAltoNetworks/minemeld-misp) | MineMeld nodes for MISP | Not tested by MISP core team |
+| [golang-misp](https://github.com/0xrawsec/golang-misp) | Golang Library to interact with your MISP instance | Not tested by MISP core team |
+| [go-misp](https://github.com/Zenithar/go-misp) | Golang MISP [API Client](http://zenithar.org/go/misp) | Not tested by MISP core team |
+| [MISP MAR](https://github.com/mohlcyber/MISP-MAR) | Integration between MISP platform and McAfee Active Response | Not tested by MISP core team |
+| [MISP IoC Validator](https://github.com/tom8941/MISP-IOC-Validator) | Validate IOC from MISP ; Export results and iocs to SIEM and sensors using syslog and CEF format | Not tested by MISP core team |
+| [vt2misp](https://github.com/eCrimeLabs/vt2misp) | Script to fetch data from virustotal and add it to a specific event as an object | Not tested by MISP core team |
+| [Threat Pinch Lookup](https://github.com/cloudtracer/ThreatPinchLookup) | Documentation and Sharing Repository for ThreatPinch Lookup Chrome & Firefox [Extension](https://chrome.google.com/webstore/detail/threatpinch-lookup/ljdgplocfnmnofbhpkjclbefmjoikgke) | Not tested by MISP core team |
+| [dovehawk](https://github.com/tylabs/dovehawk) | Dovehawk is a Bro module that automatically imports MISP indicators and reports Sightings | Not tested by MISP core team |
+| [yara-exporter](https://github.com/CERT-Bund/yara-exporter) | Exporting MISP event attributes to yara rules usable with Thor apt scanner | Not tested by MISP core team |
+| [volatility-misp](https://github.com/CIRCL/volatility-misp) | Volatility plugin to interface with MISP | Not tested by MISP core team |
+| [misp2bro](https://github.com/thnyheim/misp2bro) | Python script that gets IOC from MISP and converts it into BRO intel files. | Not tested by MISP core team |
+| [TA-misp](https://github.com/stricaud/TA-misp) | Splunk integration with MISP | Not tested by MISP core team |
+| [MISP QRadar](https://github.com/karthikkbala/MISP-QRadar-Integration) | The Project can used to integrate QRadar with MISP Threat Sharing Platform | Not tested by MISP core team |
+| [pymisp-suricata_search](https://github.com/raw-data/pymisp-suricata_search) | Multi-threaded suricata search module for MISP | Not tested by MISP core team |
+| [MISP-ThreatExchange](https://github.com/EC-DIGIT-CSIRC/MISP-ThreatExchange) | Script to interface MISP with Facebook ThreatExchange | Not tested by MISP core team |
+| [aptc](https://github.com/jymcheong/aptc) | [Automated Payload Test Controller](https://jymcheong.github.io/aptc/) | Not tested by MISP core team |
+| [aptmap](https://github.com/3c7/aptmap) | A [map](https://aptmap.netlify.com) displaying threat actors from the [misp-galaxy](https://github.com/MISP/misp-galaxy) | Not tested by MISP core team |
+| [mispy](https://github.com/nbareil/mispy) | Another MISP module for Python | Not tested by MISP core team |
+| [MispSharp](https://github.com/DBHeise/MispSharp) | C# Library for MISP | Not tested by MISP core team |
+| [misp_btc](https://github.com/rommelfs/misp_btc) | get BTC addresses from MISP and fetch BTC transactions | Tested by MISP core team |
+| [Privacy Aware Sharing of IoCs in MISP](https://github.com/charly077/MISP-privacy-aware-sharing-master-thesis) | [Master Thesis](https://github.com/charly077/MISP-privacy-aware-sharing-master-thesis/blob/master/report/report.pdf) including MISP data. | Master thesis |
+
+<!--
+| []() |  | Not tested by MISP core team |
+| []() |  | Not tested by MISP core team |
+| []() |  | Not tested by MISP core team |
+-->
+
+# Appendix E: Other Threat Intel Ressources
+
+A brief list of online ressources that around #ThreatIntel
+
+* [Curated list of awesome cybersecurity companies and solutions.](https://github.com/Annsec/awesome-cybersecurity/blob/master/README.md) (Updated April 2017)
+* [A curated list of awesome malware analysis tools and resources](https://github.com/rshipp/awesome-malware-analysis/blob/master/README.md). Inspired by [awesome-python](https://github.com/vinta/awesome-python) and [awesome-php](https://github.com/ziadoz/awesome-php).
+* [An authoritative list of awesome devsecops tools with the help from community experiments and contributions](https://github.com/devsecops/awesome-devsecops/blob/master/README.md).[DEV.SEC.OPS](http://devsecops.org)
+* [Advance Python IoC extractor](https://github.com/InQuest/python-iocextract)

automation/README.md

@@ -1407,6 +1407,10 @@ https://<misp url>/events/restSearch/download/null/null/null/null/tag1&&tag2&&!t
 <dd>Include the attachments/encrypted samples in the export</dd>
 <dt>metadata</dt>
 <dd>Only fetch the event metadata (event data, tags, relations) and skip the attributes</dd>
+<dt>limit</dt>
+<dd>Limit the number of results returned; use together with page.</dd>
+<dt>page</dt>
+<dd>If a limit is set, sets the page to be returned, starting at 1; page 3, limit 100 will return records 201->300). When requesting a page beyond the number of available pages, the returned results list will be empty.</dd>
 </dl>
 
 The keywords false or null should be used for optional empty parameters in the URL.
@@ -1692,7 +1696,7 @@ A feed can be disabled by POSTing on the following URL (feed_id is the id of the
 /feeds/disable/feed_id
 ~~~~
 
-All feeds can fetch via the API:
+All feeds can cached via the API:
 
 ~~~~
 /feeds/cacheFeeds/all
@@ -1701,6 +1705,13 @@ All feeds can fetch via the API:
 or you can replace `all` by the feed format to fetch like `misp` or `freetext`. `all` can be replaced
 with the `id` value of the feed to fetch a specific feed.
 
+To fetch a feed or all feeds:
+
+~~~~
+/feeds/fetchFromFeed/feed_id
+/feeds/fetchFromAllFeeds
+~~~~
+
 This API can be also used to download feeds at regular interval via cronjobs or alike.
 
 

book-convention/README.md

@@ -1,3 +1,9 @@
+<!-- This is a comment.
+If you plan on contributing to misp-book, welcome and enjoy.
+In case of any and all questions, feel free to join our gitter:
+https://gitter.im/MISP/MISP
+For Aiur! -->
+
 ---
 description: Convention Used in MISP-Book
 ---
@@ -8,6 +14,33 @@ description: Convention Used in MISP-Book
 
 * Used for variable, function or menu names in MISP.
 
+## Language
+
+The language in this book is american english.
+All the screenshots and examples are in english.
+
+## CoC
+
+The same code of conduct applies to this book as for the main MISP project.
+As a book can some times be considered the inadvertent sould of a piece of software, please take good care and consideration of our `Code of Conduct`. The CoC [can be read here](https://github.com/MISP/MISP/blob/2.4/code_of_conduct.md).
+
+## Example install
+
+The examples and screenshots provided in this book have been created with the MISP Autogenerated VM.
+
+To get a copy of the latest VM [click here](https://www.circl.lu/misp-images/latest/)
+
+
+## MISP Instance
+
+In general when talking about a network of inter-connected MISP servers, each server is a MISP instance. Whilst we have no strong feelings towards anyones naming schemes, as a rule of thumb try to have a scheme that makes everyday use easy when analysts need to talk about remote MISP instances.
+<!--
+ToDo: Be more specific give some naming convention examples.
+-->
+
+The hostname used for the instance in this book is `misp.local` and we will henceforth refer to it either by name or as `local MISP instance`.
+
+
 ## Example Organisations
 
 As MISP is a platform to support information sharing, example organisations are often used within this book.
@@ -21,5 +54,15 @@ The following two organisations are regularly used as example:
 
 Starting from MISP 2.4.71, the example organisations with the above mentioned UUID are **black-listed** to avoid  
 large distribution of sample events while testing a MISP instance. If you want to test your distribution, the  
-sample organisation black-listing can be removed in `Administration` under `Manage Org Blacklists`.
+sample organisation black-listing can be removed in `Administration`/`Manage Org Blacklists`.
+
+## Example IOCs
+
+As with the example organisations, we want to make this book as useful as possible by using real life examples.
+
+The following IOC examples have been used:
+
+* [Sirefef](https://www.misp-project.org/galaxy.html#_zeroaccess) (aka ZeroAccess) Sample Event ID: #31337
+* [WannaCry](https://www.misp-project.org/galaxy.html#_wannacry) Sample Event ID: #42
+* [Dridex](https://www.misp-project.org/galaxy.html#_dridex) Sample Event ID: #23
 

book.json

@@ -3,7 +3,7 @@
     "description": "User guide of MISP Malware Information Sharing Platform, a Threat Sharing Platform.",
     "language": "en",
     "author": "MISP Contributors",
-    "plugins": ["autocover", "github", "toc", "anchors", "alerts", "advanced-emoji", "image-class"],
+    "plugins": ["autocover", "github", "toc", "anchors", "alerts", "advanced-emoji", "image-class", "last-modified", "search", "sitemap", "codesnippet", "gist", "fontsettings"],
     "links": { "sidebar": {        "MISP @ GitHub": "https://github.com/MISP/MISP", "PDF Format": "https://www.circl.lu/doc/misp/book.pdf" }},
     "pluginsConfig": {
         "github": {
@@ -12,12 +12,20 @@
         "twitter": {
                 "url": "https://www.twitter.com/MISPProject/"
         },
-     "styles": {
-        "website": "styles/website.css",
-        "ebook": "styles/ebook.css",
-        "pdf": "styles/pdf.css",
-        "mobi": "styles/mobi.css",
-        "epub": "styles/epub.css"
-    }
+        "sitemap": {
+                "hostname": "https://www.circl.lu/doc/misp/"
+        },
+        "fontsettings": {
+            "theme": "night",
+            "family": "sans",
+            "size": 2
+        },
+        "styles": {
+            "website": "styles/website.css",
+            "ebook": "styles/ebook.css",
+            "pdf": "styles/pdf.css",
+            "mobi": "styles/mobi.css",
+            "epub": "styles/epub.css"
+        }
 }
 }

create-event-report/README.md

@@ -6,7 +6,7 @@ For this example, we will use a report found on [Bleeping Computer](http://www.b
 
 ![Report title](figures/report_title.png)
 
-### The metadata
+### Adding an event
 
 First of all, we need to create a new event. To do so, we click the "Add Event" option when on the Events list view.
 
@@ -31,6 +31,8 @@ Then just press the blue "Add" button and here we have a brand new event. Empty.
 ![EMPTY EVENT YAY](figures/event_metadata.png)  
 (Displayed information can change depending on your role on the MISP instance)
 
+### Adding Attributes
+
 Now it is time to populate this event. But before even adding IoC, we are going to add global information about the report itself: the link of the report and a short explanation or introduction. To do so, we need to click on the "Add Attribute" option in the side menu. This will show us this view:
 
 ![add attribute](figures/add_attribute.png)
@@ -84,9 +86,11 @@ So we begin with the filename. No real change from before for this one, except t
 
 ![filename](figures/filename.png)
 
+### Freetext Import Tool
+
 Then we can add the hashes in a similar way. We will had them both alone and combined with the filename. In order to do it quickly, we are going to use the freetext import tool, hidden there
 
-![freetext import step 1](figures/freeeeeimport.png)
+![freetext import step 1](figures/freeeeeimport.png) 
 
 It will open a popup with a text area field where we will paste our IoC, one per line. As said previously, we add both the hashes alone and with the filename.
 
@@ -106,13 +110,29 @@ If the results of MISP were not what we expected, we can still modify it, howeve
 We only have the network indicators left, and as said before, we will let MISP determined for us which type is the best for the data we have.
 
 ![freetext import network](figures/free_network.png)
+
 ![type recognition fail](figures/surprise.png)
 
 Oh well, that was unexpected. In fact, it is not that surprising regarding the format of the tor address that look more like a filename than like a url but it is still a problem, since we can't change the type nor the category to a more consistant one. This is indeed one of the limitation of freetext import. To solve this issue, we will use a simple trick: we will add a slash at the end of the tor address so it won't be confused for a filename. 
 
 ![freetext import network](figures/free_network2.png)
+
 ![type recognition fail](figures/nomoresurprise.png)
 
 Thanks to the added character, the first string is recognised as an url which is more consistent with the reality. The second also seems okay, so we can now submit both. 
 
+### Batch Import
+
+The Freetext Import works properly only with a string of data without any spaces in one line. But if you have lines of text with spaces between values, like e.g. 
+
+![freetext_with_spaces1](figures/freetext_with_spaces1.png)
+
+you can still import them at once using the "Add Attribute" option. Click on _Add Attribute_, copy the data and paste it into the _Value_ box. Choose the right category and type. Now check both checkboxes _for Intrusion Detection System_ and _Batch Import_. The option _Batch Import_ will import your data line for line just like the _Freetext Import_ option without losing any information. Like this:      
+
+![addattribute_freetext](figures/addattribute_freetext.png)
+
 And that is all we can get for the main informations and IoC in this report. If we search more carefully, there might still be some information left in it, like the filename of the ransomnote for instance, but we will stop here for this example.
+
+### Modify the event
+
+If you want to modify your event from the home page, you can either double click on the event or click the edit symbol located in the column __Actions__ on the right side. You will be redirected to the editing mode of the selected event.    

dev-faq/README.md

@@ -0,0 +1,20 @@
+# Developer FAQ
+
+## Main Developer Resources
+
+The main developer resources can be found on GitHub in the [MISP Wiki](https://github.com/MISP/MISP/wiki).
+
+The following pages are worth inspecting closer in case you want to actively develop for MISP:
+
+- [The real FAQ](https://github.com/MISP/MISP/wiki/Frequently-Asked-Questions)
+- [Contributor Overview](https://github.com/MISP/MISP/wiki/Contributing-to-MISP-Project)
+- [Some objectives of MISP](https://github.com/MISP/MISP/wiki/Critical-aspects-or-features)
+- [Various deployment tools](https://github.com/MISP/MISP/wiki/DeploymentTools)
+- [MISP Code of Conduct](https://github.com/MISP/MISP/blob/2.4/code_of_conduct.md)
+- [UI coloring scheme](https://github.com/MISP/MISP/wiki/UserInterface)
+- [Notes on MISP and STIX 2](https://github.com/MISP/MISP/wiki/Notes:-MISP-STIX2)
+- [Commit Messages Best Practices](https://github.com/MISP/MISP/wiki/CommitMessageBestPractices)
+- [Internationalization (i18n)](https://www.circl.lu/doc/misp/translation/)
+
+Our [gitter channel](https://gitter.im/MISP/MISP) is a welcome place to ask other community developers in case you are stuck.
+

faq/README.md

@@ -0,0 +1,118 @@
+<!-- toc -->
+
+# Frequently Asked Questions
+
+The following page hosts most frequently asked questions as seen on our [issues](https://github.com/MISP/issues) and [gitter](https://gitter.im/MISP/MISP).
+
+## Permission issues
+
+If you have any permission issues, please [set the permissions](https://misp.github.io/MISP/INSTALL.ubuntu1804/#5-set-the-permissions) to something sane first.
+
+## When to update MISP?
+
+One question might be how often to update MISP.
+You can update MISP as ofte as you like. If you see the follwing:
+
+![MISP Update](./figures/misp-diag-update.png)
+
+This means that the main repository has an update available.
+
+If you want to play it safer or want to integrate it in your Weekly/Bi-Monthly update routine you can track our [Changelog](https://www.misp-project.org/Changelog.txt) a more up to date version is available [here](https://misp.github.io/MISP/Changelog/)
+
+## Hardening
+
+### How do I harden my MISP instance?
+
+You can check the [hardening section](https://misp.github.io/MISP/generic/hardening/) in the install guide.
+
+## Maintenance mode
+
+### Is there a MISP maintenance mode?
+
+Yes, you want to flip your instances "Live-mode".
+This wants to be done on the CLI if you experience issues:
+
+```bash
+$PATH_TO_MISP/app/Console/cake "MISP.live" 0
+```
+
+Other related MISP Settings
+
+Optional	MISP.maintenance_message	Great things are happening! MISP is undergoing maintenance, but will return shortly. You can contact the administration at $email or call CIRCL.	The message that users will see if the instance is not live.
+
+Critical	MISP.live	true	Unless set to true, the instance will only be accessible by site admins.
+
+## Update MISP fails
+
+If your MISP instance is outdated, meaning ONLY the core, not the modules or dashboard or python modules, you well see the following.
+
+![MISP outdated](./figures/misp-outdated.png)
+
+Once you click on update MISP you will be asked confirmation.
+
+![MISP Update Yes/No](./figures/update-misp-YN.png)
+
+If you are not on a branch, the UI will tell you this, the update will fail.
+
+![not on branch](./figures/misp-not-on-branch.png)
+
+If you cannot write the **.git** files and directory as the user running the web server (and thus PHP), the update will fail.
+The following diagnostic check will let you know if you can update or not.
+
+![.git not writeable](./figures/misp-diag-not-writeable-files-git.png)
+
+In case you get a file not found on **.git/ORIG_HEAD**, this means that you have never updated your MISP OR you have installed git from an archive file (like .zip/.tar.gz or similar)
+Try to click update MISP and see what happens.
+
+![ORIG_HEAD file not found](./figures/misp-diag-writeable-files-not_found-git.png)
+
+### What can go wrong if I update MISP?
+
+In theory nothing. We put great effort into protecting the integrity of the data stored in your MISP instance.
+DB upgrades happen upon login or on reload once you have update the repository.
+You cannot "break" anything by clicking **Update MISP** worse case it will complain about something and you will certainly find the answer on this page.
+
+IF not, please open an [issue](https://github.com/MISP/MISP/issues) on GitHub or come to our [gitter](https://gitter.im/MISP/MISP) chat to see if the community can help.
+
+### error: pathspec 'app/composer.json' did not match any file(s) known to git
+
+This is **not** an error and can be ignore. Nothing will be impacted by this.
+
+![pathspec](./figures/misp-pathspec.png)
+
+### MISP modules "Connection refused"
+
+![MISP Modules ](./figures/misp-module-system-diag.png)
+
+If you get have a **Connection refused state** on your modules one of the following might be true.
+
+- You have no [misp-modules](https://github.com/MISP/misp-modules) not installed
+- They are instaled but not running
+- Something completly different
+
+If they are not installed, check out this section of the [INSTALL guide](https://github.com/MISP/misp-modules/#how-to-install-and-start-misp-modules-in-a-python-virtualenv) of [misp-modules](https://github.com/MISP/misp-modules).
+
+In case they are not running, try this on the console:
+
+```
+sudo -u www-data /var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s &
+```
+
+OR if you were foolish enough to not install in a Python virtualenv:
+
+```
+sudo -u www-data misp-modules -l 127.0.0.1 -s &
+```
+
+> [warning] Running misp-modules like this will certainly kill it once you quit the session. Make sure it is in your **/etc/rc.local** or some ther init script that gets run on boot.
+
+## Uninstalling MISP
+
+There is no official procedure to uninstalling a MISP instance.
+
+If you want to re-use a machine where MISP was installed, wipe the machine and do a fresh install.
+Consider the data in your MISP instance as potentially confidential and if you synchronized with other instances, be respectful and wipe it clean.
+
+  <!-- 
+  Comment Place Holder
+  -->

galaxy/README.md

@@ -4,13 +4,13 @@
 
 Galaxies in MISP are a method used to express a large object called cluster that can be attached to MISP events or attributes. A cluster can be composed of one or more elements. Elements are expressed as key-values.
 
-There are default vocabularies available in MISP galaxy but those can be overwritten, replaced or updated as you wish. Vocabularies are from existing standards (like STIX, Veris, MISP and so on) or custom ones.
+There are default vocabularies available in MISP galaxy but those can be overwritten, replaced or updated as you wish. Vocabularies are from existing standards (like [STIX](https://oasis-open.github.io/cti-documentation/stix/intro), [Veris](http://veriscommunity.net/veris-overview.html), [ATT&CK](https://attack.mitre.org/), MISP and so on) or custom ones you only use for your organization.
 
 Existing clusters and vocabularies can be used as-is or as a template. MISP distribution can be applied to each cluster to permit a limited or broader distribution scheme.
 
 The objective is to have a common set of clusters for organizations starting analysis but that can be expanded to localized information (which is not shared) or additional information (that can be shared).
 
-[MISP galaxy](https://github.com/MISP/misp-galaxy) are available on Github.
+[MISP galaxy](https://github.com/MISP/misp-galaxy) is available on Github.
 
 ### Managing Galaxies in MISP
 
@@ -24,11 +24,11 @@ A list with all the galaxies existing on the server will appear.
 
 ![GalaxyView](./figures/GalaxyView.png)
 
-Each galaxy can be explored using the icon at the end of the line.
+Each galaxy can be explored using the **View** icon at the end of the line.
 
 ![GalaxyList](./figures/GalaxyList.png)
 
-Here is shown the metadata of the selected galaxy as well as a table with each available value as well as some complementary data such as a description of the value or the activity, that is to say the evolution of the use of each value.
+Here the metadata of the selected galaxy is shown. You also see a table with each available value as well as some complementary data such as a description of the value or the activity (MISP Sightings), that is to say the evolution of the use of each value.
 
 Galaxies can be reimported from the submodules by clicking the "Update Galaxies" link on either the galaxies list or while browsing a specific galaxy. A popup will appear to confirm the reimportation.
 
@@ -38,7 +38,7 @@ All galaxies will always be updated, even while browsing a specific galaxy.
 
 ### Using Galaxies in MISP Events - Example
 
-For this example, we will try to add a cluster to an existing event. This cluster will contains informations about threat actor known as Sneaky Panda.
+For this example, we will try to add a cluster to an existing event. This cluster contains information about  threat actor known as Sneaky Panda.
 
 ![EventWithoutCluster](./figures/EventWithoutCluster.png)
 
@@ -68,66 +68,102 @@ Clicking on the addition symbol on the left of Beijing Group extends the module.
 
 ### Available Galaxies
 
+<!-- NB. This list is generated dynamically with gen-doc.sh included in this directory. -->
+
 #### Clusters
 
 [Android](https://github.com/MISP/misp-galaxy/blob/master/clusters/android.json) - Android malware galaxy based on multiple open sources.
 
+[Backdoor](https://github.com/MISP/misp-galaxy/blob/master/clusters/backdoor.json) - A list of backdoor malware.
+
 [Banker](https://github.com/MISP/misp-galaxy/blob/master/clusters/banker.json) - A list of banker malware.
 
-[Exploit Kit](https://github.com/MISP/misp-galaxy/blob/master/clusters/exploit-kit.json) - Exploit-Kit is an enumeration of some exploitation kits used by adversaries. The list includes document, browser and router exploit kits. It's not meant to be totally exhaustive but aim at covering the most seen in the past 5 years.
+[Botnet](https://github.com/MISP/misp-galaxy/blob/master/clusters/botnet.json) - botnet galaxy
 
-[Microsoft Activity Group](https://github.com/MISP/misp-galaxy/blob/master/clusters/microsoft-activity-group.json) - Activity groups as described by Microsoft.
+[Branded vulnerability](https://github.com/MISP/misp-galaxy/blob/master/clusters/branded_vulnerability.json) - List of known vulnerabilities and attacks with a branding
 
-[Preventive Measure](https://github.com/MISP/misp-galaxy/blob/master/clusters/preventive-measure.json) - Preventive measures.
+[Cert eu govsector](https://github.com/MISP/misp-galaxy/blob/master/clusters/cert-eu-govsector.json) - Cert EU GovSector
 
-[Ransomware](https://github.com/MISP/misp-galaxy/blob/master/clusters/ransomware.json) - Ransomware galaxy based on https://docs.google.com/spreadsheets/d/1TWS238xacAto-fLKh1n5uTsdijWdCEsGIM0Y0Hvmc5g/pubhtml
+[Exploit kit](https://github.com/MISP/misp-galaxy/blob/master/clusters/exploit-kit.json) - Exploit-Kit is an enumeration of some exploitation kits used by adversaries. The list includes document, browser and router exploit kits.It's not meant to be totally exhaustive but aim at covering the most seen in the past 5 years
 
-[RAT](https://github.com/MISP/misp-galaxy/blob/master/clusters/rat.json) - remote administration tool or remote access tool (RAT), also called sometimes remote access trojan, is a piece of software or programming that allows a remote "operator" to control a system as if they have physical access to that system.
+[Malpedia](https://github.com/MISP/misp-galaxy/blob/master/clusters/malpedia.json) - Malware galaxy cluster based on Malpedia.
 
-[TDS](https://github.com/MISP/misp-galaxy/blob/master/clusters/tds.json) - TDS is a list of Traffic Direction System used by adversaries.
+[Microsoft activity group](https://github.com/MISP/misp-galaxy/blob/master/clusters/microsoft-activity-group.json) - Activity groups as described by Microsoft
 
-[Threat Actor](https://github.com/MISP/misp-galaxy/blob/master/clusters/threat-actor.json) - Adversary groups - Known or estimated adversary groups targeting organizations and employees. Adversary groups are regularly confused with their initial operation or campaign. MISP
+[Mitre attack pattern](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-attack-pattern.json) - ATT&CK tactic
 
-[Tool](https://github.com/MISP/misp-galaxy/blob/master/clusters/tool.json) - tool is an enumeration of tools used by adversaries. The list includes malware but also common software regularly used by the adversaries.
+[Mitre course of action](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-course-of-action.json) - ATT&CK Mitigation
 
+[Mitre enterprise attack attack pattern](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-enterprise-attack-attack-pattern.json) - ATT&CK tactic
 
-[MITRE Attack Pattern](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre_attack-pattern.json) - Attack Pattern - MITRE Adversarial Tactics, Techniques & Common Knowledge (ATT&CK)
+[Mitre enterprise attack course of action](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-enterprise-attack-course-of-action.json) - ATT&CK Mitigation
 
-[MITRE Course of Action](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre_course-of-action.json) - Course of Action - MITRE Adversarial Tactics, Techniques & Common Knowledge (ATT&CK)
+[Mitre enterprise attack intrusion set](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-enterprise-attack-intrusion-set.json) - Name of ATT&CK Group
 
-[MITRE Intrusion Set](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre_intrusion-set.json) - Intrusion Test - MITRE Adversarial Tactics, Techniques & Common Knowledge (ATT&CK)
+[Mitre enterprise attack malware](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-enterprise-attack-malware.json) - Name of ATT&CK software
 
-[MITRE Malware](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre_malware.json) - Malware - MITRE Adversarial Tactics, Techniques & Common Knowledge (ATT&CK)
+[Mitre enterprise attack tool](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-enterprise-attack-tool.json) - Name of ATT&CK software
 
-[MITRE Tool](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre_tool.json) - Tool - MITRE Adversarial Tactics, Techniques & Common Knowledge (ATT&CK)
+[Mitre intrusion set](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-intrusion-set.json) - Name of ATT&CK Group
 
+[Mitre malware](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-malware.json) - Name of ATT&CK software
 
-[Sectors](https://github.com/MISP/misp-galaxy/blob/master/clusters/sectors.json) - Activity sectors
+[Mitre mobile attack attack pattern](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-mobile-attack-attack-pattern.json) - ATT&CK tactic
 
-[CertEU Govsector](https://github.com/MISP/misp-galaxy/blob/master/clusters/cert-eu-govsector.json) - Cert EU GovSector/master/clusters/tool.json) - Enumeration of software tools used by adversaries. The list includes malware but also common software regularly used by the adversaries.
+[Mitre mobile attack course of action](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-mobile-attack-course-of-action.json) - ATT&CK Mitigation
+
+[Mitre mobile attack intrusion set](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-mobile-attack-intrusion-set.json) - Name of ATT&CK Group
+
+[Mitre mobile attack malware](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-mobile-attack-malware.json) - Name of ATT&CK software
+
+[Mitre mobile attack tool](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-mobile-attack-tool.json) - Name of ATT&CK software
+
+[Mitre pre attack attack pattern](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-pre-attack-attack-pattern.json) - ATT&CK tactic
+
+[Mitre pre attack intrusion set](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-pre-attack-intrusion-set.json) - Name of ATT&CK Group
+
+[Mitre tool](https://github.com/MISP/misp-galaxy/blob/master/clusters/mitre-tool.json) - Name of ATT&CK software
+
+[Preventive measure](https://github.com/MISP/misp-galaxy/blob/master/clusters/preventive-measure.json) - Preventive measures based on the ransomware document overview as published in https://docs.google.com/spreadsheets/d/1TWS238xacAto-fLKh1n5uTsdijWdCEsGIM0Y0Hvmc5g/pubhtml# . The preventive measures are quite generic and can fit any standard Windows infrastructure and their security measures.
+
+[Ransomware](https://github.com/MISP/misp-galaxy/blob/master/clusters/ransomware.json) - Ransomware galaxy based on https://docs.google.com/spreadsheets/d/1TWS238xacAto-fLKh1n5uTsdijWdCEsGIM0Y0Hvmc5g/pubhtml and http://pastebin.com/raw/GHgpWjar
+
+[Rat](https://github.com/MISP/misp-galaxy/blob/master/clusters/rat.json) - remote administration tool or remote access tool (RAT), also called sometimes remote access trojan, is a piece of software or programming that allows a remote "operator" to control a system as if they have physical access to that system.
+
+[Sector](https://github.com/MISP/misp-galaxy/blob/master/clusters/sector.json) - Activity sectors
+
+[Stealer](https://github.com/MISP/misp-galaxy/blob/master/clusters/stealer.json) - A list of malware stealer.
+
+[Tds](https://github.com/MISP/misp-galaxy/blob/master/clusters/tds.json) - TDS is a list of Traffic Direction System used by adversaries
+
+[Threat actor](https://github.com/MISP/misp-galaxy/blob/master/clusters/threat-actor.json) - Known or estimated adversary groups targeting organizations and employees. Adversary groups are regularly confused with their initial operation or campaign.
+
+[Tool](https://github.com/MISP/misp-galaxy/blob/master/clusters/tool.json) - threat-actor-tools is an enumeration of tools used by adversaries. The list includes malware but also common software regularly used by the adversaries.
 
 #### Vocabularies
 
 ##### Common
 
-[certainty-level](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/certainty-level.json) - Certainty level of an associated element or cluster.
+[Certainty level](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/certainty-level.json) - Certainty level of an associated element or cluster.
 
-[threat-actor-type](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/threat-actor-type.json) - threat actor type vocab as defined by Cert EU.
+[Sector](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/sector.json) - List of activity sectors
 
-[ttp-category](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/ttp-category.json) - ttp category vocab as defined by Cert EU.
+[Threat actor type](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/threat-actor-type.json) - threat actor type vocab as defined by Cert EU.
 
-[ttp-type](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/ttp-type.json) - ttp type vocab as defined by Cert EU.
+[Ttp category](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/ttp-category.json) - ttp category vocab as defined by Cert EU.
+
+[Ttp type](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/ttp-type.json) - ttp type vocab as defined by Cert EU.
 
 ##### threat-actor
 
-[cert-eu-motive](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/cert-eu-motive.json) - Motive vocab as defined by Cert EU.
+[Cert eu motive](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/cert-eu-motive.json) - Motive vocab as defined by Cert EU.
 
-[intended-effect-vocabulary](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/intended-effect.json) - The IntendedEffectVocab is the default STIX vocabulary for expressing the intended effect of a threat actor. STIX 1.2.1
+[Intended effect](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/intended-effect.json) - The IntendedEffectVocab is the default STIX vocabulary for expressing the intended effect of a threat actor
 
-[motivation-vocabulary](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/motivation.json) - The MotivationVocab is the default STIX vocabulary for expressing the motivation of a threat actor. STIX 1.2.1
+[Motivation](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/motivation.json) - The MotivationVocab is the default STIX vocabulary for expressing the motivation of a threat actor.
 
-[planning-and-operational-support-vocabulary](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/planning-and-operational-support.json) - The PlanningAndOperationalSupportVocab is the default STIX vocabulary for expressing the planning and operational support functions available to a threat actor.
+[Planning and operational support](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/planning-and-operational-support.json) - The PlanningAndOperationalSupportVocab is the default STIX vocabulary for expressing the planning and operational support functions available to a threat actor.
 
-[sophistication](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/sophistication.json) - The ThreatActorSophisticationVocab enumeration is used to define the default STIX vocabulary for expressing the subjective level of sophistication of a threat actor.
+[Sophistication](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/sophistication.json) - The ThreatActorSophisticationVocab enumeration is used to define the default STIX vocabulary for expressing the subjective level of sophistication of a threat actor.
 
-[type](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/type.json) - The ThreatActorTypeVocab enumeration is used to define the default STIX vocabulary for expressing the subjective type of a threat actor
+[Type](https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/type.json) - The ThreatActorTypeVocab enumeration is used to define the default STIX vocabulary for expressing the subjective type of a threat actor.

galaxy/gen-doc.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# This generates some of the sections of the Galaxy Documentation
+# Dependencies: git, jq, bash
+
+if [[ ! -d "misp-galaxy" ]]; then
+  git clone https://github.com/MISP/misp-galaxy.git
+  cd misp-galaxy
+else
+  cd misp-galaxy
+  git pull
+fi
+
+FOLDERS='clusters vocabularies/common vocabularies/threat-actor'
+
+for folder in ${FOLDERS}; do
+  cd $folder
+
+  for file in $(ls |grep -v README); do
+    Name=$(echo ${file} |cut -f 1 -d.| tr '-' ' '| tr '_' ' ')
+    Name=$(tr '[:lower:]' '[:upper:]' <<< ${Name:0:1})${Name:1}
+    Description=$(jq -r .description ${file})
+
+    echo "[${Name}](https://github.com/MISP/misp-galaxy/blob/master/${folder}/${file}) - ${Description}"
+    echo ""
+  done
+  echo "------------------------------------------------------------------------------------"
+  echo "$folder done"
+  echo "------------------------------------------------------------------------------------"
+  # This is ugly, but works... fix it if you do not like it.
+  if [[ $folder == *'/'* ]]; then
+    cd ../..
+  else
+    cd ..
+  fi
+done

general-layout/README.md

@@ -1,5 +1,7 @@
 <!-- Nothing else matters -->
 
+<!-- ToDo: Think about other default user roles that need to be considered and documented or at least hinted too. like: the role Read-only and user have these differences, if any -->
+
 ## General Layout
 
 ### The top bar
@@ -11,9 +13,13 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Home button:** This button will return you to the start screen of the application, which is the event index page (more about this later).
 *   **Event Actions:** All the malware data entered into MISP is made up of an event object that is described by its connected attributes. The Event actions menu gives access to all the functionality that has to do with the creation, modification, deletion, publishing, searching and listing of events and attributes.
+*   **Galaxies:** Shortcut to the list of [MISP Galaxies](../galaxy/) on the MISP instance.
 *   **Input Filters:** Input filters alter what and how data can be entered into this instance. Apart from the basic validation of attribute entry by type, it is possible for the site administrators to define regular expression replacements and blacklists for certain values in addition to blocking certain values from being exportable. Users can view these replacement and blacklist rules here whilst administrator can alter them.
 *   **Global Actions:** This menu gives you access to information about MISP and this instance. You can view and edit your own profile, view the manual, read the news or the terms of use again, see a list of the active organizations on this instance and a histogram of their contributions by attribute type.
-*   **Discussions:** Link to the discussion threads.
+*   **MISP:** Simple link to your BASEURL
+*   **Steve:** Name (Auto generated from Mail address) of current logged in user
+*   **Envelope:** Link to User Dashboard where you can consult some of your notifications and changes since last visit. Like some of the proposals recieved for your organisation.
+*   **Log out:** The Log out button to end your session immediatly.
 
 #### Admin Menu Bar
 ![Some additional buttons that will appear on top of these when a view provides it.](figures/MenuBarAdmin.jpg)
@@ -21,6 +27,8 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Event Actions:** ibidem
 
+*   **Galaxies:** You can aditionally update the Galaxies.
+
 *   **Input Filters:** Ibidem
 
 *   **Global Actions:** Ibidem
@@ -31,9 +39,14 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Audit:** If you have audit permissions, you can view the logs for your organization (or for site admins for the entire system) here or even search the logs if you are interested in something specific.
 
-*   **Proposal Notifications:** This shows how many proposals your organization has received and across how many events they are spread out. Clicking this will take you to the list of proposals.
+*   **MISP:** ibidem
+
+*   **Admin:** ibidem
+
+*   **Envelope:** Link to User Dashboard where you can consult some of your notifications and changes since last visit. Like some of the proposals recieved for your organisation.
+
+*   **Log out:** The Log out button to end your session immediatly.
 
-*   **Log out:** Logs you out of the system.
 
 ### A list of the contents of each of the above drop-down menus
 
@@ -48,6 +61,8 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Search Attributes:** You can set search terms for a filtered attribute index view here.
 
+*   **REST client** MISP Online REST client where you can make calls directly to the API via a Web UI.
+
 *   **View Proposals:** Shows a list of all proposals that you are eligible to see.
 
 *   **Events with proposals:** Shows all of the events created by your organisation that has pending proposals.
@@ -56,6 +71,8 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Add Tag:** Create a new tag.
 
+*   **List Taxonomies:** List all of the taxonomies installed on the MISP instance. This is also the place to activate the taxonomies as a Org Admin/Site Admin.
+
 *   **List Templates:** List all of the templates created by users with template creation rights on this instance.
 
 *   **Add Template:** Create a new template.
@@ -72,8 +89,9 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Signature Whitelist:** You can view the whitelist rules, which contains the values that are blocked from being used for exports and automation on this instance. Site administrators have access to editing this list.
 
-*  **List Warninglists:**
-MISP warninglists are lists of well-known indicators that can be associated to potential false positives, errors or mistakes. The warning lists are integrated in MISP to display an info/warning box at the event and attribute level.
+*  **List Warninglists:** MISP warninglists are lists of well-known indicators that can be associated to potential false positives, errors or mistakes. The warning lists are integrated in MISP to display an info/warning box at the event and attribute level.
+
+*  **List Noticelists:** MISP noticelists are lists of #Todo: Double check description from repo!!!
 
 ##### Global Actions
 
@@ -86,8 +104,6 @@ MISP warninglists are lists of well-known indicators that can be associated to p
 
 *   **Dashboard:** allow you to see your Notifications of Proposals, Events with proposals and Delegation request. Your can see the last changes since your last visit, as Events updates and Events publications.
 
-*   **Members List:** View the number of users per organization and get some statistics about the currently stored attributes.
-
 *   **Organizations:** View the organizations having a presence on this instance, with some useful informations as contact's name.
 
 *   **Role Permissions:** You can view the role permissions here.
@@ -98,28 +114,24 @@ MISP warninglists are lists of well-known indicators that can be associated to p
 
 *   **User Guide:** A link to this user guide.
 
+*   **Categories & Types:** Quick overview of Attribute Categories and Type. e.g: md5 -> Payload delivery, Artifacts dropped, Payload installation, External analysis
+
+*   **Terms & Conditions:** General terms and conditions which can be configured in Administration -> Server Settings -> MISP Settings: MISP.terms_file . From the UI: "The filename of the terms and conditions file. Make sure that the file is located in your MISP/app/files/terms directory"
+
 *   **Statistics:** View a series of statistics about the users and the data on this instance.
 
+*   **List Discussions:** List threads of discussions created on the MISP instance by the organisations connected to this local community.
+
+*   **Start Discussion:** Create a new discussion thread. 
+<!-- #ToDo: Make a reference to best practices on how to use this form of messaging. Sense/non-sense etc... -->
 
 ##### Sync Actions
 
-
 ![Sync Actions](figures/SyncActions.png)
 
 *   **List Servers:** Connect your MISP instance to other instances, or view and modify the currently established connections.
-<!-- Fix provided by elhoim -->
-It may be that you have an Error Message in the page (if you enabled debug or site_admin_debug settings). An example of error message:
-
-![Error message](figures/pb-list-server.png)
-
-An easy first step to make most of them go away is to use the clean cache feature on the server settings menu, diagnostics tab.
-
-![cleanscript](figures/cleanscript1.png)
-
-You must then scroll down the page.
-
-![cleanscript](figures/cleanscript2.png)
 
+<!-- Include example of error message/stack trace differences. -->
 
 *  **List Feeds:** Follow the RSS feeds of other organization or CERTs worldwide.  
 
@@ -129,7 +141,7 @@ You must then scroll down the page.
 
 *   **List Users:** View, modify or delete the currently registered users.
 
-*   **New User:** Create an account for a new user for your organisation. Site administrators can create users for any organisation.
+*   **Add User:** Create an account for a new user for your organisation. Site administrators can create users for any organisation.
 
 *  **Contact Users:** You can use this view to send messages to your current or future users or send them a temporary password.
 
@@ -143,33 +155,37 @@ In the case of a new user, you can specify the future user's GnuPG key, to send
 
 The system will automatically generate a message for you, but it is also possible to write a custom message if you tick the check-box, but don't worry about assigning a temporary password manually, the system will do that for you, right after your custom message.
 
-*  **List Organizations:** View the organizations having a presence on this instance, with some useful informations.
+*  **List Organisations:** View the organizations having a presence on this instance, with some useful informations.
 
-*  **Add Organization:**
+*  **Add Organisation:**
 
 *   **List Roles:** List, modify or delete currently existing roles.
 
 *   **Add Role:** Create a new role group for the users of this instance, controlling their privileges to create, modify, delete and to publish events and to access certain features such as the logs or automation.
 
-*   **Administrative Tools:** Various tools, upgrade scripts that can help a site-admin run the instance.
-
-*   **Server Settings:** Set up and diagnose your MISP installation.
+*   **Server Settings & Maintenance:** Various tools, upgrade scripts that can help a site-admin run the instance & Set up and diagnose your MISP installation.
 
 *   **Jobs:** View the background jobs and their progress
 
 *   **Scheduled Tasks:** Schedule the pre-defined tasks for your instance (this currently includes export caching, server pull and server push).
 
+*   **Blacklist Event:** Link to form where you can quickly add an event to a blacklist with it's UUID.
+<!-- #Todo: Double check if blacklists and their impacts are explained at all -->
+
+*   **Manage Event Blacklists:** List of blacklisted events on MISP instance.
+
+*   **Blacklists Organisation:** Link to for where you can quickly add an organisation to a blacklist with it's UUID.
+
+<!-- #Todo: Double check if blacklists and their impacts are explained at all -->
+
+*   **Manage Org Blacklists:** List of blacklisted Organisations on this instance.
+
 ##### Audit
 
 ![Audit](figures/Audit.png)
 *   **List Logs:** View the logs of the instance.
 *   **Search Logs:** Search the logs by various attributes.
 
-##### Discussions
-
-*   **List Discussions:** List all of the discussion threads.
-*   **Start Discussion:** Create a new discussion thread.
-
 ### The left bar
 
 This bar changes based on each page-group. The blue selection shows you what page you are on.

get-your-instance/README.md

@@ -1,4 +1,3 @@
-
 ## Get your own MISP instance
 
 The intention of this chapter is to support you in getting your own MISP instance up and running.
@@ -6,18 +5,16 @@ The intention of this chapter is to support you in getting your own MISP instanc
 
 ### MISP Virtual Machine
 
-CIRCL maintains the image of a recent MISP virtual machine online.
+CIRCL maintains the image of a recent MISP virtual machine online. This VM is generated after every commit to the main MISP repository on Github.
 
-This is a very easy out of the box solution, optimized for product evaluation and to support trainings hold by CIRCL staff.
-
-The images is updated on a regular base. You should frequently re-visit the online resources to get the latest versions including bug fixes and new features.
+This is a very easy out of the box solution, optimized for product evaluation and to support trainings held by CIRCL staff.
 
 
 #### MISP VM Download
 
 The best place to get the latest version of the MISP virtual machine, as well as all the available training materials is the [MISP training materials page] [1] on the CIRCL website.
 
-If you do not remember the direct link to the MISP training materials here are the very easy to remember step you have to follow to reach the right place:
+If you do not remember the direct link to the MISP training materials here are the very easy to remember steps you have to follow to reach the right place:
 
 1. Access the [CIRCL homepage] [2]
 2. Navigate to the [Training area] [3]
@@ -35,10 +32,12 @@ In VirtualBox use the "Import Appliance..." functionality to import the virtual
 
 The instructions in this manual covers VirtualBox only. If you prefer another virtualization solution like VMWare you can find some quick instruction on the [MISP training materials page] [1].
 
+ESXi Servers have been tested too. Should work without problem but some manual changing of the ATA-Bus is needed.
+
 
 #### MISP VM Credentials
 
-The MISP image is pre-configured to be reachable on the private IP address **192.168.56.50** by SSH. The GUI is reachable by **http://192.168.56.50/**.
+The MISP image is pre-configured to be reachable on the private IP address **localhost** by SSH on port 2222. The GUI is reachable by **http://localhost:8080/**.
 
 You should have two interfaces on your VirtualBox configuration (NAT and host-only). You can also configure access to the MISP instance by doing port forwarding on the NAT interface.
 
@@ -46,26 +45,44 @@ MISP credentials:
 
 *   **GUI Admin:** admin@admin.test:admin  (it's the site admin account with full rights, feel free to create other users)
 *   **Shell/SSH:** misp : Password1234
+*   **MySQL:**     The credentials are generated during the VM generator. The details are located in ~misp/mysql.txt
+
+
+#### Networking on the VM
+
+Virtualbox has a neat feature to forward ports from your Host machine to the Guest VM.
+We forward the following ports:
+
+* **ssh** Forward from 2222 on Host -> 22 on guest
+* **http** Main WebUI - 8080 on Host -> 80 on guest
+* **https** Not in use - 8443 on Host -> 443 on guest
+* **8001** MISP Dashboard - 8001 on Host -> 8001 on guest
+* **8888** Viper Web UI - 8888 on Host -> 8888 on guest
+* **1666** misp-modules used to poll the misp-modules API - 1666 on Host -> 6666 on guest
+
+If the port is already used on your host, virtualbox will still boot and all the other ports will work.
+
+To change the port forwarding select the running VM in the UI and click on `Settings` -> `Network` -> `Advanced` -> `Port forwarding`
+
+![Overview of Network settings](figures/vbox-settings-forwarding.png)
+
+Overview of default port forwards
+
+![Overview of forwarded ports](figures/port-forwards-vbox.png)
+
+The reason that some entries have `0.0.0.0` and other are left blank is due to a virtualbox bug where traffic would not be sent to the Guest VM.
+
+:warning: VMWare users will need to connect to whatever IP the VM has on your host. There is NO port forwarding done fo r VMWare.
+
 
 #### Potential issues
 
-During life trainings we see in rare cases that some users could not reach the virtual machine over the virtual network.
-
-Some investigations discover that this always happens with user whom already had VirtualBox in use before and had already one or more **Host-only Adapter** configured in advance.
-
-The MISP image is pre-configured to use **Host-only Adapter** with the Name **vboxnet0**.
-
-![Host-only Adapter vboxnet0](figures/host-only-1.png)
-
-If this is already occupied by previous VirtualBox projects, try to attach the network adapter to the next available **Host-only** network.
-
-![Host-only Adapter vboxnet0](figures/host-only-2.png)
-
+You might have a very old VM installed and the ports are not be forwarded.
+Either configure the port forwards manually or download a new VM.
 
 
 
 [1]: https://www.circl.lu/services/misp-training-materials/ "MISP training materials page"
 [2]: https://www.circl.lu/ "CIRCL homepage"
 [3]: https://www.circl.lu/services/training/ "Training area"
-[4]: https://www.circl.lu/services/training/#misp-malware-information-sharing-platform-threat-sharing "Malware Information Sharing Platform"
-
+[4]: https://www.circl.lu/services/training/#misp-malware-information-sharing-platform-threat-sharing "Malware Information Sharing Platform"

misp-zmq/README.md

@@ -32,6 +32,13 @@ Prior to enabling it, make sure that you have the pyzmq installed by running
 
 ~~~~
 sudo pip install pyzmq
+sudo pip install redis
+~~~~
+
+If you have problems and the plugin does not start, the logfile may be helpful.
+
+~~~~
+sudo cat /var/www/MISP/app/tmp/logs/mispzmq.error.log 
 ~~~~
 
 ![ZeroMQ configuration](./figures/zmq-config.png)

publish.sh

@@ -1,10 +1,16 @@
 #!/bin/bash
+echo "1/4 html"
 gitbook build
+echo "2/4 pdf"
 gitbook pdf
+echo "3/4 epub"
 gitbook epub
+echo "4/4 mobi"
 gitbook mobi
-cp book.pdf _book
-cp book.epub _book
-cp book.mobi _book
+echo "Done"
+mv book.pdf _book
+mv book.epub _book
+mv book.mobi _book
+chmod +r _book/book.pdf _book/book.epub _book/book.mobi
 cd _book
-rsync -av . circl@cpab.circl.lu:/var/www/nwww.circl.lu/doc/misp/
+rsync -azv . circl@cpab.circl.lu:/var/www/nwww.circl.lu/doc/misp/ && rm -rf _book

quick-start/README.md

@@ -21,10 +21,11 @@ MISP default credentials:
 
 ## Tasks to do after first Start
 
-1. [Change admin password](https://misp.gitbooks.io/misp-book/content/quick-start/#password-policy)
+<!-- TODO: Consollidate all sources to misp-book -->
+1. [Change site admin password](https://misp.gitbooks.io/misp-book/content/quick-start/#password-policy)
 2. [Activate Feeds](https://www.circl.lu/doc/misp/managing-feeds/)
 3. [Setup your User](https://misp.gitbooks.io/misp-book/content/user-management/#first-run-of-the-system)
-3. [Administrate MISP](https://www.circl.lu/doc/misp/administration/)
+4. [MISP Administration](https://www.circl.lu/doc/misp/administration/)
 
 ## Password Policy 
 - [12]: Ensure that the password is at least 12 characters long
@@ -38,11 +39,25 @@ If you need a password generator use:
 - Ubuntu / Debian: [pwgen](https://linux.die.net/man/1/pwgen)
 - Website: [LastPass PW Generator](https://lastpass.com/generatepassword.php)
 - Built-in generator in Keepass* and other passwort manager
+- Built-in generator in various web browsers
 
 **All Generator tools are only possibilities without any guarantee!**
 
 <div class="pagebreak"></div>
 
+## tl;dr
+
+### Create an Event
+![Overview create an event in MISP](figures/quick_create.jpg)
+
+### Browse Past Events
+![Overview browse past evente in MISP](figures/quick_browse.jpg)
+
+### Export Events for logsearches
+![Overview export events for logsearches](figures/quick_export.jpg)
+
+<div class="pagebreak"></div>
+
 ## Create an Event
 
 ![Create an Event in MISP](figures/AddEvent.jpg)
@@ -51,7 +66,7 @@ You only have to add a few pieces of information to register your Event. Further
 
 ## Describe Event
 
-Red is totally normal. No worries. (In future releases this will potentially change to a more harmonious color)
+Red is totally normal. No worries. (In future releases this will change to a more harmonious color)
 
 ![Describe Event](figures/AddEventOK.jpg)
 

requirements/README.md

@@ -0,0 +1,28 @@
+# MISP Instance requirements
+
+<!-- toc -->
+
+## Intro
+
+There are various ways you can run a MISP instance.
+
+- Virtualized with docker/ansible/packer etc
+- VMware/Virtualbox/Xen etc
+- Dedicated hardware
+- Road warrior setups
+- Air-gapped setups
+
+Whilst there is never an ultimate answer to what specifications a system needs, we try to give an approximate answer depending on your use case.
+
+## The biggie
+
+Having millions of events with millions of attributes (indicators) will eventually result in sub-par performance.
+Ideally you have millions of attributes and thousands of events. But this also depends on how you ingest the data.
+With millions of attributes a bottleneck could be the correlation engine.
+Especially if you have many duplicates in your events. (Use the feed matrix to see if feeds are massively overlapping)
+
+### Tool assisted sizing
+
+During a hackathon [misp-sizer](https://www.misp-project.org/MISP-sizer/) was conceived. ([code](https://github.com/MISP/MISP-sizer))
+This can give you a very rough estimate and needs some more [improvements](https://github.com/MISP/MISP-sizer/issues).
+

serve.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+# Kill any existing instances of "gitbook serve"
+ps -Af | grep node | grep "gitbook serve" | awk '{print $2}' | xargs kill 2>/dev/null
+if [ "$1" = "shutdown" ]; then echo "Goodbye!"; exit 0; fi;
+
+# Launch the server
+gitbook serve . &

taxonomy/README.md

@@ -214,26 +214,6 @@ Tags can be used to:
 
 * Enrich IDS export with tags to fit your NIDS deployment.
 
-## MISP warning lists: The dilemma of false-positive
-
-- False-positive is a common issue in threat intelligence sharing.
-
-- It’s often a contextual issue:
-   - false-positive might be different per community of users sharing
-information.
-
-   - organization might have their own view on false-positive.
-
-- Based on the success of the MISP taxonomy model, we build misp-warninglists. They are lists of well-known indicators that can be
-associated to potential false positives, errors or mistakes. They are Simple JSON files.
-
-![MISP warning lists](./figures/MISPwarninglist.png)
-
-The warning lists are integrated in MISP to display an info/warning box at the event and attribute level. This can be enabled at MISP instance level. Default warning lists can be enabled or disabled like known public
-resolver, multicast IP addresses, hashes for empty values, rfc1918, TLDs or known google domains. The warning lists can be expanded or added in JSON locally or via
-pull requests (https://github.com/MISP/misp-warninglists). Warning lists can be also used for critical or core infrastructure
-warning, personally identifiable information...
-
 ## Future functionalities related to MISP taxonomies
 
 - Sighting support (thanks to NCSC-NL) is integrated in MISP allowing to auto expire IOC based on user detection.

translation/README.md

@@ -2,7 +2,7 @@
 
 ## Requirements
 
-Please read the following CakePHP documentation about i18n & l10n: https://book.cakephp.org/2.0/en/core-libraries/internationalization-and-localization.html
+Please read the following [CakePHP documentation about i18n & l10n](https://book.cakephp.org/2.0/en/core-libraries/internationalization-and-localization.html).
 
 ## Add one .md per translation effort
 

using-the-system/README.md

@@ -52,7 +52,7 @@ Keep in mind that the system searches for regular expressions in the value field
 *   **Contextual Comment:** Add a comment to the attribute. This will not be used for correlation.
 *   **Value:** The actual value of the attribute, enter data about the value based on what is valid for the chosen attribute type. For example, for an attribute of type ip-src (source IP address), 11.11.11.11 would be a valid value. For more information on types and values, [click here](../categories-and-types)
 *   **Contextual Comment:** You can add some comments to the attribute that will not be used for correlation but instead serves as purely an informational field.
-*   **For Intrusion Detection System:** This option allows the attribute to be used as an IDS signature when exporting the NIDS data, unless it is being overruled by the white-list. For more information about the whitelist, head over to the [administration](#administration) section.
+*   **For Intrusion Detection System:** This option allows the attribute to be used as an IDS signature when exporting the NIDS data, unless it is being overruled by the white-list. For more information about the white-list, head over to the [administration](#administration) section. If the IDS flag is not set, the attribute is considered as contextual information and not to be used for automatic detection.
 *   **Batch import:** If there are several attributes of the same type to enter (such as a list of IP addresses, it is possible to enter them all into the same value-field, separated by a line break between each line. This will allow the system to create separate lines for the each attribute.
 
 
@@ -329,7 +329,7 @@ The last option is a checkbox that restricts all of the results to attributes th
 ## Updating and modifying events and attributes:
 
 Every event and attribute can easily be edited. First of all it is important to find the event or attribute that is to be edited, using any of the methods mentioned in the section on [browsing past events](#browsing_events).
-Once it is found, the edit button (whether it be under actions when events/attributes get listed or simply on the event view) will bring up the same screen as what is used to create the entry of the same type (for an event it would be the event screen as [seen here](#Creating an event), for an attribute the attribute screen as [described here](#add-attributes-to-the-event)).
+Once it is found, the edit button (whether it be under actions when events/attributes get listed or simply on the event view) will bring up the same screen as what is used to create the entry of the same type (for an event it would be the event screen as [seen here](#Creating an event), for an attribute the attribute screen as [described here](#add-attributes-to-the-event)). You can also simply double-click on the event you wish to edit and enter the edit mode. 
 Keep in mind that editing any event (either directly or indirectly through an attribute) will unpublish it, meaning that you'll have to publish it (through the event view) again once you are done.
 
 ## Tagging:

warninglists/README.md

@@ -3,3 +3,22 @@ MISP warninglists are lists of well-known indicators that can be associated to p
 There is a Python module available to work with warninglists in a Pythonic way called [PyMISPWarningLists](https://github.com/MISP/PyMISPWarningLists).
 [MISP warninglists GitHub Repo](https://github.com/MISP/misp-warninglists)
 
+## MISP warning lists: The dilemma of false-positive
+
+- False-positive is a common issue in threat intelligence sharing.
+
+- It’s often a contextual issue:
+   - false-positive might be different per community of users sharing
+information.
+
+   - organization might have their own view on false-positive.
+
+- Based on the success of the MISP taxonomy model, we build misp-warninglists. They are lists of well-known indicators that can be
+associated to potential false positives, errors or mistakes. They are Simple JSON files.
+
+![MISP warning lists](./figures/MISPwarninglist.png)
+
+The warning lists are integrated in MISP to display an info/warning box at the event and attribute level. This can be enabled at MISP instance level. Default warning lists can be enabled or disabled like known public
+resolver, multicast IP addresses, hashes for empty values, rfc1918, TLDs or known google domains. The warning lists can be expanded or added in JSON locally or via
+pull requests (https://github.com/MISP/misp-warninglists). Warning lists can be also used for critical or core infrastructure
+warning, personally identifiable information...