Merge pull request #128 from SteveClement/master

chg: [doc] Sections updated: administration, GLOSSARY, USAGE, Book conventions, general layout update, get your instance

GLOSSARY.md

@@ -1,6 +1,6 @@
 ## MISP Glossary
 This glossary is meant as a quick lookup document in case of any need of clarification of any threat sharing, threat-intel lingo.
-Be careful when adding terms to the glossary. Adding a generic term like: *MISP* will prevent terms like *MISP noticelist* to be addded. As a matter of definition please use the singular of for any terms.
+Be careful when adding terms to the glossary. Adding a generic term like: *MISP* will prevent terms like *MISP noticelist* to be addded. As a matter of definition please use the singular for any terms.
 In case you use any CCBYSA licensed content, or other pieces that are subject to licensing, make sure to add it as a by-line at the end of the mention.
 
 ## API

USAGE.md

@@ -5,6 +5,16 @@ Install notes
 :warning: Make sure to be in the *misp-book* repository directory for the *npm magic*.
 Also: The *npm* plugin *autocover* is broken. It pulls an incompatible *canvas* module version. Thus patched repository used (forked from original)
 Finally: You will get a few errors on Ubuntu 18.04 which you can ignore for now. In the rather near future we need to think about an alternative as gitbook glides towards obsoletion and security risk.
+Reason for concern:
+```
+npm WARN deprecated ignore@3.1.2: several bugs fixed in v3.2.1
+npm WARN deprecated tough-cookie@2.2.2: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
+npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
+npm WARN deprecated hoek@2.16.3: The major version is no longer supported. Please update to 4.x or newer
+npm WARN deprecated datauri@0.2.1: Potential REDOS vulnerability removed in v1.1.0
+npm WARN deprecated coffee-script@1.12.7: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
+```
+
 
 Tested on: *Ubuntu 18.04 LTS* *Debian 9.5/sid/testing*
 [Terminal Recording of npm install lines on Ubuntu 18.04](https://asciinema.org/a/84JZMuGu2QlFH59q6mK8jbdQS)

administration/README.md

@@ -7,19 +7,19 @@
 * [Roles](#roles)
 * [Tools](#tools)
 * [Server Settings](#server-settings)
-* Jobs
-* Scheduled Tasks
+* [Jobs](#jobs)
+* [Scheduled Tasks](#scheduled-tasks)
 
 
 > [warning] This page is under modification for updating the content. Current status:
 
-- [x] Users
-- [x] Organisations
-- [x] Roles
-- [x] Tools
-- [ ] Server Settings
-- [ ] Jobs
-- [ ] Scheduled Tasks
+- [x] Users - Reviewed/Updated on: ?
+- [x] Organisations - Reviewed/Updated on: ?
+- [x] Roles - Reviewed/Updated on: ?
+- [x] Tools - Reviewed/Updated on: ?
+- [ ] Server Settings - Reviewed/Updated on: ?
+- [ ] Jobs aka. Background processing - Reviewed/Updated on: ?
+- [ ] Scheduled Tasks aka. Background processing - Reviewed/Updated on: ?
 
 - - -
 
@@ -329,13 +329,22 @@ If enabled, MISP can delegate a lot of the time intensive tasks to the backgroun
 #### Command Line Tools for the Background Workers
 
 The background workers are powered by [CakeResque](https://github.com/kamisama/Cake-Resque), so all of the CakeResque commands work.
-To start all of the workers needed by MISP go to your `/var/www/MISP/app/Console/worker` (assuming a standard installation path) and execute start.sh.
-To interact with the workers, here is a list of useful commands. Go to your `/var/www/MISP/app/Console` (assuming a standard installation path) and execute one of the following commands as a parameter to `./cake CakeResque.CakeResque` (for example: `./cake CakeResque.CakeResque tail`):
+To start all of the workers needed by MISP go to your `/var/www/MISP/app/Console/worker` (assuming a standard installation path) and execute `start.sh`.
+To interact with the workers, here is a list of useful commands. Go to your `/var/www/MISP/app/Console` (assuming a standard installation path) and execute one of the following commands as a parameter to `./cake CakeResque` (for example: `./cake CakeResque tail`):
 
-*   **tail**: tail the various log files that CakeResque creates, just choose the one from the list that you are interested in.
-*   **cleanup**: terminate the job that a worker is working on with immediate effect. You will be presented with a choice of workers to choose from when executing this command.
-*   **clear**: Clear the queue of a worker immediately.
-*   **stats**: Display some statistics about your workers including the count of successful and failed jobs.
+*   **start**:           Start a new worker.
+*   **startscheduler**:  Start a new scheduler worker.
+*   **stop**:            Stop a worker.
+*   **pause**:           Pause a worker.
+*   **resume**:          Resume a paused worker.
+*   **cleanup**:         Terminate the job that a worker is working on with immediate effect. You will be presented with a choice of workers to choose from when executing this command.
+*   **restart**:         Stop all Resque workers, and start a new one.
+*   **clear**:           Clear all jobs inside a queue
+*   **reset**:           Reset CakeResque internal worker's saved status
+*   **stats**:           Display some statistics about your workers including the count of successful and failed jobs.
+*   **tail**:            Tail the various (workers) log files that CakeResque creates, just choose the one from the list that you are interested in.
+*   **track**:           Track a job status.
+*   **load**:            Load a set of predefined workers.
 
 The other commands should not be required, instead of starting / stopping or restarting workers use the supplied start.sh (it stops all workers and starts them all up again). For further instructions on how to use the console commands for the workers, visit the [CakeResque list of commands](http://cakeresque.kamisama.me/commands#cleanup).
 
@@ -585,3 +594,33 @@ An example of error message:
 ```
 Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'Task.job_id' in 'field list'
 ```
+
+### Jobs
+
+The Jobs tab gives you an overview on any currently running jobs or jobs that were previously completed and their status.
+
+![Running Jobs](figures/jobs-running.png)
+
+Typically this is one of the places you would turn to even some background process might not complete as expected to get an indication on any issues related to user initiated Jobs.
+
+For ease of use, you can filter the Jobs by 'All', 'Default', 'Email', 'Cache'
+##### Todo: Explain differences Default, Email, Cache
+
+You can also purge the entries, either only by completed status or purge all.
+This is not automated and needs to be done manually.
+
+### Scheduled Tasks
+
+Straight from the UI:
+
+"""
+Here you can schedule pre-defined tasks that will be executed every x hours. You can alter the date and time of the next scheduled execution and the frequency at which it will be repeated (expressed in hours). If you set the frequency to 0 then the task will not be repeated. To change and of the above mentioned settings just click on the appropriate field and hit update all when you are done editing the scheduled tasks.
+
+Warning: Scheduled tasks come with a lot of caveats and little in regards of customisations / granularity. You can instead simply create cron jobs out of the console commands as described here: Automating certain console tasks
+"""
+
+The task scheduler is a sub-par component to enable minimal functionality in terms of automating certain MISP tasks.
+If you have a dedicated and concious MISP Site Admin she can keep an eye on the Scheduler to make sure everything runs smoothly.
+
+For better performance please use a real scheduler like your systems' crontab.
+As a rule of thumb: If you can click on it, MISP can automate it.

book-convention/README.md

@@ -1,3 +1,9 @@
+<!-- This is a comment.
+If you plan on contributing to misp-book, welcome and enjoy.
+In case of any and all questions, feel free to join our gitter:
+https://gitter.im/MISP/MISP
+For Aiur! -->
+
 ---
 description: Convention Used in MISP-Book
 ---
@@ -8,6 +14,33 @@ description: Convention Used in MISP-Book
 
 * Used for variable, function or menu names in MISP.
 
+## Language
+
+The language in this book is american english.
+All the screenshots and examples are in english.
+
+## CoC
+
+The same code of conduct applies to this book as for the main MISP project.
+As a book can some times be considered the inadvertent sould of a piece of software, please take good care and consideration of our `Code of Conduct`. The CoC [can be read here](https://github.com/MISP/MISP/blob/2.4/code_of_conduct.md).
+
+## Example install
+
+The examples and screenshots provided in this book have been created with the MISP Autogenerated VM.
+
+To get a copy of the latest VM [click here](https://www.circl.lu/misp-images/latest/)
+
+
+## MISP Instance
+
+In general when talking about a network of inter-connected MISP servers, each server is a MISP instance. Whilst we have no strong feelings towards anyones naming schemes, as a rule of thumb try to have a scheme that makes everyday use easy when analysts need to talk about remote MISP instances.
+<!--
+ToDo: Be more specific give some naming convention examples.
+-->
+
+The hostname used for the instance in this book is `misp.local` and we will henceforth refer to it either by name or as `local MISP instance`.
+
+
 ## Example Organisations
 
 As MISP is a platform to support information sharing, example organisations are often used within this book.
@@ -21,5 +54,15 @@ The following two organisations are regularly used as example:
 
 Starting from MISP 2.4.71, the example organisations with the above mentioned UUID are **black-listed** to avoid  
 large distribution of sample events while testing a MISP instance. If you want to test your distribution, the  
-sample organisation black-listing can be removed in `Administration` under `Manage Org Blacklists`.
+sample organisation black-listing can be removed in `Administration`/`Manage Org Blacklists`.
+
+## Example IOCs
+
+As with the example organisations, we want to make this book as useful as possible by using real life examples.
+
+The following IOC examples have been used:
+
+* [Sirefef](https://www.misp-project.org/galaxy.html#_zeroaccess) (aka ZeroAccess) Sample Event ID: #31337
+* [WannaCry](https://www.misp-project.org/galaxy.html#_wannacry) Sample Event ID: #42
+* [Dridex](https://www.misp-project.org/galaxy.html#_dridex) Sample Event ID: #23
 

general-layout/README.md

@@ -1,5 +1,7 @@
 <!-- Nothing else matters -->
 
+<!-- ToDo: Think about other default user roles that need to be considered and documented or at least hinted too. like: the role Read-only and user have these differences, if any -->
+
 ## General Layout
 
 ### The top bar
@@ -11,9 +13,13 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Home button:** This button will return you to the start screen of the application, which is the event index page (more about this later).
 *   **Event Actions:** All the malware data entered into MISP is made up of an event object that is described by its connected attributes. The Event actions menu gives access to all the functionality that has to do with the creation, modification, deletion, publishing, searching and listing of events and attributes.
+*   **Galaxies:** Shortcut to the list of [MISP Galaxies](../galaxy/) on the MISP instance.
 *   **Input Filters:** Input filters alter what and how data can be entered into this instance. Apart from the basic validation of attribute entry by type, it is possible for the site administrators to define regular expression replacements and blacklists for certain values in addition to blocking certain values from being exportable. Users can view these replacement and blacklist rules here whilst administrator can alter them.
 *   **Global Actions:** This menu gives you access to information about MISP and this instance. You can view and edit your own profile, view the manual, read the news or the terms of use again, see a list of the active organizations on this instance and a histogram of their contributions by attribute type.
-*   **Discussions:** Link to the discussion threads.
+*   **MISP:** Simple link to your BASEURL
+*   **Steve:** Name (Auto generated from Mail address) of current logged in user
+*   **Envelope:** Link to User Dashboard where you can consult some of your notifications and changes since last visit. Like some of the proposals recieved for your organisation.
+*   **Log out:** The Log out button to end your session immediatly.
 
 #### Admin Menu Bar
 ![Some additional buttons that will appear on top of these when a view provides it.](figures/MenuBarAdmin.jpg)
@@ -21,6 +27,8 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Event Actions:** ibidem
 
+*   **Galaxies:** You can aditionally update the Galaxies.
+
 *   **Input Filters:** Ibidem
 
 *   **Global Actions:** Ibidem
@@ -31,9 +39,14 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Audit:** If you have audit permissions, you can view the logs for your organization (or for site admins for the entire system) here or even search the logs if you are interested in something specific.
 
-*   **Proposal Notifications:** This shows how many proposals your organization has received and across how many events they are spread out. Clicking this will take you to the list of proposals.
+*   **MISP:** ibidem
+
+*   **Admin:** ibidem
+
+*   **Envelope:** Link to User Dashboard where you can consult some of your notifications and changes since last visit. Like some of the proposals recieved for your organisation.
+
+*   **Log out:** The Log out button to end your session immediatly.
 
-*   **Log out:** Logs you out of the system.
 
 ### A list of the contents of each of the above drop-down menus
 
@@ -48,6 +61,8 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Search Attributes:** You can set search terms for a filtered attribute index view here.
 
+*   **REST client** MISP Online REST client where you can make calls directly to the API via a Web UI.
+
 *   **View Proposals:** Shows a list of all proposals that you are eligible to see.
 
 *   **Events with proposals:** Shows all of the events created by your organisation that has pending proposals.
@@ -56,6 +71,8 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Add Tag:** Create a new tag.
 
+*   **List Taxonomies:** List all of the taxonomies installed on the MISP instance. This is also the place to activate the taxonomies as a Org Admin/Site Admin.
+
 *   **List Templates:** List all of the templates created by users with template creation rights on this instance.
 
 *   **Add Template:** Create a new template.
@@ -72,8 +89,9 @@ This menu contains all of the main functions of the site as a series of dropdown
 
 *   **Signature Whitelist:** You can view the whitelist rules, which contains the values that are blocked from being used for exports and automation on this instance. Site administrators have access to editing this list.
 
-*  **List Warninglists:**
-MISP warninglists are lists of well-known indicators that can be associated to potential false positives, errors or mistakes. The warning lists are integrated in MISP to display an info/warning box at the event and attribute level.
+*  **List Warninglists:** MISP warninglists are lists of well-known indicators that can be associated to potential false positives, errors or mistakes. The warning lists are integrated in MISP to display an info/warning box at the event and attribute level.
+
+*  **List Noticelists:** MISP noticelists are lists of #Todo: Double check description from repo!!!
 
 ##### Global Actions
 
@@ -86,8 +104,6 @@ MISP warninglists are lists of well-known indicators that can be associated to p
 
 *   **Dashboard:** allow you to see your Notifications of Proposals, Events with proposals and Delegation request. Your can see the last changes since your last visit, as Events updates and Events publications.
 
-*   **Members List:** View the number of users per organization and get some statistics about the currently stored attributes.
-
 *   **Organizations:** View the organizations having a presence on this instance, with some useful informations as contact's name.
 
 *   **Role Permissions:** You can view the role permissions here.
@@ -98,28 +114,24 @@ MISP warninglists are lists of well-known indicators that can be associated to p
 
 *   **User Guide:** A link to this user guide.
 
+*   **Categories & Types:** Quick overview of Attribute Categories and Type. e.g: md5 -> Payload delivery, Artifacts dropped, Payload installation, External analysis
+
+*   **Terms & Conditions:** General terms and conditions which can be configured in Administration -> Server Settings -> MISP Settings: MISP.terms_file . From the UI: "The filename of the terms and conditions file. Make sure that the file is located in your MISP/app/files/terms directory"
+
 *   **Statistics:** View a series of statistics about the users and the data on this instance.
 
+*   **List Discussions:** List threads of discussions created on the MISP instance by the organisations connected to this local community.
+
+*   **Start Discussion:** Create a new discussion thread. 
+<!-- #ToDo: Make a reference to best practices on how to use this form of messaging. Sense/non-sense etc... -->
 
 ##### Sync Actions
 
-
 ![Sync Actions](figures/SyncActions.png)
 
 *   **List Servers:** Connect your MISP instance to other instances, or view and modify the currently established connections.
-<!-- Fix provided by elhoim -->
-It may be that you have an Error Message in the page (if you enabled debug or site_admin_debug settings). An example of error message:
-
-![Error message](figures/pb-list-server.png)
-
-An easy first step to make most of them go away is to use the clean cache feature on the server settings menu, diagnostics tab.
-
-![cleanscript](figures/cleanscript1.png)
-
-You must then scroll down the page.
-
-![cleanscript](figures/cleanscript2.png)
 
+<!-- Include example of error message/stack trace differences. -->
 
 *  **List Feeds:** Follow the RSS feeds of other organization or CERTs worldwide.  
 
@@ -129,7 +141,7 @@ You must then scroll down the page.
 
 *   **List Users:** View, modify or delete the currently registered users.
 
-*   **New User:** Create an account for a new user for your organisation. Site administrators can create users for any organisation.
+*   **Add User:** Create an account for a new user for your organisation. Site administrators can create users for any organisation.
 
 *  **Contact Users:** You can use this view to send messages to your current or future users or send them a temporary password.
 
@@ -143,33 +155,37 @@ In the case of a new user, you can specify the future user's GnuPG key, to send
 
 The system will automatically generate a message for you, but it is also possible to write a custom message if you tick the check-box, but don't worry about assigning a temporary password manually, the system will do that for you, right after your custom message.
 
-*  **List Organizations:** View the organizations having a presence on this instance, with some useful informations.
+*  **List Organisations:** View the organizations having a presence on this instance, with some useful informations.
 
-*  **Add Organization:**
+*  **Add Organisation:**
 
 *   **List Roles:** List, modify or delete currently existing roles.
 
 *   **Add Role:** Create a new role group for the users of this instance, controlling their privileges to create, modify, delete and to publish events and to access certain features such as the logs or automation.
 
-*   **Administrative Tools:** Various tools, upgrade scripts that can help a site-admin run the instance.
-
-*   **Server Settings:** Set up and diagnose your MISP installation.
+*   **Server Settings & Maintenance:** Various tools, upgrade scripts that can help a site-admin run the instance & Set up and diagnose your MISP installation.
 
 *   **Jobs:** View the background jobs and their progress
 
 *   **Scheduled Tasks:** Schedule the pre-defined tasks for your instance (this currently includes export caching, server pull and server push).
 
+*   **Blacklist Event:** Link to form where you can quickly add an event to a blacklist with it's UUID.
+<!-- #Todo: Double check if blacklists and their impacts are explained at all -->
+
+*   **Manage Event Blacklists:** List of blacklisted events on MISP instance.
+
+*   **Blacklists Organisation:** Link to for where you can quickly add an organisation to a blacklist with it's UUID.
+
+<!-- #Todo: Double check if blacklists and their impacts are explained at all -->
+
+*   **Manage Org Blacklists:** List of blacklisted Organisations on this instance.
+
 ##### Audit
 
 ![Audit](figures/Audit.png)
 *   **List Logs:** View the logs of the instance.
 *   **Search Logs:** Search the logs by various attributes.
 
-##### Discussions
-
-*   **List Discussions:** List all of the discussion threads.
-*   **Start Discussion:** Create a new discussion thread.
-
 ### The left bar
 
 This bar changes based on each page-group. The blue selection shows you what page you are on.

get-your-instance/README.md

@@ -1,4 +1,3 @@
-
 ## Get your own MISP instance
 
 The intention of this chapter is to support you in getting your own MISP instance up and running.
@@ -6,18 +5,16 @@ The intention of this chapter is to support you in getting your own MISP instanc
 
 ### MISP Virtual Machine
 
-CIRCL maintains the image of a recent MISP virtual machine online.
+CIRCL maintains the image of a recent MISP virtual machine online. This VM is generated after every commit to the main MISP repository on Github.
 
-This is a very easy out of the box solution, optimized for product evaluation and to support trainings hold by CIRCL staff.
-
-The images is updated on a regular base. You should frequently re-visit the online resources to get the latest versions including bug fixes and new features.
+This is a very easy out of the box solution, optimized for product evaluation and to support trainings held by CIRCL staff.
 
 
 #### MISP VM Download
 
 The best place to get the latest version of the MISP virtual machine, as well as all the available training materials is the [MISP training materials page] [1] on the CIRCL website.
 
-If you do not remember the direct link to the MISP training materials here are the very easy to remember step you have to follow to reach the right place:
+If you do not remember the direct link to the MISP training materials here are the very easy to remember steps you have to follow to reach the right place:
 
 1. Access the [CIRCL homepage] [2]
 2. Navigate to the [Training area] [3]
@@ -35,10 +32,12 @@ In VirtualBox use the "Import Appliance..." functionality to import the virtual
 
 The instructions in this manual covers VirtualBox only. If you prefer another virtualization solution like VMWare you can find some quick instruction on the [MISP training materials page] [1].
 
+ESXi Servers have been tested too. Should work without problem but some manual changing of the ATA-Bus is needed.
+
 
 #### MISP VM Credentials
 
-The MISP image is pre-configured to be reachable on the private IP address **192.168.56.50** by SSH. The GUI is reachable by **http://192.168.56.50/**.
+The MISP image is pre-configured to be reachable on the private IP address **localhost** by SSH on port 2222. The GUI is reachable by **http://localhost:8080/**.
 
 You should have two interfaces on your VirtualBox configuration (NAT and host-only). You can also configure access to the MISP instance by doing port forwarding on the NAT interface.
 
@@ -46,26 +45,44 @@ MISP credentials:
 
 *   **GUI Admin:** admin@admin.test:admin  (it's the site admin account with full rights, feel free to create other users)
 *   **Shell/SSH:** misp : Password1234
+*   **MySQL:**     The credentials are generated during the VM generator. The details are located in ~misp/mysql.txt
+
+
+#### Networking on the VM
+
+Virtualbox has a neat feature to forward ports from your Host machine to the Guest VM.
+We forward the following ports:
+
+* **ssh** Forward from 2222 on Host -> 22 on guest
+* **http** Main WebUI - 8080 on Host -> 80 on guest
+* **https** Not in use - 8443 on Host -> 443 on guest
+* **8001** MISP Dashboard - 8001 on Host -> 8001 on guest
+* **8888** Viper Web UI - 8888 on Host -> 8888 on guest
+* **1666** misp-modules used to poll the misp-modules API - 1666 on Host -> 6666 on guest
+
+If the port is already used on your host, virtualbox will still boot and all the other ports will work.
+
+To change the port forwarding select the running VM in the UI and click on `Settings` -> `Network` -> `Advanced` -> `Port forwarding`
+
+![Overview of Network settings](figures/vbox-settings-forwarding.png)
+
+Overview of default port forwards
+
+![Overview of forwarded ports](figures/port-forwards-vbox.png)
+
+The reason that some entries have `0.0.0.0` and other are left blank is due to a virtualbox bug where traffic would not be sent to the Guest VM.
+
+:warning: VMWare users will need to connect to whatever IP the VM has on your host. There is NO port forwarding done fo r VMWare.
+
 
 #### Potential issues
 
-During life trainings we see in rare cases that some users could not reach the virtual machine over the virtual network.
-
-Some investigations discover that this always happens with user whom already had VirtualBox in use before and had already one or more **Host-only Adapter** configured in advance.
-
-The MISP image is pre-configured to use **Host-only Adapter** with the Name **vboxnet0**.
-
-![Host-only Adapter vboxnet0](figures/host-only-1.png)
-
-If this is already occupied by previous VirtualBox projects, try to attach the network adapter to the next available **Host-only** network.
-
-![Host-only Adapter vboxnet0](figures/host-only-2.png)
-
+You might have a very old VM installed and the ports are not be forwarded.
+Either configure the port forwards manually or download a new VM.
 
 
 
 [1]: https://www.circl.lu/services/misp-training-materials/ "MISP training materials page"
 [2]: https://www.circl.lu/ "CIRCL homepage"
 [3]: https://www.circl.lu/services/training/ "Training area"
-[4]: https://www.circl.lu/services/training/#misp-malware-information-sharing-platform-threat-sharing "Malware Information Sharing Platform"
-
+[4]: https://www.circl.lu/services/training/#misp-malware-information-sharing-platform-threat-sharing "Malware Information Sharing Platform"