Merge branch 'master' of https://github.com/MISP/misp-book

Conflicts:
	using-the-system/README.md

.travis.yml

@@ -10,6 +10,8 @@ install:
     - npm install gitbook-plugin-autocover
     - npm install gitbook-plugin-github
     - npm install gitbook-plugin-toc
+    - npm install gitbook-plugin-anchors
 
 script:
+    - gitbook install
     - gitbook build

GLOSSARY.md

@@ -1,5 +1,5 @@
 # MISP
-Malware Information Sharing Platform
+Malware Information Sharing Platform and Threat Sharing
 
 # IOC
 Indicator of compromise (IOC or IoC) is an artifact observed on a network or in an operating system or information channel that could reference an intrusion or a reference to a technique used by an attacker.

README.md

@@ -17,6 +17,10 @@ The MISP user guide is a collaborative effort between all the contributors to [M
 - Cthulhu Solutions
 - [CERT-EU](https://cert.europa.eu)
 
+## Contributing
+
+We welcome contributions to the MISP book. If you want to contribute, clone the [misp-book](https://github.com/MISP/misp-book) repository and pull a request with your changes.
+
 ## License
 
 The MISP user guide is dual-licensed under [GNU Affero General Public License version 3](http://www.gnu.org/licenses/agpl-3.0.html) and [CC-BY-SA 4.0 international](https://creativecommons.org/licenses/by-sa/4.0/).
@@ -24,7 +28,7 @@ The MISP user guide is dual-licensed under [GNU Affero General Public License ve
 * Copyright (C) 2012 Christophe Vandeplas
 * Copyright (C) 2012 Belgian Defence
 * Copyright (C) 2012 NATO / NCIRC
-* Copyright (C) 2013-2015 Andras Iklody
-* Copyright (C) 2015 Alexandre Dulaunoy
-* Copyright (C) 2014-2015 CIRCL - Computer Incident Response Center Luxembourg
+* Copyright (C) 2013-2016 Andras Iklody
+* Copyright (C) 2015-2016 Alexandre Dulaunoy
+* Copyright (C) 2014-2016 CIRCL - Computer Incident Response Center Luxembourg
 

SUMMARY.md

@@ -7,5 +7,9 @@
 * [Using the System](using-the-system/README.md)
 * [Administration](administration/README.md)
 * [Automation and MISP API](automation/README.md)
+* [PyMISP - Python Library to access MISP](pymisp/README.md)
+* [Create an event based on a report](create-event-report/README.md)
+* [Taxonomies](taxonomy/README.md)
+* [Galaxies](galaxy/README.md) - in progress
 * [Categories and Types](categories-and-types/README.md)
 * [Appendices](appendices/README.md)

USAGE

@@ -6,6 +6,8 @@ Install notes
 npm install gitbook-plugin-autocover
 npm install gitbook-plugin-github
 npm install gitbook-plugin-toc
+npm install gitbook-plugin-anchors
+gitbook install
 ~~~
 
 Usage

administration/README.md

@@ -2,9 +2,210 @@
 
 ## Administration
 
+* [Users](#users)
+* [Organisations](#organisations)
+* [Roles](#roles)
+* [Tools](#tools)
+* [Server Settings](#server-settings)
+* Jobs
+* Scheduled Tasks
+
+
+:warning: This page is under modification for updating the content. Current status:
+
+- [x] Users
+- [x] Organisations
+- [x] Roles
+- [x] Tools
+- [ ] Server Settings
+- [ ] Jobs
+- [ ] Scheduled Tasks 
+
+- - -
+
+### Users
+As an admin, you can set up new accounts for users, edit the profiles of users, delete them, or just have a look at all the viewers’ profiles. Organisation admins are restricted to executing the same actions on their organisation’s users only.
+
+#### Adding a new user:
+
+To add a new user, click on the Add User button in the administration menu to the left and fill out the following fields in the view that is loaded:
+
+![Fill this form out to add a new user. Keep in mind that the drop-down menu titled Role controls the privileges the user will have.](figures/add_user.png)
+
+*   **Email:** The user's e-mail address, this will be used as his/her login name and as an address to send all the automatic e-mails and e-mails sent by contacting the user as the reporter of an event.
+*   **Set password:** Tick the box if you want to define a temporary password for the user. If you don't, you'll should use the action button 'reset password' on 'List Users' view for generating one and send it by email to the user.
+*   **Password:** *This textbox is displayed only when 'Set password' is ticked.* A Temporary password for the user that he/she should change after the first login. Make sure that it is at least 6 characters long, includes a digit or a special character and contains at least one upper-case and at least one lower-case character.
+*   **Confirm Password:** *This textbox is displayed only when 'Set password' is ticked.* This should be an exact copy of the Password field.
+*   **Organisation:** A drop-down list allows you to choose an organisation for the user. To learn more about organisation, [click here](#organisation).
+*   **Roles:** A drop-down list allows you to choose a role-group that the user should belong to. Roles define the privileges of the user. To learn more about roles, [click here](#managing-the-roles).
+*   **Authkey:** This is assigned automatically and is the unique authentication key of the user (he/she will be able to reset this and receive a new key). It is used for exports and for connecting one server to another, but it requires the user to be assigned to a role that has auth permission enabled.
+*   **NIDS Sid:** ID of network intrusion detection systems.
+*   **Sync user for:** Use this option for granted the user the right to synchronize the event between MISP server. This option is available for admin, Org Admin and Sync user role.
+*   **Gpgkey:** The key used for encrypting e-mails sent through the system.
+*   **Fetch GPG key:** Fetch GPG public key.
+*   **Receive alerts when events are published:** This option will subscribe the new user to automatically generated e-mails whenever an event is published.
+*   **Receive alerts from "contact reporter" requests:** This option will subscribe the new user to e-mails that are generated when another user tries to get in touch with an event's reporting organisation that matches that of the new user.
+*   **Disable this user account:** Tick it if you want to disable this user account.
+
+#### Listing all users:
+
+To list all current users of the system, just click on List Users under the administration menu to the left. A view will be loaded with a list of all users and the following columns of information:
+
+![View, Edit or Delete a user using the action buttons to the right.](figures/list_users.png)
+
+*   **Id:** The user's automatically assigned ID number.
+*   **Org:** The organisation that the user belongs to.
+*   **Email:** The e-mail address (and login name) of the user.
+*   **Authkey:** Unique authentication key of the user. 
+*   **Autoalert:** Shows whether the user has subscribed to auto-alerts and is always receiving the mass-emails regarding newly published events that he/she is eligible for.
+*   **Contactalert:** Shows whether the user has the subscription to contact reporter e-mails directed at his/her organisation turned on or off.
+*   **Gpgkey:** Shows whether the user has entered a Gpgkey yet.
+*   **Nids Sid:** Shows the currently assigned NIDS ID.
+*   **Termsaccepted:** This flag indicates whether the user has accepted the terms of use or not.
+*   **Last login:** Date of last login.
+*   **Disabled:** Show the user status. Enabled or disabled.
+*   **Action Buttons:** There are 4 options available: reset the password, edit the user, delete the user or display user's information. These options are also available on the left menu.
+	*    **Reset Password:** Use this action for reseting password. If you have created a new user without password, tick the 'First time registration' checkbox for sending a welcome message. Otherwise a  reset password message will be sent.
+![Reset password.](figures/reset_pwd.png)
+	*    **Edit the user:** Same options of create user's view. Few options are only available here:
+		*   **Terms accepted:** Indicates whether the user has accepted the terms of use already or not.
+		*   **Change Password:** Setting this flag will require the user to change password after the next login.
+		*   **Reset Auth Key:** Use this link for generate a new AuthKey.
+![Edit user.](figures/edit_users.png)
+	*    **Delete the user:** If you want to delete a user.
+![delete user.](figures/delete_user.png)
+	*    **Display the user:** Display all user's information.<br />
+![display user.](figures/display_user.png)
+
+#### Contacting a user:
+
+Site admins can use the "Contact users" feature to send all or individual user an e-mail. Users that have a PGP key set will receive their e-mails encrypted. When clicking this button on the left, you'll be presented with a form that allows you to specify the type of the e-mail, who it should reach and what the content is using the following options:
+
+![Contact your users here.](figures/contact.png)
+
+*   **Action:** This defines the type of the e-mail, which can be a custom message or a password reset. Password resets automatically include a new temporary password at the bottom of the message and will automatically change the user's password accordingly.
+*   **Subject:** In the case of a custom e-mail, you can enter a subject line here.
+*   **Recipient:** The recipient toggle lets you contact all your users, a single user (which creates a second drop-down list with all the e-mail addresses of the users) and potential future users (which opens up a text field for the e-mail address and a text area field for a PGP public key).
+*   **Custom message checkbox:** This is available for password resets or for welcome message, you can either write your own message (which will be appended with a temporary key and the signature), or let the system generate one automatically.
+
+Keep in mind that all e-mails sent through this system will, in addition to your own message, will be signed in the name of the instance's host organisation's support team, will include the e-mail address of the instance's support (if the contact field is set in the bootstrap file), and will include the instance's PGP signature for users that have a PGP key set (and thus are eligible for an encrypted e-mail).
+
+- - -
+
+### Organisations
+
+Each users belongs to an organisation. As admin, you can manage these organisations.
+
+#### Adding a new organisation:
+
+To add a new organisation, click on the Add Organisation button in the administration menu to the left and fill out the following fields in the view that is loaded:
+
+![Fill this form out to add a new organisation.](figures/add_org.png)
+
+*   **Local organisation:** If the organisation should have access to this instance, tick this checkbox. If you would only like to add a known external organisation for inclusion in sharing groups, uncheck it.
+*   **Organisation Identifier:** Name your organisation. If you want to add a picture, you should add a file on the webserver using the 'Server Settings menu'. Picture should have the same name. To learn more about server settings menu, [click here](#server-settings).
+*   **Uuid:** Unique identifier. If you want to share organisation between MISP multi-instance, use the same Uuid.
+*   **A brief description of the organisation:** A word for describing the organisation.
+*   **Nationality:** A drop-down list for selecting the country of organisation.
+*   **Sector:** Define the sector of organisation (financial, transport, telecom...)
+*   **Type of organisation:** Define the type of the organisation.
+*   **Contacts:** You can add some contact details for the organisation.
+
+#### Listing all organisation:
+
+To list all current organisation of the system, just click on List Organisations under the administration menu to the left. There are 3 tabs in this view for filtering the local organisations, remote organisations and both. Default view display local organisations. For all views the following columns of information are available:
+
+![List of organisations.](figures/list_org.png)
+
+*   **Id:** The organisation's automatically assigned ID number.
+*   **Logo:** Picture of the organisation.
+*   **Name:** Name of the organisation.
+*   **Uuid:** Unique identifier of orgnisation. Share this Uuid for using it between MISP's multi-instance.
+*   **Description:** Description of the organisation.
+*   **Nationality:** Country of the organisation.
+*   **Sector:** Sector defined for the organisation.
+*   **Type:** Type of organisation.
+*   **Contacts:** Contacts of orgnisation.
+*   **Added by:** Login of the user which have added the organisation
+*   **Local:** Flag defined if the organisation is local or remote.
+*   **Actions:**  There are 3 options available: edit, delete or display organisation's information. These options are also available on the left menu when you are on the display view.
+	*    **Edit Organisation:** Same options of create organisation's view.
+![Edit organisation.](figures/edit_org.png)	
+	*    **Delete Organisation:** Use this option for deleting organisation.<br />
+![Delete organisation.](figures/delete_org.png)	 
+	*    **View Organisation:** Use this option for displaying information about organisation selected. In this view, you can display the user belongs to this organisation and events published by organisation.
+![View organisation.](figures/view_org.png)	
+
+#### Merge organisations:
+Merge Organisation menu is available only in the view organisation, under the left menu. Merge one organisation to another will transfer all users and data from one to another. On the left the organisation to merge, on the right the target one.
+
+![Merge organisations.](figures/merge_org.png)
+
+- - -
+
+### Roles
+
+Privileges are assigned to users by assigning them to rule groups, which use one of four options determining what they can do with events and four additional privilege elevating settings. The four options for event manipulation are: Read Only, Manage My Own Events, Manage Organisation Events, Manage & Publish Organisation Events. A short description is provided below:
+
+*   **Read Only:** This allows the user to browse events that his organisation has access to, but doesn't allow any changes to be made to the database.
+*   **Manage My Own Events:** The second option, gives its users rights to create, modify or delete their own events, but they cannot publish them.
+*   **Manage Organization Events:** Allows users to create events or modify and delete events created by a member of their organisation.
+*   **Manage & Publish Organisation Events:** This last setting, gives users the right to do all of the above and also to publish the events of their organisation.
+
+The extra permissions are defined below:
+
+*   **Perm Admin:** Gives the user limited administrator privileges, this setting is used for the organisation admins.
+*   **Perm Audit:** Grants access to the logs. With the exception of site admins, only logs generated by the user's own org are visible.
+*   **Perm Tagger:** Allow user to assign tags to events.
+*   **Perm Sharing Group:** Grant access to edit or create sharing groups.
+*   **Perm Site Admin:** Gives the user full administrator privileges, this setting is used for the site admins.
+*   **Perm Auth:** This setting enables the authentication key of the role's users to be used for rest requests.
+*   **Perm Tag Editor:** Grand access to edit or create new local tags or from taxonomies. 
+*   **Perm Delegate:** Grant access to delegate the publication of an event to a third-party organization.
+*   **Perm Sync:** This setting allows the users of the role to be used as a synchronisation user. The authentication key of this user can be handed out to the administrator of a remote MISP instance to allow the synchronisation features to work.
+*   **Perm Regexp Access:** Allows the users with this permission enabled to edit the regular expression table. Be careful when giving out this permission, incorrect regular expressions can be very harmful (infinite loops, loss of data, etc.).
+*   **Perm Template:** Grant access to create or modify templates.
+
+#### Adding a new role:
+
+When creating a new role, you will have to enter a name for the role to be created and set up the permissions (as described above) using the drop-down menu and the check-boxes.
+
+![Add a new role.](figures/add_role.png)
+
+#### Listing roles:
+
+By clicking on the List Roles button, you can view a list of all the currently registered roles and a list of the permission flags turned on for each. In addition, you can find buttons that allow you to edit and delete the roles. Keep in mind that you will need to first remove every member from a role before you can delete it.
+
+![You can Edit or Delete roles using the action buttons to the right in each row. Keep in mind that a role has to be no members before it can be deleted.](figures/list_roles.png)
+
+*   **Id:** The role's automatically assigned ID number.
+*   **Name:** The name of role.
+*   **Permission:** One of the 4 permissions: Read Only, Manage My Own Events, Manage Organization Events, Manage & Publish Organisation Events. 
+*   **Extra Permissions flag:** Flag for each extra permissions: Admin, Site Admin, Sync Actions, Audit Actions, Auth key access, Regex Actions, Tagger, Tag Editor, Template Editor, Sharing Group Editor, Deletagions Access.
+*   **Action Buttons:** There are 2 options available: Edit Role or Delete it.
+	*    **Edit Role:** Same options of create role's view.<br />
+![Edit Role.](figures/edit_roles.png)
+	*    **Delete Role:** Use this option for deleting a role.<br />
+![Delete Role.](figures/delete_roles.png)
+
+- - -
+
+### Tools
+
+MISP has a couple of administrative tools that help administrators keep their instance up to date and healthy. The list of these small tools can change rapidly with each new version, but they should be self-explanatory. Make sure to check this section after upgrading to a new version, just in case there is a new upgrade script in there - though if this is the case it will be mentioned in the upgrade instructions.
+
+![Administrative Tools.](figures/tools.png)
+
+- - -
+
+### Server Settings
+
+Since version 2.3, MISP has a settings and diagnostics tool that allows site-admins to manage and diagnose their MISP installation. You can access this by navigating to Administration - Server settings.
+
+
 ### Server settings and diagnostics
 
-Since version 2.3, MISP has a settings and diagnostics tool that allows site-admins to manage and diagnose their MISP installation. You can access this by navigating to Administration - Server settings
+
 
 ![Server settings overview with all of the tabs explained.](figures/settings_1.png)
 
@@ -32,7 +233,7 @@ Each of the setting pages is a table with each row representing a setting. Colou
 
 The workers tab shows a list of the workers that MISP can use. You can restart the workers using the restart all workers, If the button doesn't work, make sure that the workers were started using the apache user. This can however only be done using the command line, refer to the INSTALL.txt documentation on how to let the workers automatically start on each boot.
 
-*   **Worker Type**: The worker type is determined by the queue it monitors. MISP currently has 4 queues (cache, default, email and a special _schdlr_ queue).
+*   **Worker Type**: The worker type is determined by the queue it monitors. MISP currently has 5 queues (cache, default, prio, email and a special _schdlr_ queue).
 *   **Worker Id**: The ID is made up of the machine name, the PID of the worker and the queue it monitors.
 *   **Status**: Displays OK if the worker is running. If the _schdlr_ worker is the only one not running make sure that you copy the config file into the cakeresque directory as described in the INSTALL.txt documentation.
 
@@ -74,101 +275,6 @@ When viewing the list of whitelisted addresses, the following pieces of informat
 
 ![You can edit or delete currently white-listed addresses using the action buttons on this list.](figures/whitelist.png)
 
-### Managing the users:
-
-As an admin, you can set up new accounts for users, edit the profiles of users, delete them, or just have a look at all the viewers' profiles. Organisation admins are restricted to executing the same actions on their organisation's users only.
-
-#### Adding a new user:
-
-To add a new user, click on the New User button in the administration menu to the left and fill out the following fields in the view that is loaded:
-
-![Fill this form out to add a new user. Keep in mind that the drop-down menu titled Role controls the privileges the user will have.](figures/add_user.png)
-	
-*   **Email:** The user's e-mail address, this will be used as his/her login name and as an address to send all the automatic e-mails and e-mails sent by contacting the user as the reporter of an event.
-*   **Password:** A temporary password for the user that he/she should change after the first login. Make sure that it is at least 6 characters long, includes a digit or a special character and contains at least one upper-case and at least one lower-case character.
-*   **Confirm Password:** This should be an exact copy of the Password field.
-*   **Org:**The organisation of the user. Entering ADMIN into this field will give administrator privileges to the user. If you are an organisation admin, then this field will be unchangeable and be set to your own organisation.
-*   **Roles:** A drop-down list allows you to choose a role-group that the user should belong to. Roles define the privileges of the user. To learn more about roles, [click here](#managing-the-roles).
-*   **Receive alerts when events are published:** This option will subscribe the new user to automatically generated e-mails whenever an event is published.
-*   **Receive alerts from "contact reporter" requests:** This option will subscribe the new user to e-mails that are generated when another user tries to get in touch with an event's reporting organisation that matches that of the new user.
-*   **Authkey:** This is assigned automatically and is the unique authentication key of the user (he/she will be able to reset this and receive a new key). It is used for exports and for connecting one server to another, but it requires the user to be assigned to a role that has auth permission enabled.
-*   **NIDS Sid:** Nids ID, not yet implemented.
-*   **Gpgkey:** The key used for encrypting e-mails sent through the system. 
-	
-#### Listing all users:
-
-To list all current users of the system, just click on List Users under the administration menu to the left. A view will be loaded with a list of all users and the following columns of information:
-
-![View, Edit or Delete a user using the action buttons to the right.](figures/list_users.png)
-	
-*   **Id:** The user's automatically assigned ID number.
-*   **Org:** The organisation that the user belongs to.
-*   **Email:** The e-mail address (and login name) of the user.
-*   **Autoalert:** Shows whether the user has subscribed to auto-alerts and is always receiving the mass-emails regarding newly published events that he/she is eligible for.
-*   **ontactalert:** Shows whether the user has the subscription to contact reporter e-mails directed at his/her organisation turned on or off.
-*   **Gpgkey:** Shows whether the user has entered a Gpgkey yet.
-*   **Nids Sid:** Shows the currently assigned NIDS ID.
-*   **Termsaccepted:** This flag indicates whether the user has accepted the terms of use or not.
-*   **Newsread:** The last point in time when the user has looked at the news section of the system.
-*   **Action Buttons:** Here you can view a detailed view of a user, edit the basic details of a user (same view as the one used for creating a new user, but all the fields come filled out by default) or remove a user completely. 
-	
-#### Editing a user:
-
-To add a new user, click on the New User button in the administration menu to the left and fill out the following fields in the view that is loaded:
-	
-*   **Email:** The user's e-mail address, this will be used as his/her login name and as an address to send all the automatic e-mails and e-mails sent by contacting the user as the reporter of an event.
-*   **Password:** It is possible to assign a new password manually for a user. For example, in case that he/she forgot the old one a new temporary one can be assigned. Make sure to check the "Change password" field if you do give out a temporary password, so that the user will be forced to change it after login.
-*   **Confirm Password:** This should be an exact copy of the Password field.
-*   **Org:**The organisation of the user. Entering ADMIN into this field will give administrator privileges to the user. If you are an organisation admin, then this field will be unchangeable and be set to your own organisation.
-*   **Roles:** A drop-down list allows you to choose a role-group that the user should belong to. Roles define the privileges of the user. To learn more about roles, [click here](#managing-the-roles).
-*   **Receive alerts when events are published:** This option will subscribe the user to automatically generated e-mails whenever an event is published.
-*   **Receive alerts from "contact reporter" requests:** This option will subscribe the user to e-mails that are generated when another user tries to get in touch with an event's reporting organisation that matches that of the user.
-*   **Authkey:** It is possible to request a new authentication key for the user. 
-*   **NIDS Sid:** Nids ID, not yet implemented.
-*   **Termsaccepted:** Indicates whether the user has accepted the terms of use already or not.
-*   **Change Password:** Setting this flag will require the user to change password after the next login.
-*   **Gpgkey:** The key used for encrypting e-mails sent through the system. 
-	
-#### Contacting a user:
-
-Site admins can use the "Contact users" feature to send all or an individual user an e-mail. Users that have a PGP key set will receive their e-mails encrypted. When clicking this button on the left, you'll be presented with a form that allows you to specify the type of the e-mail, who it should reach and what the content is using the following options:
-
-![Contact your users here.](figures/contact.png)
-	
-*   **Action:** This defines the type of the e-mail, which can be a custom message or a password reset. Password resets automatically include a new temporary password at the bottom of the message and will automatically change the user's password accordingly.
-*   **Recipient:** The recipient toggle lets you contact all your users, a single user (which creates a second drop-down list with all the e-mail addresses of the users) and potential future users (which opens up a text field for the e-mail address and a text area field for a PGP public key).
-*   **Subject:** In the case of a custom e-mail, you can enter a subject line here.
-*   **Subject:** In the case of a custom e-mail, you can enter a subject line here.
-*   **Custom message checkbox:** This is available for password resets, you can either write your own message (which will be appended with a temporary key and the signature), or let the system generate one automatically.
-	
-Keep in mind that all e-mails sent through this system will, in addition to your own message, will be signed in the name of the instance's host organisation's support team, will include the e-mail address of the instance's support (if the contact field is set in the bootstrap file), and will include the instance's PGP signature for users that have a PGP key set (and thus are eligible for an encrypted e-mail).
-	
-### Managing the roles
-
-Privileges are assigned to users by assigning them to rule groups, which use one of four options determining what they can do with events and four additional privilege elevating settings. The four options for event manipulation are: Read Only, Manage My Own Events, Manage Organisation Events, Manage & Publish Organisation Events. The extra privileges are admin, sync, authentication key usage and audit permission
-	
-*   **Read Only:** This allows the user to browse events that his organisation has access to, but doesn't allow any changes to be made to the database.
-*   **Manage My Own Events:** The second option, gives its users rights to create, modify or delete their own events, but they cannot publish them.
-*   **Manage Organization Events:** allows users to create events or modify and delete events created by a member of their organisation.
-*   **Manage & Publish Organisation Events:** This last setting, gives users the right to do all of the above and also to publish the events of their organisation.
-*   **Perm sync:** This setting allows the users of the role to be used as a synchronisation user. The authentication key of this user can be handed out to the administrator of a remote MISP instance to allow the synchronisation features to work.
-*   **Perm auth:** This setting enables the authentication key of the role's users to be used for rest requests. 
-*   **Perm admin:** Gives the user limited administrator privileges, this setting is used for the organisation admins. 
-*   **Perm site admin:** Gives the user full administrator privileges, this setting is used for the site admins. 
-*   **Perm audit:** Grants access to the logs. With the exception of site admins, only logs generated by the user's own org are visible. 
-*   **Perm regexp access:** Allows the users with this permission enabled to edit the regular expression table. Be careful when giving out this permission, incorrect regular expressions can be very harmful (infinite loops, loss of data, etc.).
-*   **Perm tagger:** Allows the user with this permission to create custom tags and assign them to events. 
-	
-#### Creating roles:
-
-When creating a new role, you will have to enter a name for the role to be created and set up the permissions (as described above) using the radio toggle and the four check-boxes.
-
-#### Listing roles:
-
-By clicking on the List Roles button, you can view a list of all the currently registered roles and a list of the permission flags turned on for each. In addition, you can find buttons that allow you to edit and delete the roles. Keep in mind that you will need to first remove every member from a role before you can delete it.
-
-![You can View, Edit or Delete roles using the action buttons to the right in each row. Keep in mind that a role has to be devoid of members before it can be deleted.](figures/list_groups.png)
-	
 ### Using the logs of MISP
 
 Users with audit permissions are able to browse or search the logs that MISP automatically appends each time certain actions are taken (actions that modify data or if a user logs in and out).
@@ -214,10 +320,6 @@ Another way to browse the logs is to search it by filtering the results accordin
 *   **Change:** With the help of this field, you can search for various specific changes or changes to certain variables (such as published will find all the log entries where an event has gotten published, ip-src will find all attributes where a source IP address has been entered / edited, etc).
 
 
-### Administrative Tools
-
-MISP has a couple of administrative tools that help administrators keep their instance up to date and healthy. The list of these small tools can change rapidly with each new version, but they should be self-explanatory. Make sure to check this section after upgrading to a new version, just in case there is a new upgrade script in there - though if this is the case it will be mentioned in the upgrade instructions.
-	
 ### Background Processing
 
 If enabled, MISP can delegate a lot of the time intensive tasks to the background workers. These will then be executed in order, allowing the users of the instance to keep using the system without a hiccup and without having to wait for the process to finish. It also allows for certain tasks to be scheduled and automated.
@@ -474,4 +576,10 @@ Two ways to fix it:
 2) Comment the localhost mapping to IPv6 address in /etc/hosts
 
 
+#### Errors about fields or tables
 
+If you have errors with fields or tables that you can see in the error.log or in the page (if you enabled _debug_ or _site_admin_debug_ settings), an easy first them to make most of them go away is to use the **clean cache** feature on the _server settings_ menu, _diagnostics_ tab.
+An example of error message:
+```
+Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'Task.job_id' in 'field list'
+```

automation/README.md

@@ -136,8 +136,20 @@ https://<misp url>/events/csv/download
 
 You can specify additional flags for CSV exports as follows:
 
+POST to: 
 ~~~~
-https://<misp url>/events/csv/download/[eventid]/[ignore]/[tags]/[category]/[type]/[includeContext]/[from]/[to]/[last]
+https://<misp url>/events/csv/download
+~~~~
+
+Headers: 
+~~~~
+Authorization: <your auth key>
+Content-type: application/json
+~~~~
+
+Body:
+~~~~json
+{"parameter1":"value1", "parameter2":1, "parameter3":["value3", "value4", "!value5"]}
 ~~~~
 
 <dl>
@@ -146,19 +158,7 @@ https://<misp url>/events/csv/download/[eventid]/[ignore]/[tags]/[category]/[typ
 <dt>ignore</dt>
 <dd>Setting this flag to true will include attributes that are not marked "to_ids".</dd>
 <dt>tags</dt>
-<dd>To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag
-commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).</dd>
-</dl>
-
-For example, to include tag1 and tag2 but exclude tag3 you would use:
-
-For example, to only download a csv generated of the "domain" type and the "Network activity" category attributes all events except for the one and further restricting it to events that are tagged "tag1" or "tag2" but not "tag3", only allowing attributes that are IDS flagged use the following syntax:
-
-~~~~
-https://<misp url>/events/csv/download/false/false/tag1&&tag2&&!tag3/Network%20activity/domain
-~~~~
-
-<dl>
+<dd>Simply add a list of tags that should be included or negated (by prepending the tag name with a "!"). Any event with a negated tag will be ignored, even if an included tag is matching. An example is included further down.</dd>
 <dt>category</dt>
 <dd>The attribute category, any valid MISP attribute category is accepted.</dd>
 <dt>type</dt>
@@ -173,10 +173,30 @@ https://<misp url>/events/csv/download/false/false/tag1&&tag2&&!tag3/Network%20a
 <dd>Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.</dd>
 </dl>
 
-The keywords false or null should be used for optional empty parameters in the URL.
+For example, to only download a csv generated of the "domain" type and the "Network activity" category attributes all events except for the one and further restricting it to events that are tagged "tag1" or "tag2" but not "tag3", only allowing attributes that are IDS flagged use the following syntax:
 
+POST to: 
+~~~~
+https://<misp url>/events/csv/download
+~~~~
+
+Headers: 
+~~~~
+Authorization: <your auth key>
+Content-type: application/json
+~~~~
+
+Body:
+~~~~json
+{"tags":["tag1", "tag2", "!tag3"], "category":"Network activity", "type": "domain"}
+~~~~
+
+Alternatively you can fall back to the deprecated syntax of passing parameters in a GET request via the URL, however this is discouraged:
+~~~~
+https://<misp url>/events/csv/download/[eventid]/[ignore]/[tags]/[category]/[type]/[includeContext]/[from]/[to]/[last]
+~~~~
+If you use the deprecated URL parameter method, keep in mind that the keywords false or null should be used for optional empty parameters.
 To export the attributes of all events that are of the type "domain", use the following syntax:
-
 ~~~~
 https://<misp url>/events/csv/download/false/false/false/false/domain
 ~~~~
@@ -487,7 +507,7 @@ To restrict the results by tags, use the usual syntax. Please be aware the colon
 https://<misp url>/attributes/text/download/ip-src/tag1&&
 ~~~~
 
-As of version 2.3.38, it is possible to restrict the text exports on two additional flags. The first allows the user to restrict based on event ID,
+It is possible to restrict the text exports on additional flags. The first allows the user to restrict based on event ID,
 whilst the second is a boolean switch allowing non IDS flagged attributes to be exported. Additionally, choosing "all" in the type field will return
 all eligible attributes.
 
@@ -500,6 +520,14 @@ https://<misp url>/attributes/text/download/[type]/[tags]/[event_id]/[allowNonID
 <dd>The attribute type, any valid MISP attribute type is accepted.</dd>
 <dt>tags</dt>
 <dd>To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).</dd>
+<dt>allowNonIDS</dt>
+<dd>Include attributes that would normally be excluded due to the IDS flag not being set or due to being whitelisted</dd>
+<dt>from</dt>
+<dd>Set the lowest "date" field value that should be included in the export (format YYYY-MM-DD)</dd>
+<dt>to</dt>
+<dd>Set the highest "date" field value that should be included in the export (format YYYY-MM-DD)</dd>
+<dt>last</dt>
+<dd>Set the timeframe of the export based on the "timestamp" value. The parameter uses a time + metric notation (valid examples: "2w", "60m", "24h")</dd>
 </dl>
 
 For example, to include tag1 and tag2 but exclude tag3 you would use:
@@ -536,7 +564,7 @@ It is possible to search the database for attributes based on a list of criteria
 To return an event with all of its attributes, relations, shadowAttributes, use the following syntax:
 
 ~~~~
-https://<misp url>/events/restSearch/download/[value]/[type]/[category]/[org]/[tag]/[quickfilter]/[from]/[to]/[last]
+https://<misp url>/events/restSearch/download/[value]/[type]/[category]/[org]/[tag]/[quickfilter]/[from]/[to]/[last]/[eventid]/[withAttachments]/[metadata]/[uuid]
 ~~~~
 
 <dl>
@@ -570,6 +598,10 @@ https://<misp url>/events/restSearch/download/null/null/null/null/tag1&&tag2&&!t
 <dd>Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.</dd>
 <dt>eventid</dt>
 <dd>The events that should be included / excluded from the search</dd>
+<dt>withAttachments</dt>
+<dd>Include the attachments/encrypted samples in the export</dd>
+<dt>metadata</dt>
+<dd>Only fetch the event metadata (event data, tags, relations) and skip the attributes</dd>
 </dl>
 
 The keywords false or null should be used for optional empty parameters in the URL.
@@ -614,13 +646,15 @@ To just return a list of attributes, use the following syntax:
 <dt>last</dt>
 <dd>Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.</dd>
 <dt>eventid</dt>
-<dd>The events that should be included / excluded from the search</dd>
+<dd>The events that should be included / excluded from the search.</dd>
+<dt>uuid</dt>
+<dd>The returned events must include an attribute with the given UUID, or alternatively the event's UUID must match the value(s) passed.</dd>
 </dl>
 
 The keywords false or null should be used for optional empty parameters in the URL.
 
 ~~~~
-https://<misp url>/attributes/restSearch/download/[value]/[type]/[category]/[org]/[tag]/[from]/[to]/[last]/[eventid]
+https://<misp url>/attributes/restSearch/download/[value]/[type]/[category]/[org]/[tag]/[from]/[to]/[last]/[eventid]/[withattachments]/[uuid]
 ~~~~
 
 Value, type, category and org are optional. It is possible to search for several terms in each category by joining them with the '&&' operator. It is
@@ -652,6 +686,58 @@ sigOnly is an optional flag that will block all attributes from being exported t
 https://<misp url>/attributes/returnAttributes/download/25/md5&&sha256&&!filename/true
 ~~~~
 
+## Filtering event metadata
+
+As described in the REST section, it is possible to retrieve a list of events along with their metadata by sending a GET request to the /events API. However, this API in particular is a bit more versatile. You can pass search parameters along to search among the events on various fields and retrieve a list of matching events (along with their metadata). Use the following URL:
+
+~~~~
+https://<misp url>/events/index 
+~~~~
+
+POST a JSON object with the desired lookup fields and values to receive a JSON back.
+An example for a valid lookup:
+
+~~~~
+Authorization: <your API key>
+Accept: application/json
+Content-type: application/json
+~~~~
+
+Body: 
+
+~~~~json
+{"searchinfo":"Locky", "searchpublished":1, "searchdistribution":0}
+~~~~
+
+
+The list of valid parameters:
+<dl>
+<dt>searchpublished:</dt>
+<dd>Filters on published or unpulished events [0,1] - negatable</dd>
+<dt>searchinfo:</dt>
+<dd>Filters on strings found in the event info - negatable</dd>
+<dt>searchtag:</dt>
+<dd>Filters on attached tag names - negatable</dd>
+<dt>searcheventid:</dt>
+<dd>Filters on specific event IDs - negatable</dd>
+<dt>searchthreatlevel:</dt>
+<dd>Filters on a given event threat level [1,2,3,4] - negatable</dd>
+<dt>searchdistribution:</dt>
+<dd>Filters on the distribution level [0,1,2,3] - negatable</dd>
+<dt>searchanalysis:</dt>
+<dd>Filters on the given analysis phase of the event [0,1,2,3] - negatable</dd>
+<dt>searchattribute:</dt>
+<dd>Filters on a contained attribute value - negatable</dd>
+<dt>searchorg:</dt>
+<dd>Filters on the creator organisation - negatable</dd>
+<dt>searchemail:</dt>
+<dd>Filters on the creator user's email address (admin only) - negatable</dd>
+<dt>searchDatefrom:</dt>
+<dd>Filters on the date, anything newer than the given date in YYYY-MM-DD format is taken - non-negatable</dd>
+<dt>searchDateuntil:</dt>
+<dd>Filters on the date, anything older than the given date in YYYY-MM-DD format is taken - non-negatable</dd>
+</dl>
+
 ## Download attachment or malware sample
 
 If you know the attribute ID of a malware-sample or an attachment, you can download it with the following syntax:
@@ -793,6 +879,32 @@ XML:
 
 None of the above fields are mandatory, but at least one of them has to be provided.
 
+## Sharing groups
+
+MISP allows sharing groups to be retrieved via the API.
+
+~~~~
+https://<misp url>/sharing_groups/index.json
+~~~~
+
+Based on the API key used, the list of visible sharing groups will be returned in a JSON file. The JSON includes the organization parts of a given sharing group along with the associated server.
+
+## Enable and disable feeds via the API
+
+The MISP feeds can be enabled via the API.
+
+A feed can be enabled by POSTing on the following url (feed_id is the id of the feed):
+
+~~~~
+/feeds/enable/feed_id
+~~~~
+
+A feed can be disabled by POSTing on the following url (feed_id is the id of the feed):
+
+~~~~
+/feeds/disable/feed_id
+~~~~
+
 ## Sightings API
 
 MISP allows Sightings data to be conveyed in several ways. 
@@ -906,7 +1018,7 @@ An example STIX sightings document:
 </stix:STIX_Package>
 ~~~~
 
-POSTing this as the message's body to MISP will sight any attributes visible to the user witht he value "malicious2.example.com". For composite types, a match on a component will also trigger a sighting (so for example for attributes of type domain|ip a domain match would be sufficient).
+POSTing this as the message's body to MISP will sight any attributes visible to the user with he value "malicious2.example.com". For composite types, a match on a component will also trigger a sighting (so for example for attributes of type domain|ip a domain match would be sufficient).
 
 If no Related observables are set in the Sighting itself, MISP will fall back to the observable directly contained in the indicator. So in the following example:
 
@@ -964,6 +1076,176 @@ If no Related observables are set in the Sighting itself, MISP will fall back to
 
 MISP would create sightings for attributes matching any of the following: malicious1.example.com, malicious2.example.com, malicious3.example.com
 
+# Describe types API
+
+MISP can procedurally describe all attribute types and attribute categories it currently supports including the category - type mappings. To access this information simply send a GET request to:
+
+~~~~
+https://<misp url>/attributes/describeTypes
+~~~~
+
+Depending on the headers passed the returned data will be a JSON object or an XML, with 3 main sections: types, categories, category\_type\_mappings.
+
+# Attribute statistics API
+
+If you are interested in the attribute type or attribute category data distribution on your instance, MISP offers an API that will create an aggregates list. To access the API, simple sent a GET request to:
+
+~~~~
+https://<misp url>/attributes/attributeStatistics/[context]/[percentage]
+~~~~
+
+Where the following parameters can be set:
+
+<dl>
+<dt>Context</dt>
+<dd>Set whether you are interested in the type or category statistics of your instance. This parameter can be either set to "type" or "category", with type being the default setting if the parameter is not set.</dd>
+<dt>Percentage</dt>
+<dd>An optional field, if set, it will return the results in percentages instead of the count.</dd>
+</dl>
+
+The results are always returned as JSON.
+
+Sample output of the types in percentages from CIRCL's MISP instance:
+
+~~~~json
+{
+    "AS": "0.015%",
+    "attachment": "0.177%",
+    "btc": "0.005%",
+    "campaign-name": "0.005%",
+    "comment": "1.47%",
+    "domain": "15.992%",
+    "domain|ip": "0.005%",
+    "email-attachment": "0.207%",
+    "email-dst": "0.121%",
+    "email-src": "0.192%",
+    "email-subject": "0.146%",
+    "filename": "3.698%",
+    "filename|md5": "0.349%",
+    "filename|sha1": "0.894%",
+    "filename|sha256": "0.652%",
+    "hostname": "17.558%",
+    "http-method": "0.045%",
+    "ip-dst": "7.087%",
+    "ip-src": "2.707%",
+    "link": "5.748%",
+    "malware-sample": "0.702%",
+    "malware-type": "0.005%",
+    "md5": "21.064%",
+    "mutex": "0.278%",
+    "named pipe": "0.03%",
+    "other": "1.495%",
+    "pattern-in-file": "0.192%",
+    "pattern-in-memory": "0.303%",
+    "pattern-in-traffic": "0.051%",
+    "regkey": "0.126%",
+    "regkey|value": "0.187%",
+    "sha1": "8.921%",
+    "sha256": "5.597%",
+    "snort": "0.045%",
+    "target-machine": "0.248%",
+    "target-org": "0.01%",
+    "target-user": "0.106%",
+    "text": "0.934%",
+    "threat-actor": "0.005%",
+    "url": "2.258%",
+    "user-agent": "0.081%",
+    "vulnerability": "0.182%",
+    "whois-registrant-email": "0.01%",
+    "x509-fingerprint-sha1": "0.01%",
+    "yara": "0.086%"
+}
+~~~~
+
+# User management
+
+MISP allows administrators to create and manage users via its REST API
+
+The API is available in JSON format so make sure you use the following headers:
+
+~~~~
+Authorization: [Your auth key]
+Content-type: application/json
+Accept: application/json
+
+~~~~
+
+To fetch all users send a GET request to:
+
+~~~~
+https://<misp url>/admin/users
+~~~~
+
+To view a user simply send a GET request to the following url:
+
+~~~~
+https://<misp url>/admin/users/view/[user id]
+~~~~
+
+To create a new user, send a POST request to:
+
+~~~~
+https://<misp url>/admin/users/add
+~~~~
+
+Sample input:
+
+~~~~
+{
+    "email":"andras.iklody@circl.lu",
+    "org\_id":1,
+    "role\_id":1
+}
+~~~~
+
+To view the mandatory and optional fields, use a GET request on the above URL.
+
+Sample output:
+
+~~~~
+{
+    "name": "\/admin\/users\/add API description",
+    "description": "POST a User object in JSON format to this API to create a new user.",
+    "mandatory_fields": [
+        "email",
+        "org_id",
+        "role_id"
+    ],
+    "optional_fields": [
+        "password",
+        "external_auth_required",
+        "external_auth_key",
+        "enable_password",
+        "nids_sid",
+        "server_id",
+        "gpgkey",
+        "certif_public",
+        "autoalert",
+        "contactalert",
+        "disabled",
+        "change_pw",
+        "termsaccepted",
+        "newsread"
+    ],
+    "url": "\/admin\/users\/add"
+}
+~~~~
+
+To edit an existing user send a POST request to: 
+
+~~~~
+https://<misp url>/admin/users/edit/[user id]
+~~~~
+
+Only the fields POSTed will be updated, the rest is left intact. To view all possible parameters, simply send a GET request to the above URL.
+
+
+You can also delete users by POSTing to the below URL, but keep in mind that disabling users (by setting the disabled flag via an edit) is always prefered to keep user associations to events intact.
+
+~~~~
+https://<misp url>/admin/users/delete/[user id]
+~~~~
+
 # Automation using PyMISP
 
 PyMISP is a Python library to access MISP platforms via their REST API.
@@ -972,4 +1254,3 @@ PyMISP allows you to fetch events, add or update events/attributes, add or updat
 
 [PyMISP is available](https://github.com/MISP/PyMISP) including a documentation with various examples.
 
-

book.json

@@ -3,7 +3,7 @@
     "description": "User guide of MISP Malware Information Sharing Platform, a Threat Sharing Platform.",
     "language": "en",
     "author": "MISP Contributors",
-    "plugins": ["autocover", "github", "toc"],
+    "plugins": ["autocover", "github", "toc", "anchors"],
     "links": { "sidebar": {        "MISP @ GitHub": "https://github.com/MISP/MISP", "PDF Format": "https://www.circl.lu/doc/misp/book.pdf" }},
     "pluginsConfig": {
         "autocover": {

categories-and-types/README.md

@@ -1,7 +1,6 @@
 <!-- toc -->
 
-# MISP Attribute Categories vs Types (MISP version 2.4)
-
+## MISP Attribute Categories vs Types
 
 |Category| Internal reference | Targeting data | Antivirus detection | Payload delivery | Artifacts dropped | Payload installation |
 | --- |:---:|:---:|:---:|:---:|:---:|:---:|
@@ -65,21 +64,21 @@
 |ssdeep| | | | X | X | X |
 |imphash| | | | X | X | X |
 |pehash| | | | X | | X |
-|sha-224| | | | | | |
-|sha-384| | | | | | |
-|sha-512| | | | | | |
-|sha-512/224| | | | | | |
-|sha-512/256| | | | | | |
+|sha224| | | | X | X | X |
+|sha384| | | | X | X | X |
+|sha512| | | | X | X | X |
+|sha512/224| | | | X | X | X |
+|sha512/256| | | | X | X | X |
 |tlsh| | | | X | | X |
 |filename&#124;authentihash| | | | X | X | X |
 |filename&#124;ssdeep| | | | X | X | X |
 |filename&#124;imphash| | | | X | X | X |
 |filename&#124;pehash| | | | X | X | X |
-|filename&#124;sha-224| | | | | | |
-|filename&#124;sha-384| | | | | | |
-|filename&#124;sha-512| | | | | | |
-|filename&#124;sha-512/224| | | | | | |
-|filename&#124;sha-512/256| | | | | | |
+|filename&#124;sha224| | | | X | X | X |
+|filename&#124;sha384| | | | X | X | X |
+|filename&#124;sha512| | | | X | X | X |
+|filename&#124;sha512/224| | | | X | X | X |
+|filename&#124;sha512/256| | | | X | X | X |
 |filename&#124;tlsh| | | | X | X | X |
 |windows-scheduled-task| | | | | X | |
 |windows-service-name| | | | | X | |
@@ -89,11 +88,53 @@
 |whois-registrant-name| | | | | | |
 |whois-registrar| | | | | | |
 |whois-creation-date| | | | | | |
-|targeted-threat-index| | | | | | |
-|mailslot| | | | | | |
-|pipe| | | | | | |
-|ssl-cert-attributes| | | | | | |
 |x509-fingerprint-sha1| | | | X | X | X |
+|dns-soa-email| | | | | | |
+|size-in-bytes| | | | | | |
+|counter| | | | | | |
+|datetime| | | | | | |
+|cpe| | | | | | |
+|port| | | | | | |
+|ip-dst&#124;port| | | | X | | |
+|ip-src&#124;port| | | | X | | |
+|hostname&#124;port| | | | X | | |
+|email-dst-display-name| | | | X | | |
+|email-src-display-name| | | | X | | |
+|email-header| | | | X | | |
+|email-reply-to| | | | X | | |
+|email-x-mailer| | | | X | | |
+|email-mime-boundary| | | | X | | |
+|email-thread-index| | | | X | | |
+|email-message-id| | | | X | | |
+|github-username| | | | | | |
+|github-repository| | | | | | |
+|github-organisation| | | | | | |
+|jabber-id| | | | | | |
+|twitter-id| | | | | | |
+|first-name| | | | | | |
+|middle-name| | | | | | |
+|last-name| | | | | | |
+|date-of-birth| | | | | | |
+|place-of-birth| | | | | | |
+|gender| | | | | | |
+|passport-number| | | | | | |
+|passport-country| | | | | | |
+|passport-expiration| | | | | | |
+|redress-number| | | | | | |
+|nationality| | | | | | |
+|visa-number| | | | | | |
+|issue-date-of-the-visa| | | | | | |
+|primary-residence| | | | | | |
+|country-of-residence| | | | | | |
+|special-service-request| | | | | | |
+|frequent-flyer-number| | | | | | |
+|travel-details| | | | | | |
+|payment-details| | | | | | |
+|place-port-of-original-embarkation| | | | | | |
+|place-port-of-clearance| | | | | | |
+|place-port-of-onward-foreign-destination| | | | | | |
+|passenger-name-record-locator-number| | | | | | |
+|mobile-application-id| | | | X | | X |
 
 |Category| Persistence mechanism | Network activity | Payload type | Attribution | External analysis | Financial fraud |
 | --- |:---:|:---:|:---:|:---:|:---:|:---:|
@@ -157,21 +198,21 @@
 |ssdeep| | | | | | |
 |imphash| | | | | | |
 |pehash| | | | | | |
-|sha-224| | | | | | |
-|sha-384| | | | | | |
-|sha-512| | | | | | |
-|sha-512/224| | | | | | |
-|sha-512/256| | | | | | |
+|sha224| | | | | | |
+|sha384| | | | | | |
+|sha512| | | | | | |
+|sha512/224| | | | | | |
+|sha512/256| | | | | | |
 |tlsh| | | | | | |
 |filename&#124;authentihash| | | | | | |
 |filename&#124;ssdeep| | | | | | |
 |filename&#124;imphash| | | | | | |
 |filename&#124;pehash| | | | | | |
-|filename&#124;sha-224| | | | | | |
-|filename&#124;sha-384| | | | | | |
-|filename&#124;sha-512| | | | | | |
-|filename&#124;sha-512/224| | | | | | |
-|filename&#124;sha-512/256| | | | | | |
+|filename&#124;sha224| | | | | | |
+|filename&#124;sha384| | | | | | |
+|filename&#124;sha512| | | | | | |
+|filename&#124;sha512/224| | | | | | |
+|filename&#124;sha512/256| | | | | | |
 |filename&#124;tlsh| | | | | | |
 |windows-scheduled-task| | | | | | |
 |windows-service-name| | | | | | |
@@ -181,108 +222,192 @@
 |whois-registrant-name| | | | X | | |
 |whois-registrar| | | | X | | |
 |whois-creation-date| | | | X | | |
-|targeted-threat-index| | | | | | |
-|mailslot| | | | | | |
-|pipe| | | | | | |
-|ssl-cert-attributes| | | | | | |
 |x509-fingerprint-sha1| | X | | X | X | |
+|dns-soa-email| | | | | | |
+|size-in-bytes| | | | | | |
+|counter| | | | | | |
+|datetime| | | | | | |
+|cpe| | | | | | |
+|port| | | | | | |
+|ip-dst&#124;port| | X | | | X | |
+|ip-src&#124;port| | X | | | X | |
+|hostname&#124;port| | | | | | |
+|email-dst-display-name| | | | | | |
+|email-src-display-name| | | | | | |
+|email-header| | | | | | |
+|email-reply-to| | | | | | |
+|email-x-mailer| | | | | | |
+|email-mime-boundary| | | | | | |
+|email-thread-index| | | | | | |
+|email-message-id| | | | | | |
+|github-username| | | | | | |
+|github-repository| | | | | X | |
+|github-organisation| | | | | | |
+|jabber-id| | | | | | |
+|twitter-id| | | | | | |
+|first-name| | | | | | |
+|middle-name| | | | | | |
+|last-name| | | | | | |
+|date-of-birth| | | | | | |
+|place-of-birth| | | | | | |
+|gender| | | | | | |
+|passport-number| | | | | | |
+|passport-country| | | | | | |
+|passport-expiration| | | | | | |
+|redress-number| | | | | | |
+|nationality| | | | | | |
+|visa-number| | | | | | |
+|issue-date-of-the-visa| | | | | | |
+|primary-residence| | | | | | |
+|country-of-residence| | | | | | |
+|special-service-request| | | | | | |
+|frequent-flyer-number| | | | | | |
+|travel-details| | | | | | |
+|payment-details| | | | | | |
+|place-port-of-original-embarkation| | | | | | |
+|place-port-of-clearance| | | | | | |
+|place-port-of-onward-foreign-destination| | | | | | |
+|passenger-name-record-locator-number| | | | | | |
+|mobile-application-id| | | | | | |
 
-|Category| Other |
-| --- |:---:|
-|md5| |
-|sha1| |
-|sha256| |
-|filename| |
-|pdb| |
-|filename&#124;md5| |
-|filename&#124;sha1| |
-|filename&#124;sha256| |
-|ip-src| |
-|ip-dst| |
-|hostname| |
-|domain| |
-|domain&#124;ip| |
-|email-src| |
-|email-dst| |
-|email-subject| |
-|email-attachment| |
-|url| |
-|http-method| |
-|user-agent| |
-|regkey| |
-|regkey&#124;value| |
-|AS| |
-|snort| |
-|pattern-in-file| |
-|pattern-in-traffic| |
-|pattern-in-memory| |
-|yara| |
-|vulnerability| |
-|attachment| |
-|malware-sample| |
-|link| |
-|comment| X |
-|text| X |
-|other| X |
-|named pipe| |
-|mutex| |
-|target-user| |
-|target-email| |
-|target-machine| |
-|target-org| |
-|target-location| |
-|target-external| |
-|btc| |
-|iban| |
-|bic| |
-|bank-account-nr| |
-|aba-rtn| |
-|bin| |
-|cc-number| |
-|prtn| |
-|threat-actor| |
-|campaign-name| |
-|campaign-id| |
-|malware-type| |
-|uri| |
-|authentihash| |
-|ssdeep| |
-|imphash| |
-|pehash| |
-|sha-224| |
-|sha-384| |
-|sha-512| |
-|sha-512/224| |
-|sha-512/256| |
-|tlsh| |
-|filename&#124;authentihash| |
-|filename&#124;ssdeep| |
-|filename&#124;imphash| |
-|filename&#124;pehash| |
-|filename&#124;sha-224| |
-|filename&#124;sha-384| |
-|filename&#124;sha-512| |
-|filename&#124;sha-512/224| |
-|filename&#124;sha-512/256| |
-|filename&#124;tlsh| |
-|windows-scheduled-task| |
-|windows-service-name| |
-|windows-service-displayname| |
-|whois-registrant-email| |
-|whois-registrant-phone| |
-|whois-registrant-name| |
-|whois-registrar| |
-|whois-creation-date| |
-|targeted-threat-index| |
-|mailslot| |
-|pipe| |
-|ssl-cert-attributes| |
-|x509-fingerprint-sha1| |
+|Category| Suport Tool | Social network | Person | Other |
+| --- |:---:|:---:|:---:|:---:|
+|md5| | | | |
+|sha1| | | | |
+|sha256| | | | |
+|filename| | | | |
+|pdb| | | | |
+|filename&#124;md5| | | | |
+|filename&#124;sha1| | | | |
+|filename&#124;sha256| | | | |
+|ip-src| | | | |
+|ip-dst| | | | |
+|hostname| | | | |
+|domain| | | | |
+|domain&#124;ip| | | | |
+|email-src| | X | | |
+|email-dst| | X | | |
+|email-subject| | | | |
+|email-attachment| | | | |
+|url| | | | |
+|http-method| | | | |
+|user-agent| | | | |
+|regkey| | | | |
+|regkey&#124;value| | | | |
+|AS| | | | |
+|snort| | | | |
+|pattern-in-file| | | | |
+|pattern-in-traffic| | | | |
+|pattern-in-memory| | | | |
+|yara| | | | |
+|vulnerability| | | | |
+|attachment| X | | | |
+|malware-sample| | | | |
+|link| X | | | |
+|comment| X | X | X | X |
+|text| X | X | X | X |
+|other| X | X | X | X |
+|named pipe| | | | |
+|mutex| | | | |
+|target-user| | | | |
+|target-email| | | | |
+|target-machine| | | | |
+|target-org| | | | |
+|target-location| | | | |
+|target-external| | | | |
+|btc| | | | |
+|iban| | | | |
+|bic| | | | |
+|bank-account-nr| | | | |
+|aba-rtn| | | | |
+|bin| | | | |
+|cc-number| | | | |
+|prtn| | | | |
+|threat-actor| | | | |
+|campaign-name| | | | |
+|campaign-id| | | | |
+|malware-type| | | | |
+|uri| | | | |
+|authentihash| | | | |
+|ssdeep| | | | |
+|imphash| | | | |
+|pehash| | | | |
+|sha224| | | | |
+|sha384| | | | |
+|sha512| | | | |
+|sha512/224| | | | |
+|sha512/256| | | | |
+|tlsh| | | | |
+|filename&#124;authentihash| | | | |
+|filename&#124;ssdeep| | | | |
+|filename&#124;imphash| | | | |
+|filename&#124;pehash| | | | |
+|filename&#124;sha224| | | | |
+|filename&#124;sha384| | | | |
+|filename&#124;sha512| | | | |
+|filename&#124;sha512/224| | | | |
+|filename&#124;sha512/256| | | | |
+|filename&#124;tlsh| | | | |
+|windows-scheduled-task| | | | |
+|windows-service-name| | | | |
+|windows-service-displayname| | | | |
+|whois-registrant-email| | | | |
+|whois-registrant-phone| | | | |
+|whois-registrant-name| | | | |
+|whois-registrar| | | | |
+|whois-creation-date| | | | |
+|x509-fingerprint-sha1| | | | |
+|dns-soa-email| | | | |
+|size-in-bytes| | | | X |
+|counter| | | | X |
+|datetime| | | | X |
+|cpe| | | | X |
+|port| | | | X |
+|ip-dst&#124;port| | | | |
+|ip-src&#124;port| | | | |
+|hostname&#124;port| | | | |
+|email-dst-display-name| | | | |
+|email-src-display-name| | | | |
+|email-header| | | | |
+|email-reply-to| | | | |
+|email-x-mailer| | | | |
+|email-mime-boundary| | | | |
+|email-thread-index| | | | |
+|email-message-id| | | | |
+|github-username| | X | | |
+|github-repository| | X | | |
+|github-organisation| | X | | |
+|jabber-id| | X | | |
+|twitter-id| | X | | |
+|first-name| | | X | |
+|middle-name| | | X | |
+|last-name| | | X | |
+|date-of-birth| | | X | |
+|place-of-birth| | | X | |
+|gender| | | X | |
+|passport-number| | | X | |
+|passport-country| | | X | |
+|passport-expiration| | | X | |
+|redress-number| | | X | |
+|nationality| | | X | |
+|visa-number| | | X | |
+|issue-date-of-the-visa| | | X | |
+|primary-residence| | | X | |
+|country-of-residence| | | X | |
+|special-service-request| | | X | |
+|frequent-flyer-number| | | X | |
+|travel-details| | | X | |
+|payment-details| | | X | |
+|place-port-of-original-embarkation| | | X | |
+|place-port-of-clearance| | | X | |
+|place-port-of-onward-foreign-destination| | | X | |
+|passenger-name-record-locator-number| | | X | |
+|mobile-application-id| | | | |
 
 ### Categories
 
 *   **Internal reference**: Reference used by the publishing party (e.g. ticket number)
-*   **Targeting data**: Targeting information to include recipient email, infected machines, department, and or locations.<br/>
+*   **Targeting data**: Targeting information to include recipient email, infected machines, department, and or locations.
 *   **Antivirus detection**: List of anti-virus vendors detecting the malware or information on detection performance (e.g. 13/43 or 67%). Attachment with list of detection or link to VirusTotal could be placed here as well.
 *   **Payload delivery**: Information about the way the malware payload is initially delivered, for example information about the email or web-site, vulnerability used, originating IP etc. Malware sample itself should be attached here.
 *   **Artifacts dropped**: Any artifact (files, registry keys etc.) dropped by the malware or other modifications to the system
@@ -293,7 +418,10 @@
 *   **Attribution**: Identification of the group, organisation, or country behind the attack
 *   **External analysis**: Any other result from additional analysis of the malware like tools output Examples: pdf-parser output, automated sandbox analysis, reverse engineering report.
 *   **Financial fraud**: Financial Fraud indicators, for example: IBAN Numbers, BIC codes, Credit card numbers, etc.
-*   **Other**: Attributes that are not part of any other category
+*   **Suport Tool**: Tools supporting analysis or detection of the event
+*   **Social network**: Social networks and platforms
+*   **Person**: A human being - natural person
+*   **Other**: Attributes that are not part of any other category or are meant to be used as a component in MISP objects in the future
 
 ### Types
 
@@ -357,21 +485,21 @@
 *   **ssdeep**: You are encouraged to use filename|ssdeep instead. A checksum in the SSDeep format, only use this if you don't know the correct filename
 *   **imphash**: You are encouraged to use filename|imphash instead. A hash created based on the imports in the sample, only use this if you don't know the correct filename
 *   **pehash**: PEhash - a hash calculated based of certain pieces of a PE executable file
-*   **sha-224**: You are encouraged to use filename|sha224 instead. A checksum in sha224 format, only use this if you don't know the correct filename
-*   **sha-384**: You are encouraged to use filename|sha384 instead. A checksum in sha384 format, only use this if you don't know the correct filename
-*   **sha-512**: You are encouraged to use filename|sha512 instead. A checksum in sha512 format, only use this if you don't know the correct filename
-*   **sha-512/224**: You are encouraged to use filename|sha512/224 instead. A checksum in sha512/224 format, only use this if you don't know the correct filename
-*   **sha-512/256**: You are encouraged to use filename|sha512/256 instead. A checksum in sha512/256 format, only use this if you don't know the correct filename
+*   **sha224**: You are encouraged to use filename|sha224 instead. A checksum in sha224 format, only use this if you don't know the correct filename
+*   **sha384**: You are encouraged to use filename|sha384 instead. A checksum in sha384 format, only use this if you don't know the correct filename
+*   **sha512**: You are encouraged to use filename|sha512 instead. A checksum in sha512 format, only use this if you don't know the correct filename
+*   **sha512/224**: You are encouraged to use filename|sha512/224 instead. A checksum in sha512/224 format, only use this if you don't know the correct filename
+*   **sha512/256**: You are encouraged to use filename|sha512/256 instead. A checksum in sha512/256 format, only use this if you don't know the correct filename
 *   **tlsh**: You are encouraged to use filename|tlsh instead. A checksum in the Trend Micro Locality Sensitive Hash format, only use this if you don't know the correct filename
 *   **filename|authentihash**: A checksum in md5 format
 *   **filename|ssdeep**: A checksum in ssdeep format
 *   **filename|imphash**: Import hash - a hash created based on the imports in the sample.
 *   **filename|pehash**: A filename and a PEhash separated by a |
-*   **filename|sha-224**: A filename and a sha-224 hash separated by a |
-*   **filename|sha-384**: A filename and a sha-384 hash separated by a |
-*   **filename|sha-512**: A filename and a sha-512 hash separated by a |
-*   **filename|sha-512/224**: A filename and a sha-512/224 hash separated by a |
-*   **filename|sha-512/256**: A filename and a sha-512/256 hash separated by a |
+*   **filename|sha224**: A filename and a sha-224 hash separated by a |
+*   **filename|sha384**: A filename and a sha-384 hash separated by a |
+*   **filename|sha512**: A filename and a sha-512 hash separated by a |
+*   **filename|sha512/224**: A filename and a sha-512/224 hash separated by a |
+*   **filename|sha512/256**: A filename and a sha-512/256 hash separated by a |
 *   **filename|tlsh**: A filename and a Trend Micro Locality Sensitive Hash separated by a |
 *   **windows-scheduled-task**: A scheduled task in windows
 *   **windows-service-name**: A windows service name. This is the name used internally by windows. Not to be confused with the windows-service-displayname.
@@ -381,8 +509,51 @@
 *   **whois-registrant-name**: The name of a domain's registrant, obtained from the WHOIS information.
 *   **whois-registrar**: The registrar of the domain, obtained from the WHOIS information.
 *   **whois-creation-date**: The date of domain's creation, obtained from the WHOIS information.
-*   **targeted-threat-index**:
-*   **mailslot**: MailSlot interprocess communication
-*   **pipe**: Pipeline (for named pipes use the attribute type "named pipe")
-*   **ssl-cert-attributes**: SSL certificate attributes
 *   **x509-fingerprint-sha1**: X509 fingerprint in SHA-1 format
+*   **dns-soa-email**: RFC1035 mandates that DNS zones should have a SOA (Statement Of Authority) record that contains an email address where a PoC for the domain could be contacted. This can sometimes be used for attribution/linkage between different domains even if protected by whois privacy
+*   **size-in-bytes**: Size expressed in bytes
+*   **counter**: An integer counter, generally to be used in objects
+*   **datetime**: Datetime in the ISO 8601 format
+*   **cpe**: Common platform enumeration
+*   **port**: Port number
+*   **ip-dst|port**: IP destination and port number seperated by a |
+*   **ip-src|port**: IP source and port number seperated by a |
+*   **hostname|port**: Hostname and port number seperated by a |
+*   **email-dst-display-name**: Email destination display name
+*   **email-src-display-name**: Email source display name
+*   **email-header**: Email header
+*   **email-reply-to**: Email reply to header
+*   **email-x-mailer**: Email x-mailer header
+*   **email-mime-boundary**: The email mime boundary separating parts in a multipart email
+*   **email-thread-index**: The email thread index header
+*   **email-message-id**:
+*   **github-username**: A github user name
+*   **github-repository**: A github repository
+*   **github-organisation**: A github organisation
+*   **jabber-id**: Jabber ID
+*   **twitter-id**: Twitter ID
+*   **first-name**: First name of a natural person
+*   **middle-name**: Middle name of a natural person
+*   **last-name**: Last name of a natural person
+*   **date-of-birth**: Date of birth of a natural person (in YYYY-MM-DD format)
+*   **place-of-birth**: Place of birth of a natural person
+*   **gender**: The gender of a natural person (Male, Female, Other, Prefer not to say)
+*   **passport-number**: The passport number of a natural person
+*   **passport-country**: The country in which the passport was issued
+*   **passport-expiration**: The expiration date of a passport
+*   **redress-number**: The Redress Control Number is the record identifier for people who apply for redress through the DHS Travel Redress Inquiry Program (DHS TRIP). DHS TRIP is for travelers who have been repeatedly identified for additional screening and who want to file an inquiry to have erroneous information corrected in DHS systems
+*   **nationality**: The nationality of a natural person
+*   **visa-number**: Visa number
+*   **issue-date-of-the-visa**: The date on which the visa was issued
+*   **primary-residence**: The primary residence of a natural person
+*   **country-of-residence**: The country of residence of a natural person
+*   **special-service-request**: A Special Service Request is a function to an airline to provide a particular facility for A Passenger or passengers.
+*   **frequent-flyer-number**: The frequent flyer number of a passenger
+*   **travel-details**: Travel details
+*   **payment-details**: Payment details
+*   **place-port-of-original-embarkation**: The orignal port of embarkation
+*   **place-port-of-clearance**: The port of clearance
+*   **place-port-of-onward-foreign-destination**: A Port where the passenger is transiting to
+*   **passenger-name-record-locator-number**: The Passenger Name Record Locator is a key under which the reservation for a trip is stored in the system. The PNR contains, among other data, the name, flight segments and address of the passenger. It is defined by a combination of five or six letters and numbers.
+*   **mobile-application-id**: The application id of a mobile application
+

create-event-report/README.md

@@ -0,0 +1,118 @@
+## Create an event based on a report
+
+:warning: We need specific permission to create an event. 
+
+For this example, we will use a report found on [Bleeping Computer](http://www.bleepingcomputer.com/news/security/researcher-finds-the-karma-ransomware-being-distributed-via-pay-per-install-network/), so considered as OSINT.
+
+![Report title](figures/report_title.png)
+
+### The metadata
+
+First of all, we need to create a new event. To do so, we click the "Add Event" option when on the Events list view.
+
+![Add Event Option](figures/menu_add_event.png)
+
+Then we get the add event form.
+
+![Add Event form](figures/add_event_form.png)
+
+Let's fill it with the data we already have:
+* Date: Here we will put the date of the report, so 2016-11-14
+* Distribution: Depending on the event, we might want it to be more or less spread accross the MISP instances. For this one, since it is a public report, there is no reason to limit the diffusion so "All communities".
+* Threat Level: Self explainatory. Since the ransomware in the report is not using a huge exploit, we can use low, or undefined as we don't really know. we'll go for the latter since it can be edited. 
+* Analysis: Give the current stage of the analysis. Since the report is published, we can assume that the analysis is completed.
+* Event Info: The event's info is in fact the name or title of the event, so it seems legit to put the title of the report here as well. Since it is public information, we also prefix it with "OSINT".
+* GFI sandbox: Since we don't have any sample or anything here, we leave this alone.
+
+![Add Event form filled](figures/add_event_form_filled.png)
+
+Then just press the blue "Add" button and here we have a brand new event. Empty.
+
+![EMPTY EVENT YAY](figures/event_metadata.png)  
+(Displayed information can change depending on your role on the MISP instance)
+
+Now it is time to populate this event. But before even adding IoC, we are going to add global information about the report itself: the link of the report and a short explanation or introduction. To do so, we need to click on the "Add Attribute" option in the side menu. This will show us this view:
+
+![add attribute](figures/add_attribute.png)
+
+* First we are going to add the link of the report. Since it has been written by an other researcher, it will be considered as an "External analysis", we choose this category. 
+* Concerning the type, regarding the kind of data we are adding it is obvious that we will choose the "link" type.
+* The distribution field can be a little tricky. We can either choose one of the option that was already available at event level or "Inherit event". If we choose the latter, the attribute will be shared the same way as the event it is included in (here to "All communities"). On the other hand, if we choose manually a distribution for the attribute, the most restritive between event distribution and attribute distribution will be applied. That is to say: if both event and attribute distributions are the same, there will be no change (similar to "Inherit event"). However, if for instance the event distribution is "all communities" while the attribute is limited to "This community only", the event will indeed be distributed to all communities but without this particular attribute which will be limited to this community only. The same works the other way around, if the attribute can be distributed to "all communities" while the related event is limited to this community, the attribute being dependant of the event, it will be shared to this community only, basing its distribution on the event (most restrictive) one.
+* The value is simply the data we want to add, here it is the link of the report.
+* The contextual comment is a field that will not be used for correlation and is mainly there to add some complementary information on the attribute. Can be a port for an IP, or an indication of any type. Here there is no perticular information to add, except maybe tell that it is the source of the report, so let us put this information.
+* "for Intrusion Detection System" is used to set the IDS flag or not. If set, the attribute will be used as an IDS signature when exporting the NIDS data. In this case, we have no reason to check it.
+* The Batch Import is a useful option when we need to add several IoC of the same category/type which allow you to add them at once by separated by a line break between each line in the value field. However it is of no use here.
+
+![add attribute filled](figures/add_attribute_filled.png)
+
+All fields are properly filled ? Then let's press the "submit" button, and Ta-dah !
+
+![attribute added YAY](figures/added_attribute.png)
+
+Now we can do a similar procedure to add an introduction to the report (that is to say the first paragraph of the report). We will simply change the type for text. But this time, we will access the add attribute form by clicking on the small + symbol next to the attribute table.
+
+![Noooo you found meeeeee](figures/hidden_add_attr.png)
+
+The same form as before will appear in a popup.
+
+![add attribute popup](figures/add_attr_popup.png)
+
+Again, we fill it with the required data.
+
+![add attribute popup](figures/filled_popup.png)
+
+Then we submit it by clicking on the blue button
+_Et voilà!_
+
+![MAGIC](figures/popadded.png)
+
+Okay, now it is time to add some Indicators of Compromise. In this report, they are mainly listed at the end.
+
+![OMG IOC](figures/IoC_from_report.png)
+
+Let's try to define which category/type those IoC belong to.
+
+First, Windows-TuneUp.exe is without a doubt a _filename_, and the associated category may be _Payload delivery_.
+
+Second the registry entries (type _regkey_) seems to be from _Artifacts dropped_ category
+
+Then the hashes that are already said to be _SHA 256_, and a quick test on VirusTotal also reveals that they correspond to the filename seen earlier. so we can add both as an association _filename|SHA256_. Once again, the category will be _Payload delivery_.
+
+And finally the network communication. No doubt here for the category: _Network activity_, and the type might be _url_ but for the example, we will let MISP decide for us.
+
+So we begin with the filename. No real change from before for this one, except that we will set the IDS flag to true.
+
+![filename](figures/filename.png)
+
+Then we can add the hashes in a similar way. We will had them both alone and combined with the filename. In order to do it quickly, we are going to use the freetext import tool, hidden there
+
+![freetext import step 1](figures/freeeeeimport.png)
+
+It will open a popup with a text area field where we will paste our IoC, one per line. As said previously, we add both the hashes alone and with the filename.
+
+![freetext import step 2](figures/freetxtimpooort.png)
+
+Then when we press the submit button, we are redirected on this page to control the sent data.
+
+![freetext import step 3](figures/freeresults.png)
+
+Here, MISP detected by itself what should be the category and type associated to our IoC and surprise! It matches our suppositions. Plus, it also put the IDS flag, so it is perfect. But before submitting, please double check to be sure all the values are correct and no information was lost (That can happen when the data are not formatted as expected by MISP). 
+
+If the results of MISP were not what we expected, we can still modify it, however MISP will only suggest suitable category/type regarding the format of your data. We can change for each attribute individually or all at the same time using the option on the bottom right of the form. The same principle also applies for the comments, individually or for all.
+
+![freetext import suggestions](figures/freesuggest.png)  
+(Yes I have two cursors, MISP is magic!)
+
+We only have the network indicators left, and as said before, we will let MISP determined for us which type is the best for the data we have.
+
+![freetext import network](figures/free_network.png)
+![type recognition fail](figures/surprise.png)
+
+Oh well, that was unexpected. In fact, it is not that surprising regarding the format of the tor address that look more like a filename than like a url but it is still a problem, since we can't change the type nor the category to a more consistant one. This is indeed one of the limitation of freetext import. To solve this issue, we will use a simple trick: we will add a slash at the end of the tor address so it won't be confused for a filename. 
+
+![freetext import network](figures/free_network2.png)
+![type recognition fail](figures/nomoresurprise.png)
+
+Thanks to the added character, the first string is recognised as an url which is more consistent with the reality. The second also seems okay, so we can now submit both. 
+
+And that is all we can get for the main informations and IoC in this report. If we search more carefully, there might still be some information left in it, like the filename of the ransomnote for instance, but we will stop here for this example.

galaxy/README.md

@@ -0,0 +1,83 @@
+<!-- toc -->
+
+## Galaxies
+
+Galaxies in MISP are a method used to express a large object called cluster that can be attached to MISP events or attributes. A cluster can be composed of one or more elements. Elements are expressed as key-values.
+
+There are default vocabularies available in MISP galaxy but those can be overwritten, replaced or updated as you wish. Vocabularies are from existing standards (like STIX, Veris, MISP and so on) or custom ones.
+
+Existing clusters and vocabularies can be used as-is or as a template. MISP distribution can be applied to each cluster to permit a limited or broader distribution scheme.
+
+The objective is to have a comment set of clusters for organizations starting analysis but that can be expanded to localized information (which is not shared) or additional information (that can be shared).
+
+WIP
+
+[MISP galaxy](https://github.com/MISP/misp-galaxy)
+
+### Managing Galaxies in MISP
+
+WIP
+
+### Using Galaxies in MISP Events - Example
+
+For this example, we will try to add a cluster to an existing event. This cluster will contains informations about threath actor known as Sneaky Panda.
+
+![EventWithoutCluster](./figures/EventWithoutCluster.png)
+
+Here on the event view, we notice a blue frame under the metadatas with the title "Galaxies" and a button "Add new cluster". Let's click on the latter to begin.
+
+![GalaxyPopup](./figures/GalaxyPopup.png)
+
+A popup will appear proposising to explore a particular galaxy or all at the same time. Here, as we know we want to as a threat actor, we will choose the second option and scroll to find Sneaky Panda (We are courageous, aren't we?).
+
+![NoSneakyPanda](./figures/NoSneakyPanda.png)
+
+Wait. No Sneaky Panda? Hm that's strange. Or maybe it is only registred as a alias. Let's have a look! To do so we will use the search field which stay on top of the list. So what do we get? Beijing Group, is it an alias of our threat actor.
+
+![Search](./figures/Search.png)
+
+Pointing the cursor on it will give us the answer.
+
+![Alias](./figures/Alias.png)
+
+We have a match. So we select it and here we go.
+
+![NewThreatActor](./figures/NewThreatActor.png)
+
+Clicking on the magnifying glass next to Threat actor redirects to the list of all threat actors
+Clicking on the magnifying glass next to Beijing Group redirects us to a page about this group
+Clicking on the addition symbole on the left of Beijing Group extends the module.
+
+### Available Galaxies
+
+#### Clusters
+
+[Exploit-kit](https://github.com/MISP/misp-galaxy/blob/master/clusters/exploit-kit.json) - Exploit-Kit is an enumeration of some exploitation kits used by adversaries. The list includes document, browser and router exploit kits. It's not meant to be totally exhaustive but aim at covering the most seen in the past 5 years.
+
+[Microsoft Activity Group](https://github.com/MISP/misp-galaxy/blob/master/clusters/microsoft-activity-group.json) - Activity groups as described by Microsoft.
+
+[TDS - Traffic Direction System](clusters/tds.json) - TDS is a list of Traffic Direction System used by adversaries.
+
+[Threats Actors](https://github.com/MISP/misp-galaxy/blob/master/clusters/threat-actor.json) - Known or estimated adversary groups targeting organizations and employees. Adversary groups are regularly confused with their initial operation or campaign. Threat actors are characteristics of malicious actors (or adversaries) representing a cyber attack threat including presumed intent and historically observed behaviour.
+
+[Tools](https://github.com/MISP/misp-galaxy/blob/master/clusters/tool.json) - Enumeration of software tools used by adversaries. The list includes malware but also common software regularly used by the adversaries.
+
+
+#### Vocabularies
+
+##### Common
+	[certainty-level]
+	(https://github.com/MISP/misp-galaxy/blob/master/vocabularies/common/certainty-level.json) -
+Certainty level of an associated element or cluster
+
+##### threat-actor
+ 	[intended-effect]
+	(https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/intended-effect.json) - default STIX vocabulary for expressing the intended effect of a threat actor
+	[motivation]
+	(https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/motivation.json) - default STIX vocabulary for expressing the motivation of a threat actor.
+	[planning-and-operational-support]
+	(https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/planning-and-operational-support.json) - default STIX vocabulary for expressing the planning and operational support functions available to a threat actor.
+	[sophistication]
+	(https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/sophistication.json) - default STIX vocabulary for expressing the subjective level of sophistication of a threat actor.
+	[type]
+	(https://github.com/MISP/misp-galaxy/blob/master/vocabularies/threat-actor/type.json) - default STIX vocabulary for expressing the subjective type of a threat actor.

general-layout/README.md

@@ -1,75 +1,167 @@
-<!-- toc -->
+<!-- Nothing else matters -->
 
 ## General Layout
 
 ### The top bar
 
-![This is the main menu that will be accessible from all of the views. In some instances, some additional buttons that will appear on top of these when a view provides it.](figures/menu_image.png)
+#### Simple User
+![This is the main menu that will be accessible from all of the views. In some instances, some additional buttons that will appear on top of these when a view provides it.](figures/MenuBarUser.jpg)
 
-This menu contains all of the main functions of the site as a series of dropdown menues. These contains all (from the current user's perspective) accessible functions sorted into several groups.
+This menu contains all of the main functions of the site as a series of dropdown menus. These contains all (from the current simple user's perspective) accessible functions sorted into several groups.
 
 *   **Home button:** This button will return you to the start screen of the application, which is the event index page (more about this later).
 *   **Event Actions:** All the malware data entered into MISP is made up of an event object that is described by its connected attributes. The Event actions menu gives access to all the functionality that has to do with the creation, modification, deletion, publishing, searching and listing of events and attributes.
 *   **Input Filters:** Input filters alter what and how data can be entered into this instance. Apart from the basic validation of attribute entry by type, it is possible for the site administrators to define regular expression replacements and blacklists for certain values in addition to blocking certain values from being exportable. Users can view these replacement and blacklist rules here whilst administrator can alter them.
-*   **Global Actions:** This menu gives you access to information about MISP and this instance. You can view and edit your own profile, view the manual, read the news or the terms of use again, see a list of the active organisations on this instance and a histogram of their contributions by attribute type.
-*   **Sync Actions:** With administrator access rights, shows a list of the connected instances and allows the initiation of a push and a pull (more about the synchronisation mechanisms later).
-*   **Administration:** Administrators can add, edit or remove user accounts and user roles. Roles define the access rights to certain features such as publishing of events, usage of the REST interface or synchronisation of any user belonging to the given role. Site administrators can also access a contact form, through which it is possible to reset the passwords of users, or to just get in touch with them via encrypted e-mails.
-*   **Audit:** If you have audit permissions, you can view the logs for your organisation (or for site admins for the entire system) here or even search the logs if you are interested in something specific.
+*   **Global Actions:** This menu gives you access to information about MISP and this instance. You can view and edit your own profile, view the manual, read the news or the terms of use again, see a list of the active organizations on this instance and a histogram of their contributions by attribute type.
 *   **Discussions:** Link to the discussion threads.
-*   **Proposal Notifications:** This shows how many proposals your organisation has received and across how many events they are spread out. Clicking this will take you to the list of proposals.
+
+#### Admin Menu Bar
+![Some additional buttons that will appear on top of these when a view provides it.](figures/MenuBarAdmin.jpg)
+*   **Home button:** idem as user.
+
+*   **Event Actions:** ibidem
+
+*   **Input Filters:** Ibidem
+
+*   **Global Actions:** Ibidem
+
+*   **Sync Actions:** With administrator access rights, shows a list of the connected instances and allows the initiation of a push and a pull (more about the synchronization mechanisms later).
+
+*   **Administration:** Administrators can add, edit or remove user accounts and user roles. Roles define the access rights to certain features such as publishing of events, usage of the REST interface or synchronization of any user belonging to the given role. Site administrators can also access a contact form, through which it is possible to reset the passwords of users, or to just get in touch with them via encrypted e-mails.
+
+*   **Audit:** If you have audit permissions, you can view the logs for your organization (or for site admins for the entire system) here or even search the logs if you are interested in something specific.
+
+*   **Proposal Notifications:** This shows how many proposals your organization has received and across how many events they are spread out. Clicking this will take you to the list of proposals.
+
 *   **Log out:** Logs you out of the system.
 
-### A list of the contents of each of the above drop-down menues
+### A list of the contents of each of the above drop-down menus
 
 ##### Event actions
 
+![List Event Actions](figures/Event Actions.jpg)
 *   **List Events:** Lists all the events in the system that are not private or belong to your organisation. You can add, modify, delete, publish or view individual events from this view.
-*   **Add Event:** Allows you to fill out an event creation form and create the event object, which you can start populating with attributes.
+
+*   **Add Event:** Allows you to fill out an event creation form and create the event object, which you can start adding attributes.
+
 *   **List Attributes:** Lists all the attributes in the system that are not private or belong to your organisation. You can modify, delete or view each individual attribute from this view.
+
 *   **Search Attributes:** You can set search terms for a filtered attribute index view here.
+
 *   **View Proposals:** Shows a list of all proposals that you are eligible to see.
-*   **Events with proposals:** Shows all of the events created by your organsiation that has pending proposals.
-*   **List Tags:**List all the tags that have been created by users with tag creation rights on this instance.
-*   **Add Tag:**Create a new tag.
-*   **List Templates:**List all of the templates created by users with template creation rights on this instance.
-*   **Add Template:**Create a new template.
+
+*   **Events with proposals:** Shows all of the events created by your organisation that has pending proposals.
+
+*   **List Tags:** List all the tags that have been created by users with tag creation rights on this instance.
+
+*   **Add Tag:** Create a new tag.
+
+*   **List Templates:** List all of the templates created by users with template creation rights on this instance.
+
+*   **Add Template:** Create a new template.
+
 *   **Export:** Export the data accessible to you in various formats.
+
 *   **Automation:** If you have authentication key access, you can view how to use your key to use the REST interface for automation here.
 
 ##### Input filters
 
-*   **Import Regexp:** You can view the Regular Expression rules, which modify the data that can be entered into the system. This can and should be used to help filter out personal information from automatic imports (such as removing the username from windows file paths), having unified representation for certain common values for easier correlation or simply standardising certain input. It is also possible to block certain values from being inserted. As a site administrator or a user with regex permission, you can also edit these rules.
-*   **Signature Whitelist:** You can view the whitelist rules, which contain the values that are blocked from being used for exports and automation on this instance. Site administrators have access to editing this list.
+![Input filters](figures/InputFilter.png)
+
+*   **Import Regexp:** You can view the Regular Expression rules, which modify the data that can be entered into the system. This can and should be used to help filter out personal information from automatic imports (such as removing the username from windows file paths), having unified representation for certain common values for easier correlation or simply standardizing certain input. It is also possible to block certain values from being inserted. As a site administrator or a user with regex permission, you can also edit these rules.
+
+*   **Signature Whitelist:** You can view the whitelist rules, which contains the values that are blocked from being used for exports and automation on this instance. Site administrators have access to editing this list.
+
+*  **List Warninglists:**
+MISP warninglists are lists of well-known indicators that can be associated to potential false positives, errors or mistakes. The warning lists are integrated in MISP to display an info/warning box at the event and attribute level.
 
 ##### Global Actions
 
+
+![Global Actions](figures/GlobalActions.png)
+
 *   **News:** Read about the latest news regarding the MISP system
+
 *   **My Profile:** Manage your user account.
-*   **Members List:** View the number of users per organisation and get some statistics about the currently stored attributes.
+
+*   **Dashboard:** allow you to see your Notifications of Proposals, Events with proposals and Delegation request. Your can see the last changes since your last visit, as Events updates and Events publications.
+
+*   **Members List:** View the number of users per organization and get some statistics about the currently stored attributes.
+
+*   **Organizations:** View the organizations having a presence on this instance, with some useful informations as contact's name.
+
 *   **Role Permissions:** You can view the role permissions here.
+
+*  **List Sharing Groups:** You can view the list of existing Sharing Groups who you or your organization have access.
+
+*  **Add Sharing Group:** You can create a sharing group.
+
 *   **User Guide:** A link to this user guide.
-*   **Terms & Conditions:** View the terms & conditions again.
+
 *   **Statistics:** View a series of statistics about the users and the data on this instance.
-*   **Log out:** Logs the current user out.
+
 
 ##### Sync Actions
 
+
+![Sync Actions](figures/SyncActions.png)
+
 *   **List Servers:** Connect your MISP instance to other instances, or view and modify the currently established connections.
+<!-- Fix provided by elhoim -->
+It may be that you have an Error Message in the page (if you enabled debug or site_admin_debug settings). An example of error message:
+
+![Error message](figures/pb-list-server.png)
+
+An easy first them to make most of them go away is to use the clean cache feature on the server settings menu, diagnostics tab.
+
+![cleanscript](figures/cleanscript1.png)
+
+You must then scroll down the page.
+
+![cleanscript](figures/cleanscript2.png)
+
+
+*  **List Feeds:** Follow the RSS feeds of other organization or CERTs worldwide.  
 
 ##### Administration
 
-*   **New User:** Create an account for a new user for your organisation. Site administrators can create users for any organisation.
+![Administration](figures/Administration.png)
+
 *   **List Users:** View, modify or delete the currently registered users.
-*   **New Role:** Create a new role group for the users of this instance, controlling their privileges to create, modify, delete and to publish events and to access certain features such as the logs or automation.
+
+*   **New User:** Create an account for a new user for your organisation. Site administrators can create users for any organisation.
+
+*  **Contact Users:** You can use this view to send messages to your current or future users or send them a temporary password.
+
+When adding a new user to the system, or when you want to manually reset the password for a user, just use the "Send temporary password" setting.
+
+After selecting the action, choose who the target of the e-mails should be (all users, a single user or a user not yet in the system).
+
+You can then specify (if eligible) what the e-mail address of the target is (for existing users you can choose from a dropdown menu).
+
+In the case of a new user, you can specify the future user's GPG key, to send his/her new key in an encrypted e-mail.
+
+The system will automatically generate a message for you, but it is also possible to write a custom message if you tick the check-box, but don't worry about assigning a temporary password manually, the system will do that for you, right after your custom message.
+
+*  **List Organizations:** View the organizations having a presence on this instance, with some useful informations.
+
+*  **Add Organization:**
+
 *   **List Roles:** List, modify or delete currently existing roles.
-*   **Contact Users:** You can use this view to send messages to your current or future users or send them a new temporary password.
-*   **Administrative Tools:** Various tools, upgrade scripts that can help a site-admin run the instance
-*   **Server Settings:** Set up and diagnose your MISP installation
+
+*   **Add Role:** Create a new role group for the users of this instance, controlling their privileges to create, modify, delete and to publish events and to access certain features such as the logs or automation.
+
+*   **Administrative Tools:** Various tools, upgrade scripts that can help a site-admin run the instance.
+
+*   **Server Settings:** Set up and diagnose your MISP installation.
+
 *   **Jobs:** View the background jobs and their progress
+
 *   **Scheduled Tasks:** Schedule the pre-defined tasks for your instance (this currently includes export caching, server pull and server push).
 
 ##### Audit
 
+![Audit](figures/Audit.png)
 *   **List Logs:** View the logs of the instance.
 *   **Search Logs:** Search the logs by various attributes.
 
@@ -81,4 +173,3 @@ This menu contains all of the main functions of the site as a series of dropdown
 ### The left bar
 
 This bar changes based on each page-group. The blue selection shows you what page you are on.
-

pymisp/README.md

@@ -0,0 +1,337 @@
+## PyMISP - Python Library to access MISP
+
+PyMISP is a Python library to access MISP platforms via their REST API.
+
+PyMISP allows you to fetch events, add or update events/attributes, add or update samples or search for attributes.
+
+Note that you need to have Auth Key access in your MISP instance to use PyMISP
+
+### Capabilities
+
+* Add, get, update, publish, delete events
+* Add or remove tags
+* Add file attributes:  hashes, registry key, patterns, pipe, mutex
+* Add network attributes:  IP dest/src, hostname, domain, url, UA, ...
+* Add Email attributes:  source, destination, subject, attachment, ...
+* Upload/download samples
+* Update sightings
+* Proposals:  add, edit, accept, discard
+* Full text search and search by attributes
+* Get STIX event
+* Export statistics  
+And even more, just look at the api.py file
+
+### Installation
+
+You can install PyMISP by either using pip or by getting the last version from the [GitHub repository](https://github.com/MISP/PyMISP) 
+
+#### Install from pip
+~~~~
+pip install pymisp
+~~~~
+
+#### Install the lastest version from repo
+~~~~
+git clone https://github.com/CIRCL/PyMISP.git && cd PyMISP
+python setup.py install
+~~~~
+
+Note that you will also need to install [requests](http://docs.python-requests.org/) if you don't have it already.
+
+### Getting started
+
+You now need to get your automation key. You can find it on the automation page:
+
+~~~~
+https://<misp url>/events/automation
+~~~~
+
+or on your profile
+
+~~~~
+https://<misp url>/users/view/me
+~~~~
+
+If you did not install using the repository, you can still fetch it to get examples to work on:
+~~~~
+git clone https://github.com/CIRCL/PyMISP.git
+~~~~
+
+In order to use these, you need to create a file named keys.py in the examples folder and edit it to put the url of your MISP instance and your automation key.
+~~~~
+cd examples
+cp keys.py.sample keys.py
+vim keys.py
+~~~~
+
+Once you are done with it, you are ready to start.
+
+### Using PyMISP
+
+To have a better understanding of how to use PyMISP, we will have a look at one of the existing examples: add\_named\_attribute.py
+This script allow us to add an attribute to an existing event while knowing only its type (the category is determined by default).
+~~~~python
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from pymisp import PyMISP
+from keys import misp_url, misp_key
+import argparse
+~~~~
+First of all, it is obvious that we need to import PyMISP.
+Then we also need to know both the instance with which we will work and the API key to use: Both should be stored in the keys.py file.  
+Finally we import argparse library so the script can handle arguments.
+~~~~python
+# For python2 & 3 compat, a bit dirty, but it seems to be the least bad one
+try:
+    input = raw_input
+except NameError:
+    pass
+~~~~
+Just a few lines to be sure that pyhon 2 and 3 are supported
+~~~~python
+def init(url, key):
+    return PyMISP(url, key, True, 'json', debug=True)
+~~~~
+This function will create a PyMISP object that will be used later to interact with the MISP instance.
+As seen in the [api.py](https://github.com/CIRCL/PyMISP/blob/master/pymisp/api.py#L85), a PyMISP object need to know both the url of the MISP instance and the API key to use. It can also take additionnal and not mandatory data, such as the use or not of SSL or the name of the export format.
+~~~~python
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Create an event on MISP.')
+    parser.add_argument("-e", "--event", type=int, help="The id of the event to update.")
+    parser.add_argument("-t", "--type", help="The type of the added attribute")
+    parser.add_argument("-v", "--value", help="The value of the attribute")
+    args = parser.parse_args()
+~~~~
+Then the function starts by preparing the awaited arguments:
+* event: The event that will get a new attribute
+* type: The type of the attribute that will be added. See [here](../categories-and-types/README.md) for more informations 
+* value: The value of the new attribute
+~~~~python
+    misp = init(misp_url, misp_key)
+~~~~
+Thanks to the previously created function, we create a PyMISP object.
+~~~~python
+    event = misp.get_event(args.event)
+    event = misp.add_named_attribute(event, args.type, args.value)
+~~~~
+In order to add the new argument, we first need to fetch the event in the MISP database using the [get\_event](https://github.com/CIRCL/PyMISP/blob/master/pymisp/api.py#L223) function which only need the event\_id. Then only once we have it, we can call the function [add\_named\_attribute](https://github.com/CIRCL/PyMISP/blob/master/pymisp/api.py#L372) that will add the argument. 
+~~~~python
+	print(event)
+~~~~
+Finally the new event is printed, so we can check that the attribute was correctly added, and that a category was attached to it automatically.
+
+### Existing examples
+
+As the name implies you will find several example scripts in the examples folder. For each you can get help if you do `scriptname.py -h`
+
+Let us have a look at some of these examples:
+
+#### add_named_attribute.py
+
+Allow to add an argument to an existing event by giving only the type of the attribute. The category will be set with a default value.
+
+Arguments:
+* **event**: The id of the event to update.
+* **type**: The type of the added attribute.
+* **value**: The value of the attribute.
+
+#### add_user.py
+
+Allow to add a user by giving the mandatory fields as entries.
+
+Arguments:
+* **email**: Email linked to the account.
+* **org_id**: Organisation linked to the user.
+* **role_id**: Role linked to the user.
+
+#### add_user_json.py
+
+Add the user described in the given json. If no file is provided, returns a json listing all the fields used to describe a user.
+
+Arguments:
+* **json_file**: The name of the json file describing the user you want to create.
+
+#### create_events.py
+
+Allow a user to create a new event on the MISP instance.
+
+Arguments:
+* **distrib**: The distribution setting used for the attributes and for the newly created event, if relevant. [0-3].
+* **info**: Used to populate the event info field if no event ID supplied.
+* **analysis**: The analysis level of the newly created event, if applicable. [0-2]
+* **threat**: The threat level ID of the newly created event, if applicable. [1-4]
+
+#### del.py
+
+Delete an event or an attribute from a MISP instance. The event has the priority: if both are set, only the event will be deleted. 
+
+Arguments:
+* **event**: Event ID to delete.
+* **attribute**: Attribute ID to delete.
+
+#### delete_user.py
+
+Delete the user with the given id. Keep in mind that disabling users (by setting the disabled flag via an edit) is always prefered to keep user associations to events intact.
+
+Arguments:
+* **user_id**: The id of the user you want to delete.
+
+#### edit_user.py
+
+Edit the email of the user designed by the user_id.
+
+Arguments:
+* **user_id**: The name of the json file describing the user you want to modify.
+* **email**: Email linked to the account.
+
+#### edit_user_json.py
+
+Edit the user designed by the user_id. If no file is provided, returns a json listing all the fields used to describe a user.
+
+Arguments:
+* **user_id**: The name of the json file describing the user you want to modify.
+* **json_file**: The name of the json file describing your modifications.
+
+#### get.py
+
+Get an event from a MISP instance in json format.
+
+Arguments:
+* **event**: Event ID to get.
+* **output**: Output file
+
+#### last.py
+
+Download latest events from a MISP instance. A output file can be created to store these events.
+
+Arguments:
+* **last**: can be defined in days, hours, minutes (for example 5d or 12h or 30m).
+* **output**: Output file
+
+#### searchall.py
+
+Get all the events matching a value.
+
+Arguments:
+* **search**: String to search.
+* **quiet**: Only display URLs to MISP
+* **output**: Output file
+
+#### sharing_groups.py
+
+Get a list of the sharing groups from the MISP instance.  
+No argument.
+
+#### sighting.py
+
+Add sighting.
+
+Arguments:
+* **json_file**: The name of the json file describing the attribute you want to add sighting to.
+
+#### stats.py
+
+Output attributes statistics from a MISP instance.  
+No argument.
+
+#### suricata.py
+
+Download Suricata events.
+
+Arguments:
+* **all**: Download all suricata rules available.
+* **event**: Download suricata rules from one event.
+
+#### tags.py
+
+Get tags from MISP instance.  
+No argument.
+
+#### tagstatistics.py
+
+Get statistics from tags.
+
+Arguments:
+* **percentage**: An optional field, if set, it will return the results in percentages, otherwise it returns exact count.
+* **namesort**: An optional field, if set, values are sort by the namespace, otherwise the sorting will happen on the value.
+
+#### up.py
+
+Update an existing event regarding the data inside a given json file.
+
+Arguments:
+* **event**: Event ID to modify.
+* **input**: Input file
+
+#### upload.py
+
+Send malware sample to MISP.
+
+Arguments:
+* **upload**: File or directory of files to upload.
+* **event**: Not supplying an event ID will cause MISP to create a single new event for all of the POSTed malware samples.
+* **distrib**: The distribution setting used for the attributes and for the newly created event, if relevant. [0-3].
+* **ids**: You can flag all attributes created during the transaction to be marked as \"to_ids\" or not.
+* **categ**: The category that will be assigned to the uploaded samples. Valid options are: Payload delivery, Artifacts dropped, Payload Installation, External Analysis.
+* **info**: Used to populate the event info field if no event ID supplied.
+* **analysis**: The analysis level of the newly created event, if applicatble. [0-2]
+* **threat**: The threat level ID of the newly created event, if applicatble. [1-4]
+* **comment**: Comment for the uploaded file(s).
+
+#### users_list.py 
+
+Get a list of the sharing groups from the MISP instance.  
+No argument.
+
+### Going further
+
+#### feed-generator
+
+It is used to generate the CIRCL OSINT feed. This script export the events as json, based on tags, organisation, events, ...
+It automatically update the dumps and the metadata file.
+
+Here is an example of a config file:
+~~~~
+url = ''
+key = ''
+ssl = True
+outputdir = 'output'
+# filters = {'tag' : 'tlp : white|feed-export|!privint', 'org':'CIRCL'}
+filters = {}
+
+valid_attribute_distribution_levels = ['0', '1', '2', '3', '4', '5'] 
+
+~~~~
+
+#### Consuming feed
+
+As the feed is a simple set of MISP json files, the files can be easily imported
+directly into any MISP instance. The script below processes the manifest file of an OSINT
+feed and reimport them in a MISP directly.
+
+~~~~python
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from pymisp import PyMISP
+import requests
+
+url = 'https://www.circl.lu/doc/misp/feed-osint/'
+osintcircl = requests.get('{}manifest.json'.format(url))
+
+misp = PyMISP('http://misp.test/', 'key', False, 'json')
+for uri in osintcircl.json():
+        req = requests.get('{}{}.json'.format(url,uri))
+        misp.add_event(req.json())
+~~~~
+
+#### ioc-2-misp
+
+Allow to import OpenIOC files into MISP easily. It is also possible to set specific tags on these events.
+ 
+#### Situational Awareness
+
+* attribute_treemap.py generate a treemap showing the distribution of the attributes on the misp instance.
+* tags_* : these functions help having statistics and graphs about the tag repartition.
+

quick-start/README.md

@@ -1,20 +1,92 @@
-Quick Start
------------
+<!-- This is a comment.
+And Justice for All! -->
 
-The Malware Information Sharing Platform (MISP) is the tool which will be used to facilitate the exchange of Indicator of Compromise (IOC) about targeted malware and attacks within your community of trusted members. It is a distributed Indicator of Compromise (IOC) database with technical and non-technical information. Exchanging this information should result in faster detection of targeted attacks and improve the detection ratio, while also reducing the number of false positives.
+# Quick Start
+The Malware Information Sharing Platform (MISP) tool facilitates the exchange of Indicators of Compromise (IOCs) about targeted malware and attacks, within your community of trusted members. MISP is a distributed IOC database containing technical and non-technical information. Exchanging such information should result in faster detection of targeted attacks and improve the detection ratio, whilst also reducing the number of false positives.
 
-Create an Event
-===============
 
-![Create an Event in MISP](figures/quick_create.jpg)
+## Create an Event
 
-Browsing Events
-====================
+![Create an Event in MISP](figures/AddEvent.jpg)
 
-![Browsing Events](figures/quick_browse.jpg)
+You only have to add a few pieces of information to register your Event. Further details will be specified after the Event has been added.
 
-Export Events for Log Search
-============================
+## Describe Event
 
-![Quick Export](figures/quick_export.jpg)
 
+Red is totally normal. No worries.
+
+![Describe Event](figures/AddEventOK.jpg)
+
+Now you can specify the information for your Event (you will need to scroll the window).
+
+### Free-Text Import Tool
+
+![Use Freetext import](figures/AddEventDescription.jpg)
+
+If you have a list of indicators from which you would like to quickly generate attributes then the **Free-text import tool** is
+just what you need. Simply paste your list of indicators (separated by line-breaks) into this tool.
+
+![FreeText Import result](figures/FreeTextImportResult.jpg)
+
+The tool will help you to find similarities between your import and other issues already registered in MISP.
+
+![FreeText Suggest](figures/FreeTextSuggest.jpg)
+
+For example, you can see the ID of all related Events and view their information.
+
+### Tags and Taglist
+
+#### Using existing Data
+
+Another easy way to add information is to use Tags. You can see the result of adding existing Tags (circl:incident-classification=XSS ans circl:incident-classification="information-leak).
+
+![Add Tag](figures/SelectTag.jpg)
+
+By clicking the button, you can add more tags from an existing Taglist.
+
+![Taglist](figures/AddEventTagsList.jpg)
+
+In particular the "Taxonomy Library: circl" Taglist is very complete, as you can see:
+
+![Select Tag from Taglis](figures/AddEventSelectTag.jpg)
+
+#### Make your own Taglist
+
+If you want make your own Taglist, select Add Tag.
+
+![Select Add New Tag](figures/SelectAddNewTag.jpg)
+
+You will see the following window:
+
+![Define Tag](figures/AddTag.jpg)
+
+Then, when you add the new tag it will appear in the Custom Taglist.
+
+### Suggestions
+
+The following attribute types should be added for each Event:
+- ip-src: source IP of attacker
+- email-src: email used to send malware
+- md5/sha1/sha256: checksum
+- Hostname: full host/dnsname of attacker
+- Domain: domain name used in malware
+
+## Browsing Events
+To see your Event, select List Events from the menu Events Action. You can click any row and select a filter.
+
+![Browsing Events](figures/ListEvents.png)
+
+If you click on your Event's number, you can see all the information related to your Event.
+
+![See Event](figures/SeeEvent.jpg)
+
+## Export Events for Log Search
+
+Export functionality is designed to automatically generate signatures for intrusion detection systems. To enable signature generation for a given attribute, the Signature field of this attribute must be set to Yes. Note that not all attribute types are applicable for signature generation, currently we only support NIDS signature generation for IP, domains, host names, user agents etc., and hash list generation for MD5/SHA1 values of file artifacts. Support for more attribute types is planned.
+
+![Quick Export](figures/Export.jpg)
+
+Simply click on any of the following buttons to download the appropriate data for log correlation.
+
+![Select Format](figures/Select Export.jpg)

taxonomy/README.md

@@ -8,32 +8,131 @@ You can access the taxonomy by going into 'Event Actions' and select 'List Taxon
 
 ![MISP Taxonomy index](./figures/taxonomies-index.png)
 
-12 default taxonomies are available:
+The following taxonomies can be used in MISP (as local or distributed tags) or in other tools willing to share common taxonomies among security information sharing tools.
 
-- [Admiralty Scale](https://github.com/MISP/misp-taxonomies/tree/master/admiralty-scale)
-- CIRCL [Taxonomy - Schemes of Classification in Incident Response and Detection](https://github.com/MISP/misp-taxonomies/tree/master/circl)
-- DE German (DE) [Government classification markings (VS)](https://github.com/MISP/misp-taxonomies/tree/master/de-vs)
-- [eCSIRT](https://github.com/MISP/misp-taxonomies/tree/master/ecsirt) and IntelMQ incident classification
-- [EUCI](https://github.com/MISP/misp-taxonomies/tree/master/euci) - EU classified information marking
-- [FIRST CSIRT Case](https://github.com/MISP/misp-taxonomies/tree/master/first_csirt_case_classification) classification
-- [Information Security Marking Metadata](https://github.com/MISP/misp-taxonomies/tree/master/dni-ism) from DNI (Director of National Intelligence - US)
-- [Malware](https://github.com/MISP/misp-taxonomies/tree/master/malware) classification based on a SANS document
-- [NATO Classification Marking](https://github.com/MISP/misp-taxonomies/tree/master/nato)
-- [OSINT Open Source Intelligence - Classification](https://github.com/MISP/misp-taxonomies/tree/master/osint)
-- [TLP - Traffic Light Protocol](https://github.com/MISP/misp-taxonomies/tree/master/tlp)
-- Vocabulary for Event Recording and Incident Sharing [VERIS](https://github.com/MISP/misp-taxonomies/tree/master/veris)
+![Overview of the MISP taxonomies](./figures/taxonomy-explanation.png)
 
-A taxonomy contains a series of tags that can use as normal tags in your MISP instance. The advantage is that you even set a specific tag as being
-exportable. This means that you can export your classification with other MISP instance and share the same taxonomies.
 
-If you want to enable a specific taxonomy, you can click on the cross to enable it. Then you can even cherry-pick the tags you want to use on the system. If you want to use the whole taxonomy, select all and then click on the cross in the top left.
+The following taxonomies are described:
 
-## Contributing a taxonomy
+1. [Admiralty Scale](https://github.com/MISP/misp-taxonomies/tree/master/admiralty-scale): The Admiralty Scale (also called the NATO System) is used to rank the reliability of a source and the credibility of an information.
 
-It is quite easy. Create a JSON file describing your taxonomy as triple tags (e.g. check an existing one like [Admiralty Scale](https://github.com/MISP/misp-taxonomies/tree/master/admiralty-scale)), create a directory matching your name space, put your machinetag file in the directory and pull your request. That's it. Everyone can benefit from your taxonomy and can be automatically enabled in information sharing tools like [MISP](https://www.github.com/MISP/MISP).
+2. [adversary](https://github.com/MISP/misp-taxonomies/tree/master/adversary) An overview and description of the adversary infrastructure.
+
+3. CIRCL [Taxonomy - Schemes of Classification in Incident Response and Detection](https://github.com/MISP/misp-taxonomies/tree/master/circl) CIRCL Taxonomy is a simple scheme for incident classification and area topic where the incident took place.
+
+4. [Cyber Kill Chain](https://github.com/MISP/misp-taxonomies/tree/master/kill-chain) from Lockheed Martin as described in [Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains](http://www.lockheedmartin.com/content/dam/lockheed/data/corporate/documents/LM-White-Paper-Intel-Driven-Defense.pdf).
+
+5. DE German (DE) [Government classification markings (VS)](https://github.com/MISP/misp-taxonomies/tree/master/de-vs) Taxonomy for the handling of protectively marked information in MISP with German (DE) Government classification markings (VS).
+
+6. [DHS CIIP Sectors](https://github.com/MISP/misp-taxonomies/tree/master/dhs-ciip-sectors) DHS critical sectors as described in https://www.dhs.gov/critical-infrastructure-sectors.
+
+7. [Diamond Model for Intrusion Analysis](https://github.com/MISP/misp-taxonomies/tree/master/diamond-model), a phase-based model developed by Lockheed Martin, aims to help categorise and identify the stage of an attack.
+
+8. [Domain Name Abuse](https://github.com/MISP/misp-taxonomies/tree/master/domain-abuse) - taxonomy to tag domain names used for cybercrime. Use europol-incident to tag abuse-activity
+
+9. [eCSIRT](https://github.com/MISP/misp-taxonomies/tree/master/ecsirt) eCSIRT incident classification Appendix C of the eCSIRT EU project including IntelMQ updates.
+
+10. [ENISA](https://github.com/MISP/misp-taxonomies/tree/master/enisa) ENISA Threat Taxonomy - A tool for structuring threat information [as published](https://www.enisa.europa.eu/topics/threat-risk-management/threats-and-trends/enisa-threat-landscape/etl2015/enisa-threat-taxonomy-a-tool-for-structuring-threat-information)
+
+11. [Estimative Language](https://github.com/MISP/misp-taxonomies/tree/master/estimative-language) Estimative language - including likelihood or probability of event based on the Intelligence Community Directive 203 (ICD 203) (6.2.(a)).
+
+12. [EU Marketop and Publicadmin][EU critical sectors](https://github.com/MISP/misp-taxonomies/tree/master/eu-marketop-and-publicadmin) Market operators and public administrations that must comply to some notifications requirements under EU NIS directive.
+
+13. [EUCI](https://github.com/MISP/misp-taxonomies/tree/master/euci) EU classified information (EUCI) means any information or material designated by a EU security classification, the unauthorised disclosure of which could cause varying degrees of prejudice to the interests of the European Union or of one or more of the Member States [as described](http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32013D0488&from=EN).
+
+14. [Europol Incident](https://github.com/MISP/misp-taxonomies/tree/master/europol-incident) EUROPOL class of incident taxonomy
+
+15. [Europol Events](https://github.com/MISP/misp-taxonomies/tree/master/europol-events) - EUROPOL type of events taxonomy
+
+16. [FIRST CSIRT Case](https://github.com/MISP/misp-taxonomies/tree/master/csirt_case_classification) FIRST CSIRT Case Classification.
+
+17. [FIRST Information Exchange Policy (IEP)](https://github.com/MISP/misp-taxonomies/tree/master/iep) framework
+
+18. [French gov information classification system](https://github.com/MISP/misp-taxonomies/tree/master/fr-classif)
+
+19. [Information Security Indicators](https://github.com/MISP/misp-taxonomies/tree/master/information-security-indicators) Information security indicators have been standardized by the [ETSI Industrial Specification Group (ISG) ISI](http://www.etsi.org/technologies-clusters/technologies/information-security-indicators). These indicators provide the basis to switch from a qualitative to a quantitative culture in IT Security Scope of measurements: External and internal threats (attempt and success), user's deviant behaviours, nonconformities and/or vulnerabilities (software, configuration, behavioural, general security framework).
+
+20. [Information Security Marking Metadata](https://github.com/MISP/misp-taxonomies/tree/master/dni-ism) (ISM)  [V13](http://www.dni.gov/index.php/about/organization/chief-information-officer/information-security-marking-metadata) as described by DNI.gov.
+
+21. [Malware](https://github.com/MISP/misp-taxonomies/tree/master/malware_classification) classification based on different categories. Based on a [SANS whitepaper about malware](https://www.sans.org/reading-room/whitepapers/incident/malware-101-viruses-32848).
+
+22. Malware Type and Platform classification based on Microsoft's implementation of the [Computer Antivirus Research Organization (CARO)](https://github.com/MISP/misp-taxonomies/tree/master/ms-caro-malware-full) Naming Scheme and Malware Terminology. Based on [Microsoft Malware naming convntions](https://www.microsoft.com/en-us/security/portal/mmpc/shared/malwarenaming.aspx), [Microsoft Glossary](https://www.microsoft.com/security/portal/mmpc/shared/glossary.aspx), [Microsoft Objective Criteria](https://www.microsoft.com/security/portal/mmpc/shared/objectivecriteria.aspx), and [CARO's definitions](http://www.caro.org/definitions/index.html). Malware families are extracted from Microsoft SIRs since 2008 based on [Microsoft Malware, virus, and threat encyclopedia](https://www.microsoft.com/en-us/security/portal/threat/threats.aspx). Note that SIRs do NOT include all Microsoft malware families.
+
+23. [MISP taxonomy](https://github.com/MISP/misp-taxonomies/tree/master/misp) to infer with MISP behavior or operation.
+
+24. [ms-caro-malware](https://github.com/MISP/misp-taxonomies/tree/master/ms-caro-malware) Malware Type and Platform classification based on Microsoft's implementation of the Computer Antivirus Research Organization (CARO) Naming Scheme and Malware Terminology.
+
+25. [NATO Classification Marking](https://github.com/MISP/misp-taxonomies/tree/master/nato) Marking of Classified and Unclassified materials as described by the North Atlantic Treaty Organization, NATO.
+
+26. [Open Threat Taxonomy v1.1 (SANS)](https://github.com/MISP/misp-taxonomies/tree/master/open-threat) based on James Tarala of SANS (http://www.auditscripts.com/resources/open_threat_taxonomy_v1.1a.pdf).
+
+27. [OSINT Open Source Intelligence - Classification](https://github.com/MISP/misp-taxonomies/tree/master/osint)
+
+28. [The Permissible Actions Protocol - or short: PAP](https://github.com/MISP/misp-taxonomies/tree/master/pap) PAP was designed to indicate how the received information can be used. It's a protocol/taxonomy similar to TLP informing the recipients of information what they can do with the received information.
+
+29. Status of events used in [Request Tracker](https://github.com/MISP/misp-taxonomies/tree/master/rt_event_status).
+
+30. Classification based on [malware stealth](https://github.com/MISP/misp-taxonomies/tree/master/stealth_malware) techniques. Described in [Introducing Stealth Malware Taxonomy](https://vxheaven.org/lib/pdf/Introducing%20Stealth%20Malware%20Taxonomy.pdf)
+
+31. [TLP - Traffic Light Protocol](https://github.com/MISP/misp-taxonomies/tree/master/tlp) The Traffic Light Protocol - or short: TLP - was designed with the objective to create a favorable classification scheme for sharing sensitive information while keeping the control over its distribution at the same time.
+
+32. Vocabulary for Event Recording and Incident Sharing [VERIS](https://github.com/MISP/misp-taxonomies/tree/master/veris)
+
+A taxonomy contains a series of tags that can be used as normal tags in your MISP instance.
+
+Tagging is a simple way to attach a classification to an event. In the early version of MISP, tagging was local to an instance. Classification must be globally used to be efficient. After evaluating different solutions of classification, we build a new scheme using the concept of machine tags.
+
+Taxonomy is a classification of informations. Her, we classified Tags. Taxonomies are implemented in a simple JSON format. Anyone can create their own taxonomy or reuse an existing one.
+
+Taxonomys are in an independent git repository [https://github.com/MISP/misp-taxonomies]
+
+These can be **freely reused** and **integrated** in other threat intel tools.
+
+The advantage is that you even set a specific tag as being
+exportable. This means that you can **export** your classification with other MISP instance and **share** the same taxonomies. Tagging is a simple way to attach a classification to an event.
+
+**Classification must be globally used to be efficient.**
+
+If you want to enable a specific taxonomy, you can click on the cross to enable it.
+
+![enableTaxonomy](./figures/enableTaxonomy.png)
+
+Then you can even cherry-pick the tags you want to use on the system. If you want to use the whole taxonomy, select all and then click on the cross in the top left.
+
+## Contributing to Taxonomy
+
+It is quite easy. Create a JSON file describing your taxonomy as triple tags.
+
+![](./figures/writeTaxonomy.png)
+
+![](./figures/writeTaxonomy2.png)
+
+ (e.g. check an existing one like [Admiralty Scale](https://github.com/MISP/misp-taxonomies/tree/master/admiralty-scale)), create a directory matching your name space, put your machinetag file in the directory and pull your request. Publishing your taxonomy is as easy as a simple git pull request on misp-taxonomies (https://github.com/MISP/misp-taxonomies). That's it. Everyone can benefit from your taxonomy and can be automatically enabled in information sharing tools like [MISP](https://www.github.com/MISP/MISP).
+
+
+## Reserved Taxonomy
+
+ The following taxonomy namespaces are reserved and used internally to MISP.
+
+ - [galaxy](./galaxy/) mapping taxonomy with cluster:element:"value".
+
+## Adding Taxonomy in MISP
+
+How are taxonomies integrated in MISP?
+
+MISP administrator have only to import (or even cherry pick) the namespace or predicates they want to use as tag.
+
+Tags can be exported to other instances.
+
+Tags are also accessible via the MISP REST API.
+
+For more information, "[Information Sharing and Taxonomies Practical Classification of Threat Indicators using MISP](https://www.circl.lu/assets/files/misp-training/first2016/2-MISP-Taxonomies.pdf)" presentation given to the last MISP training in Luxembourg.
 
 ## Adding a private taxonomy
 
+<!-- ========================== I don't know if this part is nor yet relevant? ================================================= -->
+
 ~~~~ shell
 $ cd /var/www/MISP/app/files/taxonomies/
 $ mkdir privatetaxonomy
@@ -44,4 +143,71 @@ Create a JSON file Create a JSON file describing your taxonomy as triple tags.
 
 Once you are happy with your file go to MISP Web GUI taxonomies/index and update the taxonomies, the newly created taxonomy should be visible, now you need to activate the tags within your taxonomy.
 
+## How using Taxonomy in MISP
 
+### Filtering the distribution of events among MISP instances
+
+Applying rules for distribution based on tags:
+
+### MISP Taxonomies - tools
+
+[machinetag.py](https://github.com/MISP/misp-taxonomies/blob/master/tools/machinetag.py) is a parsing tool to dump taxonomies expressed in Machine Tags (Triple Tags) and list all valid tags from a specific taxonomy.
+
+~~~~shell
+% cd tools
+% python machinetag.py
+        admiralty-scale:source-reliability="a"
+        admiralty-scale:source-reliability="b"
+        admiralty-scale:source-reliability="c"
+        admiralty-scale:source-reliability="d"
+        admiralty-scale:source-reliability="e"
+        admiralty-scale:source-reliability="f"
+        admiralty-scale:information-credibility="1"
+        admiralty-scale:information-credibility="2"
+        admiralty-scale:information-credibility="3"
+        admiralty-scale:information-credibility="4"
+        admiralty-scale:information-credibility="5"
+        admiralty-scale:information-credibility="6"
+        ...
+~~~~
+
+### Other use cases using MISP taxonomies
+
+Tags can be used to set events for further processing by external tools (e.g. VirusTotal auto-expansion using Viper).
+
+Ensuring a classification manager classes the events before release (e.g. release of information from air-gapped/classified networks).
+
+Enriching IDS export with tags to fit your NIDS deployment.
+
+## MISP warning lists: The dilemma of false-positive
+
+- False-positive is a common issue in threat intelligence sharing.
+
+- It’s often a contextual issue:
+   - false-positive might be different per community of users sharing
+information.
+
+   - organization might have their own view on false-positive.
+
+- Based on the success of the MISP taxonomy model, we build misp-warninglists. They are lists of well-known indicators that can be
+associated to potential false positives, errors or mistakes. They are Simple JSON files.
+
+![MISP warning lists](./figures/MISPwarninglist.png)
+
+The warning lists are integrated in MISP to display an info/warning box at the event and attribute level. This can be enabled at MISP instance level. Default warning lists can be enabled or disabled like known public
+resolver, multicast IP addresses, hashes for empty values, rfc1918, TLDs or known google domains. The warning lists can be expanded or added in JSON locally or via
+pull requests (https://github.com/MISP/misp-warninglists). Warning lists can be also used for critical or core infrastructure
+warning, personally identifiable information...
+
+## Future functionalities related to MISP taxonomies
+
+- Sighting support (thanks to NCSC-NL) is integrated in MISP allowing to auto expire IOC based on user detection.
+
+- Adjusting taxonomies (adding/removing tags) based on their score or visibility via sighting.
+
+- Simple taxonomy editors to help non-technical users to create their
+taxonomies.
+
+- Filtering mechanisms in MISP to rename or replace taxonomies/tags at pull and push synchronisation.
+
+- More public taxonomies to be included