diff --git a/administration/README.md b/administration/README.md index f651764..b15dd1e 100644 --- a/administration/README.md +++ b/administration/README.md @@ -86,9 +86,9 @@ Site admins can use the "Contact users" feature to send all or individual user a * **Action:** This defines the e-mail type, which can be a custom message or a password reset. Password resets automatically include a new temporary password at the bottom of the message and will automatically change the user's password accordingly. * **Subject:** In the case of a custom e-mail, you can enter a subject line here. * **Recipient:** The recipient toggle lets you contact all your users, a single user (which creates a second drop-down list with all the e-mail addresses of the users) and potential future users (which opens up a text field for the e-mail address and a text area field for a GnuPG public key). -* **Custom message checkbox:** This is available for password resets or for welcome message, you can either write your own message (which will be appended with a temporary key and the signature), or let the system generate one automatically. +* **Custom message checkbox:** This is available for password resets and for welcome messages. You can either write your own message (which will be appended with a temporary key and the signature), or let the system generate one automatically. -Keep in mind that all e-mails sent through this system, in addition to your own message, will be signed in the name of the instance's host organisation's support team, the e-ail will also include the e-mail address of the instance's support (if the contact field is set in the bootstrap file), and will include the instance's GnuPG signature for users that have a GnuPG key set (and thus are eligible for an encrypted e-mail). +Keep in mind that all e-mails sent through this system, in addition to your own message, will be signed in the name of the instance's host organisation's support team, the e-mail will also include the e-mail address of the instance's support (if the contact field is set in the bootstrap file), and will include the instance's GnuPG signature for users that have a GnuPG key set (and thus are eligible for an encrypted e-mail). :warning: GnuPG instance key is the GnuPG key used by the MISP instance and which is only used to sign notification. The GnuPG key used in the MISP instance must not be used anywhere else and should not be valuable. @@ -113,9 +113,9 @@ To add a new organisation, click on the "Add Organisation" button in the adminis * **Type of organisation:** Define the type of the organisation. * **Contacts:** You can add some contact details for the organisation. -#### Listing all organisation +#### Listing all organisations -To list all current organisation of the system, just click on List Organisations under the administration menu to the left. There are 3 tabs in this view to filter local organisations, remote organisations or both. The default view displays local organisations. For all views the following columns of information are available: +To list all current organisations of the system, just click on List Organisations under the administration menu to the left. There are 3 tabs in this view to filter local organisations, remote organisations or both. The default view displays local organisations. For all views the following columns of information are available: ![List of organisations.](figures/list_org.png) @@ -130,6 +130,7 @@ To list all current organisation of the system, just click on List Organisations * **Contacts:** Contacts of organisation. * **Added by:** Login of the user who added the organisation * **Local:** Flag defined if the organisation is local or remote. +* **Users:** The amount of users on this instance belonging to the organisation. * **Actions:** There are 3 options available: edit, delete or display an organisation's information. These options are also available on the left menu when you are on the display view. * **Edit Organisation:** Same options of create organisation's view. ![Edit organisation.](figures/edit_org.png) @@ -139,7 +140,7 @@ To list all current organisation of the system, just click on List Organisations ![View organisation.](figures/view_org.png) #### Merge organisations -Merge Organisation menu is available only in the organisation view, under the left menu. Merge one organisation to another will transfer all users and data from one to another. On the left the organisation to merge, on the right the target one. +Merge Organisation menu is available only in the organisation view, under the left menu. Merging one organisation into another will transfer all users and data from one organisation to a different one. The organisation of which the users and data will be transferred is displayed on the left, the target organisation is displayed on the right. ![Merge organisations.](figures/merge_org.png) @@ -149,10 +150,10 @@ Merge Organisation menu is available only in the organisation view, under the le Privileges are assigned to users by assigning them to rule groups. Rule groups use one of four options determining what they can do with events as well as four additional privilege elevation settings. These are the four options to edit the full options available in the Roles section: Read Only, Manage My Own Events, Manage Organisation Events, Manage & Publish Organisation Events. A short description is provided below: -* **Read Only:** This allows a user to browse events that his organisation has access to, but doesn't allow any changes to be made to the database. -* **Manage My Own Events:** The second option, gives its users the rights to create, modify or delete their own events, but they cannot publish them. +* **Read Only:** Allows a user to browse events that his organisation has access to, but doesn't allow any changes to be made to the database. +* **Manage My Own Events:** Allows users to create, modify or delete their own events, but they cannot publish them. * **Manage Organization Events:** Allows users to create events or modify and delete events created by a member of their organisation. -* **Manage & Publish Organisation Events:** This last setting, gives users the right to do all of the above and also to publish the events of their organisation. +* **Manage & Publish Organisation Events:** Gives users the right to do all of the above and to publish the events of their organisation. The extra permissions are defined below: @@ -176,7 +177,7 @@ When creating a new role, you will have to enter a name for the role to be creat #### Listing roles -By clicking on the List Roles button, you can view a list of all currently registered roles and a list of the permission flags enabled for each. In addition, you can find buttons that allow you to edit and delete said roles. Keep in mind that you will need to first remove every member from a role before you can delete it. +By clicking on the List Roles button, you can view a list of all currently registered roles and their enabled permissions. In addition, you can find buttons that allow you to edit and delete said roles. Keep in mind that you will need to first remove every member from a role before you can delete it. ![You can Edit or Delete roles using the action buttons to the right in each row. Keep in mind that in order to Delete a role, all members of a Role must be removed from said role before it can be deleted.](figures/list_roles.png) @@ -219,7 +220,7 @@ The settings and diagnostics tool is split up into several aspects, all accessib * **Misc settings**: Settings controlling debug options, please ensure that debug is always disabled on a production system. * **Diagnostics**: The diagnostics tool checks if all directories that MISP uses to store data are writeable by the apache user. Also, the tool checks whether the STIX libraries and GnuPG are working as intended. * **Workers**: Shows the background workers (if enabled) and shows a warning if they are not running. Admins can also restart the workers here. -* **Download report**: Download a report in JSON format, compiled of all of the settings visible in the tool. +* **Download report**: Download a report of all the settings visible in the tool, in JSON format. ![The settings tabs explained.](figures/settings_2.png) @@ -228,13 +229,13 @@ Each of the setting pages is a table with each row representing a setting. Colou * **Setting**: The setting name. * **Value**: The current value of the setting. * **Description**: A description of what the setting does. -* **Error Message**: If the setting is incorrect / not set, then this field will let the user know what is wrong. +* **Error Message**: If the setting is incorrect / not set, this field will let the user know what is wrong. ![The workers tab.](figures/settings_3.png) The workers tab shows a list of the workers that MISP can use. You can restart workers using the "restart all workers" button. If the button doesn't work, make sure that the workers were started using the apache user. This can however only be done using the command line, refer to the INSTALL.txt documentation on how to let the workers automatically start on each boot. -* **Worker Type**: The worker type is determined by the queue it monitors. MISP currently has 5 queues (cache, default, prio, email and a special _schdlr_ queue). +* **Worker Type**: The worker type is determined by the queue it monitors. MISP currently has 6 queues (cache, default, prio, email, update and a special _schdlr_ queue). * **Worker Id**: The ID is made up of the machine name, the PID of the worker and the queue it monitors. * **Status**: Displays OK if the worker is running. If the _schdlr_ worker is the only one not running, make sure that you copy the config file into the cakeresque directory as described in the INSTALL.txt documentation. @@ -258,6 +259,12 @@ The workers tab shows a list of the workers that MISP can use. You can restart w Interdependence: +**update** + + Role: + Interdependence: + + **prio** Role: @@ -265,9 +272,11 @@ The workers tab shows a list of the workers that MISP can use. You can restart w **scheduler** + Role: Interdependence: + #### Workers dead Even if the workers are dead, any actions related to them are on-hold. Nothing is lost. diff --git a/automation/README.md b/automation/README.md index 8a6e529..3aea3fd 100644 --- a/automation/README.md +++ b/automation/README.md @@ -82,6 +82,77 @@ curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" {"name":"Not Found","message":"Not Found","url":"\/servers\/gaaa"} ~~~~ +## Search + +It is possible to search in the database for a list of attributes or events based on a list of criterias. + +To return attributes or events in a desired format, use the following URL and header settings: + +URL: +~~~~ +YOUR_MISP_URL/attributes/restSearch +YOUR_MISP_URL/events/restSearch +~~~~ + +Headers: +~~~~ +Accept: application/json +Content-type: application/json +Authorization: YOUR_API_KEY +~~~~ + +The next feature to take care of then is the body of the query. This is where you are going to put your filters. +As an example, if we want to export all the IP addresses that have a TLP marking and not marked as TLP:red, you can find below the corresponding filters to use: +~~~~json +{ + "returnFormat": "json", + "type": { + "OR": [ + "ip-src", + "ip-dst" + ] + }, + "tags": { + "NOT": [ + "tlp:red" + ], + "OR": [ + "tlp:%" + ] + } +} +~~~~ + +Find below a non exhaustive list of parameters that can be used to filter data in your search (some parameters specific to given export formats are not mentioned): +- **returnFormat**: Set the return format of the search (Currently supported: json, xml, openioc, suricata, snort - more formats are being moved to restSearch with the goal being that all searches happen through this API). Can be passed as the first parameter after restSearch or via the JSON payload. +- **limit**: Limit the number of results returned, depending on the scope (for example 10 attributes or 10 full events). +- **page**: If a limit is set, sets the page to be returned. page 3, limit 100 will return records 201->300). +- **value**: Search for the given value in the attributes' value field. +- **type**: The attribute type, any valid MISP attribute type is accepted. +- **category**: The attribute category, any valid MISP attribute category is accepted. +- **org**: Search by the creator organisation by supplying the organisation identifier. +- **tags**: To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. +- **quickfilter**: Enabling this (by passing "1" as the argument) will make the search ignore all of the other arguments, except for the auth key and value. MISP will return an xml / json (depending on the header sent) of all events that have a sub-string match on value in the event info, event orgc, or any of the attribute value1 / value2 fields, or in the attribute comment. +- **from**: Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event. +- **to**: Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event. +- **eventid**: The events that should be included / excluded from the search +- **withAttachments**: If set, encodes the attachments / zipped malware samples as base64 in the data field within each attribute +- **metadata**: Only the metadata (event, tags, relations) is returned, attributes and proposals are omitted. +- **uuid**: Restrict the results by uuid. +- **publish_timestamp**: Restrict the results by the timestamp of the last publishing of the event. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]). +- **last**: (Deprecated synonym for publish_timestamp) Restrict the results by the timestamp of the last publishing of the event. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]). +- **timestamp**: Restrict the results by the timestamp (last edit). Any event with a timestamp newer than the given timestamp will be returned. In case you are dealing with /attributes as scope, the attribute's timestamp will be used for the lookup. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]). +- **published**: Set whether published or unpublished events should be returned. Do not set the parameter if you want both. +- **enforceWarninglist**: Remove any attributes from the result that would cause a hit on a warninglist entry. +- **to_ids**: By default (0) all attributes are returned that match the other filter parameters, irregardless of their to_ids setting. To restrict the returned data set to to_ids only attributes set this parameter to 1. You can only use the special "exclude" setting to only return attributes that have the to_ids flag disabled. +- **deleted**: If this parameter is set to 1, it will return soft-deleted attributes along with active ones. By using "only" as a parameter it will limit the returned data set to soft-deleted data only. +- **includeEventUuid**: Instead of just including the event ID, also include the event UUID in each of the attributes. +- **event_timestamp**: Only return attributes from events that have received a modification after the given timestamp. The input can be a timetamp or a short-hand time description (7d or 24h for example). You can also pass a list with two values to set a time range (for example ["14d", "7d"]). +- **sgReferenceOnly**: If this flag is set, sharing group objects will not be included, instead only the sharing group ID is set. +- **eventinfo**: Filter on the event's info field. +- **searchall**: Search for a full or a substring (delimited by % for substrings) in the event info, event tags, attribute tags, attribute values or attribute comment fields. +- **attackGalaxy**: Select the ATT&CK matrix like galaxy to use when using returnFormat = attack. Defaults to the Mitre ATT&CK library via mitre-attack-pattern. + ## Events management ### /events diff --git a/faq/README.md b/faq/README.md index f841f31..6daa5fe 100644 --- a/faq/README.md +++ b/faq/README.md @@ -902,6 +902,120 @@ Created symlink from /etc/systemd/system/multi-user.target.wants/php73-php-fpm.s A galaxy can be assigned like a tag. You can use the add tag function and copy the full conntector-tag. Example `misp-galaxy:ransomware=“Locky”`, which can be found in `/galaxy_clusters/view/` +## Updating PHP from 7.2 to 7.4.5 on Ubuntu 18.04 + +### Installation + +1. Disable and Uninstall Currently Installed SSDEEP +```bash +sudo phpdismod ssdeep +sudo pecl uninstall ssdeep +sudo apt purge ssdeep +sudo rm -rf /etc/php/7.2/mods-available/ssdeep.ini +``` + +2. Install PHP 7.4.5 +```bash +sudo apt install software-properties-common -qy +sudo add-apt-repository ppa:ondrej/php -y +sudo apt update +sudo apt install -qy \ + libapache2-mod-php7.4 \ + php7.4 \ + php7.4-cli \ + php7.4-dev \ + php7.4-json \ + php7.4-xml \ + php7.4-mysql \ + php7.4-opcache \ + php7.4-readline \ + php7.4-mbstring \ + php-redis \ + php-gnupg \ + php-gd +sudo apt update +sudo apt upgrade -y +``` + +3. Install SSDEEP +```bash +cd /usr/local/src +sudo rm -rf ssdeep-2.14.1.tar.gz ssdeep-2.14.1 +sudo wget https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz +sudo tar zxvf ssdeep-2.14.1.tar.gz +cd ssdeep-2.14.1 +sudo ./configure --datadir=/usr --prefix=/usr --localstatedir=/var --sysconfdir=/etc +sudo make +sudo make install +``` + +4. Test SSDEEP +```bash +ssdeep -h +``` + +5. Install ssdeep_php +```bash + sudo pecl channel-update pecl.php.net + sudo pecl install ssdeep +``` + +6. Enable SSDEEP in both 7.2 and 7.4 (** as root** `sudo su`) +```bash +echo 'extension=ssdeep.so' > /etc/php/7.2/mods-available/ssdeep.ini +echo 'extension=ssdeep.so' > /etc/php/7.4/mods-available/ssdeep.ini +``` + +7. Enable SSDEEP PHP Mod +```bash +sudo phpenmod ssdeep +``` + +8. Set PHP 7.4.5 to default PHP +```bash +sudo a2dismod php7.2 +sudo a2enmod php7.4 +sudo update-alternatives --set php /usr/bin/php7.4 +``` + +9. [Optional] Set better values for defaults +```bash +sudo sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.4/apache2/php.ini ; \ +sudo sed -i "s/memory_limit = 128M/memory_limit = 2048M/" /etc/php/7.4/apache2/php.ini ; \ +sudo sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 500M/" /etc/php/7.4/apache2/php.ini ; \ +sudo sed -i "s/post_max_size = 8M/post_max_size = 500M/" /etc/php/7.4/apache2/php.ini ; \ +sudo sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.4/cli/php.ini ; \ +sudo sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 500M/" /etc/php/7.4/cli/php.ini ; \ +sudo sed -i "s/post_max_size = 8M/post_max_size = 5000M/" /etc/php/7.4/cli/php.ini ; +``` + +10. Restart Apache to implement changes +```bash +sudo sudo systemctl restart apache2 +``` + +### Verification of php 7.2 to 7.4 + +1. **Administration** > **Server Settings & Maintenance** + +2. **Diagnostics** + +3. Scroll down to the **PHP Settings** section and verify + + +### What are the required steps after a MISP installation to have a properly running instance? + +- First login with the installation credentials and change the password immediatly (especially if your instance is publicly accessible) +- Set the base_url to the hostname of your machine (apache virtualhost name) +- Create a new organisation which will be the host organisation running the MISP instance +- Set the new organisation in `MISP.host_org_id` to replace the default one +- Set messages like `MISP.footermidleft` and alike to a proper message to help your users +- Create a new user as `admin` role with the new organisation +- Log with the new user, if successful, remove the default user used during the installation such as `admin@admin.test` +- Select and enable required taxonomies for your sharing community +- Select and enable the external feeds (as caching only if you don't want full events but you can get the full feeds too) +- Select and enable the warning-list (if you don't know what to enable, select all) +- Add the remote MISP instances where you have access to (either caching only or full pull if you want the complete events)