General questions
This FAQ is regularly published in misp-book.
1. Where can I get support?
If you have feature requests or you found a bug you can open a ticket on MISP's github repository issues.
If you want to discuss about something related to MISP, want help from the community, etc... You have the MISP Users mailing list or the MISP Gitter channel.
A number of companies are also offering custom development, consulting, and support around MISP, please check the support page of the MISP Project website.
2. What are the hardware requirements?
From a hardware perspective, MISP's requirements are quite humble, a web server with 2+ cores and 8-16 GB of memory should be plenty, though more is always better of course. A lot of it depends on the data set and the number of users you are dealing with.
Some considerations for what might affect your requirements:
- How highly correlating your data is (correlations are generally memory and computation intensive), if you have a high correlation ratio, consider either lowering this with better management of the data (correlate flag on attributes) or by increasing the memory and CPU available
- Number of samples and attachments directly affect the disk usage
- Concurrent user counts affect the memory usage and CPU utilisation, especially if you have a list of API users querying MISP frequently
- Number of remote feeds and servers cached and kept in memory will also increase the memory requirements of the system.
- the amount of logging / activity / longevity of the server can increase the disk requirements both on the database as well as the local log file stash
To give some indications of some of the operational servers:
- 16GB memory and 2 vcpus are quite common for smaller sharing hub and end-point MISPs
- large sharing communities (such as the CIRCL private sector community) use 128 GB of memory with 8 physical CPU cores on modern Xeon CPUs
- The COVID misp community (https://covid-19.iglocska.eu) runs on 8GB of memory with 4 vcpus and serves over a thousand users
- The training instances we use run on a meager 2GB of memory and a single vcpu (though we would not recommend using this for anything besides trainings / experimentation)
For software we recommend a standard LAMP stack on top of Ubuntu 18.04. For details on the exact dependencies please refer to the installation guide as well as the requirements for the MISP modules.
During a Hackathon a small tool called MISP-Sizer was conceived. It will give you a very rough idea on what requirements are if you have a bigger installation. source-code is here
Specific questions
1. Can I configure MISP encrypted notification emails to contain more information in the subject?
The setting MISP.extended_alert_subject allows you to have an extended subject. One word of warning though. If you’re using encryption : the subject will not be encrypted. Be aware that you might leak some sensitive information this way. Below is an example how the two subject types look like. First with the option disabled, then with the option enabled.
Event 7 - Low - TLP Amber
Event 8 - OSINT - Dissecting XXX... - Low - TLP Amber
(Taken from http://www.vanimpe.eu/2015/05/31/getting-started-misp-malware-information-sharing-platform-threat-sharing-part-3/)
2. How can I restart the workers?
The workers can be restarted from the web interface:
administration -> server settings -> workers -> restart all
You can also follow the manual process below.
If you are on Ubuntu / Debian based systems:
sudo su -l www-data -s /bin/bash -c "bash /var/www/MISP/app/Console/worker/start.sh"
If you are on RHEL / Fedora based systems:
su -s /bin/bash apache -c 'bash /var/www/MISP/app/Console/worker/start.sh'
3. How can I redirect HTTP to HTTPs?
<VirtualHost *:80>
ServerAdmin misp@misp.misp
ServerName misp.misp.misp
ServerAlias misp-int.misp.misp
Redirect permanent / https://misp.misp.misp
LogLevel warn
ErrorLog /var/log/apache2/misp.local_error.log
CustomLog /var/log/apache2/misp.local_access.log combined
ServerSignature Off
</VirtualHost>
<VirtualHost *:443>
ServerAdmin misp@misp.misp
ServerName misp.misp.misp
ServerAlias misp-int.misp.misp
DocumentRoot /var/www/MISP/app/webroot
<Directory /var/www/MISP/app/webroot>
Options -Indexes
AllowOverride all
Order allow,deny
allow from all
</Directory>
SSLEngine On
SSLCertificateFile /etc/ssl/misp.misp.misp/misp.crt
SSLCertificateKeyFile /etc/ssl/misp.misp.misp/misp.key
SSLCertificateChainFile /etc/ssl/misp.misp.misp/mispCA.crt
LogLevel warn
ErrorLog /var/log/apache2/misp.local_error.log
CustomLog /var/log/apache2/misp.local_access.log combined
ServerSignature Off
</VirtualHost>
(Taken from http://www.vanimpe.eu/2015/05/31/getting-started-misp-malware-information-sharing-platform-threat-sharing-part-3/)
4. When I try to access my new installation, I am redirected to localhost:8443 and get an error.
By default, MISP runs on a local instance and is setup for local access upon installation. This allows you to setup security and customizations before making it available elsewhere. If you would like to access the MISP instance from a remote host (including another VM host/client), assign an IP to the MISP host and point your browser accordingly. Upon login, you may get the “localhost:8443” redirection. Change that piece of the URL back to the IP assigned to the MISP host (or associated DNS name) and refresh the browser. Once in, go to Administration - Server Settings and Maintenance - MISP settings. You can change the top two items to your MISP IP or DNS name and the redirect will start using that address instead of 'localhost'.
5. How can I define the default sharing level?
MISP allows you to define the group of people with whom you want to share your threat data. If you do not set it to your preferred default then it’s likely that at one given moment you’ll make an error and share your intel with the wrong group. Defining the sharing level is done with the setting default_event_distribution in the configuration file. There are three levels
0 : Your organisation only (default)
1 : This community only
2 : Connected communities
3 : All communities
You can set a similar configuration setting for the attributes. The setting default_attribute_distribution has the same values as default_event_distribution. Additionally it has the value event which allows the attribute to get the setting from the event to which it belongs.
(Taken from http://www.vanimpe.eu/2015/05/31/getting-started-misp-malware-information-sharing-platform-threat-sharing-part-3/)
6. How can I add organisation logos?
MISP can be made more appealing to the eye by adding some graphics. You can set your organisation logo by adding an image (.png) that has the same name as your organisation in the directory /var/www/MISP/app/webroot/img/orgs/. Similarly you can add a footer logo. Add an image to the directory /var/www/MISP/app/webroot/img/custom/ and define the footer logo in the config file (config.php).
Another way of doing it is by logging in your MISP instance with administrator rights, go in the menu Administration, sub-menu Server Settings, tab Manage files.
(Partially taken from http://www.vanimpe.eu/2015/05/31/getting-started-misp-malware-information-sharing-platform-threat-sharing-part-3/)
7. All workers are starting correctly except schdlr . How can I fix this?
This can happen if the FQDN of the server hosting the instance has changed. A way to fix this is to flush temporary data stored in redis. This can be done by logging in redis, for example when logging in with redis-cli, and issuing a flushall command.
8. How can I import data directly from PDF reports?
You can use a generic script called IOC parser (https://github.com/armbues/ioc_parser) or use a script published by Palo Alto to convert IOC parser output to a MISP event (https://github.com/PaloAltoNetworks-BD/report_to_misp/). You have also the option to select all the text and paste it in the free-text import form.
Another option is the new OCR import module that can be used via the import modules. You will need to install the OCR software tesseract.
9. I am having trouble updating beyond version 2.4.50 (stuck loading any page beyond the login), what can I do?
This is most likely due to the fact that MISP did not clean up expired sessions prior to version 2.4.51 automatically and relied on a site admin occasionally cleaning it up using the button found on the diagnostics page. Once you upgrade to 2.4.51, MISP will try to cull the table with each page load by a site-admin, which in some cases if the table has grown to extreme sizes it will get stuck on. To resolve the issue, log into mysql:
mysql -u [misp-db-user-name] -p [misp-db-name];
and execute the following commands:
DROP cake_sessions;
CREATE TABLE IF NOT EXISTS `cake_sessions` (
`id` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '',
`data` text COLLATE utf8_bin NOT NULL,
`expires` int(11) NOT NULL,
PRIMARY KEY (`id`),
INDEX `expires` (`expires`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
After this everything should work and the session table will be trimmed each time a site admin loads a page.
10. I have many failed jobs when doing email notification. What should I do?
This is most probably due to some encryption failing for some users. We strongly advise to review the current PGP keys and to ensure that they keys are not expired or not supported. The keys can be reviewed at the following location in MISP:
https://<YOUR MISP URL>/users/verifyGPG
11. I have issues with pushing events
-
What does connection test for the specific server telling?
-
Is the event you assume is ready to push published?
-
Is the distribution level set to not to restricted?
-
Have you enabled push in the server config you want to push to?
-
Do you have any limitations to the push rules e.g. limited to a certain TLP Level tag?
-
What is written in your job log?
https:///jobs/index
Have a look at: /var/www/MISP/app/tmp/logs and /var/log/apache2/misp
12. I have many users or API access, what's the best PHP session handler?
We strongly recommend production-level MISP installation to rely on PHP session in Redis. As Redis is already part of a standard MISP setup, we recommend to enable the redis session handling. To configure the redis session handling in PHP, edit :
session.save_handler = redis
session.save_path = "tcp://127.0.0.1:6379
13. Upgrading from MISP 2.4.65 to MISP 2.4.66 - Unable to merge due to the Composer file.
In MISP 2.4.66, Composer is included by default to avoid the risk of downloading a rogue PHP Composer version (if the composer repository is compromised or MiTM are performed) via the download and php execution. But when upgrading (via a git pull), the git merge process might complain about the composer phar file still being there. You can safely remove that file and git pull origin 2.4 again.
14. Is there TAXII support?
A TAXII 1 implementation can be found at https://github.com/MISP/MISP-Taxii-Server .
This is mostly a TAXII server hooked up to MISP, meant to receive STIX files to its in box and uploading them to MISP.
There is also an experimental feature to push MISP events to the TAXII server when they're published - that's in scripts/push_published_to_taxii.py. It seems to work, but may occasionally re-upload duplicate events to MISP.
TAXII 2 support will be provided in the future once the specification, which is at time of writing in draft, reaches a stable form.
15. Wipe MISP data - Remove all data
If you need to start from scratch with your MISP database and remove all data you can use the misp-wipe script provided in the tools/ folder.
16. Constantly acknowledging my self-signed certificate drives me nuts
You want to add it in 2 places: Your browser(s) and your OS.
The following steps can be performed on the CLI to install the Certificate:
sudo mkdir -m 0755 /usr/local/share/ca-certificates/MISP
sudo cp /etc/ssl/private/misp.local.crt /usr/local/share/ca-certificates/MISP
sudo chmod 0644 /usr/local/share/ca-certificates/MISP/misp.local.crt
sudo update-ca-certificates
For the Chrome Browser:
- Visit: "Advanced Settings" -> chrome://settings/?search=Manage+certificates
- Scroll down to: Manage Certificates (click)
- Select: "Authorities"
- Click: "Import"
- Browse to your .crt file and import it.
- On the next screen tick: "Trust this certificate for identifying websites"
- Done, enjoy the new gained quality of life
Note: Chrome might expect a Subject Alternative Name make sure you created your certificate with '-extension san'.
To allow insecure localhost connections enable this option: chrome://flags/#allow-insecure-localhost
Sources: CLI and Chrome/Chrome insecure localhost
17. How can I change the theme?
MISP uses bootstrap.css the specific CSS file can be found on a typical MISP install at /var/www/MISP/app/webroot/css/bootstrap.css.
You can customize this for your own needs. There are also pre-made boostrap themes which you can use as-is or build upon.
Before making any changes, confirm the version of boostrap currenlty used by running head -5 /var/www/MISP/app/webroot/css/bootstrap.css. You can find themes on sites like Bootswatch.
To replace the current theme with a theme you found on bootsplash, run: wget https://bootswatch.com/2/readable/bootstrap.css -O /var/www/MISP/app/webroot/css/bootstrap.css , replacing the URL as needed.
Some bootswatch themes applied on MISP:
- https://i.imgur.com/usONTLk.png
- https://i.imgur.com/5XMjB7o.png
- https://i.imgur.com/5gc57VU.png
- https://i.imgur.com/4AJCPgf.png
- https://i.imgur.com/JuMGm8U.png
- https://i.imgur.com/v1Wu6xW.png
18. How can I deal with a MISP instance that has pulled in feeds over and over into new events, generating hundreds of GBs of junk correlations, rendering the instance unusable?
Step 1: ensure that all your CSV/freetext source_format feeds are using the fixed event setting. If you want to make sure this is the case, you can run this SQL query instead of doing it manually:
UPDATE feeds SET fixed_event = 1 WHERE source_format="csv" OR source_format="freetext";
Step 2: purge all of your correlations (this will make the next steps much faster), for which you have two methods at your disposal:
- either go to your administration -> server settings -> MISP tab and set
MISP.completely_disable_correlationto true - via MYSQL run
TRUNCATE correlations;
Step 3: purge all of your feed data that have been pulled into multiple events. The easiest way of doing this: check which feeds are enabled (ignore misp source format feeds, they are not causing issues) and note down the IDs. Afterwards, use the CLI cleanup tool to remove all the feed events:
/var/www/MISP/app/Console/cake Admin purgeFeedEvents [user_id] [feed_id]
Execute this for each feed that you had enabled, replacing user_id with your admin user's ID and feed_id with the individual feed IDs on your list.
Step 4: recorrelate your data, depending on which method you've used in Step 2 you have two options:
- either go to your administration -> server settings -> MISP tab and set
MISP.completely_disable_correlationto false - recorrelate your current data-set via the recorrelate attributes tool on
/pages/display/administration
19. I can no longer log in. How do I reset the admin password?
You can reset the password via the console. See https://github.com/MISP/MISP/issues/1160
/var/www/MISP/app/Console/cake Password [email] [password]
20. I can no longer sync large events / fetch the event index of a remote instance while syncing
If your logs show that the remote is not in fact sending JSON as a response, one thing to check is whether the remote uses a reverse proxy that has message size limitations. nginx seems to have such a default limitation, so even if your php post_max_size is configured correctly, messages may fail and return an error message from the reverse proxy instead of the desired JSON.
In the case of nginx the setting is client_max_body_size, make sure that you set it to a sufficiently high number.
21. How do I install a new (self-signed/organization internal) to the MISP trust store? How do I update what root CA's MISP trusts? How do I make MISP use my system certificate/trust store?
MISP will default to the bundled Mozilla certificate bundle shipped with the CakePHP framework, which may be rather stale. It may sometimes be useful to configure MISP to use the host system trust store.
If you wish to use an alternate bundle, you can set MISP.ca_path to the path to the bundle to use. This setting can only be modified via the CLI. This is also the recommended way to add new certificates for MISP to trust.
The MISP settings GUI may complain that MISP.ca_path is not set to a PEM file. Both the suggested system certificate stores are PEM files with a *.crt filename extension.
Please be aware that after changing this setting you will have to terminate and restart all your MISP workers.
Debian/Ubuntu and deratives:
To add a new (i.e. self-signed) certificate:
# cp new-root.crt /usr/share/ca-certificates/
# update-ca-certificates
Using system certificates:
Set MISP.ca_path to /etc/ssl/certs/ca-certificates.crt. This must be changed via the cake CLI or directly in config.php, it can not be changed in the web GUI.
Red Hat/Rocky/Alma and deratives:
To add a new (i.e. self-signed) certificate:
# cp new-root.crt /usr/share/pki/ca-trust-source/anchors/
# update-ca-trust
Using system certificates:
Set MISP.ca_path to /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt. This must be changed via the cake CLI or directly in config.php, it can not be changed in the web GUI.
22. How do I solve the Connection refused error with Redis / CakeResque?
In this troubleshooting guide we assume you are running a default MISP installation located in /var/www/MISP with the user www-data running on a Ubuntu/Debian system.
If your setup differs, please adapt the paths and commands accordingly, the troubleshooting steps remain the same.
-
Check
redisis running:$ sudo service redis status ● redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-12-01 08:27:32 CET; 1 day 1h ago Docs: http://redis.io/documentation, man:redis-server(1) Process: 70007 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS) Main PID: 70100 (redis-server) Tasks: 4 (limit: 38094) Memory: 123.6M CGroup: /system.slice/redis-server.service └─70100 /usr/bin/redis-server 127.0.0.1:6379 déc. 01 08:27:32 lucho-e14 systemd[1]: Starting Advanced key-value store... déc. 01 08:27:32 lucho-e14 systemd[1]: redis-server.service: Can't open PID file /run/redis/redis-server.pid (yet?) after start: Operation not permitted déc. 01 08:27:32 lucho-e14 systemd[1]: Started Advanced key-value store.In the following section we can see the host and port where
redis-serveris bind to:CGroup: /system.slice/redis-server.service └─70100 /usr/bin/redis-server 127.0.0.1:6379host: 127.0.0.1 port: 6379
-
Restart
redisand check the status a few seconds after:$ sudo service redis restart # wait 30 seconds $ sudo service redis status # output should contain: Active: active (running) -
If
redisfails to start skip to jump 7. -
Test the connection using
redis-cli:$ redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379>If using non-default host, port or password:
$ redis-cli -h 10.0.0.1 -p 63379 -a 'YOUR_SECRET_PASSWORD' 10.0.0.1:63379> ping PONG 10.0.0.1:63379> -
Check
redisconfiguration settings inMISP:/var/www/MISP/app/Config/config.php'redis_host' => '127.0.0.1', 'redis_port' => 6379, 'redis_database' => 13, 'redis_password' => '',These values must match the values set in the
CakeResqueconfiguration file:/var/www/MISP/app/Plugin/CakeResque/Config/config.php'Redis' => array( 'host' => 'localhost', // Redis server hostname 'port' => 6379, // Redis server port 'database' => 0, // Redis database number 'namespace' => 'resque', // Redis keys namespace 'password' => null // Redis password ),And these values must match the host and port redis is running, as stated in the
redis statuscommand output from 1.redis-serversettings can be found and changed in the following configuration file:sudo cat /etc/redis/redis.conf -
Check if you have a firewall running:
$ sudo ufw status Status: inactiveIf active, check the rules allow connections from your
MISPinstance to yourredishost and port.To view traffic to
redis:$ sudo tcpdump -n -i lo port 6379 -
If problem persists, attach the following logs:
/var/www/MISP/app/tmp/logs/error.log/var/www/MISP/app/tmp/logs/resque-worker-error.log/var/www/MISP/app/tmp/logs/resque-scheduler-error.logjournalctl -xe
23. I have issues with MISP Workers, how do I solve them?
- Check for file permissions issues:
sudo chown -R www-data:www-data /var/www/MISP
sudo chmod -R 750 /var/www/MISP
sudo chmod -R g+ws /var/www/MISP/app/tmp
sudo chmod -R g+ws /var/www/MISP/app/files
sudo chmod -R g+ws /var/www/MISP/app/files/scripts/tmp
sudo service redis restart # restart redis
/var/www/MISP/app/Console/cake Admin restartWorkers # restart workers
-
MISP Workers rely on Redis and CakeResque, check FAQ question #22 to solve related issues.
-
Since MISP version 2.4.151 there is a new Background Jobs / Workers backend available, but it requires a manual setup for now, a migration guide can be found here: https://github.com/MISP/MISP/blob/2.4/docs/background-jobs-migration-guide.md.
24. What if I want to remove authentication?
In MISP you cannot actually remove authentication, but it is possible to bypass it forcing access.
In the MISP configuration file, add the following line
RequestHeader add dummy-auth "foobar"
where foobar is some value you choose.
Note: do not use "_", instead use "-" then PHP will convert into "_"
Then configure the CustomAuth plugin in the following way (IMPORTANT: dummy_auth must be capitalized in the plugin configuration):
And set a readonly user for authentication (in this way the access will not be administrative)
Finally, restart apache2
systemctl restart apache2
25. I am getting weird errors when trying to do pretty much anything on MISP, complaining about datetime issues
Example:
[PDOException] SQLSTATE[22007]: Invalid datetime format: 1292 Incorrect datetime value: '1684152909' for column misp.logs.created at row 1
Explanation and solution:
After loads of frustrations and heroic debugging efforts from https://github.com/3c7 we now know what causes it.
MariaDB tables created in older versions of MariaDB used a different internal datetime field mechanism. When you updated your MariaDB to a more recent version, the tables did not get updated, and MariaDB handles this by trying to "warn" the users that the field is still in the old datetime format.
What this looks like in practice when you run a describe?
MariaDB [misp]> describe event_blocklists;
+------------+----------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+----------------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| event_uuid | varchar(40) | NO | UNI | NULL | |
| created | datetime /* mariadb-5.3 */ | YES | | NULL | |
| event_info | text | NO | | NULL | |
| comment | text | YES | | NULL | |
| event_orgc | varchar(255) | NO | MUL | NULL | |
+------------+----------------------------+------+-----+---------+----------------+
For some reason they figured that barfing this directly back in the type column would be helpful - sadly in our case it's quite the opposite. CakePHP does column introspection for each model, matching each column type to formatters for data entry. "datetime" gets mapped to a handler that will pass back datetime strings. "datetime /* mariadb-5.3 */" doesn't match anything and thus it seems to fall back to integers.
tl;dr:
ALTER TABLE `logs` MODIFY `created` datetime;
ALTER TABLE `event_blocklists` MODIFY `created` datetime;
will convert the datetime fields to the correct format and get rid of the dodgy name in the describe output, solving the issue.
We'll be implementing a diagnostic tool for this in the future, for now this should be a quick fix though.