MISP-Taxii-Server/README.md

226 lines
7.4 KiB
Markdown
Raw Normal View History

2016-11-18 12:57:01 +01:00
# MISP Taxii Server
![Build Status ](https://travis-ci.org/MISP/MISP-Taxii-Server.svg?branch=master)
2017-06-29 14:54:15 +02:00
[![Code Health](https://landscape.io/github/MISP/MISP-Taxii-Server/master/landscape.svg?style=flat)](https://landscape.io/github/MISP/MISP-Taxii-Server/master)
2016-11-18 12:57:01 +01:00
A set of configuration files to use with EclecticIQ's OpenTAXII implementation,
along with a callback for when data is sent to the TAXII Server's inbox.
2016-11-18 13:22:12 +01:00
## Installation
2018-10-24 18:40:59 +02:00
### Manual install
2016-11-18 13:22:12 +01:00
```bash
2019-02-12 18:11:36 +01:00
git clone https://github.com/MISP/MISP-Taxii-Server
cd MISP-Taxii-Server
apt-get install libmysqlclient-dev # for mysql_config
2019-02-12 18:11:36 +01:00
pip3 install -r REQUIREMENTS.txt
2016-11-18 13:22:12 +01:00
```
You'll then need to set up your TAXII database. As you're using MISP, you'll likely
already have a MySQL environment running.
```bash
mysql -u [database user] -p
# Enter Database password
mysql> create database taxiiauth;
mysql> create database taxiipersist;
mysql> grant all on taxiiauth.* to 'taxii'@'%' identified by 'some_password';
mysql> grant all on taxiipersist.* to 'taxii'@'%' identified by 'some_password';
mysql> exit;
```
2019-02-12 18:11:36 +01:00
Now configure your TAXII server
```bash
cp config/config.default.yaml config/config.yaml
```
Now, with that data, copy `config/config.default.yaml` over to `config/config.yaml` and open it. Edit the `db_connection` parameters to match your environment. Change `auth_api -> parameters -> secret` whilst you're here as well.
2019-02-12 18:11:36 +01:00
2017-05-30 14:00:59 +02:00
Do not forget to set your MISP server's URL and API key at the bottom.
2016-11-18 13:22:12 +01:00
2019-02-12 18:12:32 +01:00
If you wish, you can edit the taxii service definitions and collections in
`config/data-configuration.yaml`; full documentation on how this is set up is available at [OpenTaxii's docs](https://opentaxii.readthedocs.io/en/stable/configuration.html).
2016-11-18 13:22:12 +01:00
Now it's time to create all your SQL tables. Luckily OpenTaxii comes with commands for this.
You're going to want to export your configuration file to a variable as well.
```bash
2016-11-23 16:31:56 +01:00
# An example of this config is in the config directory
2016-11-18 13:22:12 +01:00
export OPENTAXII_CONFIG=/path/to/config.yaml
2017-05-30 14:00:59 +02:00
export PYTHONPATH=.
2016-11-18 13:22:12 +01:00
2019-02-12 18:12:32 +01:00
opentaxii-sync-data config/data-configuration.yaml
2016-11-18 13:22:12 +01:00
```
2017-05-30 14:00:59 +02:00
OpenTaxii is now ready to roll, we've just gotta do one more thing.
2016-11-18 13:22:12 +01:00
2017-05-30 14:00:59 +02:00
In the repository root directory, run
2016-11-18 13:22:12 +01:00
```bash
sudo python3 setup.py install
```
This will install the TAXII hooks to run when we have new data.
Now we should be ready to go!
```bash
opentaxii-run-dev
```
This should tell you that there is now a server running on `localhost:9000` (maybe a different port if you changed it). If there are no errors, you're good!
If you want to test everything is working, run
```bash
2019-02-12 18:12:32 +01:00
taxii-push --path http://localhost:9000/services/inbox -f tests/test.xml \
--dest my_collection --username admin --password admin
2016-11-18 13:22:12 +01:00
```
Obviously replace anything that differs in your system.
2017-03-31 11:45:28 +02:00
The client should say "Content Block Pushed Successfully" if all went well.
2016-11-18 13:22:12 +01:00
Now you have a TAXII server hooked up to MISP, you're able to send STIX files to the inbox and have them uploaded directly to MISP. So that's nice <3
2016-12-28 11:53:45 +01:00
There is also an experimental feature to push MISP events to the TAXII server when they're published - that's in `scripts/push_published_to_taxii.py`. It seems to work, but may occasionally re-upload duplicate events to MISP.
## Automated TAXII -> MISP Sync
2017-03-31 13:45:56 +02:00
If you want, there is the ability to synchronise between a remote TAXII server and the local MISP server.
```bash
$ install-remote-server.sh
[MISP-TAXII-SERVER]
POLLING SERVER INSTALLATION
FRIENDLY SERVER NAME:
< Add a unique server name here, can be anything >
```
This will then install 2 files to `~/.misptaxii`, one for a local server and one for the remote servers.
Edit these files as needed. Run `install-remote-server.sh` once for each remote server you want to add.
You'll probably want to put the sync script on a crontab,
First, run
```bash
echo `which python3` `which run-taxii-poll.py`
```
to get the path of your script, copy it. Then
```bash
crontab -e
```
This will open your crontab. Paste in
```cron
0 */6 * * * <the output of that echo command you just ran>
```
This will run the polling script every 6 hours to keep things all synced up.
## Troubleshooting
### Data truncated for column...
```python
Warning: (1265, "Data truncated for column 'original_message' at row 1")
Warning: (1265, "Data truncated for column 'content' at row 1")
```
If you encounter the error above, this means you tried to push a STIX file bigger than 65,535 bytes. To fix it run the following commands.
```bash
mysql -u [database user] -p
# Enter Database password
mysql> use taxiipersist;
mysql> alter table `inbox_messages` modify `original_message` LONGTEXT;
mysql> alter table `content_blocks` modify `content` LONGTEXT;
mysql> exit;
```
### Specified key was too long
```python
Warning: (1071, 'Specified key was too long; max key length is 767 bytes')
```
If you encounter the error above, try the following after creating the databases as per [this issue](https://github.com/MISP/MISP-Taxii-Server/issues/3#issuecomment-291875813):
```SQL
ALTER DATABASE taxiipersist CHARACTER SET latin1 COLLATE latin1_general_ci;
ALTER DATABASE taxiiauth CHARACTER SET latin1 COLLATE latin1_general_ci;
```
### Nothing appears in MISP
Take note of the user you did `export OPENTAXII_CONFIG=/path/to/config.yaml` with. If you `sudo`, this env will be lost. Use `sudo -E` to preserve env instead.
### InsecureRequestWarning
PyMISP complains about missing certificate verification. Under the misp-options in `config.yaml` do not simply set `verifySSL = False`. You can provide the CA bundle, a concatenation of all certificates in the chain, as `verifySSL = /path/to/ca_bundle`. Alternatively, you can `export REQUESTS_CA_BUNDLE=/path/to/ca_bundle`.
## Verifying the database
To verify that the `opentaxii-sync-data` worked, check the tables of database `taxiipersist`:
```
MariaDB [taxiipersist]> show tables;
+-----------------------------+
| Tables_in_taxiipersist |
+-----------------------------+
| collection_to_content_block |
| content_blocks |
| data_collections |
| inbox_messages |
| result_sets |
| service_to_collection |
| services |
| subscriptions |
+-----------------------------+
```
To verify whether the account-creation worked, check database `taxiiauth`:
```
MariaDB [taxiiauth]> select * from accounts;
+----+----------+-----------------------------------------------------------------------------------------------+
| id | username | password_hash |
+----+----------+-----------------------------------------------------------------------------------------------+
| 1 | ltaxii | pbkdf2:sha256:50000$99999999$1111111111111111111111111111111111111111111111111111111111111111 |
+----+----------+-----------------------------------------------------------------------------------------------+
```
### Ambigious Polling Service
In the case that the server you want to poll has multiple `POLL` services,
run
```bash
taxii-discovery \
--host <HOST TO POLL>
--port <POLLING PORT>
--discovery <DISCOVERY PATH, sometimes /taxii-discovery-service, may vary>
```
It'll show you the services available on the server. You'll *probably*
see two POLL services, for different version of TAXII (message binding)
Find the one relevent to you, copy its `Service Address`,
and modify `~/.misptaxii/remote-servers.yml` to resemble
```yaml
- name: "my server"
taxii_version: "1.1"
...
uri: <SERVICE ADDRESS>
```
now try polling again