MISP
/
MISP
mirror of https://github.com/MISP/MISP
2
2
Fork 0
Code Issues Releases Wiki Activity

Merge branch '2.4' of github.com:MISP/MISP into stix2

pull/3707/head
chrisr3d 2018-08-12 19:22:17 +02:00
parent 9c9e2960a9 8d34205778
commit 11faddc07a
25 changed files with 839 additions and 444 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

135
INSTALL/INSTALL.ubuntu1804.with.webmin.txt
View File

@ -1,6 +1,8 @@
INSTALLATION INSTRUCTIONS
-------------------------
# For Ubuntu 16.04-server with Webmin preinstalled
# For Ubuntu 18.04.1 server with Webmin
# Why Webmin/Virtualmin?
# Some may not be full time sysadmin and prefer a platform that once it has been setup works and is decently easy to manage.
# Assuming you created the subdomanin misp.yourserver.tld to where MISP will be installed
# and that the user "misp" is in the sudoers group
@ -9,15 +11,20 @@ INSTALLATION INSTRUCTIONS
1/ Minimal Ubuntu install
-------------------------
# Login as misp
# Make sure your system is up2date:
sudo apt-get update
sudo apt-get upgrade
# Get Virtualmin
wget http://software.virtualmin.com/gpl/scripts/install.sh
2/ Install LAMP & dependencies
# Install it
chmod +x install.sh
./install.sh
# Grab a coffee while it does its magic
2/ Configure basic Virtualmin environment
------------------------------
Once the system is installed you can perform the following steps:
@ -26,17 +33,33 @@ sudo apt-get install curl gcc git gnupg-agent make python openssl redis-server s
# Stop MySQL and install MariaDB (a MySQL fork/alternative)
# MariaDB will replace MySQL and it will work with the latests versions of Webmin without modifications
# WARNING: Databases and data may be lost! It is assumed you are installing on a new server with no existing DBs
# WARNING: Databases and data will be lost! It is assumed you are installing on a new server with no existing DBs
# NOTE: at present, a simple...
# 'sudo service mysql stop && sudo apt-get install mariadb-client mariadb-server'
# ... doesn't work well with 18.04.1 so you should do the following:
sudo apt purge mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server-5.7 mysql-server-core-5.7 mysql-server
# Issues may crop up if you leave MySQL configuration there so remove also config files in /etc/mysql.
# Remove and cleanup packages
sudo apt autoremove && sudo apt -f install
# Add repositories for Mariadb 10.3 and install it
sudo apt-get install software-properties-common
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://mariadb.mirrors.ovh.net/MariaDB/repo/10.3/ubuntu bionic main'
sudo apt update
sudo apt install mariadb-server
sudo service mysql stop
sudo apt-get install mariadb-client mariadb-server
# Secure the MariaDB installation (especially by setting a strong root password)
# Secure the MariaDB installation (especially by setting a strong root password) if it hasn't been asked during the setup process.
sudo mysql_secure_installation
# Go through the post-installation Wizard and configure your misp.yourdomain.tld virtual server
# That should create the misp user and related directories
# Add the misp user to the sudo group
# Install PHP and dependencies
sudo apt-get install libapache2-mod-php php php-cli php-gnupg php-dev php-json php-mysql php-opcache php-readline php-redis php-xml
sudo pear channel-update pear.php.net
sudo pear install Crypt_GPG
# Apply all changes
sudo systemctl restart apache2
@ -44,7 +67,10 @@ sudo systemctl restart apache2
3/ MISP code
------------
# Assuming you created the subdomanin misp.yourserver.tld
# Download MISP using git in the /home/misp/public_html/.
# Download MISP using git in the /home/misp/public_html/ as misp
sudo - misp
# or log out root and log back in as misp
git clone https://github.com/MISP/MISP.git /home/misp/public_html/MISP
cd /home/misp/public_html/MISP
@ -58,10 +84,10 @@ git submodule init
git submodule update
# Make git ignore filesystem permission differences
git config core.filemode false
git submodule foreach git config core.filemode false
# install Mitre's STIX and its dependencies by running the following commands:
sudo apt-get install python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev python-setuptools
sudo apt-get install python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev python-setuptools python-pip
cd /home/misp/public_html/MISP/app/files/scripts
git clone https://github.com/CybOXProject/python-cybox.git
git clone https://github.com/STIXProject/python-stix.git
@ -77,15 +103,15 @@ cd /home/misp/public_html/MISP/app/files/scripts/mixbox
sudo python3 setup.py install
# install PyMISP
cd /var/www/MISP/PyMISP
pip install jsonschema
cd /home/misp/public_html/MISP/PyMISP
sudo python3 setup.py install
4/ CakePHP
-----------
# CakePHP is included as a submodule of MISP, execute the following commands to let git fetch it:
cd /home/misp/public_html/MISP
# CakePHP is included as a submodule of MISP
# Once done, install CakeResque along with its dependencies if you intend to use the built in background jobs:
# Install CakeResque along with its dependencies if you intend to use the built in background jobs:
cd /home/misp/public_html/MISP/app
php composer.phar require kamisama/cake-resque:4.1.2
php composer.phar config vendor-dir Vendor
@ -102,7 +128,7 @@ cp -fa /home/misp/public_html/MISP/INSTALL/setup/config.php /home/misp/public_ht
----------------------
# Check if the permissions are set correctly using the following commands:
sudo chmod -R 750 /home/misp/public_html/MISP
sudo chmod -R 770 /home/misp/public_html/MISP
sudo chmod -R g+ws /home/misp/public_html/MISP/app/tmp
sudo chmod -R g+ws /home/misp/public_html/MISP/app/files
sudo chmod -R g+ws /home/misp/public_html/MISP/app/files/scripts/tmp
@ -113,14 +139,17 @@ sudo chmod -R g+ws /home/misp/public_html/MISP/app/files/scripts/tmp
# Enter the mysql shell
sudo mysql -u root -p
MariaDB [(none)]> create database misp;
MariaDB [(none)]> grant usage on *.* to misp@localhost identified by 'XXXXdbpasswordhereXXXXX';
MariaDB [(none)]> grant all privileges on misp.* to misp@localhost;
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> exit
# If all went well when you created the misp user in Virtualmin you should already have a misp database
# otherwise create it with:
create database misp;
# Make sure password and all privileges are set
grant usage on *.* to misp@localhost identified by 'XXXXdbpasswordhereXXXXX';
grant all privileges on misp.* to misp@localhost;
flush privileges;
exit
# Import the empty MISP database from MYSQL.sql
sh -c "mysql -u misp -p misp < /home/misp/public_html/MISP/INSTALL/MYSQL.sql"
mysql -u misp -p misp < /home/misp/public_html/MISP/INSTALL/MYSQL.sql
# enter the password you set previously
@ -164,7 +193,7 @@ cp -a /home/misp/public_html/MISP/app/Config/core.default.php /home/misp/public_
cp -a /home/misp/public_html/MISP/app/Config/config.default.php /home/misp/public_html/MISP/app/Config/config.php
# Configure the fields in the newly created files:
vim /home/misp/public_html/MISP/app/Config/database.php
vi /home/misp/public_html/MISP/app/Config/database.php
# DATABASE_CONFIG has to be filled
# With the default values provided in section 6, this would look like:
# class DATABASE_CONFIG {
@ -181,14 +210,16 @@ vim /home/misp/public_html/MISP/app/Config/database.php
# );
#}
# Important! Change the salt key in /var/www/MISP/app/Config/config.php
# Important! Change the salt key in /home/misp/public_html/MISP/app/Config/config.php
# see line 7 (may change)
# 'salt' => 'yoursaltkeyhere'
# The salt key must be a string at least 32 bytes long.
# The admin user account will be generated on the first login, make sure that the salt is changed before you create that user
# If you forget to do this step, and you are still dealing with a fresh installation, just alter the salt,
# delete the user from mysql and log in again using the default admin credentials (admin@admin.test / admin)
# Change base url in config.php
vim /home/misp/public_html/MISP/app/Config/config.php
vi /home/misp/public_html/MISP/app/Config/config.php
# example: 'baseurl' => 'https://<your.FQDN.here>',
# alternatively, you can leave this field empty if you would like to use relative pathing in MISP
# 'baseurl' => '',
@ -218,13 +249,53 @@ gpg --homedir /home/misp/public_html/MISP/.gnupg --gen-key
# and return to the "install" shell
# Export the public key to the webroot
sh -c "gpg --homedir /home/misp/public_html/MISP/.gnupg --export --armor YOUR-KEYS-EMAIL-HERE > /home/misp/public_html/MISP/app/webroot/gpg.asc"
gpg --homedir /home/misp/public_html/MISP/.gnupg --export --armor YOUR-KEYS-EMAIL-HERE > /home/misp/public_html/MISP/app/webroot/gpg.asc
# To make the background workers start on boot
chmod +x /home/misp/public_html/MISP/app/Console/worker/start.sh
sudo vim /etc/rc.local
# Add the following line before the last line (exit 0). Make sure that you replace www-data with your apache user:
sudo -u www-data bash /home/misp/public_html/MISP/app/Console/worker/start.sh
# Activate rc.local in systemd
# Systemd developers, in their wisdom, decided to complicate things a bit so you'll have to
# create the rc-local.service
sudo vi /etc/systemd/system/rc-local.service
# and paste the following in it
[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99
[Install]
WantedBy=multi-user.target
# Hit the "esc" button then type :wq! to write the file and exit vi
# Create/edit /etc/rc.local
sudo vi /etc/rc.local
# If the file is empty add the following including the #
#!/bin/bash
# Then add this
sudo -u misp bash /home/misp/public_html/MISP/app/Console/worker/start.sh
# If the file was empty add this as the last line
exit 0
# save, quit vi and set permissions
sudo chmod +x /etc/rc.local
# Enable it in systemd
sudo systemctl enable rc-local
#Start the rc-local compatibility layer and check if AOK
sudo systemctl start rc-local.service
sudo systemctl status rc-local.service
# Now log in using the webinterface:
# The default user/pass = admin@admin.test/admin
@ -240,8 +311,8 @@ sudo -u www-data bash /home/misp/public_html/MISP/app/Console/worker/start.sh
# If any of the directories that MISP uses to store files is not writeable to the apache user, change the permissions
# you can do this by running the following commands:
sudo chmod -R 750 /home/misp/public_html/MISP/<directory path with an indicated issue>
sudo chown -R misp:misp /home/misp/public_html/MISP/<directory path with an indicated issue>
sudo chmod -R 770 /home/misp/public_html/MISP/<directory path with an indicated issue>
sudo chown -R misp:www-data /home/misp/public_html/MISP/<directory path with an indicated issue>
# Make sure that the STIX libraries and GnuPG work as intended, if not, refer to INSTALL.txt's paragraphs dealing with these two items

2
VERSION.json
View File

@ -1 +1 @@
{"major":2, "minor":4, "hotfix":93}
{"major":2, "minor":4, "hotfix":94}

1
app/Controller/Component/ACLComponent.php
View File

@ -323,6 +323,7 @@ class ACLComponent extends Component
'pull' => array(),
'purgeSessions' => array(),
'push' => array(),
'rest' => array('perm_auth'),
'restartWorkers' => array(),
'serverSettings' => array(),
'serverSettingsEdit' => array(),

5
app/Controller/EventsController.php
View File

@ -1536,8 +1536,11 @@ class EventsController extends AppController
if (isset($this->request->data['response'])) {
$this->request->data = $this->request->data['response'];
}
if (isset($this->request->data['request'])) {
$this->request->data = $this->request->data['request'];
}
if (!isset($this->request->data['Event'])) {
$this->request->data['Event'] = $this->request->data;
$this->request->data = array('Event' => $this->request->data);
}
// Distribution, reporter for the events pushed will be the owner of the authentication key

81
app/Controller/ServersController.php
View File

@ -1580,4 +1580,85 @@ class ServersController extends AppController
{
return $this->RestResponse->viewData(array('uuid' => Configure::read('MISP.uuid')), $this->response->type());
}
public function rest() {
if ($this->request->is('post')) {
$request = $this->request->data;
if (!empty($request['Server'])) {
$request = $this->request->data['Server'];
}
$result = $this->__doRestQuery($request);
if (!$result) {
$this->Flash->error('Something went wrong. Make sure you set the http method, body (when sending POST requests) and URL correctly.');
} else {
$this->set('data', $result);
}
}
$header =
'Authorization: ' . $this->Auth->user('authkey') . PHP_EOL .
'Accept: application/json' . PHP_EOL .
'Content-Type: application/json';
$this->set('header', $header);
}
private function __doRestQuery($request) {
App::uses('SyncTool', 'Tools');
$params = array(
);
if (!empty($request['url'])) {
$path = parse_url($request['url'], PHP_URL_PATH);
$query = parse_url($request['url'], PHP_URL_QUERY);
if (!empty($query)) {
$path .= '?' . $query;
}
$url = Configure::read('MISP.baseurl') . '/' . $path;
} else {
throw new InvalidArgumentException('Url not set.');
}
App::uses('HttpSocket', 'Network/Http');
$HttpSocket = new HttpSocket($params);
$view_data = array();
$temp_headers = explode("\n", $request['header']);
$request['header'] = array(
'Authorization' => $this->Auth->user('authkey'),
'Accept' => 'application/json',
'Content-Type' => 'application/json'
);
foreach ($temp_headers as $header) {
$header = explode(':', $header);
$header[0] = trim($header[0]);
$header[1] = trim($header[1]);
$request['header'][$header[0]] = $header[1];
}
$start = microtime(true);
if (
!empty($request['method']) &&
$request['method'] === 'GET'
) {
$response = $HttpSocket->get($url, false, array('header' => $request['header']));
} else if (
!empty($request['method']) &&
$request['method'] === 'POST' &&
!empty($request['body'])
) {
$response = $HttpSocket->post($url, $request['body'], array('header' => $request['header']));
} else {
return false;
}
$view_data['duration'] = microtime(true) - $start;
$view_data['duration'] = round($view_data['duration'] * 1000, 2) . 'ms';
$view_data['code'] = $response->code;
$view_data['headers'] = $response->headers;
if (!empty($request['show_result'])) {
$view_data['data'] = $response->body;
} else {
if ($response->isOk()) {
$view_data['data'] = 'Success.';
} else {
$view_data['data'] = 'Something went wrong.';
}
}
return $view_data;
}
}

4
app/Controller/UsersController.php
View File

@ -1838,12 +1838,12 @@ class UsersController extends AppController
}
}
public function verifyGPG()
public function verifyGPG($full = false)
{
if (!self::_isSiteAdmin()) {
throw new NotFoundException();
}
$user_results = $this->User->verifyGPG();
$user_results = $this->User->verifyGPG($full);
$this->set('users', $user_results);
}

2
app/Model/Event.php
View File

@ -860,7 +860,7 @@ class Event extends AppModel
private function __executeRestfulEventToServer($event, $server, $resourceId, &$newLocation, &$newTextBody, $HttpSocket)
{
$result = $this->restfulEventToServer($event, $server, null, $newLocation, $newTextBody, $HttpSocket);
$result = $this->restfulEventToServer($event, $server, $resourceId, $newLocation, $newTextBody, $HttpSocket);
if (is_numeric($result)) {
$error = $this->__resolveErrorCode($result, $event, $server);
if ($error) {

1
app/Model/User.php
View File

@ -472,6 +472,7 @@ class User extends AppModel
$currentTimestamp = time();
$temp = $gpg->importKey($user['User']['gpgkey']);
$key = $gpg->getKeys($temp['fingerprint']);
$result[5] = $temp['fingerprint'];
$subKeys = $key[0]->getSubKeys();
$sortedKeys = array('valid' => 0, 'expired' => 0, 'noEncrypt' => 0);
foreach ($subKeys as $subKey) {

4
app/View/Elements/Events/View/sighting_field.ctp
View File

@ -28,8 +28,8 @@
</td>
<td class="short">
<?php
if (!empty($temp)) {
echo $this->element('sparkline', array('id' => $object['id'], 'csv' => $temp));
if (!empty($sightingsData['csv'][$object['id']])) {
echo $this->element('sparkline', array('id' => $object['id'], 'csv' => $sightingsData['csv'][$object['id']]));
}
?>
</td>

1
app/View/Elements/global_menu.ctp
View File

@ -27,6 +27,7 @@
<?php endif; ?>
<li><a href="<?php echo $baseurl;?>/attributes/index"><?php echo __('List Attributes');?></a></li>
<li><a href="<?php echo $baseurl;?>/attributes/search"><?php echo __('Search Attributes');?></a></li>
<li><a href="<?php echo $baseurl;?>/servers/rest"><?php echo __('REST client');?></a></li>
<li class="divider"></li>
<li><a href="<?php echo $baseurl;?>/shadow_attributes/index"><?php echo __('View Proposals');?></a></li>
<li><a href="<?php echo $baseurl;?>/events/proposalEventIndex">Events with proposals</a></li>

5
app/View/Elements/side_menu.ctp
View File

@ -200,6 +200,11 @@
),
'text' => __('Import from…')
));
echo $this->element('/side_menu_link', array(
'element_id' => 'rest',
'url' => '/servers/rest',
'text' => __('REST client')
));
}
echo $this->element('/side_menu_divider');
echo $this->element('/side_menu_link', array(

2
app/View/Elements/view_mitre_attack_matrix.ctp
View File

@ -11,7 +11,7 @@ foreach($attackTactic as $tactic):
</div>
<div class="attack-matrix-options matrix-div-submit">
<span class="btn btn-inverse btn-matrix-submit" role="button" style="padding: 1px 5px !important;font-size: 12px !important;font-weight: bold;"><?php echo _('Submit'); ?></span>
<span class="btn btn-inverse btn-matrix-submit" role="button" style="padding: 1px 5px !important;font-size: 12px !important;font-weight: bold;"><?php echo __('Submit'); ?></span>
</div>
<div class="attack-matrix-options">

4
app/View/Events/proposal_event_index.ctp
View File

@ -20,14 +20,14 @@
<th class="filter">
<?php echo $this->Paginator->sort('published');?>
</th>
<th><?php echo $this->Paginator->sort('id');?></th>
<th><?php echo $this->Paginator->sort('id', 'Id', array('direction' => 'desc'));?></th>
<th><?php echo $this->Paginator->sort('attribute_count', __('Proposals'));?></th>
<th><?php echo __('Contributors');?></th>
<?php if ($isSiteAdmin): ?>
<th><?php echo $this->Paginator->sort('user_id', __('Email'));?></th>
<?php endif; ?>
<th class="filter">
<?php echo $this->Paginator->sort('date');?>
<?php echo $this->Paginator->sort('date', 'Date', array('direction' => 'desc'));?>
</th>
<th class="filter">
<?php echo $this->Paginator->sort('info');?>

66
app/View/Servers/rest.ctp Normal file
View File

@ -0,0 +1,66 @@
<div class="servers form">
<?php echo $this->Form->create('Server');?>
<fieldset>
<legend><?php echo __('REST client');?></legend>
<?php
echo $this->Form->input('method', array(
'label' => __('Relative path to query'),
'options' => array(
'GET' => 'GET',
'POST' => 'POST'
)
));
?>
<div class="input clear" style="width:100%;">
<?php
echo $this->Form->input('url', array(
'label' => __('Relative path to query'),
'class' => 'input-xxlarge'
));
?>
<div class="input clear" style="width:100%;">
<?php
echo $this->Form->input('show_result', array(
'type' => 'checkbox'
));
?>
<div class="input clear" style="width:100%;">
<?php
echo $this->Form->input('header', array(
'type' => 'textarea',
'div' => 'input clear',
'class' => 'input-xxlarge',
'default' => !empty($this->request->data['Server']['header']) ? $this->request->data['Server']['header'] : $header
));
?>
<div class="input clear" style="width:100%;">
<?php
echo $this->Form->input('body', array(
'type' => 'textarea',
'div' => 'input clear',
'class' => 'input-xxlarge'
));
?>
<div class="input clear" style="width:100%;">
<?php
echo $this->Form->submit('Run query', array('class' => 'btn btn-primary'));
echo $this->Form->end();
?>
<hr />
</fieldset>
<?php
if (!empty($data['data'])) {
echo sprintf('<h3>%s</h3>', __('Response'));
echo sprintf('<div><span class="bold">%s</span>: %d</div>', __('Response code'), h($data['code']));
echo sprintf('<div><span class="bold">%s</span>: %s</div>', __('Request duration'), h($data['duration']));
echo sprintf('<div class="bold">%s</div>', __('Headers'));
foreach ($data['headers'] as $header => $value) {
echo sprintf('&nbsp;&nbsp;<span class="bold">%s</span>: %s<br />', h($header), h($value));
}
echo sprintf('<pre>%s</pre>', h($data['data']));
}
?>
</div>
<?php
echo $this->element('side_menu', array('menuList' => 'event-collection', 'menuItem' => 'rest'));
?>

2
app/View/ShadowAttributes/index.ctp
View File

@ -51,7 +51,7 @@
<?php echo $this->Paginator->sort('type', __('Type'));?>
</th>
<th>
<?php echo $this->Paginator->sort('timestamp', __('Created'));?>
<?php echo $this->Paginator->sort('timestamp', __('Created'), array('direction' => 'desc'));?>
</th>
</tr>
<?php foreach ($shadowAttributes as $event):?>

8
app/View/Users/verify_g_p_g.ctp
View File

@ -4,11 +4,13 @@
<?php foreach ($users as $k => $user) {
echo '<a href="'.$baseurl.'/admin/users/view/' . $k . '">' . $k . ' (' . h($user[1]) . ')</a>:';
if (isset($user[0])) {
echo '-> <span style="color:red;"><span style="font-weight:bold">Invalid.</span> (' . h($user[2]) . ')</span><br />';
echo '-> <span style="color:red;"><span style="font-weight:bold">Invalid.</span> (' . h($user[2]) . ')</span>';
} else {
echo '-> <span style="color:green;">OK</span><br />';
echo '-> <span style="color:green;">OK</span>';
}
echo '------------------------------------------------------------------------------<br />';
echo ' (' . $user[5] . ')';
echo '<br />------------------------------------------------------------------------------<br />';
}
?>
</ul>

2
app/files/misp-galaxy

@ -1 +1 @@
Subproject commit a0dfdd65ae2aeab3e9552535cd576c7399694e88
Subproject commit 9059a85eed4294bfaa45a30a025ab69c5b97cde3

176
app/files/scripts/misp2stix.py
View File

@ -32,9 +32,9 @@ from cybox.objects.artifact_object import Artifact, RawArtifact
from cybox.objects.memory_object import Memory
from cybox.objects.email_message_object import EmailMessage, EmailHeader, EmailRecipients, Attachments
from cybox.objects.domain_name_object import DomainName
from cybox.objects.win_registry_key_object import *
from cybox.objects.win_registry_key_object import RegistryValue, RegistryValues, WinRegistryKey
from cybox.objects.system_object import System, NetworkInterface, NetworkInterfaceList
from cybox.objects.http_session_object import RegistryValue, RegistryValues, WinRegistryKey
from cybox.objects.http_session_object import HTTPClientRequest, HTTPRequestHeader, HTTPRequestHeaderFields, HTTPRequestLine, HTTPRequestResponse, HTTPSession
from cybox.objects.as_object import AutonomousSystem
from cybox.objects.socket_address_object import SocketAddress
from cybox.objects.network_connection_object import NetworkConnection
@ -136,6 +136,7 @@ class StixBuilder(object):
except TypeError:
idgen.set_id_namespace(Namespace(namespace[0], namespace[1], "MISP"))
self.namespace_prefix = idgen.get_id_namespace_alias()
self.objects_to_parse = defaultdict(dict)
## MAPPING FOR ATTRIBUTES
self.simple_type_to_method = {"port": self.generate_port_observable, "domain|ip": self.generate_domain_ip_observable}
self.simple_type_to_method.update(dict.fromkeys(hash_type_attributes["single"] + hash_type_attributes["composite"] + ["filename"], self.resolve_file_observable))
@ -159,6 +160,8 @@ class StixBuilder(object):
"ip-port": self.parse_ip_port_object,
"network-connection": self.parse_network_connection_object,
"network-socket": self.parse_network_socket_object,
"pe": self.store_pe,
"pe-section": self.store_pe,
"process": self.parse_process_object,
"registry-key": self.parse_regkey_object,
"url": self.parse_url_object,
@ -269,41 +272,15 @@ class StixBuilder(object):
self.handle_attribute(incident, attribute, tags)
def resolve_objects(self, incident, tags):
objects_to_parse = defaultdict(dict)
for misp_object in self.misp_event.objects:
category = misp_object.get('meta-category')
name = misp_object.name
if name in ('pe', 'pe-section'):
objects_to_parse[name][misp_object.uuid] = misp_object
continue
elif name == 'file':
if misp_object.references:
to_parse = False
for reference in misp_object.references:
if reference.relationship_type == 'included-in' and reference.Object['name'] == "pe":
objects_to_parse[name][misp_object.uuid] = misp_object
to_parse = True
break
if to_parse:
continue
try:
to_ids, observable = self.objects_mapping[name](misp_object.attributes, misp_object.uuid)
to_ids, observable = self.objects_mapping[name](misp_object)
except KeyError:
try:
to_ids, observable = self.create_custom_observable(misp_object.name, misp_object.attributes, misp_object.uuid)
except:
continue
if name == "process" and misp_object.references:
for reference in misp_object.references:
if reference.relationship_type == "connected-to":
related_object = RelatedObject()
try:
referenced_attribute_type = reference.Object['name']
except AttributeError:
references_attribute_type = reference.Attribute['type']
related_object.idref = "{}:{}-{}".format(self.namespace_prefix, referenced_attribute_type, reference.referenced_uuid)
related_object.relationship = "Connected_To"
observable.object_.related_objects.append(related_object)
to_ids, observable = self.create_custom_observable(name, misp_object.attributes, misp_object.uuid)
except TypeError:
continue
if to_ids:
indicator = self.create_indicator(misp_object, observable, tags)
related_indicator = RelatedIndicator(indicator, relationship=category)
@ -311,11 +288,12 @@ class StixBuilder(object):
else:
related_observable = RelatedObservable(observable, relationship=category)
incident.related_observables.append(related_observable)
if objects_to_parse: self.resolve_objects2parse(objects_to_parse, incident, tags)
if self.objects_to_parse:
self.resolve_objects2parse(incident, tags)
def resolve_objects2parse(self, objects2parse, incident, tags):
for uuid, file_object in objects2parse['file'].items():
def resolve_objects2parse(self, incident, tags):
for uuid, file_object in self.objects_to_parse['file'].items():
category = file_object.get('meta-category')
to_ids_file, file_dict = self.create_attributes_dict(file_object.attributes)
to_ids_list = [to_ids_file]
@ -325,7 +303,7 @@ class StixBuilder(object):
if reference.relationship_type == "included-in" and reference.Object['name'] == "pe":
pe_uuid = reference.referenced_uuid
break
pe_object = objects2parse['pe'][pe_uuid]
pe_object = self.objects_to_parse['pe'][pe_uuid]
pe_headers = PEHeaders()
pe_file_header = PEFileHeader()
pe_sections = PESectionList()
@ -333,7 +311,7 @@ class StixBuilder(object):
to_ids_list.append(to_ids_pe)
for reference in pe_object.references:
if reference.Object['name'] == "pe-section":
pe_section_object = objects2parse['pe-section'][reference.referenced_uuid]
pe_section_object = self.objects_to_parse['pe-section'][reference.referenced_uuid]
to_ids_section, section_dict = self.create_attributes_dict(pe_section_object.attributes)
to_ids_list.append(to_ids_section)
if reference.relationship_type == "included-in":
@ -659,8 +637,8 @@ class StixBuilder(object):
ttp.exploit_targets.append(ET)
return ttp
def parse_asn_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict(attributes)
def parse_asn_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict(misp_object.attributes)
auto_sys = AutonomousSystem()
if 'asn' in attributes_dict:
asn = attributes_dict['asn']
@ -670,13 +648,14 @@ class StixBuilder(object):
auto_sys.number = asn
if 'description' in attributes_dict:
auto_sys.name = attributes_dict['description']
uuid = misp_object.uuid
auto_sys.parent.id_ = "{}:AutonomousSystemObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(auto_sys)
observable.id_ = "{}:AutonomousSystem-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def parse_credential_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict_multiple(attributes)
def parse_credential_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict_multiple(misp_object.attributes)
account = Account()
if 'text' in attributes_dict:
account.description = attributes_dict.pop('text')[0]
@ -685,10 +664,10 @@ class StixBuilder(object):
for attribute_relation in ('username', 'origin', 'notification'):
if attribute_relation in attributes_dict:
for attribute in attributes_dict.pop(attribute_relation):
property = Property()
property.name = attribute_relation
property.value = attribute
custom_properties.append(property)
prop = Property()
prop.name = attribute_relation
prop.value = attribute
custom_properties.append(prop)
account.custom_properties = custom_properties
if attributes_dict:
authentication = Authentication()
@ -697,9 +676,9 @@ class StixBuilder(object):
struct_auth_meca.description = attributes_dict['format'][0]
authentication.structured_authentication_mechanism = struct_auth_meca
if 'type' in attributes_dict and 'password' in attributes_dict and len(attributes_dict['type']) == len(attributes_dict['password']):
for type, password in zip(attributes_dict['type'], attributes_dict['password']):
for p_type, password in zip(attributes_dict['type'], attributes_dict['password']):
auth = deepcopy(authentication)
auth.authentication_type = type
auth.authentication_type = p_type
auth.authentication_data = password
account.authentication.append(auth)
else:
@ -730,13 +709,14 @@ class StixBuilder(object):
account.authentication.append(authentication)
except:
pass
uuid = misp_object.uuid
account.parent.id_ = "{}:AccountObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(account)
observable.id_ = "{}:Account-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def parse_domain_ip_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict_multiple(attributes, with_uuid=True)
def parse_domain_ip_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict_multiple(misp_object.attributes, with_uuid=True)
composition = []
if 'domain' in attributes_dict:
domain = attributes_dict['domain'][0]
@ -746,10 +726,10 @@ class StixBuilder(object):
composition.append(self.create_ip_observable(ip['value'], ip['uuid']))
if len(composition) == 1:
return to_ids, composition[0]
return to_ids, self.create_observable_composition(composition, uuid, "domain-ip")
return to_ids, self.create_observable_composition(composition, misp_object.uuid, "domain-ip")
def parse_email_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict_multiple(attributes, with_uuid=True)
def parse_email_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict_multiple(misp_object.attributes, with_uuid=True)
email_object = EmailMessage()
email_header = EmailHeader()
if 'from' in attributes_dict:
@ -786,14 +766,25 @@ class StixBuilder(object):
attachment_file = self.create_file_attachment(attachment['value'], attachment['uuid'])
email_object.add_related(attachment_file, "Contains", inline=True)
email_object.attachments.append(attachment_file.parent.id_)
uuid = misp_object.uuid
email_object.header = email_header
email_object.parent.id_ = "{}:EmailMessageObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(email_object)
observable.id_ = "{}:EmailMessage-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def parse_file_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict(attributes)
def parse_file_object(self, misp_object):
uuid = misp_object.uuid
if misp_object.references:
to_parse = False
for reference in misp_object.references:
if reference.relationship_type == 'included-in' and reference.Object['name'] == "pe":
self.objects_to_parse[misp_object.name][uuid] = misp_object
to_parse = True
break
if to_parse:
return
to_ids, attributes_dict = self.create_attributes_dict(misp_object.attributes)
file_object = File()
self.fill_file_object(file_object, attributes_dict)
file_object.parent.id_ = "{}:FileObject-{}".format(self.namespace_prefix, uuid)
@ -801,8 +792,8 @@ class StixBuilder(object):
file_observable.id_ = "{}:File-{}".format(self.namespace_prefix, uuid)
return to_ids, file_observable
def parse_ip_port_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict_multiple(attributes, with_uuid=True)
def parse_ip_port_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict_multiple(misp_object.attributes, with_uuid=True)
composition = []
if 'domain' in attributes_dict:
for domain in attributes_dict['domain']:
@ -821,27 +812,31 @@ class StixBuilder(object):
composition.append(self.create_ip_observable(ip['value'], ip['uuid']))
if len(composition) == 1:
return to_ids, composition[0]
return to_ids, self.create_observable_composition(composition, uuid, "ip-port")
return to_ids, self.create_observable_composition(composition, misp_object.uuid, "ip-port")
def parse_network_connection_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict(attributes)
def parse_network_connection_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict(misp_object.attributes)
network_connection_object = NetworkConnection()
src_args, dst_args = self.parse_src_dst_args(attributes_dict)
if src_args: network_connection_object.source_socket_address = self.create_socket_address_object('src', **src_args)
if dst_args: network_connection_object.destination_socket_address = self.create_socket_address_object('dst', **dst_args)
if src_args:
network_connection_object.source_socket_address = self.create_socket_address_object('src', **src_args)
if dst_args:
network_connection_object.destination_socket_address = self.create_socket_address_object('dst', **dst_args)
if 'layer3-protocol' in attributes_dict:
network_connection_object.layer3_protocol = attributes_dict['layer3-protocol']
if 'layer4-protocol' in attributes_dict:
network_connection_object.layer4_protocol = attributes_dict['layer4-protocol']
if 'layer7-protocol' in attributes_dict:
network_connection_object.layer7_protocol = attributes_dict['layer7-protocol']
uuid = misp_object.uuid
network_connection_object.parent.id_ = "{}:NetworkConnectionObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(network_connection_object)
observable.id_ = "{}:NetworkConnection-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def parse_network_socket_object(self, attributes, uuid):
def parse_network_socket_object(self, misp_object):
listening, blocking = [False] * 2
attributes = misp_object.attributes
for attribute in attributes:
if attribute.object_relation == "state":
if attribute.value == "listening":
@ -851,8 +846,10 @@ class StixBuilder(object):
to_ids, attributes_dict = self.create_attributes_dict(attributes)
network_socket_object = NetworkSocket()
src_args, dst_args = self.parse_src_dst_args(attributes_dict)
if src_args: network_socket_object.local_address = self.create_socket_address_object('src', **src_args)
if dst_args: network_socket_object.remote_address = self.create_socket_address_object('dst', **dst_args)
if src_args:
network_socket_object.local_address = self.create_socket_address_object('src', **src_args)
if dst_args:
network_socket_object.remote_address = self.create_socket_address_object('dst', **dst_args)
if 'protocol' in attributes_dict:
network_socket_object.protocol = attributes_dict['protocol']
network_socket_object.is_listening = True if listening else False
@ -861,12 +858,14 @@ class StixBuilder(object):
network_socket_object.address_family = attributes_dict['address-family']
if 'domain-family' in attributes_dict:
network_socket_object.domain = attributes_dict['domain-family']
uuid = misp_object.uuid
network_socket_object.parent.id_ = "{}:NetworkSocketObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(network_socket_object)
observable.id_ = "{}:NetworkSocket-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def parse_process_object(self, attributes, uuid):
def parse_process_object(self, misp_object):
attributes = misp_object.attributes
to_ids, attributes_dict = self.create_attributes_dict_multiple(attributes)
process_object = Process()
if 'creation-time' in attributes_dict:
@ -886,13 +885,25 @@ class StixBuilder(object):
# if 'port' in attributes_dict:
# for port in attributes['port']:
# process_object.port_list.append(self.create_port_object(port['value']))
uuid = misp_object.uuid
process_object.parent.id_ = "{}:ProcessObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(process_object)
observable.id_ = "{}:Process-{}".format(self.namespace_prefix, uuid)
if misp_object.references:
for reference in misp_object.references:
if reference.relationship_type == "connected-to":
related_object = RelatedObject()
try:
referenced_attribute_type = reference.Object['name']
except AttributeError:
references_attribute_type = reference.Attribute['type']
related_object.idref = "{}:{}-{}".format(self.namespace_prefix, referenced_attribute_type, reference.referenced_uuid)
related_object.relationship = "Connected_To"
observable.object_.related_objects.append(related_object)
return to_ids, observable
def parse_regkey_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict(attributes)
def parse_regkey_object(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict(misp_object.attributes)
reg_object = WinRegistryKey()
registry_values = False
reg_value_object = RegistryValue()
@ -920,14 +931,15 @@ class StixBuilder(object):
registry_values = True
if registry_values:
reg_object.values = RegistryValues(reg_value_object)
uuid = misp_object.uuid
reg_object.parent.id_ = "{}:WinRegistryKeyObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(reg_object)
observable.id_ = "{}:WinRegistryKey-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def parse_url_object(self, attributes, uuid):
def parse_url_object(self, misp_object):
observables = []
to_ids, attributes_dict = self.create_attributes_dict(attributes, with_uuid=True)
to_ids, attributes_dict = self.create_attributes_dict(misp_object.attributes, with_uuid=True)
if 'url' in attributes_dict:
url = attributes_dict['url']
observables.append(self.create_url_observable(url['value'], url['uuid']))
@ -939,10 +951,10 @@ class StixBuilder(object):
observables.append(self.create_hostname_observable(hostname['value'], hostname['uuid']))
if len(observables) == 1:
return observables[0]
return to_ids, self.create_observable_composition(observables, uuid, "url")
return to_ids, self.create_observable_composition(observables, misp_object.uuid, "url")
def parse_whois(self, attributes, uuid):
to_ids, attributes_dict = self.create_attributes_dict_multiple(attributes)
def parse_whois(self, misp_object):
to_ids, attributes_dict = self.create_attributes_dict_multiple(misp_object.attributes)
n_attribute = len(attributes_dict)
whois_object = WhoisEntry()
for attribute in attributes_dict:
@ -972,11 +984,15 @@ class StixBuilder(object):
whois_object.remarks = attributes_dict['comment']
elif 'text' in attributes_dict:
whois_object.remarks = attributes_dict['text']
uuid = misp_object.uuid
whois_object.parent.id_ = "{}:WhoisObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(whois_object)
observable.id_ = "{}:Whois-{}".format(self.namespace_prefix, uuid)
return to_ids, observable
def store_pe(self, misp_object):
self.objects_to_parse[misp_object.name][misp_object.uuid] = misp_object
@staticmethod
def fill_whois_registrants(attributes):
registrants = WhoisRegistrants()
@ -992,8 +1008,8 @@ class StixBuilder(object):
registrants.append(registrant)
return registrants
def parse_x509_object(self, attributes, uuid):
to_ids, attributes_dict = self.create_x509_attributes_dict(attributes)
def parse_x509_object(self, misp_object):
to_ids, attributes_dict = self.create_x509_attributes_dict(misp_object.attributes)
x509_object = X509Certificate()
if 'raw_certificate' in attributes_dict:
raw_certificate = attributes_dict.pop('raw_certificate')
@ -1016,6 +1032,7 @@ class StixBuilder(object):
if attributes_dict:
x509_cert.subject_public_key = self.fill_x509_pubkey(**attributes_dict)
x509_object.certificate = x509_cert
uuid = misp_object.uuid
x509_object.parent.id_ = "{}:x509CertificateObject-{}".format(self.namespace_prefix, uuid)
observable = Observable(x509_object)
observable.id_ = "{}:x509Certificate-{}".format(self.namespace_prefix, uuid)
@ -1384,11 +1401,12 @@ class StixBuilder(object):
custom_object = Custom()
custom_object.custom_properties = CustomProperties()
for attribute in attributes:
property = Property()
property.name = "{} {}: {}".format(name, attribute.type, attribute.object_relation)
property.value = attribute.value
custom_object.custom_properties.append(property)
if attribute.to_ids: to_ids = True
prop = Property()
prop.name = "{} {}: {}".format(name, attribute.type, attribute.object_relation)
prop.value = attribute.value
custom_object.custom_properties.append(prop)
if attribute.to_ids:
to_ids = True
return to_ids, custom_object
@staticmethod

99
app/files/scripts/stix2/stix2misp.py
View File

@ -34,78 +34,39 @@ class StixParser():
self.misp_event['Galaxy'] = []
def loadEvent(self, args):
try:
filename = os.path.join(os.path.dirname(args[0]), args[1])
with open(filename, 'r', encoding='utf-8') as f:
event = json.loads(f.read())
self.filename = filename
self.stix_version = 'stix {}'.format(event.get('spec_version'))
for o in event.get('objects'):
try:
try:
parsed_object = stix2.parse(o, allow_custom=True)
object_type = parsed_object._type
except:
parsed_object = self.parse_custom_stix(o)
object_type = parsed_object['type']
except:
pass
object_uuid = parsed_object['id'].split('--')[1]
if object_type.startswith('x-misp-object'):
object_type = 'x-misp-object'
self.event[object_type][object_uuid] = parsed_object
if not self.event:
print(json.dumps({'success': 0, 'message': 'There is no valid STIX object to import'}))
sys.exit(1)
filename = os.path.join(os.path.dirname(args[0]), args[1])
with open(filename, 'r', encoding='utf-8') as f:
event = json.loads(f.read())
self.filename = filename
self.stix_version = 'stix {}'.format(event.get('spec_version'))
for o in event.get('objects'):
parsed_object = stix2.parse(o, allow_custom=True)
try:
event_distribution = args[2]
if not isinstance(event_distribution, int):
event_distribution = int(event_distribution) if event_distribution.isdigit() else 5
except IndexError:
event_distribution = 5
try:
attribute_distribution = args[3]
if attribute_distribution != 'event' and not isinstance(attribute_distribution, int):
attribute_distribution = int(attribute_distribution) if attribute_distribution.isdigit() else 5
except IndexError:
attribute_distribution = 5
self.misp_event.distribution = event_distribution
self.__attribute_distribution = event_distribution if attribute_distribution == 'event' else attribute_distribution
self.load_mapping()
except:
print(json.dumps({'success': 0, 'message': 'The STIX file could not be read'}))
object_type = parsed_object._type
except AttributeError:
object_type = parsed_object['type']
object_uuid = parsed_object['id'].split('--')[1]
if object_type.startswith('x-misp-object'):
object_type = 'x-misp-object'
self.event[object_type][object_uuid] = parsed_object
if not self.event:
print(json.dumps({'success': 0, 'message': 'There is no valid STIX object to import'}))
sys.exit(1)
def parse_custom_stix(self, obj):
custom_object_type = obj.pop('type')
labels = obj['labels']
try:
@stix2.CustomObject(custom_object_type,[('id', stix2.properties.StringProperty(required=True)),
('x_misp_timestamp', stix2.properties.StringProperty(required=True)),
('labels', stix2.properties.ListProperty(labels, required=True)),
('x_misp_value', stix2.properties.StringProperty(required=True)),
('created_by_ref', stix2.properties.StringProperty(required=True)),
('x_misp_comment', stix2.properties.StringProperty()),
('x_misp_category', stix2.properties.StringProperty())
])
class Custom(object):
def __init__(self, **kwargs):
return
custom = Custom(**obj)
except:
@stix2.CustomObject(custom_object_type,[('id', stix2.properties.StringProperty(required=True)),
('x_misp_timestamp', stix2.properties.StringProperty(required=True)),
('labels', stix2.properties.ListProperty(labels, required=True)),
('x_misp_values', stix2.properties.DictionaryProperty(required=True)),
('created_by_ref', stix2.properties.StringProperty(required=True)),
('x_misp_comment', stix2.properties.StringProperty()),
('x_misp_category', stix2.properties.StringProperty())
])
class Custom(object):
def __init__(self, **kwargs):
return
custom = Custom(**obj)
return stix2.parse(custom)
event_distribution = args[2]
if not isinstance(event_distribution, int):
event_distribution = int(event_distribution) if event_distribution.isdigit() else 5
except IndexError:
event_distribution = 5
try:
attribute_distribution = args[3]
if attribute_distribution != 'event' and not isinstance(attribute_distribution, int):
attribute_distribution = int(attribute_distribution) if attribute_distribution.isdigit() else 5
except IndexError:
attribute_distribution = 5
self.misp_event.distribution = event_distribution
self.__attribute_distribution = event_distribution if attribute_distribution == 'event' else attribute_distribution
self.load_mapping()
def load_mapping(self):
self.objects_mapping = {'asn': {'observable': observable_asn, 'pattern': pattern_asn},

10
app/files/scripts/stix2/stix2misp_mapping.py
View File

@ -390,7 +390,7 @@ def pattern_ip_port(pattern):
def observable_process(observable):
attributes = []
observable_object = observable['0'] if len(observable) == 1 else parse_process_observable(observable)
observable_object = dict(observable['0']) if len(observable) == 1 else parse_process_observable(observable)
try:
parent_key = observable_object.pop('parent_ref')
attributes.append({'type': 'text', 'value': observable[parent_key]['pid'], 'object_relation': 'parent-pid'})
@ -409,7 +409,7 @@ def parse_process_observable(observable):
for key in observable:
observable_object = observable[key]
if observable_object['type'] == 'process' and ('parent_ref' in observable_object or 'child_refs' in observable_object):
return observable_object
return dict(observable_object)
def pattern_process(pattern):
attributes = []
@ -458,7 +458,7 @@ def pattern_regkey(pattern):
return attributes
def observable_socket(observable):
observable_object = observable['0'] if len(observable) == 1 else parse_socket_observable(observable)
observable_object = dict(observable['0']) if len(observable) == 1 else parse_socket_observable(observable)
try:
extension = observable_object.pop('extensions')
attributes = parse_socket_extension(extension['socket-ext'])
@ -487,7 +487,7 @@ def parse_socket_observable(observable):
for key in observable:
observable_object = observable[key]
if observable_object['type'] == 'network-traffic':
return observable_object
return dict(observable_object)
def parse_socket_extension(extension):
attributes = []
@ -499,7 +499,7 @@ def parse_socket_extension(extension):
if element in ('is_listening', 'is_blocking'):
attribute_value = element.split('_')[1]
else:
attribute_value = extension['element']
attribute_value = extension[element]
attributes.append({'type': mapping['type'], 'object_relation': mapping['relation'],
'value': attribute_value})
return attributes

390
app/files/scripts/stix2misp.py
View File

@ -15,25 +15,17 @@
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import sys, json, os, time, uuid
import mixbox.namespaces as mixbox_ns
import sys
import json
import os
import time
import uuid
import stix2misp_mapping
from operator import attrgetter
from pymisp import MISPEvent, MISPObject, MISPAttribute, __path__
from stix.core import STIXPackage
from collections import defaultdict
file_object_type = {"type": "filename", "relation": "filename"}
eventTypes = {"ArtifactObjectType": {"type": "attachment", "relation": "attachment"},
"DomainNameObjectType": {"type": "domain", "relation": "domain"},
"FileObjectType": file_object_type,
"HostnameObjectType": {"type": "hostname", "relation": "host"},
"MutexObjectType": {"type": "mutex", "relation": "mutex"},
"PDFFileObjectType": file_object_type,
"PortObjectType": {"type": "port", "relation": "port"},
"URIObjectType": {"type": "url", "relation": "url"},
"WindowsExecutableFileObjectType": file_object_type,
"WindowsRegistryKeyObjectType": {"type": "regkey", "relation": ""}}
cybox_to_misp_object = {"Account": "credential", "AutonomousSystem": "asn",
"EmailMessage": "email", "NetworkConnection": "network-connection",
"NetworkSocket": "network-socket", "Process": "process",
@ -62,11 +54,11 @@ class StixParser():
try:
import maec
print(2)
except:
except ModuleNotFoundError:
print(3)
sys.exit(0)
title = event.stix_header.title
fromMisp = (title is not None and "Export from " in title and "MISP" in title)
fromMISP = (title is not None and "Export from " in title and "MISP" in title)
if fromMISP:
package = event.related_packages.related_package[0].item
self.event = package.incidents[0]
@ -163,14 +155,17 @@ class StixParser():
def parse_journal_entries(self):
for entry in self.event.history.history_items:
journal_entry = entry.journal_entry.value
entry_type, entry_value = journal_entry.split(': ')
if entry_type == "MISP Tag":
self.parse_tag(entry_value)
elif entry_type.startswith('attribute['):
_, category, attribute_type = entry_type.split('[')
self.misp_event.add_attribute(**{'type': attribute_type[:-1], 'category': category[:-1], 'value': entry_value})
elif entry_type == "Event Threat Level":
self.misp_event.threat_level_id = threat_level_mapping[entry_value]
try:
entry_type, entry_value = journal_entry.split(': ')
if entry_type == "MISP Tag":
self.parse_tag(entry_value)
elif entry_type.startswith('attribute['):
_, category, attribute_type = entry_type.split('[')
self.misp_event.add_attribute(**{'type': attribute_type[:-1], 'category': category[:-1], 'value': entry_value})
elif entry_type == "Event Threat Level":
self.misp_event.threat_level_id = threat_level_mapping[entry_value]
except ValueError:
continue
def parse_tag(self, entry):
if entry.startswith('misp-galaxy:'):
@ -201,10 +196,9 @@ class StixParser():
# Parse a DNS object
def resolve_dns_objects(self):
for domain in self.dns_objects['domain']:
domain_object = self.dns_objects['domain'][domain]
ip_reference = domain_object['related']
domain_attribute = domain_object['data']
for domain, domain_dict in self.dns_objects['domain'].items():
ip_reference = domain_dict['related']
domain_attribute = domain_dict['data']
if ip_reference in self.dns_objects['ip']:
misp_object = MISPObject('passive-dns')
domain_attribute['object_relation'] = "rrname"
@ -218,10 +212,9 @@ class StixParser():
self.misp_event.add_object(**misp_object)
else:
self.misp_event.add_attribute(**domain_attribute)
for ip in self.dns_objects['ip']:
for ip, ip_dict in self.dns_objects['ip'].items():
if ip not in self.dns_ips:
# print(ip)
self.misp_event.add_attribute(**self.dns_objects['ip'][ip])
self.misp_event.add_attribute(**ip_dict)
def set_distribution(self):
for attribute in self.misp_event.attributes:
@ -344,17 +337,17 @@ class StixParser():
# Define type & value of an attribute or object in MISP
def handle_attribute_type(self, properties, is_object=False, title=None, observable_id=None):
xsi_type = properties._XSI_TYPE
try:
args = [properties]
if xsi_type in ("FileObjectType", "PDFFileObjectType"):
args.append(is_object)
elif xsi_type == "ArtifactObjectType":
args.append(title)
return self.attribute_types_mapping[xsi_type](*args)
except AttributeError:
# ATM USED TO TEST TYPES
print("Unparsed type: {}".format(xsi_type))
sys.exit(1)
# try:
args = [properties]
if xsi_type in ("FileObjectType", "PDFFileObjectType"):
args.append(is_object)
elif xsi_type == "ArtifactObjectType":
args.append(title)
return self.attribute_types_mapping[xsi_type](*args)
# except AttributeError:
# # ATM USED TO TEST TYPES
# print("Unparsed type: {}".format(xsi_type))
# sys.exit(1)
# Return type & value of an ip address attribute
@staticmethod
@ -366,23 +359,15 @@ class StixParser():
return ip_type, properties.address_value.value, "ip"
def handle_as(self, properties):
attributes = []
if properties.number:
attributes.append(['AS', properties.number.value, 'asn'])
if properties.handle:
attributes.append(['AS', properties.handle.value, 'asn'])
if properties.name:
attributes.append(['text', properties.name.value, 'description'])
if len(attributes) == 1:
return attributes[0]
return 'asn', self.return_attributes(attributes), ''
attributes = self.fetch_attributes_with_partial_key_parsing(properties, stix2misp_mapping._as_mapping)
return attributes[0] if len(attributes) == 1 else ('asn', self.return_attributes(attributes), '')
# Return type & value of an attachment attribute
@staticmethod
def handle_attachment(properties, title):
if properties.hashes:
return "malware-sample", "{}|{}".format(title, properties.hashes[0], properties.raw_artifact.value)
return eventTypes[properties._XSI_TYPE]['type'], title, properties.raw_artifact.value
return stix2misp_mapping.eventTypes[properties._XSI_TYPE]['type'], title, properties.raw_artifact.value
# Return type & attributes of a credential object
def handle_credential(self, properties):
@ -390,38 +375,13 @@ class StixParser():
if properties.description:
attributes.append(["text", properties.description.value, "text"])
if properties.authentication:
l_password, l_type, l_format = [], [], []
for authentication in properties.authentication:
if authentication.authentication_type:
auth_type = authentication.authentication_type.value
if auth_type not in l_type:
l_type.append(auth_type)
if authentication.authentication_data:
auth_data = authentication.authentication_data.value
if auth_data not in l_password:
l_password.append(auth_data)
if authentication.structured_authentication_mechanism:
auth_format = authentication.structured_authentication_mechanism.description.value
if auth_format not in l_format:
l_format.append(auth_format)
if l_type:
# auth_type = l_type[0] if len(l_type) == 1 else "".join(["{}, ".format(t) for t in l_type])[:-2]
for auth_type in l_type:
attributes.append(["text", auth_type, "type"])
if l_format:
# auth_format = l_format[0] if len(l_format) == 1 else "".join(["{}, ".format(f) for f in l_format])[:-2]
for auth_format in l_format:
attributes.append(["text", auth_format, "format"])
if l_password:
for password in l_password:
attributes.append(["text", password, "password"])
attributes += self.fetch_attributes_with_key_parsing(authentication, stix2misp_mapping._credential_authentication_mapping)
if properties.custom_properties:
for prop in properties.custom_properties:
if prop.name in ("username", "origin", "notification"):
attributes.append(["text", prop.value, prop.name])
if len(attributes) == 1:
return attributes[0]
return "credential", self.return_attributes(attributes), ""
if prop.name in stix2misp_mapping._credential_custom_types:
attributes.append(['text', prop.value, prop.name])
return attributes[0] if len(attributes) == 1 else ("credential", self.return_attributes(attributes), "")
# Return type & attributes (or value) of a Custom Object
def handle_custom(self, properties):
@ -464,43 +424,31 @@ class StixParser():
# Return type & value of a domain or url attribute
@staticmethod
def handle_domain_or_url(properties):
event_types = eventTypes[properties._XSI_TYPE]
event_types = stix2misp_mapping.eventTypes[properties._XSI_TYPE]
return event_types['type'], properties.value.value, event_types['relation']
# Return type & value of an email attribute
def handle_email_attribute(self, properties):
attributes = []
if properties.header:
header = properties.header
if header.from_:
attributes.append(["email-src", header.from_.address_value.value, "from"])
attributes = self.fetch_attributes_with_key_parsing(header, stix2misp_mapping._email_mapping)
if header.to:
for to in header.to:
attributes.append(["email-dst", to.address_value.value, "to"])
if header.cc:
for cc in header.cc:
attributes.append(["email-dst", cc.address_value.value, "cc"])
if header.reply_to:
attributes.append(["email-reply-to", header.reply_to.address_value.value, "reply-to"])
if header.subject:
attributes.append(["email-subject", header.subject.value, "subject"])
if header.x_mailer:
attributes.append(["email-x-mailer", header.x_mailer.value, "x-mailer"])
if header.boundary:
attributes.append(["email-mime-boundary", header.boundary.value, "mime-boundary"])
if header.user_agent:
attributes.append(["text", header.user_agent.value, "user-agent"])
else:
attributes = []
if properties.attachments:
attributes.append([self.handle_email_attachment(properties.parent)])
if len(attributes) == 1:
return attributes[0]
return "email", self.return_attributes(attributes), ""
attributes.append(self.handle_email_attachment(properties.parent))
return attributes[0] if len(attributes) == 1 else ("email", self.return_attributes(attributes), "")
# Return type & value of an email attachment
@staticmethod
def handle_email_attachment(indicator_object):
properties = indicator_object.related_objects[0].properties
return "email-attachment", properties.file_name.value, "attachment"
return ["email-attachment", properties.file_name.value, "attachment"]
# Return type & attributes of a file object
def handle_file(self, properties, is_object):
@ -510,26 +458,13 @@ class StixParser():
b_hash = True
for h in properties.hashes:
attributes.append(self.handle_hashes_attribute(h))
if properties.file_format and properties.file_format.value:
attributes.append(["mime-type", properties.file_format.value, "mimetype"])
if properties.file_name:
value = properties.file_name.value
if value:
b_file = True
event_types = eventTypes[properties._XSI_TYPE]
attributes.append([event_types['type'], value, event_types['relation']])
if properties.file_path:
value = properties.file_path.value
if value:
attributes.append(['text', value, 'path'])
if properties.byte_runs:
attribute_type = "pattern-in-file"
attributes.append([attribute_type, properties.byte_runs[0].byte_run_data, attribute_type])
if properties.size_in_bytes and properties.size_in_bytes.value:
attribute_type = "size-in-bytes"
attributes.append([attribute_type, properties.size_in_bytes.value, attribute_type])
if properties.peak_entropy and properties.peak_entropy.value:
attributes.append(["float", properties.peak_entropy.value, "entropy"])
attribute_type, relation = stix2misp_mapping.eventTypes[properties._XSI_TYPE]
attributes.append([attribute_type, value, relation])
self.fetch_attributes_with_keys(properties, stix2misp_mapping._file_mapping, attributes)
if len(attributes) == 1:
return attributes[0]
if len(attributes) == 2:
@ -585,7 +520,7 @@ class StixParser():
# Return type & value of a hostname attribute
@staticmethod
def handle_hostname(properties):
event_types = eventTypes[properties._XSI_TYPE]
event_types = stix2misp_mapping.eventTypes[properties._XSI_TYPE]
return event_types['type'], properties.hostname_value.value, event_types['relation']
# Return type & value of a http request attribute
@ -607,42 +542,25 @@ class StixParser():
# Return type & value of a mutex attribute
@staticmethod
def handle_mutex(properties):
event_types = eventTypes[properties._XSI_TYPE]
event_types = stix2misp_mapping.eventTypes[properties._XSI_TYPE]
return event_types['type'], properties.name.value, event_types['relation']
# Return type & attributes of a network connection object
def handle_network_connection(self, properties):
attributes = []
if properties.source_socket_address:
self.handle_socket(attributes, properties.source_socket_address, "src")
if properties.destination_socket_address:
self.handle_socket(attributes, properties.destination_socket_address, "dst")
if properties.layer3_protocol:
attributes.append(["text", properties.layer3_protocol.value, "layer3-protocol"])
if properties.layer4_protocol:
attributes.append(["text", properties.layer4_protocol.value, "layer4-protocol"])
if properties.layer7_protocol:
attributes.append(["text", properties.layer7_protocol.value, "layer7-protocol"])
attributes = self.fetch_attributes_from_sockets(properties, stix2misp_mapping._network_connection_addresses)
for prop in ('layer3_protocol', 'layer4_protocol', 'layer7_protocol'):
if getattr(properties, prop):
attributes.append(['text', attrgetter("{}.value".format(prop))(properties), prop.replace('_', '-')])
if attributes:
return "network-connection", self.return_attributes(attributes), ""
# Return type & attributes of a network socket objet
def handle_network_socket(self, properties):
attributes = []
if properties.local_address:
self.handle_socket(attributes, properties.local_address, "src")
if properties.remote_address:
self.handle_socket(attributes, properties.remote_address, "dst")
if properties.protocol:
attributes.append(["text", properties.protocol.value, "protocol"])
if properties.is_listening:
attributes.append(["text", "listening", "state"])
if properties.is_blocking:
attributes.append(["text", "blocking", "state"])
if properties.address_family:
attributes.append(["text", properties.address_family.value, "address-family"])
if properties.domain:
attributes.append(["text", properties.domain.value, "domain-family"])
attributes = self.fetch_attributes_from_sockets(properties, stix2misp_mapping._network_socket_addresses)
self.fetch_attributes_with_keys(properties, stix2misp_mapping._network_socket_mapping, attributes)
for prop in ('is_listening', 'is_blocking'):
if getattr(properties, prop):
attributes.append(["text", prop.split('_')[1], "state"])
if attributes:
return "network-socket", self.return_attributes(attributes), ""
@ -650,7 +568,7 @@ class StixParser():
@staticmethod
def handle_port(*kwargs):
properties = kwargs[0]
event_types = eventTypes[properties._XSI_TYPE]
event_types = stix2misp_mapping.eventTypes[properties._XSI_TYPE]
relation = event_types['relation']
if len(kwargs) > 1:
observable_id = kwargs[1]
@ -662,21 +580,10 @@ class StixParser():
# Return type & attributes of a process object
def handle_process(self, properties):
attributes = []
if properties.creation_time:
attributes.append(["datetime", properties.creation_time.value, "creation-time"])
if properties.start_time:
attributes.append(["datetime", properties.creation_time.value, "start-time"])
attribute_type = "text"
if properties.name:
attributes.append([attribute_type, properties.name.value, "name"])
if properties.pid:
attributes.append([attribute_type, properties.pid.value, "pid"])
if properties.parent_pid:
attributes.append([attribute_type, properties.parent_pid.value, "parent-pid"])
attributes = self.fetch_attributes_with_partial_key_parsing(properties, stix2misp_mapping._process_mapping)
if properties.child_pid_list:
for child in properties.child_pid_list:
attributes.append([attribute_type, child.value, "child-pid"])
attributes.append(["text", child.value, "child-pid"])
# if properties.port_list:
# for port in properties.port_list:
# attributes.append(["src-port", port.port_value.value, "port"])
@ -695,33 +602,20 @@ class StixParser():
# Return type & value of a regkey attribute
def handle_regkey(self, properties):
attributes = []
if properties.hive:
attributes.append(["text", properties.hive.value, "hive"])
if properties.key:
attributes.append(["regkey", properties.key.value, "key"])
attributes = self.fetch_attributes_with_partial_key_parsing(properties, stix2misp_mapping._regkey_mapping)
if properties.values:
values = properties.values
value = values[0]
if value.data:
attributes.append(["text", value.data.value, "data"])
if value.datatype:
attributes.append(["text", value.datatype.value, "data-type"])
if value.name:
attributes.append(["text", value.name.value, "name"])
attributes += self.fetch_attributes_with_partial_key_parsing(value, stix2misp_mapping._regkey_value_mapping)
return "registry-key", self.return_attributes(attributes), ""
# Parse a socket address object into a network connection or socket object,
# in order to add its attributes
@staticmethod
def handle_socket(attributes, socket, s_type):
if socket.ip_address:
ip_type = "ip-{}".format(s_type)
attributes.append([ip_type, socket.ip_address.address_value.value, ip_type])
if socket.port:
attributes.append(["port", socket.port.port_value.value, "{}-port".format(s_type)])
if socket.hostname:
attributes.append(["hostname", socket.hostname.hostname_value.value, "hostname-{}".format(s_type)])
for prop, mapping in stix2misp_mapping._socket_mapping.items():
if getattr(socket, prop):
attribute_type, properties_key, relation = mapping
attribute_type, relation = [elem.format(s_type) for elem in (attribute_type, relation)]
attributes.append([attribute_type, attrgetter('{}.{}.value'.format(prop, properties_key))(socket), relation])
# Parse a socket address object in order to return type & value
# of a composite attribute ip|port or hostname|port
@ -743,22 +637,11 @@ class StixParser():
# Return type & attributes of a whois object if we have the required fields
# Otherwise create attributes and return type & value of the last attribute to avoid crashing the parent function
def handle_whois(self, properties):
required_one_of = False
attributes = []
if properties.registrar_info:
attribute_type = "whois-registrar"
attributes.append([attribute_type, properties.registrar_info.value, attribute_type])
required_one_of = True
attributes = self.fetch_attributes_with_key_parsing(properties, stix2misp_mapping._whois_mapping)
required_one_of = True if attributes else False
if properties.registrants:
registrant = properties.registrants[0]
if registrant.email_address:
attributes.append(["whois-registrant-email", registrant.email_address.address_value.value, "registrant-email"])
if registrant.name:
attributes.append(["whois-registrant-name", registrant.name.value, "registrant-name"])
if registrant.phone_number:
attributes.append(["whois-registrant-phone", registrant.phone_number.value, "registrant-phone"])
if registrant.organization:
attributes.append(["whois-registrant-org", registrant.organization.value, "registrant-org"])
attributes += self.fetch_attributes_with_key_parsing(registrant, stix2misp_mapping._whois_registrant_mapping)
if properties.creation_date:
attributes.append(["datetime", properties.creation_date.value.strftime('%Y-%m-%d'), "creation-date"])
required_one_of = True
@ -769,13 +652,6 @@ class StixParser():
if properties.nameservers:
for nameserver in properties.nameservers:
attributes.append(["hostname", nameserver.value.value, "nameserver"])
if properties.ip_address:
attributes.append(["ip-src", properties.ip_address.address_value.value, "ip-address"])
required_one_of = True
if properties.domain_name:
attribute_type = "domain"
attributes.append([attribute_type, properties.domain_name.value.value, attribute_type])
required_one_of = True
if properties.remarks:
attribute_type = "text"
relation = "comment" if attributes else attribute_type
@ -785,16 +661,15 @@ class StixParser():
if required_one_of:
# if yes, we return the object type and the attributes
return "whois", self.return_attributes(attributes), ""
else:
# otherwise, attributes are added in the event, and one attribute is returned to not make the function crash
if len(attributes) == 1:
return attributes[0]
last_attribute = attributes.pop(-1)
for attribute in attributes:
attribute_type, attribute_value, attribute_relation = attribute
misp_attributes = {"comment": "Whois {}".format(attribute_relation)}
self.misp_event.add_attribute(attribute_type, attribute_value, **misp_attributes)
return last_attribute
# otherwise, attributes are added in the event, and one attribute is returned to not make the function crash
if len(attributes) == 1:
return attributes[0]
last_attribute = attributes.pop(-1)
for attribute in attributes:
attribute_type, attribute_value, attribute_relation = attribute
misp_attributes = {"comment": "Whois {}".format(attribute_relation)}
self.misp_event.add_attribute(attribute_type, attribute_value, **misp_attributes)
return last_attribute
# Return type & value of a windows service object
@staticmethod
@ -803,33 +678,7 @@ class StixParser():
return "windows-service-name", properties.name.value, ""
def handle_x509(self, properties):
attributes = []
if properties.certificate:
certificate = properties.certificate
if certificate.validity:
validity = certificate.validity
if validity.not_before:
attributes.append(["datetime", validity.not_before.value, "validity-not-before"])
if validity.not_after:
attributes.append(["datetime", validity.not_after.value, "validity-not-after"])
if certificate.subject_public_key:
subject_pubkey = certificate.subject_public_key
if subject_pubkey.rsa_public_key:
rsa_pubkey = subject_pubkey.rsa_public_key
if rsa_pubkey.exponent:
attributes.append(["text", rsa_pubkey.exponent.value, "pubkey-info-exponent"])
if rsa_pubkey.modulus:
attributes.append(["text", rsa_pubkey.modulus.value, "pubkey-info-modulus"])
if subject_pubkey.public_key_algorithm:
attributes.append(["text", subject_pubkey.public_key_algorithm.value, "pubkey-info-algorithm"])
if certificate.version:
attributes.append(["text", certificate.version.value, "version"])
if certificate.serial_number:
attributes.append(["text", certificate.serial_number.value, "serial-number"])
if certificate.issuer:
attributes.append(["text", certificate.issuer.value, "issuer"])
if certificate.subject:
attributes.append(["text", certificate.subject.value, "subject"])
attributes = self.handle_x509_certificate(properties.certificate) if properties.certificate else []
if properties.raw_certificate:
raw = properties.raw_certificate.value
try:
@ -846,6 +695,28 @@ class StixParser():
attributes.append([attribute_type, signature.signature.value, attribute_type])
return "x509", self.return_attributes(attributes), ""
@staticmethod
def handle_x509_certificate(certificate):
attributes = []
if certificate.validity:
validity = certificate.validity
for prop in stix2misp_mapping._x509_datetime_types:
if getattr(validity, prop):
attributes.append(['datetime', attrgetter('{}.value'.format(prop))(validity), 'validity-{}'.format(prop.replace('_', '-'))])
if certificate.subject_public_key:
subject_pubkey = certificate.subject_public_key
if subject_pubkey.rsa_public_key:
rsa_pubkey = subject_pubkey.rsa_public_key
for prop in stix2misp_mapping._x509__x509_pubkey_types:
if getattr(rsa_pubkey, prop):
attributes.append(['text', attrgetter('{}.value'.format(prop))(rsa_pubkey), 'pubkey-info-{}'.format(prop)])
if subject_pubkey.public_key_algorithm:
attributes.append(["text", subject_pubkey.public_key_algorithm.value, "pubkey-info-algorithm"])
for prop in stix2misp_mapping._x509_certificate_types:
if getattr(certificate, prop):
attributes.append(['text', attrgetter('{}.value'.format(prop))(certificate), prop.replace('_', '-')])
return attributes
# Return type & attributes of the file defining a portable executable object
def handle_pe(self, properties):
pe_uuid = self.parse_pe(properties)
@ -973,7 +844,7 @@ class StixParser():
continue
if properties:
attribute_type, attribute_value, compl_data = self.handle_attribute_type(properties)
if type(attribute_value) in (str, int):
if isinstance(attribute_value, (str, int)):
# if the returned value is a simple value, we build an attribute
attribute = {'to_ids': True}
if indicator.timestamp:
@ -1000,8 +871,7 @@ class StixParser():
# print("Error with an object of type: {}\n{}".format(properties._XSI_TYPE, observable.to_json()))
continue
object_uuid = self.fetch_uuid(observable_object.id_)
attr_type = type(attribute_value)
if attr_type in (str, int):
if isinstance(attribute_value, (str, int)):
# if the returned value is a simple value, we build an attribute
attribute = {'to_ids': False, 'uuid': object_uuid}
if observable_object.related_objects:
@ -1014,7 +884,9 @@ class StixParser():
self.dns_ips.append(related_ip)
continue
if attribute_type in ('ip-src', 'ip-dst'):
self.dns_objects['ip'][object_uuid] = {"type": attribute_type, "value": attribute_value}
attribute['type'] = attribute_type
attribute['value'] = attribute_value
self.dns_objects['ip'][object_uuid] = attribute
continue
self.handle_attribute_case(attribute_type, attribute_value, compl_data, attribute)
else:
@ -1061,6 +933,39 @@ class StixParser():
misp_object.add_reference(uuid, 'connected-to')
self.misp_event.add_object(**misp_object)
def fetch_attributes_from_sockets(self, properties, mapping_dict):
attributes = []
for prop, s_type in zip(mapping_dict, stix2misp_mapping._s_types):
address_property = getattr(properties, prop)
if address_property:
self.handle_socket(attributes, address_property, s_type)
return attributes
@staticmethod
def fetch_attributes_with_keys(properties, mapping_dict, attributes):
for prop, mapping in mapping_dict.items():
if getattr(properties,prop):
attribute_type, properties_key, relation = mapping
attributes.append([attribute_type, attrgetter(properties_key)(properties), relation])
@staticmethod
def fetch_attributes_with_key_parsing(properties, mapping_dict):
attributes = []
for prop, mapping in mapping_dict.items():
if getattr(properties, prop):
attribute_type, properties_key, relation = mapping
attributes.append([attribute_type, attrgetter('{}.{}'.format(prop, properties_key))(properties), relation])
return attributes
@staticmethod
def fetch_attributes_with_partial_key_parsing(properties, mapping_dict):
attributes = []
for prop, mapping in mapping_dict.items():
if getattr(properties, prop):
attribute_type, relation = mapping
attributes.append([attribute_type, attrgetter('{}.value'.format(prop))(properties), relation])
return attributes
# Extract the uuid from a stix id
@staticmethod
def fetch_uuid(object_id):
@ -1102,6 +1007,7 @@ class StixParser():
attribute = {'type': 'text', 'object_relation': 'name',
'value': coa.title}
misp_object.add_attribute(**attribute)
# for prop in stix2misp_mapping.
if coa.type_:
attribute = {'type': 'text', 'object_relation': 'type',
'value': coa.type_.value}

57
app/files/scripts/stix2misp_mapping.py Normal file
View File

@ -0,0 +1,57 @@
_file_attribute_type = ('filename', 'filename')
_network_socket_addresses = ['local_address', 'remote_address']
_network_connection_addresses = ['source_socket_address', 'destination_socket_address']
_s_types = ['src', 'dst']
eventTypes = {"ArtifactObjectType": {"type": "attachment", "relation": "attachment"},
"DomainNameObjectType": {"type": "domain", "relation": "domain"},
"FileObjectType": _file_attribute_type,
"HostnameObjectType": {"type": "hostname", "relation": "host"},
"MutexObjectType": {"type": "mutex", "relation": "mutex"},
"PDFFileObjectType": _file_attribute_type,
"PortObjectType": {"type": "port", "relation": "port"},
"URIObjectType": {"type": "url", "relation": "url"},
"WindowsExecutableFileObjectType": _file_attribute_type,
"WindowsRegistryKeyObjectType": {"type": "regkey", "relation": ""}}
_AS_attribute = ('AS', 'asn')
_as_mapping = {'number': _AS_attribute, 'handle': _AS_attribute, 'name': ('text', 'description')}
_credential_authentication_mapping = {'authentication_type': ('text', 'value', 'type'),
'authentication_data': ('text', 'value', 'password'),
'structured_authentication_mechanism': ('text', 'description.value', 'format')}
_credential_custom_types = ("username", "origin", "notification")
_email_mapping = {'from_': ("email-src", "address_value.value", "from"),
'reply_to': ("email-reply-to", 'address_value.value', "reply-to"),
'subject': ("email-subject", 'value', "subject"),
'x_mailer': ("email-x-mailer", 'value', "x-mailer"),
'boundary': ("email-mime-boundary", 'value', "mime-boundary"),
'user_agent': ("text", 'value', "user-agent")}
_file_mapping = {'file_path': ('text', 'file_path.value', 'path'),
'file_format': ('mime-type', 'file_format.value', 'mimetype'),
'byte_runs': ('pattern-in-file', 'byte_runs[0].byte_run_data', 'pattern-in-file'),
'size_in_bytes': ('size-in-bytes', 'size_in_bytes.value', 'size-in-bytes'),
'peak_entropy': ('float', 'peak_entropy.value', 'entropy')}
_network_socket_mapping = {'protocol': ('text', 'protocol.value', 'protocol'),
'address_family': ('text', 'address_family.value', 'address-family'),
'domain': ('text', 'domain.value', 'domain-family')}
_process_mapping = {'creation_time': ('datetime', 'creation-time'),
'start_time': ('datetime', 'start-time'),
'name': ('text', 'name'),
'pid': ('text', 'pid'),
'parent_pid': ('text', 'parent-pid')}
_regkey_mapping = {'hive': ('text', 'hive'), 'key': ('regkey', 'key')}
_regkey_value_mapping = {'data': ('text', 'data'), 'datatype': ('text', 'data-type'), 'name': ('text', 'name')}
_socket_mapping = {'ip_address': ('ip-{}', 'address_value', 'ip-{}'),
'port': ('port', 'port_value', '{}-port'),
'hostname': ('hostname', 'hostname_value', 'hostname-{}')}
_whois_registrant_mapping = {'email_address': ('whois-registrant-email', 'address_value.value', 'registrant-email'),
'name': ('whois-registrant-name', 'value', 'registrant-name'),
'phone_number': ('whois-registrant-phone', 'value', 'registrant-phone'),
'organization': ('whois-registrant-org', 'value', 'registrant-org')}
_whois_mapping = {'registrar_info': ('whois-registrar', 'value', 'whois-registrar'),
'ip_address': ('ip-src', 'address_value.value', 'ip-address'),
'domain_name': ('domain', 'value.value', 'domain')}
_x509_datetime_types = ('not_before', 'not_after')
_x509_pubkey_types = ('exponent', 'modulus')
_x509_certificate_types = ('version', 'serial_number', 'issuer', 'subject')

2
app/files/taxonomies

@ -1 +1 @@
Subproject commit 99f2ebd6eed6a4b48c3d1935badb412c25036e7f
Subproject commit 91d08ccf060ca18bbbeae3d8645d4446be5ddcf0

2
app/files/warninglists

@ -1 +1 @@
Subproject commit e597831a25c2a2f8bf08320420efee4dbb386389
Subproject commit 0ebe53940cb31fe674ea0caf35cfe81269b7b32a

222
tools/gen_misp_types_categories.py Executable file
View File

@ -0,0 +1,222 @@
#!/usr/bin/env python3
'''
by Christophe Vandeplas
Takes the MISP categories and types and saves them in the MISP-book, misp-website and PyMISP format.
# TODO - do some more data-validation between the mappings and defaults, and alert if we're missing something
'''
import json
import os
import re
import subprocess
def make_matrix_header(pos, max_cols):
out = []
out.append('|Category|')
cur_pos = 0
for category in categories:
cur_pos += 1
# skip if we are not there yet
if cur_pos < pos + 1:
continue
# skip if we have reached the max cols
if cur_pos > pos + max_cols:
continue
# we are in the right range
out.append(' {} |'.format(category.replace('|', '&#124;')))
out.append('\n')
out.append('| --- |')
cur_pos = 0
for category in categories:
cur_pos += 1
# skip if we are not there yet
if cur_pos < pos + 1:
continue
# skip if we have reached the max cols
if cur_pos > pos + max_cols:
continue
# we are in the right range
out.append(':---:|')
out.append('\n')
return out
def make_matrix_content(pos, max_cols):
out = []
for t in types:
cur_pos = 0
out.append('|{}|'.format(t.replace('|', '&#124;')))
for category in categories:
cur_pos += 1
# skip if we are not there yet
if cur_pos < pos + 1:
continue
# skip if we have reached the max cols
if cur_pos > pos + max_cols:
continue
# we are in the right range
if t in category_definitions[category]['types']:
out.append(' X |')
else:
out.append(' |')
out.append('\n')
return out
# verify if the folders exist before continuing
folders = ['PyMISP', 'misp-book', 'misp-website', 'misp-rfc']
for folder in folders:
if not os.path.isdir('../../' + folder):
exit("Make sure you git clone all the folders before running the script: {}".format(folders))
# Extract categoryDefinitions and typeDefinitions
#################################################
# We do this by:
# - reading out the PHP file
# - extracting the variables in PHP code
# - using PHP to convert it to a JSON
# - read the JSON in python
with open('../app/Model/Attribute.php', 'r') as f:
attribute_php_file = f.read()
re_match = re.search(r'\$categoryDefinitions\s?=\s?([^;]+);', attribute_php_file)
php_code = re_match.group(1)
category_definitions_binary = subprocess.run(['php', '-r', 'echo json_encode({});'.format(php_code)], stdout=subprocess.PIPE).stdout
category_definitions = json.loads(category_definitions_binary.decode('utf-8'))
categories = list(category_definitions.keys())
categories.sort()
re_match = re.search(r'\$typeDefinitions\s?=\s?([^;]+);', attribute_php_file)
php_code = re_match.group(1)
type_definitions_binary = subprocess.run(['php', '-r', 'echo json_encode({});'.format(php_code)], stdout=subprocess.PIPE).stdout
type_definitions = json.loads(type_definitions_binary.decode('utf-8'))
types = list(type_definitions.keys())
types.sort()
# Generate matrix and list
##########################
matrix_and_list = []
# build the matrix
col_count = len(categories)
col_max = 6
col_pos = 0
while col_pos < col_count:
# make the header
matrix_and_list += make_matrix_header(col_pos, col_max)
# make the content
matrix_and_list += make_matrix_content(col_pos, col_max)
matrix_and_list.append('\n')
col_pos += col_max
# build the Categories list
matrix_and_list.append("\n### Categories\n\n")
for category in categories:
matrix_and_list.append("* **{}**: {}\n".format(category.replace('|', '&#124;'), category_definitions[category]['desc'].replace('|', '&#124;')))
# build the Types list
matrix_and_list.append("\n### Types\n\n")
for t in types:
matrix_and_list.append("* **{}**: {}\n".format(t.replace('|', '&#124;'), type_definitions[t]['desc'].replace('|', '&#124;')))
# MISP-book
#############
# overwrite full file
print("Updating MISP book - ../misp-book/categories-and-types/README.md")
misp_book = ('<!-- toc -->\n'
'\n'
'### Attribute Categories vs. Types\n\n')
misp_book += ''.join(matrix_and_list)
with open('../../misp-book/categories-and-types/README.md', 'w') as f:
f.write(misp_book)
# MISP-website
##############
# Replace the select content of the file
# Find the offset of the start header: "### MISP default attributes and categories"
# Find the offset of the end/next header: "## MISP objects"
# Replace our new content in between
print("Updating MISP website - ../../misp-website/_pages/datamodels.md")
misp_website = []
store_lines = True
with open('../../misp-website/_pages/datamodels.md', 'r') as f:
for line in f:
# start marker
if store_lines:
misp_website.append(line)
if line.startswith('### MISP default attributes and categories'):
store_lines = False
misp_website.append('\n### Attribute Categories vs. Types\n\n')
misp_website += matrix_and_list
misp_website.append('\n')
elif line.startswith('## MISP objects'):
store_lines = True
misp_website.append(line)
with open('../../misp-website/_pages/datamodels.md', 'w') as f:
f.write(''.join(misp_website))
# MISP-rfc
##########
# Replace the select content of the file
# Find the offset of the start header: "The list of valid category-type combinations is as follows:"
# Find the offset of the end/next header: "Attributes are based on the usage within their different communities"
# Replace our new content in between
print("Updating MISP RFC - ../../misp-rfc/misp-core-format/raw.md")
misp_rfc = []
rfc_list = []
for category in categories:
rfc_list.append('\n**{}**\n'.format(category))
rfc_list.append(': ')
rfc_list.append(', '.join(category_definitions[category]['types']))
rfc_list.append('\n')
with open('../../misp-rfc/misp-core-format/raw.md', 'r') as f:
for line in f:
# start marker
if store_lines:
misp_rfc.append(line)
if 'The list of valid category-type combinations is as follows:' in line:
store_lines = False
misp_rfc += rfc_list
misp_rfc.append('\n')
elif 'Attributes are based on the usage within their different communities' in line:
store_lines = True
misp_rfc.append(line)
with open('../../misp-rfc/misp-core-format/raw.md', 'w') as f:
f.write(''.join(misp_rfc))
# PyMISP
########
print("Updating PyMISP - ../../PyMISP/pymisp/data/describeTypes.json")
describe_types = {'result': {}}
describe_types['result']['types'] = types
describe_types['result']['categories'] = categories
describe_types['result']['category_type_mappings'] = {}
for category in categories:
describe_types['result']['category_type_mappings'][category] = category_definitions[category]['types']
describe_types['result']['sane_defaults'] = {}
for t in types:
if t not in describe_types['result']['sane_defaults']:
describe_types['result']['sane_defaults'][t] = {}
describe_types['result']['sane_defaults'][t]['default_category'] = type_definitions[t]['default_category']
describe_types['result']['sane_defaults'][t]['to_ids'] = type_definitions[t]['to_ids']
with open('../../PyMISP/pymisp/data/describeTypes.json', 'w') as f:
json.dump(describe_types, f, sort_keys=True, indent=2)
print("\nPlease initiate the git commit and push for each repository!")
print("- misp-book")
print("- misp-website")
print("- misp-rfc")
print("- PyMISP")