MISP
/
MISP
mirror of https://github.com/MISP/MISP
2
2
Fork 0
Code Issues Releases Wiki Activity

Merge branch '2.4' into feature/sightings 

Conflicts:
	app/Controller/SightingsController.php
	app/Model/Sighting.php
pull/1092/head
iglocska 2015-12-25 00:49:23 +01:00
parent 2ccdef8626 a265553f71
commit 61ea39a6ca
19 changed files with 400 additions and 368 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
INSTALL/screenshots/misp-panorama.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 360 KiB

47
README.md
View File

@ -5,49 +5,60 @@ MISP - Malware Information Sharing Platform
![logo](./INSTALL/logos/misp-logo.png?raw=true "MISP")
The problem that we experienced in the past was the difficulty to exchange information about (targeted) malwares and attacks within a group of trusted partners, or a bilateral agreement.
Even today much of the information exchange happens in unstructured reports where you have to copy-paste the information in your own text-files that you then have to parse to export to (N)IDS and systems like log-searches, etc...
MISP, Malware Information Sharing Platform and Threat Sharing, is an open source software solution for collecting, storing, distributing and sharing cyber security indicators and threat about cyber security incidents analysis and malware analysis. MISP is designed by and for incident analysts, security and ICT professionals or malware reverser to support their day-to-day operations to share structured informations efficiently.
A huge challenge in the Cyber Security domain is the information sharing inside and between organizations.
This Malware Information Sharing Platform has as goal to facilitate:
- **central IOC database**: storing technical and non-technical information about malwares and attacks, ... Data from external instances is also imported into your local instance
- **correlation**: automatically creating relations between malwares, events and attributes
- **storing data** in a structured format (allowing automated use of the database for various purposes)
- **export**: generating IDS, OpenIOC, plain text, xml output to integrate with other systems (network IDS, host IDS, custom tools, …)
- **import**: bulk-import, batch-import, import from OpenIOC, GFI sandbox, ThreatConnect CSV, ...
- **data-sharing**: automatically exchange and synchronization with other parties and trust-groups using MISP
- **STIX support**: export data in the STIX format (XML and json)
The objective of MISP is to foster the sharing of structured information within the security community and abroad. MISP provides functionalities to support the exchange of information but also the consumption of the information by Network Detection Intrusion System (NIDS), LIDS but also log analysis tools, SIEMs.
Exchanging info results in *faster detection* of targeted attacks and improves the detection ratio while reducing the false positives. We also avoid reversing similar malware as we know very fast that others already worked on this malware.
The Red October malware for example gives a similar view:
MISP, Malware Information Sharing Platform and Threat Sharing, core functionalities are:
- An **efficient IOC and indicators** database allowing to store technical and non-technical information about malware samples, incidents, attackers and intelligence.
- Automatic **correlation** finding relationships between attributes and indicators from malware, attacks campaigns or analysis.
- Built-in **sharing functionality** to ease data sharing using different model of distributions. MISP can synchronize automatically events and attributes among different MISP. Advanced filtering functionalities can be used to meet each organization sharing policy including a **flexible sharing group** capacity and an attribute level distribution mechanisms.
- An **intuitive user-interface** for end-users to create, update and collaborate on events and attributes/indicators. A **graphical interface** to navigate seamlessly between events and their correlations.
- **storing data** in a structured format (allowing automated use of the database for various purposes) with an extensive support of cyber security indicators along fraud indicators as in the financial sector.
- **export**: generating IDS, OpenIOC, plain text, CSV, MISP XML or JSON output to integrate with other systems (network IDS, host IDS, custom tools)
- **import**: bulk-import, batch-import, import from OpenIOC, GFI sandbox, ThreatConnect CSV.
- Flexible **free text import** tool to ease the integration of unstructured reports into MISP.
- A gentle system to **collaborate** on events and attributes allowing MISP users to propose changes or updates to attributes/indicators.
- **data-sharing**: automatically exchange and synchronization with other parties and trust-groups using MISP.
- Flexible **API** to integrate MISP with your own solutions. MISP is bundled with [PyMISP](https://github.com/MISP/PyMISP) which is a flexible Python Library to fetch, add or update events attributes, handle malware samples or search for attributes.
- **Adjustable taxonomy** to classify and tag events following your own classification schemes or [existing classification](https://github.com/MISP/misp-taxonomies). The taxonomy can be local to your MISP but also shareable among MISP instances.
- **STIX support**: export data in the STIX format (XML and JSON).
Exchanging info results in *faster detection* of targeted attacks and improves the detection ratio while reducing the false positives. We also avoid reversing similar malware as we know very fast that others team or organizations who already analyzed a specific malware.
![MISP 2.4 overview](https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/screenshots/misp-panorama.png)
A sample event encoded in MISP:
![red october](http://3.bp.blogspot.com/-B3h0xbX7RjI/Uftvmq05rHI/AAAAAAAAApo/I0OEYOAFUI4/s1600/red-oct-1.jpg)
![red october](http://1.bp.blogspot.com/-LnMVhq4Rpyk/UftvmguodBI/AAAAAAAAAps/e22fomGL2MU/s1600/red-oct-2.jpg)
Some people might think about CIF (Collective Intelligence Framework) and CRITs (Collaborative Research Into Threats), however those tools are different. Each one has its strenghts and weaknesses, but in the end MISP will rule the world of course.
Website / Support
------------------
Checkout the [website](http://www.misp-project.org) for more information about MISP like [features](http://www.misp-project.org/#features), [roadmap](http://www.misp-project.org/#roadmap), <small>(commercial)</small> [support](http://www.misp-project.org/#support), ... : http://misp-project.org
Documentation
-------------
[MISP user-guide](https://github.com/MISP/misp-book) is available [online](https://www.circl.lu/doc/misp/) or as [PDF](https://www.circl.lu/doc/misp/book.pdf) or as [EPUB](https://www.circl.lu/doc/misp/book.epub) or as [MOBI/Kindle](https://www.circl.lu/doc/misp/book.mobi).
Contributing
------------
Feel free to fork the code, play with it, make some patches and send us the pull requests.
Feel free to fork the code, play with it, make some patches and send us the pull requests via the [issues](https://github.com/MISP/MISP/issues).
Feel free to contact us, create [issues](https://github.com/MISP/MISP/issues), if you have questions, remarks or bug reports.
There are 2 branches:
- develop: (very active development) new features and improvements
- main: what we consider as stable
- develop: (very active development) new features and improvements.
- 2.4 (current stable version): what we consider as stable with frequent updates as hot-fixes.
License
-------

7
app/Console/Command/EventShell.php
View File

@ -210,8 +210,7 @@ class EventShell extends AppShell
$this->Job->id = $id;
$format = $this->args[2];
$sid = $this->args[3];
// TEMP: change to passing an options array with the user!!
$eventIds = $this->Event->fetchEventIds($user);
$eventIds = array_values($this->Event->fetchEventIds($user, false, false, false, true));
$eventCount = count($eventIds);
$dir = new Folder(APP . DS . '/tmp/cached_exports/' . $format);
if ($user['Role']['perm_site_admin']) {
@ -222,9 +221,9 @@ class EventShell extends AppShell
$file->write('');
foreach ($eventIds as $k => $eventId) {
if ($k == 0) {
$temp = $this->Attribute->nids($user, $format, $eventId['Event']['id']);
$temp = $this->Attribute->nids($user, $format, $eventId);
} else {
$temp = $this->Attribute->nids($user, $format, $eventId['Event']['id'], true);
$temp = $this->Attribute->nids($user, $format, $eventId, true);
}
foreach ($temp as $line) {
$file->append($line . PHP_EOL);

10
app/Controller/AppController.php
View File

@ -89,7 +89,9 @@ class AppController extends Controller {
// send users away that are using ancient versions of IE
// Make sure to update this if IE 20 comes out :)
if(preg_match('/(?i)msie [2-8]/',$_SERVER['HTTP_USER_AGENT']) && !strpos($_SERVER['HTTP_USER_AGENT'], 'Opera')) throw new MethodNotAllowedException('You are using an unsecure and outdated version of IE, please download Google Chrome, Mozilla Firefox or update to a newer version of IE. If you are running IE9 or newer and still receive this error message, please make sure that you are not running your browser in compatibility mode. If you still have issues accessing the site, get in touch with your administration team at ' . Configure::read('MISP.contact'));
if (isset($_SERVER['HTTP_USER_AGENT'])) {
if(preg_match('/(?i)msie [2-8]/',$_SERVER['HTTP_USER_AGENT']) && !strpos($_SERVER['HTTP_USER_AGENT'], 'Opera')) throw new MethodNotAllowedException('You are using an unsecure and outdated version of IE, please download Google Chrome, Mozilla Firefox or update to a newer version of IE. If you are running IE9 or newer and still receive this error message, please make sure that you are not running your browser in compatibility mode. If you still have issues accessing the site, get in touch with your administration team at ' . Configure::read('MISP.contact'));
}
// REST authentication
if ($this->_isRest() || $this->_isAutomation()) {
@ -439,7 +441,7 @@ class AppController extends Controller {
));
foreach ($attributes as $k => $attribute) {
if ($k > 0) {
$attribute['Attribute']['uuid'] = $this->{$this->alias}->generateUuid();
$attribute['Attribute']['uuid'] = $this->Attribute->generateUuid();
$this->Attribute->save($attribute);
$counter++;
}
@ -499,7 +501,7 @@ class AppController extends Controller {
$this->loadModel('Server');
if (!Configure::read('MISP.background_jobs')) {
$this->Server->upgrade2324($this->Auth->user('id'));
$this->Session->setFlash('Done. For more details check the audit logs.');
$this->Session->setFlash('Done. For more details check the audit logs.');
$this->redirect(array('controller' => 'pages', 'action' => 'display', 'administration'));
} else {
$job = ClassRegistry::init('Job');
@ -510,7 +512,7 @@ class AppController extends Controller {
'job_input' => 'Old database',
'status' => 0,
'retries' => 0,
'org_id' => $this->Auth->user('org_id'),
'org_id' => 0,
'message' => 'Job created.',
);
$job->save($data);

206
app/Controller/EventsController.php
View File

@ -1039,177 +1039,17 @@ class EventsController extends AppController {
}
if ($this->request->is('post') || $this->request->is('put')) {
if ($this->_isRest()) {
$saveEvent = false;
if ($this->_isRest()) {
if (isset($this->request->data['response'])) $this->request->data = $this->Event->updateXMLArray($this->request->data, true);
else $this->request->data = $this->Event->updateXMLArray($this->request->data, false);
}
// Workaround for different structure in XML/array than what CakePHP expects
if (isset($this->request->data['response'])) $this->request->data = $this->request->data['response'];
$this->request->data = $this->Event->cleanupEventArrayFromXML($this->request->data);
// the event_id field is not set (normal) so make sure no validation errors are thrown
// LATER do this with $this->validator()->remove('event_id');
unset($this->Event->Attribute->validate['event_id']);
unset($this->Event->Attribute->validate['value']['unique']); // otherwise gives bugs because event_id is not set
// http://book.cakephp.org/2.0/en/models/saving-your-data.html
// Creating or updating is controlled by the models id field.
// If $Model->id is set, the record with this primary key is updated.
// Otherwise a new record is created
// reposition to get the event.id with given uuid
$existingEvent = $this->Event->findByUuid($this->request->data['Event']['uuid']);
// If the event exists...
$dateObj = new DateTime();
$date = $dateObj->getTimestamp();
if (count($existingEvent)) {
$this->request->data['Event']['id'] = $existingEvent['Event']['id'];
// Conditions affecting all:
// user.org == event.org
// edit timestamp newer than existing event timestamp
if (!isset($this->request->data['Event']['timestamp'])) $this->request->data['Event']['timestamp'] = $date;
if ($this->request->data['Event']['timestamp'] > $existingEvent['Event']['timestamp']) {
if ($this->request->data['Event']['distribution'] == 4) {
$this->request->data['Event']['sharing_group_id'] = $this->Event->SharingGroup->captureSG($this->request->data['Event']['SharingGroup'], $this->Auth->user());
unset ($this->request->data['Event']['SharingGroup']);
}
// If the above is true, we have two more options:
// For users that are of the creating org of the event, always allow the edit
// For users that are sync users, only allow the edit if the event is locked
if ($existingEvent['Event']['orgc_id'] === $this->_checkOrg()
|| ($this->userRole['perm_sync'] && $existingEvent['Event']['locked']) || $this->_isSiteAdmin()) {
if ($this->userRole['perm_sync']) {
if ($this->Event->data['Event']['distribution'] == 4 && !$this->Event->SharingGroup->checkIfAuthorisedExtend($this->Auth->user(), $this->Event->data['Event']['sharing_group_id'])) {
throw new MethodNotAllowedException('Event could not be saved: The sync user has to either be an extender of the sharing group or be the sync user that has first synchronised the sharing group to this instance.');
}
}
// Only allow an edit if this is true!
$saveEvent = true;
} else throw new MethodNotAllowedException('Event could not be saved: The user used to edit the event is not authorised to do so. This can be caused by the user not being of the same organisation as the original creator of the event whilst also not being a site administrator.');
} else throw new MethodNotAllowedException('Event could not be saved: Event in the request not newer than the local copy.');
// If a field is not set in the request, just reuse the old value
$recoverFields = array('analysis', 'threat_level_id', 'info', 'distribution', 'date');
foreach ($recoverFields as $rF) if (!isset($this->request->data['Event'][$rF])) $this->request->data['Event'][$rF] = $existingEvent['Event'][$rF];
} else throw new MethodNotAllowedException('Event could not be saved: Could not find the local event.');
$fieldList = array(
'Event' => array('date', 'threat_level_id', 'analysis', 'info', 'published', 'uuid', 'distribution', 'timestamp'),
'Attribute' => array('event_id', 'category', 'type', 'value', 'value1', 'value2', 'to_ids', 'uuid', 'revision', 'distribution', 'timestamp', 'comment', 'sharing_group_id'),
'ShadowAttribute' => array('event_id', 'category', 'type', 'value', 'value1', 'value2', 'org_id', 'event_org_id', 'comment', 'event_uuid', 'deleted', 'to_ids', 'uuid')
);
$saveResult = $this->Event->save(array('Event' => $this->request->data['Event']), array('fieldList' => $fieldList['Event']));
$this->loadModel('Log');
if ($saveResult) {
$validationErrors = array();
if (isset($this->request->data['Event']['Attribute'])) {
foreach ($this->request->data['Event']['Attribute'] as $k => $attribute) {
if (isset($attribute['uuid'])) {
$existingAttribute = $this->Event->Attribute->findByUuid($attribute['uuid']);
if (count($existingAttribute)) {
if ($existingAttribute['Attribute']['event_id'] != $id) {
$result = $this->Log->save(array(
'org' => $this->Auth->user('Organisation')['name'],
'model' => 'Event',
'model_id' => $id,
'email' => $this->Auth->user('email'),
'action' => 'edit',
'user_id' => $this->Auth->user('id'),
'title' => 'Duplicate UUID found in attribute',
'change' => 'An attribute was blocked from being saved due to a duplicate UUID. The uuid in question is: ' . $attribute['uuid'],
));
unset($this->request->data['Event']['Attribute'][$k]);
} else {
// If a field is not set in the request, just reuse the old value
$recoverFields = array('value', 'to_ids', 'distribution', 'category', 'type', 'comment', 'sharing_group_id');
foreach ($recoverFields as $rF) if (!isset($attribute[$rF])) $this->request->data['Event']['Attribute'][$c][$rF] = $existingAttribute['Attribute'][$rF];
$this->request->data['Event']['Attribute'][$k]['id'] = $existingAttribute['Attribute']['id'];
// Check if the attribute's timestamp is bigger than the one that already exists.
// If yes, it means that it's newer, so insert it. If no, it means that it's the same attribute or older - don't insert it, insert the old attribute.
// Alternatively, we could unset this attribute from the request, but that could lead with issues if we decide that we want to start deleting attributes that don't exist in a pushed event.
if (isset($this->request->data['Event']['Attribute'][$k]['timestamp'])) {
if ($this->request->data['Event']['Attribute'][$k]['timestamp'] <= $existingAttribute['Attribute']['timestamp']) {
unset($this->request->data['Event']['Attribute'][$k]);
continue;
}
} else $this->request->data['Event']['timestamp'] = $date;
}
} else {
$this->Event->Attribute->create();
}
} else {
$this->Event->Attribute->create();
}
$this->request->data['Event']['Attribute'][$k]['event_id'] = $this->Event->id;
if ($this->request->data['Event']['Attribute'][$k]['distribution'] == 4) {
$sid = $this->Event->SharingGroup->captureSG($this->request->data['Event']['Attribute'][$k]['SharingGroup'], $this->Auth->user());
$this->request->data['Event']['Attribute'][$k]['sharing_group_id'] = $this->Event->SharingGroup->captureSG($this->request->data['Event']['Attribute'][$k]['SharingGroup'], $this->Auth->user());
}
if (!$this->Event->Attribute->save($this->request->data['Event']['Attribute'][$k], array('fieldList' => $fieldList))) {
$validationErrors['Attribute'][$k] = $this->Event->Attribute->validationErrors;
$attribute_short = (isset($this->request->data['Event']['Attribute'][$k]['category']) ? $this->request->data['Event']['Attribute'][$k]['category'] : 'N/A') . '/' . (isset($this->request->data['Event']['Attribute'][$k]['type']) ? $this->request->data['Event']['Attribute'][$k]['type'] : 'N/A') . ' ' . (isset($this->request->data['Event']['Attribute'][$k]['value']) ? $this->request->data['Event']['Attribute'][$k]['value'] : 'N/A');
$this->Log->create();
$this->Log->save(array(
'org' => $this->Auth->user('Organisation')['name'],
'model' => 'Attribute',
'model_id' => 0,
'email' => $this->Auth->user('email'),
'action' => 'edit',
'user_id' => $this->Auth->user('id'),
'title' => 'Attribute dropped due to validation for Event ' . $this->id . ' failed: ' . $attribute_short,
'change' => json_encode($this->Event->Attribute->validationErrors),
));
}
}
}
// check if the exact proposal exists, if yes check if the incoming one is deleted or not. If it is deleted, remove the old proposal and replace it with the one marked for being deleted
// otherwise throw the new one away.
if (isset($this->request->data['Event']['ShadowAttribute'])) {
foreach ($this->request->data['Event']['ShadowAttribute'] as $k => &$proposal) {
$existingProposal = $this->Event->ShadowAttribute->find('first', array(
'recursive' => -1,
'conditions' => array(
'value' => $proposal['value'],
'category' => $proposal['category'],
'to_ids' => $proposal['to_ids'],
'type' => $proposal['type'],
'event_uuid' => $proposal['event_uuid'],
'uuid' => $proposal['uuid']
)
));
if (!empty($existingProposal)) {
if ($existingProposal['ShadowAttribute']['deleted'] == 1) {
$this->Event->ShadowAttribute->delete($existingProposal['ShadowAttribute']['id'], false);
} else {
unset($this->request->data['ShadowAttribute'][$k]);
continue;
}
$this->Event->ShadowAttribute->create();
}
$this->Event->ShadowAttribute->save($this->request->data['Event']['ShadowAttribute'][$k], array('fieldList' => $fieldList));
}
}
// this saveAssociated() function will save not only the event, but also the attributes
// from the attributes attachments are also saved to the disk thanks to the afterSave() fonction of Attribute
if ($saveEvent) {
$saveResult = $this->Event->saveAssociated($this->request->data, array('validate' => true, 'fieldList' => $fieldList));
} else {
throw new MethodNotAllowedException();
}
$message = 'Saved';
$this->set('event', $this->Event->data);
//if published -> do the actual publishing
if ((!empty($this->request->data['Event']['published']) && 1 == $this->request->data['Event']['published'])) {
// do the necessary actions to publish the event (email, upload,...)
if ('true' != Configure::read('MISP.disablerestalert')) {
$this->Event->sendAlertEmailRouter($id, $this->Auth->user());
}
$this->Event->publish($existingEvent['Event']['id']);
}
$result = $this->Event->_edit($this->request->data, $this->Auth->user(), $id);
if ($result === true) {
// REST users want to see the newly created event
$results = $this->Event->fetchEvent($this->Auth->user(), array('eventid' => $id));
$event = &$results[0];
if (!empty($validationErrors)) {
$event['errors'] = $validationErrors;
}
$this->set('event', $event);
$this->render('view');
return true;
@ -1218,8 +1058,9 @@ class EventsController extends AppController {
if ($this->_isRest()) {
App::uses('JSONConverterTool', 'Tools');
$converter = new JSONConverterTool();
$errors = $converter->arrayPrinter($this->Event->validationErrors);
$this->set('name', 'Add event failed.');
if (isset($result['error'])) $errors = $result['error'];
else $errors = $converter->arrayPrinter($result);
$this->set('name', 'Edit event failed.');
$this->set('message', $message);
$this->set('errors', $errors);
$this->set('url', '/events/edit/' . $id);
@ -1265,7 +1106,6 @@ class EventsController extends AppController {
// even if the SG is not local, we still want the option to select the currently assigned SG
$sgs = $this->Event->SharingGroup->fetchAllAuthorised($this->Auth->user(), 'name', 1);
if (!isset($sgs[$this->Event->data['SharingGroup']['id']])) $sgs[$this->Event->data['SharingGroup']['id']] = $this->Event->data['SharingGroup']['name'];
$this->set('sharingGroups', $sgs);
$distributionLevels = $this->Event->distributionLevels;
@ -1796,7 +1636,7 @@ class EventsController extends AppController {
foreach ($simpleFalse as $sF) {
if (!is_array(${$sF}) && (${$sF} === 'null' || ${$sF} == '0' || ${$sF} === false || strtolower(${$sF}) === 'false')) ${$sF} = false;
}
$exportType = $eventid;
if ($from) $from = $this->Event->dateFieldCheck($from);
if ($to) $to = $this->Event->dateFieldCheck($to);
if ($tags) $tags = str_replace(';', ':', $tags);
@ -1832,29 +1672,36 @@ class EventsController extends AppController {
foreach ($attributes as &$attribute) {
$list[] = $attribute['Attribute']['id'];
}
} else if ($eventid === false) {
$events = $this->Event->fetchEventIds($this->Auth->user(), false, false, false, true);
if (empty($events)) $events = array(0 => -1);
}
$attributes = $this->Event->csv($user, $eventid, $ignore, $list, $tags, $category, $type, $includeContext, $from, $to, $last);
$this->loadModel('Whitelist');
if (!isset($events)) $events = array(0 => false);
$final = array();
$attributes = $this->Whitelist->removeWhitelistedFromArray($attributes, true);
foreach ($attributes as $attribute) {
$line = $attribute['Attribute']['uuid'] . ',' . $attribute['Attribute']['event_id'] . ',' . $attribute['Attribute']['category'] . ',' . $attribute['Attribute']['type'] . ',' . $attribute['Attribute']['value'] . ',' . $attribute['Attribute']['comment'] . ',' . intval($attribute['Attribute']['to_ids']) . ',' . $attribute['Attribute']['timestamp'];
if ($includeContext) {
foreach($this->Event->csv_event_context_fields_to_fetch as $header => $field) {
if ($field['object']) $line .= ',' . $attribute['Event'][$field['object']][$field['var']];
else $line .= ',' . $attribute['Event'][$field['var']];
$this->loadModel('Whitelist');
foreach ($events as $eventid) {
$attributes = $this->Event->csv($user, $eventid, $ignore, $list, $tags, $category, $type, $includeContext, $from, $to, $last);
$attributes = $this->Whitelist->removeWhitelistedFromArray($attributes, true);
foreach ($attributes as $attribute) {
$line = $attribute['Attribute']['uuid'] . ',' . $attribute['Attribute']['event_id'] . ',' . $attribute['Attribute']['category'] . ',' . $attribute['Attribute']['type'] . ',' . $attribute['Attribute']['value'] . ',' . $attribute['Attribute']['comment'] . ',' . intval($attribute['Attribute']['to_ids']) . ',' . $attribute['Attribute']['timestamp'];
if ($includeContext) {
foreach($this->Event->csv_event_context_fields_to_fetch as $header => $field) {
if ($field['object']) $line .= ',' . $attribute['Event'][$field['object']][$field['var']];
else $line .= ',' . $attribute['Event'][$field['var']];
}
}
$final[] = $line;
}
$final[] = $line;
}
$this->response->type('csv'); // set the content type
if (!$eventid) {
if (!$exportType) {
$this->header('Content-Disposition: download; filename="misp.all_attributes.csv"');
} else if ($eventid === 'search') {
} else if ($exportType === 'search') {
$this->header('Content-Disposition: download; filename="misp.search_result.csv"');
} else {
$this->header('Content-Disposition: download; filename="misp.event_' . $eventid . '.csv"');
$this->header('Content-Disposition: download; filename="misp.event_' . $exportType . '.csv"');
}
$this->layout = 'text/default';
$headers = array('uuid', 'event_id', 'category', 'type', 'value', 'comment', 'to_ids', 'date');
@ -2824,6 +2671,7 @@ class EventsController extends AppController {
'email-dst' => 'Payload delivery',
'text' => 'Other',
);
$this->set('event', $event);
$this->set('typeList', array_keys($this->Event->Attribute->typeDefinitions));
$this->set('defaultCategories', $defaultCategories);
$this->set('typeCategoryMapping', $typeCategoryMapping);

16
app/Controller/ShadowAttributesController.php
View File

@ -1008,9 +1008,19 @@ class ShadowAttributesController extends AppController {
}
$this->ShadowAttribute->recursive = -1;
$temp = $this->ShadowAttribute->findAllByEventUuid($uuid);
if ($temp == null) throw new NotFoundException(__('Invalid event'));
$this->set('proposal', $temp);
$this->render('get_proposals_by_uuid');
if ($temp == null) {
$this->response->statusCode(404);
$this->set('name', 'Invalid Event.');
$this->set('message', 'Invalid Event');
$this->set('errors', 'Invalid Event');
$this->set('url', '/shadow_attributes/getProposalsByUuid/edit/' . $uuid);
$this->set('_serialize', array('name', 'message', 'url', 'errors'));
$this->response->send();
return false;
} else {
$this->set('proposal', $temp);
$this->render('get_proposals_by_uuid');
}
}
public function fetchEditForm($id, $field = null) {

4
app/Controller/TemplatesController.php
View File

@ -276,7 +276,7 @@ class TemplatesController extends AppController {
if ($this->request->is('post')) {
$errors = array();
$this->set('template', $this->request->data);
$result = $this->Event->Attribute->checkTemplateAttributes($template, $this->request->data, $event_id, $event['Event']['distribution']);
$result = $this->Event->Attribute->checkTemplateAttributes($template, $this->request->data, $event_id);
if (isset($this->request->data['Template']['modify']) || !empty($result['errors'])) {
$fileArray = $this->request->data['Template']['fileArray'];
$this->set('fileArray', $fileArray);
@ -288,7 +288,7 @@ class TemplatesController extends AppController {
$this->set('attributes', $result['attributes']);
$fileArray = $this->request->data['Template']['fileArray'];
$this->set('fileArray', $fileArray);
$this->set('distributionLevels', $this->Event->distributionLevels);
$this->set('distributionLevels', $this->Event->Attribute->distributionLevels);
$this->render('populate_event_from_template_attributes');
}
} else {

14
app/Controller/UsersController.php
View File

@ -389,9 +389,10 @@ class UsersController extends AppController {
$conditions = array();
if (!$this->_isSiteAdmin()) $conditions['Server.org_id LIKE'] = $this->Auth->user('org_id');
$temp = $this->Server->find('all', array('conditions' => $conditions, 'recursive' => -1, 'fields' => array('id', 'name')));
$servers = array(0 => 'Nothing');
$servers = array(0 => 'Not bound to a server');
if (!empty($temp)) foreach ($temp as $t) {
$servers[$t['Server']['id']] = $t['Server']['name'];
if (!empty($t['Server']['name'])) $servers[$t['Server']['id']] = $t['Server']['name'];
else $servers[$t['Server']['id']] = $t['Server']['url'];
}
$this->set('servers', $servers);
$this->set(compact('roles'));
@ -518,10 +519,11 @@ class UsersController extends AppController {
$this->loadModel('Server');
$conditions = array();
if (!$this->_isSiteAdmin()) $conditions['Server.org_id LIKE'] = $this->Auth->user('org_id');
$temp = $this->Server->find('all', array('conditions' => $conditions, 'recursive' => -1, 'fields' => array('id', 'name')));
$servers = array(0 => 'Nothing');
$temp = $this->Server->find('all', array('conditions' => $conditions, 'recursive' => -1, 'fields' => array('id', 'name', 'url')));
$servers = array(0 => 'Not bound to a server');
foreach ($temp as $t) {
$servers[$t['Server']['id']] = $t['Server']['name'];
if (!empty($t['Server']['name'])) $servers[$t['Server']['id']] = $t['Server']['name'];
else $servers[$t['Server']['id']] = $t['Server']['url'];
}
$this->set('servers', $servers);
$this->set('orgs', $orgs);
@ -836,7 +838,7 @@ class UsersController extends AppController {
// write to syslogd as well
App::import('Lib', 'SysLog.SysLog');
$syslog = new SysLog();
if ($fieldsResult) $syslog->write('notice', $description . ' -- ' . $action . ' -- ' . $fieldsResult);
if (isset($fieldsResult) && $fieldResult) $syslog->write('notice', $description . ' -- ' . $action . ' -- ' . $fieldsResult);
else $syslog->write('notice', $description . ' -- ' . $action);
}

10
app/Lib/Tools/JSONConverterTool.php
View File

@ -1,19 +1,19 @@
<?php
class JSONConverterTool {
public function event2JSON($event, $isSiteAdmin=false) {
$event['Event']['Org'][0] = $event['Org'];
$event['Event']['Orgc'][0] = $event['Orgc'];
$event['Event']['Org'] = $event['Org'];
$event['Event']['Orgc'] = $event['Orgc'];
if (isset($event['SharingGroup']['SharingGroupOrg'])) {
foreach ($event['SharingGroup']['SharingGroupOrg'] as $key => $sgo) {
$event['SharingGroup']['SharingGroupOrg'][$key]['Organisation'] = array(0 => $event['SharingGroup']['SharingGroupOrg'][$key]['Organisation']);
$event['SharingGroup']['SharingGroupOrg'][$key]['Organisation'] = $event['SharingGroup']['SharingGroupOrg'][$key]['Organisation'];
}
}
if (isset($event['SharingGroup']['SharingGroupServer'])) {
foreach ($event['SharingGroup']['SharingGroupServer'] as $key => $sgs) {
$event['SharingGroup']['SharingGroupServer'][$key]['Server'] = array(0 => $event['SharingGroup']['SharingGroupServer'][$key]['Server']);
$event['SharingGroup']['SharingGroupServer'][$key]['Server'] = $event['SharingGroup']['SharingGroupServer'][$key]['Server'];
}
}
if (isset($event['SharingGroup'])) $event['Event']['SharingGroup'][0] = $event['SharingGroup'];
if (isset($event['SharingGroup'])) $event['Event']['SharingGroup'] = $event['SharingGroup'];
$event['Event']['Attribute'] = $event['Attribute'];
$event['Event']['ShadowAttribute'] = $event['ShadowAttribute'];
$event['Event']['RelatedEvent'] = $event['RelatedEvent'];

2
app/Model/AppModel.php
View File

@ -142,7 +142,7 @@ class AppModel extends Model {
case 'indexTables':
$fieldsToIndex = array(
'attributes' => array(array('value1', 'FULLTEXT'), array('value2', 'FULLTEXT'), array('event_id', 'INDEX'), array('sharing_group_id', 'INDEX'), array('uuid', 'INDEX')),
'correlations' => array(array('org_id', 'INDEX'), array('event_id', 'INDEX'), array('attribute_id', 'INDEX'), array('sharing_group_id', 'INDEX'), array('1_event_id', 'INDEX'), array('1_attribute_id', 'INDEX'), array('a_sharing_group_id', 'INDEX'), array('org_id', 'INDEX')),
'correlations' => array(array('org_id', 'INDEX'), array('event_id', 'INDEX'), array('attribute_id', 'INDEX'), array('sharing_group_id', 'INDEX'), array('1_event_id', 'INDEX'), array('1_attribute_id', 'INDEX'), array('a_sharing_group_id', 'INDEX'), array('org_id', 'INDEX'), array('value', 'FULLTEXT')),
'events' => array(array('info', 'FULLTEXT'), array('sharing_group_id', 'INDEX'), array('org_id', 'INDEX'), array('orgc_id', 'INDEX'), array('uuid', 'INDEX')),
'event_tags' => array(array('event_id', 'INDEX'), array('tag_id', 'INDEX')),
'organisations' => array(array('uuid', 'INDEX'), array('name', 'FULLTEXT')),

18
app/Model/Attribute.php
View File

@ -1087,7 +1087,6 @@ class Attribute extends AppModel {
public function __afterSaveCorrelation($a, $full = false, $event = false) {
// Don't do any correlation if the type is a non correlating type
if (!in_array($a['type'], $this->nonCorrelatingTypes)) {
$start = microtime(true);
if (!$event) {
$event = $this->Event->find('first', array(
'recursive' => -1,
@ -1363,11 +1362,6 @@ class Attribute extends AppModel {
$tag = ClassRegistry::init('Tag');
$args = $this->dissectArgs($tags);
$tagArray = $tag->fetchEventTagIds($args[0], $args[1]);
if ($id) {
foreach ($eventIds as $k => $v) {
if ($v['Event']['id'] !== $id) unset($eventIds[$k]);
}
}
if (!empty($tagArray[0])) {
foreach ($eventIds as $k => $v) {
if (!in_array($v['Event']['id'], $tagArray[0])) unset($eventIds[$k]);
@ -1379,6 +1373,12 @@ class Attribute extends AppModel {
}
}
}
if ($id) {
foreach ($eventIds as $k => $v) {
if ($v['Event']['id'] !== $id) unset($eventIds[$k]);
}
}
if ($format == 'suricata') App::uses('NidsSuricataExport', 'Export');
else App::uses('NidsSnortExport', 'Export');
@ -1486,8 +1486,6 @@ class Attribute extends AppModel {
array('type' => $v),
),
'fields' => array('Attribute.value'), //array of field names
'order' => array('Attribute.value'), //string or array defining order
'group' => array('Attribute.value'), //fields to GROUP BY
)
);
if ($k == 'hostname') {
@ -1601,7 +1599,7 @@ class Attribute extends AppModel {
}
public function checkTemplateAttributes($template, &$data, $event_id, $distribution) {
public function checkTemplateAttributes($template, &$data, $event_id) {
$result = array();
$errors = array();
$attributes = array();
@ -1629,7 +1627,7 @@ class Attribute extends AppModel {
} else {
foreach ($result['attributes'] as &$a) {
$a['event_id'] = $event_id;
$a['distribution'] = $distribution;
$a['distribution'] = 5;
$test = $this->checkForValidationIssues(array('Attribute' => $a));
if ($test) {
foreach ($test['value'] as $e) {

238
app/Model/Event.php
View File

@ -594,16 +594,20 @@ class Event extends AppModel {
*/
public function cleanupEventArrayFromXML(&$data) {
$objects = array('Attribute', 'ShadowAttribute');
foreach ($objects as $object) {
// Workaround for different structure in XML/array than what CakePHP expects
if (isset($data['Event'][$object]) && is_array($data['Event'][$object]) && count($data['Event'][$object])) {
if (!is_numeric(implode(array_keys($data['Event'][$object]), ''))) {
// single attribute
$data['Event'][$object][0] = $data['Event'][$object];
//$data['Event'][$object] = array(0 => $data['Event'][$object]);
}
}
}
$objects = array('Org', 'Orgc', 'SharingGroup');
foreach ($objects as $object) {
if (isset($data['Event'][$object][0])) $data['Event'][$object] = $data['Event'][$object][0];
}
return $data;
}
@ -954,8 +958,8 @@ class Event extends AppModel {
$request = array(
'header' => array(
'Authorization' => $authkey,
'Accept' => 'application/xml',
'Content-Type' => 'application/xml',
'Accept' => 'application/json',
'Content-Type' => 'application/json',
//'Connection' => 'keep-alive' // LATER followup cakephp ticket 2854 about this problem http://cakephp.lighthouseapp.com/projects/42648-cakephp/tickets/2854
)
);
@ -966,9 +970,7 @@ class Event extends AppModel {
}
$response = $HttpSocket->get($uri, $data = '', $request);
if ($response->isOk()) {
$xmlArray = Xml::toArray(Xml::build($response->body));
$xmlArray = $this->updateXMLArray($xmlArray);
return $xmlArray['response'];
return(json_decode($response->body, true));
} else {
// TODO parse the XML response and keep the reason why it failed
return null;
@ -1132,6 +1134,7 @@ class Event extends AppModel {
array('fields' => array('SharingGroup.id','SharingGroup.name', 'SharingGroup.releasability', 'SharingGroup.description')),
array(
'fields' => array('SharingGroup.*'),
'Organisation' => array('fields' => $fieldsOrg),
'SharingGroupOrg' => array(
'Organisation' => array('fields' => $fieldsOrg),
),
@ -1204,6 +1207,13 @@ class Event extends AppModel {
$attribute['data'] = $encodedFile;
}
}
if (isset($attribute['SharingGroup']['SharingGroupServer'])) {
foreach ($attribute['SharingGroup']['SharingGroupServer'] as &$sgs) {
if ($sgs['server_id'] == 0) {
$sgs['Server'] = array('id' => '0', 'url' => Configure::read('MISP.baseurl'), 'name' => Configure::read('MISP.baseurl'));
}
}
}
$attribute['ShadowAttribute'] = array();
// If a shadowattribute can be linked to an attribute, link it to it then remove it from the event
// This is to differentiate between proposals that were made to an attribute for modification and between proposals for new attributes
@ -1611,6 +1621,7 @@ class Event extends AppModel {
if ($data['Event']['distribution'] == 4) $sgs[$data['Event']['SharingGroup']['uuid']] = $data['Event']['SharingGroup'];
if (isset($data['Event']['Attribute'])) {
foreach ($data['Event']['Attribute'] as &$attribute) {
if (isset($attribute['SharingGroup']) && !empty($attribute['SharingGroup']) && isset($attribute['SharingGroup'][0])) $attribute['SharingGroup'] = $attribute['SharingGroup'][0];
if ($attribute['distribution'] == 4 && !isset($sgs[$attribute['SharingGroup']['uuid']])) $sgs[$attribute['SharingGroup']['uuid']] = $attribute['SharingGroup'];
}
}
@ -1652,6 +1663,13 @@ class Event extends AppModel {
}
$data['Event']['EventTag'] = $eventTags;
}
if (!isset($data['Event']['EventTag'])) $data['Event']['EventTag'] = array();
if (isset($data['Event']['Tag'])) {
if (isset($event['Event']['Tag']['id'])) $event['Event']['Tag'] = array($event['Event']['Tag']);
foreach ($data['Event']['Tag'] as $tag) {
$data['Event']['EventTag'][] = array('tag_id' => $this->EventTag->Tag->captureTag($tag, $user));
}
}
return $data;
}
@ -1769,66 +1787,171 @@ class Event extends AppModel {
}
public function _edit(&$data, $user, $id, $jobId = null) {
if ($jobId) {
App::import('Component','Auth');
}
$this->Log = ClassRegistry::init('Log');
$localEvent = $this->find('first', array('conditions' => array('Event.id' => $id), 'recursive' => -1, 'contain' => array('Attribute', 'ThreatLevel', 'ShadowAttribute')));
if (!isset($data['Event']['orgc_id']) && !isset($data['Event']['orgc'])) $data['Event']['orgc_id'] = $data['Event']['org_id'];
if ($localEvent['Event']['timestamp'] < $data['Event']['timestamp']) {
} else {
return 'Event exists and is the same or newer.';
}
if (!$localEvent['Event']['locked']) {
return 'Event originated on this instance, any changes to it have to be done locally.';
}
$fieldList = array(
'Event' => array('date', 'threat_level_id', 'analysis', 'info', 'published', 'uuid', 'from', 'distribution', 'timestamp'),
'Attribute' => array('event_id', 'category', 'type', 'value', 'value1', 'value2', 'to_ids', 'uuid', 'distribution', 'timestamp', 'comment')
);
$data['Event']['id'] = $localEvent['Event']['id'];
$data = $this->cleanupEventArrayFromXML($data);
$saveResult = $this->save($data, array('fieldList' => $fieldList['Event']));
unset($this->Attribute->validate['event_id']);
unset($this->Attribute->validate['value']['unique']); // otherwise gives bugs because event_id is not set
// reposition to get the event.id with given uuid
$existingEvent = $this->findByUuid($data['Event']['uuid']);
// If the event exists...
$dateObj = new DateTime();
$date = $dateObj->getTimestamp();
if (count($existingEvent)) {
$data['Event']['id'] = $existingEvent['Event']['id'];
// Conditions affecting all:
// user.org == event.org
// edit timestamp newer than existing event timestamp
if (!isset($data['Event']['timestamp'])) $data['Event']['timestamp'] = $date;
if ($data['Event']['timestamp'] > $existingEvent['Event']['timestamp']) {
if ($data['Event']['distribution'] == 4) {
$data['Event']['sharing_group_id'] = $this->SharingGroup->captureSG($data['Event']['SharingGroup'], $user);
if ($data['Event']['sharing_group_id'] === false) return (array('error' => 'Event could not be saved: User not authorised to create the associated sharing group.'));
unset ($data['Event']['SharingGroup']);
}
// If the above is true, we have two more options:
// For users that are of the creating org of the event, always allow the edit
// For users that are sync users, only allow the edit if the event is locked
if ($existingEvent['Event']['orgc_id'] === $user['org_id']
|| ($user['Role']['perm_sync'] && $existingEvent['Event']['locked']) || $user['Role']['perm_site_admin']) {
if ($user['Role']['perm_sync']) {
if ($data['Event']['distribution'] == 4 && !$this->SharingGroup->checkIfAuthorisedExtend($user, $data['Event']['sharing_group_id'])) {
return (array('error' => 'Event could not be saved: The sync user has to either be an extender of the sharing group or be the sync user that has first synchronised the sharing group to this instance.'));
}
}
// Only allow an edit if this is true!
$saveEvent = true;
} else return (array('error' => 'Event could not be saved: The user used to edit the event is not authorised to do so. This can be caused by the user not being of the same organisation as the original creator of the event whilst also not being a site administrator.'));
} else return (array('error' => 'Event could not be saved: Event in the request not newer than the local copy.'));
// If a field is not set in the request, just reuse the old value
$recoverFields = array('analysis', 'threat_level_id', 'info', 'distribution', 'date');
foreach ($recoverFields as $rF) if (!isset($data['Event'][$rF])) $data['Event'][$rF] = $existingEvent['Event'][$rF];
} else return (array('error' => 'Event could not be saved: Could not find the local event.'));
$fieldList = array(
'Event' => array('date', 'threat_level_id', 'analysis', 'info', 'published', 'uuid', 'distribution', 'timestamp', 'sharing_group_id'),
'Attribute' => array('event_id', 'category', 'type', 'value', 'value1', 'value2', 'to_ids', 'uuid', 'revision', 'distribution', 'timestamp', 'comment', 'sharing_group_id'),
'ShadowAttribute' => array('event_id', 'category', 'type', 'value', 'value1', 'value2', 'org_id', 'event_org_id', 'comment', 'event_uuid', 'deleted', 'to_ids', 'uuid')
);
$saveResult = $this->save(array('Event' => $data['Event']), array('fieldList' => $fieldList['Event']));
$this->Log = ClassRegistry::init('Log');
if ($saveResult) {
$validationErrors = array();
if (isset($data['Event']['Attribute'])) {
foreach ($data['Event']['Attribute'] as $k => &$attribute) {
$mode = 'add';
$data['Event']['Attribute'][$k]['event_id'] = $this->id;
$existingAttribute = $this->__searchUuidInAttributeArray($attribute['uuid'], $localEvent);
if (count($existingAttribute)) {
// Check if the attribute's timestamp is bigger than the one that already exists.
// If yes, it means that it's newer, so insert it. If no, it means that it's the same attribute or older - don't insert it, insert the old attribute.
// Alternatively, we could unset this attribute from the request, but that could lead with issues if we decide that we want to start deleting attributes that don't exist in a pushed event.
if ($data['Event']['Attribute'][$k]['timestamp'] > $existingAttribute['Attribute']['timestamp']) {
$data['Event']['Attribute'][$k]['id'] = $existingAttribute['Attribute']['id'];
foreach ($data['Event']['Attribute'] as $k => $attribute) {
if (isset($attribute['uuid'])) {
$existingAttribute = $this->Attribute->findByUuid($attribute['uuid']);
if (count($existingAttribute)) {
if ($existingAttribute['Attribute']['event_id'] != $id) {
$result = $this->Log->save(array(
'org' => $user['Organisation']['name'],
'model' => 'Event',
'model_id' => $id,
'email' => $user['email'],
'action' => 'edit',
'user_id' => $user['id'],
'title' => 'Duplicate UUID found in attribute',
'change' => 'An attribute was blocked from being saved due to a duplicate UUID. The uuid in question is: ' . $attribute['uuid'],
));
unset($data['Event']['Attribute'][$k]);
} else {
// If a field is not set in the request, just reuse the old value
$recoverFields = array('value', 'to_ids', 'distribution', 'category', 'type', 'comment', 'sharing_group_id');
foreach ($recoverFields as $rF) if (!isset($attribute[$rF])) $data['Event']['Attribute'][$c][$rF] = $existingAttribute['Attribute'][$rF];
$data['Event']['Attribute'][$k]['id'] = $existingAttribute['Attribute']['id'];
// Check if the attribute's timestamp is bigger than the one that already exists.
// If yes, it means that it's newer, so insert it. If no, it means that it's the same attribute or older - don't insert it, insert the old attribute.
// Alternatively, we could unset this attribute from the request, but that could lead with issues if we decide that we want to start deleting attributes that don't exist in a pushed event.
if (isset($data['Event']['Attribute'][$k]['timestamp'])) {
if ($data['Event']['Attribute'][$k]['timestamp'] <= $existingAttribute['Attribute']['timestamp']) {
unset($data['Event']['Attribute'][$k]);
continue;
}
} else $data['Event']['timestamp'] = $date;
}
} else {
unset($data['Event']['Attribute'][$k]);
continue;
$this->Attribute->create();
}
} else {
unset($data['Event']['Attribute'][$k]['id']);
$this->Attribute->create();
}
if (!$this->Attribute->save($data['Event']['Attribute'][$k], array('fieldList' => $fieldList['Attribute']))) {
$validationErrors[] = $this->Attribute->validationErrors;
$data['Event']['Attribute'][$k]['event_id'] = $this->id;
if ($data['Event']['Attribute'][$k]['distribution'] == 4) {
$sid = $this->SharingGroup->captureSG($data['Event']['Attribute'][$k]['SharingGroup'], $user);
$data['Event']['Attribute'][$k]['sharing_group_id'] = $this->SharingGroup->captureSG($data['Event']['Attribute'][$k]['SharingGroup'], $user);
}
if (!$this->Attribute->save($data['Event']['Attribute'][$k], array('fieldList' => $fieldList))) {
$validationErrors['Attribute'][$k] = $this->Attribute->validationErrors;
$attribute_short = (isset($data['Event']['Attribute'][$k]['category']) ? $data['Event']['Attribute'][$k]['category'] : 'N/A') . '/' . (isset($data['Event']['Attribute'][$k]['type']) ? $data['Event']['Attribute'][$k]['type'] : 'N/A') . ' ' . (isset($data['Event']['Attribute'][$k]['value']) ? $data['Event']['Attribute'][$k]['value'] : 'N/A');
$this->Log->create();
$this->Log->save(array(
'org' => $user['Organisation']['name'],
'model' => 'Attribute',
'model_id' => 0,
'email' => $user['email'],
'action' => 'validation_error',
'user_id' => $user['id'],
'title' => 'Attribute validation for Event ' . $this->id . ' failed: ' . $attribute_short,
'change' => json_encode($this->Attribute->validationErrors),
'org' => $user['Organisation']['name'],
'model' => 'Attribute',
'model_id' => 0,
'email' => $user['email'],
'action' => 'edit',
'user_id' => $user['id'],
'title' => 'Attribute dropped due to validation for Event ' . $this->id . ' failed: ' . $attribute_short,
'change' => json_encode($this->Attribute->validationErrors),
));
}
}
}
return 'success';
} else return 'Saving the event has failed.';
if (isset($data['Event']['Tag']) && $user['Role']['perm_tagger']) {
foreach ($data['Event']['Tag'] as $tag) {
$tag_id = $this->EventTag->Tag->captureTag($tag, $user);
if ($tag_id) {
$this->EventTag->attachTagToEvent($this->id, $tag_id);
} else {
$this->Log->create();
$this->Log->save(array(
'org' => $user['Organisation']['name'],
'model' => 'Event',
'model_id' => $this->id,
'email' => $user['email'],
'action' => 'edit',
'user_id' => $user['id'],
'title' => 'Failed create or attach Tag ' . $tag['name'] . ' to the event.',
'change' => ''
));
}
}
}
// check if the exact proposal exists, if yes check if the incoming one is deleted or not. If it is deleted, remove the old proposal and replace it with the one marked for being deleted
// otherwise throw the new one away.
if (isset($data['Event']['ShadowAttribute'])) {
foreach ($data['Event']['ShadowAttribute'] as $k => &$proposal) {
$existingProposal = $this->ShadowAttribute->find('first', array(
'recursive' => -1,
'conditions' => array(
'value' => $proposal['value'],
'category' => $proposal['category'],
'to_ids' => $proposal['to_ids'],
'type' => $proposal['type'],
'event_uuid' => $proposal['event_uuid'],
'uuid' => $proposal['uuid']
)
));
if (!empty($existingProposal)) {
if ($existingProposal['ShadowAttribute']['deleted'] == 1) {
$this->ShadowAttribute->delete($existingProposal['ShadowAttribute']['id'], false);
} else {
unset($data['ShadowAttribute'][$k]);
continue;
}
$this->ShadowAttribute->create();
}
$this->ShadowAttribute->save($data['Event']['ShadowAttribute'][$k], array('fieldList' => $fieldList));
}
}
//if published -> do the actual publishing
if ((!empty($data['Event']['published']) && 1 == $data['Event']['published'])) {
// do the necessary actions to publish the event (email, upload,...)
if (true != Configure::read('MISP.disablerestalert')) {
$this->sendAlertEmailRouter($id, $user);
}
$this->publish($existingEvent['Event']['id']);
}
return true;
} return $this->validationErrors;
}
private function __searchUuidInAttributeArray($uuid, &$attr_array) {
@ -2203,10 +2326,21 @@ class Event extends AppModel {
}
public function checkIfNewer($incomingEvent) {
$localEvent = $this->find('first', array('conditions' => array('uuid' => $incomingEvent['uuid']), 'recursive' => -1));
$localEvent = $this->find('first', array('conditions' => array('uuid' => $incomingEvent['uuid']), 'recursive' => -1, 'fields' => array('Event.uuid', 'Event.timestamp')));
if (empty($localEvent) || $incomingEvent['timestamp'] > $localEvent['Event']['timestamp']) return true;
return false;
}
public function removeOlder(&$eventArray) {
$uuidsToCheck = array();
foreach ($eventArray as $k => &$event) {
$uuidsToCheck[$event['uuid']] = $k;
}
$localEvents = $this->find('list', array('recursive' => -1, 'fields' => array('Event.uuid', 'Event.timestamp')));
foreach ($uuidsToCheck as $uuid => $eventArrayId) {
if (isset($localEvents[$uuid]) && $localEvents[$uuid] >= $eventArray[$eventArrayId]['timestamp']) unset($eventArray[$eventArrayId]);
}
}
public function stix($id, $tags, $attachments, $user, $returnType, $from = false, $to = false, $last = false) {
$eventIDs = $this->Attribute->dissectArgs($id);

15
app/Model/EventTag.php
View File

@ -52,4 +52,19 @@ class EventTag extends AppModel {
$eventIDs = array_unique($eventIDs);
return $eventIDs;
}
public function attachTagToEvent($event_id, $tag_id) {
$existingAssociation = $this->find('first', array(
'recursive' => -1,
'conditions' => array(
'tag_id' => $tag_id,
'event_id' => $event_id
)
));
if (empty($existingAssociation)) {
$this->create();
if (!$this->save(array('event_id' => $event_id, 'tag_id' => $tag_id))) return false;
}
return true;
}
}

98
app/Model/Server.php
View File

@ -933,49 +933,12 @@ class Server extends AppModel {
$event['Event']['distribution'] = '0';
break;
}
// correct $event if just one Attribute
if (is_array($event['Event']['Attribute']) && isset($event['Event']['Attribute']['id'])) {
$tmp = $event['Event']['Attribute'];
unset($event['Event']['Attribute']);
$event['Event']['Attribute'][0] = $tmp;
}
if (is_array($event['Event']['Attribute'])) {
$size = count($event['Event']['Attribute']);
if ($size == 0) {
$fails[$eventId] = 'Empty event received.';
continue;
}
for ($i = 0; $i < $size; $i++) {
if (!isset($event['Event']['Attribute'][$i]['distribution'])) { // version 1
$event['Event']['Attribute'][$i]['distribution'] = 1;
}
switch($event['Event']['Attribute'][$i]['distribution']) {
case 1:
case 'This community only': // backwards compatibility
// if community falseonly, downgrade to org only after pull
$event['Event']['Attribute'][$i]['distribution'] = '0';
break;
case 2:
case 'Connected communities': // backwards compatibility
// if connected communities downgrade to community only
$event['Event']['Attribute'][$i]['distribution'] = '1';
break;
case 'All communities': // backwards compatibility
$event['Event']['Attribute'][$i]['distribution'] = '3';
break;
case 'Your organisation only': // backwards compatibility
$event['Event']['Attribute'][$i]['distribution'] = '0';
break;
}
}
$event['Event']['Attribute'] = array_values($event['Event']['Attribute']);
} else {
if (!is_array($event['Event']['Attribute']) || empty($event['Event']['Attribute'])) {
$fails[$eventId] = 'Empty event received.';
continue;
}
$event['Event']['Attribute'] = array_values($event['Event']['Attribute']);
} else {
$fails[$eventId] = 'Empty event received.';
$fails[$eventId] = 'Event blocked by blacklist.';
continue;
}
// Distribution, set reporter of the event, being the admin that initiated the pull
@ -993,9 +956,12 @@ class Server extends AppModel {
}
} else {
$result = $eventModel->_edit($event, $user, $existingEvent['Event']['id'], $jobId);
if ($result === 'success') $successes[] = $eventId;
else $fails[$eventId] = $result;
$tempUser = $user;
$tempUser['Role']['perm_site_admin'] = false;
$result = $eventModel->_edit($event, $tempUser, $existingEvent['Event']['id'], $jobId);
if ($result === true) $successes[] = $eventId;
else if (isset($result['error'])) $fails[$eventId] = $result['error'];
else $fails[$eventId] = json_encode($result);
}
} else {
// error
@ -1103,6 +1069,7 @@ class Server extends AppModel {
* @return array of event_ids
*/
public function getEventIdsFromServer($server, $all = false, $HttpSocket=null, $force_uuid=false, $ignoreFilterRules = false) {
$start = microtime(true);
$url = $server['Server']['url'];
$authkey = $server['Server']['authkey'];
if ($ignoreFilterRules) $filter_rules = array();
@ -1143,19 +1110,20 @@ class Server extends AppModel {
} else {
// multiple events, iterate over the array
$this->Event = ClassRegistry::init('Event');
foreach ($eventArray as &$event) {
foreach ($eventArray as $k => &$event) {
if (1 != $event['published']) {
continue; // do not keep non-published events
}
// get rid of events that are the same timestamp as ours or older, we don't want to transfer the attributes for those
// The event's timestamp also matches the newest attribute timestamp by default
if ($this->Event->checkIfNewer($event)) {
if ($force_uuid) $eventIds[] = $event['uuid'];
else $eventIds[] = $event['id'];
}
unset($eventArray[$k]); // do not keep non-published events
}
}
return $eventIds;
$this->Event->removeOlder($eventArray);
if (!empty($eventArray)) {
foreach ($eventArray as $event) {
if ($force_uuid) $eventIds[] = $event['uuid'];
else $eventIds[] = $event['id'];
}
}
}
return $eventIds;
}
if ($response->code == '403') {
return 403;
@ -1888,6 +1856,7 @@ class Server extends AppModel {
}
public function captureServer($server, $user) {
if (isset($server[0])) $server = $server[0];
if ($server['url'] == Configure::read('MISP.baseurl')) return 0;
$existingServer = $this->find('first', array(
'recursive' => -1,
@ -2297,9 +2266,30 @@ class Server extends AppModel {
}
if (Configure::read('MISP.background_jobs') && $jobId) {
$this->Job->saveField('progress', 40);
$this->Job->saveField('message', 'Rebuilding all correlations (this will take a while...)');
$this->Job->saveField('message', 'Rebuilding all correlations.');
}
//$this->Attribute->generateCorrelation($jobId, 40);
// upgrade correlations. No need to recorrelate, we can be a bit trickier here
// Private = 0 attributes become distribution 1 for both the event and attribute.
// For all intents and purposes, this oversimplification works fine when upgrading from 2.3
// Even though the distribution values stored in the correlation won't be correct, they will provide the exact same realeasability
// Event1 = distribution 0 and Attribute1 distribution 3 would lead to private = 1, so setting distribution = 0 and a_distribution = 0
// will result in the same visibility, etc. Once events / attributes get put into a sharing group this will get recorrelated anyway
// Also by unsetting the org field after the move the changes we ensure that these correlations won't get hit again by the script if we rerun it
// and that we don't accidentally "upgrade" a 2.4 correlation
$this->query('UPDATE `correlations` SET `distribution` = 1, `a_distribution` = 1 WHERE `org` != "" AND `private` = 0');
foreach ($orgMapping as $old => $new) {
$this->query('UPDATE `correlations` SET `org_id` = "' . $new . '", `org` = "" WHERE `org` = "' . $old . '";');
}
if (Configure::read('MISP.background_jobs') && $jobId) {
$this->Job->saveField('progress', 60);
$this->Job->saveField('message', 'Correlations rebuilt. Indexing all tables.');
}
$this->updateDatabase('indexTables');
if (Configure::read('MISP.background_jobs') && $jobId) {
$this->Job->saveField('progress', 100);
$this->Job->saveField('message', 'Upgrade complete.');
}
$this->Attribute->generateCorrelation($jobId, 40);
}

57
app/Model/SharingGroup.php
View File

@ -198,29 +198,27 @@ class SharingGroup extends AppModel {
if ($this->checkIfOwner($user, $id)) return true;
$this->id = $id;
if (!$this->exists()) return false;
$sgo = $this->SharingGroupOrg->find('first', array(
'conditions' => array(
'sharing_group_id' => $id,
'org_id' => $user['org_id'],
'extend' => 1,
),
'recursive' => -1,
'fields' => array('id', 'org_id', 'extend')
));
if (empty($sgo)) {
if ($user['Role']['perm_sync']) {
$sg = $this->find('first', array(
'conditions' => array(
'id' => $id,
'sync_user_id' => $user['id'],
),
'recursive' => -1,
));
if (empty($sg)) return false;
else return true;
}
return false;
if ($user['Role']['perm_sync']) {
$sg = $this->find('first', array(
'conditions' => array(
'id' => $id,
'sync_user_id' => $user['id'],
),
'recursive' => -1,
));
if (empty($sg)) return false;
else return true;
}
$sgo = $this->SharingGroupOrg->find('first', array(
'conditions' => array(
'sharing_group_id' => $id,
'org_id' => $user['org_id'],
'extend' => 1,
),
'recursive' => -1,
'fields' => array('id', 'org_id', 'extend')
));
if (empty($sgo)) return false;
else return true;
}
@ -327,13 +325,21 @@ class SharingGroup extends AppModel {
));
$force = false;
if (empty($existingSG)) {
if (!$user['Role']['perm_sharing_group']) throw new Exception('User not authorised to create sharing groups.');
$this->create();
$newSG = array();
$attributes = array('name', 'releasability', 'description', 'uuid', 'organisation_uuid', 'created', 'modified');
foreach ($attributes as $a) $newSG[$a] = $sg[$a];
$newSG['local'] = 0;
$newSG['sync_user_id'] = $user['id'];
$newSG['org_id'] = $this->Organisation->captureOrg($sg['Organisation'], $user);
if (!isset($sg['Organisation'])) {
foreach ($sg['SharingGroupOrg'] as $k => $org) {
if (isset($org['Organisation'][0])) $org['Organisation'] = $org['Organisation'][0];
if ($org['Organisation']['uuid'] == $sg['uuid']) $newSG['org_id'] = $this->Organisation->captureOrg($org['Organisation'], $user);
}
} else {
$newSG['org_id'] = $this->Organisation->captureOrg($sg['Organisation'], $user);
}
if (!$this->save($newSG)) return false;
$sgids = $this->id;
} else {
@ -353,7 +359,7 @@ class SharingGroup extends AppModel {
if ($force) {
$sgids = $existingSG['SharingGroup']['id'];
$editedSG = $existingSG['SharingGroup'];
$attributes = array('name', 'releasability', 'description', 'uuid', 'organisation_uuid', 'created', 'modified');
$attributes = array('name', 'releasability', 'description', 'created', 'modified');
foreach ($attributes as &$a) $editedSG[$a] = $sg[$a];
$this->save($editedSG);
} else {
@ -363,7 +369,6 @@ class SharingGroup extends AppModel {
return $existingSG['SharingGroup']['id'];
}
}
$sg['org_id'] = $this->Organisation->captureOrg($sg['Organisation'], $user, $force);
unset ($sg['Organisation']);
if (isset($sg['SharingGroupOrg']['id'])) {
@ -372,6 +377,7 @@ class SharingGroup extends AppModel {
$sg['SharingGroupOrg'][0] = $temp;
}
foreach ($sg['SharingGroupOrg'] as $k => $org) {
if (isset($org['Organisation'][0])) $org['Organisation'] = $org['Organisation'][0];
$sg['SharingGroupOrg'][$k]['org_id'] = $this->Organisation->captureOrg($org['Organisation'], $user, $force);
unset ($sg['SharingGroupOrg'][$k]['Organisation']);
if ($force) {
@ -399,6 +405,7 @@ class SharingGroup extends AppModel {
$sg['SharingGroupServer'][0] = $temp;
}
foreach ($sg['SharingGroupServer'] as $k => $server) {
if (isset($server[0])) $server = $server[0];
$sg['SharingGroupServer'][$k]['server_id'] = $this->SharingGroupServer->Server->captureServer($server['Server'], $user, $force);
if ($sg['SharingGroupServer'][$k]['server_id'] === false) unset ($sg['SharingGroupServer'][$k]);
else {

17
app/Model/User.php
View File

@ -548,8 +548,21 @@ class User extends AppModel {
if (!$failed && $canEncrypt) {
$keyImportOutput = $gpg->importKey($user['User']['gpgkey']);
try {
$gpg->addEncryptKey($keyImportOutput['fingerprint']); // use the key that was given in the import
$body = $gpg->encrypt($body, true);
$key = $gpg->getKeys($keyImportOutput['fingerprint']);
$subKeys = $key[0]->getSubKeys();
$canEncrypt = false;
$currentTimestamp = time();
foreach ($subKeys as $subKey) {
$expiration = $subKey->getExpirationDate();
if (($expiration == 0 || $currentTimestamp < $expiration) && $subKey->canEncrypt()) $canEncrypt = true;
}
if ($canEncrypt) {
$gpg->addEncryptKey($keyImportOutput['fingerprint']); // use the key that was given in the import
$body = $gpg->encrypt($body, true);
} else {
$failed = true;
$failureReason = " the message could not be encrypted because the provided key is either expired or cannot be used for encryption.";
}
} catch (Exception $e){
// despite the user having a PGP key and the signing already succeeding earlier, we get an exception. This must mean that there is an issue with the user's key.
$failureReason = " the message could not be encrypted because there was an issue with the user's PGP key. The following error message was returned by gpg: " . $e->getMessage();

5
app/View/Elements/side_menu.ctp
View File

@ -14,6 +14,10 @@
if ($isAclPublish) $mayPublish = true;
}
?>
<?php if ($menuItem === 'freetextResults'): ?>
<li id='lifreetextResults'><a href="#">Freetext Import Results</a></li>
<li class="divider"></li>
<?php endif;?>
<li id='liviewEvent'><a href="<?php echo $baseurl;?>/events/view/<?php echo h($event['Event']['id']);?>">View Event</a></li>
<li id='liviewEventGraph'><a href="<?php echo $baseurl;?>/events/viewGraph/<?php echo h($event['Event']['id']);?>">View Correlation Graph</a></li>
<li id='lieventLog'><a href="<?php echo $baseurl;?>/logs/event_index/<?php echo h($event['Event']['id']);?>">View Event History</a></li>
@ -108,7 +112,6 @@
<li id='liedit'><?php echo $this->Html->link(__('Edit User', true), array('action' => 'edit', $user['User']['id'])); ?></li>
<li class="divider"></li>
<?php endif; ?>
<li id='linews'><a href="<?php echo $baseurl;?>/users/news">News</a></li>
<li id='liview'><a href="<?php echo $baseurl;?>/users/view/me">My Profile</a></li>
<li id='limembers'><a href="<?php echo $baseurl;?>/users/memberslist">Members List</a></li>
<li id='liindexOrg'><a href="<?php echo $baseurl;?>/organisations/index">List Organisations</a></li>

2
app/View/Events/free_text_results.ctp
View File

@ -151,5 +151,5 @@
</script>
<?php
endif;
echo $this->element('side_menu', array('menuList' => 'regexp', 'menuItem' => 'index'));
echo $this->element('side_menu', array('menuList' => 'event', 'menuItem' => 'freetextResults'));
?>

2
app/View/Servers/index.ctp
View File

@ -78,7 +78,7 @@ foreach ($servers as $server):
<?php
echo $this->Html->link('', array('action' => 'previewIndex', $server['Server']['id']), array('class' => 'icon-search', 'title' => 'Explore'));
if ($server['Server']['pull'])
echo $this->Html->link('', array('action' => 'pull', $server['Server']['id'], 'update'), array('class' => 'icon-refresh', 'title' => 'Pull updates only'));
echo $this->Html->link('', array('action' => 'pull', $server['Server']['id'], 'update'), array('class' => 'icon-refresh', 'title' => 'Pull updates to events that already exist locally'));
echo $this->Html->link('', array('action' => 'pull', $server['Server']['id'], 'full'), array('class' => 'icon-download', 'title' => 'Pull all'));
if ($server['Server']['push'])
echo $this->Html->link('', array('action' => 'push', $server['Server']['id'], 'full'), array('class' => 'icon-upload', 'title' => 'Push all'));