CSV export changes

- It is now possible to restrict the CSV automation export by type / category

- updated the automation page to describe how the syntax works

- fixed an issue with line breaks not being sanitized for the CSV export

app/Controller/EventsController.php

@@ -1334,10 +1334,12 @@ class EventsController extends AppController {
 	// csv function
 	// Usage: csv($key, $eventid)   - key can be a valid auth key or the string 'download'. Download requires the user to be logged in interactively and will generate a .csv file
 	// $eventid can be one of 3 options: left empty it will get all the visible to_ids attributes,
-	public function csv($key, $eventid=0, $ignore=0, $tags = '') {
+	public function csv($key, $eventid=0, $ignore=0, $tags = '', $category=null, $type=null) {
+		if ($category == 'null') $category = null;
+		if ($type == 'null') $type = null;
+		if ($tags == 'null') $tags = '';
 		if ($tags != '') $tags = str_replace(';', ':', $tags);
 		$list = array();
-		
 		if ($key != 'download') {
 			// check if the key is valid -> search for users based on key
 			$user = $this->checkAuthUser($key);
@@ -1353,14 +1355,13 @@ class EventsController extends AppController {
 			$isSiteAdmin = $this->_isSiteAdmin();
 			$org = $this->Auth->user('org');
 		}		
-		
 		// if it's a search, grab the attributeIDList from the session and get the IDs from it. Use those as the condition
 		// We don't need to look out for permissions since that's filtered by the search itself
 		// We just want all the attributes found by the search
 		if ($eventid === 'search') {
 			$list = $this->Session->read('search_find_attributeidlist');
 		}
-		$attributes = $this->Event->csv($org, $isSiteAdmin, $eventid, $ignore, $list, $tags);
+		$attributes = $this->Event->csv($org, $isSiteAdmin, $eventid, $ignore, $list, $tags, $category, $type);
 		$this->loadModel('Whitelist');
 		$final = array();
 		$attributes = $this->Whitelist->removeWhitelistedFromArray($attributes, true);

app/Model/Event.php

@@ -941,9 +941,8 @@ class Event extends AppModel {
 		}
 		return $results;
 	}
-	public function csv($org, $isSiteAdmin, $eventid=0, $ignore=0, $attributeIDList = array(), $tags = '') {
+	public function csv($org, $isSiteAdmin, $eventid=0, $ignore=0, $attributeIDList = array(), $tags = '', $category = null, $type = null) {
 		$final = array();
-	 	
 		$attributeList = array();
 		$conditions = array();
 	 	$econditions = array();
@@ -997,6 +996,15 @@ class Event extends AppModel {
 	 		if ($ignore == 0) {
 	 			$conditions['AND'][] = array('Attribute.to_ids =' => 1);
 	 		}
+	 		
+	 		if ($type!=null) {
+	 			$conditions['AND'][] = array('Attribute.type' => $type);
+	 		}
+	 		
+	 		if ($category!=null) {
+	 			$conditions['AND'][] = array('Attribute.category' => $category);
+	 		}
+	 		
 	 		if (!$isSiteAdmin) {
 	 			$temp = array();
 	 			$distribution = array();
@@ -1015,9 +1023,8 @@ class Event extends AppModel {
 	 			'fields' => array('Attribute.event_id', 'Attribute.distribution', 'Attribute.category', 'Attribute.type', 'Attribute.value', 'Attribute.uuid'),
 	 	);
 	 	$attributes = $this->Attribute->find('all', $params);
-	 	foreach ($attributes as $attribute) {
-	 		$attribute['Attribute']['value'] = str_replace("\r", "", $attribute['Attribute']['value']);
-	 		$attribute['Attribute']['value'] = str_replace("\n", "", $attribute['Attribute']['value']);
+	 	foreach ($attributes as &$attribute) {
+	 		$attribute['Attribute']['value'] = str_replace(array("\r\n", "\n", "\r"), "", $attribute['Attribute']['value']);
 	 	}
 	 	return $attributes;
 	 }

app/View/Events/automation.ctp

@@ -18,9 +18,18 @@ You can <?php echo $this->Html->link('reset', array('controller' => 'users', 'ac
 <pre><?php echo Configure::read('MISP.baseurl');?>/events/xml/download/1</pre>
 <p>The xml download also accepts two additional (optional) parameters: a boolean field that determines whether attachments should be encoded and a second parameter that controls the eligible tags. To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, to include tag1 and tag2 but exclude tag3 you would use:</p>
 <pre><?php echo Configure::read('MISP.baseurl');?>/events/xml/download/null/true/tag1&&tag2&&!tag3</pre>
-<p>Also check out the <?php echo $this->Html->link(__('User Guide', true), array('controller' => 'pages', 'action' => 'display', 'using_the_system', '#' => 'rest')); ?> to read about the REST API.</p>
+<p>Also check out the <a href="/pages/display/doc/using_the_system#rest">User Guide</a> to read about the REST API.</p>
 <p></p>
-
+<h3>CSV Export</h3>
+<p>An automatic export of attributes is available as CSV. Only attributes that are flagged "to_ids" will get exported.</p>
+<p>You can configure your tools to automatically download the following file:</p>
+<pre><?php echo Configure::read('MISP.baseurl');?>/events/csv/download/</pre>
+<p>You can specify additional flags for CSV exports as follows::</p>
+<pre><?php echo Configure::read('MISP.baseurl');?>/events/csv/download/[event_id]/[event_id_ignore]/[tags]/[type]</pre>
+<p>For example, to only download a csv generated of the "domain" type and the "Network Activity" category attributes all events except for the one with ID 7 and further restricting it to events that are tagged "tag1" or "tag2" but not "tag3", use the following syntax:</p>
+<pre><?php echo Configure::read('MISP.baseurl');?>/events/csv/download/0/7/tag1&&tag2&&!tag3/Network%20Activity/domain</pre>
+<p>To export the attributes of all events that are of the type "domain", use the following syntax:</p>
+<pre><?php echo Configure::read('MISP.baseurl');?>/events/csv/download/0/0/null/null/domain</pre>
 <h3>NIDS rules export</h3>
 <p>Automatic export of all network related attributes is available under the Snort rule format. Only <em>published</em> events and attributes marked as <em>IDS Signature</em> are exported.</p>
 <p>You can configure your tools to automatically download the following file:</p>