- download event as JSON now has the option to include attachments
- switched to using the restsearch api instead of the deprecated /events/xml API
- added attachment inclusion to both restsearch apis
- fixed some bugs with the API
- New generic fetch attribute method was mistakenly using the order field as a condition, resulting in some exports only displaying a subset of the data
- the fix to this fixes the issue described in #790 for text exports
- Fix to the RPZ exports not working correctly
- Fix to the horrible performance of RPZ exports
- Fix to several background worker issues with exports
- defanged URL type attributes are refanged on input
- admin script to do the same for all existing attributes
- admin tool doesn't recognise a word followed by a . as a url
- called Add MISP export now
- can be an XML / JSON file
- result browser with explanations of failures
- REST XML/JSON add/edit of events returns errors instead of the partially succeeding event
- as discovered and reported by Egidio Romano of Minded Security
- Lacking checks of HTTP methods in some functionality could lead to a site admin uploading and executing malicious scripts
- Tightened HTTP method verification across the board for actions that modify data
- Turned some administrative tasks to POST only actions
Also, reworked a lot of remaining distribution checks not handled by the main fetch methods
Conflicts:
VERSION.json
app/Controller/AttributesController.php
app/Controller/ShadowAttributesController.php
app/View/ShadowAttributes/add.ctp
app/View/ShadowAttributes/edit.ctp
- vulnerability reported by Airbus Group CERT
- Deprecated ajax attribute view had inverse access control logic
- removed ajax path
- added XML/JSON view
- Users can now propose a deletion to an attribute
- also tied into the mass accept mechanism
- new UI elements to go along with this
- Code refactoring for category list retrievals
- Until now, several methods got the list of categories from the validation code
- Was awkward with a fake empty element that had to be removed
- altered the validation code to read the categoryDefinitions array instead
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- fixed an issue where pushing a single event would fail
- both event and attribute edits via the API work without providing a timestamp. The current timestamp is instead attached
- both event and attribute edits fill the required fields from the data in the database if not supplied (as long as the uuid is found)
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- eventid a new parameter for both event and attribute restsearch
- these APIs now accept arrays in both json and xml format (you can send "eventid": ["15", "16"] instead of "eventid": "15&&16" in addition to the old functionality
- enter a UUID in the event ID field of the attribute search to find attributes belonging to a certain event
- use event IDs / UUIDs to filter events on the event index
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- as explained on the automation page
- also, better error handling
- all API calls that fail during authentication will now return a JSON/XML error message instead of redirecting to the login page
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- added the new flag "last" to the list of parameters
- exports affected: XML, CSV, NIDS, HIDS, STIX, Text, RestSearch
- Valid values: number + format where format can be d, m, h for day, minute, hour (examples: 5d or 12h or 30m)
- documented in automation view
- right now it follows the simple rule of user > admin settings > default values when generating the export
- Parameters can be passed via url / JSON object / XML object
- filters include filter on event ID, date range, tags
TODO:
- buttons for a per event download via the UI
- introduce new export option for normal users (via background workers and the old style export)
William Robinet