Also, reworked a lot of remaining distribution checks not handled by the main fetch methods
Conflicts:
VERSION.json
app/Controller/AttributesController.php
app/Controller/ShadowAttributesController.php
app/View/ShadowAttributes/add.ctp
app/View/ShadowAttributes/edit.ctp
- vulnerability reported by Airbus Group CERT
- Deprecated ajax attribute view had inverse access control logic
- removed ajax path
- added XML/JSON view
- Users can now propose a deletion to an attribute
- also tied into the mass accept mechanism
- new UI elements to go along with this
- Code refactoring for category list retrievals
- Until now, several methods got the list of categories from the validation code
- Was awkward with a fake empty element that had to be removed
- altered the validation code to read the categoryDefinitions array instead
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- fixed an issue where pushing a single event would fail
- both event and attribute edits via the API work without providing a timestamp. The current timestamp is instead attached
- both event and attribute edits fill the required fields from the data in the database if not supplied (as long as the uuid is found)
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- eventid a new parameter for both event and attribute restsearch
- these APIs now accept arrays in both json and xml format (you can send "eventid": ["15", "16"] instead of "eventid": "15&&16" in addition to the old functionality
- enter a UUID in the event ID field of the attribute search to find attributes belonging to a certain event
- use event IDs / UUIDs to filter events on the event index
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- as explained on the automation page
- also, better error handling
- all API calls that fail during authentication will now return a JSON/XML error message instead of redirecting to the login page
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- added the new flag "last" to the list of parameters
- exports affected: XML, CSV, NIDS, HIDS, STIX, Text, RestSearch
- Valid values: number + format where format can be d, m, h for day, minute, hour (examples: 5d or 12h or 30m)
- documented in automation view
- right now it follows the simple rule of user > admin settings > default values when generating the export
- Parameters can be passed via url / JSON object / XML object
- filters include filter on event ID, date range, tags
TODO:
- buttons for a per event download via the UI
- introduce new export option for normal users (via background workers and the old style export)
- server settings and validation work
- configurable template via settings
- configurable via API as well
- Also trying to define the structure for future Plugin settings
- The idea is to have them in a separate tab all prepended with the plugin name
- since this is not yet part of the future flexible plugin system, it is still kept in the main codebase, but the idea is to get the naming conventions ready for the future version
- removed some junk
- more work on the background workers
- rewrote the correlation background job - should work correctly now and be a lot more memory efficient
- Unified the way exports accept negated parameters
- Fixed the documentation
- Most exports are now restrictable by the event date (From/To parameters)
- none cached XML export now writes to file after converting each event, clearing the memory and resolving any potential memory issues
- Deleting attributes on connected MISP instances can cause serious performance issues on multiple interconnected instnaces, temporarily removed
- Version number incremented
iglocska