- Fixed a critical bug in the XML export
- As of recently XML exports include relations as they were missing before
- the sanitisation of the event info field in related attributes was incorrectly sanitized of unicode characters
- this can lead to the XML export breaking and also for affected events to be blocked from synchronisation
- Proposal fixes
- fixed an invalid uuid generation that lead to an exception
- fixed the attachments for proposals still using the old attachment system that disallows most filenames
- added the automatic creation of hashes for attachment proposals
- As RichieB2B noted, get_current_user() gets the owner of the script in CentOS / RHEL not the user executing the script (as in Ubuntu)
- Current solution uses posix_getpwuid and posix_geteuid if the php-posix package is installed
- if not, it uses whoami
- for some users the workers appeared to be dead even though the worker processes were functional and started by the correct user
- this was due to access to /proc being blocked by open_basedir directive settings
- added a check and the corresponding view changes to this being the case
- Under these distros, php is blocked from seeing concurrently running php processes even under the same user
- instead of running ps, the diagnostic now checks the existance of the pid file in /proc/
- diagnostic tool would throw exceptions because the db session tables are still missing in some older instances
- if a different session handler is used, the test is skipped
- fixed an issue where pushing a single event would fail
- both event and attribute edits via the API work without providing a timestamp. The current timestamp is instead attached
- both event and attribute edits fill the required fields from the data in the database if not supplied (as long as the uuid is found)
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- indexes were not created if they already existed
- this was an issue if a non unique index was present
- also made the process more verbose and added a generic method that deals with index removal
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- Threat level ID options correctly set
- Threat level ID validation tightened to reject anything but the existing threat levels
- The upload malware API now logs validation issues during the failed creation of attributes / events
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- Instead of fetching all events at once for the export, events are fetched one by one
- Greatly reduces memory footprint (It mostly depends on the event with the most eligible attributes now, instead of the combined list of all events)
- Because of the lower memory usage, the time taken for the export is also slashed to a fragment of what it was before
- fixed some issues with unset variables (from, to, last) when triggered by the background workers
- reduced memory usage of the hids exports (removed storing the hashes twice in memory, drastically removed the data retrieved from the db when preparing the export)
- some errors in the format (wrong comment character used, rpz-ip not appended to IP addresses, missing semi-colon)
- removed hostnames that are on domains blocked by the rules based on domain attributes
- python server running in the background doing the publishing
- MISP -> python script communication via redis
- configurable / controllable via the admin UI
- by installing the requirements described in the update and the install instructions (ubuntu only for now, centos/red-hat versions to be tested and described), administrators can enable the pub/sub feature
- assign a port to the service via the interface
- each time an event is published, MISP will use ZMQ's PUB feature to push out a MISP JSON package using the "misp_json" prefix
iglocska