- also added a new setting to set the default posture when an event containing a tag is pushed (via the API/sync/etc)
- new setting allows to automatically set new tags to hidden
- the hidden setting only hides the tags from the tag selection when tagging an event
- tied into auto upgrade system
- tied into server settings
- some cleanup of overly verbose debug
- Enforcing enable/disable everywhere
- Changed temporary file structure
- New permission flag: perm_tag_editor
- taggers can tag events with existing tags
- tag editors can create / edit / delete tags
- Fixed several misleading UI elements for tagging
- tagging users that don't own an event and aren't creators thereof cannot tag them
- this was enforced before but the UI elements were present and threw errors
- Migration is automatic
- all existing tagger roles will automatically become tag editors
- restricting current roles takes manual admin action, but the functionality should remain unchanged for those that just update
- The password change forced on users by administrators couldn't save new passwords
- instead it reset the password to a new random password
- Resetting the password of such users via the admin interface should fix the issue
- Alternatively manually setting the password also fixes it
- tied into automatic datamodel updates
- correlation is one way only (from proposal to attribute)
- proposals don't correlate with one another
- all distribution rules are adhered to
- further improvements on the upgrade mechanism pipeline
- not authenticated users now automatically get redirected to the login page, no matter what action they requested
- This as a nice side effect also removed the bug that was caused by a site admin looking at an admin function before logging out / timing out and being incorrectly redirected to /admin/users/login
- added indeces to the MYSQL.sql file
- contributors now looks for shadow attributes instead of log entries (should make the event view much faster and resolve some timeout issues on sync when the log is massive)
- Running a stix export for a specific ID that doesn't exist results in a full STIX export for the user (events visible to the user)
- This leads for an unnecesarily long export process when a quick export is expected
- New generic fetch attribute method was mistakenly using the order field as a condition, resulting in some exports only displaying a subset of the data
- the fix to this fixes the issue described in #790 for text exports
- Fix to the RPZ exports not working correctly
- Fix to the horrible performance of RPZ exports
- Fix to several background worker issues with exports
- Removed the OpenIOC Indicator UUID persistence and moved it to a comment
- this allows for the same OpenIOC report to be imported into separate events and won't result in a UUID collision
- Reworked the composite indicator resolver
- more generic, allows for 3 part composites (to allow for regkeypath/regkey/regvalue combinations)
- Registry values now correctly recognised
- Changing the auth key now creates a log entry that inclues the user's ID, e-mail address old and new autkeys
- Also removed the logging of the hashed password for newly created users
- also added comment field for attributes
- until now multi line fields were both escaped and the line breaks removed
- this was overkill, linebreaks are now kept intact
- Added logging of failed login attempts
- Added (optional) logging of successful authentications
- admin setting that has to be enabled
- will log all API calls (both HTTP method and target url)
- optional logging of user IP address for all logs
- each log entry created while this setting is enabled will log the IP address of the client
- disabling it also hides the IPs from the interface
- added new IP field for the log search (only if enabled)
Also, reworked a lot of remaining distribution checks not handled by the main fetch methods
Conflicts:
VERSION.json
app/Controller/AttributesController.php
app/Controller/ShadowAttributesController.php
app/View/ShadowAttributes/add.ctp
app/View/ShadowAttributes/edit.ctp
- APIs for the following actions:
- Add new proposed attribute to an event
- Add proposed change to an attribute
- View a proposal
- Accept a proposal
- Discard a proposal
- new APIs described on the automation page
- As RichieB2B noted, get_current_user() gets the owner of the script in CentOS / RHEL not the user executing the script (as in Ubuntu)
- Current solution uses posix_getpwuid and posix_geteuid if the php-posix package is installed
- if not, it uses whoami
- for some users the workers appeared to be dead even though the worker processes were functional and started by the correct user
- this was due to access to /proc being blocked by open_basedir directive settings
- added a check and the corresponding view changes to this being the case
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- Under these distros, php is blocked from seeing concurrently running php processes even under the same user
- instead of running ps, the diagnostic now checks the existance of the pid file in /proc/
- Event blacklist functionality extended
- Several context fields added
- edit existing entries to change the context fields
- removed the deprecated news page
- hash attribute types get validated against empty values
- fixed an excepion on REST add of attributes when the validation stops an attribute from being entered
- fixed the parameters in some exports being ignored after a recent patch
- added an admin tool to prune orphaned attributes
- cleanup and move of the database update methods - they are now accessible from any model
- Footer now shows MISP version including sub version
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- indexes were not created if they already existed
- this was an issue if a non unique index was present
- also made the process more verbose and added a generic method that deals with index removal
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- OpenIOC import now correctly sets IDS flags based on type
- OpenIOC import specifies the source file in the comments
- Fixed a blackhole issue with the password reset popups
- eventid a new parameter for both event and attribute restsearch
- these APIs now accept arrays in both json and xml format (you can send "eventid": ["15", "16"] instead of "eventid": "15&&16" in addition to the old functionality
- added support for SHA types
- fixed an issue that caused the import to fail with duplicate attributes (the list gets pruned now)
- fixed an issue where no supplied contextual fields would lead to empty attributes being created
- removed the requirement for the files to have the .ioc extension
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- as explained on the automation page
- also, better error handling
- all API calls that fail during authentication will now return a JSON/XML error message instead of redirecting to the login page
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- Threat level ID options correctly set
- Threat level ID validation tightened to reject anything but the existing threat levels
- The upload malware API now logs validation issues during the failed creation of attributes / events
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible