- tied into auto upgrade system
- tied into server settings
- some cleanup of overly verbose debug
- Enforcing enable/disable everywhere
- Changed temporary file structure
- New permission flag: perm_tag_editor
- taggers can tag events with existing tags
- tag editors can create / edit / delete tags
- Fixed several misleading UI elements for tagging
- tagging users that don't own an event and aren't creators thereof cannot tag them
- this was enforced before but the UI elements were present and threw errors
- Migration is automatic
- all existing tagger roles will automatically become tag editors
- restricting current roles takes manual admin action, but the functionality should remain unchanged for those that just update
- The password change forced on users by administrators couldn't save new passwords
- instead it reset the password to a new random password
- Resetting the password of such users via the admin interface should fix the issue
- Alternatively manually setting the password also fixes it
- tied into automatic datamodel updates
- correlation is one way only (from proposal to attribute)
- proposals don't correlate with one another
- all distribution rules are adhered to
- further improvements on the upgrade mechanism pipeline
- not authenticated users now automatically get redirected to the login page, no matter what action they requested
- This as a nice side effect also removed the bug that was caused by a site admin looking at an admin function before logging out / timing out and being incorrectly redirected to /admin/users/login
- added indeces to the MYSQL.sql file
- contributors now looks for shadow attributes instead of log entries (should make the event view much faster and resolve some timeout issues on sync when the log is massive)
- Running a stix export for a specific ID that doesn't exist results in a full STIX export for the user (events visible to the user)
- This leads for an unnecesarily long export process when a quick export is expected
- New generic fetch attribute method was mistakenly using the order field as a condition, resulting in some exports only displaying a subset of the data
- the fix to this fixes the issue described in #790 for text exports
- Fix to the RPZ exports not working correctly
- Fix to the horrible performance of RPZ exports
- Fix to several background worker issues with exports
- Removed the OpenIOC Indicator UUID persistence and moved it to a comment
- this allows for the same OpenIOC report to be imported into separate events and won't result in a UUID collision
- Reworked the composite indicator resolver
- more generic, allows for 3 part composites (to allow for regkeypath/regkey/regvalue combinations)
- Registry values now correctly recognised
- Changing the auth key now creates a log entry that inclues the user's ID, e-mail address old and new autkeys
- Also removed the logging of the hashed password for newly created users
- also added comment field for attributes
- until now multi line fields were both escaped and the line breaks removed
- this was overkill, linebreaks are now kept intact
- Added logging of failed login attempts
- Added (optional) logging of successful authentications
- admin setting that has to be enabled
- will log all API calls (both HTTP method and target url)
- optional logging of user IP address for all logs
- each log entry created while this setting is enabled will log the IP address of the client
- disabling it also hides the IPs from the interface
- added new IP field for the log search (only if enabled)
Also, reworked a lot of remaining distribution checks not handled by the main fetch methods
Conflicts:
VERSION.json
app/Controller/AttributesController.php
app/Controller/ShadowAttributesController.php
app/View/ShadowAttributes/add.ctp
app/View/ShadowAttributes/edit.ctp
- APIs for the following actions:
- Add new proposed attribute to an event
- Add proposed change to an attribute
- View a proposal
- Accept a proposal
- Discard a proposal
- new APIs described on the automation page
- As RichieB2B noted, get_current_user() gets the owner of the script in CentOS / RHEL not the user executing the script (as in Ubuntu)
- Current solution uses posix_getpwuid and posix_geteuid if the php-posix package is installed
- if not, it uses whoami
- for some users the workers appeared to be dead even though the worker processes were functional and started by the correct user
- this was due to access to /proc being blocked by open_basedir directive settings
- added a check and the corresponding view changes to this being the case
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- Under these distros, php is blocked from seeing concurrently running php processes even under the same user
- instead of running ps, the diagnostic now checks the existance of the pid file in /proc/
- Event blacklist functionality extended
- Several context fields added
- edit existing entries to change the context fields
- removed the deprecated news page
- hash attribute types get validated against empty values
- fixed an excepion on REST add of attributes when the validation stops an attribute from being entered
- fixed the parameters in some exports being ignored after a recent patch
- added an admin tool to prune orphaned attributes
- cleanup and move of the database update methods - they are now accessible from any model
- Footer now shows MISP version including sub version
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- indexes were not created if they already existed
- this was an issue if a non unique index was present
- also made the process more verbose and added a generic method that deals with index removal
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- OpenIOC import now correctly sets IDS flags based on type
- OpenIOC import specifies the source file in the comments
- Fixed a blackhole issue with the password reset popups
- eventid a new parameter for both event and attribute restsearch
- these APIs now accept arrays in both json and xml format (you can send "eventid": ["15", "16"] instead of "eventid": "15&&16" in addition to the old functionality
- added support for SHA types
- fixed an issue that caused the import to fail with duplicate attributes (the list gets pruned now)
- fixed an issue where no supplied contextual fields would lead to empty attributes being created
- removed the requirement for the files to have the .ioc extension
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- as explained on the automation page
- also, better error handling
- all API calls that fail during authentication will now return a JSON/XML error message instead of redirecting to the login page
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- Threat level ID options correctly set
- Threat level ID validation tightened to reject anything but the existing threat levels
- The upload malware API now logs validation issues during the failed creation of attributes / events
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- Instead of fetching all events at once for the export, events are fetched one by one
- Greatly reduces memory footprint (It mostly depends on the event with the most eligible attributes now, instead of the combined list of all events)
- Because of the lower memory usage, the time taken for the export is also slashed to a fragment of what it was before
- fixed some issues with unset variables (from, to, last) when triggered by the background workers
- reduced memory usage of the hids exports (removed storing the hashes twice in memory, drastically removed the data retrieved from the db when preparing the export)
- some errors in the format (wrong comment character used, rpz-ip not appended to IP addresses, missing semi-colon)
- removed hostnames that are on domains blocked by the rules based on domain attributes
- python server running in the background doing the publishing
- MISP -> python script communication via redis
- configurable / controllable via the admin UI
- by installing the requirements described in the update and the install instructions (ubuntu only for now, centos/red-hat versions to be tested and described), administrators can enable the pub/sub feature
- assign a port to the service via the interface
- each time an event is published, MISP will use ZMQ's PUB feature to push out a MISP JSON package using the "misp_json" prefix
- contact reporter first tries to contact orgc users on the instance, if they don't exist, it will contact the owner (instead of going straight to the owner)
- hostname / domain name validation change broke validation of hostnames/domain names / email addresses with a "-"
- Some documentation changes for the REST API (more coming)
- some tuning of the freetext import
- added the new flag "last" to the list of parameters
- exports affected: XML, CSV, NIDS, HIDS, STIX, Text, RestSearch
- Valid values: number + format where format can be d, m, h for day, minute, hour (examples: 5d or 12h or 30m)
- Popover_form purged after the form has been submitted
- a duplicate hidden div was created for confirmation popups within the attribute creation popup and clicking publish populated the wrong div
- comma separated values now correctly parsed
- Ports in IP/url/link/domain/hostname now added as a comment
- virustotal now automatically recognised as external analysis / link
- Events published / pushed will now refuse to sync if the situation arises where no attributes would be eligible to be synced
- Events pulled that contain no attributes will be thrown away
- Reworking the way e-mails are sent - all of it goes through a centralised e-mail method
- just pass the recipient, recipient encryption key collection, body, alternate body if the message cannot be encrypted, subject, reply to address and pgp key for reply to along and the method will do the rest
- encrypt if possible, check if sending without encryption is allowed, signing, adding attachment for reply to encryption key, using alternate sanitised body if it is enforced for accounts that cannot use encryption is all done in one place
- easy to maintain and expand with future changes (such as the S/MIME pull request on github)
- Users being logged on would not be able to use the actions that are also used for automation
- Those actions trigger a check of the authorization header, which in certain use cases can be set with values that is outside of the scope of MISP
- MISP will now try to only detect MISP auth keys in the headers and if it detects something else it ignores it
Iglocska