- just pass /sql:1 to any query via the API to see a dump of all queries
- Response isn't very clean, JSON pushed infront of whatever the output is
- requires debug mode = 2
- authenticate user via URL params if not already authenticated (to support legacy APIs)
- harvest parameters in a standardised way for filtering all export APIs
- for legacy tools that cannot pass headers in HTTP requests for some insane reason
- Needs to be enabled by a site admin - default is that it is disabled
- MISP's diagnostic tool WILL complain if this is ever enabled
- blanket disabling the security component for API requests clashes with explicit disabling of certain security component features in the objects controller causing exceptions
- Organisation merge is now offered to the user by the edit page if a UUID was used to edit an organisation that is already in use
- Merging a local org with 1+ user(s) into an external organisation converts the target organisation into a local one
- Merging a local organisation with a logo into an organisation without one will move the current logo to over
- caveat: this will only happen for organisations already using the new logo naming ([id].png as opposed to [name].png)
- use the feed uuid caches to link directly to affected MISP events
- various UI improvements
- Feed preview pagination / POSTed event ID filters added
- Fixed a reflected XSS in the sharing group editor that requires malicious organisation names
- Low impact due to the following requirements:
- organisation names with malicious org names (JS in the orgname)
- sharing group editor user has to manually add an organisation to the list that has javascript in the org name
- only vulnerable view is the editor itself, so the impact is limited to
users that manually add organisations with malicious names to the list themselves / edit such sharing groups
- As reported by Dawid Czarnecki
When server setting MISP.terms_download=true and MISP.terms_file exists under MISP/app/files/terms directory user wasn't able to download terms and conditions before accepting it.
- Added method to upgrade all passwords to blowfish transparently
- All profile edit pages (/users/edit, /admin/users/edit, /users/change_pw) now require the user's password to be confirmed
- Thanks to cert.govt.nz for the security report.
- simply use the multi select on the event index via the UI
- for the API, simply POST to /events/delete with a payload in the following format:
`{"id": [15, 16, 17]}`
- if you've accidentally deleted all your events using this functionality, feel free to contact @rommelfs or contact the NSA for backups
- Popups get scrollbars / realligned for potato resolutions
- General cleanup of popup related functions in the JS
- Added version querystring to the css files, no more ctrl+f5ing after some updates
- right now it's pretty dumb, it simply pulls the same branch that the current user is on
- Any failure is shown but not acted upon, if the git pull fails the user will see it but it needs to be resolved via the command line
- Fixed file header formatting for the export to Bro format (tabs used consistently).
- Computing the time needed for generating the export to Bro format when done using a background job.
- When generating the Bro export from the UI all the attributes are generated in one single text file similar to the CSV export instead of a zip file with different files inside.
- Changed the file extension of Bro export files from ".intel" to ".txt".
- Removed the allowNonIDS option from the Bro export as it doesn’t make sense to have it (Bro is an IDS).
- Fixed some of the API endpoints which were not accepted (ACL issues).
- Added support for a list of events that should be / should not be included in the export.
- Added a new "meta.desc" column (added in Bro 2.5, see https://www.bro.org/sphinx/frameworks/intel.html) containing the description of the event and of the attribute.
- Sanitized the exported data for Bro.
- Fixed a number of value substitutions which were imported from Snort/Suricata and which were not working for Bro. Did instead substitutions needed for Bro.
- Moved the bruteforce protection directly to the login action
- Fixed the datetime format used by the protection
- Cleaned up the logging of failed attempts
- Only allow enabling internal mode if the host organisation is set and it is chosen as the remote organisation when adding the server sync
- This ensures that internal sync only happens when the same organisation owns both instances
- removed incorrect, useless boiler plate comments
- kept useful comments intact
- added some missing line breaks to make the codebase a bit more uniform
- removed some obviously obsolete TODO comments
- tied into auto upgrade system
- tied into server settings
- some cleanup of overly verbose debug
- Enforcing enable/disable everywhere
- Changed temporary file structure
- New permission flag: perm_tag_editor
- taggers can tag events with existing tags
- tag editors can create / edit / delete tags
- Fixed several misleading UI elements for tagging
- tagging users that don't own an event and aren't creators thereof cannot tag them
- this was enforced before but the UI elements were present and threw errors
- Migration is automatic
- all existing tagger roles will automatically become tag editors
- restricting current roles takes manual admin action, but the functionality should remain unchanged for those that just update
- new syntax
- old syntax still accepted
- new tool for rearranging request data to allow the APIs to automatically catch and correct typical rearrange errors
- sync users that have not accepted the terms / have had a password reset initiated were redirected to the login page
- fixes to the issue
- if a user with automation/sync access uses the API and gets blocked because the terms weren't accepted or there is a pending password change they will be notified in a JSON/XML response
- the sync test now takes this into consideration starting with this version and will report the cause of the failure
- Both instances have to be 2.4.24+ for this to be reported correctly
- needs further fixes, sadly the version has always just showed the major and minor version in the exports
- This masked an issue that would block the import of events that are even a hotfix away
- As a temporary fix, I reverted the changes and the XML version field will now only show the major and minor version to restore compatibility (so 2.4.0 instead of 2.4.19)
- not authenticated users now automatically get redirected to the login page, no matter what action they requested
- This as a nice side effect also removed the bug that was caused by a site admin looking at an admin function before logging out / timing out and being incorrectly redirected to /admin/users/login
- users can now be disabled by an admin
- disabled users cannot login (via the UI or the API) and will be informed
- login attempts by disabled users are logged
- also added the expiration field for later use
- some json actions worked by passing the .json extension in the url
- these pages were correctly returning JSONs but were often internally running through the HTML code-path thanks to an invalid detection
- the new correct detection should provide a major speed boost for certain json requests
- It seems like relying on the Accept header can lead to the data type detection failing when accessing .json extension views
- this issue seems to have gone unnoticed since until now the data passed to the json view was the same as that passed to the html view
- this means that all the additional UI only features may have triggered in the background previously on .json views
- as discovered and reported by Egidio Romano of Minded Security
- Lacking checks of HTTP methods in some functionality could lead to a site admin uploading and executing malicious scripts
- Tightened HTTP method verification across the board for actions that modify data
- Turned some administrative tasks to POST only actions
Merge and upgrade of several new features
Conflicts:
VERSION.json
app/Controller/ShadowAttributesController.php
app/Controller/TagsController.php
app/Model/AppModel.php
app/Model/Event.php
app/Plugin/SysLogLogable/Model/Behavior/SysLogLogableBehavior.php
- Added logging of failed login attempts
- Added (optional) logging of successful authentications
- admin setting that has to be enabled
- will log all API calls (both HTTP method and target url)
- optional logging of user IP address for all logs
- each log entry created while this setting is enabled will log the IP address of the client
- disabling it also hides the IPs from the interface
- added new IP field for the log search (only if enabled)
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- .sql file to add all the new fields / tables
- admin tool to convert the old organisation fields to the new objects
- still missing a cleanup method (to remove the old organisation fields once the conversion is done)
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- added a new entry to the admin tools (Administartion -> Administrative tools)
- converts title and change columns in the logs table to text from varchar(255)
- user will get an explanation of the csrf error and that going back and refreshing the form will fix it
- also, there is a link that will take the user to the baseurl (which will redirect to the login page if the csrf issue occured on the login page)
- Users being logged on would not be able to use the actions that are also used for automation
- Those actions trigger a check of the authorization header, which in certain use cases can be set with values that is outside of the scope of MISP
- MISP will now try to only detect MISP auth keys in the headers and if it detects something else it ignores it
- The event export buttons have been unified into a single download as... button
- clicking it loads a popup with all of the export formats
- added snort, suricata, text dump to the export options
- added the option for an extra setting for some exports (such as including non IDS flagged attributes, encoding attachments)
- easily extendable system
- moved the hidden popup divs into the general layout, can be easily reused anywhere
- removed the auth refresh option that was re-enabled recently as it seems to sometimes cause issues
- text exports now allow "all" to be specified as type, which will dump all attribute values that the user can see
- text exports now allow restricting the results based on event id
MYSQL.sql and upgrade_2.3.sql updated
Fixed incorrect proposal counts showing up due to attributes that are flagged for deletion also being counted
Added some extra fields to the view proposal view to make it more useful
- Templates can now be created and populated
- Users can populate an event using a template (still needs work)
- File type elements are not yet implemented
- Authentication via headers was only allowed if _isRest() returned true
- this only happened for pages returning JSON or XML content
- a new check, _isAutomation() was added that allows authentication via headers for certain methods used by the automation system
- xml version now included in the xml exports
- MISP will now check the xml version on all imports related to sync / add MISP XML and try to update the incoming info if it detects an older version
- exports now take tag names as a parameter (affected exports: XML, text, HIDS, NIDS)
- eventtags now correctly get removed when an event is deleted
- authorization key should be sent through headers.
- passing it in the url is deprecated
- updated automation page to reflect the changes
- csv export now has headers
- Contextual comments for proposals
- shows proposal count in the top bar
- new view showing all of the events of the user's organisation with an active proposal
Also, more work on the background jobs
- started work on publishing
- started making the background jobs an optional setting in bootstrap
Conflicts:
app/Controller/AppController.php
app/Controller/EventsController.php
- adds JSON example to shell scripts
- adds sample JSON event
- ??? for some redundant Attribute model conditions
- updates travis with CakePHP installation
Some small travins changes too.
FYI there's an automated travis build available at
https://travis-ci.org/MISP/MISP
We don't have unit testing and travis setup is subpar so everything will fail
for now.
- ADMIN org removed.
- Siteadmins are now identified by the perm_site_admin flag
- Siteadmins can now be of any organisation
- editing the regexp / whitelist rules can now be done by a special user with the perm_regexp_access in his/her role
- Executing a mass replace of attribute values based on the regexp rules cannot be initiated by a regexp/whitelist user, only by a site admin
- If the login page is reached without any users / roles defined they are automatically created (perviously it was only the user that was created)
- Org admins are restricted from assigning perm_site_admin, perm_sync and perm_regexp_access roles to users. This can only be done by a site admin.
- users can search RESTfully for attributes based on various filtering mechanisms and get either an event that includes the located attribute(s) or just an array of attributes returned.
- users can also request all attributes of a (or several) types and get them returned as an XML
- we have introduced the "locked" flag for events to protect events of the original creator from being edited by a sync user
- IMPORTANT: before running the script below, make sure to create the locked field for the event table (see INSTALL/LOCKED.sql)
- This script (generateLocked found in the Administrative tools menu) will attempt to set the locked value for existing events to ease the transition
- The default value for locked is 0, and all events created on the instance should be set to this value
- events that were synced from another instance should have their locked value set to 1
- this script checks for local organisations and sets the locked field to 1 for all events not created by them
- a local organisation, as defined for the scope of this scrips is: an organisation with at least 2 members or an organisation with a single member that is not a sync user.
- The script is only accessible by site admins and will return a notification about the number of events altered.
- until now checkAction was used to check permissions of a user
- but since all of the role permissions are checked beforefilter in
appcontroller and saved into a public array, doing a lookup of the
array saves an SQL call for each permission check.
- Orgs can propose new attributes or changes to existing attributes for
events that they do not own
- publishing users of the owner organisation can see, accept or discard
them
- Reworked the access control
- minor fixes
- User/Role not looked up recursively anymore for authorisation checks -
improves performance significantly. Also, checking perm_add and
perm_modify instead of doing a lookup in the ACL tables
- To restrict the authentication key from being used by interactive users,
implemented a new export page that uses the uses cake's user
authentication
- the old export features still exist for users with perm_auth enabled
accounts - renamed to automation
- Exporting the events that found attributes belong to in a search
attributes result page
- exporting of individual events to file by clicking a link in event view
- generateCount used to just run through all attributes and save them, to
generate the count. It led to VERY long execution times on larger
databases (25k+ attributes). With the extra processing that each save()
does for attributes, this was horribly slow.
- new generateCount just saves the events based on the number of
associated attributes, only having to save the events (of which there
are considerably less).
- some changes to the access control
- re-renabled regexp and blacklists, will need a closer look though
- editing a role should update ACL
- some other minor things
- Fixed issues with the sync
- Secondary publishes on remote servers failed
- Introduced new fields in events to stop backward traverse of
edit information that lead to low performance and eroneous
distribution information updates when more than 2 servers were
linked
- Deletion of an attribute now deletes on remote servers
- Changes to the event ownership
- Original creator org now noted in the event itself
- Only original creator org can change distribution
- Events will show up with the original creator org for users
(admins can see both that and the owner of the event on the
local instance)
- Server.organization now used in junction with the connecting
user's org and the instance's org (from the bootstrap) to
determine distribution flow control and access rights
- Lots of minor changes
- perm_auth new toggle, can disable auth key usage for a role
- prevents sync / rest with a perm_auth == false key
- some changes to sync to provide better feedback on why it failed
- rewording of distribution options
- moved and fixed the aros_acos creation on the new role creation
- new method in appController that sets all the aros_acos from scratch
(for example for a new instance, or a changed acos / aros table)
- some minor changes, redirects to the terms page on invalid events
removed, etc.
- Admins cannot manually change anyone's authkey, they need to generate a
new one via the reset link
- Some pages could be accessed by changing the url - fixed (though needs
further testing)
- Edited a change in the manual that may have been confusing
- Some changes to the way ACL is set up - still needs more work
-Analaysis levels setable for events as per milestone item 94
-Password change forced as per milestone item 109
-Added feedback on entered search terms for search attributes
-fixed the authentication issue
-some minor fixes
iglocska
Steve Clement
Alexandre Dulaunoy