- tied into auto upgrade system
- tied into server settings
- some cleanup of overly verbose debug
- Enforcing enable/disable everywhere
- Changed temporary file structure
- New permission flag: perm_tag_editor
- taggers can tag events with existing tags
- tag editors can create / edit / delete tags
- Fixed several misleading UI elements for tagging
- tagging users that don't own an event and aren't creators thereof cannot tag them
- this was enforced before but the UI elements were present and threw errors
- Migration is automatic
- all existing tagger roles will automatically become tag editors
- restricting current roles takes manual admin action, but the functionality should remain unchanged for those that just update
- new syntax
- old syntax still accepted
- new tool for rearranging request data to allow the APIs to automatically catch and correct typical rearrange errors
- sync users that have not accepted the terms / have had a password reset initiated were redirected to the login page
- fixes to the issue
- if a user with automation/sync access uses the API and gets blocked because the terms weren't accepted or there is a pending password change they will be notified in a JSON/XML response
- the sync test now takes this into consideration starting with this version and will report the cause of the failure
- Both instances have to be 2.4.24+ for this to be reported correctly
- needs further fixes, sadly the version has always just showed the major and minor version in the exports
- This masked an issue that would block the import of events that are even a hotfix away
- As a temporary fix, I reverted the changes and the XML version field will now only show the major and minor version to restore compatibility (so 2.4.0 instead of 2.4.19)
- not authenticated users now automatically get redirected to the login page, no matter what action they requested
- This as a nice side effect also removed the bug that was caused by a site admin looking at an admin function before logging out / timing out and being incorrectly redirected to /admin/users/login
- users can now be disabled by an admin
- disabled users cannot login (via the UI or the API) and will be informed
- login attempts by disabled users are logged
- also added the expiration field for later use
- some json actions worked by passing the .json extension in the url
- these pages were correctly returning JSONs but were often internally running through the HTML code-path thanks to an invalid detection
- the new correct detection should provide a major speed boost for certain json requests
- It seems like relying on the Accept header can lead to the data type detection failing when accessing .json extension views
- this issue seems to have gone unnoticed since until now the data passed to the json view was the same as that passed to the html view
- this means that all the additional UI only features may have triggered in the background previously on .json views
- as discovered and reported by Egidio Romano of Minded Security
- Lacking checks of HTTP methods in some functionality could lead to a site admin uploading and executing malicious scripts
- Tightened HTTP method verification across the board for actions that modify data
- Turned some administrative tasks to POST only actions
Merge and upgrade of several new features
Conflicts:
VERSION.json
app/Controller/ShadowAttributesController.php
app/Controller/TagsController.php
app/Model/AppModel.php
app/Model/Event.php
app/Plugin/SysLogLogable/Model/Behavior/SysLogLogableBehavior.php
- Added logging of failed login attempts
- Added (optional) logging of successful authentications
- admin setting that has to be enabled
- will log all API calls (both HTTP method and target url)
- optional logging of user IP address for all logs
- each log entry created while this setting is enabled will log the IP address of the client
- disabling it also hides the IPs from the interface
- added new IP field for the log search (only if enabled)
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- .sql file to add all the new fields / tables
- admin tool to convert the old organisation fields to the new objects
- still missing a cleanup method (to remove the old organisation fields once the conversion is done)
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- added a new entry to the admin tools (Administartion -> Administrative tools)
- converts title and change columns in the logs table to text from varchar(255)
- user will get an explanation of the csrf error and that going back and refreshing the form will fix it
- also, there is a link that will take the user to the baseurl (which will redirect to the login page if the csrf issue occured on the login page)
- Users being logged on would not be able to use the actions that are also used for automation
- Those actions trigger a check of the authorization header, which in certain use cases can be set with values that is outside of the scope of MISP
- MISP will now try to only detect MISP auth keys in the headers and if it detects something else it ignores it
- The event export buttons have been unified into a single download as... button
- clicking it loads a popup with all of the export formats
- added snort, suricata, text dump to the export options
- added the option for an extra setting for some exports (such as including non IDS flagged attributes, encoding attachments)
- easily extendable system
- moved the hidden popup divs into the general layout, can be easily reused anywhere
- removed the auth refresh option that was re-enabled recently as it seems to sometimes cause issues
- text exports now allow "all" to be specified as type, which will dump all attribute values that the user can see
- text exports now allow restricting the results based on event id
MYSQL.sql and upgrade_2.3.sql updated
Fixed incorrect proposal counts showing up due to attributes that are flagged for deletion also being counted
Added some extra fields to the view proposal view to make it more useful
- Templates can now be created and populated
- Users can populate an event using a template (still needs work)
- File type elements are not yet implemented
- Authentication via headers was only allowed if _isRest() returned true
- this only happened for pages returning JSON or XML content
- a new check, _isAutomation() was added that allows authentication via headers for certain methods used by the automation system
- xml version now included in the xml exports
- MISP will now check the xml version on all imports related to sync / add MISP XML and try to update the incoming info if it detects an older version
- exports now take tag names as a parameter (affected exports: XML, text, HIDS, NIDS)
- eventtags now correctly get removed when an event is deleted
- authorization key should be sent through headers.
- passing it in the url is deprecated
- updated automation page to reflect the changes
- csv export now has headers
- Contextual comments for proposals
- shows proposal count in the top bar
- new view showing all of the events of the user's organisation with an active proposal
Also, more work on the background jobs
- started work on publishing
- started making the background jobs an optional setting in bootstrap
Conflicts:
app/Controller/AppController.php
app/Controller/EventsController.php
- adds JSON example to shell scripts
- adds sample JSON event
- ??? for some redundant Attribute model conditions
- updates travis with CakePHP installation
Some small travins changes too.
FYI there's an automated travis build available at
https://travis-ci.org/MISP/MISP
We don't have unit testing and travis setup is subpar so everything will fail
for now.
- ADMIN org removed.
- Siteadmins are now identified by the perm_site_admin flag
- Siteadmins can now be of any organisation
- editing the regexp / whitelist rules can now be done by a special user with the perm_regexp_access in his/her role
- Executing a mass replace of attribute values based on the regexp rules cannot be initiated by a regexp/whitelist user, only by a site admin
- If the login page is reached without any users / roles defined they are automatically created (perviously it was only the user that was created)
- Org admins are restricted from assigning perm_site_admin, perm_sync and perm_regexp_access roles to users. This can only be done by a site admin.
- users can search RESTfully for attributes based on various filtering mechanisms and get either an event that includes the located attribute(s) or just an array of attributes returned.
- users can also request all attributes of a (or several) types and get them returned as an XML
- we have introduced the "locked" flag for events to protect events of the original creator from being edited by a sync user
- IMPORTANT: before running the script below, make sure to create the locked field for the event table (see INSTALL/LOCKED.sql)
- This script (generateLocked found in the Administrative tools menu) will attempt to set the locked value for existing events to ease the transition
- The default value for locked is 0, and all events created on the instance should be set to this value
- events that were synced from another instance should have their locked value set to 1
- this script checks for local organisations and sets the locked field to 1 for all events not created by them
- a local organisation, as defined for the scope of this scrips is: an organisation with at least 2 members or an organisation with a single member that is not a sync user.
- The script is only accessible by site admins and will return a notification about the number of events altered.
- until now checkAction was used to check permissions of a user
- but since all of the role permissions are checked beforefilter in
appcontroller and saved into a public array, doing a lookup of the
array saves an SQL call for each permission check.
- Orgs can propose new attributes or changes to existing attributes for
events that they do not own
- publishing users of the owner organisation can see, accept or discard
them
- Reworked the access control
- minor fixes
- User/Role not looked up recursively anymore for authorisation checks -
improves performance significantly. Also, checking perm_add and
perm_modify instead of doing a lookup in the ACL tables
- To restrict the authentication key from being used by interactive users,
implemented a new export page that uses the uses cake's user
authentication
- the old export features still exist for users with perm_auth enabled
accounts - renamed to automation
- Exporting the events that found attributes belong to in a search
attributes result page
- exporting of individual events to file by clicking a link in event view
- generateCount used to just run through all attributes and save them, to
generate the count. It led to VERY long execution times on larger
databases (25k+ attributes). With the extra processing that each save()
does for attributes, this was horribly slow.
- new generateCount just saves the events based on the number of
associated attributes, only having to save the events (of which there
are considerably less).
- some changes to the access control
- re-renabled regexp and blacklists, will need a closer look though
- editing a role should update ACL
- some other minor things
- Fixed issues with the sync
- Secondary publishes on remote servers failed
- Introduced new fields in events to stop backward traverse of
edit information that lead to low performance and eroneous
distribution information updates when more than 2 servers were
linked
- Deletion of an attribute now deletes on remote servers
- Changes to the event ownership
- Original creator org now noted in the event itself
- Only original creator org can change distribution
- Events will show up with the original creator org for users
(admins can see both that and the owner of the event on the
local instance)
- Server.organization now used in junction with the connecting
user's org and the instance's org (from the bootstrap) to
determine distribution flow control and access rights
- Lots of minor changes
- perm_auth new toggle, can disable auth key usage for a role
- prevents sync / rest with a perm_auth == false key
- some changes to sync to provide better feedback on why it failed
- rewording of distribution options
- moved and fixed the aros_acos creation on the new role creation
- new method in appController that sets all the aros_acos from scratch
(for example for a new instance, or a changed acos / aros table)
- some minor changes, redirects to the terms page on invalid events
removed, etc.
- Admins cannot manually change anyone's authkey, they need to generate a
new one via the reset link
- Some pages could be accessed by changing the url - fixed (though needs
further testing)
- Edited a change in the manual that may have been confusing
- Some changes to the way ACL is set up - still needs more work
-Analaysis levels setable for events as per milestone item 94
-Password change forced as per milestone item 109
-Added feedback on entered search terms for search attributes
-fixed the authentication issue
-some minor fixes
respect distribution Org in correlations.
(for this
add correlations.1_private conform MYSQL.correlaton.sql
and
AppController::generateCorrelation() must be run)
now should be okay on the checkGroup.
(mind, we have a PHP 5.3.10 (dev) and 5.2.10 (f.a.) difference.
for CakePHP should be php > 5.2.8, pear > 1.9.0 and phpunit 3.5.0)
Add "Pull only" as a sharing state where,
everybody does see an event, is pullable,
but will never be pushed.
Has a generatePrivate for db conversion now.
Private events are true private and
running a server in 2 modes (private and sync),
so real private (red) or private to server (amber)
or full distributable (green).
Mind this needs a change to tables events, attributes and correlation.
These are in MYSQL.private.sql.
i have an idea this blackholeCallback seems to overcome a lot of
blackhole situations we got.
Notably during deleting multiple events from the index,
this improved not getting a blackhole a lot.
Forgot to clean View/Helper/AppHelper.php.
Changed underscore method names to private and protected where
appropriate given phpcs code standards errors.
in Config/bootstrap.php add
Configure::write('CyDefSIG.correlation', 'sql');
possible values:
- default, like it was
- db, correlation in database
- sql, selection on attributes i.s.o. per attribute
(sql improvement possible if result conform db above)
Network activity, ip-src
30 class-C network ip addresses
(7650 tupels) (time in ms)
default db sql
all 25366 16601 15941
24839 16604 15611
paginated 16759 8447 6615
17734 8639 8846
this is used in both:
- events/view/<id>
- attributes/event/<id>
Iglocska