- New generic fetch attribute method was mistakenly using the order field as a condition, resulting in some exports only displaying a subset of the data
- the fix to this fixes the issue described in #790 for text exports
- Fix to the RPZ exports not working correctly
- Fix to the horrible performance of RPZ exports
- Fix to several background worker issues with exports
- scheduled pulls would fail because of invalid user object passed
- invalid permissions checks / org checks would cause the RPZ export to fail when using background workers
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- finished preview feature
- can now view events and attributes remotely
- can copy over new event to local instance
- new sync mode (update)
- allows to only pull changes to events that exist locally already
- works well with the manual pull of events, no need to pull events that we didn't manually confirm, but can still update all events that we pulled over
- Fixed an issue with background tasks causing the logging to fail
- reworked connection test showing version numbers of both instances
- also telling the admin whether the sync is compatible or not
- Further refactoring / tweaking of the vent view
usage:
/var/www/MISP/app/Console/cake userInit -q
returns the created auth key or an error message if users already exist
The created account is an admin user, with the login being admin@admin.test / admin
- .sql file to add all the new fields / tables
- admin tool to convert the old organisation fields to the new objects
- still missing a cleanup method (to remove the old organisation fields once the conversion is done)
- removed some junk
- more work on the background workers
- rewrote the correlation background job - should work correctly now and be a lot more memory efficient
- decision to be revised: exports don't expose Sharing groups / org uuids to users unless they are admin (for the future: at least sync users have to be added for the new sync)
- CSV caching was saving to file on each attribute, creating extremely high amounts of I/O
- reduced it to saving to file / event
- fixed incorrect pathing
- Scheduled pulls should work correctly now
- Scheduled pushes and pulls correctly display in the logs
- Scheduled caching correctly sets the next date of execution
- MYSQL.sql file now correctly includes the task entries
- GenerateCorrelation admin task is now a background job
- Organisation of events pulled now get the org in the server object as the owner instead of the one who initiates the pull
- Small fix to wrapping text in the pivot graph
- XML export was slow, replaced SimpleXML with a simple script that outputs XML for massive performance gains
- New option in bootstrap to allow the cached XML export to also include the attachments
- CSV caching slightly rearranged, it's much more memory efficient now
- Some fixes to relatedevent orgs being shown even if showorg is disabled
- Added a new site admin action to generate several 3k events for load testing (slow)
- event level exports from the event view now export all attributes regardless of to_ids value
- to_ids value now has its own column in the csv exports
- during the event id pull, the local server already checks the timestamps, removing the ids of events that are not newer than the local version
- this results in only the event metadata being pulled for all events, and the attributes of only those events that need to be updated are pulled resulting in much quicker pulls
- Fixed an issue with proposals that got pulled not finding the attribute that they are proposals to (for proposals that belong to an attribute)
- regexp structural changes added to the upgrade script (type)
- Added publish / alert to the background jobs
- fixed a misalignment with the statistics
- Scheduled tasks for pull / push now working as intended
- Rescheduling of all tasks fixed
- protection against the rescheduled task ending up in the past
- further event history fixes
- fixed lots of erroneous logging
- performance improvement with logging (no longer loading controllers for no reason)
- logging extra actions that weren't logged before (proposal accept / discard, server pull / push)
- you can now upload a certificate file and allow a server link to use a provided self signed certificate. This should solve the issues that some organisations are having when trying to connect their instances
- Corrected some weak notifications on background jobs
- Changed the view slightly to view background jobs
- fixed an issue where editing a sync server setting would cause an error due to the id not being passed to the logging plugin
- added scheduler to the export caching
- site admins can set up the intervals of the automated caches, and the exact times at which they should be executed.
Also, more work on the background jobs
- started work on publishing
- started making the background jobs an optional setting in bootstrap
Conflicts:
app/Controller/AppController.php
app/Controller/EventsController.php
William Robinet