- Fixed file header formatting for the export to Bro format (tabs used consistently).
- Computing the time needed for generating the export to Bro format when done using a background job.
- When generating the Bro export from the UI all the attributes are generated in one single text file similar to the CSV export instead of a zip file with different files inside.
- Changed the file extension of Bro export files from ".intel" to ".txt".
- Removed the allowNonIDS option from the Bro export as it doesn’t make sense to have it (Bro is an IDS).
- Fixed some of the API endpoints which were not accepted (ACL issues).
- Added support for a list of events that should be / should not be included in the export.
- Added a new "meta.desc" column (added in Bro 2.5, see https://www.bro.org/sphinx/frameworks/intel.html) containing the description of the event and of the attribute.
- Sanitized the exported data for Bro.
- Fixed a number of value substitutions which were imported from Snort/Suricata and which were not working for Bro. Did instead substitutions needed for Bro.
- redundant var initialisation
- for the automatically created organization the "created_by" is 0, which produces a Notice error in /View/Organization/view.ctp
- Tightened the rules for export generation when no valid published events exist
- Corrected various issues with the progress bars
- Added the missing JSON export to the caches
- XML/JSON caches now correctly take into account the cached attachent inclusion setting
- MISP will now show the users browsing the export page whether attachments will be cached with the current settings or not
- Added correct progress bar to the HIDS export
I also added this to the contactemail(), publish(), postsemail() and alertemail(). But it's commented out as it's not part of the issue. I can commit it again w/ the lines uncommented.
- tied into automatic datamodel updates
- correlation is one way only (from proposal to attribute)
- proposals don't correlate with one another
- all distribution rules are adhered to
- further improvements on the upgrade mechanism pipeline
- New generic fetch attribute method was mistakenly using the order field as a condition, resulting in some exports only displaying a subset of the data
- the fix to this fixes the issue described in #790 for text exports
- Fix to the RPZ exports not working correctly
- Fix to the horrible performance of RPZ exports
- Fix to several background worker issues with exports
- scheduled pulls would fail because of invalid user object passed
- invalid permissions checks / org checks would cause the RPZ export to fail when using background workers
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- finished preview feature
- can now view events and attributes remotely
- can copy over new event to local instance
- new sync mode (update)
- allows to only pull changes to events that exist locally already
- works well with the manual pull of events, no need to pull events that we didn't manually confirm, but can still update all events that we pulled over
- Fixed an issue with background tasks causing the logging to fail
- reworked connection test showing version numbers of both instances
- also telling the admin whether the sync is compatible or not
- Further refactoring / tweaking of the vent view
usage:
/var/www/MISP/app/Console/cake userInit -q
returns the created auth key or an error message if users already exist
The created account is an admin user, with the login being admin@admin.test / admin
- .sql file to add all the new fields / tables
- admin tool to convert the old organisation fields to the new objects
- still missing a cleanup method (to remove the old organisation fields once the conversion is done)
- removed some junk
- more work on the background workers
- rewrote the correlation background job - should work correctly now and be a lot more memory efficient
- decision to be revised: exports don't expose Sharing groups / org uuids to users unless they are admin (for the future: at least sync users have to be added for the new sync)
- CSV caching was saving to file on each attribute, creating extremely high amounts of I/O
- reduced it to saving to file / event
- fixed incorrect pathing
- Scheduled pulls should work correctly now
- Scheduled pushes and pulls correctly display in the logs
- Scheduled caching correctly sets the next date of execution
- MYSQL.sql file now correctly includes the task entries
- GenerateCorrelation admin task is now a background job
- Organisation of events pulled now get the org in the server object as the owner instead of the one who initiates the pull
- Small fix to wrapping text in the pivot graph
- XML export was slow, replaced SimpleXML with a simple script that outputs XML for massive performance gains
- New option in bootstrap to allow the cached XML export to also include the attachments
- CSV caching slightly rearranged, it's much more memory efficient now
- Some fixes to relatedevent orgs being shown even if showorg is disabled
- Added a new site admin action to generate several 3k events for load testing (slow)
- event level exports from the event view now export all attributes regardless of to_ids value
- to_ids value now has its own column in the csv exports
- during the event id pull, the local server already checks the timestamps, removing the ids of events that are not newer than the local version
- this results in only the event metadata being pulled for all events, and the attributes of only those events that need to be updated are pulled resulting in much quicker pulls
- Fixed an issue with proposals that got pulled not finding the attribute that they are proposals to (for proposals that belong to an attribute)
- regexp structural changes added to the upgrade script (type)
- Added publish / alert to the background jobs
- fixed a misalignment with the statistics
- Scheduled tasks for pull / push now working as intended
- Rescheduling of all tasks fixed
- protection against the rescheduled task ending up in the past
- further event history fixes
- fixed lots of erroneous logging
- performance improvement with logging (no longer loading controllers for no reason)
- logging extra actions that weren't logged before (proposal accept / discard, server pull / push)
- you can now upload a certificate file and allow a server link to use a provided self signed certificate. This should solve the issues that some organisations are having when trying to connect their instances
- Corrected some weak notifications on background jobs
- Changed the view slightly to view background jobs
- fixed an issue where editing a sync server setting would cause an error due to the id not being passed to the logging plugin
- added scheduler to the export caching
- site admins can set up the intervals of the automated caches, and the exact times at which they should be executed.
Also, more work on the background jobs
- started work on publishing
- started making the background jobs an optional setting in bootstrap
Conflicts:
app/Controller/AppController.php
app/Controller/EventsController.php
Liviu Valsan