- removed incorrect, useless boiler plate comments
- kept useful comments intact
- added some missing line breaks to make the codebase a bit more uniform
- removed some obviously obsolete TODO comments
- Tightened the rules for export generation when no valid published events exist
- Corrected various issues with the progress bars
- Added the missing JSON export to the caches
- XML/JSON caches now correctly take into account the cached attachent inclusion setting
- MISP will now show the users browsing the export page whether attachments will be cached with the current settings or not
- Added correct progress bar to the HIDS export
- it seems that for some reason some conditions in the correlation lookup massacred the performance of the correlation
- doing that additional filter on a PHP level fixes it for now, but it would be interesting to investigate this further and potentially reuse the findings to improve other queries
- also fixed an issue with the indexing script failing on some fulltext fields if it has to fall back to regular indeces.
- some performance tuning for the restSearch API
- fixed an issue where overriding the contain parameters in the attribute fetcher would lead to an exception
- fixed an issue where accepting a proposal would try to copy the sharing group of the event incorrectly (it now simply gets set to inherit event)
- fixed an issue with the rest search API failing when some fields were not set
- New generic fetch attribute method was mistakenly using the order field as a condition, resulting in some exports only displaying a subset of the data
- the fix to this fixes the issue described in #790 for text exports
- Fix to the RPZ exports not working correctly
- Fix to the horrible performance of RPZ exports
- Fix to several background worker issues with exports
- add / delete sightings via REST
- add sightings via the UI
- View sightings info on an event and attribute level (event view only for now)
- differentiate between own sightings and that of other orgs (additional information via popover still coming)
- settings:
- 1. enable / disable sightings server wide
- 2. set sightings policy
- a. Only Event owner can see sightings + everyone sees what they themeselves contribute
- b. Anyone that contributes sightings to an event can see the sightings data
- c. Everyone that can see the event can see the sightings
- 3. Anonymisisation (in progress, data correctly retrieved in business logic)
- a. if true, then only own org + "other" is shown
- b. otherwise all orgs that submitted sightings are shown
Further improvements needed for version 1 of sightings:
- 1. Delete via the interface
- 2. View detailed sightings information
- 3. Graph the sightings data for the event
- 4. Include the Sightings data in the XML/JSON views
- 5. View sighting for attribute / event via the API
- defanged URL type attributes are refanged on input
- admin script to do the same for all existing attributes
- admin tool doesn't recognise a word followed by a . as a url
- allows for saving an event even if an attribute fails
- logs attributes that fail validation
- same for edit
- add_misp_export updated with the above in mind
- also added 2 new types: bank-account-nr and aba-rtn
- validation is completely relaxed
- idea is to add a visual notification in the view for these attributes types if they are not valid (invalid financial indicators are still interesting)
Also, reworked a lot of remaining distribution checks not handled by the main fetch methods
Conflicts:
VERSION.json
app/Controller/AttributesController.php
app/Controller/ShadowAttributesController.php
app/View/ShadowAttributes/add.ctp
app/View/ShadowAttributes/edit.ctp
- Fixed a critical bug in the XML export
- As of recently XML exports include relations as they were missing before
- the sanitisation of the event info field in related attributes was incorrectly sanitized of unicode characters
- this can lead to the XML export breaking and also for affected events to be blocked from synchronisation
- Proposal fixes
- fixed an invalid uuid generation that lead to an exception
- fixed the attachments for proposals still using the old attachment system that disallows most filenames
- added the automatic creation of hashes for attachment proposals
- Users can now propose a deletion to an attribute
- also tied into the mass accept mechanism
- new UI elements to go along with this
- Code refactoring for category list retrievals
- Until now, several methods got the list of categories from the validation code
- Was awkward with a fake empty element that had to be removed
- altered the validation code to read the categoryDefinitions array instead
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- .sql file to add all the new fields / tables
- admin tool to convert the old organisation fields to the new objects
- still missing a cleanup method (to remove the old organisation fields once the conversion is done)
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- Instead of fetching all events at once for the export, events are fetched one by one
- Greatly reduces memory footprint (It mostly depends on the event with the most eligible attributes now, instead of the combined list of all events)
- Because of the lower memory usage, the time taken for the export is also slashed to a fragment of what it was before
- fixed some issues with unset variables (from, to, last) when triggered by the background workers
- reduced memory usage of the hids exports (removed storing the hashes twice in memory, drastically removed the data retrieved from the db when preparing the export)
- some errors in the format (wrong comment character used, rpz-ip not appended to IP addresses, missing semi-colon)
- removed hostnames that are on domains blocked by the rules based on domain attributes
- contact reporter first tries to contact orgc users on the instance, if they don't exist, it will contact the owner (instead of going straight to the owner)
- hostname / domain name validation change broke validation of hostnames/domain names / email addresses with a "-"
- Some documentation changes for the REST API (more coming)
- some tuning of the freetext import
- HIDS exports did not include filename|hash types
- Sending a password reset / welcome message picked the opposite subject line
- line breaks were sent as literals.
- added the new flag "last" to the list of parameters
- exports affected: XML, CSV, NIDS, HIDS, STIX, Text, RestSearch
- Valid values: number + format where format can be d, m, h for day, minute, hour (examples: 5d or 12h or 30m)
- documented in automation view
- right now it follows the simple rule of user > admin settings > default values when generating the export
- Parameters can be passed via url / JSON object / XML object
- filters include filter on event ID, date range, tags
TODO:
- buttons for a per event download via the UI
- introduce new export option for normal users (via background workers and the old style export)
- decision to be revised: exports don't expose Sharing groups / org uuids to users unless they are admin (for the future: at least sync users have to be added for the new sync)
- Unified the way exports accept negated parameters
- Fixed the documentation
- Most exports are now restrictable by the event date (From/To parameters)
- none cached XML export now writes to file after converting each event, clearing the memory and resolving any potential memory issues
- Scheduled pulls should work correctly now
- Scheduled pushes and pulls correctly display in the logs
- Scheduled caching correctly sets the next date of execution
- The event export buttons have been unified into a single download as... button
- clicking it loads a popup with all of the export formats
- added snort, suricata, text dump to the export options
- added the option for an extra setting for some exports (such as including non IDS flagged attributes, encoding attachments)
- easily extendable system
- moved the hidden popup divs into the general layout, can be easily reused anywhere
- removed the auth refresh option that was re-enabled recently as it seems to sometimes cause issues
- text exports now allow "all" to be specified as type, which will dump all attribute values that the user can see
- text exports now allow restricting the results based on event id
- first version of templating system complete
- first version of freetext importer complete
- first version of mass attribute replace tool complete
- some UI changes
- resolved bugs with permissions
- fixed the broken mass delete tool
- Fixed an issue with the type not being chosen correctly for file type attributes when created through the templating tool
- Templates can now be created and populated
- Users can populate an event using a template (still needs work)
- File type elements are not yet implemented
- images updated in user manual
- fixed validation issues with named pipe (at the moment it's very loose)
- Fixed an issue with shadow attriubutes not showing for events that have no attributes
- some minor UI changes to make MISP a bit prettier
- xml version now included in the xml exports
- MISP will now check the xml version on all imports related to sync / add MISP XML and try to update the incoming info if it detects an older version
- exports now take tag names as a parameter (affected exports: XML, text, HIDS, NIDS)
- eventtags now correctly get removed when an event is deleted
- cleaned up the methods, they all now return results without debug mode enabled
- Added a verification method for all user GPG keys (as an expired key for example would send out empty messages)
- you can now upload a certificate file and allow a server link to use a provided self signed certificate. This should solve the issues that some organisations are having when trying to connect their instances
Also, more work on the background jobs
- started work on publishing
- started making the background jobs an optional setting in bootstrap
Conflicts:
app/Controller/AppController.php
app/Controller/EventsController.php
iglocska
Alexandre Dulaunoy