- tied into auto upgrade system
- tied into server settings
- some cleanup of overly verbose debug
- Enforcing enable/disable everywhere
- Changed temporary file structure
- new syntax
- old syntax still accepted
- new tool for rearranging request data to allow the APIs to automatically catch and correct typical rearrange errors
- freetext is now a valid return format, it will allow module developers to return an unparsed text blob which MISP will try to loop through the freetext import's detection mechanism
- still a lot of improvements to be done for the detection mechanism
- error handling for modules, instead of discarding errors they are now shown as a flash message on the freetext import result screen
- url detection would detect any word with a trailing "." as a valid url
- google. was detected as a url
- this also caused training "."s to be included in valid urls
- http://www.google.com.
- Double sanitisation when edditing an attribute/proposal comment removed
- Fixed an issue where an ip/resource was recognised as a CIDR notation IP range instead of a url
- Changed the flash message for publishing without e-mails to something less scary
- defanged URL type attributes are refanged on input
- admin script to do the same for all existing attributes
- admin tool doesn't recognise a word followed by a . as a url
- allows for saving an event even if an attribute fails
- logs attributes that fail validation
- same for edit
- add_misp_export updated with the above in mind
- called Add MISP export now
- can be an XML / JSON file
- result browser with explanations of failures
- REST XML/JSON add/edit of events returns errors instead of the partially succeeding event
- it didn't use the validation type -> validation method array to call the validation function
- resulted in CC validation not being called as expected
- Fixed a critical bug in the XML export
- As of recently XML exports include relations as they were missing before
- the sanitisation of the event info field in related attributes was incorrectly sanitized of unicode characters
- this can lead to the XML export breaking and also for affected events to be blocked from synchronisation
- Proposal fixes
- fixed an invalid uuid generation that lead to an exception
- fixed the attachments for proposals still using the old attachment system that disallows most filenames
- added the automatic creation of hashes for attachment proposals
Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- implemented a custom pagination tool for data sets that are not directly taken from teh db
- currently creates a pagination object that mocks CakePHP pagination
- supports the CakePHP pagination view helper
- supports: pagination, sorting, custom filters
- implemented first step of the remote instance browser for admins
- view an index of events on another instance
- filter the events
- uses the new pagination
- still missing:
- remote event view
- fetch event from remote instance
- reworked the event view
- separated API and UI code path
- major speedup for the API!
- cleaner code as there was almost 0 overlap
- discussions and attributes are now loaded separately from the event view
- added after the event view loads via ajax
- cleaner pagination
- attribute pagination now finally allows for sorting
- future improvement (coming soon): Show proposals only filter
- filtering on the attributes in general
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- Instead of fetching all events at once for the export, events are fetched one by one
- Greatly reduces memory footprint (It mostly depends on the event with the most eligible attributes now, instead of the combined list of all events)
- Because of the lower memory usage, the time taken for the export is also slashed to a fragment of what it was before
- fixed some issues with unset variables (from, to, last) when triggered by the background workers
- reduced memory usage of the hids exports (removed storing the hashes twice in memory, drastically removed the data retrieved from the db when preparing the export)
- some errors in the format (wrong comment character used, rpz-ip not appended to IP addresses, missing semi-colon)
- removed hostnames that are on domains blocked by the rules based on domain attributes
- python server running in the background doing the publishing
- MISP -> python script communication via redis
- configurable / controllable via the admin UI
- by installing the requirements described in the update and the install instructions (ubuntu only for now, centos/red-hat versions to be tested and described), administrators can enable the pub/sub feature
- assign a port to the service via the interface
- each time an event is published, MISP will use ZMQ's PUB feature to push out a MISP JSON package using the "misp_json" prefix
- contact reporter first tries to contact orgc users on the instance, if they don't exist, it will contact the owner (instead of going straight to the owner)
- hostname / domain name validation change broke validation of hostnames/domain names / email addresses with a "-"
- Some documentation changes for the REST API (more coming)
- some tuning of the freetext import
- HIDS exports did not include filename|hash types
- Sending a password reset / welcome message picked the opposite subject line
- line breaks were sent as literals.
- comma separated values now correctly parsed
- Ports in IP/url/link/domain/hostname now added as a comment
- virustotal now automatically recognised as external analysis / link
- documented in automation view
- right now it follows the simple rule of user > admin settings > default values when generating the export
- Parameters can be passed via url / JSON object / XML object
- filters include filter on event ID, date range, tags
TODO:
- buttons for a per event download via the UI
- introduce new export option for normal users (via background workers and the old style export)
- server settings and validation work
- configurable template via settings
- configurable via API as well
- Also trying to define the structure for future Plugin settings
- The idea is to have them in a separate tab all prepended with the plugin name
- since this is not yet part of the future flexible plugin system, it is still kept in the main codebase, but the idea is to get the naming conventions ready for the future version
- decision to be revised: exports don't expose Sharing groups / org uuids to users unless they are admin (for the future: at least sync users have to be added for the new sync)
- Performance improvements for the event search exports
- JSON view code moved to Lib
- Fixed an issue that didn't restrict the dates correctly with the from / to parameters
- Important! Logo images have now moved to a different location! Make sure that you update your settings!
- Site admins can now manage the uploaded image files and the terms of use file via the server settings interface
- add, link, delete files directly from the interface
A change in cakephp version 2.4.8+ has resulted in ajax form submitions breaking. Reason for this was a change in the SecurityComponent taking the url specified in the form into account when generating the CSRF tokens.
This is now fixed by embedding the correct url in the ajax forms.
- resolved bugs with permissions
- fixed the broken mass delete tool
- Fixed an issue with the type not being chosen correctly for file type attributes when created through the templating tool
- Templates can now be created and populated
- Users can populate an event using a template (still needs work)
- File type elements are not yet implemented
- AJAX requests now also respond with a small message at the bottom of the page, notifying the user of the result
- The following actions work now on the event page via ajax:
1. Add / remove tags
2. quick edit any attribute field if eligible
3. quickly create a proposal of any attribute field if not eligible to edit
4. popover attribute creation (also works with batch add)
5. popover proposal creation (also works with batch add)
6. delete attributes
7. accept/discard proposals
8. mass edit / delete attributes
Also, replaced the old memberslist, with a small lightweight css/js based one.
- cleaned up the methods, they all now return results without debug mode enabled
- Added a verification method for all user GPG keys (as an expired key for example would send out empty messages)
- you can now upload a certificate file and allow a server link to use a provided self signed certificate. This should solve the issues that some organisations are having when trying to connect their instances
Also, more work on the background jobs
- started work on publishing
- started making the background jobs an optional setting in bootstrap
Conflicts:
app/Controller/AppController.php
app/Controller/EventsController.php
Iglocska