Merging all the new changes from master
Conflicts:
VERSION.json
app/Console/Command/AdminShell.php
app/Controller/AttributesController.php
app/Controller/EventsController.php
app/Model/Attribute.php
app/Model/Event.php
app/Model/Log.php
app/Model/Server.php
app/Model/User.php
app/View/Elements/side_menu.ctp
app/View/Pages/administration.ctp
app/View/Users/admin_index.ctp
- Under these distros, php is blocked from seeing concurrently running php processes even under the same user
- instead of running ps, the diagnostic now checks the existance of the pid file in /proc/
- Event blacklist functionality extended
- Several context fields added
- edit existing entries to change the context fields
- removed the deprecated news page
- hash attribute types get validated against empty values
- fixed an excepion on REST add of attributes when the validation stops an attribute from being entered
- fixed the parameters in some exports being ignored after a recent patch
- added an admin tool to prune orphaned attributes
- cleanup and move of the database update methods - they are now accessible from any model
- Footer now shows MISP version including sub version
- due to a bug, setting an attribute ID in the /attributes/add API call can lead to overwriting an existing attribute
performance improvements:
- massive improvements to the correlation performance
- improvements to the attribute validation process
- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- indexes were not created if they already existed
- this was an issue if a non unique index was present
- also made the process more verbose and added a generic method that deals with index removal
- UUID uniqueness was previously not enforced
- changed the MYSQL.sql file to reflect the changes
- Added upgrade admin tool to remove duplicate events and make the database changes required
- Tweaked the tool for the attribute uuid fix so that it cannot created duplicate keys
- some minor fixes, such as automatically removing eventTag objects on event deletion
- OpenIOC import now correctly sets IDS flags based on type
- OpenIOC import specifies the source file in the comments
- Fixed a blackhole issue with the password reset popups
- eventid a new parameter for both event and attribute restsearch
- these APIs now accept arrays in both json and xml format (you can send "eventid": ["15", "16"] instead of "eventid": "15&&16" in addition to the old functionality
- added support for SHA types
- fixed an issue that caused the import to fail with duplicate attributes (the list gets pruned now)
- fixed an issue where no supplied contextual fields would lead to empty attributes being created
- removed the requirement for the files to have the .ioc extension
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- as explained on the automation page
- also, better error handling
- all API calls that fail during authentication will now return a JSON/XML error message instead of redirecting to the login page
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- Threat level ID options correctly set
- Threat level ID validation tightened to reject anything but the existing threat levels
- The upload malware API now logs validation issues during the failed creation of attributes / events
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- Instead of fetching all events at once for the export, events are fetched one by one
- Greatly reduces memory footprint (It mostly depends on the event with the most eligible attributes now, instead of the combined list of all events)
- Because of the lower memory usage, the time taken for the export is also slashed to a fragment of what it was before
- fixed some issues with unset variables (from, to, last) when triggered by the background workers
- reduced memory usage of the hids exports (removed storing the hashes twice in memory, drastically removed the data retrieved from the db when preparing the export)
- some errors in the format (wrong comment character used, rpz-ip not appended to IP addresses, missing semi-colon)
- removed hostnames that are on domains blocked by the rules based on domain attributes
- python server running in the background doing the publishing
- MISP -> python script communication via redis
- configurable / controllable via the admin UI
- by installing the requirements described in the update and the install instructions (ubuntu only for now, centos/red-hat versions to be tested and described), administrators can enable the pub/sub feature
- assign a port to the service via the interface
- each time an event is published, MISP will use ZMQ's PUB feature to push out a MISP JSON package using the "misp_json" prefix
- contact reporter first tries to contact orgc users on the instance, if they don't exist, it will contact the owner (instead of going straight to the owner)
- hostname / domain name validation change broke validation of hostnames/domain names / email addresses with a "-"
- Some documentation changes for the REST API (more coming)
- some tuning of the freetext import
- added the new flag "last" to the list of parameters
- exports affected: XML, CSV, NIDS, HIDS, STIX, Text, RestSearch
- Valid values: number + format where format can be d, m, h for day, minute, hour (examples: 5d or 12h or 30m)
- Popover_form purged after the form has been submitted
- a duplicate hidden div was created for confirmation popups within the attribute creation popup and clicking publish populated the wrong div
- comma separated values now correctly parsed
- Ports in IP/url/link/domain/hostname now added as a comment
- virustotal now automatically recognised as external analysis / link
- Events published / pushed will now refuse to sync if the situation arises where no attributes would be eligible to be synced
- Events pulled that contain no attributes will be thrown away
- Reworking the way e-mails are sent - all of it goes through a centralised e-mail method
- just pass the recipient, recipient encryption key collection, body, alternate body if the message cannot be encrypted, subject, reply to address and pgp key for reply to along and the method will do the rest
- encrypt if possible, check if sending without encryption is allowed, signing, adding attachment for reply to encryption key, using alternate sanitised body if it is enforced for accounts that cannot use encryption is all done in one place
- easy to maintain and expand with future changes (such as the S/MIME pull request on github)
- Users being logged on would not be able to use the actions that are also used for automation
- Those actions trigger a check of the authorization header, which in certain use cases can be set with values that is outside of the scope of MISP
- MISP will now try to only detect MISP auth keys in the headers and if it detects something else it ignores it
- Unified the way exports accept negated parameters
- Fixed the documentation
- Most exports are now restrictable by the event date (From/To parameters)
- none cached XML export now writes to file after converting each event, clearing the memory and resolving any potential memory issues
- Pull request by RichieB2B: CentOS 6 & 7 installation instructions
- Pull request by RichieB2B: STIX exports now include comments for indicators
- Pull request by RichieB2B: Issue fixed with md5 type attributes not generating observables correctly during a STIX export
- Password policy change-able by a site admin via a regex and a min char requirement. Old functionality assumed if not set.
- bug fixed with incorrect jobs being created appearing during a scheduled pull (designates a push)
- slight changes to the installation instructions
- database.default.php now uses localhost instead of 127.0.0.1 and the default MySQL port
- Scheduled pulls should work correctly now
- Scheduled pushes and pulls correctly display in the logs
- Scheduled caching correctly sets the next date of execution
- Deleting attributes on connected MISP instances can cause serious performance issues on multiple interconnected instnaces, temporarily removed
- Version number incremented
- Important! Logo images have now moved to a different location! Make sure that you update your settings!
- Site admins can now manage the uploaded image files and the terms of use file via the server settings interface
- add, link, delete files directly from the interface
- use terms file as before if nothing else specified
- specify a file in the app/files/terms directory via the server settings tool
- specify whether to show it inline or create a download link for users instead
- by default everything is the same as before, except that the MISP installation path is no longer exposed by a non-existing terms file
Iglocska