- CSV export ignored the tag parameters
- tagging events didn't work as expected in some cases
- timing out and clicking on an admin action results in being redirected to a non-existing admin login page
- distribution setting ignored when uploading attachments
- OpenIOC import now correctly sets IDS flags based on type
- OpenIOC import specifies the source file in the comments
- Fixed a blackhole issue with the password reset popups
- eventid a new parameter for both event and attribute restsearch
- these APIs now accept arrays in both json and xml format (you can send "eventid": ["15", "16"] instead of "eventid": "15&&16" in addition to the old functionality
- added support for SHA types
- fixed an issue that caused the import to fail with duplicate attributes (the list gets pruned now)
- fixed an issue where no supplied contextual fields would lead to empty attributes being created
- removed the requirement for the files to have the .ioc extension
- enter a UUID in the event ID field of the attribute search to find attributes belonging to a certain event
- use event IDs / UUIDs to filter events on the event index
- new functionality: Event blacklisting by UUID
- site admins cna enable this feature in the server settings
- enabling the feature will make the required db changes
- any deleted event will automatically get blacklisted
- this prevents deleted events from flowing back from a synced instance
- site admins can manually add UUIDs to the list and remove entries
- fix to UUID duplication issues for attributes
- simply run the admin script and it will regenerate the UUID of attributes that are duplicates, if any such exist
- timestamps/event published status will not be affected
- config.core.php now includes a change that prevents from 404 exceptions being logged
- the sync uses 404s to signal that an event with a given uuid does not exist when negotiating proposal synchronisation
- this causes a dangerously high amount of noise in the logs
- as explained on the automation page
- also, better error handling
- all API calls that fail during authentication will now return a JSON/XML error message instead of redirecting to the login page
- simply pass an MD5 hash along and receive a sample if available zipped and base64 encoded in a response object
- pass any hash along with a flag set and receive any samples from events that have the passed hash
- Also, fix for an issue with the freetext import not using semi-colons as separators
- Threat level ID options correctly set
- Threat level ID validation tightened to reject anything but the existing threat levels
- The upload malware API now logs validation issues during the failed creation of attributes / events
- new API for uploading malware samples
- allows the upload of several files
- can be used to populate a pre-existing event, or create a new event
- expects a JSON or an XML object with the samples base64 encoded
- new way of storing malware samples
- original filename not used any longer
- samples are renamed to their md5 hashes
- original filename preserved in a secondary txt file
- removed filename validation as it is no longer used for the command line execution
- this allows unicode name files to be uploaded!
- changed the UI attachment upload to reflect these changes
- code more centralised and extendible
- added the new flag "last" to the list of parameters
- exports affected: XML, CSV, NIDS, HIDS, STIX, Text, RestSearch
- Valid values: number + format where format can be d, m, h for day, minute, hour (examples: 5d or 12h or 30m)
- comma separated values now correctly parsed
- Ports in IP/url/link/domain/hostname now added as a comment
- virustotal now automatically recognised as external analysis / link
- documented in automation view
- right now it follows the simple rule of user > admin settings > default values when generating the export
- Parameters can be passed via url / JSON object / XML object
- filters include filter on event ID, date range, tags
TODO:
- buttons for a per event download via the UI
- introduce new export option for normal users (via background workers and the old style export)
- decision to be revised: exports don't expose Sharing groups / org uuids to users unless they are admin (for the future: at least sync users have to be added for the new sync)
- Performance improvements for the event search exports
- JSON view code moved to Lib
- Fixed an issue that didn't restrict the dates correctly with the from / to parameters
- Caused by a 1k variable / form limit imposed by php since 5.3.9
- Form data now collected by JS and passed as a single JSON in the POST request
- Allows massive IOC lists to be imported
- improved performance
- won't write to file after all, simply keeps adding to a string in memory. Should still resolve the XML conversion taking up high amounts of memory issue.
- Unified the way exports accept negated parameters
- Fixed the documentation
- Most exports are now restrictable by the event date (From/To parameters)
- none cached XML export now writes to file after converting each event, clearing the memory and resolving any potential memory issues
- The event export buttons have been unified into a single download as... button
- clicking it loads a popup with all of the export formats
- added snort, suricata, text dump to the export options
- added the option for an extra setting for some exports (such as including non IDS flagged attributes, encoding attachments)
- easily extendable system
- moved the hidden popup divs into the general layout, can be easily reused anywhere
- removed the auth refresh option that was re-enabled recently as it seems to sometimes cause issues
- text exports now allow "all" to be specified as type, which will dump all attribute values that the user can see
- text exports now allow restricting the results based on event id
- freetext import now optionally allows setting the comment field
- removing rows in the freetext import result redirects to the event view if all rows are gone
- users can now search on attributes
- attribute search returns any event that has a a sub-string match on the entered attribute
- can also be used to negate (e.g: don't show me any events that have a sub-string match on any of its attributes)
- STIX export had 2 issues as pointed out by RichieB2B:
- Incorrect name assigned to incidents due to copy-pasta fail
- Historyitems incorrectly handled
- For the OpenIOC import:
- Mapping DnsEntryItem/Host to hostname
- Mapping of hostnames to Network activity failed due to incorrect capitalistion
- Temporarily removed the ignore function on certain indicators. Ignoring an element in an AND-ed branch happens without a pruning of the element IDs
- tags not filtered correctly
- status bar showing current filters now shows actual strings for tags / analysis / distribution / threat level instead of the IDs
- server setting has to be enabled to allow for this
- can cause issues if the event gets synchronised with an instance that has a different creator organisation for the same event
- it is recommended not to use this, but in some cases it can be very helpful - the setting for it in the configuration is called MISP.take_ownership_xml_import
- XML export was slow, replaced SimpleXML with a simple script that outputs XML for massive performance gains
- New option in bootstrap to allow the cached XML export to also include the attachments
- CSV caching slightly rearranged, it's much more memory efficient now
- Some fixes to relatedevent orgs being shown even if showorg is disabled
- Added a new site admin action to generate several 3k events for load testing (slow)
- Tags are now fully shown on the event index
- can be enabled via bootstrap (the Configure::write setting is in the bootstrap.default.php file)
- shorthand distribution names
- narrowed some of the fields down
- STIX export performance greatly improved thanks to 84ce8d8be6376797053668d68e1b863713f008dd
- some junk removed
- fixed some minor pagination issues on the event view
- site admin dummy event creator now has target-* type attributes
MYSQL.sql and upgrade_2.3.sql updated
Fixed incorrect proposal counts showing up due to attributes that are flagged for deletion also being counted
Added some extra fields to the view proposal view to make it more useful
Users could only choose their own organisation in the org filter due to an overly restrictive filtering of the available options. Relaxed to all organisations that have an event that is visible to the user.
- send uuids of events to be pushed together with timestamps to the other instance
- other instance removes events that are already up to date or locally created from the array
- sends the remaining uuids back
- first instance initiates the push of events that were not filtered out
The contributor field in the event view is evaluated based on proposal log entries from the log table affecting the current event. In order to improve performance, the LIKE check for the event ID is moved to the last argument in order to avoid parsing rows that could be ignored by the other arguments quicker.
- currently to_xml() has performance issues, if it's not resolved fast, it would be a good idea to move the export to the background workers
- some UI changes
- first version of templating system complete
- first version of freetext importer complete
- first version of mass attribute replace tool complete
- some UI changes
- Events exported were enclosed in a <response> tag, which the sync automatically filters out, but a manual export and import would fail on edits
- added a conditional that removes the <response> tag if an event is encapsulated in a request to the edit method
- AJAX requests now also respond with a small message at the bottom of the page, notifying the user of the result
- The following actions work now on the event page via ajax:
1. Add / remove tags
2. quick edit any attribute field if eligible
3. quickly create a proposal of any attribute field if not eligible to edit
4. popover attribute creation (also works with batch add)
5. popover proposal creation (also works with batch add)
6. delete attributes
7. accept/discard proposals
8. mass edit / delete attributes
Also, replaced the old memberslist, with a small lightweight css/js based one.
- Exporting a JSON object erroneously included related objects which prevented the exported event from being added back to MISP via the API
- Downloading search results as XML / CSV now correctly includes all of the search results instead of just the 60 visible ones on the UI (cut off by the pagination)
- The tags parameter in the exports now correctly accepts null as a valid value even if it is the last parameter
- users can now edit all fields in an attribute whilst on the event page
- issues left to fix:
- tag changes after an attribute change run into CSRF protection
- batch add not handled gracefully yet
- going back to the event view and editing a field gives users an error message over the CSRF protection - instead, silently check if the page is loaded in a dirty way and refresh the ajax fields silently
- quickadd of attributes still missing
- upgrade script broke adding events via the rest interface if they had an xml_version included
- fixed, also, add now more flexible with directly adding events from an export encapsulated in a response tag
- events/restSearch, attributes/restSearch, events/xml, attributes/returnAttributes
- users can now POST a search array in XML / json instead of sending the parameters in the url
- event level exports from the event view now export all attributes regardless of to_ids value
- to_ids value now has its own column in the csv exports
Adds a check for a true value in GnuPG.onlyencrypted and will only display the "No GPG Key Set in your Profile" message to the user if it is missing AND MISP is set to send only encrypted email. This way orgs not using GPG will not see the banner on every index view.
- It is now possible to restrict the CSV automation export by type / category
- updated the automation page to describe how the syntax works
- fixed an issue with line breaks not being sanitized for the CSV export
- Until now, organisations that have made any change to an event in the past (even including an admin running scripts that update the event) would flag an event as having an extra contributor
- From now on, the Contributors field only shows orgs that have created proposals
- xml export now correctly exports all attachments if specified as parameter
- print view fixes
- disclaimer for old IE versions (< 10) and compatibility mode users when viewing the statistics (The heatmap calendar requires 10+)
Iglocska