MISP 2.4.83 released

2017-12-06 00:30:24 +01:00 · 2017-12-06 00:30:24 +01:00 · f83e8b4efc
parent 7e11ad3c5e
commit f83e8b4efc
1 changed files with 410 additions and 2 deletions
--- a/Changelog.txt
+++ b/Changelog.txt
@ -2,11 +2,419 @@ Changelog
 =========


-%%version%% (unreleased)
------------------------
+v2.4.83 (2017-12-05)
+--------------------
+
+New
+~~~
+- Various improvements to the CSV export. [iglocska]
+
+  - The @FloatingCode and @ilmoka care package
+  - Improved CSV performance for instances with large number of events
+  - Added "value" filter for CSV (use-case: I want all indicators for this value with context)
+  - Added attribute tags to the output of the CSV export
+- Add restrictions for e-mail addresses to certain domains. [iglocska]
+- Add attribute tag filters to the fetchEvents() functionality.
+  [iglocska]
+
+  - tag filters now filter on:
+    - all events cotaining matching tags on event + attribute level (positive lookup)
+    - all events not containing matching tags (negative lookup)
+    - filter attributes within a matched event for blocked attributes (negative lookup)
+
+  - moved tag filtering to subquery filtering - should improve performance massively on larger instances when filtering on tags
+
+  - first round of implementations, more on the way
+- Various improvements. [iglocska]
+
+  - use the feed uuid caches to link directly to affected MISP events
+  - various UI improvements
+  - Feed preview pagination / POSTed event ID filters added
+- Add the possibility to limit fields for the CSV export via POST
+  requests. [iglocska]
+- Added mac-address and mac-eui-64 attribute types. [iglocska]
+- Added full audit logging to ZMQ and Syslog, fixes #2635. [iglocska]
+
+  - syslog now includes all audit log entries and it's separated into proper severity levels
+  - ZMQ logging and syslog logging are both optional features
+- Added phone number recognition to the freetext import tool. [iglocska]
+
+  - also, changed the massaging of phone number type attributes to replace 00 with +
+- Include user action in zmq. [iglocska]
+- Added logging to galaxy attach/detach tasks. [iglocska]
+- Push the action for user updates/creations/logins along with the user
+  object to the ZMQ channel. [iglocska]
+
+Changes
+~~~~~~~
+- Version strings updated. [iglocska]
+- Bump PyMISP, again. [Raphaël Vinot]
+- Bump PyMISP. [Raphaël Vinot]
+- Wip. [chrisr3d]
+- Wip. [chrisr3d]
+- Make misp to stix export work with MISP json formatted. [chrisr3d]
+- Push MISP json formatted events to the stix exporter (pending rework)
+  instead of the direct output of fetchEvents() [iglocska]
+- Push the full user object to the ZMQ feed. [iglocska]
+
+Fix
+~~~
+- Removed unused variable. [iglocska]
+- Latest version of the MISP galaxy. [Alexandre Dulaunoy]
+- Latest version of MISP objects. [Alexandre Dulaunoy]
+- Documentation to enable cortex services. [Raphaël Vinot]
+- Don't cull the list of possible models based on existing data for the
+  search logs view. [iglocska]
+
+  - slow and useless
+- Fixed a bug with the resolved attributes list for freetext import /
+  module imports. [iglocska]
+- Fixed CSV content type. [iglocska]
+- Changed name of export popup. [iglocska]
+- Moved attribute_tags in the CSV export to the includeContext flag
+  instead of the toggle-able attributes. [iglocska]
+- Fixed some issues with the related feeds. [iglocska]
+- Fix epic snafu in Event->_add() thanks to last minute save by the
+  Travis tests. [iglocska]
+- Some minor fixes to the attribute filtering. [iglocska]
+- Fixed an issue where sharing groups were not properly attached to
+  events for sync users, potentially fixes #2653. [iglocska]
+- Added new field to MYSQL.sql. [iglocska]
+- Added db changes needed for the user domain restrictions along with
+  restricting the user self edit action. [iglocska]
+- Fixed an issue where proposal quick edits didn't work for normal
+  users, fixes #2685. [iglocska]
+- Fixed update warninglists button being available to non site admin
+  users. [iglocska]
+
+  - functionality was blocked by ACL, but button shouldn't be shown in the first place
+- Block the addition of same type/category/value attributes in one shot
+  to the same event. [iglocska]
+
+  - via the /events/add api
+- Enforce server push rules on a sync user when viewing the events.
+  [iglocska]
+
+  - user not seeing the data is a side-effect, not the intended effect
+  - serves to enforce the synchronisation rules
+  - sync user can still view the hidden attributes via attribute searches etc. Whether we want to remove this in the future is still to be decided, but for now the sync enforcement is the only intended effect.
+- Mac-eui-64 not accepted by stix validator. [chrisr3d]
+
+  By the way, it is accepted by the validator at creation..
+  .
+- Latest version of the MISP objects template imported. [Alexandre
+  Dulaunoy]
+- MISP objects updated to the latest version. [Alexandre Dulaunoy]
+- MISP galaxy updated to the latest version. [Alexandre Dulaunoy]
+- MISP taxonomies updated to the latest version. [Alexandre Dulaunoy]
+- Dns-soa-email didn't have a category. [iglocska]
+- Fixed missing entries for mac-eui-64. [iglocska]
+- Made CSV parser for freetext import tool / feed ingestion compatible
+  with escaped CSVs. [iglocska]
+
+  - "" now handled correctly
+- Vulnerability (CVE) should correlate (CIRCL and NCSC-NL are supporting
+  it) fix #2691. [Alexandre Dulaunoy]
+- Ambiguity removed from some sharing group related queries. [iglocska]
+- Graceful handling of no response during getVersion pre-sync test.
+  [iglocska]
+- Fix an issue with a double quoted integer in the correlation update
+  script during publishing, fixes #2540. [iglocska]
+- Trimp the org uuid upon entering it to avoid copy-pasta issues.
+  [iglocska]
+- Updated the duplicate attribute removal tool to actually remove
+  instead of trying to deduplicate. [iglocska]
+- Fixes notices of no SharingGroupOrg being set due to a bug in the
+  sharing group cacher for normal users. [iglocska]
+- Fixes to various issues with adding proposals via the freetext import
+  tool. [iglocska]
+
+  - no feedback on whether the resulting dataset will be stored as attributes/proposals
+  - unpublishing of the event when proposals get entered
+  - alerting the event creator of new proposals if coming from the freetext import tool
+- Quotes issue fixed. [chrisr3d]
+- MISP objects updated. [Alexandre Dulaunoy]
+- Leaking of hashed passwords in the audit logs fixed. [iglocska]
+
+  - Scope was limited due to the audit log access restrictions to site/org admins
+- Expose /users/view/me to the API, fixes #2679. [iglocska]
+- Don't verify peer name on self signed certs; don't verify self signed
+  peer if cert is missing. [Milan Pikula]
+- Settings editor not working on touch devices. [Milan Pikula]
+- Refresh rows in settings editor. [Jan Skalny]
+- Relaxed email validation. [iglocska]
+
+  - because unicode tlds / domains are such a great idea
+- Disabled pretty argument. [chrisr3d]
+
+  used while stringifying the final Bundle
+- Fixed invalid timestamp generation. [iglocska]
+- If no distribution level set, don't try to check if it's set to
+  sharing group on the attribute level. [iglocska]
+
+  - Attribute->editAttribute()
+- MISP object updated to the latest version to fix the unusable ASN
+  template. [Alexandre Dulaunoy]
+- Attribute deletes are again synced correctly. [iglocska]
+- Fixes an issue where assigning sharing groups based on existing IDs
+  didn't work for event creation via the API. [iglocska]
+
+  - expected full sharing groups as provided by the sync, references didn't work
+- Fixed the broken feed preview. [iglocska]
+- Fixed the new path for the stix files. [iglocska]
+- Moved the conversion to JSON after the massage of the data for stix.
+  [iglocska]
+- Add galaxy to valid log action list. [iglocska]
+- Shebang mixup. [Steffen Sauler]
+
+  /!bin/sh to !/bin/sh
+- 984732984th time is the charm... [iglocska]
+- Reduced the user data to just a partial user object and organisation
+  object for the zmq push. [iglocska]
+- Fixed the pubsub user push if the user object is not contained within
+  a User key. [iglocska]
+- Previous commit didn't trigger in all cases. [iglocska]
+- MISP objects updated to the latest version. [Alexandre Dulaunoy]
+- Fixed slow /tags/index calls using the API. [iglocska]
+
+  - burned the stupid out of the API
+- Fixed the downloadSamples API. [iglocska]
+- Fixed silly lookup with injected event IDs on the export page for
+  normal users. [iglocska]
+
+  - broke instances with a few hundred k events
+- Fixed a reflected XSS in the sharing group creator tool. [iglocska]
+
+  - Fixed a reflected XSS in the sharing group editor that requires malicious organisation names
+
+  - Low impact due to the following requirements:
+    - organisation names with malicious org names (JS in the orgname)
+    - sharing group editor user has to manually add an organisation to the list that has javascript in the org name
+    - only vulnerable view is the editor itself, so the impact is limited to
+      users that manually add organisations with malicious names to the list themselves / edit such sharing groups
+
+  - As reported by Dawid Czarnecki
+
+Other
+~~~~~
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [Alexandre
+  Dulaunoy]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [iglocska]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [iglocska]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [iglocska]
+- Merge pull request #2706 from Rafiot/cortex_doc. [Raphaël Vinot]
+
+  fix: documentation to enable cortex services
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [iglocska]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Merge branch 'feature/tag_filter_rework' into 2.4. [iglocska]
+- Merge branch '2.4' of github.com:MISP/MISP into
+  feature/tag_filter_rework. [iglocska]
+- Merge branch '2.4' into feature/tag_filter_rework. [iglocska]
+- Merge branch '2.4' into feature/tag_filter_rework. [iglocska]
+- Little change about SDOs generated from Galaxy. [chrisr3d]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [Alexandre
+  Dulaunoy]
+- Add: a new set of logos for the MISP project. [Alexandre Dulaunoy]
+
+  There are 3 type of logos in the set:
+
+  - core software
+  - community
+  - standard
+
+  The objective is not to replace the existing the logo but
+  to provide a clear logo when this is referencing a specific
+  sub-part of the MISP project.
+- Fixed vulnerability type. [chrisr3d]
+
+  Was generated as custom object because of a change
+  in the attributes reading function
+- Fixed assignment issues for attributes from Object. [chrisr3d]
+
+  Multiple use of the same part of the dictionary caused
+  assignment errors. Using the 'copy()' method avoid that error.
+- Added mac-eui-64 type. [chrisr3d]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
+- Merge pull request #2701 from RichieB2B/ncsc-nl/stixfix. [Andras
+  Iklody]
+
+  Fix STIX export format
+- Use threat level name instead of id in STIX. [Richard van den Berg]
+- Use new MISP JSON format (no more AttributeTags) [Richard van den
+  Berg]
+- Merge pull request #2700 from Rafiot/testdescribe2. [Raphaël Vinot]
+
+  chg: bump PyMISP, again
+- Add: MISP distributed overview in SVG format. [Alexandre Dulaunoy]
+- Merge pull request #2697 from Rafiot/testdescribe. [Raphaël Vinot]
+
+  chg: bump PyMISP
+- Little fix with 'info' field in Events. [chrisr3d]
+- Added a label to separate SDOs from Objects. [chrisr3d]
+
+  This distinction will probably be helpful for the
+  Stix2 import module to separate Attributes from
+  Objects
+- Fixed issues with dictionary keys and some objects. [chrisr3d]
+- Added Org & Orgc information for the import. [chrisr3d]
+
+  Also clarified a little part of the code
+- Added xml files parsing. [chrisr3d]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Added mac-address type. [chrisr3d]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
+- Fixed issues about observable objects and patterns. [chrisr3d]
+- Parsing attachment attributes. [chrisr3d]
+
+  Also fixed some specific issues with single quotes
+- Wip: Import of some of the most common attributes. [chrisr3d]
+
+  Work still in progress in order to:
+  - Support as many attribute types as possible
+  - Fix simple quotes (that are not json parsable)
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [Alexandre
+  Dulaunoy]
+- Merge pull request #2672 from CenturyLinkCIRT/freetext-target-email.
+  [Andras Iklody]
+
+  added target-email to FreeText Import types
+- Added target-email to FreeText Import types. [Thomas Gardner]
+- Misp-object templates updated to latest version. [Alexandre Dulaunoy]
+- Merge branch '2.4' of github.com:MISP/MISP into stix2experiments.
+  [chrisr3d]
+- Merge pull request #2671 from milankowww/return-to-orig-url. [Andras
+  Iklody]
+
+  change behavior of login page to return to original page after authen…
+- Change behavior of login page to return to original page after
+  authentication. [Milan Pikula]
+- Merge pull request #2670 from milankowww/self-signed-certificate-
+  verification. [Andras Iklody]
+
+  fix: self signed cert verification
+- Merge pull request #2669 from milankowww/support-touch-screens.
+  [Andras Iklody]
+
+  fix: settings editor not working on touch devices
+- MISP objects updated to the latest version. [Alexandre Dulaunoy]
+- Merge pull request #2668 from JanSkalny/fix_settings_editor. [Andras
+  Iklody]
+
+  fix: refresh rows in settings editor
+- Merge branch '2.4' of github.com:MISP/MISP into stix2experiments.
+  [chrisr3d]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- New relationships added. [Alexandre Dulaunoy]
+- Starting to parse info for a stix import. [chrisr3d]
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [iglocska]
+- Merge pull request #2651 from ppanero/sso_org_fix. [Andras Iklody]
+
+  Added possibility to use always default org for new users
+- Added possibility to use always default org for new users. [Pablo
+  Panero]
+- Merge branch 'feature/stixunclutter' into 2.4. [iglocska]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Merge pull request #2295 from norpol/patch-1. [Andras Iklody]
+
+  Fix gpgv2+ key generation
+- Fix gpgv2+ key generation. [Phi|eas |ebada]
+
+  This resolves failing of gpgv2 key generation with the following error message:
+  ```
+  gpg: agent_genkey failed: Permission denied
+  Key generation failed: Permission denied
+  ```
+
+  # Explanation
+  gpgv2's `pinentry-curses` requires access to a current `tty`. If you `su` or `sudo` between users, your tty's permission will stay the same as the initial login user (see illustrating below). You could, in general, work around issues like this by:
+   - `old_perms=$(stat -c "%U:%G" $(tty)); chown "www-data:tty" "$(tty)" && { sudo -u www-data gpg --gen-key; chown "${old_perms}" "$(tty)"; }` (uncertain security implications and won't probably work)
+   - starting screen/tmux within the newuser and then running `gpg --gen-key`
+   - starting a script session
+
+  But first point can't really be recommended, latter two will fail because www-data login shell is `/usr/sbin/nologin`.
+
+  Just for illustrating the problem better for you:
+  ```
+  ssh alice@somehost:
+  stat -c "%U:%G $(tty)" $(tty)
+  alice:tty /dev/pts/1
+  su - root
+  stat -c "%U:%G $(tty)" $(tty)
+  alice:tty /dev/pts/1
+  `
+- Merge pull request #2640 from SHSauler/patch-4. [Alexandre Dulaunoy]
+- Merge branch '2.4' of https://github.com/MISP/MISP into 2.4.
+  [iglocska]
+- Added reg-key objects parsing for observed data. [chrisr3d]
+
+  Still not tested as registry-key objects seem to have an issue in MISP
+- Support email objects parsing into observed data. [chrisr3d]
+
+  Currently skipping display names in observed data email-addr objects
+- Merge pull request #2639 from truckydev/patch-4. [Alexandre Dulaunoy]
+
+  update args.sleep on typeError
+- Force int for --sleep. [truckydev]
+
+  ^^
+- Update args.sleep on typeError. [truckydev]
+
+  Convert string to int for time.sleep when sub.py use with -t
+- Merge pull request #2633 from dawid-czarnecki/patch-1. [Andras Iklody]
+
+  Download terms redirect fix
+- Download terms redirect fix. [dawid-czarnecki]
+
+  When server setting MISP.terms_download=true and MISP.terms_file exists under MISP/app/files/terms directory user wasn't able to download terms and conditions before accepting it.
+- Merge pull request #2632 from PaoloVecchi/2.4. [Alexandre Dulaunoy]
+
+  Create INSTALL.ubuntu1604.with.webmin.txt
+- Create INSTALL.ubuntu1604.with.webmin.txt. [Paolo Vecchi]
+
+  Some, maybe a friend, can't be asked to configure and manage all the services on an Ubuntu 16.04 so Webmin could be useful.
+  Tested with:
+  MISP 2.4.82
+  Webmin 1.860
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
+- Merge pull request #2630 from treyka/2.4. [Andras Iklody]
+
+  add cti-python-stix2 to .gitmodules
+- Add cti-python-stix2. [Trey Darley]
+- Merge pull request #2629 from treyka/2.4. [Andras Iklody]
+
+  typo fixen
+- Typo fixen. [Trey Darley]
+- Merge pull request #2628 from Delta-Sierra/2.4. [Andras Iklody]
+
+  display "Fetch this event" button function in Servers and Feeds preview index
+- Uppercase to be consistent. [Deborah Servili]
+- Display "Fetch this event" button function in Servers and Feeds
+  preview index. [Deborah Servili]
+- Some other object types supported in Observed Data. [chrisr3d]
+
+  Object types still not supported (not in 'objectsMapping'
+  dictionary, from misp2stix2_dictionaries module) are set
+  to a basic value until the next update, so they do not
+  generate errors in Stix2 functions
+- Merge branch '2.4' of github.com:MISP/MISP into 2.4. [chrisr3d]
 - Merge branch '2.4' of github.com:MISP/MISP into 2.4. [Alexandre
  Dulaunoy]
 - 2017 even if it's not 2049 ;-) [Alexandre Dulaunoy]
+- Quick fixes. [chrisr3d]


 v2.4.82 (2017-11-10)