Commit Graph

10854 Commits (53cc2cde1f609ec34a4ce6a7c678302c65ddfe53)

Author SHA1 Message Date
Amber Brown 53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown 071206304d cleanup pep8 errors 2018-05-22 16:54:22 -05:00
Amber Brown 4abeaedcf3 tests/metrics is gone now 2018-05-22 16:45:11 -05:00
Amber Brown 85ba83eb51 fixes 2018-05-22 16:28:23 -05:00
Amber Brown 228f1f584e fix the test failures 2018-05-22 15:02:38 -05:00
Amber Brown 07bb9bdae8 update gitignore to remove some files that got put on my machine 2018-05-22 10:57:28 -05:00
Amber Brown 8f5a688d42 cleanups, self-registration 2018-05-22 10:56:03 -05:00
Amber Brown a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Richard van der Hoff 08a14b32ae
Merge pull request #3262 from matrix-org/rav/has_already_consented
Add a 'has_consented' template var to consent forms
2018-05-22 15:35:10 +01:00
Richard van der Hoff 82c2a52987
Merge pull request #3263 from matrix-org/rav/fix_jinja_dep
Fix dependency on jinja2
2018-05-22 15:31:08 +01:00
Richard van der Hoff 7b36d06a69 Add a 'has_consented' template var to consent forms
fixes #3260
2018-05-22 14:58:34 +01:00
Richard van der Hoff 669400e22f Enable auto-escaping for the consent templates
... to reduce the risk of somebody introducing an html injection attack...
2018-05-22 14:58:34 +01:00
Richard van der Hoff b5b2d5d64b Fix dependency on jinja2
Delay the import of ConsentResource, so that we can get away without jinja2 if
people don't have the consent resource enabled.

Fixes #3259
2018-05-22 14:03:45 +01:00
Richard van der Hoff 3b2def6c7a
Merge pull request #3257 from matrix-org/rav/fonx_on_no_consent
Reject attempts to send event before privacy consent is given
2018-05-22 12:01:43 +01:00
Richard van der Hoff a5e2941aad Reject attempts to send event before privacy consent is given
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
2018-05-22 12:00:47 +01:00
Richard van der Hoff 8aeb529262
Merge pull request #3236 from matrix-org/rav/consent_notice
Send users a server notice about consent
2018-05-22 11:56:09 +01:00
Richard van der Hoff 8810685df9 Stub out ServerNoticesSender on the workers
... and have the sync endpoints call it directly rather than obsure indirection
via PresenceHandler
2018-05-22 11:54:51 +01:00
Richard van der Hoff d5dca9a04f Move consent config parsing into ConsentConfig
turns out we need to reuse this, so it's better in the config class.
2018-05-22 11:54:51 +01:00
Richard van der Hoff 9ea219c514 Send users a server notice about consent
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Richard van der Hoff d14d7b8fdc Rename 'version' param on user consent config
we're going to use it for the version we require too.
2018-05-22 11:54:51 +01:00
Erik Johnston 7cfa8a87a1
Merge pull request #3258 from matrix-org/erikj/fixup_logcontext_rusage
Fix logcontext resource usage tracking
2018-05-22 11:39:56 +01:00
Erik Johnston 7948ecf234 Comment 2018-05-22 11:39:43 +01:00
Erik Johnston 020377a550 Fix logcontext resource usage tracking 2018-05-22 11:16:07 +01:00
Erik Johnston 13a8dfba0d
Merge pull request #3252 from matrix-org/erikj/in_flight_requests
Add in flight request metrics
2018-05-22 09:54:30 +01:00
Erik Johnston c435b0b441 Don't store context 2018-05-22 09:34:26 +01:00
Erik Johnston fb2806b186 Move in_flight_requests_count to be a callback metric 2018-05-22 09:31:53 +01:00
Amber Brown fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Amber Brown c60e0d5e02 don't need the resource portion 2018-05-21 17:03:20 -05:00
Amber Brown 02c1d29133 look at the Prometheus metrics instead 2018-05-21 17:02:20 -05:00
Amber Brown f258deffcb remove old metrics libs 2018-05-21 17:01:15 -05:00
Richard van der Hoff 413482f578
Merge pull request #3255 from matrix-org/rav/fix_transactions
Stop the transaction cache caching failures
2018-05-21 17:58:08 +01:00
Neil Johnson 4aac88928f
Merge pull request #3251 from matrix-org/cohort_analytics
Tighter filtering for user_daily_visits
2018-05-21 16:42:18 +00:00
Richard van der Hoff 6e1cb54a05 Fix logcontext leak in HttpTransactionCache
ONE DAY I WILL PURGE THE WORLD OF THIS EVIL
2018-05-21 16:58:20 +01:00
Richard van der Hoff 6d6e7288fe Stop the transaction cache caching failures
The transaction cache has some code which tries to stop it caching failures,
but if the callback function failed straight away, then things would happen
backwards and we'd end up with the failure stuck in the cache.
2018-05-21 16:49:59 +01:00
Michael Kaye d689e0dba1
Merge pull request #3239 from ptman/develop
Add lxml to docker image for web previews
2018-05-21 16:31:18 +01:00
Erik Johnston dfa70adc33 Add in flight request metrics
This tracks CPU and DB usage while requests are in flight, rather than
when we write the response.
2018-05-21 16:23:06 +01:00
Neil Johnson 644aac5f73 Tighter filtering for user_daily_visits 2018-05-18 17:10:35 +01:00
Neil Johnson 08462620bf
Merge pull request #3241 from matrix-org/fix_user_visits_insertion
fix psql compatability bug
2018-05-18 15:01:01 +00:00
Neil Johnson ef466b3a13 fix psql compatability bug 2018-05-18 15:51:21 +01:00
Paul Tötterman 861f8a9b21 Add lxml to docker image for web previews
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2018-05-18 16:20:08 +03:00
Neil Johnson 2725223f08 Merge branch 'master' into develop 2018-05-18 14:07:50 +01:00
Neil Johnson ab5e888927 Changes in synapse v0.29.1 (2018-05-17)
==========================================
 Changes:
 
 * Update docker documentation (PR #3222)
 
 Changes in synapse v0.29.0 (2018-05-16)
 ===========================================
 Not changes since v0.29.0-rc1
 
 Changes in synapse v0.29.0-rc1 (2018-05-14)
 ===========================================
 
 Notable changes, a docker file for running Synapse (Thanks to @kaiyou!) and a
 closed spec bug in the Client Server API. Additionally further prep for Python 3
 migration.
 
 Potentially breaking change:
 
 * Make Client-Server API return 401 for invalid token (PR #3161).
 
   This changes the Client-server spec to return a 401 error code instead of 403
   when the access token is unrecognised. This is the behaviour required by the
   specification, but some clients may be relying on the old, incorrect
   behaviour.
 
   Thanks to @NotAFile for fixing this.
 
 Features:
 
 * Add a Dockerfile for synapse (PR #2846) Thanks to @kaiyou!
 
 Changes - General:
 
 * nuke-room-from-db.sh: added postgresql option and help (PR #2337) Thanks to @rubo77!
 * Part user from rooms on account deactivate (PR #3201)
 * Make 'unexpected logging context' into warnings (PR #3007)
 * Set Server header in SynapseRequest (PR #3208)
 * remove duplicates from groups tables (PR #3129)
 * Improve exception handling for background processes (PR #3138)
 * Add missing consumeErrors to improve exception handling (PR #3139)
 * reraise exceptions more carefully (PR #3142)
 * Remove redundant call to preserve_fn (PR #3143)
 * Trap exceptions thrown within run_in_background (PR #3144)
 
 Changes - Refactors:
 
 * Refactor /context to reuse pagination storage functions (PR #3193)
 * Refactor recent events func to use pagination func (PR #3195)
 * Refactor pagination DB API to return concrete type (PR #3196)
 * Refactor get_recent_events_for_room return type (PR #3198)
 * Refactor sync APIs to reuse pagination API (PR #3199)
 * Remove unused code path from member change DB func (PR #3200)
 * Refactor request handling wrappers (PR #3203)
 * transaction_id, destination defined twice (PR #3209) Thanks to @damir-manapov!
 * Refactor event storage to prepare for changes in state calculations (PR #3141)
 * Set Server header in SynapseRequest (PR #3208)
 * Use deferred.addTimeout instead of time_bound_deferred (PR #3127, #3178)
 * Use run_in_background in preference to preserve_fn (PR #3140)
 
 Changes - Python 3 migration:
 
 * Construct HMAC as bytes on py3 (PR #3156) Thanks to @NotAFile!
 * run config tests on py3 (PR #3159) Thanks to @NotAFile!
 * Open certificate files as bytes (PR #3084) Thanks to @NotAFile!
 * Open config file in non-bytes mode (PR #3085) Thanks to @NotAFile!
 * Make event properties raise AttributeError instead (PR #3102) Thanks to @NotAFile!
 * Use six.moves.urlparse (PR #3108) Thanks to @NotAFile!
 * Add py3 tests to tox with folders that work (PR #3145) Thanks to @NotAFile!
 * Don't yield in list comprehensions (PR #3150) Thanks to @NotAFile!
 * Move more xrange to six (PR #3151) Thanks to @NotAFile!
 * make imports local (PR #3152) Thanks to @NotAFile!
 * move httplib import to six (PR #3153) Thanks to @NotAFile!
 * Replace stringIO imports with six (PR #3154, #3168) Thanks to @NotAFile!
 * more bytes strings (PR #3155) Thanks to @NotAFile!
 
 Bug Fixes:
 
 * synapse fails to start under Twisted >= 18.4 (PR #3157)
 * Fix a class of logcontext leaks (PR #3170)
 * Fix a couple of logcontext leaks in unit tests (PR #3172)
 * Fix logcontext leak in media repo (PR #3174)
 * Escape label values in prometheus metrics (PR #3175, #3186)
 * Fix 'Unhandled Error' logs with Twisted 18.4 (PR #3182) Thanks to @Half-Shot!
 * Fix logcontext leaks in rate limiter (PR #3183)
 * notifications: Convert next_token to string according to the spec (PR #3190) Thanks to @mujx!
 * nuke-room-from-db.sh: fix deletion from search table (PR #3194) Thanks to @rubo77!
 * add guard for None on purge_history api (PR #3160) Thanks to @krombel!
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlr+zv4ACgkQCTxDAAxP
 S/TKxQ/+OMQQJzk7c398zX+wFav0tJO7wSNEtsQ0lkBJq9Yy1Geaz+cXjxoLgPOT
 hthXhLZLVilONx13pnqsJCWrxufiBzWxbP9e7vnvWhP+v1uZca6kp1Pw4fpjGfl6
 DHgCXc3E2/UVDzk0aSIBdufs39qjp/WaUdcT7QuL16uCVwfX0K79nfZtGG+l2KeQ
 fi0SVwK58uZDcSwTrAB39Q3YITZrJzW8Pz1CRkmvG5efWH8KaRtnC/3UZvDb9kBl
 5xGsEkzj7A826PdMXp3ZXPc9H682NYpSD37545XU0ZqddR+RFNofZj0rRH464R+v
 lGVjHYqiiKMeX7y5VF7jZkMsb1j917txP5bcSwS+r2JtymG+RSxBQzUHNiu5gCaC
 MclIY6LbFGT9zJEzTQYBzOYzYA9RvjvlfqbdLV4D5R1Deb5z2VsjRJW7gqpWTj7c
 VpqkF7FP14JqjojMmciHa5b02GC5DPQ/ZSbL8Gp76DKvErbltYEC8GEgFx3eRJLT
 Dru/xcUQcSLUrx4DCk9ViHzpd9RBOGwERmdWilFNW/4HzXb/SK8CjEpWPNSENo+9
 ks3lRSa4J39U0Y/03c3wZMTKQFAQdIcqvAI3bJrlARa5OotvKbN1If5yf2nn1ZmM
 +JaVuV7KLqNMO0Z/O6R5DyuX6j70WAprkddExwGsb7Ha9Uq8Jig=
 =Fpbo
 -----END PGP SIGNATURE-----

Merge tag 'v0.29.1'

Changes in synapse v0.29.1 (2018-05-17)
==========================================
Changes:

* Update docker documentation (PR #3222)

Changes in synapse v0.29.0 (2018-05-16)
===========================================
Not changes since v0.29.0-rc1

Changes in synapse v0.29.0-rc1 (2018-05-14)
===========================================

Notable changes, a docker file for running Synapse (Thanks to @kaiyou!) and a
closed spec bug in the Client Server API. Additionally further prep for Python 3
migration.

Potentially breaking change:

* Make Client-Server API return 401 for invalid token (PR #3161).

  This changes the Client-server spec to return a 401 error code instead of 403
  when the access token is unrecognised. This is the behaviour required by the
  specification, but some clients may be relying on the old, incorrect
  behaviour.

  Thanks to @NotAFile for fixing this.

Features:

* Add a Dockerfile for synapse (PR #2846) Thanks to @kaiyou!

Changes - General:

* nuke-room-from-db.sh: added postgresql option and help (PR #2337) Thanks to @rubo77!
* Part user from rooms on account deactivate (PR #3201)
* Make 'unexpected logging context' into warnings (PR #3007)
* Set Server header in SynapseRequest (PR #3208)
* remove duplicates from groups tables (PR #3129)
* Improve exception handling for background processes (PR #3138)
* Add missing consumeErrors to improve exception handling (PR #3139)
* reraise exceptions more carefully (PR #3142)
* Remove redundant call to preserve_fn (PR #3143)
* Trap exceptions thrown within run_in_background (PR #3144)

Changes - Refactors:

* Refactor /context to reuse pagination storage functions (PR #3193)
* Refactor recent events func to use pagination func (PR #3195)
* Refactor pagination DB API to return concrete type (PR #3196)
* Refactor get_recent_events_for_room return type (PR #3198)
* Refactor sync APIs to reuse pagination API (PR #3199)
* Remove unused code path from member change DB func (PR #3200)
* Refactor request handling wrappers (PR #3203)
* transaction_id, destination defined twice (PR #3209) Thanks to @damir-manapov!
* Refactor event storage to prepare for changes in state calculations (PR #3141)
* Set Server header in SynapseRequest (PR #3208)
* Use deferred.addTimeout instead of time_bound_deferred (PR #3127, #3178)
* Use run_in_background in preference to preserve_fn (PR #3140)

Changes - Python 3 migration:

* Construct HMAC as bytes on py3 (PR #3156) Thanks to @NotAFile!
* run config tests on py3 (PR #3159) Thanks to @NotAFile!
* Open certificate files as bytes (PR #3084) Thanks to @NotAFile!
* Open config file in non-bytes mode (PR #3085) Thanks to @NotAFile!
* Make event properties raise AttributeError instead (PR #3102) Thanks to @NotAFile!
* Use six.moves.urlparse (PR #3108) Thanks to @NotAFile!
* Add py3 tests to tox with folders that work (PR #3145) Thanks to @NotAFile!
* Don't yield in list comprehensions (PR #3150) Thanks to @NotAFile!
* Move more xrange to six (PR #3151) Thanks to @NotAFile!
* make imports local (PR #3152) Thanks to @NotAFile!
* move httplib import to six (PR #3153) Thanks to @NotAFile!
* Replace stringIO imports with six (PR #3154, #3168) Thanks to @NotAFile!
* more bytes strings (PR #3155) Thanks to @NotAFile!

Bug Fixes:

* synapse fails to start under Twisted >= 18.4 (PR #3157)
* Fix a class of logcontext leaks (PR #3170)
* Fix a couple of logcontext leaks in unit tests (PR #3172)
* Fix logcontext leak in media repo (PR #3174)
* Escape label values in prometheus metrics (PR #3175, #3186)
* Fix 'Unhandled Error' logs with Twisted 18.4 (PR #3182) Thanks to @Half-Shot!
* Fix logcontext leaks in rate limiter (PR #3183)
* notifications: Convert next_token to string according to the spec (PR #3190) Thanks to @mujx!
* nuke-room-from-db.sh: fix deletion from search table (PR #3194) Thanks to @rubo77!
* add guard for None on purge_history api (PR #3160) Thanks to @krombel!
2018-05-18 14:06:34 +01:00
Neil Johnson f3d9dca975 Merge branch 'release-v0.29.0' of https://github.com/matrix-org/synapse into release-v0.29.0 2018-05-18 13:55:19 +01:00
Richard van der Hoff 6d9dc67139
Merge pull request #3232 from matrix-org/rav/server_notices_room
Infrastructure for a server notices room
2018-05-18 11:28:04 +01:00
Richard van der Hoff ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred
Fix error in handling receipts
2018-05-18 11:23:11 +01:00
Richard van der Hoff 67af392712
Merge pull request #3233 from matrix-org/rav/remove_dead_code
Remove unused `update_external_syncs`
2018-05-18 11:22:43 +01:00
Richard van der Hoff 011e1f4010 Better docstrings 2018-05-18 11:22:12 +01:00
Richard van der Hoff 26305788fe Make sure we reject attempts to invite the notices user 2018-05-18 11:18:39 +01:00
Richard van der Hoff d10707c810 Replace inline docstrings with "Attributes" in class docstring 2018-05-18 11:00:55 +01:00