Commit Graph

12156 Commits (787d22ed6cbc8f78fed7e316d345e54b7c6a93da)

Author SHA1 Message Date
Matthew Hodgson 787d22ed6c
Only lazy load self-members on initial sync
Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync.  Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
2018-09-25 00:49:26 +01:00
Amber Brown fbe5ba25f6 Merge branch 'master' into develop 2018-09-25 03:10:01 +10:00
Amber Brown 5121ae97f5 Internal Changes
----------------
 
 - Fix incompatibility with older Twisted version in tests. Thanks
   @OlegGirko!
 ([\#3940](https://github.com/matrix-org/synapse/issues/3940))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlupGfUACgkQIwi0edOS
 ShH/rgf5AdLpXHVmWN1YiKuwwgiK9i8/41pqcj5ZiVn+UQjzUIBNc0ERNf2gfoRU
 iwECt3rDMatB2Xxd4VFNoTn6wFtuqNV4ke7jeghMU2KxAzmwX8F6b7qJWz9sywyO
 7cOtByqVAKN2v8qLDI+wcGuBkxkFHMF8iKBQApkDn1gqvvqHs/yk9FprhLJOYvaz
 iOvrgm4T3XLiMOqDmVH1byemynLQCagol/8rn63NnnDVaoHd9Cgar0qRMta2MYBL
 xXjOJ2HS35Y7ZuIPhRAmOVhs6DXmjBnEgfxeDV+GAF1Co8qEH5Jbfq/vu/2gpKKf
 TsA5HG3mDGBit67X9evU9SN5WFlcQw==
 =CZ+X
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.5.1'

Internal Changes
----------------

- Fix incompatibility with older Twisted version in tests. Thanks
  @OlegGirko!
([\#3940](https://github.com/matrix-org/synapse/issues/3940))
2018-09-25 03:09:30 +10:00
Amber Brown fc691ca97c changelog 2018-09-25 02:54:59 +10:00
Amber Brown 6b6cb32297 bump version 2018-09-25 02:54:34 +10:00
Amber Brown e37c221b97 changelog for 3940 2018-09-25 02:51:55 +10:00
Oleg Girko 7d3f639844 Fix compatibility issue with older Twisted in tests.
Older Twisted (18.4.0) returns TimeoutError instead of
ConnectingCancelledError when connection times out.
This change allows tests to be compatible with this behaviour.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2018-09-25 02:51:18 +10:00
Amber Brown 04eed80a73 Merge branch 'master' into develop 2018-09-24 23:42:25 +10:00
Amber Brown 829213523e Features
--------
 
 - Python 3.5 and 3.6 support is now in beta.
 ([\#3576](https://github.com/matrix-org/synapse/issues/3576))
 - Implement `event_format` filter param in `/sync`
 ([\#3790](https://github.com/matrix-org/synapse/issues/3790))
 - Add synapse_admin_mau:registered_reserved_users metric to expose
 number of real reaserved users
 ([\#3846](https://github.com/matrix-org/synapse/issues/3846))
 
 Bugfixes
 --------
 
 - Remove connection ID for replication prometheus metrics, as it creates
 a large number of new series.
 ([\#3788](https://github.com/matrix-org/synapse/issues/3788))
 - guest users should not be part of mau total
 ([\#3800](https://github.com/matrix-org/synapse/issues/3800))
 - Bump dependency on pyopenssl 16.x, to avoid incompatibility with
 recent Twisted.
 ([\#3804](https://github.com/matrix-org/synapse/issues/3804))
 - Fix existing room tags not coming down sync when joining a room
 ([\#3810](https://github.com/matrix-org/synapse/issues/3810))
 - Fix jwt import check
 ([\#3824](https://github.com/matrix-org/synapse/issues/3824))
 - fix VOIP crashes under Python 3 (#3821)
 ([\#3835](https://github.com/matrix-org/synapse/issues/3835))
 - Fix manhole so that it works with latest openssh clients
 ([\#3841](https://github.com/matrix-org/synapse/issues/3841))
 - Fix outbound requests occasionally wedging, which can result in
 federation breaking between servers.
 ([\#3845](https://github.com/matrix-org/synapse/issues/3845))
 - Show heroes if room name/canonical alias has been deleted
 ([\#3851](https://github.com/matrix-org/synapse/issues/3851))
 - Fix handling of redacted events from federation
 ([\#3859](https://github.com/matrix-org/synapse/issues/3859))
 -  ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
 - Mitigate outbound federation randomly becoming wedged
 ([\#3875](https://github.com/matrix-org/synapse/issues/3875))
 
 Internal Changes
 ----------------
 
 - CircleCI tests now run on the potential merge of a PR.
 ([\#3704](https://github.com/matrix-org/synapse/issues/3704))
 - http/ is now ported to Python 3.
 ([\#3771](https://github.com/matrix-org/synapse/issues/3771))
 - Improve human readable error messages for threepid
 registration/account update
 ([\#3789](https://github.com/matrix-org/synapse/issues/3789))
 - Make /sync slightly faster by avoiding needless copies
 ([\#3795](https://github.com/matrix-org/synapse/issues/3795))
 - handlers/ is now ported to Python 3.
 ([\#3803](https://github.com/matrix-org/synapse/issues/3803))
 - Limit the number of PDUs/EDUs per federation transaction
 ([\#3805](https://github.com/matrix-org/synapse/issues/3805))
 - Only start postgres instance for postgres tests on Travis CI
 ([\#3806](https://github.com/matrix-org/synapse/issues/3806))
 - tests/ is now ported to Python 3.
 ([\#3808](https://github.com/matrix-org/synapse/issues/3808))
 - crypto/ is now ported to Python 3.
 ([\#3822](https://github.com/matrix-org/synapse/issues/3822))
 - rest/ is now ported to Python 3.
 ([\#3823](https://github.com/matrix-org/synapse/issues/3823))
 - add some logging for the keyring queue
 ([\#3826](https://github.com/matrix-org/synapse/issues/3826))
 - speed up lazy loading by 2-3x
 ([\#3827](https://github.com/matrix-org/synapse/issues/3827))
 - Improved Dockerfile to remove build requirements after building
 reducing the image size.
 ([\#3834](https://github.com/matrix-org/synapse/issues/3834))
 - Disable lazy loading for incremental syncs for now
 ([\#3840](https://github.com/matrix-org/synapse/issues/3840))
 - federation/ is now ported to Python 3.
 ([\#3847](https://github.com/matrix-org/synapse/issues/3847))
 - Log when we retry outbound requests
 ([\#3853](https://github.com/matrix-org/synapse/issues/3853))
 - Removed some excess logging messages.
 ([\#3855](https://github.com/matrix-org/synapse/issues/3855))
 - Speed up purge history for rooms that have been previously purged
 ([\#3856](https://github.com/matrix-org/synapse/issues/3856))
 - Refactor some HTTP timeout code.
 ([\#3857](https://github.com/matrix-org/synapse/issues/3857))
 - Fix running merged builds on CircleCI
 ([\#3858](https://github.com/matrix-org/synapse/issues/3858))
 - Fix typo in replication stream exception.
 ([\#3860](https://github.com/matrix-org/synapse/issues/3860))
 - Add in flight real time metrics for Measure blocks
 ([\#3871](https://github.com/matrix-org/synapse/issues/3871))
 - Disable buffering and automatic retrying in treq requests to prevent
 timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
 - mention jemalloc in the README
 ([\#3877](https://github.com/matrix-org/synapse/issues/3877))
 - Remove unmaintained "nuke-room-from-db.sh" script
 ([\#3888](https://github.com/matrix-org/synapse/issues/3888))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAluo6WIACgkQIwi0edOS
 ShHFIwf/ZURGbih+6q4jy8voc4wUbgtuNGabmFLQeXGhIXIOYltvKOrB8wcCEi/4
 jODztPqewikZoI7NFA4M44GToQKacuIEOYK/M513l3t/mEpBQYDdvLGDccM+6XGc
 RcrWqQEfJYkiNyWNmLx5TJ4bYaNelrpfUyNbbONph0wLO7o6f7ANoGyqqVKm/7pD
 ufUwwBLqzd2U2vuvNCbQLSW0LkognP91vYytHMN5NTkoAR4BSYfDxQCShUSoNSlJ
 098CEYjXZaDcY9ypGA6wlflbNfl3NC2KcTajLOCHdKe2U+FbEMc/9Ou690GbDRJ9
 6L+Bonfwx0tsF04LHsteTdyTIOVp+Q==
 =Dv5r
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.5'

Features
--------

- Python 3.5 and 3.6 support is now in beta.
([\#3576](https://github.com/matrix-org/synapse/issues/3576))
- Implement `event_format` filter param in `/sync`
([\#3790](https://github.com/matrix-org/synapse/issues/3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](https://github.com/matrix-org/synapse/issues/3846))

Bugfixes
--------

- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](https://github.com/matrix-org/synapse/issues/3788))
- guest users should not be part of mau total
([\#3800](https://github.com/matrix-org/synapse/issues/3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](https://github.com/matrix-org/synapse/issues/3804))
- Fix existing room tags not coming down sync when joining a room
([\#3810](https://github.com/matrix-org/synapse/issues/3810))
- Fix jwt import check
([\#3824](https://github.com/matrix-org/synapse/issues/3824))
- fix VOIP crashes under Python 3 (#3821)
([\#3835](https://github.com/matrix-org/synapse/issues/3835))
- Fix manhole so that it works with latest openssh clients
([\#3841](https://github.com/matrix-org/synapse/issues/3841))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](https://github.com/matrix-org/synapse/issues/3845))
- Show heroes if room name/canonical alias has been deleted
([\#3851](https://github.com/matrix-org/synapse/issues/3851))
- Fix handling of redacted events from federation
([\#3859](https://github.com/matrix-org/synapse/issues/3859))
-  ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
- Mitigate outbound federation randomly becoming wedged
([\#3875](https://github.com/matrix-org/synapse/issues/3875))

Internal Changes
----------------

- CircleCI tests now run on the potential merge of a PR.
([\#3704](https://github.com/matrix-org/synapse/issues/3704))
- http/ is now ported to Python 3.
([\#3771](https://github.com/matrix-org/synapse/issues/3771))
- Improve human readable error messages for threepid
registration/account update
([\#3789](https://github.com/matrix-org/synapse/issues/3789))
- Make /sync slightly faster by avoiding needless copies
([\#3795](https://github.com/matrix-org/synapse/issues/3795))
- handlers/ is now ported to Python 3.
([\#3803](https://github.com/matrix-org/synapse/issues/3803))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](https://github.com/matrix-org/synapse/issues/3805))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](https://github.com/matrix-org/synapse/issues/3806))
- tests/ is now ported to Python 3.
([\#3808](https://github.com/matrix-org/synapse/issues/3808))
- crypto/ is now ported to Python 3.
([\#3822](https://github.com/matrix-org/synapse/issues/3822))
- rest/ is now ported to Python 3.
([\#3823](https://github.com/matrix-org/synapse/issues/3823))
- add some logging for the keyring queue
([\#3826](https://github.com/matrix-org/synapse/issues/3826))
- speed up lazy loading by 2-3x
([\#3827](https://github.com/matrix-org/synapse/issues/3827))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](https://github.com/matrix-org/synapse/issues/3834))
- Disable lazy loading for incremental syncs for now
([\#3840](https://github.com/matrix-org/synapse/issues/3840))
- federation/ is now ported to Python 3.
([\#3847](https://github.com/matrix-org/synapse/issues/3847))
- Log when we retry outbound requests
([\#3853](https://github.com/matrix-org/synapse/issues/3853))
- Removed some excess logging messages.
([\#3855](https://github.com/matrix-org/synapse/issues/3855))
- Speed up purge history for rooms that have been previously purged
([\#3856](https://github.com/matrix-org/synapse/issues/3856))
- Refactor some HTTP timeout code.
([\#3857](https://github.com/matrix-org/synapse/issues/3857))
- Fix running merged builds on CircleCI
([\#3858](https://github.com/matrix-org/synapse/issues/3858))
- Fix typo in replication stream exception.
([\#3860](https://github.com/matrix-org/synapse/issues/3860))
- Add in flight real time metrics for Measure blocks
([\#3871](https://github.com/matrix-org/synapse/issues/3871))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
- mention jemalloc in the README
([\#3877](https://github.com/matrix-org/synapse/issues/3877))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](https://github.com/matrix-org/synapse/issues/3888))
2018-09-24 23:41:35 +10:00
Amber Brown e3aa2c0b75 towncrier 2018-09-24 23:40:27 +10:00
Amber Brown e302f40e20 update version 2018-09-24 23:40:05 +10:00
Erik Johnston ad53a5497d
Merge pull request #3927 from matrix-org/erikj/handle_background_errors
Handle exceptions thrown by background tasks
2018-09-21 09:26:30 +01:00
Matthew Hodgson a2ddaa90f2
Always LL ourselves if we're in a room to simplify clients (#3916)
Should fix https://github.com/vector-im/riot-web/issues/7209
2018-09-20 21:21:54 +01:00
Erik Johnston 94ae1dea3c Add missing logger 2018-09-20 17:05:34 +01:00
Erik Johnston 168491c412 Newsfile 2018-09-20 16:16:52 +01:00
Erik Johnston 9ea408441f Handle exceptions thrown by background tasks
Fixes #3921
2018-09-20 16:15:21 +01:00
Neil Johnson 23b53b4ef8
Merge pull request #3868 from matrix-org/neilj/fix_room_invite_mail_links
Neilj/fix room invite mail links
2018-09-20 13:32:38 +01:00
Amber Brown 1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Erik Johnston 3fd68d533b
Merge pull request #3914 from matrix-org/erikj/remove_retry_cache
Remove get_destination_retry_timings cache
2018-09-20 10:54:49 +01:00
Amber Brown 741571cf22
Add a way to run tests in PostgreSQL in Docker (#3699) 2018-09-20 18:12:45 +10:00
Amber Brown aeca5a5ed5
Add a regression test for logging on failed connections (#3912) 2018-09-20 16:28:18 +10:00
Richard van der Hoff 642199570c
Improve the logging when handling a federation transaction (#3904)
Let's try to rationalise the logging that happens when we are processing an
incoming transaction, to make it easier to figure out what is going wrong when
they take ages. In particular:

- make everything start with a [room_id event_id] prefix
- make sure we log a warning when catching exceptions rather than just turning
  them into other, more cryptic, exceptions.
2018-09-19 17:28:18 +01:00
Erik Johnston bbab6ebfd9 Fix up changelog and remove spurious comment 2018-09-19 14:45:14 +01:00
Erik Johnston 392a54128c pep8 2018-09-19 14:37:49 +01:00
Erik Johnston 83ee5592a5 Newsfile 2018-09-19 14:22:59 +01:00
Erik Johnston b9158ac2bf Remove get_destination_retry_timings cache
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.

Fixes #3798
2018-09-19 14:22:57 +01:00
Erik Johnston cb016baa37
Merge pull request #3910 from matrix-org/erikj/update_timeout
Update to use new timeout function everywhere.
2018-09-19 11:45:11 +01:00
Erik Johnston 80d2d50f47 Fixup 2018-09-19 11:19:47 +01:00
Erik Johnston 9407bcf37a Replace custom DeferredTimeoutError with defer.TimeoutError 2018-09-19 11:07:29 +01:00
Erik Johnston 6c48aa0256 Run canceller first to allow it to generate correct error 2018-09-19 11:07:27 +01:00
Erik Johnston 6bbe3d5732 Newsfile 2018-09-19 10:43:13 +01:00
Erik Johnston a334e1cace Update to use new timeout function everywhere.
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842.

The new helper function doesn't suffer from this problem.
2018-09-19 10:39:40 +01:00
Richard van der Hoff 05b9937cd7 update changelog for #3909 2018-09-19 09:17:54 +01:00
Amber Brown 47c02e6332
Merge pull request #3909 from turt2live/travis/fix-logging-1
Fix matrixfederationclient.py logging: Destination is a string
2018-09-19 18:14:47 +10:00
Amber Brown 3f0d8e6b09
Remove documentation referencing Cygwin (#3873) 2018-09-19 18:14:30 +10:00
Amber Brown 3d6b24fb1b
Merge pull request #3907 from matrix-org/rav/set_sni_to_server_name
Set SNI to the server_name, not whatever was in the SRV record
2018-09-19 17:59:33 +10:00
Amber Brown f773ecbd61
Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout
Bump timeout on get_missing_events request
2018-09-19 17:57:48 +10:00
Travis Ralston 13b31a9baf Changelog 2018-09-18 15:30:25 -06:00
Travis Ralston 35aec19f0a Destination is a string 2018-09-18 15:29:30 -06:00
Richard van der Hoff 38ead946a9 Merge remote-tracking branch 'origin/develop' into neilj/fix_room_invite_mail_links 2018-09-18 19:02:45 +01:00
Richard van der Hoff a219ce8726
Use directory server for room joins (#3899)
When we do a join, always try the server we used for the alias lookup first.

Fixes #2418
2018-09-18 18:27:37 +01:00
Richard van der Hoff 31c15dcb80
Refactor matrixfederationclient to fix logging (#3906)
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
2018-09-18 18:17:15 +01:00
Amber Brown c600886d47
Merge pull request #3894 from matrix-org/hs/phone_home_py_version
Add python_version phone home stat
2018-09-19 02:40:04 +10:00
Richard van der Hoff edabc18938 changelog 2018-09-18 17:04:20 +01:00
Richard van der Hoff b3097396e7 Set SNI to the server_name, not whatever was in the SRV record
Fixes #3843
2018-09-18 17:01:12 +01:00
Richard van der Hoff 8565d9a9fe changelog 2018-09-18 15:05:26 +01:00
Richard van der Hoff 550007cb0e Bump timeout on get_missing_events request 2018-09-18 15:02:51 +01:00
Richard van der Hoff 286d6930b7
Merge pull request #3879 from matrix-org/matthew/fix-autojoin
don't ratelimit autojoins
2018-09-18 13:07:01 +01:00
Richard van der Hoff 892432c818
Merge pull request #3882 from SimmyD/max_upload_docker_var
Add variable for changing the max upload size in Docker container
2018-09-18 13:05:09 +01:00
Richard van der Hoff 1e09a1d48a
Merge pull request #3889 from matrix-org/rav/404_on_remove_unknown_alias
Return a 404 when deleting unknown room alias
2018-09-18 12:59:30 +01:00