8e1febc6a1
Support underscores (in addition to hyphens) for charset detection. ( #10410 )
2021-07-27 17:29:42 +00:00
076deade02
allow specifying https:// proxy ( #10411 )
2021-07-27 17:31:06 +01:00
74d09a43d9
Always communicate device OTK counts to clients ( #10485 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-07-27 14:36:38 +01:00
b3a757eb3b
Support MSC2033: Device ID on whoami ( #9918 )
...
* Fix no-access-token bug in deactivation tests
* Support MSC2033: Device ID on whoami
* Test for appservices too
MSC: https://github.com/matrix-org/matrix-doc/pull/2033
The MSC has passed FCP, which means stable endpoints can be used.
2021-07-27 05:28:20 +00:00
228decfce1
Update the MSC3083 support to verify if joins are from an authorized server. ( #10254 )
2021-07-26 12:17:00 -04:00
89c4ca81bb
Add `creation_ts` to list users admin API ( #10448 )
...
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-22 16:05:16 +02:00
590cc4e888
Add type hints to additional servlet functions ( #10437 )
...
Improves type hints for:
* parse_{boolean,integer}
* parse_{boolean,integer}_from_args
* parse_json_{value,object}_from_request
And fixes any incorrect calls that resulted from unknown types.
2021-07-21 18:12:22 +00:00
b181dc402d
Synapse 1.39.0rc1 (2021-07-20)
...
==============================
The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390 ) for more information.
Features
--------
- Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884 ))
- The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298 ), [\#10305](https://github.com/matrix-org/synapse/issues/10305 ))
- Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332 ), [\#10427](https://github.com/matrix-org/synapse/issues/10427 ))
- Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360 ))
Bugfixes
--------
- Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433 ))
- Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250 ))
- Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276 ))
- Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317 ))
- Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343 ))
- Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344 ))
- Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355 ))
- Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359 ))
- Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367 ))
- Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400 ), [\#10414](https://github.com/matrix-org/synapse/issues/10414 ))
- Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404 ))
Improved Documentation
----------------------
- Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971 ))
- Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313 ))
- Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353 )), ([\#10337](https://github.com/matrix-org/synapse/issues/10337 ))
- Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368 ))
- Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370 )), ([\#10322](https://github.com/matrix-org/synapse/issues/10322 )), ([\#10399](https://github.com/matrix-org/synapse/issues/10399 ))
- Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395 ))
- Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396 ))
Deprecations and Removals
-------------------------
- Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721 ))
- The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390 ) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386 ))
Internal Changes
----------------
- Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289 ))
- Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315 ))
- Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316 ))
- Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324 ))
- Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345 ), [\#10357](https://github.com/matrix-org/synapse/issues/10357 ))
- Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347 ), [\#10348](https://github.com/matrix-org/synapse/issues/10348 ))
- Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349 ))
- Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350 ), [\#10380](https://github.com/matrix-org/synapse/issues/10380 ), [\#10381](https://github.com/matrix-org/synapse/issues/10381 ), [\#10382](https://github.com/matrix-org/synapse/issues/10382 ), [\#10418](https://github.com/matrix-org/synapse/issues/10418 ))
- Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383 ))
- Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385 ))
- When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391 ))
- Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393 ))
- Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398 ))
- Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421 ))
- Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmD20AQTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ah9EACjDbd67+fYT4CIBp1/gwRZm73qAinF
HtJQV3b4KjvF+KC3NTxyfuSo1r6UHSwVP7hYMcW4D7bueYaFVklG4Pb0L58pRdrr
nKpVmCTuzQ7KAP165Mea2ZJUihTGrghUV1NTtkgbL2UtBn/j0ymUzWC4dDu2I3V9
vSoa0QOwaBb/iE7RCfwkI/FWNdhJis5CRtzZg31OWqs7kf1FU6+iH7XEtyckkV8M
U5aryWPlVzt1c9ag7eLf4OeySyVjl9MqKDjrUd6utLM3bK7DkHSZV/L9wA/NDfwA
O6jwnSCsp5TuPjpBFvIOId5P7Ank0WFJTHZj2yzXzrP86gVmmzJi5tTbVDVAaOrw
Ept1zKZjNUNBrgB5nu5oSOZQXPrWfSQeQgOfNPkS6SN2fPEvV+koIuv2CBgttmmB
9/iFr0bairlU1LllvyO2Q9VdFb1V2dTFW5MKVW5VdzieeeXDBZYisHT9nr62CRkD
ne52Qf84IaLJhwbyFaTssBX7KrmMTfasPzSaAQiLe7i6xlzC6ysdHajrdWf/Vn5c
z4I22vaBjZ8FqPucfvC/AiAm5at5bXkVu0UTxvfqeHw6ZXan+ApBYwUVHg50Rx96
J0FkZNfDPZlFTOJpf+jiO/kx4my2pMuiTseoi3wBr21u2j07vRoeeLiGHMuICqlO
1h0ekjbYsXrvdQ==
=WSa3
-----END PGP SIGNATURE-----
Merge tag 'v1.39.0rc1' into develop
Synapse 1.39.0rc1 (2021-07-20)
==============================
The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390 ) for more information.
Features
--------
- Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884 ))
- The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298 ), [\#10305](https://github.com/matrix-org/synapse/issues/10305 ))
- Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332 ), [\#10427](https://github.com/matrix-org/synapse/issues/10427 ))
- Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360 ))
Bugfixes
--------
- Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433 ))
- Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250 ))
- Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276 ))
- Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317 ))
- Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343 ))
- Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344 ))
- Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355 ))
- Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359 ))
- Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367 ))
- Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400 ), [\#10414](https://github.com/matrix-org/synapse/issues/10414 ))
- Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404 ))
Improved Documentation
----------------------
- Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971 ))
- Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313 ))
- Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353 )), ([\#10337](https://github.com/matrix-org/synapse/issues/10337 ))
- Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368 ))
- Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370 )), ([\#10322](https://github.com/matrix-org/synapse/issues/10322 )), ([\#10399](https://github.com/matrix-org/synapse/issues/10399 ))
- Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395 ))
- Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396 ))
Deprecations and Removals
-------------------------
- Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721 ))
- The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390 ) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386 ))
Internal Changes
----------------
- Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289 ))
- Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315 ))
- Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316 ))
- Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324 ))
- Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345 ), [\#10357](https://github.com/matrix-org/synapse/issues/10357 ))
- Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347 ), [\#10348](https://github.com/matrix-org/synapse/issues/10348 ))
- Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349 ))
- Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350 ), [\#10380](https://github.com/matrix-org/synapse/issues/10380 ), [\#10381](https://github.com/matrix-org/synapse/issues/10381 ), [\#10382](https://github.com/matrix-org/synapse/issues/10382 ), [\#10418](https://github.com/matrix-org/synapse/issues/10418 ))
- Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383 ))
- Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385 ))
- When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391 ))
- Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393 ))
- Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398 ))
- Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421 ))
- Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430 ))
2021-07-20 16:47:44 +01:00
54389d5697
Fix dropping locks on shut down ( #10433 )
2021-07-20 14:24:25 +01:00
69226c1ab4
MSC3244 room capabilities implementation ( #10283 )
2021-07-20 12:59:23 +01:00
a743bf4694
Port the ThirdPartyEventRules module interface to the new generic interface ( #10386 )
...
Port the third-party event rules interface to the generic module interface introduced in v1.37.0
2021-07-20 12:39:46 +02:00
4e340412c0
Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric ( #10332 )
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-19 16:11:34 +01:00
36dc15412d
Add a module type for account validity ( #9884 )
...
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity .
To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16 18:11:53 +02:00
6a60068250
Add tests to characterise the current behaviour of R30 phone-home metrics ( #10315 )
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-15 13:51:27 +01:00
ac5c221208
Stagger send presence to remotes ( #10398 )
...
This is to help with performance, where trying to connect to thousands
of hosts at once can consume a lot of CPU (due to TLS etc).
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-07-15 11:52:56 +01:00
c7603af1d0
Allow providing credentials to `http_proxy` ( #10360 )
2021-07-15 10:37:08 +01:00
eb3beb8f12
Add type hints and comments to event auth code. ( #10393 )
2021-07-14 14:13:40 +01:00
2d16e69b4b
Show all joinable rooms in the spaces summary. ( #10298 )
...
Previously only world-readable rooms were shown. This means that
rooms which are public, knockable, or invite-only with a pending invitation,
are included in a space summary. It also applies the same logic to
the experimental room version from MSC3083 -- if a user has access
to the proper allowed rooms then it is shown in the spaces summary.
This change is made per MSC3173 allowing stripped state of a room to
be shown to any potential room joiner.
2021-07-13 08:59:27 -04:00
93729719b8
Use inline type hints in `tests/` ( #10350 )
...
This PR is tantamount to running:
python3.8 -m com2ann -v 6 tests/
(com2ann requires python 3.8 to run)
2021-07-13 11:52:58 +01:00
89cfc3dd98
[pyupgrade] `tests/` ( #10347 )
2021-07-13 11:43:15 +01:00
19d0401c56
Additional unit tests for spaces summary. ( #10305 )
2021-07-12 11:21:04 -04:00
f6767abc05
Remove functionality associated with unused historical stats tables ( #9721 )
...
Fixes #9602
2021-07-08 16:57:13 +01:00
47e28b4031
Ignore EDUs for rooms we're not in ( #10317 )
2021-07-06 14:31:13 +01:00
bcb0962a72
Fix deactivate a user if he does not have a profile ( #10252 )
2021-07-06 13:08:53 +01:00
7a5873277e
Add support for evicting cache entries based on last access time. ( #10205 )
2021-07-05 16:32:12 +01:00
8d609435c0
Move methods involving event authentication to EventAuthHandler. ( #10268 )
...
Instead of mixing them with user authentication methods.
2021-07-01 14:25:37 -04:00
6c02cca95f
Add SSO `external_ids` to Query User Account admin API ( #10261 )
...
Related to #10251
2021-07-01 11:26:24 +02:00
785bceef72
Merge branch 'release-v1.37' into develop
2021-06-29 20:25:47 +01:00
85d237eba7
Add a distributed lock ( #10269 )
...
This adds a simple best effort locking mechanism that works cross workers.
2021-06-29 19:15:47 +01:00
f55836929d
Do not recurse into non-spaces in the spaces summary. ( #10256 )
...
Previously m.child.room events in non-space rooms would be
treated as part of the room graph, but this is no longer
supported.
2021-06-29 12:00:04 -04:00
8165ba48b1
Return errors from `send_join` etc if the event is rejected ( #10243 )
...
Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
2021-06-24 16:00:08 +01:00
6e8fb42be7
Improve validation for `send_{join,leave,knock}` ( #10225 )
...
The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
2021-06-24 15:30:49 +01:00
bd4919fb72
MSC2918 Refresh tokens implementation ( #9450 )
...
This implements refresh tokens, as defined by MSC2918
This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235
The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.
Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-06-24 14:33:20 +01:00
763dba77ef
Synapse 1.37.0rc1 (2021-06-24)
...
==============================
This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface ) for more information on how to update to the new generic module interface.
This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html ) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/ )) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.
Features
--------
- Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403 ). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739 ), [\#9359](https://github.com/matrix-org/synapse/issues/9359 ), [\#10167](https://github.com/matrix-org/synapse/issues/10167 ), [\#10212](https://github.com/matrix-org/synapse/issues/10212 ), [\#10227](https://github.com/matrix-org/synapse/issues/10227 ))
- Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )). ([\#9247](https://github.com/matrix-org/synapse/issues/9247 ))
- Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062 ), [\#10206](https://github.com/matrix-org/synapse/issues/10206 ))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108 ))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191 ))
Bugfixes
--------
- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115 ))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154 ))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157 ), [\#10158](https://github.com/matrix-org/synapse/issues/10158 ))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175 ))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184 ))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195 ))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208 ))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221 ))
Improved Documentation
----------------------
- Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436 ))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122 ))
- Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180 ))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198 ))
Deprecations and Removals
-------------------------
- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface ) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062 ), [\#10210](https://github.com/matrix-org/synapse/issues/10210 ), [\#10238](https://github.com/matrix-org/synapse/issues/10238 ))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161 ))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html ) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/ )) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194 ))
Internal Changes
----------------
- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933 ))
- Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080 ))
- Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134 ), [\#10193](https://github.com/matrix-org/synapse/issues/10193 ))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143 ))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144 ))
- Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145 ))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148 ))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155 ))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156 ))
- Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160 ))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164 ))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168 ))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183 ))
- Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188 ))
- Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083 ) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189 ))
- Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190 ))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197 ))
- Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUW7UACgkQHgFcFF8Z
Fs1ZfQ//bSyzg9mB7F43JdVBH9cpGups3Ur6nmsqE608/i/OEtomx9kTZsoN/9AO
j6WoM1bhJ3j9hoE63w4DIPVhBrojtwmYWeFrRvpRVlwTzZPhfij1nIH9pPrNVwd6
/LE0uA1kW5sDZCTiOhrz84BBBz6knjBH4otl+MDDdrszYixoP3i3SrrZVJAEYyGz
Dpw2OudpoOtMdGyzFMr2ewf2mQm5UFEPs9MT0AqhQJMjyM1lJYzrq3Obv1oTdCJq
BdLx6kMhK8+MbZPrcqIMXgSMHLYeQYeJOrdTzuCozFo0DSblThpW//aa21+sZtPx
dnIE0PpjAsm2Qg1II3EUX+YycYpI5KPhMq0ydEmUD7vTuGgJCKqAV3FBRwMlF2VE
9SOr8Mjd1ip03dHz2vlg38OlrFJ52P0e1RDZwlDtWfeJjZGZ4W6VjMhT7H/Z5USg
2J1FRPTD8E9w8wBOkjVcl76Lj6AY1KU36/ZLoYUbtrt58XYfkD4OzZmAxdJrqbMg
na5T9bd9iM8vWditU+vncxU+c7Bkn55GFyDVMQYl8QtXzBvnbv8gQilGwJLkncZJ
MreQ3OIBPbLNcHml56nmmY+ab+AtrIxuFl0UcZu+LGW8+lN+UZ9G9q0NFO3HbAWN
/LuhCVpO42SLAuLlIhnO3mcHM5jzZodKIa6oXpcMS2jv0nylQWo=
=MtOo
-----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUXG8ACgkQHgFcFF8Z
Fs2buxAAg7X4mYKHwlWLn2n2W6bUKY24HAnjtzMrNW76xIHXBJJiJMnhQD3MSKVd
uiUdolc+Tt3drJYOJyurP8R3YI0OuEVbSL0Krs8f3l6jNOU0Gy7YQzc6XUHnA42J
rb6mU8gMdCusk9RHHr1r+sBefwzmxTk4naCkw2n5i+r4ZfNparJK7H9yXCPoedUN
VBZi9f09jaUIm1SiHgK24sW9XfE6ifjdVg+E/r599v2eHJxtCTHUasH9TCAFD8KL
g17NtAP95+SQWPGwUXJFZYcVee+pGbylXuwP0RXXmMvyOWVKloI0LE6phGyLQ5cH
rMRagIkvVLMdYagaX8Tf47hNLVCXRJpuryxMgY3+5fx0O7lCMXlC3mDfs9zze+Fj
KXmJtVy/vpd+zpjQw+HHPnL1dDzsKnDqLFXFCj9Sjv6A/ruckxwWofuh0z83dYtv
1Abp8k7gVTVGZqLGPo8gRoqI2jQhWDvmJd+crb05qp86SzCz1sCma2ujWQtv3c1t
6aQpfmGMhFPp+hS2lS8AcyxK4vq+GudtXUOgzdqCNvIs+eJAV5cq6uufLIy6c2XR
FAR7UowoAoDvqTdD7+i5nHEjbudjjN5KgPXlO3a9+gG5jOR3IsOLZ8FVN0ML6qIJ
wzl0wM9YIlxyleMlWb0l4CFNDKrGJVhWgOupruHS+D5jfZQsZ3g=
=AjRf
-----END PGP SIGNATURE-----
Merge tag 'v1.37.0rc1' into develop
Synapse 1.37.0rc1 (2021-06-24)
==============================
This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface ) for more information on how to update to the new generic module interface.
This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html ) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/ )) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.
Features
--------
- Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403 ). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739 ), [\#9359](https://github.com/matrix-org/synapse/issues/9359 ), [\#10167](https://github.com/matrix-org/synapse/issues/10167 ), [\#10212](https://github.com/matrix-org/synapse/issues/10212 ), [\#10227](https://github.com/matrix-org/synapse/issues/10227 ))
- Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )). ([\#9247](https://github.com/matrix-org/synapse/issues/9247 ))
- Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062 ), [\#10206](https://github.com/matrix-org/synapse/issues/10206 ))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108 ))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191 ))
Bugfixes
--------
- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115 ))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154 ))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157 ), [\#10158](https://github.com/matrix-org/synapse/issues/10158 ))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175 ))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184 ))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195 ))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208 ))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221 ))
Improved Documentation
----------------------
- Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436 ))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122 ))
- Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180 ))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198 ))
Deprecations and Removals
-------------------------
- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface ) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062 ), [\#10210](https://github.com/matrix-org/synapse/issues/10210 ), [\#10238](https://github.com/matrix-org/synapse/issues/10238 ))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161 ))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html ) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/ )) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194 ))
Internal Changes
----------------
- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933 ))
- Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080 ))
- Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134 ), [\#10193](https://github.com/matrix-org/synapse/issues/10193 ))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143 ))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144 ))
- Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145 ))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148 ))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155 ))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156 ))
- Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160 ))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164 ))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168 ))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183 ))
- Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188 ))
- Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083 ) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189 ))
- Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190 ))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197 ))
- Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199 ))
2021-06-24 11:20:28 +01:00
c955e37868
Fix wrapping of legacy check_registration_for_spam ( #10238 )
...
Fixes #10234
2021-06-23 17:22:08 +02:00
394673055d
Re-introduce "Leave out optional keys from /sync" change ( #10214 )
...
Required some fixes due to merge conflicts with #6739 , but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.
2021-06-23 15:57:41 +01:00
e19e3d452d
Improve the reliability of auto-joining remote rooms ( #10237 )
...
If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do.
Fixes #10233 (at least to some extent)
2021-06-23 16:14:52 +02:00
96f6293de5
Add endpoints for backfilling history (MSC2716) ( #9247 )
...
Work on https://github.com/matrix-org/matrix-doc/pull/2716
2021-06-22 10:02:53 +01:00
0bd968921c
Fix a missing await when in the spaces summary. ( #10208 )
...
This could cause a minor data leak if someone defined a non-restricted join rule
with an allow key or used a restricted join rule in an older room version, but this is
unlikely.
Additionally this starts adding unit tests to the spaces summary handler.
2021-06-18 18:41:33 +01:00
1b3e398bea
Standardise the module interface ( #10062 )
...
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
2021-06-18 12:15:52 +01:00
08c8469322
Remove support for ACME v1 ( #10194 )
...
Fixes #9778
ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27 ), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
fcf3c7032b
Ensure that we do not cache empty sync responses after a timeout ( #10158 )
...
Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
2021-06-17 16:23:11 +01:00
8070b893db
update black to 21.6b0 ( #10197 )
...
Reformat all files with the new version.
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2021-06-17 15:20:06 +01:00
9e5ab6dd58
Remove the experimental flag for knocking and use stable prefixes / endpoints. ( #10167 )
...
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
2021-06-15 07:45:14 -04:00
d7808a2dde
Extend `ResponseCache` to pass a context object into the callback ( #10157 )
...
This is the first of two PRs which seek to address #8518 . This first PR lays the groundwork by extending ResponseCache; a second PR (#10158 ) will update the SyncHandler to actually use it, and fix the bug.
The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
2021-06-14 10:26:09 +01:00
d936371b69
Implement knock feature ( #6739 )
...
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403
Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
11846dff8c
Limit the number of in-flight /keys/query requests from a single device. ( #10144 )
2021-06-09 07:05:32 -04:00
8942e23a69
Always update AS last_pos, even on no events ( #10107 )
...
Fixes #1834 .
`get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether.
Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list.
Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
2021-06-07 15:42:05 +01:00
5325f0308c
r0.6.1 support: /rooms/:roomId/aliases endpoint ( #9224 )
...
[MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432 ) added this endpoint originally but it has since been included in the spec for nearly a year.
This is progress towards https://github.com/matrix-org/synapse/issues/8334
2021-06-03 13:50:49 +01:00
0284d2a297
Add new admin APIs to remove media by media ID from quarantine. ( #10044 )
...
Related to: #6681 , #5956 , #10040
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-06-02 18:50:35 +01:00
fc3d2dc269
Rewrite the KeyRing ( #10035 )
2021-06-02 16:37:59 +01:00
b4b2fd2ece
add a cache to have_seen_event ( #9953 )
...
Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
2021-06-01 12:04:47 +01:00
8fb9af570f
Make reason and score optional for report_event ( #10077 )
...
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551
Signed-off-by: Callum Brown <callum@calcuode.com>
2021-05-27 18:42:23 +01:00
78b5102ae7
Fix up `BatchingQueue` ( #10078 )
...
Fixes #10068
2021-05-27 14:32:31 +01:00
fe5dad46b0
Remove redundant code to reload tls cert ( #10054 )
...
we don't need to reload the tls cert if we don't have any tls listeners.
Follow-up to #9280 .
2021-05-27 10:34:24 +01:00
224f2f949b
Combine `LruCache.invalidate` and `invalidate_many` ( #9973 )
...
* Make `invalidate` and `invalidate_many` do the same thing
... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.
* Kill off `invalidate_many`
* changelog
2021-05-27 10:33:56 +01:00
65e6c64d83
Add an admin API for unprotecting local media from quarantine ( #10040 )
...
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-05-26 11:19:47 +01:00
7adcb20fc0
Add missing type hints to synapse.util ( #9982 )
2021-05-24 15:32:01 -04:00
c0df6bae06
Remove `keylen` from `LruCache`. ( #9993 )
...
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.
The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).
Commits should be independently reviewable.
2021-05-24 14:02:01 +01:00
3e831f24ff
Don't hammer the database for destination retry timings every ~5mins ( #10036 )
2021-05-21 17:57:08 +01:00
21bd230831
Add a test for update_presence ( #10033 )
...
https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale.
The bug was fixed in #10014 , and this PR just adds a test that failed on the old code, and was used to initially verify the bug.
The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
2021-05-21 17:29:14 +01:00
7958eadcd1
Add a batching queue implementation. ( #10017 )
2021-05-21 11:20:51 +01:00
4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory ( #9823 )
2021-05-18 14:13:45 +01:00
5090f26b63
Minor `@cachedList` enhancements ( #9975 )
...
- use a tuple rather than a list for the iterable that is passed into the
wrapped function, for performance
- test that we can pass an iterable and that keys are correctly deduped.
2021-05-14 11:12:36 +01:00
a683028d81
Correctly ratelimit invites when creating a room ( #9968 )
...
* Correctly ratelimit invites when creating a room
Also allow ratelimiting for more than one action at a time.
2021-05-12 16:05:28 +02:00
7562d887e1
Change the format of access tokens away from macaroons ( #5588 )
2021-05-12 15:04:51 +01:00
63fb220e5f
Tests for to-device messages ( #9965 )
2021-05-11 18:01:11 +01:00
27c375f812
Sort child events according to MSC1772 for the spaces summary API. ( #9954 )
...
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.
This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
2021-05-11 12:57:39 -04:00
652a6b094d
Merge branch 'master' into develop
2021-05-11 14:15:30 +01:00
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
...
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io>
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
25f43faa70
Reorganise the database schema directories ( #9932 )
...
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.
It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
2021-05-07 10:22:05 +01:00
eba431c539
Revert "Leave out optional keys from /sync ( #9919 )" ( #9940 )
...
This reverts commit e9eb3549d3
2021-05-06 15:06:35 +01:00
37623e3382
Increase perf of handling presence when joining large rooms. ( #9916 )
2021-05-05 17:27:05 +01:00
e9eb3549d3
Leave out optional keys from /sync ( #9919 )
...
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2021-05-05 14:37:56 +01:00
0085dc5abc
Delete room endpoint ( #9889 )
...
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.
Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
2021-04-29 10:31:45 +01:00
e4ab8676b4
Fix tight loop handling presence replication. ( #9900 )
...
Only affects workers. Introduced in #9819 .
Fixes #9899 .
2021-04-28 14:42:50 +01:00
fe604a022a
Remove various bits of compatibility code for Python <3.6 ( #9879 )
...
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
2021-04-27 13:13:07 +01:00
3ff2251754
Improved validation for received requests ( #9817 )
...
* Simplify `start_listening` callpath
* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
84936e2264
Kill off `_PushHTTPChannel`. ( #9878 )
...
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.
Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013 , the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
2021-04-23 18:40:57 +01:00
59d24c5bef
pass a reactor into SynapseSite ( #9874 )
2021-04-23 17:06:47 +01:00
ceaa76970f
Remove room and user invite ratelimits in default unit test config ( #9871 )
2021-04-23 13:37:48 +01:00
9d25a0ae65
Split presence out of master ( #9820 )
2021-04-23 12:21:55 +01:00
51a20914a8
Limit the size of HTTP responses read over federation. ( #9833 )
2021-04-23 11:08:41 +01:00
177dae2704
Limit length of accepted email addresses ( #9855 )
2021-04-22 17:49:11 +01:00
683d6f75af
Rename handler and config modules which end in handler/config. ( #9816 )
2021-04-20 14:55:20 -04:00
6982db9651
Merge branch 'master' into develop
2021-04-20 14:55:16 +01:00
b076bc276e
Always use the name as the log ID. ( #9829 )
...
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.
This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
de0d088adc
Add presence federation stream ( #9819 )
2021-04-20 14:11:24 +01:00
495b214f4f
Fix (final) Bugbear violations ( #9838 )
2021-04-20 11:50:49 +01:00
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic ( #9832 )
...
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
5a153772c1
remove `HomeServer.get_config` ( #9815 )
...
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
2021-04-14 19:09:08 +01:00
936e69825a
Separate creating an event context from persisting it in the federation handler ( #9800 )
...
This refactoring allows adding logic that uses the event context
before persisting it.
2021-04-14 12:35:28 -04:00
e8816c6ace
Revert "Check for space membership during a remote join of a restricted room. ( #9763 )"
...
This reverts commit cc51aaaa7a
2021-04-14 12:33:37 -04:00
cc51aaaa7a
Check for space membership during a remote join of a restricted room. ( #9763 )
...
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
This only applies to an experimental room version, as defined in MSC3083.
2021-04-14 12:32:20 -04:00
00a6db9676
Move some replication processing out of generic_worker ( #9796 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
1d5f0e3529
Bump black configuration to target py36 ( #9781 )
...
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
1fc97ee876
Add an admin API to manage ratelimit for a specific user ( #9648 )
2021-04-13 10:26:37 +01:00
e300ef64b1
Require AppserviceRegistrationType ( #9548 )
...
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-12 15:13:55 +01:00
0b3112123d
Use mock from the stdlib. ( #9772 )
2021-04-09 13:44:38 -04:00
48a1f4db31
Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` ( #9401 )
...
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)
`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation
API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-04-09 09:44:40 +01:00
2ca4e349e9
Bugbear: Add Mutable Parameter fixes ( #9682 )
...
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
9e167d9c53
Merge remote-tracking branch 'origin/develop' into rav/drop_py35
2021-04-08 18:30:38 +01:00
24c58ebfc9
remove unused param on `make_tuple_comparison_clause`
2021-04-08 18:29:57 +01:00
3ada9b4264
Drop support for sqlite<3.22 as well
2021-04-08 16:42:32 +01:00
48d44ab142
Record more information into structured logs. ( #9654 )
...
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
04819239ba
Add a Synapse Module for configuring presence update routing ( #9491 )
...
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.
This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.
A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync.
The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:
* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
024f121b74
Fix reported bugbear: too broad exception assertion ( #9753 )
2021-04-06 13:48:22 +01:00
e7b769aea1
Convert storage test cases to HomeserverTestCase. ( #9736 )
2021-04-06 07:21:02 -04:00
e2b8a90897
Update mypy configuration: `no_implicit_optional = True` ( #9742 )
2021-04-05 09:10:18 -04:00
bb0fe02a52
Add `order_by` to list user admin API ( #9691 )
2021-04-01 11:28:53 +01:00
35c5ef2d24
Add an experimental room version to support restricted join rules. ( #9717 )
...
Per MSC3083.
2021-03-31 16:39:08 -04:00
f02663c4dd
Replace `room_invite_state_types` with `room_prejoin_state` ( #9700 )
...
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
2021-03-30 12:12:44 +01:00
963f4309fe
Make RateLimiter class check for ratelimit overrides ( #9711 )
...
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.
We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.
Fixes #9663
2021-03-30 12:06:09 +01:00
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. ( #9442 )
2021-03-29 12:15:33 -04:00
5fdff97719
Fix CI by ignore type for None module import ( #9709 )
2021-03-29 14:42:38 +01:00
b5efcb577e
Make it possible to use dmypy ( #9692 )
...
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.
`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
4c3827f2c1
Enable addtional flake8-bugbear linting checks. ( #9659 )
2021-03-24 09:34:30 -04:00
5b268997bd
Allow providing credentials to HTTPS_PROXY ( #9657 )
...
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70
This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization )) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.
Also added some type hints.
2021-03-22 17:20:47 +00:00
592d6305fd
Merge branch 'develop' into babolivier/msc3026
2021-03-19 16:12:40 +01:00
8dd2ea65a9
Consistently check whether a password may be set for a user. ( #9636 )
2021-03-18 12:54:08 -04:00
dd71eb0f8a
Make federation catchup send last event from any server. ( #9640 )
...
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.
Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
2021-03-18 15:52:26 +00:00
405aeb0b2c
Implement MSC3026: busy presence state
2021-03-18 16:34:47 +01:00
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event ( #9585 )
...
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)
2021-03-17 16:51:55 +00:00
27d2820c33
Enable flake8-bugbear, but disable most checks. ( #9499 )
...
* Adds B00 to ignored checks.
* Fixes remaining issues.
2021-03-16 14:19:27 -04:00
dd5e5dc1d6
Add SSO attribute requirements for OIDC providers ( #9609 )
...
Allows limiting who can login using OIDC via the claims
made from the IdP.
2021-03-16 11:46:07 -04:00
8000cf1315
Return m.change_password.enabled=false if local database is disabled ( #9588 )
...
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
2021-03-16 11:44:25 -04:00
847ecdd8fa
Pass SSO IdP information to spam checker's registration function ( #9626 )
...
Fixes https://github.com/matrix-org/synapse/issues/9572
When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.
This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16 12:41:41 +00:00
dd69110d95
Add support for stable MSC2858 API ( #9617 )
...
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. ( #9608 )
2021-03-15 11:14:39 -04:00
026503fa3b
Don't go into federation catch up mode so easily ( #9561 )
...
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492 .
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
2021-03-15 14:42:40 +00:00
55da8df078
Fix additional type hints from Twisted 21.2.0. ( #9591 )
2021-03-12 11:37:57 -05:00
e55bd0e110
Add tests for blacklisting reactor/agent. ( #9563 )
2021-03-11 09:15:22 -05:00
2a99cc6524
Use the chain cover index in get_auth_chain_ids. ( #9576 )
...
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
2021-03-10 09:57:59 -05:00
eaada74075
JWT OIDC secrets for Sign in with Apple ( #9549 )
...
Apple had to be special. They want a client secret which is generated from an EC key.
Fixes #9220 . Also fixes #9212 while I'm here.
2021-03-09 15:03:37 +00:00
7fdc6cefb3
Fix additional type hints. ( #9543 )
...
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09 07:41:32 -05:00
075c16b410
Handle image transparency better when thumbnailing. ( #9473 )
...
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
2021-03-09 07:37:09 -05:00
d6196efafc
Add ResponseCache tests. ( #9458 )
2021-03-08 14:00:07 -05:00
cb7fc7523e
Add a basic test for purging rooms. ( #9541 )
...
Unfortunately this doesn't test re-joining the room since
that requires having another homeserver to query over
federation, which isn't easily doable in unit tests.
2021-03-08 09:21:36 -05:00
7eb6e39a8f
Record the SSO Auth Provider in the login token ( #9510 )
...
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
2021-03-04 14:44:22 +00:00
33a02f0f52
Fix additional type hints from Twisted upgrade. ( #9518 )
2021-03-03 15:47:38 -05:00
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. ( #9497 )
...
By consuming the response if the headers imply that the
content is too large.
2021-03-01 12:45:00 -05:00
ddb240293a
Add support for no_proxy and case insensitive env variables ( #9372 )
...
### Changes proposed in this PR
- Add support for the `no_proxy` and `NO_PROXY` environment variables
- Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519)bdb941be42/Lib/urllib/request.py (L2488)#9306 which also uses `ProxyAgent`
Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
15090de850
SSO: redirect to public URL before setting cookies ( #9436 )
...
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
2566dc57ce
Test that we require validated email for email pushers ( #9496 )
2021-02-25 15:35:14 +00:00
2927921942
Clean up `ShardedWorkerHandlingConfig` ( #9466 )
...
* Split ShardedWorkerHandlingConfig
This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).
* Remove special cases in ShardedWorkerHandlingConfig.
`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.
To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00
1b2d6d55c5
Remove vestiges of uploads_path config ( #9462 )
...
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
2021-02-22 19:54:49 +00:00
71c9f8de6d
Add an `order_by` field to list users' media admin API. ( #8978 )
2021-02-22 14:38:51 -05:00
0a363f9ca4
Remove cache for get_shared_rooms_for_users ( #9416 )
...
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666 )). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.
The cache was added in https://github.com/matrix-org/synapse/pull/7785 , but we forgot to ensure it was invalidated appropriately.
Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
2021-02-22 16:52:45 +00:00
179c0953ff
Regenerate exact thumbnails if missing ( #9438 )
2021-02-19 17:09:57 +00:00
sri-vidyut