4b17a5ace8
Handle remote device list updates during partial join ( #13913 )
...
c.f. #12993 (comment), point 3
This stores all device list updates that we receive while partial joins are ongoing, and processes them once we have the full state.
Note: We don't actually process the device lists in the same ways as if we weren't partially joined. Instead of updating the device list remote cache, we simply notify local users that a change in the remote user's devices has happened. I think this is safe as if the local user requests the keys for the remote user and we don't have them we'll simply fetch them as normal.
2022-09-28 13:42:43 +00:00
6caa303083
fix: Push notifications for invite over federation ( #13719 )
2022-09-28 12:31:53 +00:00
8ab16a92ed
Persist CreateRoom events to DB in a batch ( #13800 )
2022-09-28 10:11:48 +00:00
a2cf66a94d
Prepatory work for batching events to send ( #13487 )
...
This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
2022-09-28 10:39:03 +01:00
f5aaa55e27
Add new columns tracking when we partial-joined ( #13892 )
2022-09-27 17:26:35 +01:00
50c92f3a69
Carry IdP Session IDs through user-mapping sessions. ( #13839 )
...
Since #11482 , we're saving sessions IDs from upstream IdPs, but we've been losing them when the user goes through a user mapping session on account registration.
2022-09-27 14:38:14 +01:00
85e161631a
Faster room joins: Fix spurious error when joining a room ( #13872 )
...
During a `lazy_load_members` `/sync`, we look through auth events in
rooms with partial state to find prior membership events. When such a
membership is not found, an error is logged.
Since the first join event for a user never has a prior membership event
to cite, the error would always be logged when one appeared in the room
timeline.
Avoid logging errors for such events.
Introduced in #13477 .
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-27 11:17:23 +01:00
41461fd4d6
typing: check origin server of typing event against room's servers ( #13830 )
...
This is also using the partial state approximation if needed so we do
not block here during a fast join.
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-09-26 17:33:32 +02:00
ac1a31740b
Only try to backfill event if we haven't tried before recently ( #13635 )
...
Only try to backfill event if we haven't tried before recently (exponential backoff). No need to keep trying the same backfill point that fails over and over.
Fix https://github.com/matrix-org/synapse/issues/13622
Fix https://github.com/matrix-org/synapse/issues/8451
Follow-up to https://github.com/matrix-org/synapse/pull/13589
Part of https://github.com/matrix-org/synapse/issues/13356
2022-09-23 14:01:29 -05:00
f49f73c0da
Faster room joins: Avoid blocking `/keys/changes` ( #13888 )
...
Part of the work for #12993 .
Once #12993 is fully resolved, we expect `/keys/changes` to behave
sensibly when joined to a room with partial state.
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-23 17:55:15 +01:00
efd108b45d
Accept & store thread IDs for receipts (implement MSC3771). ( #13782 )
...
Updates the `/receipts` endpoint and receipt EDU handler to parse a
`thread_id` from the body and insert it in the database.
2022-09-23 14:33:28 +00:00
03c2bfb7f8
Send device list updates out to servers in partially joined rooms ( #13874 )
...
Use the provided list of servers in the room from the `/send_join`
response, since we will not know which users are in the room. This
isn't sufficient to ensure that all remote servers receive the right
device list updates, since the `/send_join` response may be inaccurate
or we may calculate the membership state of new users in the room
incorrectly.
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-23 13:44:03 +01:00
c06b2b7142
Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ( #13823 )
2022-09-23 11:47:16 +01:00
8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) ( #13799 )
...
Partial implementation of MSC3881
2022-09-21 14:39:01 +00:00
16e1a9d9a7
Correct documentation for map_user_attributes of OpenID Mapping Providers ( #13836 )
...
Co-authored-by: David Robertson <davidr@element.io>
2022-09-21 13:08:16 +00:00
85fc7ea1a1
Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ( #13843 )
...
Signed-off-by: Quentin Gliech <quenting@element.io>
2022-09-20 15:18:07 +02:00
42d261c32f
Port the push rule classes to Rust. ( #13768 )
2022-09-20 12:10:31 +01:00
d64e85197a
Remove error spam when users query the keys of departed remote users ( #13826 )
...
The error message introduced in #13749 has turned out to be very spammy.
Remove it for now.
2022-09-16 16:16:05 +01:00
140af0cdb6
Record any exception when processing a pulled event ( #13814 )
...
Part of https://github.com/matrix-org/synapse/issues/13700 and https://github.com/matrix-org/synapse/issues/13356
Follow-up to https://github.com/matrix-org/synapse/pull/13589
2022-09-15 14:40:49 -05:00
957e3d74fc
Keep track when we try and fail to process a pulled event ( #13589 )
...
We can follow-up this PR with:
1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622
1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623
1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700
Fix https://github.com/matrix-org/synapse/issues/13621
Part of https://github.com/matrix-org/synapse/issues/13356
Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5 )
2022-09-14 13:57:50 -05:00
6302753012
Deduplicate `is_server_notices_room`. ( #13780 )
2022-09-14 15:53:18 +00:00
c73774467e
Fix bug in device list caching when remote users leave rooms ( #13749 )
...
When a remote user leaves the last room shared with the homeserver, we
have to mark their device list as unsubscribed, otherwise we would hold
on to a stale device list in our cache. Crucially, the device list would
remain cached even after the remote user rejoined the room, which could
lead to E2EE failures until the next change to the remote user's device
list.
Fixes #13651 .
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-14 10:42:57 +01:00
69fa29700e
Re-type hint some collections in `/sync` code as read-only ( #13754 )
...
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08 20:13:39 +01:00
f799eac7ea
Add timestamp to user's consent ( #13741 )
...
Co-authored-by: reivilibre <olivier@librepush.net>
2022-09-08 15:41:48 +00:00
d3d9ca156e
Cancel the processing of key query requests when they time out. ( #13680 )
2022-09-07 12:03:32 +01:00
bb5b47b62a
Add Admin API to Fetch Messages Within a Particular Window ( #13672 )
...
This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
2022-09-07 10:54:44 +01:00
26bc26586b
Remove the unspecced room_id field in the /hierarchy response. ( #13506 )
...
This is a re-do of 57d334a13d#13365 ),
which was backed out in 12abd72497#13501 ).
The `room_id` field represented the parent space for each room
and was made redundant by changes in the API shape where the
`children_state` is now nested underneath each `room`.
The room ID of each child is in the `state_key` field and is still
available.
2022-09-06 15:28:44 -04:00
0e99f07952
Remove support for unstable private read receipts ( #13653 )
...
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-09-01 13:31:54 +01:00
84ddcd7bbf
Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token` ( #13241 )
...
Fixes #13206
Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
2022-08-31 12:10:25 +00:00
682dfcfc0d
Fix that user cannot `/forget` rooms after the last member has left ( #13546 )
2022-08-30 09:58:38 +00:00
51d732db3b
Optimize how we calculate `likely_domains` during backfill ( #13575 )
...
Optimize how we calculate `likely_domains` during backfill because I've seen this take 17s in production just to `get_current_state` which is used to `get_domains_from_state` (see case [*2. Loading tons of events* in the `/messages` investigation issue](https://github.com/matrix-org/synapse/issues/13356 )).
There are 3 ways we currently calculate hosts that are in the room:
1. `get_current_state` -> `get_domains_from_state`
- Used in `backfill` to calculate `likely_domains` and `/timestamp_to_event` because it was cargo-culted from `backfill`
- This one is being eliminated in favor of `get_current_hosts_in_room` in this PR 🕳
1. `get_current_hosts_in_room`
- Used for other federation things like sending read receipts and typing indicators
1. `get_hosts_in_room_at_events`
- Used when pushing out events over federation to other servers in the `_process_event_queue_loop`
Fix https://github.com/matrix-org/synapse/issues/13626
Part of https://github.com/matrix-org/synapse/issues/13356
Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh )
### Query performance
#### Before
The query from `get_current_state` sucks just because we have to get all 80k events. And we see almost the exact same performance locally trying to get all of these events (16s vs 17s):
```
synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
Time: 16035.612 ms (00:16.036)
synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
Time: 4243.237 ms (00:04.243)
```
But what about `get_current_hosts_in_room`: When there is 8M rows in the `current_state_events` table, the previous query in `get_current_hosts_in_room` took 13s from complete freshness (when the events were first added). But takes 930ms after a Postgres restart or 390ms if running back to back to back.
```sh
$ psql synapse
synapse=# \timing on
synapse=# SELECT COUNT(DISTINCT substring(state_key FROM '@[^:]*:(.*)$'))
FROM current_state_events
WHERE
type = 'm.room.member'
AND membership = 'join'
AND room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
count
-------
4130
(1 row)
Time: 13181.598 ms (00:13.182)
synapse=# SELECT COUNT(*) from current_state_events where room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
count
-------
80814
synapse=# SELECT COUNT(*) from current_state_events;
count
---------
8162847
synapse=# SELECT pg_size_pretty( pg_total_relation_size('current_state_events') );
pg_size_pretty
----------------
4702 MB
```
#### After
I'm not sure how long it takes from complete freshness as I only really get that opportunity once (maybe restarting computer but that's cumbersome) and it's not really relevant to normal operating times. Maybe you get closer to the fresh times the more access variability there is so that Postgres caches aren't as exact. Update: The longest I've seen this run for is 6.4s and 4.5s after a computer restart.
After a Postgres restart, it takes 330ms and running back to back takes 260ms.
```sh
$ psql synapse
synapse=# \timing on
Timing is on.
synapse=# SELECT
substring(c.state_key FROM '@[^:]*:(.*)$') as host
FROM current_state_events c
/* Get the depth of the event from the events table */
INNER JOIN events AS e USING (event_id)
WHERE
c.type = 'm.room.member'
AND c.membership = 'join'
AND c.room_id = '!OGEhHVWSdvArJzumhm:matrix.org'
GROUP BY host
ORDER BY min(e.depth) ASC;
Time: 333.800 ms
```
#### Going further
To improve things further we could add a `limit` parameter to `get_current_hosts_in_room`. Realistically, we don't need 4k domains to choose from because there is no way we're going to query that many before we a) probably get an answer or b) we give up.
Another thing we can do is optimize the query to use a index skip scan:
- https://wiki.postgresql.org/wiki/Loose_indexscan
- Index Skip Scan, https://commitfest.postgresql.org/37/1741/
- https://www.timescale.com/blog/how-we-made-distinct-queries-up-to-8000x-faster-on-postgresql/
2022-08-30 01:38:14 -05:00
967d7bad6c
Move the execution of the retention purge_jobs to the main worker ( #13632 )
...
Fixes #9927
Signed-off-by: Brad Murray brad@beeper.com
2022-08-26 08:38:10 +01:00
0bf180cbb4
Comment about a better future where we can get the state diff between two events ( #13586 )
...
Split off from https://github.com/matrix-org/synapse/pull/13561
Part of https://github.com/matrix-org/synapse/issues/13356
Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh )
2022-08-24 18:59:27 -05:00
1a209efdb2
Update `get_users_in_room` mis-use to get hosts with dedicated `get_current_hosts_in_room` ( #13605 )
...
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:15:37 -05:00
d58615c82c
Directly lookup local membership instead of getting all members in a room first (`get_users_in_room` mis-use) ( #13608 )
...
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:13:12 -05:00
c807b814ae
Use dedicated `get_local_users_in_room` to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request ( #13606 )
...
Use dedicated `get_local_users_in_room` to find local users when calculating `join_authorised_via_users_server` ("the authorising user for joining a restricted room") of a `/make_join` request.
Found while working on https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755 but it's not related.
2022-08-24 11:14:28 -05:00
ca3d19b05f
Synapse 1.66.0rc1 (2022-08-23)
...
==============================
This release removes the ability for homeservers to delegate email ownership
verification and password reset confirmation to identity servers. This removal
was originally planned for Synapse 1.64, but was later deferred until now.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Features
--------
- Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword ), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken ), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate ) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken ). ([\#13188](https://github.com/matrix-org/synapse/issues/13188 ), [\#13563](https://github.com/matrix-org/synapse/issues/13563 ))
- Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api ). ([\#13503](https://github.com/matrix-org/synapse/issues/13503 ))
- Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852 ). ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716 ). ([\#13551](https://github.com/matrix-org/synapse/issues/13551 ))
- Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537 ))
- Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522 ), [\#13547](https://github.com/matrix-org/synapse/issues/13547 ))
Bugfixes
--------
- Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514 ))
- Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html ) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525 ))
- Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566 ))
- Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid ) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload ) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574 ))
Improved Documentation
----------------------
- Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472 ))
- Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491 ))
- Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492 ))
- Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497 ))
- Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515 ))
- Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536 ))
- Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538 ))
Deprecations and Removals
-------------------------
- Remove the ability for homeservers to delegate email ownership verification
and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Internal Changes
----------------
- Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459 ))
- Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477 ))
- Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531 ))
- Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453 ))
- Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489 ))
- Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499 ), [\#13554](https://github.com/matrix-org/synapse/issues/13554 ))
- Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533 ))
- Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534 ), [\#13541](https://github.com/matrix-org/synapse/issues/13541 ))
- Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535 ), [\#13584](https://github.com/matrix-org/synapse/issues/13584 ))
- Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544 ))
- Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545 ))
- Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024 ))
- Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471 ))
- Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474 ))
- Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479 ), [\#13488](https://github.com/matrix-org/synapse/issues/13488 ))
- Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485 ))
- Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493 ))
- Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502 ))
- Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543 ))
- Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMExBwACgkQkD7OEIo5
3t0sGA/9FLZLdTWeROLUenbInvz3POqi6DgqlE0f8IDdp3zHjjuDJIE4HZwnPhhj
c3H5FcUnTPhQoVZ8Ybno3J3KN8jfpe4tFhnhlM1x9hOi1fujfqCnFd2qWvlzkYXX
J4oN6Dz0BbhqdhqnS2tcNacZJOielJ47UtNNBZkzprpzjL3qqRUK3Hm/gGK8GtGN
BLon7JK0pKG1GFMVaoIWsr+j7880QD8DGXTjoZrjo/yGdp+zJldwJTbN2ymNdMw1
3Caq7eCk+m6jzCUJxI7tTJVioXvf7+TScIsozAXuQtBy9W1RctvVMMoR9cIlN+JJ
dj5GKhKBv8uJpJU6dJnlYpjNC5+5hPHr1gfnkoYbcyFF6PybJrHx9jfhEIEXmI1+
j0DlJJdFnX+eeq4deFd+LPU5hYaF8SkSVRK3G+/NmKI3YYTReD6ootZzV5O4Z9P0
NUwLK/OSMZrsFUfuwysUttfHq0uHGGs7mytlRgeqb/1RKRC80nxVJu3qv/xF6kKj
KF98Yr2IDgp/MFhBpGAO3ylNgK8aQ3YxED4qW+kfnDeHPkma5mTdAtJ1bpvKcec+
3awpFGWcmYVQPflwp9J3HFJHqWkj32LMfuSgDffLs8mWuj87fP7eAdjzv6M4zeAM
4qh+cmvNC7ArfQ+ngz3g+xVD1lHi84s+fMgo7L+ySGfgDqTLhT4=
=1F3P
-----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMEz10ACgkQkD7OEIo5
3t02GhAApioIpEedKaeSATI42N8rWhfuni3vaPFAXwmOD6BvBkZlP0gcg+vUoBjP
I0VIkGWZ3MKIhCSJkjwPJh1vHcpE09KYqIYRKO5vSbqHZKHYUJ+ThjcCR4IkOBPD
3Dc3IRgQaM2Nh8eMcyRY7eyxo+2y+VN2G+SXr1B+A5/4N58ZrxfV97MtLYJ1a6AH
0iIknOxb9FPArYs4ORbk6x2rvRO2QPGh88uKwJ8uHXvHlUbJ3fmHLVvRa1m0RKAW
ojkzj/Dysa+4gasMIvtzPePtIQnMyovaQLul+bGXzE0js4lqzSOIP12x1zNpLuuq
LSiyD3gMgGeogumhCAU6b23xF6jBaGtgtKKpaYuK8pqkuBTRNN8kQ6LB+7g3WQ2L
QWVos+/+wKUEHoVVD/q3u7o4CrKYqv4zE9Pj6kw5e1USOoy7803ub2ccjY1UgG3M
mZ7c48WNQQ61o9oH2+9+Ct6bab3R/Lo1MbS3zIKN8uktB4CdrhVAz3kYgV5Bl8bo
PsldxUccOTTeS5qmUGvm2/szVKjdqyCWRx9AY9eUH9Kc4u/GzmNq0UVOZF8yxM0q
WEo7GFgjM3k9IIC3K/fSmSC1iBjPSHpYvqO+c8oSNnjq2iucZeIhiEB07e89+KSa
Wiz9DDWP8Z7W6D9wRAeXyPJFB6ZdSSBgP1I90z6lQjSMzy46Kv4=
=pJ2V
-----END PGP SIGNATURE-----
Merge tag 'v1.66.0rc1' into develop
Synapse 1.66.0rc1 (2022-08-23)
==============================
This release removes the ability for homeservers to delegate email ownership
verification and password reset confirmation to identity servers. This removal
was originally planned for Synapse 1.64, but was later deferred until now.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Features
--------
- Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword ), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken ), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate ) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken ). ([\#13188](https://github.com/matrix-org/synapse/issues/13188 ), [\#13563](https://github.com/matrix-org/synapse/issues/13563 ))
- Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api ). ([\#13503](https://github.com/matrix-org/synapse/issues/13503 ))
- Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852 ). ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716 ). ([\#13551](https://github.com/matrix-org/synapse/issues/13551 ))
- Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537 ))
- Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522 ), [\#13547](https://github.com/matrix-org/synapse/issues/13547 ))
Bugfixes
--------
- Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514 ))
- Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html ) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525 ))
- Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566 ))
- Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid ) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload ) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574 ))
Improved Documentation
----------------------
- Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472 ))
- Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491 ))
- Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492 ))
- Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497 ))
- Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515 ))
- Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536 ))
- Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538 ))
Deprecations and Removals
-------------------------
- Remove the ability for homeservers to delegate email ownership verification
and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Internal Changes
----------------
- Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459 ))
- Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477 ))
- Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531 ))
- Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453 ))
- Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489 ))
- Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499 ), [\#13554](https://github.com/matrix-org/synapse/issues/13554 ))
- Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533 ))
- Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534 ), [\#13541](https://github.com/matrix-org/synapse/issues/13541 ))
- Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535 ), [\#13584](https://github.com/matrix-org/synapse/issues/13584 ))
- Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544 ))
- Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545 ))
- Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024 ))
- Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471 ))
- Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474 ))
- Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479 ), [\#13488](https://github.com/matrix-org/synapse/issues/13488 ))
- Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485 ))
- Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493 ))
- Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502 ))
- Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543 ))
- Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558 ))
2022-08-23 14:00:09 +01:00
956e015413
Drop support for delegating email validation, round 2 ( #13596 )
2022-08-23 11:40:00 +00:00
5e7847dc92
Cache user IDs instead of profile objects ( #13573 )
...
The profile objects are never used and increase cache size significantly.
2022-08-23 09:49:59 +00:00
37f329c9ad
Fix that sending server notices fail if avatar is `None` ( #13566 )
...
Indroduced in #11846 .
2022-08-23 09:48:35 +01:00
9385c41ba4
Fix Prometheus metrics being negative (mixed up start/end) ( #13584 )
...
Fix:
- https://github.com/matrix-org/synapse/pull/13535#discussion_r949582508
- https://github.com/matrix-org/synapse/pull/13533#discussion_r949577244
2022-08-23 08:47:30 +01:00
3dd175b628
`synapse.api.auth.Auth` cleanup: make permission-related methods use `Requester` instead of the `UserID` ( #13024 )
...
Part of #13019
This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.
It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00
06df5d4250
MSC2716v4 room version - remove namespace from MSC2716 event content fields ( #13551 )
...
Complement PR: https://github.com/matrix-org/complement/pull/450
As suggested in
https://github.com/matrix-org/matrix-spec-proposals/pull/2716#discussion_r941444525
2022-08-19 15:37:01 -05:00
f9f03426de
Implement MSC3852: Expose `last_seen_user_agent` to users for their own devices; also expose to Admin API ( #13549 )
2022-08-19 16:17:10 +00:00
84169a82dc
Avoid blocking lazy-loading `/sync`s during partial joins ( #13477 )
...
Use a state filter or accept partial state in a few places where we
request state, to avoid blocking.
To make lazy-loading `/sync`s work, we need to provide the memberships
of event senders, which are not guaranteed to be in the room state.
Instead we dig through auth events for memberships to present to
clients. The auth events of an event are guaranteed to contain a
passable membership event, otherwise the event would have been rejected.
Note that this only covers the common code paths encountered during
testing. There has been no exhaustive checking of all sync code paths.
Fixes #13146 .
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-18 11:53:02 +01:00
088bcb7ecb
Time how long it takes us to do backfill processing ( #13535 )
2022-08-17 10:33:19 +01:00
0a4efbc1dd
Instrument the federation/backfill part of `/messages` ( #13489 )
...
Instrument the federation/backfill part of `/messages` so it's easier to follow what's going on in Jaeger when viewing a trace.
Split out from https://github.com/matrix-org/synapse/pull/13440
Follow-up from https://github.com/matrix-org/synapse/pull/13368
Part of https://github.com/matrix-org/synapse/issues/13356
2022-08-16 12:39:40 -05:00
c3516e9dec
Faster room joins: make `/joined_members` block whilst the room is partial stated. ( #13514 )
2022-08-16 13:16:56 +01:00
344a2f767c
Instrument `FederationStateIdsServlet` - `/state_ids` ( #13499 )
...
Instrument FederationStateIdsServlet - `/state_ids` so it's easier to follow what's going on in Jaeger when viewing a trace.
2022-08-15 19:41:23 +01:00
05bc059e0d
Merge branch 'release-v1.65' into develop
2022-08-11 13:19:12 +01:00
Erik Johnston