Commit Graph

4155 Commits (f608e6c8cf48d8d5327c5ac04746f4f7f3992e31)

Author SHA1 Message Date
Eric Eastwood 9633eb2162
Allow non-member state sent in room batch to resolve for historic events (MSC2716) (#12329)
Part of https://github.com/matrix-org/synapse/issues/12110

Complement test: https://github.com/matrix-org/complement/pull/354

Previously, they didn't resolve because async `filter_events_for_client`
removes all outlier state except for out-of-band membership.

And fundamentally, we have the state at these events so they shouldn't be marked as outliers.
2022-04-04 14:54:50 -05:00
Erik Johnston 5c9e39e619
Track device list updates per room. (#12321)
This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.

There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
2022-04-04 15:25:20 +01:00
reivilibre c4cf916ed7
Default to `private` room visibility rather than `public` when a client does not specify one, according to spec. (#12350) 2022-04-01 15:55:09 +01:00
Richard van der Hoff 9b43df1f7b
Optimise `_get_state_after_missing_prev_event`: use `/state` (#12040)
If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
2022-04-01 12:53:42 +01:00
Brendan Abolivier e4409301ba
Add a module callback to react to account data changes (#12327)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-04-01 11:22:48 +02:00
Brendan Abolivier 5e88143dff
Add a callback to react to 3PID associations (#12302) 2022-03-31 18:27:21 +02:00
Patrick Cloke adbf975623
Remove an unnecessary class from the relations code. (#12338)
The PaginationChunk class attempted to bundle some properties
together, but really just caused callers to jump through hoops and
hid implementation details.
2022-03-31 07:13:49 -04:00
Patrick Cloke c31c1091d4
Remove the unused and unstable `/aggregations` endpoint. (#12293)
This endpoint was removed from MSC2675 before it was approved.
It is currently unspecified (even in any MSCs) and therefore subject to
removal. It is not implemented by any known clients.

This also changes the bundled aggregation format for `m.annotation`,
which previously included pagination tokens for the `/aggregations`
endpoint, which are no longer useful.
2022-03-30 11:45:32 -04:00
Andrew Morgan d8d0271977
Send device list updates to application services (MSC3202) - part 1 (#11881)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-30 14:39:27 +01:00
Andrew Morgan c8cbd66d3b
Start application service stream token tracking from 1 (#12193)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-03-30 11:10:12 +00:00
Richard van der Hoff 9b67715bc3
Disable proactive sends for remote joins (#12330)
Do not attempt to send remote joins out over federation. Normally, it will do
nothing; occasionally, it will do the wrong thing.
2022-03-30 12:04:35 +01:00
Brendan Abolivier 437a8ed9ef
Add a configuration to exclude rooms from sync response (#12310) 2022-03-30 09:43:04 +00:00
Eric Eastwood 6f2943714b
Remove unused `auth_event_ids` argument plumbing (#12304)
Follow-up to https://github.com/matrix-org/synapse/pull/12083

Since we are now using the new `state_event_ids` parameter to do all of the heavy lifting.
We can remove any spots where we plumbed `auth_event_ids` just for MSC2716 things in
https://github.com/matrix-org/synapse/pull/9247/files.

Removing `auth_event_ids` from following functions:

 - `create_and_send_nonmember_event`
 - `_local_membership_update`
 - `update_membership`
 - `update_membership_locked`
2022-03-29 09:18:52 +01:00
Eric Eastwood 14662d3c18
Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with MSC2716 instead of abusing `auth_event_ids` (#12083)
Spawned from https://github.com/matrix-org/synapse/pull/10975#discussion_r813183430

Part of [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716)
2022-03-25 09:21:06 -05:00
David Robertson fffb3c4c8f
Always allow the empty string as an avatar_url. (#12261)
Hopefully this fixes #12257.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-25 13:28:42 +00:00
Patrick Cloke 4df10d3214
Do not consider events by ignored users for relations (#12285)
Filter the events returned from `/relations` for the requester's ignored users
in a similar way to `/messages` (and `/sync`).
2022-03-24 10:25:42 -04:00
Brendan Abolivier 5436b014f4
Optionally include account validity in MSC3720 account status responses (#12266) 2022-03-24 11:19:41 +01:00
Richard van der Hoff afa17f0eab
Return a 404 from `/state` for an outlier (#12087)
* Replace `get_state_for_pdu` with  `get_state_ids_for_pdu` and `get_events_as_list`.
* Return a 404 from `/state` and `/state_ids` for an outlier
2022-03-21 11:23:32 +00:00
Patrick Cloke 8fe930c215
Move get_bundled_aggregations to relations handler. (#12237)
The get_bundled_aggregations code is fairly high-level and uses
a lot of store methods, we move it into the handler as that seems
like a better fit.
2022-03-18 17:49:32 +00:00
David Robertson 872dbb0181
Correct `check_username_for_spam` annotations and docs (#12246)
* Formally type the UserProfile in user searches
* export UserProfile in synapse.module_api
* Update docs

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-03-18 13:51:41 +00:00
Patrick Cloke fc9bd620ce
Add a relations handler to avoid duplication. (#12227)
Adds a handler layer between the REST and datastore layers for relations.
2022-03-16 10:39:15 -04:00
Patrick Cloke dda9b7fc4d
Use the ignored_users table to test event visibility & sync. (#12225)
Instead of fetching the raw account data and re-parsing it. The
ignored_users table is a denormalised version of the account data
for quick searching.
2022-03-15 14:06:05 -04:00
Sean Quah 605d161d7d
Add cancellation support to `ReadWriteLock` (#12120)
Also convert `ReadWriteLock` to use async context managers.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 18:49:07 +00:00
Brendan Abolivier e6a106fd5e
Implement a Jinja2 filter to extract localparts from email addresses (#12212) 2022-03-11 15:15:11 +00:00
reivilibre 4a53f35737
Improve code documentation for the typing stream over replication. (#12211) 2022-03-11 14:00:15 +00:00
Brendan Abolivier 003cc6910a
Update the SSO username picker template to comply with SIWA guidelines (#12210)
Fixes https://github.com/matrix-org/synapse/issues/12205
2022-03-11 13:20:00 +00:00
Patrick Cloke bc9dff1d95
Remove unnecessary pass statements. (#12206) 2022-03-11 07:06:21 -05:00
Patrick Cloke ea27528b5d
Support stable identifiers for MSC3440: Threading (#12151)
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00
Patrick Cloke 3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. (#12192)
Since the object it returns is a ReplicationCommandHandler.

This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
Will Hunt 15382b1afa
Add third_party module callbacks to check if a user can delete a room and deactivate a user (#12028)
* Add check_can_deactivate_user

* Add check_can_shutdown_rooms

* Documentation

* callbacks, not functions

* Various suggested tweaks

* Add tests for test_check_can_shutdown_room and test_check_can_deactivate_user

* Update check_can_deactivate_user to not take a Requester

* Fix check_can_shutdown_room docs

* Renegade and use `by_admin` instead of `admin_user_id`

* fix lint

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-03-09 18:23:57 +00:00
Richard van der Hoff dc8d825ef2
Skip attempt to get state at backwards-extremities (#12173)
We don't *have* the state at a backwards-extremity, so this is never going to
do anything useful.
2022-03-09 11:00:48 +00:00
Sean Quah 5627182788
Use `ParamSpec` in type hints for `synapse.logging.context` (#12150)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-08 15:58:14 +00:00
Patrick Cloke ca9234a9eb
Do not return allowed_room_ids from /hierarchy response. (#12175)
This field is only to be used in the Server-Server API, and not the
Client-Server API, but was being leaked when a federation response
was used in the /hierarchy API.
2022-03-08 08:09:11 -05:00
Erik Johnston 423cca9efe
Spread out sending device lists to remote hosts (#12132) 2022-03-04 11:48:15 +00:00
Richard van der Hoff 8533c8b03d
Avoid generating state groups for local out-of-band leaves (#12154)
If we locally generate a rejection for an invite received over federation, it
is stored as an outlier (because we probably don't have the state for the
room). However, currently we still generate a state group for it (even though
the state in that state group will be nonsense).

By setting the `outlier` param on `create_event`, we avoid the nonsensical
state.
2022-03-03 19:58:08 +00:00
Andrew Morgan fb0ffa9676
Rename various ApplicationServices interested methods (#11915) 2022-03-03 18:14:09 +00:00
Patrick Cloke 1d11b452b7
Use the proper serialization format when bundling aggregations. (#12090)
This ensures that the `latest_event` field of the bundled aggregation
for threads uses the same format as the other events in the response.
2022-03-03 10:43:06 -05:00
Erik Johnston 61fd2a8f59
Limit the size of the aggregation_key (#12101)
There's no reason to let people use long keys.
2022-03-03 10:52:35 +00:00
Patrick Cloke 1103c5fe8a
Check if instances are lists, not sequences. (#12128)
As a str is a sequence, the checks were not granular
enough and would allow lists or strings, when only
lists were valid.
2022-03-02 13:18:51 +00:00
Brendan Abolivier 300ed0b8a6
Add module callbacks called for reacting to deactivation status change and profile update (#12062) 2022-03-01 15:00:03 +00:00
Richard van der Hoff e2e1d90a5e
Faster joins: persist to database (#12012)
When we get a partial_state response from send_join, store information in the
database about it:
 * store a record about the room as a whole having partial state, and stash the
   list of member servers too.
 * flag the join event itself as having partial state
 * also, for any new events whose prev-events are partial-stated, note that
   they will *also* be partial-stated.

We don't yet make any attempt to interpret this data, so API calls (and a bunch
of other things) are just going to get incorrect data.
2022-03-01 12:49:54 +00:00
Richard van der Hoff 9d11fee8f2
Improve exception handling for concurrent execution (#12109)
* fix incorrect unwrapFirstError import

this was being imported from the wrong place

* Refactor `concurrently_execute` to use `yieldable_gather_results`

* Improve exception handling in `yieldable_gather_results`

Try to avoid swallowing so many stack traces.

* mark unwrapFirstError deprecated

* changelog
2022-03-01 09:34:30 +00:00
Patrick Cloke 7754af24ab
Remove the unstable `/spaces` endpoint. (#12073)
...and various code supporting it.

The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
2022-02-28 18:33:00 +00:00
Erik Johnston a711ae78a8
Add logging to `/sync` for debugging #11916 (#12068) 2022-02-23 14:22:22 +00:00
Richard van der Hoff 5b2b36809f
Remove more references to `get_datastore` (#12067)
These have snuck in since #12031 was started.

Also a couple of other cleanups while we're in the area.
2022-02-23 12:35:53 +00:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Brendan Abolivier 250104d357
Implement account status endpoints (MSC3720) (#12001)
See matrix-org/matrix-doc#3720

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
Richard van der Hoff 81364db49b
Run `_handle_queued_pdus` as a background process (#12041)
... to ensure it gets a proper log context, mostly.
2022-02-22 13:33:22 +00:00
Richard van der Hoff 3070af4809
remote join processing: get create event from state, not auth_chain (#12039)
A follow-up to #12005, in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
2022-02-21 19:27:35 +00:00
Richard van der Hoff a85dde3445
Minor typing fixes (#12034)
These started failing in
https://github.com/matrix-org/synapse/pull/12031... I'm a bit mystified by how
they ever worked.
2022-02-21 18:37:04 +00:00