605d161d7d
Add cancellation support to `ReadWriteLock` ( #12120 )
...
Also convert `ReadWriteLock` to use async context managers.
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 18:49:07 +00:00
e6a106fd5e
Implement a Jinja2 filter to extract localparts from email addresses ( #12212 )
2022-03-11 15:15:11 +00:00
4a53f35737
Improve code documentation for the typing stream over replication. ( #12211 )
2022-03-11 14:00:15 +00:00
003cc6910a
Update the SSO username picker template to comply with SIWA guidelines ( #12210 )
...
Fixes https://github.com/matrix-org/synapse/issues/12205
2022-03-11 13:20:00 +00:00
bc9dff1d95
Remove unnecessary pass statements. ( #12206 )
2022-03-11 07:06:21 -05:00
ea27528b5d
Support stable identifiers for MSC3440: Threading ( #12151 )
...
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00
3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. ( #12192 )
...
Since the object it returns is a ReplicationCommandHandler.
This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
15382b1afa
Add third_party module callbacks to check if a user can delete a room and deactivate a user ( #12028 )
...
* Add check_can_deactivate_user
* Add check_can_shutdown_rooms
* Documentation
* callbacks, not functions
* Various suggested tweaks
* Add tests for test_check_can_shutdown_room and test_check_can_deactivate_user
* Update check_can_deactivate_user to not take a Requester
* Fix check_can_shutdown_room docs
* Renegade and use `by_admin` instead of `admin_user_id`
* fix lint
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-03-09 18:23:57 +00:00
dc8d825ef2
Skip attempt to get state at backwards-extremities ( #12173 )
...
We don't *have* the state at a backwards-extremity, so this is never going to
do anything useful.
2022-03-09 11:00:48 +00:00
5627182788
Use `ParamSpec` in type hints for `synapse.logging.context` ( #12150 )
...
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-08 15:58:14 +00:00
ca9234a9eb
Do not return allowed_room_ids from /hierarchy response. ( #12175 )
...
This field is only to be used in the Server-Server API, and not the
Client-Server API, but was being leaked when a federation response
was used in the /hierarchy API.
2022-03-08 08:09:11 -05:00
423cca9efe
Spread out sending device lists to remote hosts ( #12132 )
2022-03-04 11:48:15 +00:00
8533c8b03d
Avoid generating state groups for local out-of-band leaves ( #12154 )
...
If we locally generate a rejection for an invite received over federation, it
is stored as an outlier (because we probably don't have the state for the
room). However, currently we still generate a state group for it (even though
the state in that state group will be nonsense).
By setting the `outlier` param on `create_event`, we avoid the nonsensical
state.
2022-03-03 19:58:08 +00:00
fb0ffa9676
Rename various ApplicationServices interested methods ( #11915 )
2022-03-03 18:14:09 +00:00
1d11b452b7
Use the proper serialization format when bundling aggregations. ( #12090 )
...
This ensures that the `latest_event` field of the bundled aggregation
for threads uses the same format as the other events in the response.
2022-03-03 10:43:06 -05:00
61fd2a8f59
Limit the size of the aggregation_key ( #12101 )
...
There's no reason to let people use long keys.
2022-03-03 10:52:35 +00:00
1103c5fe8a
Check if instances are lists, not sequences. ( #12128 )
...
As a str is a sequence, the checks were not granular
enough and would allow lists or strings, when only
lists were valid.
2022-03-02 13:18:51 +00:00
300ed0b8a6
Add module callbacks called for reacting to deactivation status change and profile update ( #12062 )
2022-03-01 15:00:03 +00:00
e2e1d90a5e
Faster joins: persist to database ( #12012 )
...
When we get a partial_state response from send_join, store information in the
database about it:
* store a record about the room as a whole having partial state, and stash the
list of member servers too.
* flag the join event itself as having partial state
* also, for any new events whose prev-events are partial-stated, note that
they will *also* be partial-stated.
We don't yet make any attempt to interpret this data, so API calls (and a bunch
of other things) are just going to get incorrect data.
2022-03-01 12:49:54 +00:00
9d11fee8f2
Improve exception handling for concurrent execution ( #12109 )
...
* fix incorrect unwrapFirstError import
this was being imported from the wrong place
* Refactor `concurrently_execute` to use `yieldable_gather_results`
* Improve exception handling in `yieldable_gather_results`
Try to avoid swallowing so many stack traces.
* mark unwrapFirstError deprecated
* changelog
2022-03-01 09:34:30 +00:00
7754af24ab
Remove the unstable `/spaces` endpoint. ( #12073 )
...
...and various code supporting it.
The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
2022-02-28 18:33:00 +00:00
a711ae78a8
Add logging to `/sync` for debugging #11916 ( #12068 )
2022-02-23 14:22:22 +00:00
5b2b36809f
Remove more references to `get_datastore` ( #12067 )
...
These have snuck in since #12031 was started.
Also a couple of other cleanups while we're in the area.
2022-02-23 12:35:53 +00:00
e24ff8ebe3
Remove `HomeServer.get_datastore()` ( #12031 )
...
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
2022-02-23 11:04:02 +00:00
250104d357
Implement account status endpoints (MSC3720) ( #12001 )
...
See matrix-org/matrix-doc#3720
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
81364db49b
Run `_handle_queued_pdus` as a background process ( #12041 )
...
... to ensure it gets a proper log context, mostly.
2022-02-22 13:33:22 +00:00
3070af4809
remote join processing: get create event from state, not auth_chain ( #12039 )
...
A follow-up to #12005 , in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
2022-02-21 19:27:35 +00:00
a85dde3445
Minor typing fixes ( #12034 )
...
These started failing in
https://github.com/matrix-org/synapse/pull/12031 ... I'm a bit mystified by how
they ever worked.
2022-02-21 18:37:04 +00:00
7c82da27aa
Add type hints to `synapse/storage/databases/main` ( #11984 )
2022-02-21 16:03:06 +00:00
e6acd3cf4f
Upgrade mypy to version 0.931 ( #12030 )
...
Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
2022-02-18 15:57:26 +00:00
707049c6ff
Allow modules to set a display name on registration ( #12009 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-17 16:54:16 +00:00
696acd3515
`send_join` response: get create event from `state`, not `auth_chain` ( #12005 )
...
msc3706 proposes changing the `/send_join` response:
> Any events returned within `state` can be omitted from `auth_chain`.
Currently, we rely on `m.room.create` being returned in `auth_chain`, but since
the `m.room.create` event must necessarily be part of the state, the above
change will break this.
In short, let's look for `m.room.create` in `state` rather than `auth_chain`.
2022-02-17 11:59:26 +00:00
130fd45393
Limit concurrent AS joins ( #11996 )
...
Initially introduced in matrix-org-hotfixes by e5537cfFixes #11995
See also #4826
2022-02-16 11:16:48 +00:00
2b5643b3af
Optimise calculating device_list changes in `/sync`. ( #11974 )
...
For users with large accounts it is inefficient to calculate the set of
users they share a room with (and takes a lot of space in the cache).
Instead we can look at users whose devices have changed since the last
sync and check if they share a room with the syncing user.
2022-02-15 15:01:00 +00:00
bab2394aa9
`_auth_and_persist_outliers`: drop events we have already seen ( #11994 )
...
We already have two copies of this code, in 2/3 of the callers of
`_auth_and_persist_outliers`. Before I add a third, let's push it down.
2022-02-15 14:33:28 +00:00
e44f91d678
Refactor search code to reduce function size. ( #11991 )
...
Splits the search code into a few logical functions instead of a single
unreadable function.
There are also a few additional changes for readability.
After refactoring it was clear to see there were some unused and
unnecessary variables, which were simplified.
2022-02-15 13:47:05 +00:00
55113dd5e8
Notify users, rather than rooms, of device list updates ( #11905 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-12 09:33:49 -05:00
0171fa5226
Remove deprecated user_may_create_room_with_invites callback ( #11950 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-11 13:58:11 +00:00
a121507cfe
Adds misc missing type hints ( #11953 )
2022-02-11 07:20:16 -05:00
8c94b3abe9
Experimental support to include bundled aggregations in search results (MSC3666) ( #11837 )
2022-02-08 09:21:20 -05:00
6c0984e3f0
Remove unnecessary ignores due to Twisted upgrade. ( #11939 )
...
Twisted 22.1.0 fixed some internal type hints, allowing Synapse
to remove ignore calls for parameters to connectTCP.
2022-02-08 09:15:59 -05:00
0640f8ebaa
Add a callback to allow modules to deny 3PID ( #11854 )
...
Part of the Tchap Synapse mainlining.
This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
2022-02-08 11:20:32 +01:00
fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) ( #11114 )
...
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)
1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
- Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering )).
- Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering )) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
- This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030 ) so the static archives can navigate and jump to a certain date.
- We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187 ) and [chunking](https://github.com/matrix-org/synapse/issues/3785 ) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
- Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793
Before | After
--- | ---
 | 
#### Why aren't we sorting topologically when receiving backfill events?
> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) which could change whatever implementation anyway.
>
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138
See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
cf06783d54
Remove optional state of `ApplicationService.is_interested`'s `store` parameter ( #11911 )
2022-02-07 18:26:42 +00:00
e03dde259b
Clean up an indirect reference to the homeserver datastore ( #11914 )
2022-02-07 13:25:09 +00:00
02632b3504
Stabilise MSC3231 (Token Based Registration) ( #11867 )
2022-02-04 12:15:13 +00:00
d80d39b035
Add a ratelimiter for 3pid invite ( #11892 )
2022-02-03 14:28:15 +01:00
64ec45fc1b
Send to-device messages to application services ( #11215 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
bf60da1a60
Configurable limits on avatars ( #11846 )
...
Only allow files which file size and content types match configured
limits to be set as avatar.
Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
2022-01-28 15:41:33 +01:00
2d3bd9aa67
Add a module callback to set username at registration ( #11790 )
...
This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step).
This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not.
Co-authored-by: David Robertson <davidr@element.io>
2022-01-26 14:21:13 +00:00
Sean Quah
