Commit Graph

489 Commits (3d87847ecc943c689c4587c5327d744e4a8f92c2)

Author SHA1 Message Date
David Robertson b5b5f66084
Move `StateFilter` to `synapse.types` (#14668)
* Move `StateFilter` to `synapse.types`

* Changelog
2022-12-12 16:19:30 +00:00
Shay 72f3e38137
Fix possible variable shadow in `create_new_client_event` (#14575) 2022-11-28 19:18:12 -08:00
Brendan Abolivier 86c5a710d8
Implement MSC3912: Relation-based redactions (#14260)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-03 16:21:31 +00:00
Shay b7a7ff6ee3
Add initial power level event to batch of bulk persisted events when creating a new room. (#14228) 2022-10-21 10:46:22 -07:00
Shay 847e2393f3
Prepatory work for adding power level event to batched events (#14214) 2022-10-18 09:58:47 -07:00
Shay b6baa46db0
Fix a bug where the joined hosts for a given event were not being properly cached (#14125) 2022-10-12 11:01:00 -07:00
Shay 7b7478e8b6
Batch up notifications after event persistence (#14033) 2022-10-05 10:12:48 -07:00
Shay 8ab16a92ed
Persist CreateRoom events to DB in a batch (#13800) 2022-09-28 10:11:48 +00:00
Shay a2cf66a94d
Prepatory work for batching events to send (#13487)
This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
2022-09-28 10:39:03 +01:00
reivilibre 6302753012
Deduplicate `is_server_notices_room`. (#13780) 2022-09-14 15:53:18 +00:00
Eric Eastwood d58615c82c
Directly lookup local membership instead of getting all members in a room first (`get_users_in_room` mis-use) (#13608)
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:13:12 -05:00
Quentin Gliech 3dd175b628
`synapse.api.auth.Auth` cleanup: make permission-related methods use `Requester` instead of the `UserID` (#13024)
Part of #13019

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00
reivilibre c3516e9dec
Faster room joins: make `/joined_members` block whilst the room is partial stated. (#13514) 2022-08-16 13:16:56 +01:00
Shay a648a06d52
Add some tracing spans to give insight into local joins (#13439) 2022-08-03 10:19:34 -07:00
andrew do 78a3111c41
Return 404 or member list when getting joined_members after leaving (#13374)
Signed-off-by: Andrew Doh <andrewddo@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrewm@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-03 14:26:31 +02:00
Will Hunt 502f075e96
Implement MSC3848: Introduce errcodes for specific event sending failures (#13343)
Implements MSC3848
2022-07-27 13:44:40 +01:00
Sean Quah 335ebb21cc
Faster room joins: avoid blocking when pulling events with missing prevs (#13355)
Avoid blocking on full state in `_resolve_state_at_missing_prevs` and
return a new flag indicating whether the resolved state is partial.
Thread that flag around so that it makes it into the event context.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-07-26 12:39:23 +01:00
David Robertson b977867358
Rate limit joins per-room (#13276) 2022-07-19 11:45:17 +00:00
Erik Johnston cf5fa5063d
Don't pull out full state when sending dummy events (#13310) 2022-07-18 14:19:11 +01:00
Erik Johnston 0ca4172b5d
Don't pull out state in `compute_event_context` for unconflicted state (#13267) 2022-07-14 13:57:02 +00:00
Sean Quah 68db233f0c
Handle race between persisting an event and un-partial stating a room (#13100)
Whenever we want to persist an event, we first compute an event context,
which includes the state at the event and a flag indicating whether the
state is partial. After a lot of processing, we finally try to store the
event in the database, which can fail for partial state events when the
containing room has been un-partial stated in the meantime.

We detect the race as a foreign key constraint failure in the data store
layer and turn it into a special `PartialStateConflictError` exception,
which makes its way up to the method in which we computed the event
context.

To make things difficult, the exception needs to cross a replication
request: `/fed_send_events` for events coming over federation and
`/send_event` for events from clients. We transport the
`PartialStateConflictError` as a `409 Conflict` over replication and
turn `409`s back into `PartialStateConflictError`s on the worker making
the request.

All client events go through
`EventCreationHandler.handle_new_client_event`, which is called in
*a lot* of places. Instead of trying to update all the code which
creates client events, we turn the `PartialStateConflictError` into a
`429 Too Many Requests` in
`EventCreationHandler.handle_new_client_event` and hope that clients
take it as a hint to retry their request.

On the federation event side, there are 7 places which compute event
contexts. 4 of them use outlier event contexts:
`FederationEventHandler._auth_and_persist_outliers_inner`,
`FederationHandler.do_knock`, `FederationHandler.on_invite_request` and
`FederationHandler.do_remotely_reject_invite`. These events won't have
the partial state flag, so we do not need to do anything for then.

The remaining 3 paths which create events are
`FederationEventHandler.process_remote_join`,
`FederationEventHandler.on_send_membership_event` and
`FederationEventHandler._process_received_pdu`.

We can't experience the race in `process_remote_join`, unless we're
handling an additional join into a partial state room, which currently
blocks, so we make no attempt to handle it correctly.

`on_send_membership_event` is only called by
`FederationServer._on_send_membership_event`, so we catch the
`PartialStateConflictError` there and retry just once.

`_process_received_pdu` is called by `on_receive_pdu` for incoming
events and `_process_pulled_event` for backfill. The latter should never
try to persist partial state events, so we ignore it. We catch the
`PartialStateConflictError` in `on_receive_pdu` and retry just once.

Refering to the graph of code paths in
https://github.com/matrix-org/synapse/issues/12988#issuecomment-1156857648
may make the above make more sense.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-05 16:12:52 +01:00
Shay 046a6513bc
Don't process /send requests for users who have hit their ratelimit (#13134) 2022-06-30 09:22:40 -07:00
Quentin Gliech 92103cb2c8
Decouple `synapse.api.auth_blocking.AuthBlocking` from `synapse.api.auth.Auth`. (#13021) 2022-06-14 09:51:15 +01:00
David Teller a164a46038
Uniformize spam-checker API, part 4: port other spam-checker callbacks to return `Union[Allow, Codes]`. (#12857)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-06-13 18:16:16 +00:00
Richard van der Hoff f68b5e5773 Merge branch 'rav/simplify_event_auth_interface' into develop 2022-06-13 11:34:59 +01:00
Richard van der Hoff c1b28b8842 Remove redundant `room_version` param from `check_auth_rules_from_context`
It's now implied by the room_version property on the event.
2022-06-12 23:13:10 +01:00
Richard van der Hoff 68be42f6b6 Remove `room_version` param from `validate_event_for_room_version`
Instead, use the `room_version` property of the event we're validating.

The `room_version` was originally added as a parameter somewhere around #4482,
but really it's been redundant since #6875 added a `room_version` field to `EventBase`.
2022-06-12 23:13:09 +01:00
Richard van der Hoff 7c6b2204d1
Faster joins: add issue links to the TODOs (#13004)
... to help us keep track of these things
2022-06-09 10:13:03 +00:00
Erik Johnston e3163e2e11
Reduce the amount of state we pull from the DB (#12811) 2022-06-06 09:24:12 +01:00
Erik Johnston 888a29f412
Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
Richard van der Hoff 79dadf7216
Fix 404 on `/sync` when the last event is a redaction of an unknown/purged event (#12905)
Currently, we try to pull the event corresponding to a sync token from the database. However, when
we fetch redaction events, we check the target of that redaction (because we aren't allowed to send
redactions to clients without validating them). So, if the sync token points to a redaction of an event
that we don't have, we have a problem.

It turns out we don't really need that event, and can just work with its ID and metadata, which
sidesteps the whole problem.
2022-06-01 11:29:51 +00:00
Erik Johnston 3594f6c1f3 Merge branch 'master' into develop 2022-05-31 14:48:22 +01:00
Erik Johnston 1e453053cb
Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
Brendan Abolivier 8fd87739bf
Fix import in module_api module and docs on the new check_event_for_spam signature (#12918)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-05-31 12:04:53 +02:00
David Teller af7db19e1e
Uniformize spam-checker API, part 3: Expand check_event_for_spam with the ability to return additional fields (#12846)
Signed-off-by: David Teller <davidt@element.io>
2022-05-30 18:24:56 +02:00
Erik Johnston b83bc5fab5
Pull out less state when handling gaps mk2 (#12852) 2022-05-26 09:48:12 +00:00
Erik Johnston 4660d9fdcf
Fix up `state_store` naming (#12871) 2022-05-25 12:59:04 +01:00
David Teller 28199e9357
Uniformize spam-checker API, part 2: check_event_for_spam (#12808)
Signed-off-by: David Teller <davidt@element.io>
2022-05-23 17:27:39 +00:00
Shay 71e8afe34d
Update EventContext `get_current_event_ids` and `get_prev_event_ids` to accept state filters and update calls where possible (#12791) 2022-05-20 09:54:12 +01:00
Patrick Cloke 86a515ccbf
Consolidate logic for parsing relations. (#12693)
Parse the `m.relates_to` event content field (which describes relations)
in a single place, this is used during:

* Event persistence.
* Validation of the Client-Server API.
* Fetching bundled aggregations.
* Processing of push rules.

Each of these separately implement the logic and each made slightly
different assumptions about what was valid. Some had minor / potential
bugs.
2022-05-16 12:42:45 +00:00
Patrick Cloke a4c75918b3
Remove unneeded `ActionGenerator` class. (#12691)
It simply passes through to `BulkPushRuleEvaluator`, which can be
called directly instead.
2022-05-11 07:15:21 -04:00
Erik Johnston c72d26c1e1
Refactor `EventContext` (#12689)
Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing.

The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). 

One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used.


Part of #12684
2022-05-10 19:43:13 +00:00
David Robertson 2607b3e181
Update mypy to 0.950 and fix complaints (#12650) 2022-05-06 12:35:20 +00:00
andrew do 01e625513a
remove constantly lib use and switch to enums. (#12624) 2022-05-04 11:26:11 +00:00
Nick Mills-Barrett e3a49f4784
Fix missing sync events during historical batch imports (#12319)
Discovered after much in-depth investigation in #12281.

Closes: #12281
Closes: #3305

Signed off by: Nick Mills-Barrett nick@beeper.com
2022-04-13 11:38:35 +01:00
Patrick Cloke 86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
Eric Eastwood 6f2943714b
Remove unused `auth_event_ids` argument plumbing (#12304)
Follow-up to https://github.com/matrix-org/synapse/pull/12083

Since we are now using the new `state_event_ids` parameter to do all of the heavy lifting.
We can remove any spots where we plumbed `auth_event_ids` just for MSC2716 things in
https://github.com/matrix-org/synapse/pull/9247/files.

Removing `auth_event_ids` from following functions:

 - `create_and_send_nonmember_event`
 - `_local_membership_update`
 - `update_membership`
 - `update_membership_locked`
2022-03-29 09:18:52 +01:00
Eric Eastwood 14662d3c18
Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with MSC2716 instead of abusing `auth_event_ids` (#12083)
Spawned from https://github.com/matrix-org/synapse/pull/10975#discussion_r813183430

Part of [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716)
2022-03-25 09:21:06 -05:00
Patrick Cloke ea27528b5d
Support stable identifiers for MSC3440: Threading (#12151)
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00