Commit Graph

889 Commits (d902181de98399d90c46c4e4e2cf631064757941)

Author SHA1 Message Date
Patrick Cloke 755bfeee3a
Use servlets for /key/ endpoints. (#14229)
To fix the response for unknown endpoints under that prefix.

See MSC3743.
2022-10-20 11:32:47 -04:00
Eric Eastwood 70b3396506
Explain `SynapseError` and `FederationError` better (#14191)
Explain `SynapseError` and `FederationError` better

Spawning from https://github.com/matrix-org/synapse/pull/13816#discussion_r993262622
2022-10-19 15:39:43 -05:00
Patrick Cloke 4283bd1cf9
Support filtering the /messages API by relation type (MSC3874). (#14148)
Gated behind an experimental configuration flag.
2022-10-17 11:32:11 -04:00
Eric Eastwood 40bb37eb27
Stop getting missing `prev_events` after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
Patrick Cloke 66a7857334
Use stable identifiers for MSC3771 & MSC3773. (#14050)
These are both part of Matrix 1.4 which has now been released.

For now, support both the unstable and stable identifiers.
2022-10-07 09:26:40 -04:00
Patrick Cloke b4ec4f5e71
Track notification counts per thread (implement MSC3773). (#13776)
When retrieving counts of notifications segment the results based on the
thread ID, but choose whether to return them as individual threads or as
a single summed field by letting the client opt-in via a sync flag.

The summarization code is also updated to be per thread, instead of per
room.
2022-10-04 09:47:04 -04:00
Brendan Abolivier be76cd8200
Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
reivilibre c06b2b7142
Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. (#13823) 2022-09-23 11:47:16 +01:00
Quentin Gliech eaed4e6113
Remove unused method in `synapse.api.auth.Auth`. (#13795)
Clean-up from b19060a29b (#13094)
and 73af10f419 (#13093) which removed
all callers.
2022-09-14 10:33:54 -04:00
Eric Eastwood a911ffb42c
Tag trace with instance name (#13761)
We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load.

As suggested by @clokep and @reivilibre in,

 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258
 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578
2022-09-09 11:31:37 -05:00
reivilibre d3d9ca156e
Cancel the processing of key query requests when they time out. (#13680) 2022-09-07 12:03:32 +01:00
reivilibre c2fe48a6ff
Rename the `EventFormatVersions` enum values so that they line up with room version numbers. (#13706) 2022-09-07 11:08:20 +01:00
David Robertson a4ecb8e353
Actually fix typechecking with latest types-jsonschema (#13724) 2022-09-06 14:29:16 +01:00
David Robertson 8cb9261598
Fix typechecking with latest `types-jsonschema` (#13712) 2022-09-05 11:13:13 +00:00
Šimon Brandner 0e99f07952
Remove support for unstable private read receipts (#13653)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-09-01 13:31:54 +01:00
Quentin Gliech 3dd175b628
`synapse.api.auth.Auth` cleanup: make permission-related methods use `Requester` instead of the `UserID` (#13024)
Part of #13019

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00
Eric Eastwood 06df5d4250
MSC2716v4 room version - remove namespace from MSC2716 event content fields (#13551)
Complement PR: https://github.com/matrix-org/complement/pull/450

As suggested in
https://github.com/matrix-org/matrix-spec-proposals/pull/2716#discussion_r941444525
2022-08-19 15:37:01 -05:00
Šimon Brandner ab18441573
Support stable identifiers for MSC2285: private read receipts. (#13273)
This adds support for the stable identifiers of MSC2285 while
continuing to support the unstable identifiers behind the configuration
flag. These will be removed in a future version.
2022-08-05 11:09:33 -04:00
Eric Eastwood 92d21faf12
Instrument `/messages` for understandable traces in Jaeger (#13368)
In Jaeger:

 - Before: huge list of uncategorized database calls
 - After: nice and collapsible into units of work
2022-08-03 10:57:38 -05:00
Dirk Klimpel d6e94ad9d9
Rename `RateLimitConfig` to `RatelimitSettings` (#13442) 2022-08-03 10:40:20 +01:00
Šimon Brandner 583f22780f
Use stable prefixes for MSC3827: filtering of `/publicRooms` by room type (#13370)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-07-27 19:46:57 +01:00
Will Hunt 502f075e96
Implement MSC3848: Introduce errcodes for specific event sending failures (#13343)
Implements MSC3848
2022-07-27 13:44:40 +01:00
David Robertson 599c403d99
Allow rate limiters to passively record actions they cannot limit (#13253)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-07-13 19:09:42 +00:00
Shay ad5761b65c
Add support for room version 10 (#13220) 2022-07-13 11:36:02 -07:00
David Teller 11f811470f
Uniformize spam-checker API, part 5: expand other spam-checker callbacks to return `Tuple[Codes, dict]` (#13044)
Signed-off-by: David Teller <davidt@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-07-11 16:52:10 +00:00
Šimon Brandner 13e359aec8
Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-06-29 17:12:45 +00:00
reivilibre 0dbdc39940
Fix a long-standing bug which meant that rate limiting was not restrictive enough in some cases. (#13018) 2022-06-15 14:11:55 +00:00
Quentin Gliech fe1daad672
Move the "email unsubscribe" resource, refactor the macaroon generator & simplify the access token verification logic. (#12986)
This simplifies the access token verification logic by removing the `rights`
parameter which was only ever used for the unsubscribe link in email
notifications. The latter has been moved under the `/_synapse` namespace,
since it is not a standard API.

This also makes the email verification link more secure, by embedding the
app_id and pushkey in the macaroon and verifying it. This prevents the user
from tampering the query parameters of that unsubscribe link.

Macaroon generation is refactored:

- Centralised all macaroon generation and verification logic to the
  `MacaroonGenerator`
- Moved to `synapse.utils`
- Changed the constructor to require only a `Clock`, hostname, and a secret key
  (instead of a full `Homeserver`).
- Added tests for all methods.
2022-06-14 09:12:08 -04:00
Quentin Gliech 92103cb2c8
Decouple `synapse.api.auth_blocking.AuthBlocking` from `synapse.api.auth.Auth`. (#13021) 2022-06-14 09:51:15 +01:00
Erik Johnston e3163e2e11
Reduce the amount of state we pull from the DB (#12811) 2022-06-06 09:24:12 +01:00
Patrick Cloke 7bc08f3201
Remove remaining bits of groups code. (#12936)
* Update worker docs to remove group endpoints.
* Removes an unused parameter to `ApplicationService`.
* Break dependency between media repo and groups.
* Avoid copying `m.room.related_groups` state events during room upgrades.
2022-06-01 09:41:25 -04:00
Travis Ralston d0e40dfe29
Fix M_USER_ACCOUNT_SUSPENDED error code for spec compliance (#12923) 2022-05-31 08:42:18 +01:00
Travis Ralston bcfdfeb65d Revert "Fix M_USER_ACCOUNT_SUSPENDED error code for spec compliance (#12922)"
This reverts commit e0fae823e9.
2022-05-30 20:29:40 -06:00
Travis Ralston e0fae823e9
Fix M_USER_ACCOUNT_SUSPENDED error code for spec compliance (#12922)
`M_` is a reserved namespace.
2022-05-31 02:27:19 +00:00
David Teller af7db19e1e
Uniformize spam-checker API, part 3: Expand check_event_for_spam with the ability to return additional fields (#12846)
Signed-off-by: David Teller <davidt@element.io>
2022-05-30 18:24:56 +02:00
David Teller 888eb736a1
Add code M_USER_ACCOUNT_SUSPENDED, as per MSC3823. (#12845)
Signed-off-by: David Teller <davidt@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-05-27 13:13:29 +00:00
Patrick Cloke c52abc1cfd
Additional constants for EDU types. (#12884)
Instead of hard-coding strings in many places.
2022-05-27 07:14:36 -04:00
Patrick Cloke a8db8c6eba
Remove user-visible groups/communities code (#12553)
Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove:

* All API endpoints (including Client-Server, Server-Server, and admin).
* Documented configuration options (and the experimental flag, which is now unused).
* Special handling during room upgrades.
* The `groups` section of the `/sync` response.
2022-05-25 07:53:40 -04:00
David Teller 28199e9357
Uniformize spam-checker API, part 2: check_event_for_spam (#12808)
Signed-off-by: David Teller <davidt@element.io>
2022-05-23 17:27:39 +00:00
David Teller 10280fc943
Uniformize spam-checker API, part 1: the `Code` enum. (#12703) 2022-05-20 14:53:25 +02:00
Travis Ralston 942c30b16b
Add a new room version for MSC3787's knock+restricted join rule (#12623) 2022-05-17 10:41:39 +00:00
Patrick Cloke 29f06704b8
Fix incorrect type hint in filtering code. (#12695) 2022-05-10 14:10:22 -04:00
reivilibre 07fa53ec40
Improve comments and error messages around access tokens. (#12577) 2022-05-05 13:39:59 +01:00
Patrick Cloke 7fbf42499d
Use `getClientAddress` instead of `getClientIP`. (#12599)
getClientIP was deprecated in Twisted 18.4.0, which also added
getClientAddress. The Synapse minimum version for Twisted is
currently 18.9.0, so all supported versions have the new API.
2022-05-04 14:11:21 -04:00
Šimon Brandner 116a4c8340
Implement changes to MSC2285 (hidden read receipts) (#12168)
* Changes hidden read receipts to be a separate receipt type
  (instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
2022-05-04 11:59:22 -04:00
Tulir Asokan 4bc8cb4669
Implement MSC2815: allow room moderators to view redacted event content (#12427)
Implements matrix-org/matrix-spec-proposals#2815

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2022-04-20 12:57:39 +01:00
Patrick Cloke 86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
David Robertson a2b00a4486
Bump `black` and `click` versions (#12320) 2022-03-29 10:41:19 +00:00
Patrick Cloke ea27528b5d
Support stable identifiers for MSC3440: Threading (#12151)
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00
Richard van der Hoff 6c0b44a3d7
Fix `PushRuleEvaluator` and `Filter` to work on frozendicts (#12100)
* Fix `PushRuleEvaluator` to work on frozendicts

frozendicts do not (necessarily) inherit from dict, so this needs to handle
them correctly.

* Fix event filtering for frozen events

Looks like this one was introduced by #11194.
2022-02-28 17:40:24 +00:00