Commit Graph

1458 Commits (2fc15ac718b4f48e42762deb8398af6978dc506f)

Author SHA1 Message Date
Richard van der Hoff 1f32b90b0f
Room batch: fix up handling of unknown prev_event_ids (#12316) 2022-03-29 14:56:25 +01:00
Eric Eastwood 14662d3c18
Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with MSC2716 instead of abusing `auth_event_ids` (#12083)
Spawned from https://github.com/matrix-org/synapse/pull/10975#discussion_r813183430

Part of [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716)
2022-03-25 09:21:06 -05:00
Jonathan de Jong c5776780f0
Remove mutual_rooms `update_user_directory` check, and add extra documentation (#12038)
Resolves #10339
2022-03-23 12:47:07 +00:00
Jonathan de Jong 516d092ff9
Rename shared_rooms to mutual_rooms (#12036)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-03-23 11:19:20 +00:00
Patrick Cloke 8fe930c215
Move get_bundled_aggregations to relations handler. (#12237)
The get_bundled_aggregations code is fairly high-level and uses
a lot of store methods, we move it into the handler as that seems
like a better fit.
2022-03-18 17:49:32 +00:00
David Robertson 872dbb0181
Correct `check_username_for_spam` annotations and docs (#12246)
* Formally type the UserProfile in user searches
* export UserProfile in synapse.module_api
* Update docs

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-03-18 13:51:41 +00:00
Patrick Cloke fc9bd620ce
Add a relations handler to avoid duplication. (#12227)
Adds a handler layer between the REST and datastore layers for relations.
2022-03-16 10:39:15 -04:00
Patrick Cloke ea27528b5d
Support stable identifiers for MSC3440: Threading (#12151)
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00
Patrick Cloke 88cd6f9378
Allow retrieving the relations of a redacted event. (#12130)
This is allowed per MSC2675, although the original implementation did
not allow for it and would return an empty chunk / not bundle aggregations.

The main thing to improve is that the various caches get cleared properly
when an event is redacted, and that edits must not leak if the original
event is redacted (as that would presumably leak something similar to
the original event content).
2022-03-10 09:03:59 -05:00
Patrick Cloke cd1ae3d0b4
Remove backwards compatibility with RelationPaginationToken. (#12138) 2022-03-04 07:10:10 -05:00
Patrick Cloke 1d11b452b7
Use the proper serialization format when bundling aggregations. (#12090)
This ensures that the `latest_event` field of the bundled aggregation
for threads uses the same format as the other events in the response.
2022-03-03 10:43:06 -05:00
Patrick Cloke 7754af24ab
Remove the unstable `/spaces` endpoint. (#12073)
...and various code supporting it.

The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
2022-02-28 18:33:00 +00:00
Patrick Cloke 1866fb39d7
Move experimental support for MSC3440 to /versions. (#12099)
Instead of being part of /capabilities, this matches a change to
MSC3440 to properly use these endpoints.
2022-02-28 18:29:09 +00:00
Richard van der Hoff 5b2b36809f
Remove more references to `get_datastore` (#12067)
These have snuck in since #12031 was started.

Also a couple of other cleanups while we're in the area.
2022-02-23 12:35:53 +00:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Brendan Abolivier 250104d357
Implement account status endpoints (MSC3720) (#12001)
See matrix-org/matrix-doc#3720

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
Travis Ralston d7cb0dcbaa
Use v3 endpoints for fallback auth (Matrix 1.1) (#12019) 2022-02-22 11:20:45 +00:00
Travis Ralston 99f6d79fe1
Advertise Matrix 1.2 in `/_matrix/client/versions` (#12022)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2022-02-21 08:59:29 -07:00
Travis Ralston 31a298fec7
Advertise Matrix 1.1 in `/_matrix/client/versions` (#12020) 2022-02-18 07:49:53 -05:00
Travis Ralston 19bd9cff1a
Use stable MSC3069 `is_guest` flag on `/whoami`. (#12021)
Keeping backwards compatibility with the unstable flag for now.
2022-02-18 07:48:23 -05:00
Erik Johnston 3f4d25a48b
Remove unstable MSC3283 flags (#12018)
Fixes #11962
2022-02-17 17:22:55 +00:00
Brendan Abolivier 707049c6ff
Allow modules to set a display name on registration (#12009)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-17 16:54:16 +00:00
Patrick Cloke 4d7e74b2e5
Support the MSC3715 for `/relations`. (#11941)
This adds an unstable org.matrix.msc3715.dir parameter
which acts like dir on /mesages.
2022-02-11 11:20:27 -05:00
Dirk Klimpel d36943c4df
Support the stable API endpoint for MSC3283: new settings in `/capabilities` endpoint (#11933) 2022-02-11 09:32:11 +00:00
Patrick Cloke df36945ff0
Support pagination tokens from /sync and /messages in the relations API. (#11952) 2022-02-10 10:52:48 -05:00
Brendan Abolivier 0640f8ebaa
Add a callback to allow modules to deny 3PID (#11854)
Part of the Tchap Synapse mainlining.

This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
2022-02-08 11:20:32 +01:00
Eric Eastwood fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
Jonathan de Jong 02632b3504
Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
Patrick Cloke acda9f07c8
Revert experimental push rules from #7997. (#11884)
Manually reverts the merge from cdbb8e6d6e.
2022-02-02 09:49:31 -05:00
Brendan Abolivier 2d3bd9aa67
Add a module callback to set username at registration (#11790)
This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step).
This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not.

Co-authored-by: David Robertson <davidr@element.io>
2022-01-26 14:21:13 +00:00
Patrick Cloke 2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Brendan Abolivier 95b3f952fa
Add a config flag to inhibit `M_USER_IN_USE` during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
Patrick Cloke b784299cbc
Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:31:31 +00:00
reivilibre 4c2096599c
Make the `get_global_account_data_by_type_for_user` cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Patrick Cloke 68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Patrick Cloke 0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
Olivier Wilkinson (reivilibre) 6a04767439 Merge branch 'release-v1.50' into develop 2022-01-07 14:22:13 +00:00
Patrick Cloke 6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
Patrick Cloke 6c68e874b1
Remove the /send_relation endpoint. (#11682)
This was removed from MSC2674 before that was approved
and is not used by any known clients.
2022-01-06 19:00:34 +00:00
Patrick Cloke 877b45e812
Include `io.element.thread` capability for MSC3440. (#11690) 2022-01-05 12:08:03 -05:00
Patrick Cloke b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Richard van der Hoff 2215954147
Various opentracing enhancements (#11619)
* Wrap `auth.get_user_by_req` in an opentracing span

give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.

This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.

* Emit opentracing span for encoding json responses

This can be a significant time sink.

* Rename all sync spans with a prefix

* Write an opentracing span for encoding sync response

* opentracing span to group generate_room_entries

* opentracing spans within sync.encode_response

* changelog

* Use the `trace` decorator instead of context managers
2021-12-21 11:10:36 +00:00
Patrick Cloke dd47788752
Do not bundle aggregations for APIs which shouldn't include them. (#11592)
And make bundling aggregations opt-in, instead of opt-out to avoid
having APIs to include extraneous data (and being much heavier than
necessary).
2021-12-20 14:14:38 -05:00
Richard van der Hoff 8e4083e2f6 Merge remote-tracking branch 'origin/release-v1.49' into develop 2021-12-20 16:34:01 +00:00
Richard van der Hoff 76aa5537ad
Disable aggregation bundling on `/sync` responses (#11583)
* Disable aggregation bundling on `/sync` responses

A partial revert of #11478. This turns out to have had a significant CPU impact
on initial-sync handling. For now, let's disable it, until we find a more
efficient way of achieving this.

* Fix tests.

Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-12-20 16:33:35 +00:00
Eric Eastwood 43f5cc7adc
Add MSC2716 and MSC3030 to `/versions` -> `unstable_features` (#11582)
As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
2021-12-16 11:25:37 -06:00
reivilibre e5cdb9e233
Make `get_device` return None if the device doesn't exist rather than raising an exception. (#11565)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-12-13 15:39:43 +00:00
Patrick Cloke 3b8872299a
Do not allow cross-room relations, per MSC2674. (#11516) 2021-12-09 13:16:01 -05:00
Hubert Chathi b47d10dc46
Support unprefixed versions of fallback key property names. (#11541) 2021-12-09 11:41:27 +00:00
Robert Long afa0a5e4fc
Allow guests to send state events (#11378) 2021-12-09 11:02:05 +00:00