Commit Graph

14750 Commits (5ace5d7b15b1d4604ec86f80ebcd3213d7a5550c)

Author SHA1 Message Date
Sean Quah f4ab6a4a96
Refuse to start when `faster_joins` is enabled on a worker deployment (#13531)
Synapse does not currently support faster room joins on deployments with
workers.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-16 16:21:55 +01:00
reivilibre c3516e9dec
Faster room joins: make `/joined_members` block whilst the room is partial stated. (#13514) 2022-08-16 13:16:56 +01:00
Erik Johnston 5442891cbc
Make push rules use proper structures. (#13522)
This improves load times for push rules:

| Version              | Time per user | Time for 1k users | 
| -------------------- | ------------- | ----------------- |
| Before               |       138 µs  |             138ms |
| Now (with custom)    |       2.11 µs |            2.11ms |
| Now (without custom) |       49.7 ns |           0.05 ms |

This therefore has a large impact on send times for rooms
with large numbers of local users in the room.
2022-08-16 12:22:17 +01:00
David Robertson d642ce4b32
Use Pydantic to systematically validate a first batch of endpoints in `synapse.rest.client.account`. (#13188) 2022-08-15 19:05:57 +00:00
Eric Eastwood 344a2f767c
Instrument `FederationStateIdsServlet` - `/state_ids` (#13499)
Instrument FederationStateIdsServlet - `/state_ids` so it's easier to follow what's going on in Jaeger when viewing a trace.
2022-08-15 19:41:23 +01:00
David Robertson 19e5d44886
Revert "Update locked versions of mypy and mypy-zope (#13521)"
This reverts commit f383b9b3ec. Other PRs
were seeing mypy failures that looked to be related to mypy-zope.
Confusingly, we didn't see this on #13521.

Revert this for now and investigate later.
2022-08-15 14:51:05 +01:00
Patrick Cloke 46bd7f4ed9
Clarifications for event push action processing. (#13485)
* Clarifies comments.
* Fixes an erroneous comment (about return type) added in #13455
  (ec24813220).
* Clarifies the name of a variable.
* Simplifies logic of pulling out the latest join for the requesting user.
2022-08-15 09:33:17 -04:00
David Robertson f383b9b3ec
Update locked versions of mypy and mypy-zope (#13521) 2022-08-15 11:32:30 +01:00
Germain 4390121684
Add viewport directive to HTML templates to optimise for mobile (#13493) 2022-08-11 14:04:20 +01:00
Olivier Wilkinson (reivilibre) 05bc059e0d Merge branch 'release-v1.65' into develop 2022-08-11 13:19:12 +01:00
Richard van der Hoff 507c1cb330
Update the rejected state of events during resync (#13459)
Events can be un-rejected or newly-rejected during resync, so ensure we update
the database and caches when that happens.
2022-08-11 10:42:24 +00:00
reivilibre 12abd72497
Revert 'Remove the unspecced field in the response. (#13365)' to give more time for clients to update. (#13501) 2022-08-11 10:27:48 +00:00
Sean Quah 51c01d450a
Add some miscellaneous comments around sync (#13474)
Add some miscellaneous comments to document sync, especially around
`compute_state_delta`.

Signed-off-by: Sean Quah <seanq@matrix.org>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-08-10 12:58:20 +01:00
Eric Eastwood 1b09b0832e
Allow use of both `@trace` and `@tag_args` stacked on the same function (#13453)
```py
@trace
@tag_args
async def get_oldest_event_ids_with_depth_in_room(...)
  ...
```

Before this PR, you would see a warning in the logs and the span was not exported:
```
2022-08-03 19:11:59,383 - synapse.logging.opentracing - 835 - ERROR - GET-0 - @trace may not have wrapped EventFederationWorkerStore.get_oldest_event_ids_with_depth_in_room correctly! The function is not async but returned a coroutine.
```
2022-08-09 14:32:33 -05:00
David Robertson 7a19995120
Correct a misnamed argument in state res v2 (#13467)
In state res v2, we apply two passes of iterative auth checks. The first
pass replays power events and events in their auth chains, but only
those belonging to the full conflicted set. The source code as written
suggests that we want only those belonging to the auth difference (which
is a smaller set of events).

At runtime we were doing the correct thing anyway, because the only
callsite of `_reverse_topological_power_sort` passes in the
`full_conflicted_set`. So this really is just a rename.
2022-08-08 16:59:56 +01:00
Šimon Brandner ab18441573
Support stable identifiers for MSC2285: private read receipts. (#13273)
This adds support for the stable identifiers of MSC2285 while
continuing to support the unstable identifiers behind the configuration
flag. These will be removed in a future version.
2022-08-05 11:09:33 -04:00
Matt C 026ac4486c
Update module API "update room membership" method to allow for remote joins (#13441)
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-05 09:37:58 +00:00
Erik Johnston b6a6bb4027
Add comments about how event push actions are stored. (#13445) 2022-08-04 19:38:08 +00:00
Eric Eastwood 860fdd9098
Fix `@tag_args` being off-by-one (ahead) (#13452)
Fix @tag_args being off-by-one (ahead)

Example:

```
argspec.args=[
  'self',
  'room_id'
]

args=(
  <synapse.storage.databases.main.DataStore object at 0x10d0b8d00>,
  '!HBehERstyQBxyJDLfR:my.synapse.server'
)
```

---

The previous logic was also flawed and we can end up in a situation like this:

```
argspec.args=['self', 'dest', 'room_id', 'limit', 'extremities']

args=(<synapse.federation.federation_client.FederationClient object at 0x7f1651c18160>, 'hs1', '!jAEHKIubyIfuLOdfpY:hs1')
```

From this source:
```py
async def backfill(
    self, dest: str, room_id: str, limit: int, extremities: Collection[str]
) -> Optional[List[EventBase]]:
```

And this usage:
```py
events = await self._federation_client.backfill(
    dest, room_id, limit=limit, extremities=extremities
)
```

which would previously cause this error:
```
synapse_main | 2022-08-04 06:13:12,051 - synapse.handlers.federation - 424 - ERROR - GET-5 - Failed to backfill from hs1 because tuple index out of range
synapse_main | Traceback (most recent call last):
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation.py", line 392, in try_backfill
synapse_main |     await self._federation_event_handler.backfill(
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper
synapse_main |     return await func(*args, **kwargs)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 593, in backfill
synapse_main |     events = await self._federation_client.backfill(
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper
synapse_main |     return await func(*args, **kwargs)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 827, in _wrapper
synapse_main |     with wrapping_logic(func, *args, **kwargs):
synapse_main |   File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
synapse_main |     return next(self.gen)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 922, in _wrapping_logic
synapse_main |     set_attribute("ARG_" + arg, str(args[i + 1]))  # type: ignore[index]
synapse_main | IndexError: tuple index out of range
```
2022-08-04 14:29:41 -05:00
Patrick Cloke ec24813220
Improve comments (& avoid a duplicate query) in push actions processing. (#13455)
* Adds docstrings and inline comments.
* Formats SQL queries using triple quoted strings.
* Minor formatting changes.
* Avoid fetching `event_push_summary_stream_ordering` multiple times
  in the same transactions.
2022-08-04 19:24:44 +00:00
Richard van der Hoff 96d92156d0
Update type of `EventContext.rejected` (#13460) 2022-08-04 17:45:01 +01:00
reivilibre e9e6aacfbe
Faster Room Joins: prevent Synapse from answering federated join requests for a room which it has not fully joined yet. (#13416) 2022-08-04 16:27:04 +01:00
Nick Mills-Barrett 41320a0554
Optimise async get event lookups (#13435)
Still maintains local in memory lookup optimisation, but does any external
lookup as part of the deferred that prevents duplicate lookups for the same
event at once. This makes the assumption that fetching from an external
cache is a non-zero load operation.
2022-08-04 15:49:55 +01:00
Matt C a91078200d
Add module API method to create a room (#13429)
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-04 09:34:05 +00:00
Brendan Abolivier 845732be45
Fix rooms not being properly excluded from incremental sync (#13408) 2022-08-04 11:02:29 +02:00
Shay a648a06d52
Add some tracing spans to give insight into local joins (#13439) 2022-08-03 10:19:34 -07:00
Eric Eastwood 92d21faf12
Instrument `/messages` for understandable traces in Jaeger (#13368)
In Jaeger:

 - Before: huge list of uncategorized database calls
 - After: nice and collapsible into units of work
2022-08-03 10:57:38 -05:00
andrew do 78a3111c41
Return 404 or member list when getting joined_members after leaving (#13374)
Signed-off-by: Andrew Doh <andrewddo@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrewm@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-03 14:26:31 +02:00
Dirk Klimpel d6e94ad9d9
Rename `RateLimitConfig` to `RatelimitSettings` (#13442) 2022-08-03 10:40:20 +01:00
Matt C 570bf32bbb
Add module API method to resolve a room alias to a room ID (#13428)
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-03 09:25:36 +00:00
Sean Quah 8d317f6da5
Fix error when out of servers to sync partial state with (#13432)
so that we raise the intended error instead.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-02 12:12:44 +01:00
reivilibre e17e5c97e0
Faster Room Joins: don't leave a stuck room partial state flag if the join fails. (#13403) 2022-08-01 16:45:39 +00:00
Patrick Cloke f8e7a9418a
Fix missing import in `federation_event` handler. (#13431)
#13404 removed an import of `Optional` which was still needed
due to #13413 added more usages.
2022-08-01 14:14:29 +00:00
Sean Quah 224d792dd7
Refactor `_resolve_state_at_missing_prevs` to return an `EventContext` (#13404)
Previously, `_resolve_state_at_missing_prevs` returned the resolved
state before an event and a partial state flag. These were unwieldy to
carry around would only ever be used to build an event context. Build
the event context directly instead.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-01 13:53:56 +01:00
Richard van der Hoff 23768ccb4d
Faster joins: fix rejected events becoming un-rejected during resync (#13413)
Make sure that we re-check the auth rules during state resync, otherwise
rejected events get un-rejected.
2022-08-01 11:20:05 +01:00
Richard van der Hoff d548d8f18d Synapse 1.64.0rc2 (2022-07-29)
==============================
 
 This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in a future release. ([\#13406](https://github.com/matrix-org/synapse/issues/13406))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAmLj4a8ACgkQOSor00I9
 eP8biQf/c8yY2mbeRZcBKtp6yoQCRYQvboSMEXyi+dLe1hNqdhSZwRQcAoFuAFwE
 WdScDvoTaElUxv0v6eCI1x9CoXnZ6xpDShvK39j5Yhzv+1tNsm5Uq9imyG3jK5i6
 U/3Gt6CrCsS01VkGslQ3B5I6MFtbC6ZZK9O48yg+GD8Oqw2HH/gllr5swyVbKdbc
 GGhRBHvgXn+w6d/KnKt8uRxJqIpDt9JMga+WdB8CwFR5WnWbGdw24KsyxmBuOLC3
 caQRiluJL/X4jApUpfsJMBBd/jrDod5wWDFO/4P+v0+2d3Ts+hKezZbt5h1VIYSw
 szZXbzxn5RNDkNiJDpOOOMYQ5DXGmA==
 =3/nK
 -----END PGP SIGNATURE-----

Merge tag 'v1.64.0rc2' into develop

Synapse 1.64.0rc2 (2022-07-29)
==============================

This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in a future release. ([\#13406](https://github.com/matrix-org/synapse/issues/13406))
2022-07-29 15:15:21 +01:00
3nprob 98fb610cc0
Revert "Drop support for delegating email validation (#13192)" (#13406)
Reverts commit fa71bb18b5, and tweaks documentation.

Signed-off-by: 3nprob <git@3n.anonaddy.com>
2022-07-29 10:29:23 +00:00
Šimon Brandner 583f22780f
Use stable prefixes for MSC3827: filtering of `/publicRooms` by room type (#13370)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-07-27 19:46:57 +01:00
Will Hunt 502f075e96
Implement MSC3848: Introduce errcodes for specific event sending failures (#13343)
Implements MSC3848
2022-07-27 13:44:40 +01:00
reivilibre 39be5bc550
Make minor clarifications to the error messages given when we fail to join a room via any server. (#13160) 2022-07-27 10:37:50 +00:00
Eric Eastwood 4f3082d6bf
Fix `get_pdu` asking every remote destination even after it finds an event (#13346) 2022-07-27 10:40:04 +01:00
Nick Mills-Barrett bf3115584c
Copy room serials before handling in `get_new_events_as` (#13392) 2022-07-26 17:45:27 +00:00
Patrick Cloke 57d334a13d
Remove the unspecced `room_id` field in the `/hierarchy` response. (#13365)
The `room_id` field represented the parent space for each room
and was made redundant by changes in the API shape where the
`children_state` is now nested underneath each `room`.

The room ID of each child is in the `state_key` field and is still
available.
2022-07-26 08:02:34 -04:00
Richard van der Hoff ca3db044a3
Fix infinite loop in partial-state resync (#13353)
Make sure that we only pull out events from the db once they have no
prev-events with partial state.
2022-07-26 11:47:31 +00:00
Sean Quah 335ebb21cc
Faster room joins: avoid blocking when pulling events with missing prevs (#13355)
Avoid blocking on full state in `_resolve_state_at_missing_prevs` and
return a new flag indicating whether the resolved state is partial.
Thread that flag around so that it makes it into the event context.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-07-26 12:39:23 +01:00
Patrick Cloke 8b603299bf
Remove unused argument for get_relations_for_event. (#13383) 2022-07-26 07:19:20 -04:00
Doug 549c55606a
Disable autocorrect and autocaptialisation when entering username for SSO registration. (#13350)
When registering a new account via SSO on iOS, the text field becomes pretty annoying as it autocapitalises and autocorrects your input. This PR fixes that (although I have only tested the raw HTML file on the simulator, I'm not sure how to get the complete setup available for testing in the flow).
2022-07-26 08:08:20 +00:00
Jan Schär e8519e0ed2
Support Implicit TLS for sending emails (#13317)
Previously, TLS could only be used with STARTTLS.
Add a new option `force_tls`, where TLS is used from the start.
Implicit TLS is recommended over STARTLS,
see https://datatracker.ietf.org/doc/html/rfc8314

Fixes #8046.

Signed-off-by: Jan Schär <jan@jschaer.ch>
2022-07-25 16:27:19 +01:00
Patrick Cloke 908aeac44a
Additional fixes for opentracing type hints. (#13362) 2022-07-25 08:34:06 -04:00
Erik Johnston 43adf2521c
Refactor presence so we can prune user in room caches (#13313)
See #10826 and #10786 for context as to why we had to disable pruning on
those caches.

Now that `get_users_who_share_room_with_user` is called frequently only
for presence, we just need to make calls to it less frequent and then we
can remove the various levels of caching that is going on.
2022-07-25 09:21:06 +00:00