Commit Graph

2876 Commits (29f06704b8871a44926f7c99e73cf4a978fb8e81)

Author SHA1 Message Date
Richard van der Hoff 147f098fb4
Stop writing to `event_reference_hashes` (#12679)
This table is never read, since #11794. We stop writing to it; in future we can
drop it altogether.
2022-05-10 15:35:08 +01:00
Sean Quah dbb12a0b54
Add helper class for testing request cancellation (#12630)
Also expose the `SynapseRequest` from `FakeChannel` in tests, so that
we can call `Request.connectionLost` to simulate a client disconnecting.

Signed-off-by: Sean Quah <seanq@element.io>
2022-05-10 14:06:56 +01:00
reivilibre 699192fc1a
Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. (#12654)
Co-authored-by: Shay <hillerys@element.io>
2022-05-10 11:08:45 +01:00
Erik Johnston 8dd3e0e084
Immediately retry any requests that have backed off when a server comes back online. (#12500)
Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
2022-05-10 10:39:54 +01:00
David Robertson 051a1c3f22
Convert stringy power levels to integers on room upgrade (#12657) 2022-05-07 13:37:29 +01:00
David Robertson 2607b3e181
Update mypy to 0.950 and fix complaints (#12650) 2022-05-06 12:35:20 +00:00
reivilibre c2d50e9f6c
Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452) 2022-05-06 11:43:53 +01:00
DeepBlueV7.X a377a43386
Support MSC3266 room summaries over federation (#11507)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2022-05-05 15:25:00 +01:00
Šimon Brandner 9ae0253f4e
Use `private` instead of `hidden` in MSC2285 related code. (#12635) 2022-05-05 12:31:25 +00:00
Patrick Cloke f90d381c7b
Edits/annotations should not have any bundled aggregations calculated. (#12633)
Fixes a regression from 8b309adb43 (#11660)
and b65acead42 (#11752) where events which
themselves were an edit or an annotation could have bundled aggregations calculated,
which is not allowed.
2022-05-05 08:15:12 -04:00
Will Hunt 2d74a8c178
Add `mau_appservice_trial_days` config (#12619)
* Add mau_appservice_trial_days

* Add a test

* Tweaks

* changelog

* Ensure we sync after the delay

* Fix types

* Add config statement

* Fix test

* Reinstate logging that got removed

* Fix feature name
2022-05-04 19:33:26 +01:00
Patrick Cloke 7fbf42499d
Use `getClientAddress` instead of `getClientIP`. (#12599)
getClientIP was deprecated in Twisted 18.4.0, which also added
getClientAddress. The Synapse minimum version for Twisted is
currently 18.9.0, so all supported versions have the new API.
2022-05-04 14:11:21 -04:00
Šimon Brandner 116a4c8340
Implement changes to MSC2285 (hidden read receipts) (#12168)
* Changes hidden read receipts to be a separate receipt type
  (instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
2022-05-04 11:59:22 -04:00
Patrick Cloke ba3fd54bad
Remove unstable/unspecced login types. (#12597)
* `m.login.jwt`, which was never specced and has been deprecated
  since Synapse 1.16.0. (`org.matrix.login.jwt` can be used instead.)
* `uk.half-shot.msc2778.login.application_service`, which was
  stabilized as part of the Matrix spec v1.2 release.
2022-05-04 13:53:21 +00:00
Patrick Cloke 75dff3dc98
Include bundled aggregations for the latest event in a thread. (#12273)
The `latest_event` field of the bundled aggregations for `m.thread` relations
did not include bundled aggregations itself. This resulted in clients needing to
immediately request the event from the server (and thus making it useless that
the latest event itself was serialized instead of just including an event ID).
2022-05-04 08:38:18 -04:00
Richard van der Hoff 96e0cdbc5a
Add a consistency check on events read from the database (#12620)
I've seen a few errors which can only plausibly be explained by the calculated
event id for an event being different from the ID of the event in the
database. It should be cheap to check this, so let's do so and raise an
exception.
2022-05-03 21:27:52 +01:00
Patrick Cloke aa5f5ede33
Remove unstable identifiers for MSC3069. (#12596) 2022-05-03 12:43:12 -04:00
Šimon Brandner 0d9eaa19fd Use constants for receipt types in tests. (#12582) 2022-04-28 13:34:33 -04:00
Sean Quah 0b684b59e5
Fix logging of incorrect status codes for disconnected requests (#12580)
The status code of requests must always be set, regardless of client
disconnection, otherwise they will always be logged as 200!.

Broken for `respond_with_json` in
f48792eec4.
Broken for `respond_with_json_bytes` in
3e58ce72b4.
Broken for `respond_with_html_bytes` in
ea26e9a98b.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-28 15:49:50 +00:00
Sean Quah 78b99de7c2
Prefer `make_awaitable` over `defer.succeed` in tests (#12505)
When configuring the return values of mocks, prefer awaitables from
`make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable
once, so it is inappropriate for a mock to return the same `Deferred`
multiple times.

Also update `run_in_background` to support functions that return
arbitrary awaitables.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-27 14:58:26 +01:00
Brendan Abolivier 5ef673de4f
Add a module API to allow modules to edit push rule actions (#12406)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-27 13:55:33 +00:00
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Nick Mills-Barrett 63ba9ba38b
Bound ephemeral events by key (#12544)
Co-authored-by: Brad Murray <bradtgmurray@gmail.com>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-04-26 20:14:21 +01:00
Erik Johnston f59e3f4c90
Mark remote device list updates as already handled (#12557) 2022-04-26 17:07:21 +01:00
Sean Quah 8a87b4435a
Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db` (#12529)
Multiple calls to `EventsWorkerStore._get_events_from_cache_or_db` can
reuse the same database fetch, which is initiated by the first call.
Ensure that cancelling the first call doesn't cancel the other calls
sharing the same database fetch.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-25 19:39:17 +01:00
Patrick Cloke 185da8f0f2
Misc. clean-ups to the relations code (#12519)
* Corrects some typos / copy & paste errors in tests.
* Clarifies docstrings.
* Removes an unnecessary method.
2022-04-25 08:25:56 -04:00
Sean Quah a50fb411b3
Update `delay_cancellation` to accept any awaitable (#12468)
This will mainly be useful when dealing with module callbacks, which are
all typed as returning `Awaitable`s instead of coroutines or
`Deferred`s.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-22 18:20:06 +01:00
Richard van der Hoff f5668f0b4a
Await un-partial-stating after a partial-state join (#12399)
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
2022-04-21 07:42:03 +01:00
Patrick Cloke d0c1f4ca4c
Remove unnecessary config overrides for MSC3666. (#12511) 2022-04-20 09:56:59 -04:00
Richard van der Hoff b80bb7e452
Fix `/room/.../event/...` to return the *original* event after any edits (#12476)
This is what the MSC (now) requires. Fixes https://github.com/matrix-org/synapse/issues/10310.
2022-04-19 16:42:19 +01:00
Richard van der Hoff b121a3ad2b
Back out implementation of MSC2314 (#12474)
MSC2314 has now been closed, so we're backing out its implementation, which
originally happened in #6176.

Unfortunately it's not a direct revert, as that PR mixed in a bunch of
unrelated changes to tests etc.
2022-04-19 11:17:29 +00:00
Erik Johnston 0b014eb25e
Only send out device list updates for our own users (#12465)
Broke in #12365
2022-04-14 13:05:31 +01:00
Shay 8e2759f2d8
Limit `device_id` size to 512B (#12454)
*
2022-04-13 10:04:01 -07:00
David Baker 73d8ded0b0
Prevent a sync request from removing a user's busy presence status (#12213)
In trying to use the MSC3026 busy presence status, the user's status
would be set back to 'online' next time they synced. This change makes
it so that syncing does not affect a user's presence status if it
is currently set to 'busy': it must be removed through the presence
API.

The MSC defers to implementations on the behaviour of busy presence,
so this ought to remain compatible with the MSC.
2022-04-13 16:21:07 +01:00
Nick Mills-Barrett e3a49f4784
Fix missing sync events during historical batch imports (#12319)
Discovered after much in-depth investigation in #12281.

Closes: #12281
Closes: #3305

Signed off by: Nick Mills-Barrett nick@beeper.com
2022-04-13 11:38:35 +01:00
Erik Johnston aa28110264
Process device list updates asynchronously (#12365) 2022-04-12 16:50:40 +01:00
Patrick Cloke 9535fd0f9c
Disable groups/communities by default. (#12344)
This disables the endpoints (and sync response fields) for
groups/communities by default.
2022-04-12 10:20:46 -04:00
Patrick Cloke 86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
Patrick Cloke 4586119f0b
Add missing type hints to config classes. (#12402) 2022-04-11 12:07:23 -04:00
Patrick Cloke 772bad2562
Do not consider events by ignored users for bundled aggregations (#12235)
Consider the requester's ignored users when calculating the
bundled aggregations.

See #12285 / 4df10d3214
for corresponding changes for the `/relations` endpoint.
2022-04-11 10:09:57 -04:00
David Robertson 961ee75a9b
Disallow untyped defs in synapse._scripts (#12422)
Of note: 

* No untyped defs in `register_new_matrix_user`

This one might be contraversial. `request_registration` has three
dependency-injection arguments used for testing. I'm removing the
injection of the `requests` module and using `unitest.mock.patch` in the
test cases instead.

Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the
change:

```
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any"
synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any"
```

And after:

```
synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType"
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response"
```

* Drive-by comment in `synapse.storage.types`

* No untyped defs in `synapse_port_db`

This was by far the most painful. I'm happy to break this up into
smaller pieces for review if it's not managable as-is.
2022-04-11 12:41:55 +01:00
reivilibre 85ca963c1a
Add Module API for reading and writing global account data. (#12391) 2022-04-11 10:05:43 +01:00
reivilibre e630722f11
Optimise `_update_client_ips_batch_txn` to batch together database operations. (#12252)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 15:29:13 +01:00
Jorge Florian 2e2d8cc2f9
Update the server notices user profile in room if changed. (#12115) 2022-04-08 07:51:27 -04:00
Patrick Cloke 0bcb651b3f
Support the v1 endpoint for `/relations`. (#12403)
Now that MSC2675 has passed FCP and the implementation is
compliant with the final version.
2022-04-07 07:08:23 -04:00
Eric Eastwood 793d03e2c5
Generate historic pagination token for `/messages` when no `?from` token provided (#12370) 2022-04-06 11:40:28 +01:00
Sean Quah 79e7c2c426
Fix edge case where a `Linearizer` could get stuck (#12358)
Just after a task acquires a contended `Linearizer` lock, it sleeps.
If the task is cancelled during this sleep, we need to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 17:19:16 +01:00
Sean Quah 9c4c49991d
Update docstrings for `ReadWriteLock` tests (#12354)
Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 16:54:40 +01:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
Sean Quah 41b5f72677
Convert `Linearizer` tests from `inlineCallbacks` to async (#12353)
Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 14:56:09 +01:00