Commit Graph

157 Commits (d56f48038a07fd76d2ce08220a4061f85006bf3b)

Author SHA1 Message Date
Patrick Cloke 6d47b7e325
Add a type hint for `get_device_handler()` and fix incorrect types. (#14055)
This was the last untyped handler from the HomeServer object. Since
it was being treated as Any (and thus unchecked) it was being used
incorrectly in a few places.
2022-11-22 14:08:04 -05:00
Sean Quah 9cae44f49e
Track unconverted device list outbound pokes using a position instead (#14516)
When a local device list change is added to
`device_lists_changes_in_room`, the `converted_to_destinations` flag is
set to `FALSE` and the `_handle_new_device_update_async` background
process is started. This background process looks for unconverted rows
in `device_lists_changes_in_room`, copies them to
`device_lists_outbound_pokes` and updates the flag.

To update the `converted_to_destinations` flag, the database performs a
`DELETE` and `INSERT` internally, which fragments the table. To avoid
this, track unconverted rows using a `(stream ID, room ID)` position
instead of the flag.

From now on, the `converted_to_destinations` column indicates rows that
need converting to outbound pokes, but does not indicate whether the
conversion has already taken place.

Closes #14037.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-22 16:46:52 +00:00
David Robertson c3a4780080
When restarting a partial join resync, prioritise the server which actioned a partial join (#14126) 2022-10-18 12:33:18 +01:00
Nick Mills-Barrett a466164647
Optimise get_rooms_for_user (drop with_stream_ordering) (#13787) 2022-09-29 13:55:12 +00:00
Erik Johnston 5f659d4a88
Handle local device list updates during partial join (#13934) 2022-09-28 23:22:35 +01:00
Erik Johnston 4b17a5ace8
Handle remote device list updates during partial join (#13913)
c.f. #12993 (comment), point 3

This stores all device list updates that we receive while partial joins are ongoing, and processes them once we have the full state.

Note: We don't actually process the device lists in the same ways as if we weren't partially joined. Instead of updating the device list remote cache, we simply notify local users that a change in the remote user's devices has happened. I think this is safe as if the local user requests the keys for the remote user and we don't have them we'll simply fetch them as normal.
2022-09-28 13:42:43 +00:00
Sean Quah f49f73c0da
Faster room joins: Avoid blocking `/keys/changes` (#13888)
Part of the work for #12993.

Once #12993 is fully resolved, we expect `/keys/changes` to behave
sensibly when joined to a room with partial state.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-23 17:55:15 +01:00
Sean Quah 03c2bfb7f8
Send device list updates out to servers in partially joined rooms (#13874)
Use the provided list of servers in the room from the `/send_join`
response, since we will not know which users are in the room.  This
isn't sufficient to ensure that all remote servers receive the right
device list updates, since the `/send_join` response may be inaccurate
or we may calculate the membership state of new users in the room
incorrectly.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-23 13:44:03 +01:00
Sean Quah c73774467e
Fix bug in device list caching when remote users leave rooms (#13749)
When a remote user leaves the last room shared with the homeserver, we
have to mark their device list as unsubscribed, otherwise we would hold
on to a stale device list in our cache. Crucially, the device list would
remain cached even after the remote user rejoined the room, which could
lead to E2EE failures until the next change to the remote user's device
list.

Fixes #13651.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-14 10:42:57 +01:00
reivilibre d3d9ca156e
Cancel the processing of key query requests when they time out. (#13680) 2022-09-07 12:03:32 +01:00
Eric Eastwood 1a209efdb2
Update `get_users_in_room` mis-use to get hosts with dedicated `get_current_hosts_in_room` (#13605)
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:15:37 -05:00
Andrew Morgan f9f03426de
Implement MSC3852: Expose `last_seen_user_agent` to users for their own devices; also expose to Admin API (#13549) 2022-08-19 16:17:10 +00:00
Patrick Cloke 50122754c8
Add missing types to opentracing. (#13345)
After this change `synapse.logging` is fully typed.
2022-07-21 12:01:52 +00:00
Erik Johnston 5099b5ecc7
Use new `device_list_changes_in_room` table when getting device list changes (#13045) 2022-06-17 11:42:03 +01:00
Patrick Cloke 9dc3293e0b
Consolidate the logic of delete_device/delete_devices. (#12970)
By always using delete_devices and sometimes passing a list
with a single device ID.

Previously these methods had gotten out of sync with each
other and it seems there's little benefit to the single-device
variant.
2022-06-07 07:43:35 -04:00
Erik Johnston 888a29f412
Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
Erik Johnston 1e453053cb
Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
Brendan Abolivier 28989cb301
Add a background job to automatically delete stale devices (#12855)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-05-27 17:47:32 +02:00
Patrick Cloke c52abc1cfd
Additional constants for EDU types. (#12884)
Instead of hard-coding strings in many places.
2022-05-27 07:14:36 -04:00
Erik Johnston 4660d9fdcf
Fix up `state_store` naming (#12871) 2022-05-25 12:59:04 +01:00
Carl Bordum Hansen 774ac4930d
Make sure `prev_ids` defaults to empty list (#12829)
Signed-off-by: Carl Bordum Hansen <carl@bordum.dk>
2022-05-25 09:14:45 +00:00
Andrew Morgan 83be72d76c
Add `StreamKeyType` class and replace string literals with constants (#12567) 2022-05-16 15:35:31 +00:00
Erik Johnston f59e3f4c90
Mark remote device list updates as already handled (#12557) 2022-04-26 17:07:21 +01:00
David Robertson 6d89f1239c
Comment out dodgy log-kv (#12554) 2022-04-26 15:53:06 +01:00
Erik Johnston 0b014eb25e
Only send out device list updates for our own users (#12465)
Broke in #12365
2022-04-14 13:05:31 +01:00
Erik Johnston aa28110264
Process device list updates asynchronously (#12365) 2022-04-12 16:50:40 +01:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
Erik Johnston 5c9e39e619
Track device list updates per room. (#12321)
This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.

There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
2022-04-04 15:25:20 +01:00
Patrick Cloke bc9dff1d95
Remove unnecessary pass statements. (#12206) 2022-03-11 07:06:21 -05:00
Erik Johnston 423cca9efe
Spread out sending device lists to remote hosts (#12132) 2022-03-04 11:48:15 +00:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Andrew Morgan 55113dd5e8
Notify users, rather than rooms, of device list updates (#11905)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-12 09:33:49 -05:00
David Robertson 88a78c6577
Cache empty responses from `/user/devices` (#11587)
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
2022-01-05 13:33:28 +00:00
reivilibre e5cdb9e233
Make `get_device` return None if the device doesn't exist rather than raising an exception. (#11565)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-12-13 15:39:43 +00:00
Quentin Gliech a15a893df8
Save the OIDC session ID (sid) with the device on login (#11482)
As a step towards allowing back-channel logout for OIDC.
2021-12-06 12:43:06 -05:00
Andrew Morgan 6408372234
Improve docstrings for methods related to sending EDUs to application services (#11138) 2021-10-21 17:42:25 +01:00
Sean Quah 36224e056a
Add type hints to `synapse.storage.databases.main.client_ips` (#10972) 2021-10-12 13:50:34 +01:00
Patrick Cloke eb9ddc8c2e
Remove the deprecated BaseHandler. (#11005)
The shared ratelimit function was replaced with a dedicated
RequestRatelimiter class (accessible from the HomeServer
object).

Other properties were copied to each sub-class that inherited
from BaseHandler.
2021-10-08 07:44:43 -04:00
Patrick Cloke b3590614da
Require type hints in the handlers module. (#10831)
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.

This also removes the unused construct_auth_difference method
from the FederationHandler.
2021-09-20 08:56:23 -04:00
Jonathan de Jong 98aec1cc9d
Use inline type hints in `handlers/` and `rest/`. (#10382) 2021-07-16 18:22:36 +01:00
Richard van der Hoff 69018acbd2
Clear the resync bit after resyncing device lists (#9867)
Fixes #9866.
2021-04-22 16:53:24 +01:00
Richard van der Hoff 294c675033
Remove `synapse.types.Collection` (#9856)
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Jonathan de Jong 495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke 44bb881096
Add type hints to expiring cache. (#9730) 2021-04-06 08:58:18 -04:00
Patrick Cloke b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Hubert Chathi 73dbce5523
only save remote cross-signing keys if they're different from the current ones (#9634)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-17 11:04:57 -04:00
Erik Johnston ad721fc559
Fix bad naming of storage function (#9637)
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
2021-03-17 13:20:08 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke a78016dadf
Add type hints to E2E handler. (#9232)
This finishes adding type hints to the `synapse.handlers` module.
2021-01-28 08:34:19 -05:00