Merge remote-tracking branch 'origin/release-v1.58' into matrix-org-hotfixes
commit
474a964fb7
72
CHANGES.md
72
CHANGES.md
|
@ -1,3 +1,75 @@
|
||||||
|
Synapse 1.58.0rc1 (2022-04-26)
|
||||||
|
==============================
|
||||||
|
|
||||||
|
As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](https://github.com/matrix-org/synapse/issues/11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.
|
||||||
|
|
||||||
|
Features
|
||||||
|
--------
|
||||||
|
|
||||||
|
- Implement [MSC3383](https://github.com/matrix-org/matrix-spec-proposals/pull/3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](https://github.com/matrix-org/synapse/issues/11398))
|
||||||
|
- Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](https://github.com/matrix-org/synapse/issues/11537))
|
||||||
|
- Enable processing of device list updates asynchronously. ([\#12365](https://github.com/matrix-org/synapse/issues/12365), [\#12465](https://github.com/matrix-org/synapse/issues/12465))
|
||||||
|
- Implement [MSC2815](https://github.com/matrix-org/matrix-spec-proposals/pull/2815) to allow room moderators to view redacted event content. Contributed by @tulir. ([\#12427](https://github.com/matrix-org/synapse/issues/12427))
|
||||||
|
- Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](https://github.com/matrix-org/synapse/issues/12543))
|
||||||
|
|
||||||
|
|
||||||
|
Bugfixes
|
||||||
|
--------
|
||||||
|
|
||||||
|
- Prevent a sync request from removing a user's busy presence status. ([\#12213](https://github.com/matrix-org/synapse/issues/12213))
|
||||||
|
- Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](https://github.com/matrix-org/synapse/issues/12319))
|
||||||
|
- Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](https://github.com/matrix-org/synapse/issues/12476))
|
||||||
|
- Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](https://github.com/matrix-org/synapse/issues/12496))
|
||||||
|
- Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](https://github.com/matrix-org/synapse/issues/12510))
|
||||||
|
- Fix a bug in the implementation of [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](https://github.com/matrix-org/synapse/issues/12520))
|
||||||
|
- Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](https://github.com/matrix-org/synapse/issues/12522))
|
||||||
|
|
||||||
|
|
||||||
|
Improved Documentation
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
- Fix rendering of the documentation site when using the 'print' feature. ([\#12340](https://github.com/matrix-org/synapse/issues/12340))
|
||||||
|
- Add a manual documenting config file options. ([\#12368](https://github.com/matrix-org/synapse/issues/12368), [\#12527](https://github.com/matrix-org/synapse/issues/12527))
|
||||||
|
- Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](https://github.com/matrix-org/synapse/issues/12451))
|
||||||
|
- Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](https://github.com/matrix-org/synapse/issues/12457))
|
||||||
|
- Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](https://github.com/matrix-org/synapse/issues/12475))
|
||||||
|
- Add some example configurations for workers and update architectural diagram. ([\#12492](https://github.com/matrix-org/synapse/issues/12492))
|
||||||
|
- Fix a broken link in `README.rst`. ([\#12495](https://github.com/matrix-org/synapse/issues/12495))
|
||||||
|
- Add HAProxy delegation example with CORS headers to docs. ([\#12501](https://github.com/matrix-org/synapse/issues/12501))
|
||||||
|
- Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](https://github.com/matrix-org/synapse/issues/12533))
|
||||||
|
|
||||||
|
|
||||||
|
Deprecations and Removals
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
- The groups/communities feature in Synapse is now disabled by default. ([\#12344](https://github.com/matrix-org/synapse/issues/12344))
|
||||||
|
- Remove unstable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#12382](https://github.com/matrix-org/synapse/issues/12382))
|
||||||
|
|
||||||
|
|
||||||
|
Internal Changes
|
||||||
|
----------------
|
||||||
|
|
||||||
|
- Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](https://github.com/matrix-org/synapse/issues/12394))
|
||||||
|
- Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](https://github.com/matrix-org/synapse/issues/12399))
|
||||||
|
- Remove an unstable identifier from [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083). ([\#12395](https://github.com/matrix-org/synapse/issues/12395))
|
||||||
|
- Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](https://github.com/matrix-org/synapse/issues/12425), [\#12434](https://github.com/matrix-org/synapse/issues/12434), [\#12438](https://github.com/matrix-org/synapse/issues/12438), [\#12441](https://github.com/matrix-org/synapse/issues/12441), [\#12449](https://github.com/matrix-org/synapse/issues/12449), [\#12478](https://github.com/matrix-org/synapse/issues/12478), [\#12514](https://github.com/matrix-org/synapse/issues/12514), [\#12472](https://github.com/matrix-org/synapse/issues/12472))
|
||||||
|
- Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](https://github.com/matrix-org/matrix-spec-proposals/pull/2666). ([\#12445](https://github.com/matrix-org/synapse/issues/12445))
|
||||||
|
- Fix typo in the release script help string. ([\#12450](https://github.com/matrix-org/synapse/issues/12450))
|
||||||
|
- Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](https://github.com/matrix-org/synapse/issues/12497))
|
||||||
|
- Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](https://github.com/matrix-org/synapse/issues/12455))
|
||||||
|
- Limit length of `device_id` to less than 512 characters. ([\#12454](https://github.com/matrix-org/synapse/issues/12454))
|
||||||
|
- Dockerfile-workers: reduce the amount we install in the image. ([\#12464](https://github.com/matrix-org/synapse/issues/12464))
|
||||||
|
- Dockerfile-workers: give the master its own log config. ([\#12466](https://github.com/matrix-org/synapse/issues/12466))
|
||||||
|
- complement-synapse-workers: factor out separate entry point script. ([\#12467](https://github.com/matrix-org/synapse/issues/12467))
|
||||||
|
- Back out experimental implementation of [MSC2314](https://github.com/matrix-org/matrix-spec-proposals/pull/2314). ([\#12474](https://github.com/matrix-org/synapse/issues/12474))
|
||||||
|
- Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](https://github.com/matrix-org/synapse/issues/12483))
|
||||||
|
- Remove unnecessary configuration overrides in tests. ([\#12511](https://github.com/matrix-org/synapse/issues/12511))
|
||||||
|
- Refactor the relations code for clarity. ([\#12519](https://github.com/matrix-org/synapse/issues/12519))
|
||||||
|
- Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](https://github.com/matrix-org/synapse/issues/12528))
|
||||||
|
- Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](https://github.com/matrix-org/synapse/issues/12468))
|
||||||
|
- Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](https://github.com/matrix-org/synapse/issues/12529))
|
||||||
|
|
||||||
|
|
||||||
Synapse 1.57.1 (2022-04-20)
|
Synapse 1.57.1 (2022-04-20)
|
||||||
===========================
|
===========================
|
||||||
|
|
||||||
|
|
|
@ -1 +0,0 @@
|
||||||
Implement [MSC3383](https://github.com/matrix-org/matrix-spec-proposals/pull/3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly GmbH.
|
|
|
@ -1 +0,0 @@
|
||||||
Prevent a sync request from removing a user's busy presence status.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper.
|
|
|
@ -1 +0,0 @@
|
||||||
Use poetry to manage Synapse's dependencies.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix rendering of the documentation site when using the 'print' feature.
|
|
|
@ -1 +0,0 @@
|
||||||
The groups/communities feature in Synapse has been disabled by default.
|
|
|
@ -1 +0,0 @@
|
||||||
Enable processing of device list updates asynchronously.
|
|
|
@ -1 +0,0 @@
|
||||||
Add a manual documenting config file options.
|
|
|
@ -1 +0,0 @@
|
||||||
Remove unstable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).
|
|
|
@ -1 +0,0 @@
|
||||||
Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join.
|
|
|
@ -1 +0,0 @@
|
||||||
Remove an unstable identifier from [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083).
|
|
|
@ -1 +0,0 @@
|
||||||
Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive.
|
|
|
@ -1 +0,0 @@
|
||||||
Run twisted trunk CI job in the locked poetry environment.
|
|
|
@ -1 +0,0 @@
|
||||||
Implement [MSC2815](https://github.com/matrix-org/matrix-spec-proposals/pull/2815) to allow room moderators to view redacted event content. Contributed by @tulir.
|
|
|
@ -1 +0,0 @@
|
||||||
Run lints under poetry in CI, and remove corresponding tox lint jobs.
|
|
|
@ -1 +0,0 @@
|
||||||
Run "main" trial tests under `poetry`.
|
|
|
@ -1 +0,0 @@
|
||||||
Bump twisted version in `poetry.lock` to work around [pip bug #9644](https://github.com/pypa/pip/issues/9644).
|
|
|
@ -1 +0,0 @@
|
||||||
Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current MSC iteration.
|
|
|
@ -1 +0,0 @@
|
||||||
Use `poetry` to manage the virtualenv in debian packages.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix typo in the release script help string.
|
|
|
@ -1 +0,0 @@
|
||||||
Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental.
|
|
|
@ -1 +0,0 @@
|
||||||
Limit length of device_id to less than 512 characters.
|
|
|
@ -1 +0,0 @@
|
||||||
Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development.
|
|
|
@ -1 +0,0 @@
|
||||||
Update worker documentation and replace old `federation_reader` with `generic_worker`.
|
|
|
@ -1 +0,0 @@
|
||||||
Dockerfile-workers: reduce the amount we install in the image.
|
|
|
@ -1 +0,0 @@
|
||||||
Enable processing of device list updates asynchronously.
|
|
|
@ -1 +0,0 @@
|
||||||
Dockerfile-workers: give the master its own log config.
|
|
|
@ -1 +0,0 @@
|
||||||
complement-synapse-workers: factor out separate entry point script.
|
|
|
@ -1 +0,0 @@
|
||||||
Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s.
|
|
|
@ -1 +0,0 @@
|
||||||
Add a CI job which tests Synapse against the latest version of all dependencies.
|
|
|
@ -1 +0,0 @@
|
||||||
Back out experimental implementation of [MSC2314](https://github.com/matrix-org/matrix-spec-proposals/pull/2314).
|
|
|
@ -1 +0,0 @@
|
||||||
Strongly recommend `poetry` for development.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix a long-standing bug which incorrectly caused `GET /_matrix/client/r3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original.
|
|
|
@ -1 +0,0 @@
|
||||||
Use poetry-core instead of setuptools to build wheels.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix grammatical error in federation error response when the room version of a room is unknown.
|
|
|
@ -1 +0,0 @@
|
||||||
Add some example configurations for workers and update architectural diagram.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix a broken link in `README.rst`.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix bug where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1 no matter how many extremities were deleted. Broke in v1.27.0.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix a minor typo in the Debian changelogs generated by the release script.
|
|
|
@ -1 +0,0 @@
|
||||||
Add HAProxy delegation example with CORS headers to docs.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix a long-standing bug where the image thumbanils embedded into email notifications were broken.
|
|
|
@ -1 +0,0 @@
|
||||||
Remove unnecessary configuration overrides in tests.
|
|
|
@ -1 +0,0 @@
|
||||||
Use poetry-core instead of setuptools to build wheels.
|
|
|
@ -1 +0,0 @@
|
||||||
Refactor the relations code for clarity.
|
|
|
@ -1 +0,0 @@
|
||||||
Fix a bug in the implementation of MSC3202 where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`.
|
|
|
@ -1 +0,0 @@
|
||||||
Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`.
|
|
|
@ -1 +0,0 @@
|
||||||
Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari.
|
|
|
@ -1,8 +1,9 @@
|
||||||
matrix-synapse-py3 (1.58.0+nmu1) UNRELEASED; urgency=medium
|
matrix-synapse-py3 (1.58.0~rc1) stable; urgency=medium
|
||||||
|
|
||||||
* Use poetry to manage the bundled virtualenv included with this package.
|
* Use poetry to manage the bundled virtualenv included with this package.
|
||||||
|
* New Synapse release 1.58.0rc1.
|
||||||
|
|
||||||
-- Synapse Packaging team <packages@matrix.org> Wed, 30 Mar 2022 12:21:43 +0100
|
-- Synapse Packaging team <packages@matrix.org> Tue, 26 Apr 2022 11:15:20 +0100
|
||||||
|
|
||||||
matrix-synapse-py3 (1.57.1) stable; urgency=medium
|
matrix-synapse-py3 (1.57.1) stable; urgency=medium
|
||||||
|
|
||||||
|
|
|
@ -64,7 +64,49 @@ apply if you want your config file to be read properly. A few helpful things to
|
||||||
In addition, each setting has an example of its usage, with the proper indentation
|
In addition, each setting has an example of its usage, with the proper indentation
|
||||||
shown.
|
shown.
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
[Modules](#modules)
|
||||||
|
|
||||||
|
[Server](#server)
|
||||||
|
|
||||||
|
[Homeserver Blocking](#homeserver-blocking)
|
||||||
|
|
||||||
|
[TLS](#tls)
|
||||||
|
|
||||||
|
[Federation](#federation)
|
||||||
|
|
||||||
|
[Caching](#caching)
|
||||||
|
|
||||||
|
[Database](#database)
|
||||||
|
|
||||||
|
[Logging](#logging)
|
||||||
|
|
||||||
|
[Ratelimiting](#ratelimiting)
|
||||||
|
|
||||||
|
[Media Store](#media-store)
|
||||||
|
|
||||||
|
[Captcha](#captcha)
|
||||||
|
|
||||||
|
[TURN](#turn)
|
||||||
|
|
||||||
|
[Registration](#registration)
|
||||||
|
|
||||||
|
[API Configuration](#api-configuration)
|
||||||
|
|
||||||
|
[Signing Keys](#signing-keys)
|
||||||
|
|
||||||
|
[Single Sign On Integration](#single-sign-on-integration)
|
||||||
|
|
||||||
|
[Push](#push)
|
||||||
|
|
||||||
|
[Rooms](#rooms)
|
||||||
|
|
||||||
|
[Opentracing](#opentracing)
|
||||||
|
|
||||||
|
[Workers](#workers)
|
||||||
|
|
||||||
|
[Background Updates](#background-updates)
|
||||||
|
|
||||||
## Modules
|
## Modules
|
||||||
|
|
||||||
Server admins can expand Synapse's functionality with external modules.
|
Server admins can expand Synapse's functionality with external modules.
|
||||||
|
@ -3409,4 +3451,4 @@ background_updates:
|
||||||
sleep_duration_ms: 300
|
sleep_duration_ms: 300
|
||||||
min_batch_size: 10
|
min_batch_size: 10
|
||||||
default_batch_size: 50
|
default_batch_size: 50
|
||||||
```
|
```
|
||||||
|
|
|
@ -1,11 +1,3 @@
|
||||||
[[package]]
|
|
||||||
name = "appdirs"
|
|
||||||
version = "1.4.4"
|
|
||||||
description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
|
|
||||||
category = "dev"
|
|
||||||
optional = false
|
|
||||||
python-versions = "*"
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "attrs"
|
name = "attrs"
|
||||||
version = "21.4.0"
|
version = "21.4.0"
|
||||||
|
@ -49,17 +41,6 @@ six = "*"
|
||||||
[package.extras]
|
[package.extras]
|
||||||
visualize = ["graphviz (>0.5.1)", "Twisted (>=16.1.1)"]
|
visualize = ["graphviz (>0.5.1)", "Twisted (>=16.1.1)"]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "baron"
|
|
||||||
version = "0.10.1"
|
|
||||||
description = "Full Syntax Tree for python to make writing refactoring code a realist task"
|
|
||||||
category = "dev"
|
|
||||||
optional = false
|
|
||||||
python-versions = "*"
|
|
||||||
|
|
||||||
[package.dependencies]
|
|
||||||
rply = "*"
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "bcrypt"
|
name = "bcrypt"
|
||||||
version = "3.2.0"
|
version = "3.2.0"
|
||||||
|
@ -984,20 +965,6 @@ Pygments = ">=2.5.1"
|
||||||
[package.extras]
|
[package.extras]
|
||||||
md = ["cmarkgfm (>=0.8.0)"]
|
md = ["cmarkgfm (>=0.8.0)"]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "redbaron"
|
|
||||||
version = "0.9.2"
|
|
||||||
description = "Abstraction on top of baron, a FST for python to make writing refactoring code a realistic task"
|
|
||||||
category = "dev"
|
|
||||||
optional = false
|
|
||||||
python-versions = "*"
|
|
||||||
|
|
||||||
[package.dependencies]
|
|
||||||
baron = ">=0.7"
|
|
||||||
|
|
||||||
[package.extras]
|
|
||||||
notebook = ["pygments"]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "requests"
|
name = "requests"
|
||||||
version = "2.27.1"
|
version = "2.27.1"
|
||||||
|
@ -1038,17 +1005,6 @@ python-versions = ">=3.7"
|
||||||
[package.extras]
|
[package.extras]
|
||||||
idna2008 = ["idna"]
|
idna2008 = ["idna"]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "rply"
|
|
||||||
version = "0.7.8"
|
|
||||||
description = "A pure Python Lex/Yacc that works with RPython"
|
|
||||||
category = "dev"
|
|
||||||
optional = false
|
|
||||||
python-versions = "*"
|
|
||||||
|
|
||||||
[package.dependencies]
|
|
||||||
appdirs = "*"
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "secretstorage"
|
name = "secretstorage"
|
||||||
version = "3.3.1"
|
version = "3.3.1"
|
||||||
|
@ -1597,13 +1553,9 @@ url_preview = ["lxml"]
|
||||||
[metadata]
|
[metadata]
|
||||||
lock-version = "1.1"
|
lock-version = "1.1"
|
||||||
python-versions = "^3.7"
|
python-versions = "^3.7"
|
||||||
content-hash = "964ad29eaf7fd02749a4e735818f3bc0ba729c2f4b9e3213f0daa02643508b16"
|
content-hash = "f482a4f594a165dfe01ce253a22510d5faf38647ab0dcebc35789350cafd9bf0"
|
||||||
|
|
||||||
[metadata.files]
|
[metadata.files]
|
||||||
appdirs = [
|
|
||||||
{file = "appdirs-1.4.4-py2.py3-none-any.whl", hash = "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"},
|
|
||||||
{file = "appdirs-1.4.4.tar.gz", hash = "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41"},
|
|
||||||
]
|
|
||||||
attrs = [
|
attrs = [
|
||||||
{file = "attrs-21.4.0-py2.py3-none-any.whl", hash = "sha256:2d27e3784d7a565d36ab851fe94887c5eccd6a463168875832a1be79c82828b4"},
|
{file = "attrs-21.4.0-py2.py3-none-any.whl", hash = "sha256:2d27e3784d7a565d36ab851fe94887c5eccd6a463168875832a1be79c82828b4"},
|
||||||
{file = "attrs-21.4.0.tar.gz", hash = "sha256:626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd"},
|
{file = "attrs-21.4.0.tar.gz", hash = "sha256:626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd"},
|
||||||
|
@ -1616,10 +1568,6 @@ automat = [
|
||||||
{file = "Automat-20.2.0-py2.py3-none-any.whl", hash = "sha256:b6feb6455337df834f6c9962d6ccf771515b7d939bca142b29c20c2376bc6111"},
|
{file = "Automat-20.2.0-py2.py3-none-any.whl", hash = "sha256:b6feb6455337df834f6c9962d6ccf771515b7d939bca142b29c20c2376bc6111"},
|
||||||
{file = "Automat-20.2.0.tar.gz", hash = "sha256:7979803c74610e11ef0c0d68a2942b152df52da55336e0c9d58daf1831cbdf33"},
|
{file = "Automat-20.2.0.tar.gz", hash = "sha256:7979803c74610e11ef0c0d68a2942b152df52da55336e0c9d58daf1831cbdf33"},
|
||||||
]
|
]
|
||||||
baron = [
|
|
||||||
{file = "baron-0.10.1-py2.py3-none-any.whl", hash = "sha256:befb33f4b9e832c7cd1e3cf0eafa6dd3cb6ed4cb2544245147c019936f4e0a8a"},
|
|
||||||
{file = "baron-0.10.1.tar.gz", hash = "sha256:af822ad44d4eb425c8516df4239ac4fdba9fdb398ef77e4924cd7c9b4045bc2f"},
|
|
||||||
]
|
|
||||||
bcrypt = [
|
bcrypt = [
|
||||||
{file = "bcrypt-3.2.0-cp36-abi3-macosx_10_10_universal2.whl", hash = "sha256:b589229207630484aefe5899122fb938a5b017b0f4349f769b8c13e78d99a8fd"},
|
{file = "bcrypt-3.2.0-cp36-abi3-macosx_10_10_universal2.whl", hash = "sha256:b589229207630484aefe5899122fb938a5b017b0f4349f769b8c13e78d99a8fd"},
|
||||||
{file = "bcrypt-3.2.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:c95d4cbebffafcdd28bd28bb4e25b31c50f6da605c81ffd9ad8a3d1b2ab7b1b6"},
|
{file = "bcrypt-3.2.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:c95d4cbebffafcdd28bd28bb4e25b31c50f6da605c81ffd9ad8a3d1b2ab7b1b6"},
|
||||||
|
@ -2412,10 +2360,6 @@ readme-renderer = [
|
||||||
{file = "readme_renderer-33.0-py3-none-any.whl", hash = "sha256:f02cee0c4de9636b5a62b6be50c9742427ba1b956aad1d938bfb087d0d72ccdf"},
|
{file = "readme_renderer-33.0-py3-none-any.whl", hash = "sha256:f02cee0c4de9636b5a62b6be50c9742427ba1b956aad1d938bfb087d0d72ccdf"},
|
||||||
{file = "readme_renderer-33.0.tar.gz", hash = "sha256:e3b53bc84bd6af054e4cc1fe3567dc1ae19f554134221043a3f8c674e22209db"},
|
{file = "readme_renderer-33.0.tar.gz", hash = "sha256:e3b53bc84bd6af054e4cc1fe3567dc1ae19f554134221043a3f8c674e22209db"},
|
||||||
]
|
]
|
||||||
redbaron = [
|
|
||||||
{file = "redbaron-0.9.2-py2.py3-none-any.whl", hash = "sha256:d01032b6a848b5521a8d6ef72486315c2880f420956870cdd742e2b5a09b9bab"},
|
|
||||||
{file = "redbaron-0.9.2.tar.gz", hash = "sha256:472d0739ca6b2240bb2278ae428604a75472c9c12e86c6321e8c016139c0132f"},
|
|
||||||
]
|
|
||||||
requests = [
|
requests = [
|
||||||
{file = "requests-2.27.1-py2.py3-none-any.whl", hash = "sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d"},
|
{file = "requests-2.27.1-py2.py3-none-any.whl", hash = "sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d"},
|
||||||
{file = "requests-2.27.1.tar.gz", hash = "sha256:68d7c56fd5a8999887728ef304a6d12edc7be74f1cfa47714fc8b414525c9a61"},
|
{file = "requests-2.27.1.tar.gz", hash = "sha256:68d7c56fd5a8999887728ef304a6d12edc7be74f1cfa47714fc8b414525c9a61"},
|
||||||
|
@ -2428,10 +2372,6 @@ rfc3986 = [
|
||||||
{file = "rfc3986-2.0.0-py2.py3-none-any.whl", hash = "sha256:50b1502b60e289cb37883f3dfd34532b8873c7de9f49bb546641ce9cbd256ebd"},
|
{file = "rfc3986-2.0.0-py2.py3-none-any.whl", hash = "sha256:50b1502b60e289cb37883f3dfd34532b8873c7de9f49bb546641ce9cbd256ebd"},
|
||||||
{file = "rfc3986-2.0.0.tar.gz", hash = "sha256:97aacf9dbd4bfd829baad6e6309fa6573aaf1be3f6fa735c8ab05e46cecb261c"},
|
{file = "rfc3986-2.0.0.tar.gz", hash = "sha256:97aacf9dbd4bfd829baad6e6309fa6573aaf1be3f6fa735c8ab05e46cecb261c"},
|
||||||
]
|
]
|
||||||
rply = [
|
|
||||||
{file = "rply-0.7.8-py2.py3-none-any.whl", hash = "sha256:28ffd11d656c48aeb8c508eb382acd6a0bd906662624b34388751732a27807e7"},
|
|
||||||
{file = "rply-0.7.8.tar.gz", hash = "sha256:2a808ac25a4580a9991fc304d64434e299a8fc75760574492f242cbb5bb301c9"},
|
|
||||||
]
|
|
||||||
secretstorage = [
|
secretstorage = [
|
||||||
{file = "SecretStorage-3.3.1-py3-none-any.whl", hash = "sha256:422d82c36172d88d6a0ed5afdec956514b189ddbfb72fefab0c8a1cee4eaf71f"},
|
{file = "SecretStorage-3.3.1-py3-none-any.whl", hash = "sha256:422d82c36172d88d6a0ed5afdec956514b189ddbfb72fefab0c8a1cee4eaf71f"},
|
||||||
{file = "SecretStorage-3.3.1.tar.gz", hash = "sha256:fd666c51a6bf200643495a04abb261f83229dcb6fd8472ec393df7ffc8b6f195"},
|
{file = "SecretStorage-3.3.1.tar.gz", hash = "sha256:fd666c51a6bf200643495a04abb261f83229dcb6fd8472ec393df7ffc8b6f195"},
|
||||||
|
|
|
@ -54,7 +54,7 @@ skip_gitignore = true
|
||||||
|
|
||||||
[tool.poetry]
|
[tool.poetry]
|
||||||
name = "matrix-synapse"
|
name = "matrix-synapse"
|
||||||
version = "1.57.1"
|
version = "1.58.0rc1"
|
||||||
description = "Homeserver for the Matrix decentralised comms protocol"
|
description = "Homeserver for the Matrix decentralised comms protocol"
|
||||||
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
|
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
|
||||||
license = "Apache-2.0"
|
license = "Apache-2.0"
|
||||||
|
@ -270,7 +270,6 @@ idna = ">=2.5"
|
||||||
|
|
||||||
# The following are used by the release script
|
# The following are used by the release script
|
||||||
click = "==8.1.0"
|
click = "==8.1.0"
|
||||||
redbaron = "==0.9.2"
|
|
||||||
GitPython = "==3.1.14"
|
GitPython = "==3.1.14"
|
||||||
commonmark = "==0.9.1"
|
commonmark = "==0.9.1"
|
||||||
pygithub = "==1.55"
|
pygithub = "==1.55"
|
||||||
|
|
|
@ -26,6 +26,7 @@ DISTS = (
|
||||||
"debian:sid",
|
"debian:sid",
|
||||||
"ubuntu:focal", # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
|
"ubuntu:focal", # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
|
||||||
"ubuntu:impish", # 21.10 (EOL 2022-07)
|
"ubuntu:impish", # 21.10 (EOL 2022-07)
|
||||||
|
"ubuntu:jammy", # 22.04 LTS (EOL 2027-04)
|
||||||
)
|
)
|
||||||
|
|
||||||
DESC = """\
|
DESC = """\
|
||||||
|
|
|
@ -25,13 +25,12 @@ import sys
|
||||||
import urllib.request
|
import urllib.request
|
||||||
from os import path
|
from os import path
|
||||||
from tempfile import TemporaryDirectory
|
from tempfile import TemporaryDirectory
|
||||||
from typing import List, Optional, Tuple
|
from typing import List, Optional
|
||||||
|
|
||||||
import attr
|
import attr
|
||||||
import click
|
import click
|
||||||
import commonmark
|
import commonmark
|
||||||
import git
|
import git
|
||||||
import redbaron
|
|
||||||
from click.exceptions import ClickException
|
from click.exceptions import ClickException
|
||||||
from github import Github
|
from github import Github
|
||||||
from packaging import version
|
from packaging import version
|
||||||
|
@ -100,7 +99,7 @@ def prepare():
|
||||||
repo.remote().fetch()
|
repo.remote().fetch()
|
||||||
|
|
||||||
# Get the current version and AST from root Synapse module.
|
# Get the current version and AST from root Synapse module.
|
||||||
current_version, parsed_synapse_ast, version_node = parse_version_from_module()
|
current_version = get_package_version()
|
||||||
|
|
||||||
# Figure out what sort of release we're doing and calcuate the new version.
|
# Figure out what sort of release we're doing and calcuate the new version.
|
||||||
rc = click.confirm("RC", default=True)
|
rc = click.confirm("RC", default=True)
|
||||||
|
@ -162,7 +161,7 @@ def prepare():
|
||||||
click.get_current_context().abort()
|
click.get_current_context().abort()
|
||||||
|
|
||||||
# Switch to the release branch.
|
# Switch to the release branch.
|
||||||
parsed_new_version = version.parse(new_version)
|
parsed_new_version: version.Version = version.parse(new_version)
|
||||||
|
|
||||||
# We assume for debian changelogs that we only do RCs or full releases.
|
# We assume for debian changelogs that we only do RCs or full releases.
|
||||||
assert not parsed_new_version.is_devrelease
|
assert not parsed_new_version.is_devrelease
|
||||||
|
@ -207,17 +206,15 @@ def prepare():
|
||||||
# Create the new release branch
|
# Create the new release branch
|
||||||
release_branch = repo.create_head(release_branch_name, commit=base_branch)
|
release_branch = repo.create_head(release_branch_name, commit=base_branch)
|
||||||
|
|
||||||
# Switch to the release branch and ensure its up to date.
|
# Switch to the release branch and ensure it's up to date.
|
||||||
repo.git.checkout(release_branch_name)
|
repo.git.checkout(release_branch_name)
|
||||||
update_branch(repo)
|
update_branch(repo)
|
||||||
|
|
||||||
# Update the `__version__` variable and write it back to the file.
|
# Update the version specified in pyproject.toml.
|
||||||
version_node.value = '"' + new_version + '"'
|
subprocess.check_output(["poetry", "version", new_version])
|
||||||
with open("synapse/__init__.py", "w") as f:
|
|
||||||
f.write(parsed_synapse_ast.dumps())
|
|
||||||
|
|
||||||
# Generate changelogs.
|
# Generate changelogs.
|
||||||
generate_and_write_changelog(current_version)
|
generate_and_write_changelog(current_version, new_version)
|
||||||
|
|
||||||
# Generate debian changelogs
|
# Generate debian changelogs
|
||||||
if parsed_new_version.pre is not None:
|
if parsed_new_version.pre is not None:
|
||||||
|
@ -284,7 +281,7 @@ def tag(gh_token: Optional[str]):
|
||||||
repo.remote().fetch()
|
repo.remote().fetch()
|
||||||
|
|
||||||
# Find out the version and tag name.
|
# Find out the version and tag name.
|
||||||
current_version, _, _ = parse_version_from_module()
|
current_version = get_package_version()
|
||||||
tag_name = f"v{current_version}"
|
tag_name = f"v{current_version}"
|
||||||
|
|
||||||
# Check we haven't released this version.
|
# Check we haven't released this version.
|
||||||
|
@ -362,7 +359,7 @@ def publish(gh_token: str):
|
||||||
if repo.is_dirty():
|
if repo.is_dirty():
|
||||||
raise click.ClickException("Uncommitted changes exist.")
|
raise click.ClickException("Uncommitted changes exist.")
|
||||||
|
|
||||||
current_version, _, _ = parse_version_from_module()
|
current_version = get_package_version()
|
||||||
tag_name = f"v{current_version}"
|
tag_name = f"v{current_version}"
|
||||||
|
|
||||||
if not click.confirm(f"Publish {tag_name}?", default=True):
|
if not click.confirm(f"Publish {tag_name}?", default=True):
|
||||||
|
@ -396,7 +393,7 @@ def publish(gh_token: str):
|
||||||
def upload():
|
def upload():
|
||||||
"""Upload release to pypi."""
|
"""Upload release to pypi."""
|
||||||
|
|
||||||
current_version, _, _ = parse_version_from_module()
|
current_version = get_package_version()
|
||||||
tag_name = f"v{current_version}"
|
tag_name = f"v{current_version}"
|
||||||
|
|
||||||
pypi_asset_names = [
|
pypi_asset_names = [
|
||||||
|
@ -424,7 +421,7 @@ def upload():
|
||||||
def announce():
|
def announce():
|
||||||
"""Generate markdown to announce the release."""
|
"""Generate markdown to announce the release."""
|
||||||
|
|
||||||
current_version, _, _ = parse_version_from_module()
|
current_version = get_package_version()
|
||||||
tag_name = f"v{current_version}"
|
tag_name = f"v{current_version}"
|
||||||
|
|
||||||
click.echo(
|
click.echo(
|
||||||
|
@ -455,37 +452,11 @@ Announce the release in
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def parse_version_from_module() -> Tuple[
|
def get_package_version() -> version.Version:
|
||||||
version.Version, redbaron.RedBaron, redbaron.Node
|
version_string = subprocess.check_output(["poetry", "version", "--short"]).decode(
|
||||||
]:
|
"utf-8"
|
||||||
# Parse the AST and load the `__version__` node so that we can edit it
|
)
|
||||||
# later.
|
return version.Version(version_string)
|
||||||
with open("synapse/__init__.py") as f:
|
|
||||||
red = redbaron.RedBaron(f.read())
|
|
||||||
|
|
||||||
version_node = None
|
|
||||||
for node in red:
|
|
||||||
if node.type != "assignment":
|
|
||||||
continue
|
|
||||||
|
|
||||||
if node.target.type != "name":
|
|
||||||
continue
|
|
||||||
|
|
||||||
if node.target.value != "__version__":
|
|
||||||
continue
|
|
||||||
|
|
||||||
version_node = node
|
|
||||||
break
|
|
||||||
|
|
||||||
if not version_node:
|
|
||||||
print("Failed to find '__version__' definition in synapse/__init__.py")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
# Parse the current version.
|
|
||||||
current_version = version.parse(version_node.value.value.strip('"'))
|
|
||||||
assert isinstance(current_version, version.Version)
|
|
||||||
|
|
||||||
return current_version, red, version_node
|
|
||||||
|
|
||||||
|
|
||||||
def find_ref(repo: git.Repo, ref_name: str) -> Optional[git.HEAD]:
|
def find_ref(repo: git.Repo, ref_name: str) -> Optional[git.HEAD]:
|
||||||
|
@ -565,11 +536,13 @@ def get_changes_for_version(wanted_version: version.Version) -> str:
|
||||||
return "\n".join(version_changelog)
|
return "\n".join(version_changelog)
|
||||||
|
|
||||||
|
|
||||||
def generate_and_write_changelog(current_version: version.Version):
|
def generate_and_write_changelog(current_version: version.Version, new_version: str):
|
||||||
# We do this by getting a draft so that we can edit it before writing to the
|
# We do this by getting a draft so that we can edit it before writing to the
|
||||||
# changelog.
|
# changelog.
|
||||||
result = run_until_successful(
|
result = run_until_successful(
|
||||||
"python3 -m towncrier --draft", shell=True, capture_output=True
|
f"python3 -m towncrier build --draft --version {new_version}",
|
||||||
|
shell=True,
|
||||||
|
capture_output=True,
|
||||||
)
|
)
|
||||||
new_changes = result.stdout.decode("utf-8")
|
new_changes = result.stdout.decode("utf-8")
|
||||||
new_changes = new_changes.replace(
|
new_changes = new_changes.replace(
|
||||||
|
|
|
@ -1,4 +1,4 @@
|
||||||
# Copyright 2014-2021 The Matrix.org Foundation C.I.C.
|
# Copyright 2014-2022 The Matrix.org Foundation C.I.C.
|
||||||
# Copyright 2020 Sorunome
|
# Copyright 2020 Sorunome
|
||||||
#
|
#
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
@ -15,10 +15,14 @@
|
||||||
|
|
||||||
"""Contains handlers for federation events."""
|
"""Contains handlers for federation events."""
|
||||||
|
|
||||||
|
import enum
|
||||||
|
import itertools
|
||||||
import logging
|
import logging
|
||||||
|
from enum import Enum
|
||||||
from http import HTTPStatus
|
from http import HTTPStatus
|
||||||
from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple, Union
|
from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple, Union
|
||||||
|
|
||||||
|
import attr
|
||||||
from signedjson.key import decode_verify_key_bytes
|
from signedjson.key import decode_verify_key_bytes
|
||||||
from signedjson.sign import verify_signed_json
|
from signedjson.sign import verify_signed_json
|
||||||
from unpaddedbase64 import decode_base64
|
from unpaddedbase64 import decode_base64
|
||||||
|
@ -92,6 +96,24 @@ def get_domains_from_state(state: StateMap[EventBase]) -> List[Tuple[str, int]]:
|
||||||
return sorted(joined_domains.items(), key=lambda d: d[1])
|
return sorted(joined_domains.items(), key=lambda d: d[1])
|
||||||
|
|
||||||
|
|
||||||
|
class _BackfillPointType(Enum):
|
||||||
|
# a regular backwards extremity (ie, an event which we don't yet have, but which
|
||||||
|
# is referred to by other events in the DAG)
|
||||||
|
BACKWARDS_EXTREMITY = enum.auto()
|
||||||
|
|
||||||
|
# an MSC2716 "insertion event"
|
||||||
|
INSERTION_PONT = enum.auto()
|
||||||
|
|
||||||
|
|
||||||
|
@attr.s(slots=True, auto_attribs=True, frozen=True)
|
||||||
|
class _BackfillPoint:
|
||||||
|
"""A potential point we might backfill from"""
|
||||||
|
|
||||||
|
event_id: str
|
||||||
|
depth: int
|
||||||
|
type: _BackfillPointType
|
||||||
|
|
||||||
|
|
||||||
class FederationHandler:
|
class FederationHandler:
|
||||||
"""Handles general incoming federation requests
|
"""Handles general incoming federation requests
|
||||||
|
|
||||||
|
@ -157,89 +179,51 @@ class FederationHandler:
|
||||||
async def _maybe_backfill_inner(
|
async def _maybe_backfill_inner(
|
||||||
self, room_id: str, current_depth: int, limit: int
|
self, room_id: str, current_depth: int, limit: int
|
||||||
) -> bool:
|
) -> bool:
|
||||||
oldest_events_with_depth = (
|
backwards_extremities = [
|
||||||
await self.store.get_oldest_event_ids_with_depth_in_room(room_id)
|
_BackfillPoint(event_id, depth, _BackfillPointType.BACKWARDS_EXTREMITY)
|
||||||
)
|
for event_id, depth in await self.store.get_oldest_event_ids_with_depth_in_room(
|
||||||
|
room_id
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
insertion_events_to_be_backfilled: Dict[str, int] = {}
|
insertion_events_to_be_backfilled: List[_BackfillPoint] = []
|
||||||
if self.hs.config.experimental.msc2716_enabled:
|
if self.hs.config.experimental.msc2716_enabled:
|
||||||
insertion_events_to_be_backfilled = (
|
insertion_events_to_be_backfilled = [
|
||||||
await self.store.get_insertion_event_backward_extremities_in_room(
|
_BackfillPoint(event_id, depth, _BackfillPointType.INSERTION_PONT)
|
||||||
|
for event_id, depth in await self.store.get_insertion_event_backward_extremities_in_room(
|
||||||
room_id
|
room_id
|
||||||
)
|
)
|
||||||
)
|
]
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"_maybe_backfill_inner: extremities oldest_events_with_depth=%s insertion_events_to_be_backfilled=%s",
|
"_maybe_backfill_inner: backwards_extremities=%s insertion_events_to_be_backfilled=%s",
|
||||||
oldest_events_with_depth,
|
backwards_extremities,
|
||||||
insertion_events_to_be_backfilled,
|
insertion_events_to_be_backfilled,
|
||||||
)
|
)
|
||||||
|
|
||||||
if not oldest_events_with_depth and not insertion_events_to_be_backfilled:
|
if not backwards_extremities and not insertion_events_to_be_backfilled:
|
||||||
logger.debug("Not backfilling as no extremeties found.")
|
logger.debug("Not backfilling as no extremeties found.")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
# We only want to paginate if we can actually see the events we'll get,
|
# we now have a list of potential places to backpaginate from. We prefer to
|
||||||
# as otherwise we'll just spend a lot of resources to get redacted
|
# start with the most recent (ie, max depth), so let's sort the list.
|
||||||
# events.
|
sorted_backfill_points: List[_BackfillPoint] = sorted(
|
||||||
#
|
itertools.chain(
|
||||||
# We do this by filtering all the backwards extremities and seeing if
|
backwards_extremities,
|
||||||
# any remain. Given we don't have the extremity events themselves, we
|
insertion_events_to_be_backfilled,
|
||||||
# need to actually check the events that reference them.
|
),
|
||||||
#
|
key=lambda e: -int(e.depth),
|
||||||
# *Note*: the spec wants us to keep backfilling until we reach the start
|
|
||||||
# of the room in case we are allowed to see some of the history. However
|
|
||||||
# in practice that causes more issues than its worth, as a) its
|
|
||||||
# relatively rare for there to be any visible history and b) even when
|
|
||||||
# there is its often sufficiently long ago that clients would stop
|
|
||||||
# attempting to paginate before backfill reached the visible history.
|
|
||||||
#
|
|
||||||
# TODO: If we do do a backfill then we should filter the backwards
|
|
||||||
# extremities to only include those that point to visible portions of
|
|
||||||
# history.
|
|
||||||
#
|
|
||||||
# TODO: Correctly handle the case where we are allowed to see the
|
|
||||||
# forward event but not the backward extremity, e.g. in the case of
|
|
||||||
# initial join of the server where we are allowed to see the join
|
|
||||||
# event but not anything before it. This would require looking at the
|
|
||||||
# state *before* the event, ignoring the special casing certain event
|
|
||||||
# types have.
|
|
||||||
|
|
||||||
forward_event_ids = await self.store.get_successor_events(
|
|
||||||
list(oldest_events_with_depth)
|
|
||||||
)
|
)
|
||||||
|
|
||||||
extremities_events = await self.store.get_events(
|
|
||||||
forward_event_ids,
|
|
||||||
redact_behaviour=EventRedactBehaviour.AS_IS,
|
|
||||||
get_prev_content=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
# We set `check_history_visibility_only` as we might otherwise get false
|
|
||||||
# positives from users having been erased.
|
|
||||||
filtered_extremities = await filter_events_for_server(
|
|
||||||
self.storage,
|
|
||||||
self.server_name,
|
|
||||||
list(extremities_events.values()),
|
|
||||||
redact=False,
|
|
||||||
check_history_visibility_only=True,
|
|
||||||
)
|
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"_maybe_backfill_inner: filtered_extremities %s", filtered_extremities
|
"_maybe_backfill_inner: room_id: %s: current_depth: %s, limit: %s, "
|
||||||
|
"backfill points (%d): %s",
|
||||||
|
room_id,
|
||||||
|
current_depth,
|
||||||
|
limit,
|
||||||
|
len(sorted_backfill_points),
|
||||||
|
sorted_backfill_points,
|
||||||
)
|
)
|
||||||
|
|
||||||
if not filtered_extremities and not insertion_events_to_be_backfilled:
|
|
||||||
return False
|
|
||||||
|
|
||||||
extremities = {
|
|
||||||
**oldest_events_with_depth,
|
|
||||||
# TODO: insertion_events_to_be_backfilled is currently skipping the filtered_extremities checks
|
|
||||||
**insertion_events_to_be_backfilled,
|
|
||||||
}
|
|
||||||
|
|
||||||
# Check if we reached a point where we should start backfilling.
|
|
||||||
sorted_extremeties_tuple = sorted(extremities.items(), key=lambda e: -int(e[1]))
|
|
||||||
max_depth = sorted_extremeties_tuple[0][1]
|
|
||||||
|
|
||||||
# If we're approaching an extremity we trigger a backfill, otherwise we
|
# If we're approaching an extremity we trigger a backfill, otherwise we
|
||||||
# no-op.
|
# no-op.
|
||||||
#
|
#
|
||||||
|
@ -249,6 +233,11 @@ class FederationHandler:
|
||||||
# chose more than one times the limit in case of failure, but choosing a
|
# chose more than one times the limit in case of failure, but choosing a
|
||||||
# much larger factor will result in triggering a backfill request much
|
# much larger factor will result in triggering a backfill request much
|
||||||
# earlier than necessary.
|
# earlier than necessary.
|
||||||
|
#
|
||||||
|
# XXX: shouldn't we do this *after* the filter by depth below? Again, we don't
|
||||||
|
# care about events that have happened after our current position.
|
||||||
|
#
|
||||||
|
max_depth = sorted_backfill_points[0].depth
|
||||||
if current_depth - 2 * limit > max_depth:
|
if current_depth - 2 * limit > max_depth:
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"Not backfilling as we don't need to. %d < %d - 2 * %d",
|
"Not backfilling as we don't need to. %d < %d - 2 * %d",
|
||||||
|
@ -265,31 +254,98 @@ class FederationHandler:
|
||||||
# 2. we have likely previously tried and failed to backfill from that
|
# 2. we have likely previously tried and failed to backfill from that
|
||||||
# extremity, so to avoid getting "stuck" requesting the same
|
# extremity, so to avoid getting "stuck" requesting the same
|
||||||
# backfill repeatedly we drop those extremities.
|
# backfill repeatedly we drop those extremities.
|
||||||
filtered_sorted_extremeties_tuple = [
|
#
|
||||||
t for t in sorted_extremeties_tuple if int(t[1]) <= current_depth
|
|
||||||
]
|
|
||||||
|
|
||||||
logger.debug(
|
|
||||||
"room_id: %s, backfill: current_depth: %s, limit: %s, max_depth: %s, extrems (%d): %s filtered_sorted_extremeties_tuple: %s",
|
|
||||||
room_id,
|
|
||||||
current_depth,
|
|
||||||
limit,
|
|
||||||
max_depth,
|
|
||||||
len(sorted_extremeties_tuple),
|
|
||||||
sorted_extremeties_tuple,
|
|
||||||
filtered_sorted_extremeties_tuple,
|
|
||||||
)
|
|
||||||
|
|
||||||
# However, we need to check that the filtered extremities are non-empty.
|
# However, we need to check that the filtered extremities are non-empty.
|
||||||
# If they are empty then either we can a) bail or b) still attempt to
|
# If they are empty then either we can a) bail or b) still attempt to
|
||||||
# backfill. We opt to try backfilling anyway just in case we do get
|
# backfill. We opt to try backfilling anyway just in case we do get
|
||||||
# relevant events.
|
# relevant events.
|
||||||
if filtered_sorted_extremeties_tuple:
|
#
|
||||||
sorted_extremeties_tuple = filtered_sorted_extremeties_tuple
|
filtered_sorted_backfill_points = [
|
||||||
|
t for t in sorted_backfill_points if t.depth <= current_depth
|
||||||
|
]
|
||||||
|
if filtered_sorted_backfill_points:
|
||||||
|
logger.debug(
|
||||||
|
"_maybe_backfill_inner: backfill points before current depth: %s",
|
||||||
|
filtered_sorted_backfill_points,
|
||||||
|
)
|
||||||
|
sorted_backfill_points = filtered_sorted_backfill_points
|
||||||
|
else:
|
||||||
|
logger.debug(
|
||||||
|
"_maybe_backfill_inner: all backfill points are *after* current depth. Backfilling anyway."
|
||||||
|
)
|
||||||
|
|
||||||
# We don't want to specify too many extremities as it causes the backfill
|
# For performance's sake, we only want to paginate from a particular extremity
|
||||||
# request URI to be too long.
|
# if we can actually see the events we'll get. Otherwise, we'd just spend a lot
|
||||||
extremities = dict(sorted_extremeties_tuple[:5])
|
# of resources to get redacted events. We check each extremity in turn and
|
||||||
|
# ignore those which users on our server wouldn't be able to see.
|
||||||
|
#
|
||||||
|
# Additionally, we limit ourselves to backfilling from at most 5 extremities,
|
||||||
|
# for two reasons:
|
||||||
|
#
|
||||||
|
# - The check which determines if we can see an extremity's events can be
|
||||||
|
# expensive (we load the full state for the room at each of the backfill
|
||||||
|
# points, or (worse) their successors)
|
||||||
|
# - We want to avoid the server-server API request URI becoming too long.
|
||||||
|
#
|
||||||
|
# *Note*: the spec wants us to keep backfilling until we reach the start
|
||||||
|
# of the room in case we are allowed to see some of the history. However,
|
||||||
|
# in practice that causes more issues than its worth, as (a) it's
|
||||||
|
# relatively rare for there to be any visible history and (b) even when
|
||||||
|
# there is it's often sufficiently long ago that clients would stop
|
||||||
|
# attempting to paginate before backfill reached the visible history.
|
||||||
|
|
||||||
|
extremities_to_request: List[str] = []
|
||||||
|
for bp in sorted_backfill_points:
|
||||||
|
if len(extremities_to_request) >= 5:
|
||||||
|
break
|
||||||
|
|
||||||
|
# For regular backwards extremities, we don't have the extremity events
|
||||||
|
# themselves, so we need to actually check the events that reference them -
|
||||||
|
# their "successor" events.
|
||||||
|
#
|
||||||
|
# TODO: Correctly handle the case where we are allowed to see the
|
||||||
|
# successor event but not the backward extremity, e.g. in the case of
|
||||||
|
# initial join of the server where we are allowed to see the join
|
||||||
|
# event but not anything before it. This would require looking at the
|
||||||
|
# state *before* the event, ignoring the special casing certain event
|
||||||
|
# types have.
|
||||||
|
if bp.type == _BackfillPointType.INSERTION_PONT:
|
||||||
|
event_ids_to_check = [bp.event_id]
|
||||||
|
else:
|
||||||
|
event_ids_to_check = await self.store.get_successor_events(bp.event_id)
|
||||||
|
|
||||||
|
events_to_check = await self.store.get_events_as_list(
|
||||||
|
event_ids_to_check,
|
||||||
|
redact_behaviour=EventRedactBehaviour.AS_IS,
|
||||||
|
get_prev_content=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
# We set `check_history_visibility_only` as we might otherwise get false
|
||||||
|
# positives from users having been erased.
|
||||||
|
filtered_extremities = await filter_events_for_server(
|
||||||
|
self.storage,
|
||||||
|
self.server_name,
|
||||||
|
events_to_check,
|
||||||
|
redact=False,
|
||||||
|
check_history_visibility_only=True,
|
||||||
|
)
|
||||||
|
if filtered_extremities:
|
||||||
|
extremities_to_request.append(bp.event_id)
|
||||||
|
else:
|
||||||
|
logger.debug(
|
||||||
|
"_maybe_backfill_inner: skipping extremity %s as it would not be visible",
|
||||||
|
bp,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not extremities_to_request:
|
||||||
|
logger.debug(
|
||||||
|
"_maybe_backfill_inner: found no extremities which would be visible"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.debug(
|
||||||
|
"_maybe_backfill_inner: extremities_to_request %s", extremities_to_request
|
||||||
|
)
|
||||||
|
|
||||||
# Now we need to decide which hosts to hit first.
|
# Now we need to decide which hosts to hit first.
|
||||||
|
|
||||||
|
@ -309,7 +365,7 @@ class FederationHandler:
|
||||||
for dom in domains:
|
for dom in domains:
|
||||||
try:
|
try:
|
||||||
await self._federation_event_handler.backfill(
|
await self._federation_event_handler.backfill(
|
||||||
dom, room_id, limit=100, extremities=extremities
|
dom, room_id, limit=100, extremities=extremities_to_request
|
||||||
)
|
)
|
||||||
# If this succeeded then we probably already have the
|
# If this succeeded then we probably already have the
|
||||||
# appropriate stuff.
|
# appropriate stuff.
|
||||||
|
|
|
@ -54,7 +54,7 @@ class RoomBatchHandler:
|
||||||
# it has a larger `depth` but before the successor event because the `stream_ordering`
|
# it has a larger `depth` but before the successor event because the `stream_ordering`
|
||||||
# is negative before the successor event.
|
# is negative before the successor event.
|
||||||
successor_event_ids = await self.store.get_successor_events(
|
successor_event_ids = await self.store.get_successor_events(
|
||||||
[most_recent_prev_event_id]
|
most_recent_prev_event_id
|
||||||
)
|
)
|
||||||
|
|
||||||
# If we can't find any successor events, then it's a forward extremity of
|
# If we can't find any successor events, then it's a forward extremity of
|
||||||
|
|
|
@ -695,7 +695,9 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
# Return all events where not all sets can reach them.
|
# Return all events where not all sets can reach them.
|
||||||
return {eid for eid, n in event_to_missing_sets.items() if n}
|
return {eid for eid, n in event_to_missing_sets.items() if n}
|
||||||
|
|
||||||
async def get_oldest_event_ids_with_depth_in_room(self, room_id) -> Dict[str, int]:
|
async def get_oldest_event_ids_with_depth_in_room(
|
||||||
|
self, room_id
|
||||||
|
) -> List[Tuple[str, int]]:
|
||||||
"""Gets the oldest events(backwards extremities) in the room along with the
|
"""Gets the oldest events(backwards extremities) in the room along with the
|
||||||
aproximate depth.
|
aproximate depth.
|
||||||
|
|
||||||
|
@ -708,7 +710,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
room_id: Room where we want to find the oldest events
|
room_id: Room where we want to find the oldest events
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Map from event_id to depth
|
List of (event_id, depth) tuples
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def get_oldest_event_ids_with_depth_in_room_txn(txn, room_id):
|
def get_oldest_event_ids_with_depth_in_room_txn(txn, room_id):
|
||||||
|
@ -741,7 +743,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
|
|
||||||
txn.execute(sql, (room_id, False))
|
txn.execute(sql, (room_id, False))
|
||||||
|
|
||||||
return dict(txn)
|
return txn.fetchall()
|
||||||
|
|
||||||
return await self.db_pool.runInteraction(
|
return await self.db_pool.runInteraction(
|
||||||
"get_oldest_event_ids_with_depth_in_room",
|
"get_oldest_event_ids_with_depth_in_room",
|
||||||
|
@ -751,7 +753,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
|
|
||||||
async def get_insertion_event_backward_extremities_in_room(
|
async def get_insertion_event_backward_extremities_in_room(
|
||||||
self, room_id
|
self, room_id
|
||||||
) -> Dict[str, int]:
|
) -> List[Tuple[str, int]]:
|
||||||
"""Get the insertion events we know about that we haven't backfilled yet.
|
"""Get the insertion events we know about that we haven't backfilled yet.
|
||||||
|
|
||||||
We use this function so that we can compare and see if someones current
|
We use this function so that we can compare and see if someones current
|
||||||
|
@ -763,7 +765,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
room_id: Room where we want to find the oldest events
|
room_id: Room where we want to find the oldest events
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Map from event_id to depth
|
List of (event_id, depth) tuples
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def get_insertion_event_backward_extremities_in_room_txn(txn, room_id):
|
def get_insertion_event_backward_extremities_in_room_txn(txn, room_id):
|
||||||
|
@ -778,8 +780,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
"""
|
"""
|
||||||
|
|
||||||
txn.execute(sql, (room_id,))
|
txn.execute(sql, (room_id,))
|
||||||
|
return txn.fetchall()
|
||||||
return dict(txn)
|
|
||||||
|
|
||||||
return await self.db_pool.runInteraction(
|
return await self.db_pool.runInteraction(
|
||||||
"get_insertion_event_backward_extremities_in_room",
|
"get_insertion_event_backward_extremities_in_room",
|
||||||
|
@ -1295,22 +1296,19 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
||||||
event_results.reverse()
|
event_results.reverse()
|
||||||
return event_results
|
return event_results
|
||||||
|
|
||||||
async def get_successor_events(self, event_ids: Iterable[str]) -> List[str]:
|
async def get_successor_events(self, event_id: str) -> List[str]:
|
||||||
"""Fetch all events that have the given events as a prev event
|
"""Fetch all events that have the given event as a prev event
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
event_ids: The events to use as the previous events.
|
event_id: The event to search for as a prev_event.
|
||||||
"""
|
"""
|
||||||
rows = await self.db_pool.simple_select_many_batch(
|
return await self.db_pool.simple_select_onecol(
|
||||||
table="event_edges",
|
table="event_edges",
|
||||||
column="prev_event_id",
|
keyvalues={"prev_event_id": event_id},
|
||||||
iterable=event_ids,
|
retcol="event_id",
|
||||||
retcols=("event_id",),
|
|
||||||
desc="get_successor_events",
|
desc="get_successor_events",
|
||||||
)
|
)
|
||||||
|
|
||||||
return [row["event_id"] for row in rows]
|
|
||||||
|
|
||||||
@wrap_as_background_process("delete_old_forward_extrem_cache")
|
@wrap_as_background_process("delete_old_forward_extrem_cache")
|
||||||
async def _delete_old_forward_extrem_cache(self) -> None:
|
async def _delete_old_forward_extrem_cache(self) -> None:
|
||||||
def _delete_old_forward_extrem_cache_txn(txn):
|
def _delete_old_forward_extrem_cache_txn(txn):
|
||||||
|
|
|
@ -75,7 +75,7 @@ from synapse.storage.util.id_generators import (
|
||||||
from synapse.storage.util.sequence import build_sequence_generator
|
from synapse.storage.util.sequence import build_sequence_generator
|
||||||
from synapse.types import JsonDict, get_domain_from_id
|
from synapse.types import JsonDict, get_domain_from_id
|
||||||
from synapse.util import unwrapFirstError
|
from synapse.util import unwrapFirstError
|
||||||
from synapse.util.async_helpers import ObservableDeferred
|
from synapse.util.async_helpers import ObservableDeferred, delay_cancellation
|
||||||
from synapse.util.caches.descriptors import cached, cachedList
|
from synapse.util.caches.descriptors import cached, cachedList
|
||||||
from synapse.util.caches.lrucache import LruCache
|
from synapse.util.caches.lrucache import LruCache
|
||||||
from synapse.util.iterutils import batch_iter
|
from synapse.util.iterutils import batch_iter
|
||||||
|
@ -640,42 +640,57 @@ class EventsWorkerStore(SQLBaseStore):
|
||||||
missing_events_ids.difference_update(already_fetching_ids)
|
missing_events_ids.difference_update(already_fetching_ids)
|
||||||
|
|
||||||
if missing_events_ids:
|
if missing_events_ids:
|
||||||
log_ctx = current_context()
|
|
||||||
log_ctx.record_event_fetch(len(missing_events_ids))
|
|
||||||
|
|
||||||
# Add entries to `self._current_event_fetches` for each event we're
|
async def get_missing_events_from_db() -> Dict[str, EventCacheEntry]:
|
||||||
# going to pull from the DB. We use a single deferred that resolves
|
"""Fetches the events in `missing_event_ids` from the database.
|
||||||
# to all the events we pulled from the DB (this will result in this
|
|
||||||
# function returning more events than requested, but that can happen
|
|
||||||
# already due to `_get_events_from_db`).
|
|
||||||
fetching_deferred: ObservableDeferred[
|
|
||||||
Dict[str, EventCacheEntry]
|
|
||||||
] = ObservableDeferred(defer.Deferred(), consumeErrors=True)
|
|
||||||
for event_id in missing_events_ids:
|
|
||||||
self._current_event_fetches[event_id] = fetching_deferred
|
|
||||||
|
|
||||||
# Note that _get_events_from_db is also responsible for turning db rows
|
Also creates entries in `self._current_event_fetches` to allow
|
||||||
# into FrozenEvents (via _get_event_from_row), which involves seeing if
|
concurrent `_get_events_from_cache_or_db` calls to reuse the same fetch.
|
||||||
# the events have been redacted, and if so pulling the redaction event out
|
"""
|
||||||
# of the database to check it.
|
log_ctx = current_context()
|
||||||
#
|
log_ctx.record_event_fetch(len(missing_events_ids))
|
||||||
try:
|
|
||||||
missing_events = await self._get_events_from_db(
|
|
||||||
missing_events_ids,
|
|
||||||
)
|
|
||||||
|
|
||||||
event_entry_map.update(missing_events)
|
# Add entries to `self._current_event_fetches` for each event we're
|
||||||
except Exception as e:
|
# going to pull from the DB. We use a single deferred that resolves
|
||||||
with PreserveLoggingContext():
|
# to all the events we pulled from the DB (this will result in this
|
||||||
fetching_deferred.errback(e)
|
# function returning more events than requested, but that can happen
|
||||||
raise e
|
# already due to `_get_events_from_db`).
|
||||||
finally:
|
fetching_deferred: ObservableDeferred[
|
||||||
# Ensure that we mark these events as no longer being fetched.
|
Dict[str, EventCacheEntry]
|
||||||
|
] = ObservableDeferred(defer.Deferred(), consumeErrors=True)
|
||||||
for event_id in missing_events_ids:
|
for event_id in missing_events_ids:
|
||||||
self._current_event_fetches.pop(event_id, None)
|
self._current_event_fetches[event_id] = fetching_deferred
|
||||||
|
|
||||||
with PreserveLoggingContext():
|
# Note that _get_events_from_db is also responsible for turning db rows
|
||||||
fetching_deferred.callback(missing_events)
|
# into FrozenEvents (via _get_event_from_row), which involves seeing if
|
||||||
|
# the events have been redacted, and if so pulling the redaction event
|
||||||
|
# out of the database to check it.
|
||||||
|
#
|
||||||
|
try:
|
||||||
|
missing_events = await self._get_events_from_db(
|
||||||
|
missing_events_ids,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
with PreserveLoggingContext():
|
||||||
|
fetching_deferred.errback(e)
|
||||||
|
raise e
|
||||||
|
finally:
|
||||||
|
# Ensure that we mark these events as no longer being fetched.
|
||||||
|
for event_id in missing_events_ids:
|
||||||
|
self._current_event_fetches.pop(event_id, None)
|
||||||
|
|
||||||
|
with PreserveLoggingContext():
|
||||||
|
fetching_deferred.callback(missing_events)
|
||||||
|
|
||||||
|
return missing_events
|
||||||
|
|
||||||
|
# We must allow the database fetch to complete in the presence of
|
||||||
|
# cancellations, since multiple `_get_events_from_cache_or_db` calls can
|
||||||
|
# reuse the same fetch.
|
||||||
|
missing_events: Dict[str, EventCacheEntry] = await delay_cancellation(
|
||||||
|
get_missing_events_from_db()
|
||||||
|
)
|
||||||
|
event_entry_map.update(missing_events)
|
||||||
|
|
||||||
if already_fetching_deferreds:
|
if already_fetching_deferreds:
|
||||||
# Wait for the other event requests to finish and add their results
|
# Wait for the other event requests to finish and add their results
|
||||||
|
|
|
@ -419,6 +419,13 @@ async def _event_to_memberships(
|
||||||
return {}
|
return {}
|
||||||
|
|
||||||
# for each event, get the event_ids of the membership state at those events.
|
# for each event, get the event_ids of the membership state at those events.
|
||||||
|
#
|
||||||
|
# TODO: this means that we request the entire membership list. If there are only
|
||||||
|
# one or two users on this server, and the room is huge, this is very wasteful
|
||||||
|
# (it means more db work, and churns the *stateGroupMembersCache*).
|
||||||
|
# It might be that we could extend StateFilter to specify "give me keys matching
|
||||||
|
# *:<server_name>", to avoid this.
|
||||||
|
|
||||||
event_to_state_ids = await storage.state.get_state_ids_for_events(
|
event_to_state_ids = await storage.state.get_state_ids_for_events(
|
||||||
frozenset(e.event_id for e in events),
|
frozenset(e.event_id for e in events),
|
||||||
state_filter=StateFilter.from_types(types=((EventTypes.Member, None),)),
|
state_filter=StateFilter.from_types(types=((EventTypes.Member, None),)),
|
||||||
|
|
|
@ -13,10 +13,11 @@
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
import json
|
import json
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from typing import Generator
|
from typing import Generator, Tuple
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
from twisted.enterprise.adbapi import ConnectionPool
|
from twisted.enterprise.adbapi import ConnectionPool
|
||||||
from twisted.internet.defer import ensureDeferred
|
from twisted.internet.defer import CancelledError, Deferred, ensureDeferred
|
||||||
from twisted.test.proto_helpers import MemoryReactor
|
from twisted.test.proto_helpers import MemoryReactor
|
||||||
|
|
||||||
from synapse.api.room_versions import EventFormatVersions, RoomVersions
|
from synapse.api.room_versions import EventFormatVersions, RoomVersions
|
||||||
|
@ -281,3 +282,119 @@ class DatabaseOutageTestCase(unittest.HomeserverTestCase):
|
||||||
|
|
||||||
# This next event fetch should succeed
|
# This next event fetch should succeed
|
||||||
self.get_success(self.store.get_event(self.event_ids[0]))
|
self.get_success(self.store.get_event(self.event_ids[0]))
|
||||||
|
|
||||||
|
|
||||||
|
class GetEventCancellationTestCase(unittest.HomeserverTestCase):
|
||||||
|
"""Test cancellation of `get_event` calls."""
|
||||||
|
|
||||||
|
servlets = [
|
||||||
|
admin.register_servlets,
|
||||||
|
room.register_servlets,
|
||||||
|
login.register_servlets,
|
||||||
|
]
|
||||||
|
|
||||||
|
def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer):
|
||||||
|
self.store: EventsWorkerStore = hs.get_datastores().main
|
||||||
|
|
||||||
|
self.user = self.register_user("user", "pass")
|
||||||
|
self.token = self.login(self.user, "pass")
|
||||||
|
|
||||||
|
self.room = self.helper.create_room_as(self.user, tok=self.token)
|
||||||
|
|
||||||
|
res = self.helper.send(self.room, tok=self.token)
|
||||||
|
self.event_id = res["event_id"]
|
||||||
|
|
||||||
|
# Reset the event cache so the tests start with it empty
|
||||||
|
self.store._get_event_cache.clear()
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def blocking_get_event_calls(
|
||||||
|
self,
|
||||||
|
) -> Generator[
|
||||||
|
Tuple["Deferred[None]", "Deferred[None]", "Deferred[None]"], None, None
|
||||||
|
]:
|
||||||
|
"""Starts two concurrent `get_event` calls for the same event.
|
||||||
|
|
||||||
|
Both `get_event` calls will use the same database fetch, which will be blocked
|
||||||
|
at the time this function returns.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A tuple containing:
|
||||||
|
* A `Deferred` that unblocks the database fetch.
|
||||||
|
* A cancellable `Deferred` for the first `get_event` call.
|
||||||
|
* A cancellable `Deferred` for the second `get_event` call.
|
||||||
|
"""
|
||||||
|
# Patch `DatabasePool.runWithConnection` to block.
|
||||||
|
unblock: "Deferred[None]" = Deferred()
|
||||||
|
original_runWithConnection = self.store.db_pool.runWithConnection
|
||||||
|
|
||||||
|
async def runWithConnection(*args, **kwargs):
|
||||||
|
await unblock
|
||||||
|
return await original_runWithConnection(*args, **kwargs)
|
||||||
|
|
||||||
|
with mock.patch.object(
|
||||||
|
self.store.db_pool,
|
||||||
|
"runWithConnection",
|
||||||
|
new=runWithConnection,
|
||||||
|
):
|
||||||
|
ctx1 = LoggingContext("get_event1")
|
||||||
|
ctx2 = LoggingContext("get_event2")
|
||||||
|
|
||||||
|
async def get_event(ctx: LoggingContext) -> None:
|
||||||
|
with ctx:
|
||||||
|
await self.store.get_event(self.event_id)
|
||||||
|
|
||||||
|
get_event1 = ensureDeferred(get_event(ctx1))
|
||||||
|
get_event2 = ensureDeferred(get_event(ctx2))
|
||||||
|
|
||||||
|
# Both `get_event` calls ought to be blocked.
|
||||||
|
self.assertNoResult(get_event1)
|
||||||
|
self.assertNoResult(get_event2)
|
||||||
|
|
||||||
|
yield unblock, get_event1, get_event2
|
||||||
|
|
||||||
|
# Confirm that the two `get_event` calls shared the same database fetch.
|
||||||
|
self.assertEqual(ctx1.get_resource_usage().evt_db_fetch_count, 1)
|
||||||
|
self.assertEqual(ctx2.get_resource_usage().evt_db_fetch_count, 0)
|
||||||
|
|
||||||
|
def test_first_get_event_cancelled(self):
|
||||||
|
"""Test cancellation of the first `get_event` call sharing a database fetch.
|
||||||
|
|
||||||
|
The first `get_event` call is the one which initiates the fetch. We expect the
|
||||||
|
fetch to complete despite the cancellation. Furthermore, the first `get_event`
|
||||||
|
call must not abort before the fetch is complete, otherwise the fetch will be
|
||||||
|
using a finished logging context.
|
||||||
|
"""
|
||||||
|
with self.blocking_get_event_calls() as (unblock, get_event1, get_event2):
|
||||||
|
# Cancel the first `get_event` call.
|
||||||
|
get_event1.cancel()
|
||||||
|
# The first `get_event` call must not abort immediately, otherwise its
|
||||||
|
# logging context will be finished while it is still in use by the database
|
||||||
|
# fetch.
|
||||||
|
self.assertNoResult(get_event1)
|
||||||
|
# The second `get_event` call must not be cancelled.
|
||||||
|
self.assertNoResult(get_event2)
|
||||||
|
|
||||||
|
# Unblock the database fetch.
|
||||||
|
unblock.callback(None)
|
||||||
|
# A `CancelledError` should be raised out of the first `get_event` call.
|
||||||
|
exc = self.get_failure(get_event1, CancelledError).value
|
||||||
|
self.assertIsInstance(exc, CancelledError)
|
||||||
|
# The second `get_event` call should complete successfully.
|
||||||
|
self.get_success(get_event2)
|
||||||
|
|
||||||
|
def test_second_get_event_cancelled(self):
|
||||||
|
"""Test cancellation of the second `get_event` call sharing a database fetch."""
|
||||||
|
with self.blocking_get_event_calls() as (unblock, get_event1, get_event2):
|
||||||
|
# Cancel the second `get_event` call.
|
||||||
|
get_event2.cancel()
|
||||||
|
# The first `get_event` call must not be cancelled.
|
||||||
|
self.assertNoResult(get_event1)
|
||||||
|
# The second `get_event` call gets cancelled immediately.
|
||||||
|
exc = self.get_failure(get_event2, CancelledError).value
|
||||||
|
self.assertIsInstance(exc, CancelledError)
|
||||||
|
|
||||||
|
# Unblock the database fetch.
|
||||||
|
unblock.callback(None)
|
||||||
|
# The first `get_event` call should complete successfully.
|
||||||
|
self.get_success(get_event1)
|
||||||
|
|
Loading…
Reference in New Issue