Merge remote-tracking branch 'origin/release-v1.58' into matrix-org-hotfixes
David Robertson
commit
474a964fb7
72
CHANGES.md
72
CHANGES.md
|
|
@ -1,3 +1,75 @@
|
|||
Synapse 1.58.0rc1 (2022-04-26)
|
||||
==============================
|
||||
|
||||
As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](https://github.com/matrix-org/synapse/issues/11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
- Implement [MSC3383](https://github.com/matrix-org/matrix-spec-proposals/pull/3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](https://github.com/matrix-org/synapse/issues/11398))
|
||||
- Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](https://github.com/matrix-org/synapse/issues/11537))
|
||||
- Enable processing of device list updates asynchronously. ([\#12365](https://github.com/matrix-org/synapse/issues/12365), [\#12465](https://github.com/matrix-org/synapse/issues/12465))
|
||||
- Implement [MSC2815](https://github.com/matrix-org/matrix-spec-proposals/pull/2815) to allow room moderators to view redacted event content. Contributed by @tulir. ([\#12427](https://github.com/matrix-org/synapse/issues/12427))
|
||||
- Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](https://github.com/matrix-org/synapse/issues/12543))
|
||||
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Prevent a sync request from removing a user's busy presence status. ([\#12213](https://github.com/matrix-org/synapse/issues/12213))
|
||||
- Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](https://github.com/matrix-org/synapse/issues/12319))
|
||||
- Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](https://github.com/matrix-org/synapse/issues/12476))
|
||||
- Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](https://github.com/matrix-org/synapse/issues/12496))
|
||||
- Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](https://github.com/matrix-org/synapse/issues/12510))
|
||||
- Fix a bug in the implementation of [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](https://github.com/matrix-org/synapse/issues/12520))
|
||||
- Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](https://github.com/matrix-org/synapse/issues/12522))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Fix rendering of the documentation site when using the 'print' feature. ([\#12340](https://github.com/matrix-org/synapse/issues/12340))
|
||||
- Add a manual documenting config file options. ([\#12368](https://github.com/matrix-org/synapse/issues/12368), [\#12527](https://github.com/matrix-org/synapse/issues/12527))
|
||||
- Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](https://github.com/matrix-org/synapse/issues/12451))
|
||||
- Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](https://github.com/matrix-org/synapse/issues/12457))
|
||||
- Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](https://github.com/matrix-org/synapse/issues/12475))
|
||||
- Add some example configurations for workers and update architectural diagram. ([\#12492](https://github.com/matrix-org/synapse/issues/12492))
|
||||
- Fix a broken link in `README.rst`. ([\#12495](https://github.com/matrix-org/synapse/issues/12495))
|
||||
- Add HAProxy delegation example with CORS headers to docs. ([\#12501](https://github.com/matrix-org/synapse/issues/12501))
|
||||
- Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](https://github.com/matrix-org/synapse/issues/12533))
|
||||
|
||||
|
||||
Deprecations and Removals
|
||||
-------------------------
|
||||
|
||||
- The groups/communities feature in Synapse is now disabled by default. ([\#12344](https://github.com/matrix-org/synapse/issues/12344))
|
||||
- Remove unstable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#12382](https://github.com/matrix-org/synapse/issues/12382))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](https://github.com/matrix-org/synapse/issues/12394))
|
||||
- Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](https://github.com/matrix-org/synapse/issues/12399))
|
||||
- Remove an unstable identifier from [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083). ([\#12395](https://github.com/matrix-org/synapse/issues/12395))
|
||||
- Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](https://github.com/matrix-org/synapse/issues/12425), [\#12434](https://github.com/matrix-org/synapse/issues/12434), [\#12438](https://github.com/matrix-org/synapse/issues/12438), [\#12441](https://github.com/matrix-org/synapse/issues/12441), [\#12449](https://github.com/matrix-org/synapse/issues/12449), [\#12478](https://github.com/matrix-org/synapse/issues/12478), [\#12514](https://github.com/matrix-org/synapse/issues/12514), [\#12472](https://github.com/matrix-org/synapse/issues/12472))
|
||||
- Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](https://github.com/matrix-org/matrix-spec-proposals/pull/2666). ([\#12445](https://github.com/matrix-org/synapse/issues/12445))
|
||||
- Fix typo in the release script help string. ([\#12450](https://github.com/matrix-org/synapse/issues/12450))
|
||||
- Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](https://github.com/matrix-org/synapse/issues/12497))
|
||||
- Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](https://github.com/matrix-org/synapse/issues/12455))
|
||||
- Limit length of `device_id` to less than 512 characters. ([\#12454](https://github.com/matrix-org/synapse/issues/12454))
|
||||
- Dockerfile-workers: reduce the amount we install in the image. ([\#12464](https://github.com/matrix-org/synapse/issues/12464))
|
||||
- Dockerfile-workers: give the master its own log config. ([\#12466](https://github.com/matrix-org/synapse/issues/12466))
|
||||
- complement-synapse-workers: factor out separate entry point script. ([\#12467](https://github.com/matrix-org/synapse/issues/12467))
|
||||
- Back out experimental implementation of [MSC2314](https://github.com/matrix-org/matrix-spec-proposals/pull/2314). ([\#12474](https://github.com/matrix-org/synapse/issues/12474))
|
||||
- Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](https://github.com/matrix-org/synapse/issues/12483))
|
||||
- Remove unnecessary configuration overrides in tests. ([\#12511](https://github.com/matrix-org/synapse/issues/12511))
|
||||
- Refactor the relations code for clarity. ([\#12519](https://github.com/matrix-org/synapse/issues/12519))
|
||||
- Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](https://github.com/matrix-org/synapse/issues/12528))
|
||||
- Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](https://github.com/matrix-org/synapse/issues/12468))
|
||||
- Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](https://github.com/matrix-org/synapse/issues/12529))
|
||||
|
||||
|
||||
Synapse 1.57.1 (2022-04-20)
|
||||
===========================
|
||||
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
Implement [MSC3383](https://github.com/matrix-org/matrix-spec-proposals/pull/3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly GmbH.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Prevent a sync request from removing a user's busy presence status.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Use poetry to manage Synapse's dependencies.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix rendering of the documentation site when using the 'print' feature.
|
||||
|
|
@ -1 +0,0 @@
|
|||
The groups/communities feature in Synapse has been disabled by default.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Enable processing of device list updates asynchronously.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add a manual documenting config file options.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove unstable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).
|
||||
|
|
@ -1 +0,0 @@
|
|||
Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove an unstable identifier from [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083).
|
||||
|
|
@ -1 +0,0 @@
|
|||
Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Run twisted trunk CI job in the locked poetry environment.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Implement [MSC2815](https://github.com/matrix-org/matrix-spec-proposals/pull/2815) to allow room moderators to view redacted event content. Contributed by @tulir.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Run lints under poetry in CI, and remove corresponding tox lint jobs.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Run "main" trial tests under `poetry`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Bump twisted version in `poetry.lock` to work around [pip bug #9644](https://github.com/pypa/pip/issues/9644).
|
||||
|
|
@ -1 +0,0 @@
|
|||
Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current MSC iteration.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Use `poetry` to manage the virtualenv in debian packages.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix typo in the release script help string.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Limit length of device_id to less than 512 characters.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Update worker documentation and replace old `federation_reader` with `generic_worker`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Dockerfile-workers: reduce the amount we install in the image.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Enable processing of device list updates asynchronously.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Dockerfile-workers: give the master its own log config.
|
||||
|
|
@ -1 +0,0 @@
|
|||
complement-synapse-workers: factor out separate entry point script.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add a CI job which tests Synapse against the latest version of all dependencies.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Back out experimental implementation of [MSC2314](https://github.com/matrix-org/matrix-spec-proposals/pull/2314).
|
||||
|
|
@ -1 +0,0 @@
|
|||
Strongly recommend `poetry` for development.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix a long-standing bug which incorrectly caused `GET /_matrix/client/r3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Use poetry-core instead of setuptools to build wheels.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix grammatical error in federation error response when the room version of a room is unknown.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add some example configurations for workers and update architectural diagram.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix a broken link in `README.rst`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix bug where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1 no matter how many extremities were deleted. Broke in v1.27.0.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix a minor typo in the Debian changelogs generated by the release script.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add HAProxy delegation example with CORS headers to docs.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix a long-standing bug where the image thumbanils embedded into email notifications were broken.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove unnecessary configuration overrides in tests.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Use poetry-core instead of setuptools to build wheels.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Refactor the relations code for clarity.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix a bug in the implementation of MSC3202 where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari.
|
||||
|
|
@ -1,8 +1,9 @@
|
|||
matrix-synapse-py3 (1.58.0+nmu1) UNRELEASED; urgency=medium
|
||||
matrix-synapse-py3 (1.58.0~rc1) stable; urgency=medium
|
||||
|
||||
* Use poetry to manage the bundled virtualenv included with this package.
|
||||
* New Synapse release 1.58.0rc1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Wed, 30 Mar 2022 12:21:43 +0100
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 26 Apr 2022 11:15:20 +0100
|
||||
|
||||
matrix-synapse-py3 (1.57.1) stable; urgency=medium
|
||||
|
||||
|
|
|
|||
|
|
@ -64,6 +64,48 @@ apply if you want your config file to be read properly. A few helpful things to
|
|||
In addition, each setting has an example of its usage, with the proper indentation
|
||||
shown.
|
||||
|
||||
## Contents
|
||||
[Modules](#modules)
|
||||
|
||||
[Server](#server)
|
||||
|
||||
[Homeserver Blocking](#homeserver-blocking)
|
||||
|
||||
[TLS](#tls)
|
||||
|
||||
[Federation](#federation)
|
||||
|
||||
[Caching](#caching)
|
||||
|
||||
[Database](#database)
|
||||
|
||||
[Logging](#logging)
|
||||
|
||||
[Ratelimiting](#ratelimiting)
|
||||
|
||||
[Media Store](#media-store)
|
||||
|
||||
[Captcha](#captcha)
|
||||
|
||||
[TURN](#turn)
|
||||
|
||||
[Registration](#registration)
|
||||
|
||||
[API Configuration](#api-configuration)
|
||||
|
||||
[Signing Keys](#signing-keys)
|
||||
|
||||
[Single Sign On Integration](#single-sign-on-integration)
|
||||
|
||||
[Push](#push)
|
||||
|
||||
[Rooms](#rooms)
|
||||
|
||||
[Opentracing](#opentracing)
|
||||
|
||||
[Workers](#workers)
|
||||
|
||||
[Background Updates](#background-updates)
|
||||
|
||||
## Modules
|
||||
|
||||
|
|
|
|||
|
|
@ -1,11 +1,3 @@
|
|||
[[package]]
|
||||
name = "appdirs"
|
||||
version = "1.4.4"
|
||||
description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
|
||||
category = "dev"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
|
||||
[[package]]
|
||||
name = "attrs"
|
||||
version = "21.4.0"
|
||||
|
|
@ -49,17 +41,6 @@ six = "*"
|
|||
[package.extras]
|
||||
visualize = ["graphviz (>0.5.1)", "Twisted (>=16.1.1)"]
|
||||
|
||||
[[package]]
|
||||
name = "baron"
|
||||
version = "0.10.1"
|
||||
description = "Full Syntax Tree for python to make writing refactoring code a realist task"
|
||||
category = "dev"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
|
||||
[package.dependencies]
|
||||
rply = "*"
|
||||
|
||||
[[package]]
|
||||
name = "bcrypt"
|
||||
version = "3.2.0"
|
||||
|
|
@ -984,20 +965,6 @@ Pygments = ">=2.5.1"
|
|||
[package.extras]
|
||||
md = ["cmarkgfm (>=0.8.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "redbaron"
|
||||
version = "0.9.2"
|
||||
description = "Abstraction on top of baron, a FST for python to make writing refactoring code a realistic task"
|
||||
category = "dev"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
|
||||
[package.dependencies]
|
||||
baron = ">=0.7"
|
||||
|
||||
[package.extras]
|
||||
notebook = ["pygments"]
|
||||
|
||||
[[package]]
|
||||
name = "requests"
|
||||
version = "2.27.1"
|
||||
|
|
@ -1038,17 +1005,6 @@ python-versions = ">=3.7"
|
|||
[package.extras]
|
||||
idna2008 = ["idna"]
|
||||
|
||||
[[package]]
|
||||
name = "rply"
|
||||
version = "0.7.8"
|
||||
description = "A pure Python Lex/Yacc that works with RPython"
|
||||
category = "dev"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
|
||||
[package.dependencies]
|
||||
appdirs = "*"
|
||||
|
||||
[[package]]
|
||||
name = "secretstorage"
|
||||
version = "3.3.1"
|
||||
|
|
@ -1597,13 +1553,9 @@ url_preview = ["lxml"]
|
|||
[metadata]
|
||||
lock-version = "1.1"
|
||||
python-versions = "^3.7"
|
||||
content-hash = "964ad29eaf7fd02749a4e735818f3bc0ba729c2f4b9e3213f0daa02643508b16"
|
||||
content-hash = "f482a4f594a165dfe01ce253a22510d5faf38647ab0dcebc35789350cafd9bf0"
|
||||
|
||||
[metadata.files]
|
||||
appdirs = [
|
||||
{file = "appdirs-1.4.4-py2.py3-none-any.whl", hash = "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"},
|
||||
{file = "appdirs-1.4.4.tar.gz", hash = "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41"},
|
||||
]
|
||||
attrs = [
|
||||
{file = "attrs-21.4.0-py2.py3-none-any.whl", hash = "sha256:2d27e3784d7a565d36ab851fe94887c5eccd6a463168875832a1be79c82828b4"},
|
||||
{file = "attrs-21.4.0.tar.gz", hash = "sha256:626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd"},
|
||||
|
|
@ -1616,10 +1568,6 @@ automat = [
|
|||
{file = "Automat-20.2.0-py2.py3-none-any.whl", hash = "sha256:b6feb6455337df834f6c9962d6ccf771515b7d939bca142b29c20c2376bc6111"},
|
||||
{file = "Automat-20.2.0.tar.gz", hash = "sha256:7979803c74610e11ef0c0d68a2942b152df52da55336e0c9d58daf1831cbdf33"},
|
||||
]
|
||||
baron = [
|
||||
{file = "baron-0.10.1-py2.py3-none-any.whl", hash = "sha256:befb33f4b9e832c7cd1e3cf0eafa6dd3cb6ed4cb2544245147c019936f4e0a8a"},
|
||||
{file = "baron-0.10.1.tar.gz", hash = "sha256:af822ad44d4eb425c8516df4239ac4fdba9fdb398ef77e4924cd7c9b4045bc2f"},
|
||||
]
|
||||
bcrypt = [
|
||||
{file = "bcrypt-3.2.0-cp36-abi3-macosx_10_10_universal2.whl", hash = "sha256:b589229207630484aefe5899122fb938a5b017b0f4349f769b8c13e78d99a8fd"},
|
||||
{file = "bcrypt-3.2.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:c95d4cbebffafcdd28bd28bb4e25b31c50f6da605c81ffd9ad8a3d1b2ab7b1b6"},
|
||||
|
|
@ -2412,10 +2360,6 @@ readme-renderer = [
|
|||
{file = "readme_renderer-33.0-py3-none-any.whl", hash = "sha256:f02cee0c4de9636b5a62b6be50c9742427ba1b956aad1d938bfb087d0d72ccdf"},
|
||||
{file = "readme_renderer-33.0.tar.gz", hash = "sha256:e3b53bc84bd6af054e4cc1fe3567dc1ae19f554134221043a3f8c674e22209db"},
|
||||
]
|
||||
redbaron = [
|
||||
{file = "redbaron-0.9.2-py2.py3-none-any.whl", hash = "sha256:d01032b6a848b5521a8d6ef72486315c2880f420956870cdd742e2b5a09b9bab"},
|
||||
{file = "redbaron-0.9.2.tar.gz", hash = "sha256:472d0739ca6b2240bb2278ae428604a75472c9c12e86c6321e8c016139c0132f"},
|
||||
]
|
||||
requests = [
|
||||
{file = "requests-2.27.1-py2.py3-none-any.whl", hash = "sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d"},
|
||||
{file = "requests-2.27.1.tar.gz", hash = "sha256:68d7c56fd5a8999887728ef304a6d12edc7be74f1cfa47714fc8b414525c9a61"},
|
||||
|
|
@ -2428,10 +2372,6 @@ rfc3986 = [
|
|||
{file = "rfc3986-2.0.0-py2.py3-none-any.whl", hash = "sha256:50b1502b60e289cb37883f3dfd34532b8873c7de9f49bb546641ce9cbd256ebd"},
|
||||
{file = "rfc3986-2.0.0.tar.gz", hash = "sha256:97aacf9dbd4bfd829baad6e6309fa6573aaf1be3f6fa735c8ab05e46cecb261c"},
|
||||
]
|
||||
rply = [
|
||||
{file = "rply-0.7.8-py2.py3-none-any.whl", hash = "sha256:28ffd11d656c48aeb8c508eb382acd6a0bd906662624b34388751732a27807e7"},
|
||||
{file = "rply-0.7.8.tar.gz", hash = "sha256:2a808ac25a4580a9991fc304d64434e299a8fc75760574492f242cbb5bb301c9"},
|
||||
]
|
||||
secretstorage = [
|
||||
{file = "SecretStorage-3.3.1-py3-none-any.whl", hash = "sha256:422d82c36172d88d6a0ed5afdec956514b189ddbfb72fefab0c8a1cee4eaf71f"},
|
||||
{file = "SecretStorage-3.3.1.tar.gz", hash = "sha256:fd666c51a6bf200643495a04abb261f83229dcb6fd8472ec393df7ffc8b6f195"},
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ skip_gitignore = true
|
|||
|
||||
[tool.poetry]
|
||||
name = "matrix-synapse"
|
||||
version = "1.57.1"
|
||||
version = "1.58.0rc1"
|
||||
description = "Homeserver for the Matrix decentralised comms protocol"
|
||||
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
|
||||
license = "Apache-2.0"
|
||||
|
|
@ -270,7 +270,6 @@ idna = ">=2.5"
|
|||
|
||||
# The following are used by the release script
|
||||
click = "==8.1.0"
|
||||
redbaron = "==0.9.2"
|
||||
GitPython = "==3.1.14"
|
||||
commonmark = "==0.9.1"
|
||||
pygithub = "==1.55"
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ DISTS = (
|
|||
"debian:sid",
|
||||
"ubuntu:focal", # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
|
||||
"ubuntu:impish", # 21.10 (EOL 2022-07)
|
||||
"ubuntu:jammy", # 22.04 LTS (EOL 2027-04)
|
||||
)
|
||||
|
||||
DESC = """\
|
||||
|
|
|
|||
|
|
@ -25,13 +25,12 @@ import sys
|
|||
import urllib.request
|
||||
from os import path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import List, Optional, Tuple
|
||||
from typing import List, Optional
|
||||
|
||||
import attr
|
||||
import click
|
||||
import commonmark
|
||||
import git
|
||||
import redbaron
|
||||
from click.exceptions import ClickException
|
||||
from github import Github
|
||||
from packaging import version
|
||||
|
|
@ -100,7 +99,7 @@ def prepare():
|
|||
repo.remote().fetch()
|
||||
|
||||
# Get the current version and AST from root Synapse module.
|
||||
current_version, parsed_synapse_ast, version_node = parse_version_from_module()
|
||||
current_version = get_package_version()
|
||||
|
||||
# Figure out what sort of release we're doing and calcuate the new version.
|
||||
rc = click.confirm("RC", default=True)
|
||||
|
|
@ -162,7 +161,7 @@ def prepare():
|
|||
click.get_current_context().abort()
|
||||
|
||||
# Switch to the release branch.
|
||||
parsed_new_version = version.parse(new_version)
|
||||
parsed_new_version: version.Version = version.parse(new_version)
|
||||
|
||||
# We assume for debian changelogs that we only do RCs or full releases.
|
||||
assert not parsed_new_version.is_devrelease
|
||||
|
|
@ -207,17 +206,15 @@ def prepare():
|
|||
# Create the new release branch
|
||||
release_branch = repo.create_head(release_branch_name, commit=base_branch)
|
||||
|
||||
# Switch to the release branch and ensure its up to date.
|
||||
# Switch to the release branch and ensure it's up to date.
|
||||
repo.git.checkout(release_branch_name)
|
||||
update_branch(repo)
|
||||
|
||||
# Update the `__version__` variable and write it back to the file.
|
||||
version_node.value = '"' + new_version + '"'
|
||||
with open("synapse/__init__.py", "w") as f:
|
||||
f.write(parsed_synapse_ast.dumps())
|
||||
# Update the version specified in pyproject.toml.
|
||||
subprocess.check_output(["poetry", "version", new_version])
|
||||
|
||||
# Generate changelogs.
|
||||
generate_and_write_changelog(current_version)
|
||||
generate_and_write_changelog(current_version, new_version)
|
||||
|
||||
# Generate debian changelogs
|
||||
if parsed_new_version.pre is not None:
|
||||
|
|
@ -284,7 +281,7 @@ def tag(gh_token: Optional[str]):
|
|||
repo.remote().fetch()
|
||||
|
||||
# Find out the version and tag name.
|
||||
current_version, _, _ = parse_version_from_module()
|
||||
current_version = get_package_version()
|
||||
tag_name = f"v{current_version}"
|
||||
|
||||
# Check we haven't released this version.
|
||||
|
|
@ -362,7 +359,7 @@ def publish(gh_token: str):
|
|||
if repo.is_dirty():
|
||||
raise click.ClickException("Uncommitted changes exist.")
|
||||
|
||||
current_version, _, _ = parse_version_from_module()
|
||||
current_version = get_package_version()
|
||||
tag_name = f"v{current_version}"
|
||||
|
||||
if not click.confirm(f"Publish {tag_name}?", default=True):
|
||||
|
|
@ -396,7 +393,7 @@ def publish(gh_token: str):
|
|||
def upload():
|
||||
"""Upload release to pypi."""
|
||||
|
||||
current_version, _, _ = parse_version_from_module()
|
||||
current_version = get_package_version()
|
||||
tag_name = f"v{current_version}"
|
||||
|
||||
pypi_asset_names = [
|
||||
|
|
@ -424,7 +421,7 @@ def upload():
|
|||
def announce():
|
||||
"""Generate markdown to announce the release."""
|
||||
|
||||
current_version, _, _ = parse_version_from_module()
|
||||
current_version = get_package_version()
|
||||
tag_name = f"v{current_version}"
|
||||
|
||||
click.echo(
|
||||
|
|
@ -455,37 +452,11 @@ Announce the release in
|
|||
)
|
||||
|
||||
|
||||
def parse_version_from_module() -> Tuple[
|
||||
version.Version, redbaron.RedBaron, redbaron.Node
|
||||
]:
|
||||
# Parse the AST and load the `__version__` node so that we can edit it
|
||||
# later.
|
||||
with open("synapse/__init__.py") as f:
|
||||
red = redbaron.RedBaron(f.read())
|
||||
|
||||
version_node = None
|
||||
for node in red:
|
||||
if node.type != "assignment":
|
||||
continue
|
||||
|
||||
if node.target.type != "name":
|
||||
continue
|
||||
|
||||
if node.target.value != "__version__":
|
||||
continue
|
||||
|
||||
version_node = node
|
||||
break
|
||||
|
||||
if not version_node:
|
||||
print("Failed to find '__version__' definition in synapse/__init__.py")
|
||||
sys.exit(1)
|
||||
|
||||
# Parse the current version.
|
||||
current_version = version.parse(version_node.value.value.strip('"'))
|
||||
assert isinstance(current_version, version.Version)
|
||||
|
||||
return current_version, red, version_node
|
||||
def get_package_version() -> version.Version:
|
||||
version_string = subprocess.check_output(["poetry", "version", "--short"]).decode(
|
||||
"utf-8"
|
||||
)
|
||||
return version.Version(version_string)
|
||||
|
||||
|
||||
def find_ref(repo: git.Repo, ref_name: str) -> Optional[git.HEAD]:
|
||||
|
|
@ -565,11 +536,13 @@ def get_changes_for_version(wanted_version: version.Version) -> str:
|
|||
return "\n".join(version_changelog)
|
||||
|
||||
|
||||
def generate_and_write_changelog(current_version: version.Version):
|
||||
def generate_and_write_changelog(current_version: version.Version, new_version: str):
|
||||
# We do this by getting a draft so that we can edit it before writing to the
|
||||
# changelog.
|
||||
result = run_until_successful(
|
||||
"python3 -m towncrier --draft", shell=True, capture_output=True
|
||||
f"python3 -m towncrier build --draft --version {new_version}",
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
)
|
||||
new_changes = result.stdout.decode("utf-8")
|
||||
new_changes = new_changes.replace(
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Copyright 2014-2021 The Matrix.org Foundation C.I.C.
|
||||
# Copyright 2014-2022 The Matrix.org Foundation C.I.C.
|
||||
# Copyright 2020 Sorunome
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
|
|
@ -15,10 +15,14 @@
|
|||
|
||||
"""Contains handlers for federation events."""
|
||||
|
||||
import enum
|
||||
import itertools
|
||||
import logging
|
||||
from enum import Enum
|
||||
from http import HTTPStatus
|
||||
from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple, Union
|
||||
|
||||
import attr
|
||||
from signedjson.key import decode_verify_key_bytes
|
||||
from signedjson.sign import verify_signed_json
|
||||
from unpaddedbase64 import decode_base64
|
||||
|
|
@ -92,6 +96,24 @@ def get_domains_from_state(state: StateMap[EventBase]) -> List[Tuple[str, int]]:
|
|||
return sorted(joined_domains.items(), key=lambda d: d[1])
|
||||
|
||||
|
||||
class _BackfillPointType(Enum):
|
||||
# a regular backwards extremity (ie, an event which we don't yet have, but which
|
||||
# is referred to by other events in the DAG)
|
||||
BACKWARDS_EXTREMITY = enum.auto()
|
||||
|
||||
# an MSC2716 "insertion event"
|
||||
INSERTION_PONT = enum.auto()
|
||||
|
||||
|
||||
@attr.s(slots=True, auto_attribs=True, frozen=True)
|
||||
class _BackfillPoint:
|
||||
"""A potential point we might backfill from"""
|
||||
|
||||
event_id: str
|
||||
depth: int
|
||||
type: _BackfillPointType
|
||||
|
||||
|
||||
class FederationHandler:
|
||||
"""Handles general incoming federation requests
|
||||
|
||||
|
|
@ -157,89 +179,51 @@ class FederationHandler:
|
|||
async def _maybe_backfill_inner(
|
||||
self, room_id: str, current_depth: int, limit: int
|
||||
) -> bool:
|
||||
oldest_events_with_depth = (
|
||||
await self.store.get_oldest_event_ids_with_depth_in_room(room_id)
|
||||
)
|
||||
|
||||
insertion_events_to_be_backfilled: Dict[str, int] = {}
|
||||
if self.hs.config.experimental.msc2716_enabled:
|
||||
insertion_events_to_be_backfilled = (
|
||||
await self.store.get_insertion_event_backward_extremities_in_room(
|
||||
backwards_extremities = [
|
||||
_BackfillPoint(event_id, depth, _BackfillPointType.BACKWARDS_EXTREMITY)
|
||||
for event_id, depth in await self.store.get_oldest_event_ids_with_depth_in_room(
|
||||
room_id
|
||||
)
|
||||
]
|
||||
|
||||
insertion_events_to_be_backfilled: List[_BackfillPoint] = []
|
||||
if self.hs.config.experimental.msc2716_enabled:
|
||||
insertion_events_to_be_backfilled = [
|
||||
_BackfillPoint(event_id, depth, _BackfillPointType.INSERTION_PONT)
|
||||
for event_id, depth in await self.store.get_insertion_event_backward_extremities_in_room(
|
||||
room_id
|
||||
)
|
||||
]
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: extremities oldest_events_with_depth=%s insertion_events_to_be_backfilled=%s",
|
||||
oldest_events_with_depth,
|
||||
"_maybe_backfill_inner: backwards_extremities=%s insertion_events_to_be_backfilled=%s",
|
||||
backwards_extremities,
|
||||
insertion_events_to_be_backfilled,
|
||||
)
|
||||
|
||||
if not oldest_events_with_depth and not insertion_events_to_be_backfilled:
|
||||
if not backwards_extremities and not insertion_events_to_be_backfilled:
|
||||
logger.debug("Not backfilling as no extremeties found.")
|
||||
return False
|
||||
|
||||
# We only want to paginate if we can actually see the events we'll get,
|
||||
# as otherwise we'll just spend a lot of resources to get redacted
|
||||
# events.
|
||||
#
|
||||
# We do this by filtering all the backwards extremities and seeing if
|
||||
# any remain. Given we don't have the extremity events themselves, we
|
||||
# need to actually check the events that reference them.
|
||||
#
|
||||
# *Note*: the spec wants us to keep backfilling until we reach the start
|
||||
# of the room in case we are allowed to see some of the history. However
|
||||
# in practice that causes more issues than its worth, as a) its
|
||||
# relatively rare for there to be any visible history and b) even when
|
||||
# there is its often sufficiently long ago that clients would stop
|
||||
# attempting to paginate before backfill reached the visible history.
|
||||
#
|
||||
# TODO: If we do do a backfill then we should filter the backwards
|
||||
# extremities to only include those that point to visible portions of
|
||||
# history.
|
||||
#
|
||||
# TODO: Correctly handle the case where we are allowed to see the
|
||||
# forward event but not the backward extremity, e.g. in the case of
|
||||
# initial join of the server where we are allowed to see the join
|
||||
# event but not anything before it. This would require looking at the
|
||||
# state *before* the event, ignoring the special casing certain event
|
||||
# types have.
|
||||
|
||||
forward_event_ids = await self.store.get_successor_events(
|
||||
list(oldest_events_with_depth)
|
||||
# we now have a list of potential places to backpaginate from. We prefer to
|
||||
# start with the most recent (ie, max depth), so let's sort the list.
|
||||
sorted_backfill_points: List[_BackfillPoint] = sorted(
|
||||
itertools.chain(
|
||||
backwards_extremities,
|
||||
insertion_events_to_be_backfilled,
|
||||
),
|
||||
key=lambda e: -int(e.depth),
|
||||
)
|
||||
|
||||
extremities_events = await self.store.get_events(
|
||||
forward_event_ids,
|
||||
redact_behaviour=EventRedactBehaviour.AS_IS,
|
||||
get_prev_content=False,
|
||||
)
|
||||
|
||||
# We set `check_history_visibility_only` as we might otherwise get false
|
||||
# positives from users having been erased.
|
||||
filtered_extremities = await filter_events_for_server(
|
||||
self.storage,
|
||||
self.server_name,
|
||||
list(extremities_events.values()),
|
||||
redact=False,
|
||||
check_history_visibility_only=True,
|
||||
)
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: filtered_extremities %s", filtered_extremities
|
||||
"_maybe_backfill_inner: room_id: %s: current_depth: %s, limit: %s, "
|
||||
"backfill points (%d): %s",
|
||||
room_id,
|
||||
current_depth,
|
||||
limit,
|
||||
len(sorted_backfill_points),
|
||||
sorted_backfill_points,
|
||||
)
|
||||
|
||||
if not filtered_extremities and not insertion_events_to_be_backfilled:
|
||||
return False
|
||||
|
||||
extremities = {
|
||||
**oldest_events_with_depth,
|
||||
# TODO: insertion_events_to_be_backfilled is currently skipping the filtered_extremities checks
|
||||
**insertion_events_to_be_backfilled,
|
||||
}
|
||||
|
||||
# Check if we reached a point where we should start backfilling.
|
||||
sorted_extremeties_tuple = sorted(extremities.items(), key=lambda e: -int(e[1]))
|
||||
max_depth = sorted_extremeties_tuple[0][1]
|
||||
|
||||
# If we're approaching an extremity we trigger a backfill, otherwise we
|
||||
# no-op.
|
||||
#
|
||||
|
|
@ -249,6 +233,11 @@ class FederationHandler:
|
|||
# chose more than one times the limit in case of failure, but choosing a
|
||||
# much larger factor will result in triggering a backfill request much
|
||||
# earlier than necessary.
|
||||
#
|
||||
# XXX: shouldn't we do this *after* the filter by depth below? Again, we don't
|
||||
# care about events that have happened after our current position.
|
||||
#
|
||||
max_depth = sorted_backfill_points[0].depth
|
||||
if current_depth - 2 * limit > max_depth:
|
||||
logger.debug(
|
||||
"Not backfilling as we don't need to. %d < %d - 2 * %d",
|
||||
|
|
@ -265,31 +254,98 @@ class FederationHandler:
|
|||
# 2. we have likely previously tried and failed to backfill from that
|
||||
# extremity, so to avoid getting "stuck" requesting the same
|
||||
# backfill repeatedly we drop those extremities.
|
||||
filtered_sorted_extremeties_tuple = [
|
||||
t for t in sorted_extremeties_tuple if int(t[1]) <= current_depth
|
||||
]
|
||||
|
||||
logger.debug(
|
||||
"room_id: %s, backfill: current_depth: %s, limit: %s, max_depth: %s, extrems (%d): %s filtered_sorted_extremeties_tuple: %s",
|
||||
room_id,
|
||||
current_depth,
|
||||
limit,
|
||||
max_depth,
|
||||
len(sorted_extremeties_tuple),
|
||||
sorted_extremeties_tuple,
|
||||
filtered_sorted_extremeties_tuple,
|
||||
)
|
||||
|
||||
#
|
||||
# However, we need to check that the filtered extremities are non-empty.
|
||||
# If they are empty then either we can a) bail or b) still attempt to
|
||||
# backfill. We opt to try backfilling anyway just in case we do get
|
||||
# relevant events.
|
||||
if filtered_sorted_extremeties_tuple:
|
||||
sorted_extremeties_tuple = filtered_sorted_extremeties_tuple
|
||||
#
|
||||
filtered_sorted_backfill_points = [
|
||||
t for t in sorted_backfill_points if t.depth <= current_depth
|
||||
]
|
||||
if filtered_sorted_backfill_points:
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: backfill points before current depth: %s",
|
||||
filtered_sorted_backfill_points,
|
||||
)
|
||||
sorted_backfill_points = filtered_sorted_backfill_points
|
||||
else:
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: all backfill points are *after* current depth. Backfilling anyway."
|
||||
)
|
||||
|
||||
# We don't want to specify too many extremities as it causes the backfill
|
||||
# request URI to be too long.
|
||||
extremities = dict(sorted_extremeties_tuple[:5])
|
||||
# For performance's sake, we only want to paginate from a particular extremity
|
||||
# if we can actually see the events we'll get. Otherwise, we'd just spend a lot
|
||||
# of resources to get redacted events. We check each extremity in turn and
|
||||
# ignore those which users on our server wouldn't be able to see.
|
||||
#
|
||||
# Additionally, we limit ourselves to backfilling from at most 5 extremities,
|
||||
# for two reasons:
|
||||
#
|
||||
# - The check which determines if we can see an extremity's events can be
|
||||
# expensive (we load the full state for the room at each of the backfill
|
||||
# points, or (worse) their successors)
|
||||
# - We want to avoid the server-server API request URI becoming too long.
|
||||
#
|
||||
# *Note*: the spec wants us to keep backfilling until we reach the start
|
||||
# of the room in case we are allowed to see some of the history. However,
|
||||
# in practice that causes more issues than its worth, as (a) it's
|
||||
# relatively rare for there to be any visible history and (b) even when
|
||||
# there is it's often sufficiently long ago that clients would stop
|
||||
# attempting to paginate before backfill reached the visible history.
|
||||
|
||||
extremities_to_request: List[str] = []
|
||||
for bp in sorted_backfill_points:
|
||||
if len(extremities_to_request) >= 5:
|
||||
break
|
||||
|
||||
# For regular backwards extremities, we don't have the extremity events
|
||||
# themselves, so we need to actually check the events that reference them -
|
||||
# their "successor" events.
|
||||
#
|
||||
# TODO: Correctly handle the case where we are allowed to see the
|
||||
# successor event but not the backward extremity, e.g. in the case of
|
||||
# initial join of the server where we are allowed to see the join
|
||||
# event but not anything before it. This would require looking at the
|
||||
# state *before* the event, ignoring the special casing certain event
|
||||
# types have.
|
||||
if bp.type == _BackfillPointType.INSERTION_PONT:
|
||||
event_ids_to_check = [bp.event_id]
|
||||
else:
|
||||
event_ids_to_check = await self.store.get_successor_events(bp.event_id)
|
||||
|
||||
events_to_check = await self.store.get_events_as_list(
|
||||
event_ids_to_check,
|
||||
redact_behaviour=EventRedactBehaviour.AS_IS,
|
||||
get_prev_content=False,
|
||||
)
|
||||
|
||||
# We set `check_history_visibility_only` as we might otherwise get false
|
||||
# positives from users having been erased.
|
||||
filtered_extremities = await filter_events_for_server(
|
||||
self.storage,
|
||||
self.server_name,
|
||||
events_to_check,
|
||||
redact=False,
|
||||
check_history_visibility_only=True,
|
||||
)
|
||||
if filtered_extremities:
|
||||
extremities_to_request.append(bp.event_id)
|
||||
else:
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: skipping extremity %s as it would not be visible",
|
||||
bp,
|
||||
)
|
||||
|
||||
if not extremities_to_request:
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: found no extremities which would be visible"
|
||||
)
|
||||
return False
|
||||
|
||||
logger.debug(
|
||||
"_maybe_backfill_inner: extremities_to_request %s", extremities_to_request
|
||||
)
|
||||
|
||||
# Now we need to decide which hosts to hit first.
|
||||
|
||||
|
|
@ -309,7 +365,7 @@ class FederationHandler:
|
|||
for dom in domains:
|
||||
try:
|
||||
await self._federation_event_handler.backfill(
|
||||
dom, room_id, limit=100, extremities=extremities
|
||||
dom, room_id, limit=100, extremities=extremities_to_request
|
||||
)
|
||||
# If this succeeded then we probably already have the
|
||||
# appropriate stuff.
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ class RoomBatchHandler:
|
|||
# it has a larger `depth` but before the successor event because the `stream_ordering`
|
||||
# is negative before the successor event.
|
||||
successor_event_ids = await self.store.get_successor_events(
|
||||
[most_recent_prev_event_id]
|
||||
most_recent_prev_event_id
|
||||
)
|
||||
|
||||
# If we can't find any successor events, then it's a forward extremity of
|
||||
|
|
|
|||
|
|
@ -695,7 +695,9 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
# Return all events where not all sets can reach them.
|
||||
return {eid for eid, n in event_to_missing_sets.items() if n}
|
||||
|
||||
async def get_oldest_event_ids_with_depth_in_room(self, room_id) -> Dict[str, int]:
|
||||
async def get_oldest_event_ids_with_depth_in_room(
|
||||
self, room_id
|
||||
) -> List[Tuple[str, int]]:
|
||||
"""Gets the oldest events(backwards extremities) in the room along with the
|
||||
aproximate depth.
|
||||
|
||||
|
|
@ -708,7 +710,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
room_id: Room where we want to find the oldest events
|
||||
|
||||
Returns:
|
||||
Map from event_id to depth
|
||||
List of (event_id, depth) tuples
|
||||
"""
|
||||
|
||||
def get_oldest_event_ids_with_depth_in_room_txn(txn, room_id):
|
||||
|
|
@ -741,7 +743,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
|
||||
txn.execute(sql, (room_id, False))
|
||||
|
||||
return dict(txn)
|
||||
return txn.fetchall()
|
||||
|
||||
return await self.db_pool.runInteraction(
|
||||
"get_oldest_event_ids_with_depth_in_room",
|
||||
|
|
@ -751,7 +753,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
|
||||
async def get_insertion_event_backward_extremities_in_room(
|
||||
self, room_id
|
||||
) -> Dict[str, int]:
|
||||
) -> List[Tuple[str, int]]:
|
||||
"""Get the insertion events we know about that we haven't backfilled yet.
|
||||
|
||||
We use this function so that we can compare and see if someones current
|
||||
|
|
@ -763,7 +765,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
room_id: Room where we want to find the oldest events
|
||||
|
||||
Returns:
|
||||
Map from event_id to depth
|
||||
List of (event_id, depth) tuples
|
||||
"""
|
||||
|
||||
def get_insertion_event_backward_extremities_in_room_txn(txn, room_id):
|
||||
|
|
@ -778,8 +780,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
"""
|
||||
|
||||
txn.execute(sql, (room_id,))
|
||||
|
||||
return dict(txn)
|
||||
return txn.fetchall()
|
||||
|
||||
return await self.db_pool.runInteraction(
|
||||
"get_insertion_event_backward_extremities_in_room",
|
||||
|
|
@ -1295,22 +1296,19 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
event_results.reverse()
|
||||
return event_results
|
||||
|
||||
async def get_successor_events(self, event_ids: Iterable[str]) -> List[str]:
|
||||
"""Fetch all events that have the given events as a prev event
|
||||
async def get_successor_events(self, event_id: str) -> List[str]:
|
||||
"""Fetch all events that have the given event as a prev event
|
||||
|
||||
Args:
|
||||
event_ids: The events to use as the previous events.
|
||||
event_id: The event to search for as a prev_event.
|
||||
"""
|
||||
rows = await self.db_pool.simple_select_many_batch(
|
||||
return await self.db_pool.simple_select_onecol(
|
||||
table="event_edges",
|
||||
column="prev_event_id",
|
||||
iterable=event_ids,
|
||||
retcols=("event_id",),
|
||||
keyvalues={"prev_event_id": event_id},
|
||||
retcol="event_id",
|
||||
desc="get_successor_events",
|
||||
)
|
||||
|
||||
return [row["event_id"] for row in rows]
|
||||
|
||||
@wrap_as_background_process("delete_old_forward_extrem_cache")
|
||||
async def _delete_old_forward_extrem_cache(self) -> None:
|
||||
def _delete_old_forward_extrem_cache_txn(txn):
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ from synapse.storage.util.id_generators import (
|
|||
from synapse.storage.util.sequence import build_sequence_generator
|
||||
from synapse.types import JsonDict, get_domain_from_id
|
||||
from synapse.util import unwrapFirstError
|
||||
from synapse.util.async_helpers import ObservableDeferred
|
||||
from synapse.util.async_helpers import ObservableDeferred, delay_cancellation
|
||||
from synapse.util.caches.descriptors import cached, cachedList
|
||||
from synapse.util.caches.lrucache import LruCache
|
||||
from synapse.util.iterutils import batch_iter
|
||||
|
|
@ -640,6 +640,13 @@ class EventsWorkerStore(SQLBaseStore):
|
|||
missing_events_ids.difference_update(already_fetching_ids)
|
||||
|
||||
if missing_events_ids:
|
||||
|
||||
async def get_missing_events_from_db() -> Dict[str, EventCacheEntry]:
|
||||
"""Fetches the events in `missing_event_ids` from the database.
|
||||
|
||||
Also creates entries in `self._current_event_fetches` to allow
|
||||
concurrent `_get_events_from_cache_or_db` calls to reuse the same fetch.
|
||||
"""
|
||||
log_ctx = current_context()
|
||||
log_ctx.record_event_fetch(len(missing_events_ids))
|
||||
|
||||
|
|
@ -656,15 +663,13 @@ class EventsWorkerStore(SQLBaseStore):
|
|||
|
||||
# Note that _get_events_from_db is also responsible for turning db rows
|
||||
# into FrozenEvents (via _get_event_from_row), which involves seeing if
|
||||
# the events have been redacted, and if so pulling the redaction event out
|
||||
# of the database to check it.
|
||||
# the events have been redacted, and if so pulling the redaction event
|
||||
# out of the database to check it.
|
||||
#
|
||||
try:
|
||||
missing_events = await self._get_events_from_db(
|
||||
missing_events_ids,
|
||||
)
|
||||
|
||||
event_entry_map.update(missing_events)
|
||||
except Exception as e:
|
||||
with PreserveLoggingContext():
|
||||
fetching_deferred.errback(e)
|
||||
|
|
@ -677,6 +682,16 @@ class EventsWorkerStore(SQLBaseStore):
|
|||
with PreserveLoggingContext():
|
||||
fetching_deferred.callback(missing_events)
|
||||
|
||||
return missing_events
|
||||
|
||||
# We must allow the database fetch to complete in the presence of
|
||||
# cancellations, since multiple `_get_events_from_cache_or_db` calls can
|
||||
# reuse the same fetch.
|
||||
missing_events: Dict[str, EventCacheEntry] = await delay_cancellation(
|
||||
get_missing_events_from_db()
|
||||
)
|
||||
event_entry_map.update(missing_events)
|
||||
|
||||
if already_fetching_deferreds:
|
||||
# Wait for the other event requests to finish and add their results
|
||||
# to ours.
|
||||
|
|
|
|||
|
|
@ -419,6 +419,13 @@ async def _event_to_memberships(
|
|||
return {}
|
||||
|
||||
# for each event, get the event_ids of the membership state at those events.
|
||||
#
|
||||
# TODO: this means that we request the entire membership list. If there are only
|
||||
# one or two users on this server, and the room is huge, this is very wasteful
|
||||
# (it means more db work, and churns the *stateGroupMembersCache*).
|
||||
# It might be that we could extend StateFilter to specify "give me keys matching
|
||||
# *:<server_name>", to avoid this.
|
||||
|
||||
event_to_state_ids = await storage.state.get_state_ids_for_events(
|
||||
frozenset(e.event_id for e in events),
|
||||
state_filter=StateFilter.from_types(types=((EventTypes.Member, None),)),
|
||||
|
|
|
|||
|
|
@ -13,10 +13,11 @@
|
|||
# limitations under the License.
|
||||
import json
|
||||
from contextlib import contextmanager
|
||||
from typing import Generator
|
||||
from typing import Generator, Tuple
|
||||
from unittest import mock
|
||||
|
||||
from twisted.enterprise.adbapi import ConnectionPool
|
||||
from twisted.internet.defer import ensureDeferred
|
||||
from twisted.internet.defer import CancelledError, Deferred, ensureDeferred
|
||||
from twisted.test.proto_helpers import MemoryReactor
|
||||
|
||||
from synapse.api.room_versions import EventFormatVersions, RoomVersions
|
||||
|
|
@ -281,3 +282,119 @@ class DatabaseOutageTestCase(unittest.HomeserverTestCase):
|
|||
|
||||
# This next event fetch should succeed
|
||||
self.get_success(self.store.get_event(self.event_ids[0]))
|
||||
|
||||
|
||||
class GetEventCancellationTestCase(unittest.HomeserverTestCase):
|
||||
"""Test cancellation of `get_event` calls."""
|
||||
|
||||
servlets = [
|
||||
admin.register_servlets,
|
||||
room.register_servlets,
|
||||
login.register_servlets,
|
||||
]
|
||||
|
||||
def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer):
|
||||
self.store: EventsWorkerStore = hs.get_datastores().main
|
||||
|
||||
self.user = self.register_user("user", "pass")
|
||||
self.token = self.login(self.user, "pass")
|
||||
|
||||
self.room = self.helper.create_room_as(self.user, tok=self.token)
|
||||
|
||||
res = self.helper.send(self.room, tok=self.token)
|
||||
self.event_id = res["event_id"]
|
||||
|
||||
# Reset the event cache so the tests start with it empty
|
||||
self.store._get_event_cache.clear()
|
||||
|
||||
@contextmanager
|
||||
def blocking_get_event_calls(
|
||||
self,
|
||||
) -> Generator[
|
||||
Tuple["Deferred[None]", "Deferred[None]", "Deferred[None]"], None, None
|
||||
]:
|
||||
"""Starts two concurrent `get_event` calls for the same event.
|
||||
|
||||
Both `get_event` calls will use the same database fetch, which will be blocked
|
||||
at the time this function returns.
|
||||
|
||||
Returns:
|
||||
A tuple containing:
|
||||
* A `Deferred` that unblocks the database fetch.
|
||||
* A cancellable `Deferred` for the first `get_event` call.
|
||||
* A cancellable `Deferred` for the second `get_event` call.
|
||||
"""
|
||||
# Patch `DatabasePool.runWithConnection` to block.
|
||||
unblock: "Deferred[None]" = Deferred()
|
||||
original_runWithConnection = self.store.db_pool.runWithConnection
|
||||
|
||||
async def runWithConnection(*args, **kwargs):
|
||||
await unblock
|
||||
return await original_runWithConnection(*args, **kwargs)
|
||||
|
||||
with mock.patch.object(
|
||||
self.store.db_pool,
|
||||
"runWithConnection",
|
||||
new=runWithConnection,
|
||||
):
|
||||
ctx1 = LoggingContext("get_event1")
|
||||
ctx2 = LoggingContext("get_event2")
|
||||
|
||||
async def get_event(ctx: LoggingContext) -> None:
|
||||
with ctx:
|
||||
await self.store.get_event(self.event_id)
|
||||
|
||||
get_event1 = ensureDeferred(get_event(ctx1))
|
||||
get_event2 = ensureDeferred(get_event(ctx2))
|
||||
|
||||
# Both `get_event` calls ought to be blocked.
|
||||
self.assertNoResult(get_event1)
|
||||
self.assertNoResult(get_event2)
|
||||
|
||||
yield unblock, get_event1, get_event2
|
||||
|
||||
# Confirm that the two `get_event` calls shared the same database fetch.
|
||||
self.assertEqual(ctx1.get_resource_usage().evt_db_fetch_count, 1)
|
||||
self.assertEqual(ctx2.get_resource_usage().evt_db_fetch_count, 0)
|
||||
|
||||
def test_first_get_event_cancelled(self):
|
||||
"""Test cancellation of the first `get_event` call sharing a database fetch.
|
||||
|
||||
The first `get_event` call is the one which initiates the fetch. We expect the
|
||||
fetch to complete despite the cancellation. Furthermore, the first `get_event`
|
||||
call must not abort before the fetch is complete, otherwise the fetch will be
|
||||
using a finished logging context.
|
||||
"""
|
||||
with self.blocking_get_event_calls() as (unblock, get_event1, get_event2):
|
||||
# Cancel the first `get_event` call.
|
||||
get_event1.cancel()
|
||||
# The first `get_event` call must not abort immediately, otherwise its
|
||||
# logging context will be finished while it is still in use by the database
|
||||
# fetch.
|
||||
self.assertNoResult(get_event1)
|
||||
# The second `get_event` call must not be cancelled.
|
||||
self.assertNoResult(get_event2)
|
||||
|
||||
# Unblock the database fetch.
|
||||
unblock.callback(None)
|
||||
# A `CancelledError` should be raised out of the first `get_event` call.
|
||||
exc = self.get_failure(get_event1, CancelledError).value
|
||||
self.assertIsInstance(exc, CancelledError)
|
||||
# The second `get_event` call should complete successfully.
|
||||
self.get_success(get_event2)
|
||||
|
||||
def test_second_get_event_cancelled(self):
|
||||
"""Test cancellation of the second `get_event` call sharing a database fetch."""
|
||||
with self.blocking_get_event_calls() as (unblock, get_event1, get_event2):
|
||||
# Cancel the second `get_event` call.
|
||||
get_event2.cancel()
|
||||
# The first `get_event` call must not be cancelled.
|
||||
self.assertNoResult(get_event1)
|
||||
# The second `get_event` call gets cancelled immediately.
|
||||
exc = self.get_failure(get_event2, CancelledError).value
|
||||
self.assertIsInstance(exc, CancelledError)
|
||||
|
||||
# Unblock the database fetch.
|
||||
unblock.callback(None)
|
||||
# The first `get_event` call should complete successfully.
|
||||
self.get_success(get_event1)
|
||||
|
|
|
|||
Loading…
Reference in New Issue