Merge remote-tracking branch 'origin/release-v1.76' into matrix-org-hotfixes

anoa/redirect_instances
David Robertson 2023-01-26 17:20:12 +00:00
commit 484db60cd6
No known key found for this signature in database
GPG Key ID: 903ECE108A39DEDD
68 changed files with 142 additions and 73 deletions

View File

@ -1,3 +1,96 @@
Synapse 1.76.0rc1 (2023-01-25)
==============================
This release is the first to enable faster joins ([MSC3706](https://github.com/matrix-org/matrix-spec-proposals/pull/3706) and [MSC3902](https://github.com/matrix-org/matrix-spec-proposals/pull/3902)) by default. Admins can opt-out: see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.76/upgrade.md#faster-joins-are-enabled-by-default) for more details.
The upgrade from 1.75 to 1.76 changes the account data replication streams in a backwards-incompatible manner. Server operators running a multi-worker deployment should consult [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.76/docs/upgrade.md#changes-to-the-account-data-replication-streams).
Those who are `poetry install`ing from source using our lockfile should ensure their poetry version is 1.3.2 or higher; [see upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.76/docs/upgrade.md#minimum-version-of-poetry-is-now-132).
Features
--------
- Update the default room version to [v10](https://spec.matrix.org/v1.5/rooms/v10/) ([MSC 3904](https://github.com/matrix-org/matrix-spec-proposals/pull/3904)). Contributed by @FSG-Cat. ([\#14111](https://github.com/matrix-org/synapse/issues/14111))
- Add a `set_displayname()` method to the module API for setting a user's display name. ([\#14629](https://github.com/matrix-org/synapse/issues/14629))
- Add a dedicated listener configuration for `health` endpoint. ([\#14747](https://github.com/matrix-org/synapse/issues/14747))
- Implement support for [MSC3890](https://github.com/matrix-org/matrix-spec-proposals/pull/3890): Remotely silence local notifications. ([\#14775](https://github.com/matrix-org/synapse/issues/14775))
- Implement experimental support for [MSC3930](https://github.com/matrix-org/matrix-spec-proposals/pull/3930): Push rules for ([MSC3381](https://github.com/matrix-org/matrix-spec-proposals/pull/3381)) Polls. ([\#14787](https://github.com/matrix-org/synapse/issues/14787))
- Per [MSC3925](https://github.com/matrix-org/matrix-spec-proposals/pull/3925), bundle the whole of the replacement with any edited events, and optionally inhibit server-side replacement. ([\#14811](https://github.com/matrix-org/synapse/issues/14811))
- Faster joins: always serve a partial join response to servers that request it with the stable query param. ([\#14839](https://github.com/matrix-org/synapse/issues/14839))
- Faster joins: allow non-lazy-loading ("eager") syncs to complete after a partial join by omitting partial state rooms until they become fully stated. ([\#14870](https://github.com/matrix-org/synapse/issues/14870))
- Faster joins: request partial joins by default. Admins can opt-out of this for the time being---see the upgrade notes. ([\#14905](https://github.com/matrix-org/synapse/issues/14905))
Bugfixes
--------
- Add index to improve performance of the `/timestamp_to_event` endpoint used for jumping to a specific date in the timeline of a room. ([\#14799](https://github.com/matrix-org/synapse/issues/14799))
- Fix a long-standing bug where Synapse would exhaust the stack when processing many federation requests where the remote homeserver has disconencted early. ([\#14812](https://github.com/matrix-org/synapse/issues/14812), [\#14842](https://github.com/matrix-org/synapse/issues/14842))
- Fix rare races when using workers. ([\#14820](https://github.com/matrix-org/synapse/issues/14820))
- Fix a bug introduced in Synapse 1.64.0 when using room version 10 with frozen events enabled. ([\#14864](https://github.com/matrix-org/synapse/issues/14864))
- Fix a long-standing bug where the `populate_room_stats` background job could fail on broken rooms. ([\#14873](https://github.com/matrix-org/synapse/issues/14873))
- Faster joins: Fix a bug in worker deployments where the room stats and user directory would not get updated when finishing a fast join until another event is sent or received. ([\#14874](https://github.com/matrix-org/synapse/issues/14874))
- Faster joins: Fix incompatibility with joins into restricted rooms where no local users have the ability to invite. ([\#14882](https://github.com/matrix-org/synapse/issues/14882))
- Fix a regression introduced in Synapse 1.69.0 which can result in database corruption when database migrations are interrupted on sqlite. ([\#14910](https://github.com/matrix-org/synapse/issues/14910))
Updates to the Docker image
---------------------------
- Bump default Python version in the Dockerfile from 3.9 to 3.11. ([\#14875](https://github.com/matrix-org/synapse/issues/14875))
Improved Documentation
----------------------
- Include `x_forwarded` entry in the HTTP listener example configs and remove the remaining `worker_main_http_uri` entries. ([\#14667](https://github.com/matrix-org/synapse/issues/14667))
- Remove duplicate commands from the Code Style documentation page; point to the Contributing Guide instead. ([\#14773](https://github.com/matrix-org/synapse/issues/14773))
- Add missing documentation for `tag` to `listeners` section. ([\#14803](https://github.com/matrix-org/synapse/issues/14803))
- Updated documentation in configuration manual for `user_directory.search_all_users`. ([\#14818](https://github.com/matrix-org/synapse/issues/14818))
- Add `worker_manhole` to configuration manual. ([\#14824](https://github.com/matrix-org/synapse/issues/14824))
- Fix the example config missing the `id` field in [application service documentation](https://matrix-org.github.io/synapse/latest/application_services.html). ([\#14845](https://github.com/matrix-org/synapse/issues/14845))
- Minor corrections to the logging configuration documentation. ([\#14868](https://github.com/matrix-org/synapse/issues/14868))
- Document the export user data command. Contributed by @thezaidbintariq. ([\#14883](https://github.com/matrix-org/synapse/issues/14883))
Deprecations and Removals
-------------------------
- Poetry 1.3.2 or higher is now required when `poetry install`ing from source. ([\#14860](https://github.com/matrix-org/synapse/issues/14860))
Internal Changes
----------------
- Faster remote room joins (worker mode): do not populate external hosts-in-room cache when sending events as this requires blocking for full state. ([\#14749](https://github.com/matrix-org/synapse/issues/14749))
- Enable Complement tests for Faster Remote Room Joins against worker-mode Synapse. ([\#14752](https://github.com/matrix-org/synapse/issues/14752))
- Add some clarifying comments and refactor a portion of the `Keyring` class for readability. ([\#14804](https://github.com/matrix-org/synapse/issues/14804))
- Add local poetry config files (`poetry.toml`) to `.gitignore`. ([\#14807](https://github.com/matrix-org/synapse/issues/14807))
- Add missing type hints. ([\#14816](https://github.com/matrix-org/synapse/issues/14816), [\#14885](https://github.com/matrix-org/synapse/issues/14885), [\#14889](https://github.com/matrix-org/synapse/issues/14889))
- Refactor push tests. ([\#14819](https://github.com/matrix-org/synapse/issues/14819))
- Re-enable some linting that was disabled when we switched to ruff. ([\#14821](https://github.com/matrix-org/synapse/issues/14821))
- Add `cargo fmt` and `cargo clippy` to the lint script. ([\#14822](https://github.com/matrix-org/synapse/issues/14822))
- Drop unused table `presence`. ([\#14825](https://github.com/matrix-org/synapse/issues/14825))
- Merge the two account data and the two device list replication streams. ([\#14826](https://github.com/matrix-org/synapse/issues/14826), [\#14833](https://github.com/matrix-org/synapse/issues/14833))
- Faster joins: use stable identifiers from [MSC3706](https://github.com/matrix-org/matrix-spec-proposals/pull/3706). ([\#14832](https://github.com/matrix-org/synapse/issues/14832), [\#14841](https://github.com/matrix-org/synapse/issues/14841))
- Add a parameter to control whether the federation client performs a partial state join. ([\#14843](https://github.com/matrix-org/synapse/issues/14843))
- Add check to avoid starting duplicate partial state syncs. ([\#14844](https://github.com/matrix-org/synapse/issues/14844))
- Add an early return when handling no-op presence updates. ([\#14855](https://github.com/matrix-org/synapse/issues/14855))
- Fix `wait_for_stream_position` to correctly wait for the right instance to advance its token. ([\#14856](https://github.com/matrix-org/synapse/issues/14856), [\#14872](https://github.com/matrix-org/synapse/issues/14872))
- Always notify replication when a stream advances automatically. ([\#14877](https://github.com/matrix-org/synapse/issues/14877))
- Reduce max time we wait for stream positions. ([\#14881](https://github.com/matrix-org/synapse/issues/14881))
- Faster joins: allow the resync process more time to fetch `/state` ids. ([\#14912](https://github.com/matrix-org/synapse/issues/14912))
- Bump regex from 1.7.0 to 1.7.1. ([\#14848](https://github.com/matrix-org/synapse/issues/14848))
- Bump peaceiris/actions-gh-pages from 3.9.1 to 3.9.2. ([\#14861](https://github.com/matrix-org/synapse/issues/14861))
- Bump ruff from 0.0.215 to 0.0.224. ([\#14862](https://github.com/matrix-org/synapse/issues/14862))
- Bump types-pillow from 9.4.0.0 to 9.4.0.3. ([\#14863](https://github.com/matrix-org/synapse/issues/14863))
- Bump types-opentracing from 2.4.10 to 2.4.10.1. ([\#14896](https://github.com/matrix-org/synapse/issues/14896))
- Bump ruff from 0.0.224 to 0.0.230. ([\#14897](https://github.com/matrix-org/synapse/issues/14897))
- Bump types-requests from 2.28.11.7 to 2.28.11.8. ([\#14899](https://github.com/matrix-org/synapse/issues/14899))
- Bump types-psycopg2 from 2.9.21.2 to 2.9.21.4. ([\#14900](https://github.com/matrix-org/synapse/issues/14900))
- Bump types-commonmark from 0.9.2 to 0.9.2.1. ([\#14901](https://github.com/matrix-org/synapse/issues/14901))
Synapse 1.75.0 (2023-01-17)
===========================

View File

@ -1 +0,0 @@
Update the default room version to [v10](https://spec.matrix.org/v1.5/rooms/v10/) ([MSC 3904](https://github.com/matrix-org/matrix-spec-proposals/pull/3904)). Contributed by @FSG-Cat.

View File

@ -1 +0,0 @@
Adds a `set_displayname()` method to the module API for setting a user's display name.

View File

@ -1 +0,0 @@
Include `x_forwarded` entry in the HTTP listener example configs and remove the remaining `worker_main_http_uri` entries.

View File

@ -1 +0,0 @@
Add a dedicated listener configuration for `health` endpoint.

View File

@ -1 +0,0 @@
Faster remote room joins (worker mode): do not populate external hosts-in-room cache when sending events as this requires blocking for full state.

View File

@ -1 +0,0 @@
Enable Complement tests for Faster Remote Room Joins against worker-mode Synapse.

View File

@ -1 +0,0 @@
Remove duplicate commands from the Code Style documentation page; point to the Contributing Guide instead.

View File

@ -1 +0,0 @@
Implement support for MSC3890: Remotely silence local notifications.

View File

@ -1 +0,0 @@
Implement experimental support for MSC3930: Push rules for (MSC3381) Polls.

View File

@ -1 +0,0 @@
Add index to improve performance of the `/timestamp_to_event` endpoint used for jumping to a specific date in the timeline of a room.

View File

@ -1 +0,0 @@
Add missing documentation for `tag` to `listeners` section.

View File

@ -1 +0,0 @@
Add some clarifying comments and refactor a portion of the `Keyring` class for readability.

View File

@ -1 +0,0 @@
Add local poetry config files (`poetry.toml`) to `.gitignore`.

View File

@ -1 +0,0 @@
Per [MSC3925](https://github.com/matrix-org/matrix-spec-proposals/pull/3925), bundle the whole of the replacement with any edited events, and optionally inhibit server-side replacement.

View File

@ -1 +0,0 @@
Fix a long-standing bug where Synapse would exhaust the stack when processing many federation requests where the remote homeserver has disconencted early.

View File

@ -1 +0,0 @@
Add missing type hints.

View File

@ -1 +0,0 @@
Updated documentation in configuration manual for `user_directory.search_all_users`.

View File

@ -1 +0,0 @@
Refactor push tests.

View File

@ -1 +0,0 @@
Fix rare races when using workers.

View File

@ -1 +0,0 @@
Re-enable some linting that was disabled when we switched to ruff.

View File

@ -1 +0,0 @@
Add `cargo fmt` and `cargo clippy` to the lint script.

View File

@ -1 +0,0 @@
Add `worker_manhole` to configuration manual.

View File

@ -1 +0,0 @@
Drop unused table `presence`.

View File

@ -1 +0,0 @@
Merge the two account data and the two device list replication streams.

View File

@ -1 +0,0 @@
Faster joins: use stable identifiers from [MSC3706](https://github.com/matrix-org/matrix-spec-proposals/pull/3706).

View File

@ -1 +0,0 @@
Merge the two account data and the two device list replication streams.

View File

@ -1 +0,0 @@
Faster joins: always serve a partial join response to servers that request it with the stable query param.

View File

@ -1 +0,0 @@
Faster joins: use stable identifiers from [MSC3706](https://github.com/matrix-org/matrix-spec-proposals/pull/3706).

View File

@ -1 +0,0 @@
Fix a long-standing bug where Synapse would exhaust the stack when processing many federation requests where the remote homeserver has disconencted early.

View File

@ -1 +0,0 @@
Add a parameter to control whether the federation client performs a partial state join.

View File

@ -1 +0,0 @@
Add check to avoid starting duplicate partial state syncs.

View File

@ -1 +0,0 @@
Fix the example config missing the `id` field in [application service documentation](https://matrix-org.github.io/synapse/latest/application_services.html).

View File

@ -1 +0,0 @@
Bump regex from 1.7.0 to 1.7.1.

View File

@ -1 +0,0 @@
Add an early return when handling no-op presence updates.

View File

@ -1 +0,0 @@
Fix `wait_for_stream_position` to correctly wait for the right instance to advance its token.

View File

@ -1 +0,0 @@
Poetry 1.3.2 or higher is now required when `poetry install`ing from source.

View File

@ -1 +0,0 @@
Bump peaceiris/actions-gh-pages from 3.9.1 to 3.9.2.

View File

@ -1 +0,0 @@
Bump ruff from 0.0.215 to 0.0.224.

View File

@ -1 +0,0 @@
Bump types-pillow from 9.4.0.0 to 9.4.0.3.

View File

@ -1 +0,0 @@
Fix a bug introduced in Synapse 1.64.0 when using room version 10 with frozen events enabled.

View File

@ -1 +0,0 @@
Minor corrections to the logging configuration documentation.

View File

@ -1 +0,0 @@
Faster joins: allow non-lazy-loading ("eager") syncs to complete after a partial join by omitting partial state rooms until they become fully stated.

View File

@ -1 +0,0 @@
Fix `wait_for_stream_position` to correctly wait for the right instance to advance its token.

View File

@ -1 +0,0 @@
Fix a long-standing bug where the `populate_room_stats` background job could fail on broken rooms.

View File

@ -1 +0,0 @@
Faster joins: Fix a bug in worker deployments where the room stats and user directory would not get updated when finishing a fast join until another event is sent or received.

View File

@ -1 +0,0 @@
Bump default Python version in the Dockerfile from 3.9 to 3.11.

View File

@ -1 +0,0 @@
Always notify replication when a stream advances automatically.

View File

@ -1 +0,0 @@
Reduce max time we wait for stream positions.

View File

@ -1 +0,0 @@
Faster joins: Fix incompatibility with joins into restricted rooms where no local users have the ability to invite.

View File

@ -1 +0,0 @@
Document the export user data command. Contributed by @thezaidbintariq.

View File

@ -1 +0,0 @@
Add missing type hints.

View File

@ -1 +0,0 @@
Add missing type hints.

View File

@ -1 +0,0 @@
Bump types-opentracing from 2.4.10 to 2.4.10.1.

View File

@ -1 +0,0 @@
Bump ruff from 0.0.224 to 0.0.230.

View File

@ -1 +0,0 @@
Bump types-requests from 2.28.11.7 to 2.28.11.8.

View File

@ -1 +0,0 @@
Bump types-psycopg2 from 2.9.21.2 to 2.9.21.4.

View File

@ -1 +0,0 @@
Bump types-commonmark from 0.9.2 to 0.9.2.1.

View File

@ -1 +0,0 @@
Faster joins: request partial joins by default. Admins can opt-out of this for the time being---see the upgrade notes.

View File

@ -1 +0,0 @@
Fix a regression introduced in Synapse 1.69.0 which can result in database corruption when database migrations are interrupted on sqlite.

1
changelog.d/14914.bugfix Normal file
View File

@ -0,0 +1 @@
Faster joins: Fix a bug introduced in Synapse 1.69 where device list EDUs could fail to be handled after a restart when a faster join sync is in progress.

1
changelog.d/14917.misc Normal file
View File

@ -0,0 +1 @@
Faster joins: Improve performance of looking up partial-state status of rooms.

5
debian/changelog vendored
View File

@ -1,8 +1,9 @@
matrix-synapse-py3 (1.75.1) UNRELEASED; urgency=medium
matrix-synapse-py3 (1.76.0~rc1) stable; urgency=medium
* Use Poetry 1.3.2 to manage the bundled virtualenv included with this package.
* New Synapse release 1.76.0rc1.
-- Synapse Packaging team <packages@matrix.org> Tue, 17 Jan 2023 15:08:00 +0000
-- Synapse Packaging team <packages@matrix.org> Wed, 25 Jan 2023 16:21:16 +0000
matrix-synapse-py3 (1.75.0) stable; urgency=medium

View File

@ -89,7 +89,7 @@ manifest-path = "rust/Cargo.toml"
[tool.poetry]
name = "matrix-synapse"
version = "1.75.0"
version = "1.76.0rc1"
description = "Homeserver for the Matrix decentralised comms protocol"
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
license = "Apache-2.0"

View File

@ -1383,16 +1383,21 @@ class SyncHandler:
if not sync_config.filter_collection.lazy_load_members():
# Non-lazy syncs should never include partially stated rooms.
# Exclude all partially stated rooms from this sync.
for room_id in mutable_joined_room_ids:
if await self.store.is_partial_state_room(room_id):
mutable_rooms_to_exclude.add(room_id)
results = await self.store.is_partial_state_room_batched(
mutable_joined_room_ids
)
mutable_rooms_to_exclude.update(
room_id
for room_id, is_partial_state in results.items()
if is_partial_state
)
# Incremental eager syncs should additionally include rooms that
# - we are joined to
# - are full-stated
# - became fully-stated at some point during the sync period
# (These rooms will have been omitted during a previous eager sync.)
forced_newly_joined_room_ids = set()
forced_newly_joined_room_ids: Set[str] = set()
if since_token and not sync_config.filter_collection.lazy_load_members():
un_partial_stated_rooms = (
await self.store.get_un_partial_stated_rooms_between(
@ -1401,9 +1406,14 @@ class SyncHandler:
mutable_joined_room_ids,
)
)
for room_id in un_partial_stated_rooms:
if not await self.store.is_partial_state_room(room_id):
forced_newly_joined_room_ids.add(room_id)
results = await self.store.is_partial_state_room_batched(
un_partial_stated_rooms
)
forced_newly_joined_room_ids.update(
room_id
for room_id, is_partial_state in results.items()
if not is_partial_state
)
# Now we have our list of joined room IDs, exclude as configured and freeze
joined_room_ids = frozenset(

View File

@ -1819,7 +1819,7 @@ class DatabasePool:
keyvalues: Optional[Dict[str, Any]] = None,
desc: str = "simple_select_many_batch",
batch_size: int = 100,
) -> List[Any]:
) -> List[Dict[str, Any]]:
"""Executes a SELECT query on the named table, which may return zero or
more rows, returning the result as a list of dicts.

View File

@ -99,6 +99,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
("user_signature_stream", "stream_id"),
("device_lists_outbound_pokes", "stream_id"),
("device_lists_changes_in_room", "stream_id"),
("device_lists_remote_pending", "stream_id"),
],
is_writer=hs.config.worker.worker_app is None,
)

View File

@ -60,9 +60,9 @@ from synapse.storage.util.id_generators import (
MultiWriterIdGenerator,
StreamIdGenerator,
)
from synapse.types import JsonDict, RetentionPolicy, ThirdPartyInstanceID
from synapse.types import JsonDict, RetentionPolicy, StrCollection, ThirdPartyInstanceID
from synapse.util import json_encoder
from synapse.util.caches.descriptors import cached
from synapse.util.caches.descriptors import cached, cachedList
from synapse.util.stringutils import MXC_REGEX
if TYPE_CHECKING:
@ -1255,7 +1255,7 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):
return room_servers
@cached()
@cached(max_entries=10000)
async def is_partial_state_room(self, room_id: str) -> bool:
"""Checks if this room has partial state.
@ -1274,6 +1274,27 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):
return entry is not None
@cachedList(cached_method_name="is_partial_state_room", list_name="room_ids")
async def is_partial_state_room_batched(
self, room_ids: StrCollection
) -> Mapping[str, bool]:
"""Checks if the given rooms have partial state.
Returns true for "partial-state" rooms, which means that the state
at events in the room, and `current_state_events`, may not yet be
complete.
"""
rows: List[Dict[str, str]] = await self.db_pool.simple_select_many_batch(
table="partial_state_rooms",
column="room_id",
iterable=room_ids,
retcols=("room_id",),
desc="is_partial_state_room_batched",
)
partial_state_rooms = {row_dict["room_id"] for row_dict in rows}
return {room_id: room_id in partial_state_rooms for room_id in room_ids}
async def get_join_event_id_and_device_lists_stream_id_for_partial_state(
self, room_id: str
) -> Tuple[str, int]: