MatrixSynapse/synapse/storage
David Robertson 80d44060c9
Faster joins: omit partial rooms from eager syncs until the resync completes (#14870)
* Allow `AbstractSet` in `StrCollection`

Or else frozensets are excluded. This will be useful in an upcoming
commit where I plan to change a function that accepts `List[str]` to
accept `StrCollection` instead.

* `rooms_to_exclude` -> `rooms_to_exclude_globally`

I am about to make use of this exclusion mechanism to exclude rooms for
a specific user and a specific sync. This rename helps to clarify the
distinction between the global config and the rooms to exclude for a
specific sync.

* Better function names for internal sync methods

* Track a list of excluded rooms on SyncResultBuilder

I plan to feed a list of partially stated rooms for this sync to ignore

* Exclude partial state rooms during eager sync

using the mechanism established in the previous commit

* Track un-partial-state stream in sync tokens

So that we can work out which rooms have become fully-stated during a
given sync period.

* Fix mutation of `@cached` return value

This was fouling up a complement test added alongside this PR.
Excluding a room would mean the set of forgotten rooms in the cache
would be extended. This means that room could be erroneously considered
forgotten in the future.

Introduced in #12310, Synapse 1.57.0. I don't think this had any
user-visible side effects (until now).

* SyncResultBuilder: track rooms to force as newly joined

Similar plan as before. We've omitted rooms from certain sync responses;
now we establish the mechanism to reintroduce them into future syncs.

* Read new field, to present rooms as newly joined

* Force un-partial-stated rooms to be newly-joined

for eager incremental syncs only, provided they're still fully stated

* Notify user stream listeners to wake up long polling syncs

* Changelog

* Typo fix

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Unnecessary list cast

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Rephrase comment

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Another comment

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Fixup merge(?)

* Poke notifier when receiving un-partial-stated msg over replication

* Fixup merge whoops

Thanks MV :)

Co-authored-by: Mathieu Velen <mathieuv@matrix.org>

Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-01-23 15:44:39 +00:00
..
controllers Faster joins: Update room stats and the user directory on workers when finishing join (#14874) 2023-01-23 10:31:36 +00:00
databases Faster joins: omit partial rooms from eager syncs until the resync completes (#14870) 2023-01-23 15:44:39 +00:00
engines Enable `--warn-redundant-casts` option in mypy (#14671) 2022-12-12 21:25:07 +00:00
schema Drop unused table `presence` (#14825) 2023-01-13 14:17:03 +00:00
util Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
__init__.py Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
_base.py Update all stream IDs after processing replication rows (#14723) 2023-01-04 11:49:26 +00:00
background_updates.py Handle half-created indices in receipts index background update (#14650) 2022-12-09 23:02:11 +00:00
database.py Add experimental support for MSC3391: deleting account data (#14714) 2023-01-01 03:40:46 +00:00
keys.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
prepare_database.py Snapshot schema 72 (#13873) 2022-09-26 18:28:32 +01:00
push_rule.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
roommember.py Do not include rooms with an unknown room version in a sync response. (#10644) 2021-08-19 11:12:55 -04:00
types.py Tidy up and type-hint the database engine modules (#12734) 2022-05-17 00:34:38 +01:00