73ff493dfb
Merge account data streams ( #14826 )
2023-01-13 14:57:43 +00:00
db1cfe9c80
Update all stream IDs after processing replication rows ( #14723 )
...
This creates a new store method, `process_replication_position` that
is called after `process_replication_rows`. By moving stream ID advances
here this guarantees any relevant cache invalidations will have been
applied before the stream is advanced.
This avoids race conditions where Python switches between threads mid
way through processing the `process_replication_rows` method where stream
IDs may be advanced before caches are invalidated due to class resolution
ordering.
See this comment/issue for further discussion:
https://github.com/matrix-org/synapse/issues/14158#issuecomment-1344048703
2023-01-04 11:49:26 +00:00
2888d7ec83
Faster remote room joins: invalidate caches and unblock requests when receiving un-partial-stated event notifications over replication. [rei:frrj/streams/unpsr] ( #14546 )
2022-12-19 14:57:51 +00:00
fb60cb16fe
Faster remote room joins: stream the un-partial-stating of events over replication. [rei:frrj/streams/unpsr] ( #14545 )
2022-12-14 14:47:11 +00:00
9e82caac45
Faster remote room joins: unblock tasks waiting for full room state when the un-partial-stating of that room is received over the replication stream. [rei:frrj/streams/unpsr] ( #14474 )
2022-12-06 15:48:42 +00:00
501f62d1a6
Faster remote room joins: stream the un-partial-stating of rooms over replication. [rei:frrj/streams/unpsr] ( #14473 )
2022-12-05 13:07:55 +00:00
e7132c3f81
Fix check to ignore blank lines in incoming TCP replication ( #14449 )
2022-11-17 16:09:56 +00:00
7b7478e8b6
Batch up notifications after event persistence ( #14033 )
2022-10-05 10:12:48 -07:00
efd108b45d
Accept & store thread IDs for receipts (implement MSC3771). ( #13782 )
...
Updates the `/receipts` endpoint and receipt EDU handler to parse a
`thread_id` from the body and insert it in the database.
2022-09-23 14:33:28 +00:00
8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) ( #13799 )
...
Partial implementation of MSC3881
2022-09-21 14:39:01 +00:00
32fc3b7ba4
Remove configuration options for direct TCP replication. ( #13647 )
...
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-06 07:50:02 +00:00
0e99f07952
Remove support for unstable private read receipts ( #13653 )
...
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-09-01 13:31:54 +01:00
ab18441573
Support stable identifiers for MSC2285: private read receipts. ( #13273 )
...
This adds support for the stable identifiers of MSC2285 while
continuing to support the unstable identifiers behind the configuration
flag. These will be removed in a future version.
2022-08-05 11:09:33 -04:00
b977867358
Rate limit joins per-room ( #13276 )
2022-07-19 11:45:17 +00:00
f721f1baba
Revert "Make all `process_replication_rows` methods async ( #13304 )" ( #13312 )
...
This reverts commit 5d4028f217
2022-07-18 14:28:14 +01:00
5d4028f217
Make all `process_replication_rows` methods async ( #13304 )
...
More prep work for asyncronous caching, also makes all process_replication_rows methods consistent (presence handler already is so).
Signed off by Nick @ Beeper (@Fizzadar)
2022-07-17 22:19:43 +01:00
cf05258f76
Remove groups replication code. ( #12900 )
...
The replication logic for groups is no longer used, so the message
passing infrastructure can be removed.
2022-05-31 13:04:08 -04:00
39dee30f01
Send `USER_IP` commands on a different Redis channel, in order to reduce traffic to workers that do not process these commands. ( #12809 )
2022-05-20 15:28:23 +01:00
177b884ad7
Lay some foundation work to allow workers to only subscribe to some kinds of messages, reducing replication traffic. ( #12672 )
2022-05-19 16:29:08 +01:00
83be72d76c
Add `StreamKeyType` class and replace string literals with constants ( #12567 )
2022-05-16 15:35:31 +00:00
d80a7ab151
Update `replication.md` with info on TCP module structure ( #12621 )
2022-05-09 14:46:43 -07:00
ef86cf3d28
Update `_on_new_receipts()` to work with MSC2285 changes. ( #12636 )
2022-05-05 13:25:51 +00:00
c0379d6e5b
Reduce log spam when running multiple event persisters ( #12610 )
2022-05-05 10:20:23 +01:00
d1cd96ce29
Add opentracing spans to calls to external cache ( #12380 )
2022-04-07 13:18:29 +01:00
800ba87cc8
Refactor and convert `Linearizer` to async ( #12357 )
...
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.
Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.
Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
f871222880
Move `update_client_ip` background job from the main process to the background worker. ( #12251 )
2022-04-01 13:08:55 +01:00
4a53f35737
Improve code documentation for the typing stream over replication. ( #12211 )
2022-03-11 14:00:15 +00:00
3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. ( #12192 )
...
Since the object it returns is a ReplicationCommandHandler.
This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
d8bab6793c
Fix incorrect type hints for txredis. ( #12042 )
...
Some properties were marked as RedisProtocol instead of ConnectionHandler,
which wraps RedisProtocol instance(s).
2022-03-08 07:26:05 -05:00
423cca9efe
Spread out sending device lists to remote hosts ( #12132 )
2022-03-04 11:48:15 +00:00
e24ff8ebe3
Remove `HomeServer.get_datastore()` ( #12031 )
...
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
2022-02-23 11:04:02 +00:00
d0e78af35e
Add missing type hints to synapse.replication. ( #11938 )
2022-02-08 11:03:08 -05:00
6c0984e3f0
Remove unnecessary ignores due to Twisted upgrade. ( #11939 )
...
Twisted 22.1.0 fixed some internal type hints, allowing Synapse
to remove ignore calls for parameters to connectTCP.
2022-02-08 09:15:59 -05:00
10a88ba91c
Use auto_attribs/native type hints for attrs classes. ( #11692 )
2022-01-13 13:49:28 +00:00
cbd82d0b2d
Convert all namedtuples to attrs. ( #11665 )
...
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
ffd858aa68
Add type hints to `synapse/storage/databases/main/events_worker.py` ( #11411 )
...
Also refactor the stream ID trackers/generators a bit and try to
document them better.
2021-11-26 18:41:31 +00:00
5cace20bf1
Add missing type hints to `synapse.app`. ( #11287 )
2021-11-10 15:06:54 -05:00
af54167516
Enable passing typing stream writers as a list. ( #11237 )
...
This makes the typing stream writer config match the other stream writers
that only currently support a single worker.
2021-11-03 14:25:47 +00:00
c7a5e49664
Implement an `on_new_event` callback ( #11126 )
...
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
2b82ec425f
Add type hints for most `HomeServer` parameters ( #11095 )
2021-10-22 18:15:41 +01:00
6a67f3786a
Fix logging context warnings when losing replication connection ( #10984 )
...
Instead of triggering `__exit__` manually on the replication handler's
logging context, use it as a context manager so that there is an
`__enter__` call to balance the `__exit__`.
2021-10-15 13:10:58 +01:00
f4b1a9a527
Require direct references to configuration variables. ( #10985 )
...
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
2021-10-06 10:47:41 -04:00
29364145b2
Pass str to twisted's IReactorTCP ( #10895 )
...
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.
Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
2021-09-30 12:51:47 +01:00
94b620a5ed
Use direct references for configuration variables (part 6). ( #10916 )
2021-09-29 06:44:15 -04:00
bb7fdd821b
Use direct references for configuration variables (part 5). ( #10897 )
2021-09-24 07:25:21 -04:00
01c88a09cd
Use direct references for some configuration variables ( #10798 )
...
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
84469bdac7
Remove the unused public_room_list_stream ( #10565 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-08-17 14:02:50 +01:00
d9cb658c78
Fix up type hints for Twisted 21.7 ( #10490 )
...
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
2021-07-28 12:04:11 +00:00
c3b037795a
Support for MSC2285 (hidden read receipts) ( #10413 )
...
Implementation of matrix-org/matrix-doc#2285
2021-07-28 10:05:11 +02:00
bf72d10dbf
Use inline type hints in various other places (in `synapse/`) ( #10380 )
2021-07-15 11:02:43 +01:00
Erik Johnston