a2070a2c4e
Remove unused 'stream' param of REPLICATE and update docs
2020-03-23 14:56:22 +00:00
4f2a803c66
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/catchup_on_worker
2020-03-23 14:49:07 +00:00
a564b92d37
Convert `*StreamRow` classes to inner classes ( #7116 )
...
This just helps keep the rows closer to their streams, so that it's easier to
see what the format of each stream is.
2020-03-23 13:59:11 +00:00
b3cee0ce67
Fix processing of `groups` stream, and use symbolic names for streams ( #7117 )
...
`groups` != `receipts`
Introduced in #6964
2020-03-23 11:39:36 +00:00
32c656865a
Always subscribe to all streams.
...
This already happens since the worker merge.
2020-03-20 15:31:52 +00:00
8734b75ca8
Remove unused token param from REPLICATE cmd
2020-03-20 15:31:51 +00:00
1f83255de1
Move stream catchup to workers.
2020-03-20 15:31:49 +00:00
ba90596687
Add ability to catchup on stream by talking to master.
2020-03-20 15:31:47 +00:00
811d2ecf2e
Don't panic if streams get behind.
...
The catchup will in future happen on workers, so master process won't
need to protect itself by dropping the connection.
2020-03-20 15:31:45 +00:00
fdb1344716
Remove concept of a non-limited stream. ( #7011 )
2020-03-20 14:40:47 +00:00
9ce4e344a8
Change device list replication to match new semantics.
...
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
2020-02-28 11:25:34 +00:00
1f773eec91
Port PresenceHandler to async/await ( #6991 )
2020-02-26 15:33:26 +00:00
0bd8cf435e
Increase MAX_EVENTS_BEHIND for replication clients
2020-02-21 09:04:33 +00:00
c3d4ad8afd
Fix sending server up commands from workers ( #6811 )
...
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-01-30 16:42:11 +00:00
d5275fc55f
Propagate cache invalidates from workers to other workers. ( #6748 )
...
Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers.
2020-01-27 13:47:50 +00:00
5d7a6ad223
Allow streaming cache invalidate all to workers. ( #6749 )
2020-01-22 10:37:00 +00:00
a8a50f5b57
Wake up transaction queue when remote server comes back online ( #6706 )
...
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
2020-01-17 10:27:19 +00:00
48c3a96886
Port synapse.replication.tcp to async/await ( #6666 )
...
* Port synapse.replication.tcp to async/await
* Newsfile
* Correctly document type of on_<FOO> functions as async
* Don't be overenthusiastic with the asyncing....
2020-01-16 09:16:12 +00:00
e8b68a4e4b
Fixup synapse.replication to pass mypy checks ( #6667 )
2020-01-14 14:08:06 +00:00
6964ea095b
Reduce the reconnect time when replication fails. ( #6617 )
2020-01-03 14:19:09 +00:00
cd96b4586f
lint
2019-11-08 15:45:45 +00:00
c4bdf2d785
Remove content from being sent for account data rdata stream
2019-11-08 15:44:02 +00:00
cc6243b4c0
document the REPLICATE command a bit better ( #6305 )
...
since I found myself wonder how it works
2019-11-04 12:40:18 +00:00
9c94b48bf1
Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify
2019-10-31 12:32:07 -04:00
54fef094b3
Remove usage of deprecated logger.warn method from codebase ( #6271 )
...
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
998f7fe7d4
make user signatures a separate stream
2019-10-30 17:22:52 -04:00
4548d1f87e
Remove unnecessary parentheses around return statements ( #5931 )
...
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
b5c62c6b26
Fix relations in worker mode
2019-05-16 10:38:13 +01:00
4b91c313a9
Combine the CurrentStateDeltaStream into the EventStream
2019-03-27 22:07:05 +00:00
1f6d6f918a
Make EventStream rows have a type
...
... as a precursor to combining it with the CurrentStateDelta stream.
2019-03-27 22:07:05 +00:00
015b3622eb
Skip building a ROW_TYPE when building updates
...
We're about to turn it straight into a JSON object anyway so building a
ROW_TYPE is a bit pointless, and reduces flexibility in the update_function.
2019-03-27 21:58:03 +00:00
f570916a3e
Add parse_row method to replication stream class
...
This will allow individual stream classes to override how a row is parsed.
2019-03-27 21:32:33 +00:00
71dcb275f1
move FederationStream out to its own file
2019-03-27 21:13:14 +00:00
aa1e017864
move EventsStream out to its own file
2019-03-27 21:13:14 +00:00
a5798de067
Move replication.tcp.streams into a package
2019-03-27 21:13:14 +00:00
acaa18f7dd
Fix/improve some docstrings in the replication code. ( #4949 )
2019-03-27 21:12:36 +00:00
8cbbedaa2b
Fix ClientReplicationStreamProtocol.__str__ ( #4929 )
...
`__str__` depended on `self.addr`, which was absent from
ClientReplicationStreamProtocol, so attempting to call str on such an object
would raise an exception.
We can calculate the peer addr from the transport, so there is no need for addr
anyway.
2019-03-25 16:41:51 +00:00
9bde730ef8
Fix bug where read-receipts lost their timestamps ( #4927 )
...
Make sure that they are sent correctly over the replication stream.
Fixes : #4898
2019-03-25 16:38:05 +00:00
cdb8036161
Add a config option for torture-testing worker replication. ( #4902 )
...
Setting this to 50 or so makes a bunch of sytests fail in worker mode.
2019-03-20 16:04:35 +00:00
b9f6163092
Simplify token replication logic
2019-03-05 13:58:30 +00:00
fe7bd23a85
Clean up logic and add comments
2019-03-04 15:08:15 +00:00
9f7cdf3da1
Clearer branching, fix missing list clear
2019-03-04 14:36:52 +00:00
5f0c449dd5
Prevent replication wedging
2019-03-04 14:03:18 +00:00
7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff
...
Fix tightloop over connecting to replication server
2019-02-27 11:00:59 +00:00
6bb1c028f1
Limit cache invalidation replication line length ( #4748 )
2019-02-27 10:28:37 +00:00
6870fc496f
Move connecting logic into ClientReplicationStreamProtocol
2019-02-27 10:23:51 +00:00
25814921f1
Increase the max delay between retry attempts
...
Otherwise if you have many workers they can easily take out master with
their connection attempts
2019-02-26 15:12:33 +00:00
Erik Johnston