Commit Graph

346 Commits (ce0ce1add3dde3da2ff366ee873174f5b5e70763)

Author SHA1 Message Date
Richard van der Hoff ce0ce1add3 Merge branch 'develop' into matrix-org-hotfixes 2019-03-25 16:48:56 +00:00
Richard van der Hoff 8cbbedaa2b
Fix ClientReplicationStreamProtocol.__str__ (#4929)
`__str__` depended on `self.addr`, which was absent from
ClientReplicationStreamProtocol, so attempting to call str on such an object
would raise an exception.

We can calculate the peer addr from the transport, so there is no need for addr
anyway.
2019-03-25 16:41:51 +00:00
Richard van der Hoff 9bde730ef8
Fix bug where read-receipts lost their timestamps (#4927)
Make sure that they are sent correctly over the replication stream.

Fixes: #4898
2019-03-25 16:38:05 +00:00
Erik Johnston b0bf1ea7bd Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-03-21 14:10:31 +00:00
Richard van der Hoff cdb8036161
Add a config option for torture-testing worker replication. (#4902)
Setting this to 50 or so makes a bunch of sytests fail in worker mode.
2019-03-20 16:04:35 +00:00
Erik Johnston 8f7dbbc14a Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-03-06 19:30:30 +00:00
Erik Johnston face0c5b3c Prefill client IPs cache on workers 2019-03-06 17:39:32 +00:00
Andrew Morgan 7b8a157b79
Merge pull request #4792 from matrix-org/anoa/replication_tokens
Support batch updates in the worker sender
2019-03-06 15:48:29 +00:00
Richard van der Hoff e9aa401994
Remove redundant changes from synapse/replication/tcp/streams.py (#4813)
This was some hacky code (introduced in c10c71e70d) to make the presence stream
do nothing on hotfixes. We now ensure that no replication clients subscribe to
the presence stream, so this is redundant.
2019-03-06 13:21:32 +00:00
Erik Johnston dc510e0e43 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-03-05 14:41:13 +00:00
Brendan Abolivier a4c3a361b7
Add rate-limiting on registration (#4735)
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
2019-03-05 14:25:33 +00:00
Andrew Morgan b9f6163092 Simplify token replication logic 2019-03-05 13:58:30 +00:00
Erik Johnston a84b8d56c2 Fixup slave stores 2019-03-04 18:04:57 +00:00
Andrew Morgan fe7bd23a85 Clean up logic and add comments 2019-03-04 15:08:15 +00:00
Andrew Morgan 9f7cdf3da1 Clearer branching, fix missing list clear 2019-03-04 14:36:52 +00:00
Andrew Morgan 5f0c449dd5 Prevent replication wedging 2019-03-04 14:03:18 +00:00
Richard van der Hoff ed12338f35
Remove #4733 debug (#4767)
We don't need any of this stuff now; this brings protocol.py back into line
with develop for the hotfixes branch.
2019-03-04 14:00:03 +00:00
Erik Johnston 1e315017d3 When presence is enabled don't send over replication 2019-02-27 13:53:46 +00:00
Erik Johnston 7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff
Fix tightloop over connecting to replication server
2019-02-27 11:00:59 +00:00
Richard van der Hoff 67acd1aa1b Merge branch 'develop' into matrix-org-hotfixes 2019-02-27 10:29:24 +00:00
Erik Johnston 6bb1c028f1 Limit cache invalidation replication line length (#4748) 2019-02-27 10:28:37 +00:00
Erik Johnston 6870fc496f Move connecting logic into ClientReplicationStreamProtocol 2019-02-27 10:23:51 +00:00
Erik Johnston 25814921f1 Increase the max delay between retry attempts
Otherwise if you have many workers they can easily take out master with
their connection attempts
2019-02-26 15:12:33 +00:00
Erik Johnston 313987187e Fix tightloop over connecting to replication server
If the client failed to process incoming commands during the initial set
up of the replication connection it would immediately disconnect and
reconnect, resulting in a tightloop.

This can happen, for example, when subscribing to a stream that has a
row that is too long in the backlog.

The fix here is to not consider the connection successfully set up until
the client has succesfully subscribed and caught up with the streams.
This ensures that the retry logic timers aren't reset until then,
meaning that if an error does happen during start up the client will
continue backing off before retrying again.
2019-02-26 15:05:41 +00:00
Erik Johnston b50fe65a22 Add logging when sending error 2019-02-25 15:55:21 +00:00
Erik Johnston 17009e689b
Merge pull request #4734 from matrix-org/rav/repl_debug
Add some debug to help with #4733
2019-02-25 15:52:45 +00:00
Richard van der Hoff 5d2f755d3f Add some debug to help with #4733 2019-02-25 14:37:23 +00:00
Erik Johnston 0ea52872ab Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-22 15:29:41 +00:00
Erik Johnston 80467bbac3 Fix state cache invalidation on workers 2019-02-22 14:38:14 +00:00
Erik Johnston 776fe6c184 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-20 09:52:24 +00:00
Erik Johnston dbdc565dfd Fix registration on workers (#4682)
* Move RegistrationHandler init to HomeServer

* Move post registration actions to RegistrationHandler

* Add post regisration replication endpoint

* Newsfile
2019-02-20 18:47:31 +11:00
Erik Johnston 0e07d2c7d5 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-19 13:24:37 +00:00
Erik Johnston 90ec885805 Revert "Merge pull request #4654 from matrix-org/hawkowl/registration-worker"
This reverts commit 5bd2e2c31d, reversing
changes made to d97c3a6ce6.
2019-02-19 13:23:17 +00:00
Erik Johnston 5a28154c4d Revert "Merge pull request #4655 from matrix-org/hawkowl/registration-worker"
This reverts commit 93555af5c9, reversing
changes made to 5bd2e2c31d.
2019-02-19 13:23:14 +00:00
Erik Johnston a9b5ea6fc1 Batch cache invalidation over replication
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.

Hopefully this will reduce load on the master process by substantially
reducing traffic.
2019-02-18 17:53:31 +00:00
Erik Johnston af691e415c Move register_device into handler 2019-02-18 16:49:38 +00:00
Erik Johnston eb2b8523ae Split out registration to worker
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.

Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
Erik Johnston c10c71e70d Emergency changes 2019-02-15 18:15:21 +00:00
Amber Brown 06622e4110 fix 2019-02-16 05:11:09 +11:00
Amber Brown d95252c01f use a device replication thingy 2019-02-16 05:08:58 +11:00
Amber Brown e4381ed514 pep8 2019-02-16 04:42:04 +11:00
Amber Brown ce5f3b1ba5 add all the files 2019-02-16 04:35:58 +11:00
Amber Brown 7b5c04312e isort 2019-02-16 04:35:27 +11:00
Amber Brown f5bafd70f4 add cache remover endpoint and wire it up 2019-02-16 04:34:23 +11:00
Erik Johnston 05413d4e20 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-01-30 14:27:19 +00:00
Erik Johnston a4f52a33fe Fix replication for room v3 (#4523)
* Fix replication for room v3

We were not correctly quoting the path fragments over http replication,
which meant that it exploded when the event IDs had a slash in them

* Newsfile
2019-01-30 14:19:52 +00:00
Erik Johnston d351be1567 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-01-30 11:48:29 +00:00
Erik Johnston b6b73a0bcf Fix receiving events from federation via a worker
This bug was introduced in PR #4470, commit 678a92cb56
2019-01-29 10:30:26 +00:00
Andrew Morgan c7f2eaf4f4 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-01-29 10:07:13 +00:00
Erik Johnston 678a92cb56 Replace missed usages of FrozenEvent 2019-01-25 10:32:30 +00:00