7b8a157b79
Merge pull request #4792 from matrix-org/anoa/replication_tokens
...
Support batch updates in the worker sender
2019-03-06 15:48:29 +00:00
e9aa401994
Remove redundant changes from synapse/replication/tcp/streams.py ( #4813 )
...
This was some hacky code (introduced in c10c71e70d
2019-03-06 13:21:32 +00:00
dc510e0e43
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-03-05 14:41:13 +00:00
a4c3a361b7
Add rate-limiting on registration ( #4735 )
...
* Rate-limiting for registration
* Add unit test for registration rate limiting
* Add config parameters for rate limiting on auth endpoints
* Doc
* Fix doc of rate limiting function
Co-Authored-By: babolivier <contact@brendanabolivier.com>
* Incorporate review
* Fix config parsing
* Fix linting errors
* Set default config for auth rate limiting
* Fix tests
* Add changelog
* Advance reactor instead of mocked clock
* Move parameters to registration specific config and give them more sensible default values
* Remove unused config options
* Don't mock the rate limiter un MAU tests
* Rename _register_with_store into register_with_store
* Make CI happy
* Remove unused import
* Update sample config
* Fix ratelimiting test for py2
* Add non-guest test
2019-03-05 14:25:33 +00:00
b9f6163092
Simplify token replication logic
2019-03-05 13:58:30 +00:00
a84b8d56c2
Fixup slave stores
2019-03-04 18:04:57 +00:00
fe7bd23a85
Clean up logic and add comments
2019-03-04 15:08:15 +00:00
9f7cdf3da1
Clearer branching, fix missing list clear
2019-03-04 14:36:52 +00:00
5f0c449dd5
Prevent replication wedging
2019-03-04 14:03:18 +00:00
ed12338f35
Remove #4733 debug ( #4767 )
...
We don't need any of this stuff now; this brings protocol.py back into line
with develop for the hotfixes branch.
2019-03-04 14:00:03 +00:00
1e315017d3
When presence is enabled don't send over replication
2019-02-27 13:53:46 +00:00
7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff
...
Fix tightloop over connecting to replication server
2019-02-27 11:00:59 +00:00
67acd1aa1b
Merge branch 'develop' into matrix-org-hotfixes
2019-02-27 10:29:24 +00:00
6bb1c028f1
Limit cache invalidation replication line length ( #4748 )
2019-02-27 10:28:37 +00:00
6870fc496f
Move connecting logic into ClientReplicationStreamProtocol
2019-02-27 10:23:51 +00:00
25814921f1
Increase the max delay between retry attempts
...
Otherwise if you have many workers they can easily take out master with
their connection attempts
2019-02-26 15:12:33 +00:00
313987187e
Fix tightloop over connecting to replication server
...
If the client failed to process incoming commands during the initial set
up of the replication connection it would immediately disconnect and
reconnect, resulting in a tightloop.
This can happen, for example, when subscribing to a stream that has a
row that is too long in the backlog.
The fix here is to not consider the connection successfully set up until
the client has succesfully subscribed and caught up with the streams.
This ensures that the retry logic timers aren't reset until then,
meaning that if an error does happen during start up the client will
continue backing off before retrying again.
2019-02-26 15:05:41 +00:00
b50fe65a22
Add logging when sending error
2019-02-25 15:55:21 +00:00
17009e689b
Merge pull request #4734 from matrix-org/rav/repl_debug
...
Add some debug to help with #4733
2019-02-25 15:52:45 +00:00
5d2f755d3f
Add some debug to help with #4733
2019-02-25 14:37:23 +00:00
0ea52872ab
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-02-22 15:29:41 +00:00
80467bbac3
Fix state cache invalidation on workers
2019-02-22 14:38:14 +00:00
776fe6c184
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-02-20 09:52:24 +00:00
dbdc565dfd
Fix registration on workers ( #4682 )
...
* Move RegistrationHandler init to HomeServer
* Move post registration actions to RegistrationHandler
* Add post regisration replication endpoint
* Newsfile
2019-02-20 18:47:31 +11:00
0e07d2c7d5
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-02-19 13:24:37 +00:00
90ec885805
Revert "Merge pull request #4654 from matrix-org/hawkowl/registration-worker"
...
This reverts commit 5bd2e2c31dd97c3a6ce6
2019-02-19 13:23:17 +00:00
5a28154c4d
Revert "Merge pull request #4655 from matrix-org/hawkowl/registration-worker"
...
This reverts commit 93555af5c95bd2e2c31d
2019-02-19 13:23:14 +00:00
a9b5ea6fc1
Batch cache invalidation over replication
...
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.
Hopefully this will reduce load on the master process by substantially
reducing traffic.
2019-02-18 17:53:31 +00:00
af691e415c
Move register_device into handler
2019-02-18 16:49:38 +00:00
eb2b8523ae
Split out registration to worker
...
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.
Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
c10c71e70d
Emergency changes
2019-02-15 18:15:21 +00:00
06622e4110
fix
2019-02-16 05:11:09 +11:00
d95252c01f
use a device replication thingy
2019-02-16 05:08:58 +11:00
e4381ed514
pep8
2019-02-16 04:42:04 +11:00
ce5f3b1ba5
add all the files
2019-02-16 04:35:58 +11:00
7b5c04312e
isort
2019-02-16 04:35:27 +11:00
f5bafd70f4
add cache remover endpoint and wire it up
2019-02-16 04:34:23 +11:00
05413d4e20
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-01-30 14:27:19 +00:00
a4f52a33fe
Fix replication for room v3 ( #4523 )
...
* Fix replication for room v3
We were not correctly quoting the path fragments over http replication,
which meant that it exploded when the event IDs had a slash in them
* Newsfile
2019-01-30 14:19:52 +00:00
d351be1567
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-01-30 11:48:29 +00:00
b6b73a0bcf
Fix receiving events from federation via a worker
...
This bug was introduced in PR #4470 , commit 678a92cb56
2019-01-29 10:30:26 +00:00
c7f2eaf4f4
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2019-01-29 10:07:13 +00:00
678a92cb56
Replace missed usages of FrozenEvent
2019-01-25 10:32:30 +00:00
be6a7e47fa
Revert "Require event format version to parse or create events"
2019-01-25 10:23:51 +00:00
e8c9f15397
Replace missed usages of FrozenEvent
2019-01-24 11:14:07 +00:00
f8281f42c8
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-10-29 18:16:58 +00:00
a163b748a5
Don't truncate command name in metrics
2018-10-29 17:34:21 +00:00
7171bdf279
Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes
2018-10-29 23:14:47 +11:00
c4b3698a80
Make the replication logger quieter ( #4108 )
2018-10-29 22:59:44 +11:00
33551be61b
Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes
2018-10-15 20:15:27 +11:00
381d2cfdf0
Make workers work on Py3 ( #4027 )
2018-10-13 00:14:08 +11:00
b5eef203f4
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-09-14 18:25:55 +01:00
f1a7264663
Fix minor typo in exception
2018-09-13 11:51:12 -06:00
7c27c4d51c
merge ( #3576 )
2018-09-14 03:11:11 +10:00
17201abd53
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-09-11 14:17:33 +01:00
3e242dc149
Remove conn_id
2018-09-04 11:45:52 +01:00
b13836da7f
Remove conn_id from repl prometheus metrics
...
`conn_id` gets set to a random string, and so we end up filling up
prometheus with tonnes of data series, which is bad.
2018-09-03 17:22:49 +01:00
f5abc10724
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-08-20 11:12:18 +01:00
2aa7cc6a46
Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging
...
Fix logging bug in EDU handling over replication
2018-08-20 10:51:45 +01:00
3b2dcfff78
Fix logging bug in EDU handling over replication
2018-08-17 11:11:06 +01:00
0e8d78f6aa
Logcontexts for replication command handlers
...
Run the handlers for replication commands as background processes. This should
improve the visibility in our metrics, and reduce the number of "running db
transaction from sentinel context" warnings.
Ideally it means converting the things that fire off deferreds into the night
into things that actually return a Deferred when they are done. I've made a bit
of a stab at this, but it will probably be leaky.
2018-08-17 00:43:43 +01:00
4dd0604f61
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-08-15 15:37:05 +01:00
488ffe6fdb
Use federation handler function rather than duplicate
...
This involves renaming _persist_events to be a public function.
2018-08-15 14:17:18 +01:00
773db62a22
Rename slave TransactionStore to SlaveTransactionStore
2018-08-15 14:17:06 +01:00
b179537f2a
Move clean_room_for_join to master
2018-08-09 10:37:38 +01:00
72d1902bbe
Fixup doc comments
2018-08-09 10:23:49 +01:00
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
2bdafaf3c1
Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet
...
Add helper base class for generating new replication endpoints
2018-08-09 10:06:23 +01:00
62564797f5
Fixup wording and remove dead code
2018-08-09 09:56:10 +01:00
bebe325e6c
Rename POST param to METHOD
2018-08-08 10:36:18 +01:00
5011417632
Fixup logging and docstrings
2018-08-08 10:29:58 +01:00
1e2bed9656
Import all functions from TransactionStore
2018-08-06 15:23:38 +01:00
a3f5bf79a0
Add EDU/query handling over replication
2018-08-06 15:23:31 +01:00
e26dbd82ef
Add replication APIs for persisting federation events
2018-08-06 15:02:28 +01:00
051a99c400
Fix isort
2018-08-06 14:29:31 +01:00
49a3163958
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-08-06 13:33:54 +01:00
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
cb298ff623
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet
2018-08-03 09:25:15 +01:00
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
443da003bc
Use new helper base class for membership requests
2018-07-31 14:32:23 +01:00
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
d81602b75a
Add helper base class for generating new replication endpoints
...
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
f59be4eb0e
Fix unit tests
...
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
e1eb147f2a
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-07-23 16:45:22 +01:00
0faa3223cd
Fix missing attributes on workers.
...
This was missed during the transition from attribute to getter for
getting state from context.
2018-07-23 16:28:00 +01:00
7c79f2cb72
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-07-12 09:59:58 +01:00
f04e35c170
Merge branch 'develop' into matrix-org-hotfixes
2018-07-10 18:04:03 +01:00
05f5dabc10
Use stream cache in get_linearized_receipts_for_room
...
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
82781f5838
Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes
2018-06-28 21:09:28 +00:00
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
4c22c9b0b6
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2018-06-25 14:37:13 +01:00
33fdcfa957
Merge pull request #3441 from matrix-org/erikj/redo_erasure
...
Fix user erasure and re-enable
2018-06-25 14:37:01 +01:00
eb50c44eaf
Add UserErasureWorkerStore to workers
2018-06-25 14:22:24 +01:00
07cad26d65
Remove all global reactor imports & pass it around explicitly ( #3424 )
2018-06-25 14:08:28 +01:00
7a1406d144
Prefill client_ip_last_seen in replication
2018-06-22 15:58:15 +01:00
1766a5fdc0
Increase MAX_EVENTS_BEHIND for replication clients
2018-06-22 15:58:14 +01:00
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
99b77aa829
Fix tcp protocol metrics naming ( #3410 )
2018-06-21 09:39:27 +01:00
Andrew Morgan