Commit Graph

8494 Commits (3a1f3f838862cfd2486773079d01e30c2237e8c2)

Author SHA1 Message Date
Erik Johnston 3a1f3f8388 Change slave storage to use new replication interface
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
Erik Johnston 52bfa604e1 Add basic replication client handler and factory 2017-04-03 15:34:13 +01:00
Erik Johnston 0a6a966e2b Always advance stream tokens 2017-04-03 15:22:56 +01:00
Erik Johnston 1df7c28661 Use callbacks to notify tcp replication rather than deferreds 2017-03-31 15:42:51 +01:00
Erik Johnston 36d2b66f90 Add a timestamp to USER_SYNC command
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
Erik Johnston 9d0170ac6c Fix up presence 2017-03-31 11:36:32 +01:00
Erik Johnston b4276a3896 Add a brief list of commands to docs 2017-03-31 11:34:45 +01:00
Erik Johnston bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston 63fcc42990 Remove user from process_presence when stops syncing 2017-03-30 14:26:08 +01:00
Erik Johnston 31e0fe9031 Fix indentation in docs/ 2017-03-30 13:54:15 +01:00
Erik Johnston 3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Erik Johnston e9dd8370b0 Add functions to presence to support remote syncs
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
2017-03-30 13:25:14 +01:00
Erik Johnston 4d7fc7f977 Add server side resource for tcp replication 2017-03-30 13:24:45 +01:00
Erik Johnston 7450693435 Initial TCP protocol implementation
This defines the low level TCP replication protocol
2017-03-30 12:54:46 +01:00
Erik Johnston 8da6f0be48 Define the various streams we will replicate 2017-03-30 12:54:46 +01:00
Erik Johnston 11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston 7984708a55 Add a simple hook to wait for replication traffic 2017-03-30 11:57:52 +01:00
Erik Johnston 24d35ab47b Add new storage functions for new replication
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston 30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Anant Prakash 6cdca71079 synctl.py: wait for synapse to stop before restarting (#2020) 2017-03-29 19:20:13 +01:00
Erik Johnston 3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston b9c2ae6788 Merge pull request #1849 from matrix-org/erikj/state_typo
Fix bug where current_state_events renamed to current_state_ids
2017-03-29 11:53:42 +01:00
Erik Johnston 85be3dde81 Bail early if remote wouldn't be retried (#2064)
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
2017-03-29 11:48:27 +01:00
Erik Johnston 2f8b580b64 Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert
Don't user upsert to persist new one time keys
2017-03-29 11:44:23 +01:00
Erik Johnston c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
Erik Johnston e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston 4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Richard van der Hoff 276af7b59b Merge pull request #2037 from ricco386/fix_readme_centos_issues
Fix installation issues
2017-03-29 08:52:39 +02:00
Erik Johnston d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
Erik Johnston 58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston dc56a6b8c8 Merge pull request #2070 from matrix-org/erikj/perf_send
Short circuit if all new events have same state group
2017-03-27 18:33:54 +01:00
Erik Johnston bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston 35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston 2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston 281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
Erik Johnston 987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston 31d56c3fb5 Merge pull request #2061 from matrix-org/erikj/add_transaction_store
Add slave transaction store to workers who send federation requests
2017-03-24 13:24:27 +00:00
Erik Johnston 09f79aaad0 Use presence replication stream to invalidate cache
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
Erik Johnston 23e0ff840a Merge pull request #2060 from matrix-org/erikj/cache_hosts_in_room
Cache hosts in room
2017-03-24 13:07:22 +00:00
Erik Johnston 48e7697911 Add slave transaction store 2017-03-24 13:05:30 +00:00
Richard van der Hoff f136c89d5e Merge pull request #2058 from matrix-org/rav/logcontext_leaks_2
try not to drop context after federation requests
2017-03-24 12:47:26 +00:00
Richard van der Hoff 01fc847f7f Merge pull request #2057 from matrix-org/rav/missing_yield_2
Add another missing yield on check_device_registered
2017-03-24 12:46:43 +00:00
Erik Johnston 7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston 57cfa513f5 Merge pull request #2054 from matrix-org/erikj/user_iter_cursor
Reduce some CPU work on DB threads
2017-03-24 11:40:08 +00:00
Erik Johnston d58b1ffe94 Replace some calls to cursor_to_dict
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
Erik Johnston e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
David Baker e36950dec5 Merge pull request #2055 from matrix-org/dbkr/fix_add_msisdn_requestToken
Fix token request for addition of phone numbers
2017-03-24 10:32:57 +00:00
David Baker f902e89d4b Merge pull request #2056 from matrix-org/dbkr/fix_invite_reject
Fix rejection of invites not reaching sync
2017-03-24 10:32:51 +00:00