e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
ef344b10e5
Don't log errors propogated from send_event
2018-02-15 11:03:49 +00:00
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
5c3c32f16f
Metrics for number of RDATA commands received
...
I found myself wishing we had this.
2018-01-15 17:45:55 +00:00
0edf085b68
Fix some logcontext leaks in replication resource
...
The @measure_func annotations rely on the wrapped function respecting the
logcontext rules. Add the necessary yields to make this work.
2017-11-23 23:19:43 +00:00
35a4b63240
Pull out bits of StateStore to a mixin
...
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.
fixes https://github.com/matrix-org/synapse/issues/2655 .
2017-11-14 11:43:58 +00:00
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
f807f7f804
log when we get an exception handling replication updates
2017-10-12 11:51:24 +01:00
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
925b3638ff
Reduce log levels in tcp replication
2017-07-11 10:04:21 +01:00
27f26e48b7
Serialize user ip command as json
2017-06-27 16:25:38 +01:00
8c23221666
Fix up
2017-06-27 15:53:45 +01:00
78cefd78d6
Make workers report to master for user ip updates
2017-06-27 14:58:10 +01:00
dae9a00a28
Initialise exclusive_user_regex
2017-06-21 14:19:33 +01:00
8177563ebe
Fix for workers
2017-06-21 13:57:49 +01:00
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
a837765e8c
Merge pull request #2266 from matrix-org/erikj/host_in_room
...
Change is_host_joined to use current_state table
2017-06-12 09:49:51 +01:00
8060974344
Fix replication
2017-06-09 16:40:52 +01:00
2cac7623a5
Add missing notifier
2017-06-09 11:24:41 +01:00
298d83b340
Fix replication
2017-06-09 11:01:28 +01:00
dfbda5e025
Faster cache for get_joined_hosts
2017-05-25 17:24:44 +01:00
f85a415279
Add missing storage function to slave store
2017-05-22 16:31:24 +01:00
9ac263ed1b
Add new storage functions to slave store
2017-05-04 14:29:03 +01:00
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
9c712a366f
Move get_presence_list_* to SlaveStore
2017-04-11 16:07:33 +01:00
28a4649785
Remove HTTP replication APIs
2017-04-11 09:52:11 +01:00
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
2e6f5a4910
Typo
2017-04-10 16:17:40 +01:00
efcb6db688
Merge pull request #2109 from matrix-org/erikj/send_queue_fix
...
Fix up federation SendQueue and document types
2017-04-10 13:09:25 +01:00
0364d23210
Up replication ping timeout
2017-04-10 11:32:05 +01:00
ab904caf33
Comments
2017-04-10 10:02:17 +01:00
98ce212093
Merge pull request #2103 from matrix-org/erikj/no-double-encode
...
Don't double encode replication data
2017-04-07 09:39:52 +01:00
ad544c803a
Document types of the replication streams
2017-04-06 13:28:52 +01:00
69b3fd485d
Fix incorrect type when using InvalidateCacheCommand
2017-04-06 09:36:38 +01:00
fcc803b2bf
Add log lines
2017-04-05 17:13:44 +01:00
3f213d908d
Rearrange metrics
2017-04-05 14:15:09 +01:00
1ca0e78ca1
Fix typo
2017-04-05 13:43:39 +01:00
b43d3267e2
Fixup some metrics for tcp repl
2017-04-05 13:34:54 +01:00
a5c401bd12
Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
...
Move to using TCP replication
2017-04-05 09:36:21 +01:00
a76886726b
Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix
...
Advance replication streams even if nothing is listening
2017-04-04 15:40:51 +01:00
4264ceb31c
Fiddle tcp replication logging
2017-04-04 14:14:03 +01:00
023ee197be
Advance replication streams even if nothing is listening
...
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
2017-04-04 13:19:26 +01:00
3a1f3f8388
Change slave storage to use new replication interface
...
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
52bfa604e1
Add basic replication client handler and factory
2017-04-03 15:34:13 +01:00
0a6a966e2b
Always advance stream tokens
2017-04-03 15:22:56 +01:00
1df7c28661
Use callbacks to notify tcp replication rather than deferreds
2017-03-31 15:42:51 +01:00
36d2b66f90
Add a timestamp to USER_SYNC command
...
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
4d7fc7f977
Add server side resource for tcp replication
2017-03-30 13:24:45 +01:00
7450693435
Initial TCP protocol implementation
...
This defines the low level TCP replication protocol
2017-03-30 12:54:46 +01:00
8da6f0be48
Define the various streams we will replicate
2017-03-30 12:54:46 +01:00
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
09f79aaad0
Use presence replication stream to invalidate cache
...
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
d58b1ffe94
Replace some calls to cursor_to_dict
...
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
aac6d1fc9b
PEP8
2017-03-20 13:47:56 +00:00
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
45c7f12d2a
Add new storage function to slave store
2017-03-13 16:26:44 +00:00
8f267fa8a8
Fix it for the workers
2017-03-10 11:22:25 +00:00
e933a2712d
Don't log unknown cache warnings in workers
2017-02-28 16:22:41 +00:00
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
9e617cd4c2
Cache get_presence storage
2017-02-13 13:50:03 +00:00
6bba80241c
Merge pull request #1912 from matrix-org/markjh/roominitialsync
...
Add db functions needed for room initial sync to slave
2017-02-13 12:20:21 +01:00
3a46280ca3
Add db functions needed for room initial sync to slave
2017-02-13 11:16:53 +00:00
0f3e296cb7
Fix replication
2017-02-02 15:02:03 +00:00
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
3670025e64
Rename func
2017-01-30 14:11:31 +00:00
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
09cbcb78d3
Add cache to get_public_room_ids_at_stream_id
2016-12-12 14:41:51 +00:00
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
26072df6af
Ensure only main or federation_sender process can send federation traffic
2016-11-23 14:09:47 +00:00
4c79a63fd7
Explicit federation ack
2016-11-23 10:40:44 +00:00
90565d015e
Invalidate retry cache in both directions
2016-11-22 17:45:44 +00:00
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
668f91d707
Fix check of wrong variable
2016-10-11 13:57:22 +01:00
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
995f2f032f
Fix public room pagination for client_reader app
2016-09-16 14:48:21 +01:00
418bcd4309
Add new storage function to slave store
2016-09-16 08:37:39 +01:00
cb3edec6af
Use stream_change cache to make get_forward_extremeties_for_room cache more effective
2016-09-15 14:28:13 +01:00
55e6fc917c
Add cache to get_forward_extremeties_for_room
2016-09-15 14:04:28 +01:00
211786ecd6
Stream public room changes down replication
2016-09-15 11:47:23 +01:00
a4339de9de
Correctly handle typing stream id resetting
2016-09-09 16:44:26 +01:00
ab80d5e0a9
Drop replication log levels
2016-09-09 14:56:50 +01:00
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
fa9d36e050
Merge branch 'develop' into markjh/direct_to_device_federation
2016-09-08 13:43:43 +01:00
2a0159b8ae
Fix the stream change cache to work over replication
2016-09-07 15:58:00 +01:00
a99e933550
Add upgrade script that will slowly prune state_groups_state entries
2016-09-05 10:05:36 +01:00
44982606ee
Merge pull request #1060 from matrix-org/erikj/state_ids
...
Assign state groups in state handler.
2016-09-01 14:20:42 +01:00
826ca61745
Add storage function to SlaveStore
2016-08-31 14:45:04 +01:00
1aa3e1d287
Add a replication stream for direct to device messages
2016-08-31 10:38:58 +01:00
3e784eff74
Remove state replication stream
2016-08-30 16:51:36 +01:00
55fc17cf4b
Merge pull request #1049 from matrix-org/erikj/presence_users_in_room
...
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-30 10:50:37 +01:00
41788bba50
Add to slave store
2016-08-30 09:55:17 +01:00
873f870e5a
Add new direct message storage functions to slave store
2016-08-30 09:40:32 +01:00
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
4daa397a00
Add is_host_joined to slave storage
2016-08-26 13:10:56 +01:00
50943ab942
Add new state storage funcs to replication
2016-08-26 09:57:32 +01:00
07229bbdae
Add appservice worker
2016-08-18 14:59:55 +01:00
a2427981b7
Use cached get_user_by_access_token in slaves
2016-08-16 11:24:32 +01:00
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
0be963472b
Use cached version of get_aliases_for_room
2016-08-15 11:24:12 +01:00
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
5aa024e501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader
2016-07-29 11:24:56 +01:00
0a7d3cd00f
Create separate methods for getting messages to push
...
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.
The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.
The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
ec8b217722
Add destination retry to slave store
2016-07-28 17:35:53 +01:00
76b89d0edb
Add slace storage functions for public room list
2016-07-28 17:03:40 +01:00
0fcbca531f
Add get_auth_chain to slave store
2016-07-28 16:36:28 +01:00
6ede23ff1b
Add more key storage funcs into slave store
2016-07-28 15:41:26 +01:00
aede7248ab
Split out a FederationReader process
2016-07-21 17:37:44 +01:00
46b7362304
pep8
2016-06-24 11:44:57 +01:00
870c45913e
Use similar naming we use in email notifs for push
...
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
05e01f21d7
Remove event fetching from DB threads
2016-06-03 17:22:13 +01:00
20eccd84d4
Merge pull request #827 from matrix-org/markjh/more_slaved_methods
...
Add methods to events, account data and receipt slaves
2016-06-03 11:46:21 +01:00
b09f348530
Merge pull request #824 from matrix-org/markjh/slaved_presence_store
...
Add a slaved store for presence
2016-06-03 11:26:33 +01:00
81cf449daa
Add methods to events, account data and receipt slaves
...
Adds the methods needed by /sync to the slaved events,
account data and receipt stores.
2016-06-03 11:19:27 +01:00
3ae915b27e
Add a slaved store for presence
2016-06-03 11:05:53 +01:00
f88d747f79
Add a comment explaining why the filter cache doesn't need exipiring
2016-06-03 11:03:10 +01:00
0eae075723
Add slaved stores for filters, tokens, and push rules
2016-06-03 10:58:03 +01:00
70599ce925
Allow external processes to mark a user as syncing. ( #812 )
...
* Add infrastructure to the presence handler to track sync requests in external processes
* Expire stale entries for dead external processes
* Add an http endpoint for making users as syncing
Add some docstrings and comments.
* Fixes
2016-06-02 15:20:15 +01:00
0cb441fedd
Move typing handler out of the Handlers object
2016-05-17 15:58:46 +01:00
3b86ecfa79
Move the presence handler out of the Handlers object
2016-05-16 18:56:37 +01:00
0466454b00
Assert that stream replicated stream positions are ints
2016-05-13 17:33:44 +01:00
1f71f386f6
Merge branch 'develop' into dbkr/email_notifs_on_pusher
2016-05-13 16:59:56 +01:00
206eb9fd94
Shift some of the state_group methods into the SlavedEventStore
2016-05-13 16:58:14 +01:00
3abab26458
Add a slaved datastore for account data
2016-05-13 15:34:06 +01:00
0e792e7903
Log the stream IDs in an order that makes sense
2016-05-13 11:54:44 +01:00
3547e66bc6
Make sure we advance our stream position
2016-05-13 11:53:00 +01:00
5f46be19a7
Pass through get_events to pusher too
2016-05-10 18:43:40 +02:00
d46b18a00f
Pass through _get_event_txn
2016-05-10 18:27:06 +02:00
8940281d1b
Don't warn
2016-05-05 15:10:03 +01:00
5d8a93a10e
Add some log information at returned replication streams
2016-05-05 10:29:21 +01:00
8d7ad44331
Report per request metrics for all of the things using request_handler
2016-04-28 10:57:49 +01:00
8a65666454
Fix backfill replication to advance the stream correctly
2016-04-27 15:38:43 +01:00
78741cf025
Merge pull request #743 from matrix-org/markjh/slave_pushers
...
Replicate the pushers
2016-04-21 17:21:29 +01:00
31719ad124
Merge pull request #744 from matrix-org/markjh/replication_remove_pusher
...
Add a replication endpoint for deleting pushers
2016-04-21 17:10:49 +01:00
cfe1ff4bdb
Add a replication endpoint for deleting pushers
2016-04-21 16:33:05 +01:00
d4823efad9
Replicate the pushers
2016-04-21 16:18:00 +01:00
c0d8e0eb63
Replicate push actions
2016-04-21 15:25:58 +01:00
f52dd35ac3
Merge pull request #738 from matrix-org/markjh/slaved_receipts
...
Add a slaved receipts store
2016-04-19 17:31:59 +01:00
5bbd424ee0
Add a slaved receipts store
2016-04-19 17:14:08 +01:00
e99365f601
Replicate get_invited_rooms_for_user
2016-04-19 15:22:14 +01:00
ceb599e789
Add tests for redactions
2016-04-07 16:52:07 +01:00
60ec9793fb
Add tests for get_latest_event_ids_in_room and get_current_state
2016-04-07 13:17:56 +01:00
6bfec56796
Test that room membership is replicated
2016-04-06 16:20:13 +01:00
75fb9ac1be
Add a slaved events store class
...
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
2016-04-06 14:18:35 +01:00
1d4deff25a
Separate generating the replication response...
...
from doing the http request parsing to make it easier
to write unit tests for replication.
2016-04-05 11:23:57 +01:00
62e395f0e3
Merge pull request #676 from matrix-org/markjh/replicate_stateIII
...
Add replication streams for ex outliers and current state resets
2016-03-31 11:20:57 +01:00
2ec5426035
Use a namedtuple rather than tuple unpacking
2016-03-31 10:33:02 +01:00
9113316b0e
typo
2016-03-30 17:29:42 +01:00
1fbb094c6f
Add replication streams for ex outliers and current state resets
2016-03-30 17:19:56 +01:00
31a9eceda5
Add a replication stream for state groups
2016-03-30 16:01:58 +01:00
b6e8420aee
Add replication stream for pushers
2016-03-15 17:33:10 +00:00
ebcbb23226
s/stream_ordering/event_stream_ordering/ in push
2016-03-04 16:15:23 +00:00
2223204eba
Hook push rules up to the replication API
2016-03-02 17:26:20 +00:00
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Erik Johnston