a828a64b75
Comment
2017-04-07 11:54:03 +01:00
d4d176e5d0
Add logging
2017-04-07 11:51:28 +01:00
449d1297ca
Fix up federation SendQueue and document types
2017-04-07 11:48:33 +01:00
d72667fcce
Speed up get_current_state_ids
...
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).
get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
2017-04-07 10:10:49 +01:00
a41fe500d6
Bump version and changelog
2017-04-07 10:03:48 +01:00
54f59bd7d4
Merge pull request #2107 from HarHarLinks/patch-1
...
fix typo in synctl help
2017-04-07 09:54:37 +01:00
98ce212093
Merge pull request #2103 from matrix-org/erikj/no-double-encode
...
Don't double encode replication data
2017-04-07 09:39:52 +01:00
8a1137ceab
fix typo in synctl help
2017-04-06 17:10:20 +02:00
877c029c16
Use iteritems
2017-04-06 15:51:22 +01:00
944692ef69
Merge pull request #2106 from matrix-org/erikj/reduce_user_sync
...
Reduce rate of USER_SYNC repl commands
2017-04-06 13:35:31 +01:00
391712a4f9
Comment
2017-04-06 13:35:00 +01:00
ad544c803a
Document types of the replication streams
2017-04-06 13:28:52 +01:00
dbf87282d3
Docs
2017-04-06 13:11:21 +01:00
69b3fd485d
Fix incorrect type when using InvalidateCacheCommand
2017-04-06 09:36:38 +01:00
5058292537
Support authenticated SMTP
...
Closes (SYN-714) #1385
Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
2017-04-05 21:01:08 -07:00
fcc803b2bf
Add log lines
2017-04-05 17:13:44 +01:00
3f213d908d
Rearrange metrics
2017-04-05 14:15:09 +01:00
1ca0e78ca1
Fix typo
2017-04-05 13:43:39 +01:00
b43d3267e2
Fixup some metrics for tcp repl
2017-04-05 13:34:54 +01:00
b5cb6347a4
Don't immediately notify the master about users whose syncs have gone away
2017-04-05 13:25:40 +01:00
96b9b6c127
Don't double json encode typing replication data
2017-04-05 11:34:20 +01:00
f10ce8944b
Don't double json encode federation replication data
2017-04-05 11:10:28 +01:00
a5c401bd12
Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
...
Move to using TCP replication
2017-04-05 09:36:21 +01:00
d1d5362267
Add comment
2017-04-04 16:41:03 +01:00
9f26d3b75b
Deduplicate new deviceinbox rows for replication
2017-04-04 16:21:21 +01:00
a76886726b
Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix
...
Advance replication streams even if nothing is listening
2017-04-04 15:40:51 +01:00
ac66e11f2b
Add the appropriate amount of preserve_fn
2017-04-04 15:22:54 +01:00
4264ceb31c
Fiddle tcp replication logging
2017-04-04 14:14:03 +01:00
023ee197be
Advance replication streams even if nothing is listening
...
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
2017-04-04 13:19:26 +01:00
d1605794ad
Remove unused worker config option
2017-04-04 11:17:00 +01:00
3376f16012
Shuffle and comment synchrotron presence
2017-04-04 11:14:16 +01:00
6ce6bbedcb
Move where we ack federation
2017-04-04 11:02:44 +01:00
62b89daac6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server
2017-04-04 09:46:16 +01:00
773e64cc1a
Merge pull request #2095 from matrix-org/rav/cull_log_preserves
...
Cull spurious PreserveLoggingContexts
2017-04-03 17:02:25 +01:00
2d05eb3cf5
Merge remote-tracking branch 'origin/release-v0.20.0' into develop
2017-04-03 16:23:23 +01:00
30bcbf775a
Accept join events from all servers
...
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.
(Fixes #1893 ).
2017-04-03 15:58:07 +01:00
7eb9f34cc3
Remove spurious yield
...
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
2017-04-03 15:44:19 +01:00
0b08c48fc5
Remove more spurious `PreserveLoggingContext`s
...
Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`;
there is no problem if the logcontext is set when calling it.
2017-04-03 15:43:37 +01:00
65e1683680
Remove spurious PreserveLoggingContext
...
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
2017-04-03 15:42:38 +01:00
feb496056e
preserve_fn some deferred-returning things
...
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
2017-04-03 15:41:17 +01:00
e2eebf1696
Fix fixme in preserve_fn
...
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
2017-04-03 15:38:02 +01:00
36c28bc467
Update all the workers and master to use TCP replication
2017-04-03 15:35:52 +01:00
3a1f3f8388
Change slave storage to use new replication interface
...
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
52bfa604e1
Add basic replication client handler and factory
2017-04-03 15:34:13 +01:00
0a6a966e2b
Always advance stream tokens
2017-04-03 15:22:56 +01:00
773e1c6d68
Remove spurious @preserve_fn decorators
...
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
2017-04-03 15:14:11 +01:00
0d1c85e643
Merge branch 'release-v0.20.0' of github.com:matrix-org/synapse into develop
2017-04-03 14:58:14 +01:00
1df7c28661
Use callbacks to notify tcp replication rather than deferreds
2017-03-31 15:42:51 +01:00
36d2b66f90
Add a timestamp to USER_SYNC command
...
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
8a240e4f9c
Merge pull request #2078 from APwhitehat/assertuserfriendly
...
add user friendly report of assertion error in synctl.py
2017-03-31 14:41:49 +01:00
142b6b4abf
Merge pull request #2011 from matrix-org/matthew/turn_allow_guests
...
add setting (on by default) to support TURN for guests
2017-03-31 14:37:09 +01:00
2a06b44be2
Merge pull request #1986 from matrix-org/matthew/enable_guest_3p
...
enable guest access for the 3pl/3pid APIs
2017-03-31 14:36:03 +01:00
9ee397b440
switch to allow_guest=True for authing 3Ps as per PR feedback
2017-03-31 13:54:26 +01:00
9d0170ac6c
Fix up presence
2017-03-31 11:36:32 +01:00
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
9cee0ce7db
Merge pull request #2075 from matrix-org/erikj/cache_speed
...
Speed up cached function access
2017-03-31 10:10:56 +01:00
350333a09a
Merge pull request #2076 from matrix-org/erikj/as_perf
...
Make AS's faster
2017-03-31 09:43:10 +01:00
4d17add8de
Remove unused instance variable
2017-03-31 09:38:27 +01:00
27b1b4a2c9
Speed up copy_and_replace
2017-03-30 17:50:31 +01:00
5b5b171f3e
Docs
2017-03-30 17:05:53 +01:00
b282fe7170
Revert log context change
2017-03-30 17:03:59 +01:00
9ff4e0e91b
Bump version and changelog
2017-03-30 16:37:40 +01:00
0834d1a70c
Merge pull request #2079 from matrix-org/erikj/push_regex_cache
...
Cache glob to regex at a higher level for push
2017-03-30 16:28:18 +01:00
63fcc42990
Remove user from process_presence when stops syncing
2017-03-30 14:26:08 +01:00
6194a64ae9
Doc new instance variables
2017-03-30 14:19:10 +01:00
014fee93b3
Manually calculate cache key as getcallargs is expensive
...
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
2017-03-30 14:14:46 +01:00
86780a8bc3
Don't convert to deferreds when not necessary
2017-03-30 14:14:36 +01:00
3ba2859e0c
Add tcp replication listener type and hook it up
2017-03-30 13:31:10 +01:00
e9dd8370b0
Add functions to presence to support remote syncs
...
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
2017-03-30 13:25:14 +01:00
4d7fc7f977
Add server side resource for tcp replication
2017-03-30 13:24:45 +01:00
f9b4bb05e0
Fix the logcontext handling in the cache wrappers ( #2077 )
...
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
2017-03-30 13:22:24 +01:00
7450693435
Initial TCP protocol implementation
...
This defines the low level TCP replication protocol
2017-03-30 12:54:46 +01:00
8da6f0be48
Define the various streams we will replicate
2017-03-30 12:54:46 +01:00
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
7984708a55
Add a simple hook to wait for replication traffic
2017-03-30 11:57:52 +01:00
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
30348c924c
Use txn.fetchall() so we can reuse txn
2017-03-30 10:30:05 +01:00
6cdca71079
synctl.py: wait for synapse to stop before restarting ( #2020 )
2017-03-29 19:20:13 +01:00
305d16d612
add user friendly report of assertion error in synctl.py
...
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
2017-03-29 20:41:39 +05:30
a3810136fe
Cache glob to regex at a higher level for push
2017-03-29 15:53:14 +01:00
3ce8d59176
Increase cache size for _get_state_group_for_event
2017-03-29 14:31:46 +01:00
b9c2ae6788
Merge pull request #1849 from matrix-org/erikj/state_typo
...
Fix bug where current_state_events renamed to current_state_ids
2017-03-29 11:53:42 +01:00
85be3dde81
Bail early if remote wouldn't be retried ( #2064 )
...
* Bail early if remote wouldn't be retried
* Don't always return true
* Just use get_retry_limiter
* Spelling
2017-03-29 11:48:27 +01:00
2f8b580b64
Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert
...
Don't user upsert to persist new one time keys
2017-03-29 11:44:23 +01:00
c5b0bdd542
Merge pull request #2067 from matrix-org/erikj/notify_on_fed
...
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
e4df0e189d
Decrank last commit
2017-03-29 11:02:35 +01:00
4ad613f6be
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert
2017-03-29 10:57:19 +01:00
ac6bc55512
Correctly look up key
2017-03-29 10:56:26 +01:00
69efd77749
Add comment
2017-03-29 09:50:05 +01:00
51b156d48a
Cache whether an AS is interested based on members
2017-03-28 13:27:21 +01:00
30f5ffdca2
Remove param and cast at call site
2017-03-28 13:27:21 +01:00
650f0e69f2
Compile the regex's used in ASes
2017-03-28 13:27:21 +01:00
d28db583da
Merge pull request #2063 from matrix-org/erikj/device_list_batch
...
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
58a35366be
The algorithm is part of the key id
2017-03-28 11:34:37 +01:00
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
35b4aa04be
Notify on new federation traffic
2017-03-27 14:07:47 +01:00
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
281553afe6
Merge pull request #2062 from matrix-org/erikj/presence_replication
...
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
987f4945b4
Actually call invalidate
2017-03-24 13:28:20 +00:00
Erik Johnston