82475a18d9
Merge pull request #2180 from matrix-org/rav/fix_timeout_on_timeout
...
Instantiate DeferredTimedOutError correctly
2017-05-02 13:32:58 +01:00
2e996271fe
Instantiate DeferredTimedOutError correctly
...
Call `super` correctly, so that we correctly initialise the `errcode` field.
Fixes https://github.com/matrix-org/synapse/issues/2179 .
2017-05-02 13:26:17 +01:00
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
3033261891
Merge pull request #2080 from matrix-org/erikj/filter_speed
...
Speed up filtering of a single event in push
2017-04-28 14:17:13 +01:00
2347efc065
Fixup
2017-04-28 12:46:53 +01:00
9b147cd730
Remove unncessary call in _get_missing_events_for_pdu
2017-04-28 11:55:25 +01:00
3a9f5bf6dd
Don't fetch state for missing events that we fetched
2017-04-28 11:26:46 +01:00
ab37bef83b
Remove unused import
2017-04-28 09:57:23 +01:00
ad8b316939
We don't care about forgotten rooms
2017-04-28 09:52:36 +01:00
421fdf7460
Speed up filtering of a single event in push
2017-04-28 09:52:36 +01:00
25a96e0c63
Merge pull request #2163 from matrix-org/erikj/fix_invite_state
...
Fix invite state to always include all events
2017-04-27 17:36:30 +01:00
46826bb078
Comment and remove spurious logging
2017-04-27 17:25:44 +01:00
f87b287291
Merge pull request #2127 from APwhitehat/alreadystarted
...
print something legible if synapse already running
2017-04-27 15:46:53 +01:00
bb9246e525
Merge pull request #2131 from matthewjwolff/develop
...
web_client_location documentation fix
2017-04-27 15:46:40 +01:00
c84770b877
Fix bgupdate error if index already exists ( #2167 )
...
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135 .
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
380fb87ecc
Merge pull request #2168 from matrix-org/erikj/federation_logging
...
Add some extra logging for edge cases of federation
2017-04-27 15:19:12 +01:00
87ae59f5e9
Typo
2017-04-27 15:16:21 +01:00
e42b4ebf0f
Add some extra logging for edge cases of federation
2017-04-27 14:38:21 +01:00
d3c150411c
Merge pull request #2130 from APwhitehat/roomexists
...
Check that requested room_id exists
2017-04-27 09:20:26 +01:00
1e166470ab
Fix tests
2017-04-26 16:23:30 +01:00
34e682d385
Fix invite state to always include all events
2017-04-26 16:18:08 +01:00
5fd12dce01
Remove debugging
2017-04-26 12:36:26 +01:00
82ae0238f9
Revert accidental commit
2017-04-26 11:43:16 +01:00
81804909d3
Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error
2017-04-26 11:31:55 +01:00
c366276056
Fix get_json
2017-04-26 10:07:01 +01:00
1a9255c12e
Use CodeMessageException subclass instead
...
Parse json errors from get_json client methods and throw special
errors.
2017-04-25 19:30:55 +01:00
f053a1409e
Make state caches cache in ascii
2017-04-25 17:22:55 +01:00
acb58bfb6a
fix up
2017-04-25 15:39:19 +01:00
f7181615f2
Don't specify default as dict
2017-04-25 15:22:59 +01:00
f144365281
Comment
2017-04-25 15:18:26 +01:00
d9aa645f86
Reduce size of joined_user cache
...
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.
We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
22f3d3ae76
Reduce _get_state_group_for_event cache size
2017-04-25 11:43:03 +01:00
b4da08cad8
Merge pull request #2158 from matrix-org/erikj/reduce_cache_size
...
Reduce cache size by not storing deferreds
2017-04-25 11:18:07 +01:00
efab1dadde
Remove DEBUG_CACHES
2017-04-25 10:54:09 +01:00
119cb9bbcf
Reduce cache size by not storing deferreds
...
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.
However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.
This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.
As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
e6e2627636
Fix code for reporting old verify keys in synapse
2017-04-24 18:51:25 +01:00
30f7bfa121
Merge pull request #2145 from matrix-org/rav/reject_invite_to_unreachable_server
...
Fix rejection of invites to unreachable servers
2017-04-24 15:20:52 +01:00
7af825bae4
Merge pull request #2155 from matrix-org/erikj/string_intern
...
Only intern ascii strings
2017-04-24 14:28:41 +01:00
d134d0935e
Only intern ascii strings
2017-04-24 14:07:48 +01:00
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
a46982cee9
Need the HTTP status code
2017-04-21 16:20:12 +01:00
70caf49914
Do the same for get_json
2017-04-21 16:09:03 +01:00
a1595cec78
Don't error for 3xx responses
2017-04-21 11:51:17 +01:00
2e165295b7
Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error
2017-04-21 11:35:52 +01:00
a90a0f5c8a
Propagate errors sensibly from proxied IS requests
...
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
2017-04-21 11:32:48 +01:00
91b3981800
Try harder when sending leave events
...
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
2017-04-21 01:50:36 +01:00
0cdb32fc43
Remove redundant try/except clauses
...
The `except SynapseError` clauses were pointless because the wrapped functions
would never throw a `SynapseError` (they either throw a `CodeMessageException`
or a `RuntimeError`).
The `except CodeMessageException` is now also pointless because the caller
treats all exceptions equally, so we may as well just throw the
`CodeMessageException`.
2017-04-21 01:32:01 +01:00
838810b76a
Broaden the conditions for locally_rejecting invites
...
The logic for marking invites as locally rejected was all well and good, but
didn't happen when the remote server returned a 500, or wasn't reachable, or
had no DNS, or whatever.
Just expand the except clause to catch everything.
Fixes https://github.com/matrix-org/synapse/issues/761 .
2017-04-21 01:31:37 +01:00
736b9a4784
Remove redundant function
...
inline `reject_remote_invite`, which only existed to make tracing the callflow
more difficult.
2017-04-21 01:31:09 +01:00
4903ccf159
Fix some lies, and other clarifications, in docstrings
...
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
d4040e9e28
Queried CONDITIONAL_REQUIREMENTS
2017-04-18 16:19:48 -05:00
3fb8784c92
m.read_marker -> m.fully_read ( #2128 )
...
Also:
- change the REST endpoint to have a "S" on the end (so it's now /read_markers)
- change the content of the m.read_up_to event to have the key "event_id" instead of "marker".
2017-04-18 17:46:15 +01:00
54f9a4cb59
Fixed travis build failure
...
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 01:38:27 -05:00
8e780b113d
web_server_root documentation fix
...
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 00:49:11 -05:00
574d573ac2
Check that requested room_id exists
2017-04-14 23:50:59 +05:30
78f0ddbfad
Merge pull request #2120 from matrix-org/luke/read-markers
...
Implement Read Marker API
2017-04-13 14:21:31 +01:00
6a70647d45
Correct logic in is_event_after
2017-04-13 13:46:17 +01:00
c1f52a321d
synctl.py: Check if synapse is already running
2017-04-13 18:00:02 +05:30
b9557064bf
Simplify is_event_after logic
2017-04-12 14:36:20 +01:00
cf6121e3da
More null-guard changes
2017-04-12 14:02:03 +01:00
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
8fbc0d29ee
Merge pull request #2121 from matrix-org/paul/sent-transactions-metric
...
Add a counter metric for successfully-sent transactions
2017-04-12 11:04:31 +01:00
c06c00190f
Merge pull request #2116 from matrix-org/erikj/dedupe_federation_repl2
...
Dedupe KeyedEdu and Devices federation repl traffic
2017-04-12 10:57:24 +01:00
c0aba0a23e
Remove Unused ref to hs
2017-04-12 10:52:11 +01:00
b9676a75f6
Move a space
2017-04-12 10:51:17 +01:00
69a18514e9
Only notify user, not entire room
2017-04-12 10:50:37 +01:00
122cd52ce4
Remove comment, simplify null-guard
2017-04-12 10:48:32 +01:00
26ae5178a4
Add some comments
2017-04-12 10:36:29 +01:00
bf9060156a
Merge pull request #2117 from matrix-org/erikj/remove_http_replication
...
Remove HTTP replication APIs
2017-04-12 10:21:42 +01:00
1745069543
Comment
2017-04-12 10:17:10 +01:00
c7ddb5ef7a
Reuse get_interested_parties
2017-04-12 10:16:26 +01:00
7b41013102
Merge pull request #2118 from matrix-org/erikj/no_devices
...
Fix getting latest device IP for user with no devices
2017-04-12 10:14:32 +01:00
77fb2b72ae
Handle no previous RM
2017-04-12 09:47:29 +01:00
7f94709066
travis flake8..
2017-04-11 18:35:45 +01:00
867822fa1e
flake8
2017-04-11 17:36:04 +01:00
73880268ef
Refactor event ordering check to events store
2017-04-11 17:34:09 +01:00
131485ef66
Copyright
2017-04-11 17:33:51 +01:00
11dbceb761
Add a counter metric for successfully-sent transactions
2017-04-11 17:16:12 +01:00
0127423027
flake8
2017-04-11 17:07:07 +01:00
85657eedf8
Bail on where clause instead
2017-04-11 16:24:31 +01:00
b48045a8f5
Don't bother with outer check for now
2017-04-11 16:23:24 +01:00
9c712a366f
Move get_presence_list_* to SlaveStore
2017-04-11 16:07:33 +01:00
a8c8e4efd4
Comment
2017-04-11 15:35:49 +01:00
414522aed5
Move get_interested_parties
2017-04-11 15:33:26 +01:00
2be8a281d2
Comments
2017-04-11 15:28:24 +01:00
6308ac45b0
Move get_interested_remotes back to presence handler
2017-04-11 15:19:26 +01:00
b9b72bc6e2
Comments
2017-04-11 15:15:34 +01:00
d892079844
Finish implementing RM endpoint
...
- This change causes a 405 to be sent if "m.read_marker" is set via /account_data
- This also fixes-up the RM endpoint so that it actually Works.
2017-04-11 15:01:39 +01:00
e263c26690
Initial commit of RM server-side impl
...
(See https://docs.google.com/document/d/1UWqdS-e1sdwkLDUY0wA4gZyIkRp-ekjsLZ8k6g_Zvso/edit#heading=h.lndohpg8at5u )
2017-04-11 11:55:30 +01:00
f3cf3ff8b6
Merge branch 'master' of github.com:matrix-org/synapse into develop
2017-04-11 11:13:32 +01:00
34840cdcef
Fix getting latest device IP for user with no devices
2017-04-11 09:56:54 +01:00
28a4649785
Remove HTTP replication APIs
2017-04-11 09:52:11 +01:00
7c551ec445
trust a hypothetical future riot.im IS
2017-04-10 17:58:36 +01:00
84fbb80c8f
Use generators
2017-04-10 16:55:56 +01:00
40453b3f84
Dedupe KeyedEdu and Devices federation repl traffic
2017-04-10 16:49:51 +01:00
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
2e6f5a4910
Typo
2017-04-10 16:17:40 +01:00
405ba4178a
Merge pull request #2102 from DanielDent/add-auth-email
...
Support authenticated SMTP
2017-04-10 15:42:16 +01:00
efcb6db688
Merge pull request #2109 from matrix-org/erikj/send_queue_fix
...
Fix up federation SendQueue and document types
2017-04-10 13:09:25 +01:00
0018491af2
Rename variable
2017-04-10 12:44:43 +01:00
0364d23210
Up replication ping timeout
2017-04-10 11:32:05 +01:00
8c5f03cec7
Revert to sending the same data type as before
2017-04-10 10:07:18 +01:00
f8434db549
Change name
2017-04-10 10:03:07 +01:00
ab904caf33
Comments
2017-04-10 10:02:17 +01:00
64765e5199
When we do an invite rejection, save the signed leave event to the db
...
During a rejection of an invite received over federation, we ask a remote
server to make us a `leave` event, then sign it, then send that with
`send_leave`.
We were saving the *unsigned* version of the event (which has a different event
id to the signed version) to our db (and sending it to the clients), whereas
other servers in the room will have seen the *signed* version. We're not aware
of any actual problems that caused, except that it makes the database confusing
to look at and generally leaves the room in a weird state.
2017-04-07 14:39:32 +01:00
2a3e822f44
Comment
2017-04-07 13:47:04 +01:00
a828a64b75
Comment
2017-04-07 11:54:03 +01:00
d4d176e5d0
Add logging
2017-04-07 11:51:28 +01:00
449d1297ca
Fix up federation SendQueue and document types
2017-04-07 11:48:33 +01:00
d72667fcce
Speed up get_current_state_ids
...
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).
get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
2017-04-07 10:10:49 +01:00
a41fe500d6
Bump version and changelog
2017-04-07 10:03:48 +01:00
54f59bd7d4
Merge pull request #2107 from HarHarLinks/patch-1
...
fix typo in synctl help
2017-04-07 09:54:37 +01:00
98ce212093
Merge pull request #2103 from matrix-org/erikj/no-double-encode
...
Don't double encode replication data
2017-04-07 09:39:52 +01:00
8a1137ceab
fix typo in synctl help
2017-04-06 17:10:20 +02:00
877c029c16
Use iteritems
2017-04-06 15:51:22 +01:00
944692ef69
Merge pull request #2106 from matrix-org/erikj/reduce_user_sync
...
Reduce rate of USER_SYNC repl commands
2017-04-06 13:35:31 +01:00
391712a4f9
Comment
2017-04-06 13:35:00 +01:00
ad544c803a
Document types of the replication streams
2017-04-06 13:28:52 +01:00
dbf87282d3
Docs
2017-04-06 13:11:21 +01:00
69b3fd485d
Fix incorrect type when using InvalidateCacheCommand
2017-04-06 09:36:38 +01:00
5058292537
Support authenticated SMTP
...
Closes (SYN-714) #1385
Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
2017-04-05 21:01:08 -07:00
fcc803b2bf
Add log lines
2017-04-05 17:13:44 +01:00
3f213d908d
Rearrange metrics
2017-04-05 14:15:09 +01:00
1ca0e78ca1
Fix typo
2017-04-05 13:43:39 +01:00
b43d3267e2
Fixup some metrics for tcp repl
2017-04-05 13:34:54 +01:00
b5cb6347a4
Don't immediately notify the master about users whose syncs have gone away
2017-04-05 13:25:40 +01:00
96b9b6c127
Don't double json encode typing replication data
2017-04-05 11:34:20 +01:00
f10ce8944b
Don't double json encode federation replication data
2017-04-05 11:10:28 +01:00
a5c401bd12
Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
...
Move to using TCP replication
2017-04-05 09:36:21 +01:00
d1d5362267
Add comment
2017-04-04 16:41:03 +01:00
9f26d3b75b
Deduplicate new deviceinbox rows for replication
2017-04-04 16:21:21 +01:00
a76886726b
Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix
...
Advance replication streams even if nothing is listening
2017-04-04 15:40:51 +01:00
ac66e11f2b
Add the appropriate amount of preserve_fn
2017-04-04 15:22:54 +01:00
4264ceb31c
Fiddle tcp replication logging
2017-04-04 14:14:03 +01:00
023ee197be
Advance replication streams even if nothing is listening
...
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
2017-04-04 13:19:26 +01:00
d1605794ad
Remove unused worker config option
2017-04-04 11:17:00 +01:00
3376f16012
Shuffle and comment synchrotron presence
2017-04-04 11:14:16 +01:00
6ce6bbedcb
Move where we ack federation
2017-04-04 11:02:44 +01:00
62b89daac6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server
2017-04-04 09:46:16 +01:00
773e64cc1a
Merge pull request #2095 from matrix-org/rav/cull_log_preserves
...
Cull spurious PreserveLoggingContexts
2017-04-03 17:02:25 +01:00
2d05eb3cf5
Merge remote-tracking branch 'origin/release-v0.20.0' into develop
2017-04-03 16:23:23 +01:00
30bcbf775a
Accept join events from all servers
...
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.
(Fixes #1893 ).
2017-04-03 15:58:07 +01:00
7eb9f34cc3
Remove spurious yield
...
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
2017-04-03 15:44:19 +01:00
0b08c48fc5
Remove more spurious `PreserveLoggingContext`s
...
Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`;
there is no problem if the logcontext is set when calling it.
2017-04-03 15:43:37 +01:00
65e1683680
Remove spurious PreserveLoggingContext
...
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
2017-04-03 15:42:38 +01:00
feb496056e
preserve_fn some deferred-returning things
...
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
2017-04-03 15:41:17 +01:00
e2eebf1696
Fix fixme in preserve_fn
...
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
2017-04-03 15:38:02 +01:00
36c28bc467
Update all the workers and master to use TCP replication
2017-04-03 15:35:52 +01:00
3a1f3f8388
Change slave storage to use new replication interface
...
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
Richard van der Hoff