78f306a6f7
Revert "Speed up filtering of a single event in push"
...
This reverts commit 421fdf7460
2017-05-08 13:07:41 +01:00
66f371e8b8
Merge pull request #2176 from matrix-org/erikj/faster_get_joined
...
Make get_joined_users faster
2017-05-05 10:59:55 +01:00
e3aa8a7aa8
Merge pull request #2185 from matrix-org/erikj/smaller_caches
...
Optimise caches for single key
2017-05-05 10:19:05 +01:00
cf589f2c1e
Fixes
2017-05-05 10:17:56 +01:00
8af4569583
Merge pull request #2174 from matrix-org/erikj/current_cache_hosts
...
Add cache for get_current_hosts_in_room
2017-05-05 10:15:24 +01:00
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
aa93cb9f44
Add comment
2017-05-04 14:59:28 +01:00
537dbadea0
Intern host strings
2017-05-04 14:55:28 +01:00
07a07588a0
Make caches bigger
2017-05-04 14:52:28 +01:00
dfaa58f72d
Fix comment and num args
2017-05-04 14:50:24 +01:00
5d8290429c
Reduce size of get_users_in_room
2017-05-04 13:43:19 +01:00
7ebf518c02
Make get_joined_users faster
2017-05-03 15:55:54 +01:00
2c2dcf81d0
Update comment
2017-05-03 10:00:29 +01:00
1827057acc
Comments
2017-05-03 09:56:05 +01:00
8346e6e696
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-03 09:46:40 +01:00
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
3033261891
Merge pull request #2080 from matrix-org/erikj/filter_speed
...
Speed up filtering of a single event in push
2017-04-28 14:17:13 +01:00
ab37bef83b
Remove unused import
2017-04-28 09:57:23 +01:00
ad8b316939
We don't care about forgotten rooms
2017-04-28 09:52:36 +01:00
421fdf7460
Speed up filtering of a single event in push
2017-04-28 09:52:36 +01:00
c84770b877
Fix bgupdate error if index already exists ( #2167 )
...
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135 .
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
f053a1409e
Make state caches cache in ascii
2017-04-25 17:22:55 +01:00
f144365281
Comment
2017-04-25 15:18:26 +01:00
d9aa645f86
Reduce size of joined_user cache
...
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.
We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
22f3d3ae76
Reduce _get_state_group_for_event cache size
2017-04-25 11:43:03 +01:00
119cb9bbcf
Reduce cache size by not storing deferreds
...
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.
However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.
This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.
As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
78f0ddbfad
Merge pull request #2120 from matrix-org/luke/read-markers
...
Implement Read Marker API
2017-04-13 14:21:31 +01:00
6a70647d45
Correct logic in is_event_after
2017-04-13 13:46:17 +01:00
b9557064bf
Simplify is_event_after logic
2017-04-12 14:36:20 +01:00
7f94709066
travis flake8..
2017-04-11 18:35:45 +01:00
73880268ef
Refactor event ordering check to events store
2017-04-11 17:34:09 +01:00
85657eedf8
Bail on where clause instead
2017-04-11 16:24:31 +01:00
b48045a8f5
Don't bother with outer check for now
2017-04-11 16:23:24 +01:00
34840cdcef
Fix getting latest device IP for user with no devices
2017-04-11 09:56:54 +01:00
2a3e822f44
Comment
2017-04-07 13:47:04 +01:00
d72667fcce
Speed up get_current_state_ids
...
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).
get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
2017-04-07 10:10:49 +01:00
9f26d3b75b
Deduplicate new deviceinbox rows for replication
2017-04-04 16:21:21 +01:00
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
30348c924c
Use txn.fetchall() so we can reuse txn
2017-03-30 10:30:05 +01:00
3ce8d59176
Increase cache size for _get_state_group_for_event
2017-03-29 14:31:46 +01:00
e4df0e189d
Decrank last commit
2017-03-29 11:02:35 +01:00
4ad613f6be
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert
2017-03-29 10:57:19 +01:00
ac6bc55512
Correctly look up key
2017-03-29 10:56:26 +01:00
d28db583da
Merge pull request #2063 from matrix-org/erikj/device_list_batch
...
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
58a35366be
The algorithm is part of the key id
2017-03-28 11:34:37 +01:00
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
281553afe6
Merge pull request #2062 from matrix-org/erikj/presence_replication
...
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
987f4945b4
Actually call invalidate
2017-03-24 13:28:20 +00:00
09f79aaad0
Use presence replication stream to invalidate cache
...
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
7fc1f1e2b6
Cache hosts in room
2017-03-24 11:46:24 +00:00
d58b1ffe94
Replace some calls to cursor_to_dict
...
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
6ebe2d23b1
Raise a more helpful exception
2017-03-23 13:48:30 +00:00
e64655c25d
Don't user upsert to persist new one time keys
...
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
2017-03-23 13:17:00 +00:00
64778693be
fix up some key verif docstrings
2017-03-21 13:27:50 +00:00
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
eddce9d74a
Merge pull request #2027 from matrix-org/rav/logcontext_leaks
...
A few fixes to logcontext things
2017-03-20 11:53:36 +00:00
994d7ae7c5
Remove broken use of clock.call_later
...
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.
We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
0c01f829ae
Avoid resetting state on rejected events
...
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.
This should fix https://github.com/matrix-org/synapse/issues/1935 .
2017-03-17 15:06:08 +00:00
5068fb16a5
Refactoring and cleanups
...
A few non-functional changes:
* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
mind-bending conditional.
2017-03-17 15:06:07 +00:00
da146657c9
Comments
2017-03-16 13:04:07 +00:00
6957bfdca6
Don't recreate so many sets
2017-03-16 11:54:26 +00:00
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
bb256ac96f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed
2017-03-14 11:35:05 +00:00
cc7a294e2e
Fix current_state_events table to not lie
...
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
2017-03-14 10:57:43 +00:00
bbeeb97f75
Implement _simple_delete_many_txn, use it to delete devices
...
(But this doesn't implement the same for deleting access tokens or e2e keys.
Also respond to code review.
2017-03-13 17:53:23 +00:00
79926e016e
Assume rooms likely haven't changed
2017-03-13 09:50:10 +00:00
8ffbe43ba1
Get current state by using current_state_events table
2017-03-10 17:39:35 +00:00
64d62e41b8
Noop repated delete device inbox calls from /sync
2017-03-10 10:36:43 +00:00
6ad71cc29d
Remove spurious SQL logging ( #1972 )
...
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
2017-03-08 18:00:44 +00:00
ac5491f563
Select distinct devices from DB
...
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
2017-03-06 11:10:14 +00:00
da52d3af31
Fix up
2017-03-03 15:29:13 +00:00
d766343668
Add index to device_lists_stream
2017-03-01 15:56:30 +00:00
3365117151
Clobber old device list stream entries
2017-03-01 10:21:30 +00:00
d638a7484b
Merge pull request #1959 from matrix-org/erikj/intern_once
...
Intern table column names once
2017-02-28 15:15:16 +00:00
b84907bdbb
Intern table column names once
2017-02-28 14:38:16 +00:00
e4919b9329
Add stream_id index to device_lists_outbound_pokes
...
As this is used for replication streaming
2017-02-28 11:19:06 +00:00
8a12b6f1eb
Fix up txn name
2017-02-28 10:15:50 +00:00
a41dce8f8a
Remove needless check
2017-02-27 18:54:43 +00:00
c0d6045776
It should be all
2017-02-27 18:45:24 +00:00
49f4bc4709
Don't fetch current state in common case
...
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
2017-02-27 18:33:41 +00:00
fc5d876dba
Merge pull request #1954 from matrix-org/erikj/cache_device2
...
Cache get_user_devices_from_cache
2017-02-27 16:36:40 +00:00
f58dbb02a6
Cache get_user_devices_from_cache
2017-02-27 16:22:12 +00:00
1a4f8022e6
Strip newlines from SQL queries
2017-02-23 11:15:31 +00:00
b2d20e94fa
Remove lock from rotate notifs
2017-02-22 14:24:02 +00:00
b7442c3e2b
Store looping call
2017-02-21 13:59:25 +00:00
699be7d1be
Fix up notif rotation
2017-02-18 14:42:39 +00:00
4aa29508af
Use TEXT rather than VARCHAR
...
While we're changing anyway
2017-02-17 10:51:49 +00:00
b4017539d4
Make the pushers lang field column longer
...
To accommodate things like zh-Hans-CN
Fixes https://github.com/vector-im/riot-ios/issues/1031
2017-02-17 10:42:57 +00:00
138e030cfe
Comment
2017-02-16 15:03:36 +00:00
502ae6c663
Comment
2017-02-16 14:47:11 +00:00
e6acf0c399
Store the default push actions in a more efficient manner
2017-02-16 14:40:24 +00:00
04eca2589d
Merge pull request #1916 from matrix-org/erikj/push_actions_delete
...
Aggregate event push actions
2017-02-16 15:28:58 +01:00
7dcbcca68c
Merge pull request #1921 from matrix-org/rav/fix_key_changes
...
Fix bugs in the /keys/changes api
2017-02-15 11:25:16 +00:00
fc2f29c1d0
Fix bugs in the /keys/changes api
...
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
passing a `RoomStreamToken`, which meant that it returned the *current*
extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
that a `TypeError` was thrown and we ended up acting as though there was *no*
prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
dictionary - just looking up the state keys in it again meant that we acted
as though there was no prev state. We now check if each member's state has
changed since *any* of the extremities.
Also add/fix some comments.
2017-02-14 13:59:50 +00:00
ce3c8df6df
Less aggressive timers
2017-02-14 13:41:24 +00:00
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
795f8e3fe7
Merge pull request #1873 from matrix-org/erikj/delete_push_actions
...
Be more agressive about purging old room event_push_actions
2017-02-14 14:29:04 +01:00
9e617cd4c2
Cache get_presence storage
2017-02-13 13:50:03 +00:00
52cd019a54
Make None check explicit
2017-02-08 16:04:29 +00:00
f20cd34858
Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem
...
Ignore new rejected events when working out forward extremities.
2017-02-08 16:59:06 +01:00
7723b4caa4
Ignore new rejected events when working out forward extremeties.
2017-02-08 14:48:06 +00:00
af6da6db2d
Merge pull request #1784 from morteza-araby/user-admin
...
Administration functionalities
2017-02-06 16:21:10 +01:00
38258a0976
Bump cache sizes for common membership queries
2017-02-02 18:45:55 +00:00
832e9c52ca
Comment
2017-02-02 13:09:56 +00:00
54a79c1d37
Make presence.get_new_events a bit faster
...
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
2849d3f29d
admin,storage: added more administrator functionalities
...
administrators can now:
- Set displayname of users
- Update user avatars
- Search for users by user_id
- Browse all users in a paginated API
- Reset user passwords
- Deactivate users
Helpers for doing paginated queries has also been added to storage
Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
fac3c03087
Be more agressive about purging old room event_push_actions
2017-02-01 18:27:24 +00:00
fbfe44bb4d
Doc args
2017-02-01 17:52:57 +00:00
d61a04583e
Comment
2017-02-01 17:35:23 +00:00
7e919bdbd0
Include newly joined users in /keys/changes API
2017-02-01 17:33:16 +00:00
ebfaff84c9
Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
...
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
62f6b86ba7
Merge pull request #1868 from matrix-org/erikj/replication_cache
...
Only invalidate membership caches based on the cache stream
2017-02-01 13:12:30 +00:00
f6124311fd
Add m.room.member type to query
2017-02-01 11:59:17 +00:00
88a4d54883
Merge pull request #1867 from matrix-org/erikj/member_index
...
Add an index to make membership queries faster
2017-02-01 11:44:27 +00:00
368c88c487
Add a small cache get_all_new_events
2017-02-01 10:50:44 +00:00
5deaf9e30b
Up get_latest_event_ids_in_room cache
2017-02-01 10:39:41 +00:00
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
ac001dabdc
Merge pull request #1864 from matrix-org/erikj/device_list_fixes
...
Fix clearing out old device list outbound pokes
2017-01-31 13:35:35 +00:00
bfb3d255b1
Merge pull request #1862 from matrix-org/erikj/presence_update
...
Use DB cache of joined users for presence
2017-01-31 13:23:24 +00:00
ab55794b6f
Fix deletion of old sent devices correctly
2017-01-31 13:22:41 +00:00
d3169e8d28
Only fetch with row ts and count > 1
2017-01-31 11:20:03 +00:00
05b9f48ee5
Fix clearing out old device list outbound pokes
2017-01-31 10:08:55 +00:00
4b3403ca9b
Stream cache invalidations for room membership storage functions
2017-01-30 17:28:22 +00:00
1c13c9f6b6
Don't have such a large cache
2017-01-30 17:12:14 +00:00
c7a26b7c32
Fix unit tests
2017-01-30 17:11:24 +00:00
c2c9a78db9
Noop device key changes if they're the same
2017-01-30 16:55:04 +00:00
e75a779d9e
Fix query
2017-01-30 16:38:20 +00:00
828db669ec
Use get_users_in_room and declare it iterable
2017-01-30 16:37:22 +00:00
3670025e64
Rename func
2017-01-30 14:11:31 +00:00
d360c97ae1
Clear out old destination pokes.
2017-01-30 10:14:37 +00:00
76100203ab
Always use the latest stream_id, sent or unsent
2017-01-30 10:14:25 +00:00
d1e1fd6210
Add ts column to device_lists_outbound_pokes
2017-01-27 15:23:48 +00:00
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
84a35f32c7
Comment
2017-01-27 10:35:12 +00:00
c517a19c2d
Comment
2017-01-27 10:33:26 +00:00
738a2867c8
SQL param ordering
2017-01-27 10:31:29 +00:00
755adff0e4
User if rather than for
2017-01-27 10:31:06 +00:00
888c59c955
Better name
2017-01-27 10:29:47 +00:00
f25a4a4692
Remove unused param
2017-01-27 10:27:39 +00:00
31aca5589c
Fix on sqlite: use left rather than outer join
2017-01-26 16:55:50 +00:00
Erik Johnston