Commit Graph

3890 Commits (79c1f973cee73f2d24c1b9140500e1999b25a479)

Author SHA1 Message Date
Erik Johnston bbb3c8641c
Make MultiWriterIDGenerator work for streams that use negative stream IDs (#8203)
This is so that we can use it for the backfill events stream.
2020-09-01 13:36:25 +01:00
Richard van der Hoff aa07c37cf0
Move and rename `get_devices_with_keys_by_user` (#8204)
* Move `get_devices_with_keys_by_user` to `EndToEndKeyWorkerStore`

this seems a better fit for it.

This commit simply moves the existing code: no other changes at all.

* Rename `get_devices_with_keys_by_user`

to better reflect what it does.

* get_device_stream_token abstract method

To avoid referencing fields which are declared in the derived classes, make
`get_device_stream_token` abstract, and define that in the classes which define
`_device_list_id_gen`.
2020-09-01 12:41:21 +01:00
Richard van der Hoff 45e8f7726f
Rename `get_e2e_device_keys` to better reflect its purpose (#8205)
... and to show that it does something slightly different to
`_get_e2e_device_keys_txn`.

`include_all_devices` and `include_deleted_devices` were never used (and
`include_deleted_devices` was broken, since that would cause `None`s in the
result which were not handled in the loop below.

Add some typing too.
2020-08-29 00:14:17 +01:00
Andrew Morgan b4826d6eb1 Fix incorrect return signature 2020-08-28 17:39:48 +01:00
Patrick Cloke d58fda99ff
Convert `event_push_actions`, `registration`, and `roommember` datastores to async (#8197) 2020-08-28 11:34:50 -04:00
Richard van der Hoff 22b926c284
Only return devices with keys from `/federation/v1/user/devices/` (#8198)
There's not much point in returning all the others, and some people have a
silly number of devices.
2020-08-28 15:59:28 +01:00
Patrick Cloke aec7085179
Convert state and stream stores and related code to async (#8194) 2020-08-28 09:37:55 -04:00
Patrick Cloke 5c03134d0f
Convert additional database code to async/await. (#8195) 2020-08-28 07:54:27 -04:00
Patrick Cloke b49a5b9307
Convert stats and related calls to async/await (#8192) 2020-08-27 17:24:37 -04:00
Patrick Cloke b71d4a094c
Convert simple_delete to async/await. (#8191) 2020-08-27 14:16:41 -04:00
Patrick Cloke 9b7ac03af3
Convert calls of async database methods to async (#8166) 2020-08-27 13:38:41 -04:00
Patrick Cloke c9fa696ea2
simple_search_list_txn should return None, not 0. (#8187) 2020-08-27 12:07:13 -04:00
Erik Johnston 5649b7f3d0
Fix missing _add_persisted_position (#8179)
This was forgotten in #8164.
2020-08-27 13:20:34 +01:00
Patrick Cloke 30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) (#8168) 2020-08-27 07:41:01 -04:00
Patrick Cloke 4a739c73b4
Convert simple_update* and simple_select* to async (#8173) 2020-08-27 07:08:38 -04:00
Erik Johnston e3c91a3c55
Make SlavedIdTracker.advance have same interface as MultiWriterIDGenerator (#8171) 2020-08-26 13:15:20 +01:00
Patrick Cloke 4c6c56dc58
Convert simple_select_one and simple_select_one_onecol to async (#8162) 2020-08-26 07:19:32 -04:00
Erik Johnston eba98fb024
Add functions to `MultiWriterIdGen` used by events stream (#8164) 2020-08-25 17:32:30 +01:00
Erik Johnston 2231dffee6
Make StreamIdGen `get_next` and `get_next_mult` async (#8161)
This is mainly so that `StreamIdGenerator` and `MultiWriterIdGenerator`
will have the same interface, allowing them to be used interchangeably.
2020-08-25 15:10:08 +01:00
Andrew Morgan 74bf8d4d06
Wording fixes to 'name' user admin api filter (#8163)
Some fixes to wording I noticed after merging #7377.
2020-08-25 15:03:24 +01:00
Manuel Stahl 97962ad17b
Search in columns 'name' and 'displayname' in the admin users endpoint (#7377)
* Search in columns 'name' and 'displayname' in the admin users endpoint

Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-08-25 14:18:14 +01:00
Brendan Abolivier 3f49f74610
Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on (#7991)
* Don't raise session_id errors on submit_token if request_token_inhibit_3pid_errors is set

* Changelog

* Also wait some time before responding to /requestToken

* Incorporate review

* Update synapse/storage/databases/main/registration.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Incorporate review

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-08-24 11:33:55 +01:00
Richard van der Hoff 09fd0eda81
Micro-optimisations to get_auth_chain_ids (#8132) 2020-08-21 10:06:45 +01:00
Patrick Cloke 3f91638da6
Allow denying or shadow banning registrations via the spam checker (#8034) 2020-08-20 15:42:58 -04:00
Richard van der Hoff 318f4e738e
Be more tolerant of membership events in unknown rooms (#8110)
It turns out that not all out-of-band membership events are labelled as such,
so we need to be more accepting here.
2020-08-20 16:42:12 +01:00
Patrick Cloke dbc630a628
Use the JSON encoder without whitespace in more places. (#8124) 2020-08-20 10:32:33 -04:00
Patrick Cloke 5eac0b7e76
Add more types to synapse.storage.database. (#8127) 2020-08-20 09:00:59 -04:00
Patrick Cloke 76c43f086a
Do not assume calls to runInteraction return Deferreds. (#8133) 2020-08-20 06:39:55 -04:00
Richard van der Hoff 12aebdfa5a
Close the database connection we create during startup (#8131)
... otherwise it gets leaked.
2020-08-19 20:41:53 +01:00
Erik Johnston c9c544cda5
Remove `ChainedIdGenerator`. (#8123)
It's just a thin wrapper around two ID gens to make `get_current_token`
and `get_next` return tuples. This can easily be replaced by calling the
appropriate methods on the underlying ID gens directly.
2020-08-19 13:41:51 +01:00
Patrick Cloke eebf52be06
Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
Patrick Cloke d89692ea84
Convert runWithConnection to async. (#8121) 2020-08-19 07:09:24 -04:00
Erik Johnston 76d21d14a0
Separate `get_current_token` into two. (#8113)
The function is used for two purposes: 1) for subscribers of streams to
get a token they can use to get further updates with, and 2) for
replication to track position of the writers of the stream.

For streams with a single writer the two scenarios produce the same
result, however the situation becomes complicated for streams with
multiple writers. The current `MultiWriterIdGenerator` does not
correctly handle the first case (which is not an issue as its only used
for the `caches` stream which nothing subscribes to outside of
replication).
2020-08-19 10:39:31 +01:00
Patrick Cloke f40645e60b
Convert events worker database to async/await. (#8071) 2020-08-18 16:20:49 -04:00
Patrick Cloke 3c01724b33
Fix the return type of send_nonmember_events. (#8112) 2020-08-18 09:53:13 -04:00
Patrick Cloke 050e20e7ca
Convert some of the general database methods to async (#8100) 2020-08-17 12:18:01 -04:00
Patrick Cloke ad6190c925
Convert stream database to async/await. (#8074) 2020-08-17 07:24:46 -04:00
Patrick Cloke ac77cdb64e
Add a shadow-banned flag to users. (#8092) 2020-08-14 12:37:59 -04:00
Patrick Cloke b069b78bb4
Convert pusher databases to async/await. (#8075) 2020-08-14 10:30:16 -04:00
Patrick Cloke e8861957d9
Convert receipts and events databases to async/await. (#8076) 2020-08-14 10:05:19 -04:00
Patrick Cloke 6b7ce1d332
Remove some unused database functions. (#8085) 2020-08-14 09:25:40 -04:00
Patrick Cloke 894dae74fe
Convert misc database code to async (#8087) 2020-08-14 07:24:26 -04:00
Patrick Cloke fbe930dad2
Convert the roommember database to async/await. (#8070) 2020-08-12 12:14:34 -04:00
Patrick Cloke 5ecc8b5825
Convert devices database to async/await. (#8069) 2020-08-12 10:51:42 -04:00
Patrick Cloke d68e10f308
Convert account data, device inbox, and censor events databases to async/await (#8063) 2020-08-12 09:29:06 -04:00
Patrick Cloke a3a59bab7b
Convert appservice, group server, profile and more databases to async (#8066) 2020-08-12 09:28:48 -04:00
Erik Johnston 6ba621d786
Merge pull request #8060 from matrix-org/erikj/type_server
Change HomeServer definition to work with typing.
2020-08-11 22:32:14 +01:00
Patrick Cloke 04faa0bfa9
Convert tags and metrics databases to async/await (#8062) 2020-08-11 17:21:20 -04:00
Patrick Cloke a0acdfa9e9
Converts event_federation and registration databases to async/await (#8061) 2020-08-11 17:21:13 -04:00
Erik Johnston 0f1afbe8dc Change HomeServer definition to work with typing.
Duplicating function signatures between server.py and server.pyi is
silly. This commit changes that by changing all `build_*` methods to
`get_*` methods and changing the `_make_dependency_method` to work work
as a descriptor that caches the produced value.

There are some changes in other files that were made to fix the typing
in server.py.
2020-08-11 18:00:17 +01:00
Brendan Abolivier cdbb8e6d6e
Implement new experimental push rules (#7997)
With an undocumented configuration setting to enable them for specific users.
2020-08-10 11:48:01 +01:00
Patrick Cloke 7f837959ea
Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users database to async (#8042) 2020-08-07 13:36:29 -04:00
Patrick Cloke f3fe6961b2
Convert additional database stores to async/await (#8045) 2020-08-07 12:17:17 -04:00
David Vo 4dd27e6d11
Reduce unnecessary whitespace in JSON. (#7372) 2020-08-07 08:02:55 -04:00
Brendan Abolivier 2ffd6783c7
Revert #7736 (#8039) 2020-08-06 17:15:35 +01:00
Patrick Cloke d4a7829b12
Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Brendan Abolivier 118a9eafb3 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules 2020-08-06 10:52:50 +01:00
Brendan Abolivier dd11f575a2
Incorporate review 2020-08-06 10:52:26 +01:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke e19de43eb5
Convert streams to async. (#8014) 2020-08-04 07:21:47 -04:00
Andrew Morgan 5d92a1428c
Prevent join->join membership transitions changing member count (#7977)
`StatsHandler` handles updates to the `current_state_delta_stream`, and updates room stats such as the amount of state events, joined users, etc.

However, it counts every new join membership as a new user entering a room (and that user being in another room), whereas it's possible for a user's membership status to go from join -> join, for instance when they change their per-room profile information.

This PR adds a check for join->join membership transitions, and bails out early, as none of the further checks are necessary at that point.

Due to this bug, membership stats in many rooms have ended up being wildly larger than their true values. I am not sure if we also want to include a migration step which recalculates these statistics (possibly using the `_populate_stats_process_rooms` bg update).

Bug introduced in the initial implementation https://github.com/matrix-org/synapse/pull/4338.
2020-08-03 21:54:24 +01:00
Brendan Abolivier 1678057b56
Back out the database hack and replace it with a temporary config setting 2020-08-03 11:22:22 +01:00
Brendan Abolivier cf42d0a60c
Fix cache name 2020-07-31 15:06:41 +01:00
Brendan Abolivier 79d991eff0
Fix cache invalidation calls 2020-07-31 13:58:42 +01:00
Brendan Abolivier 69158e554f Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules 2020-07-30 19:00:29 +01:00
Olivier Wilkinson (reivilibre) 3aa36b782c Merge branch 'master' into develop 2020-07-30 15:18:36 +01:00
Patrick Cloke b3a97d6dac
Convert some of the data store to async. (#7976) 2020-07-30 07:20:41 -04:00
Erik Johnston 2c1b9d6763
Update worker docs with recent enhancements (#7969) 2020-07-29 23:22:13 +01:00
Patrick Cloke d90087cffa
Remove from the event_relations table when purging historical events. (#7978) 2020-07-29 13:55:01 -04:00
Brendan Abolivier 8dff4a1242
Re-implement unread counts (#7736) 2020-07-29 18:26:55 +01:00
Patrick Cloke 3345c166a4
Convert storage layer to async/await. (#7963) 2020-07-28 16:09:53 -04:00
Brendan Abolivier 9725c59247
Implement new experimental push rules with a database hack to enable them 2020-07-28 19:20:55 +01:00
Andrew Morgan 8a25332d94
Move some log lines from default logger to sql/transaction loggers (#7952)
Idea from matrix-org/synapse-dinsic#49
2020-07-28 18:52:13 +01:00
Patrick Cloke 8144bc26a7
Convert push to async/await. (#7948) 2020-07-27 12:21:34 -04:00
Patrick Cloke b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Patrick Cloke 13d77464c9
Follow-up to admin API to re-activate accounts (#7908) 2020-07-22 12:33:19 -04:00
Erik Johnston 2d2acc1cf2
Stop using 'device_max_stream_id' (#7882)
It serves no purpose and updating everytime we write to the device inbox
stream means all such transactions will conflict, causing lots of
transaction failures and retries.
2020-07-17 17:03:27 +01:00
Erik Johnston 4642fd66df
Change "unknown room ver" logging to warning. (#7881)
It's somewhat expected for us to have unknown room versions in the
database due to room version experiments.
2020-07-17 12:10:43 +01:00
Patrick Cloke f460da6031
Consistently use `db_to_json` to convert from database values to JSON objects. (#7849) 2020-07-16 11:32:19 -04:00
Richard van der Hoff a827838706
Merge pull request #7866 from matrix-org/rav/fix_guest_user_id
Fix guest user registration with lots of client readers
2020-07-16 13:54:45 +01:00
Richard van der Hoff c445bc0cad Use a postgres sequence to generate guest user IDs 2020-07-16 13:00:25 +01:00
Richard van der Hoff 3c36ae17a5 Use SequenceGenerator for state group ID allocation 2020-07-16 11:25:08 +01:00
Richard van der Hoff 42509b8fb6 Use `PostgresSequenceGenerator` from `MultiWriterIdGenerator`
partly just to show it works, but alwo to remove a bit of code duplication.
2020-07-16 11:25:08 +01:00
Richard van der Hoff 90b0cdda42 Add some helper classes for generating ID sequences 2020-07-16 11:25:08 +01:00
Richard van der Hoff a57df9b827
Avoid brand new rooms in `delete_old_current_state_events` (#7854)
When considering rooms to clean up in `delete_old_current_state_events`, skip
rooms which we are creating, which otherwise look a bit like rooms we have
left.

Fixes #7834.
2020-07-15 18:33:03 +01:00
Richard van der Hoff 1d9dca02f9
remove `retry_on_integrity_error` wrapper for persist_events (#7848)
As far as I can tell from the sentry logs, the only time this has actually done
anything in the last two years is when we had two master workers running at
once, and even then, it made a bit of a mess of it (see
https://github.com/matrix-org/synapse/issues/7845#issuecomment-658238739).

Generally I feel like this code is doing more harm than good.
2020-07-15 10:34:53 +01:00
Patrick Cloke 8d0097bef1
Fix bug in per-room message retention policies. (#7850) 2020-07-14 15:51:13 -04:00
Dirk Klimpel 491f0dab1b
Add delete room admin endpoint (#7613)
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-14 12:36:23 +01:00
Richard van der Hoff 6cef918a4b Merge branch 'release-v1.17.0' into develop 2020-07-10 18:38:50 +01:00
Richard van der Hoff 8ccb7f08d9 Merge branch 'master' into release-v1.17.0 2020-07-10 18:38:18 +01:00
Erik Johnston f299441cc6
Add ability to shard the federation sender (#7798) 2020-07-10 18:26:36 +01:00
Richard van der Hoff b1beb3ff59 fix migration, again 2020-07-10 12:18:35 +01:00
Richard van der Hoff cbabcec05c
Drop incorrectly-added table `local_rejections_stream`. (#7816) 2020-07-10 12:07:59 +01:00
Richard van der Hoff 2ab0b021f1
Generate real events when we reject invites (#7804)
Fixes #2181. 

The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
2020-07-09 10:40:19 +01:00
Patrick Cloke e7efd8f827
Do not use simplejson in Synapse. (#7800) 2020-07-08 07:15:08 -04:00
Richard van der Hoff 76dbd7b8d6
Stop populating unused table `local_invites`. (#7793)
This table is no longer used, so we may as well stop populating it. Removing it
would prevent people rolling back to older releases of Synapse, so that can
happen in a future release.
2020-07-07 14:20:40 +01:00
Erik Johnston 67d7756fcf
Refactor getting replication updates from database v2. (#7740) 2020-07-07 12:11:35 +01:00
Will Hunt 62b1ce8539
isort 5 compatibility (#7786)
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
2020-07-05 16:32:02 +01:00
Richard van der Hoff f01e2ca039
Use symbolic names for replication stream names (#7768)
This makes it much easier to find where streams are referenced.
2020-07-01 16:35:40 +01:00
Richard van der Hoff 244dbb04f7
Fix incorrect error message when database CTYPE was set incorrectly. (#7760) 2020-07-01 13:56:16 +01:00
Brendan Abolivier 74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Patrick Cloke 95e41f368b
Allow local media to be marked as safe from being quarantined. (#7718) 2020-06-22 08:04:14 -04:00
Brendan Abolivier 5a5cf6460e
Fix unread counts in sync
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
2020-06-17 15:10:44 +01:00
Brendan Abolivier 46613aaf79
Implement unread counter (MSC2625) (#7673)
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
Erik Johnston f6f7511a4c
Refactor getting replication updates from database. (#7636)
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
2020-06-16 17:10:28 +01:00
Patrick Cloke 231252516c
Fix "argument of type 'ObservableDeferred' is not iterable" error (#7708) 2020-06-16 12:01:18 -04:00
Dagfinn Ilmari Mannsåker a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Brendan Abolivier 6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Patrick Cloke bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Brendan Abolivier fed493c5fd
Incorporate review 2020-06-15 09:58:55 +01:00
Patrick Cloke 2d11ea385c
Fix warnings about losing log context during UI auth. (#7688) 2020-06-12 15:01:00 -04:00
Brendan Abolivier e186c660b1
Lint 2020-06-12 15:31:59 +01:00
Brendan Abolivier e47e5a2dcd
Incorporate review bits 2020-06-12 15:13:12 +01:00
Brendan Abolivier 1e5a50302f
Pre-populate the unread_count column 2020-06-12 15:05:47 +01:00
Brendan Abolivier 9549d557ea
Don't update the schema version 2020-06-12 15:03:26 +01:00
Brendan Abolivier cf92fbb8aa
Use attr instead of a dict 2020-06-12 15:02:15 +01:00
Brendan Abolivier 3cc7f43e8d
Fix summary rotation 2020-06-12 11:07:26 +01:00
Brendan Abolivier cb6d4d07b1
Log for invalid values of notif 2020-06-11 18:30:31 +01:00
Brendan Abolivier 803291728c
Fix SQL 2020-06-11 18:25:25 +01:00
Brendan Abolivier 34fd1f7ab5
Fix schema update 2020-06-11 18:12:12 +01:00
Brendan Abolivier d0f095625c
Lint 2020-06-11 18:04:43 +01:00
Brendan Abolivier ce74a6685d
Save the count of unread messages to event_push_summary 2020-06-11 17:58:26 +01:00
Brendan Abolivier df3323a7cf
Use temporary prefixes as per the MSC 2020-06-10 20:32:01 +01:00
Brendan Abolivier c7b99a1180
Use a more efficient way of calculating counters 2020-06-10 17:54:33 +01:00
Brendan Abolivier ef345c5a7b
Add a new unread_counter to sync responses 2020-06-10 16:21:16 +01:00
Brendan Abolivier 6f6a4bfc07
Rename dont_push into mark_unread 2020-06-10 14:24:01 +01:00
Brendan Abolivier ec0a7b9034 Merge branch 'develop' into babolivier/mark_unread 2020-06-10 11:42:30 +01:00
Erik Johnston 664409b169
Fix bug in account data replication stream. (#7656)
* Ensure account data stream IDs are unique.

The account data stream is shared between three tables, and the maximum
allocated ID was tracked in a dedicated table. Updating the max ID
happened outside the transaction that allocated the ID, leading to a
race where if the server was restarted then the same ID could be
allocated but the max ID failed to be updated, leading it to be reused.

The ID generators have support for tracking across multiple tables, so
we may as well use that instead of a dedicated table.

* Fix bug in account data replication stream.

If the same stream ID was used in both global and room account data then
the getting updates for the replication stream would fail due to
`heapq.merge(..)` trying to compare a `str` with a `None`. (This is
because you'd have two rows like `(534, '!room')` and `(534, None)` from
the room and global account data tables).

Fix is just to order by stream ID, since we don't rely on the ordering
beyond that. The bug where stream IDs can be reused should be fixed now,
so this case shouldn't happen going forward.

Fixes #7617
2020-06-09 16:28:57 +01:00
Andrew Morgan e91abfd291
async/await get_user_id_by_threepid (#7620)
Based on #7619 

async's `get_user_id_by_threepid` and its call stack.
2020-06-03 17:15:57 +01:00
Dagfinn Ilmari Mannsåker df8a3cef6b
Improve performance of _get_state_groups_from_groups_txn (#7567)
The query keeps showing up in my slow query log.

This changes the plan under the top-level Sort node from

```
    WindowAgg  (cost=280335.88..292963.15 rows=561212 width=80) (actual time=138.651..160.562 rows=27112 loops=1)
      ->  Sort  (cost=280335.88..281738.91 rows=561212 width=84) (actual time=138.597..140.622 rows=27112 loops=1)
            Sort Key: state_groups_state.type, state_groups_state.state_key, state_groups_state.state_group
            Sort Method: quicksort  Memory: 4581kB
            ->  Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=21.548..47.657 rows=27112 loops=1)
                  ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=21.526..21.535 rows=20 loops=1)
                        Group Key: state.state_group
                        ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=21.280..21.493 rows=20 loops=1)
                  ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.005..0.991 rows=1356 loops=20)
                        Index Cond: (state_group = state.state_group)
```

to

```
    Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=24.194..52.834 rows=27112 loops=1)
      ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=24.130..24.138 rows=20 loops=1)
            Group Key: state.state_group
            ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=23.887..24.113 rows=20 loops=1)
      ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.016..1.159 rows=1356 loops=20)
            Index Cond: (state_group = state.state_group)
```

This cuts the execution time from ~190ms to ~130ms, i.e. a reduction
of ~30%.

The full plans are visualised at https://explain.depesz.com/s/WpbT and
https://explain.depesz.com/s/KlEk

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-06-01 15:23:43 +01:00
Dagfinn Ilmari Mannsåker 2dc430d36e
Use upsert when inserting read receipts (#7607)
Fixes #7469

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-06-01 10:53:06 +01:00
Andrew Morgan 0a6e837aaa
Fix incorrect placeholder syntax in database prepartion code (#7575)
We were using `logger` syntax which isn't supported by `Exception`s.
2020-05-27 16:26:59 +01:00
Richard van der Hoff edd9a7214c
Replace device_27_unique_idx bg update with a fg one (#7562)
The bg update never managed to complete, because it kept being interrupted by
transactions which want to take a lock.

Just doing it in the foreground isn't that bad, and is a good deal simpler.
2020-05-26 11:43:17 +01:00
Richard van der Hoff d14c4d6b6d
Simplify reap_monthly_active_users (#7558)
we can use `make_in_list_sql_clause` rather than doing our own half-baked
equivalent, which has the benefit of working just fine with empty lists.

(This has quite a lot of tests, so I think it's pretty safe)
2020-05-23 01:20:10 +01:00
Richard van der Hoff f4269694ce
Optimise some references to hs.config (#7546)
These are surprisingly expensive, and we only really need to do them at startup.
2020-05-22 21:47:07 +01:00
Erik Johnston e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Erik Johnston 1531b214fc
Add ability to wait for replication streams (#7542)
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
Brendan Abolivier d1ae1015ec
Retry to sync out of sync device lists (#7453)
When a call to `user_device_resync` fails, we don't currently mark the remote user's device list as out of sync, nor do we retry to sync it.

https://github.com/matrix-org/synapse/pull/6776 introduced some code infrastructure to mark device lists as stale/out of sync.

This commit uses that code infrastructure to mark device lists as out of sync if processing an incoming device list update makes the device handler realise that the device list is out of sync, but we can't resync right now.

It also adds a looping call to retry all failed resync every 30s. This shouldn't cause too much spam in the logs as this commit also removes the "Failed to handle device list update for..." warning logs when catching `NotRetryingDestination`.

Fixes #7418
2020-05-21 17:41:12 +02:00
Erik Johnston f6f92845f8
Fix bug in persist events when dealing with non member types. (#7548)
`_is_server_still_joined` will throw if it is given state updates with non-user ID state keys with local user leaves. This is actually rarely a problem since local leaves almost always get persisted by themselves.

(I discovered this on a branch that was otherwise broken, so I haven't seen this in the wild)
2020-05-21 13:20:10 +01:00
Richard van der Hoff 4d1afb1dfe
Merge pull request #7519 from matrix-org/rav/kill_py2_code
Kill off some old python 2 code
2020-05-18 10:45:30 +01:00
Richard van der Hoff e6027562e2 remove `builtins.buffer` code from storage code
this is no longer needed on python 3
2020-05-15 19:37:41 +01:00
Richard van der Hoff 65902e08c3 remove to_ascii
this is a no-op on python 3.
2020-05-15 19:12:03 +01:00
Richard van der Hoff 08fa96f030 Remove `exception_to_unicode`
this is a no-op on python 3.
2020-05-15 19:07:24 +01:00
Richard van der Hoff 6c1f7c722f
Fix limit logic for AccountDataStream (#7384)
Make sure that the AccountDataStream presents complete updates, in the right
order.

This is much the same fix as #7337 and #7358, but applied to a different stream.
2020-05-15 19:03:25 +01:00
Andrew Morgan 34a43f0084 Fix a couple of small typos 2020-05-15 18:54:32 +01:00
Erik Johnston 03aff4c75e
Add a worker store for search insertion. (#7516)
This is required as both event persistence and the background update needs access to this function. It should be perfectly safe for two workers to write to that table at the same time.
2020-05-15 17:22:47 +01:00
Andrew Morgan 16090a077f
Prevent 0-member/null room_version rooms from appearing in group room queries (#7465) 2020-05-15 17:17:42 +01:00
Erik Johnston 1f36ff69e8
Move event stream handling out of slave store. (#7491)
This allows us to have the logic on both master and workers, which is necessary to move event persistence off master.

We also combine the instantiation of ID generators from DataStore and slave stores to the base worker stores. This allows us to select which process writes events independently of the master/worker splits.
2020-05-15 16:43:59 +01:00
Andrew Morgan 86614e251f
Fix a small typo in the arguments of simple_update in update_remote_profile_cache (#7511) 2020-05-15 16:17:12 +01:00
Richard van der Hoff dede23ff1e Synapse 1.13.0rc2 (2020-05-14)
==============================
 
 Bugfixes
 --------
 
 - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
 - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
 
 Internal Changes
 ----------------
 
 - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl69IQ8ACgkQOSor00I9
 eP87lAf8DK+v6cs2U0BoD5opzQ7ZazJT6JYTmnMBaTzHU6Wx20V2ttkF7Vpwm3WU
 Zsz0048tdYtHFyYBQ1kF5RNIBBJwV8SA/QUcPkR7FVpwZMLR2q4aJn0EE7kC9OMf
 tYsmdbHeBdyfLXpXzazxWlgHquLyEIt52ykAcCphjx/Jl2fAExFEhtfsxpECoJ2f
 8Dqhjg3WFjd6QWU6AFkElbwHUYCdIWdJOcsC8N1p8OvBmDz5QXv/RlYipHE00Cpx
 QQQOgEjdRc6dlz2mbetMklnfII3p2kO9bzNdmEpOzT0Zt7nFaGdntW4I1QA0yJfa
 gows9bYMzhqYk7YSiyTYOZ4qyavVtw==
 =N/zZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.13.0rc2' into develop

Synapse 1.13.0rc2 (2020-05-14)
==============================

Bugfixes
--------

- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))

Internal Changes
----------------

- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
2020-05-14 11:46:38 +01:00