a319cb1dd1
Change device list streams to have one row per ID ( #7010 )
...
* Add 'device_lists_outbound_pokes' as extra table.
This makes sure we check all the relevant tables to get the current max
stream ID.
Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.
* Change device lists stream to have one row per id.
This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.
* Change device list replication to match new semantics.
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
* Newsfile
* Remove handling of multiple rows per ID
* Fix worker handling
* Comments from review
2020-03-19 11:36:53 +00:00
4a17a647a9
Improve get auth chain difference algorithm. ( #7095 )
...
It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.
Instead, lets try and fetch the auth chains at the same time and
calculate the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.
2020-03-18 16:46:41 +00:00
1d66dce83e
Break down monthly active users by appservice_id ( #7030 )
...
* Break down monthly active users by appservice_id and emit via prometheus.
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2020-03-06 18:14:19 +00:00
f70f44abc7
Remove handling of multiple rows per ID
2020-02-28 11:45:35 +00:00
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 ( #6957 )
...
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
43b2be9764
Replace _event_dict_property with DictProperty
...
this amounts to the same thing, but replaces `_event_dict` with `_dict`, and
removes some of the function layers generated by `property`.
2020-02-14 16:42:37 +00:00
b9391c9575
Add typing to SyncHandler ( #6821 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-02-03 18:05:44 +00:00
5a246611e3
Type defintions for use in refactoring for redaction changes ( #6803 )
...
* Bump signedjson to 1.1
... so that we can use the type definitions
* Fix breakage caused by upgrade to signedjson 1.1
Thanks, @illicitonion...
2020-01-30 11:25:59 +00:00
8df862e45d
Add `rooms.room_version` column ( #6729 )
...
This is so that we don't have to rely on pulling it out from `current_state_events` table.
2020-01-27 14:30:57 +00:00
fa4d609e20
Make 'event.redacts' never raise. ( #6771 )
...
There are quite a few places that we assume that a redaction event has a
corresponding `redacts` key, which is not always the case. So lets
cheekily make it so that event.redacts just returns None instead.
2020-01-23 15:19:03 +00:00
28c98e51ff
Add `local_current_membership` table ( #6655 )
...
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
2020-01-15 14:59:33 +00:00
d2906fe666
Allow admin users to create or modify users without a shared secret ( #6495 )
...
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-01-09 13:31:00 +00:00
d20c346544
port BackgroundUpdateTestCase to HomeserverTestCase ( #6653 )
2020-01-07 14:09:07 +00:00
dc41fbf0dd
Remove unused get_prev_events_and_hashes_for_room
2020-01-06 13:45:33 +00:00
5a04781643
rename get_prev_events_for_room to get_prev_events_and_hashes_for_room
...
... to make way for a new method which just returns the event ids
2020-01-06 13:45:33 +00:00
75d8f26ac8
Split state groups into a separate data store ( #6296 )
2019-12-20 10:48:24 +00:00
2284eb3a53
Add database config class ( #6513 )
...
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
40eda84933
Fix race which caused deleted devices to reappear ( #6514 )
...
Stop the `update_client_ips` background job from recreating deleted devices.
2019-12-10 16:22:29 +00:00
852f80d8a6
Fixup tests
2019-12-06 16:02:50 +00:00
9a4fb457cf
Change DataStores to accept 'database' param.
2019-12-06 13:30:06 +00:00
4a33a6dd19
Move background update handling out of store
2019-12-05 11:11:26 +00:00
756d4942f5
Move DB pool and helper functions into dedicated Database class
2019-12-05 10:46:37 +00:00
ee86abb2d6
Remove underscore from SQLBaseStore functions
2019-12-04 16:23:43 +00:00
0f87b912ab
Implementation of MSC2314 ( #6176 )
2019-11-28 08:54:07 +11:00
0d27aba900
add etag and count to key backup endpoints ( #5858 )
2019-11-27 16:14:44 -05:00
6a0092d371
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history
2019-11-04 13:29:35 +00:00
6f4bc6d01d
Merge branch 'develop' into cross-signing_federation
2019-10-31 22:38:21 -04:00
cd581338cf
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history
2019-10-31 15:19:26 +00:00
bb6cec27a5
rename get_devices_by_remote to get_device_updates_by_remote
2019-10-30 14:57:34 -04:00
7c8c97e635
Split purge API into events vs state
2019-10-30 15:23:37 +00:00
69f0054ce6
Port to use state storage
2019-10-30 14:46:54 +00:00
7d7eac61be
Merge branch 'develop' into cross-signing_federation
2019-10-30 10:17:10 -04:00
ec6de1cc7d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store
2019-10-30 13:37:04 +00:00
326b3dace7
Make ObservableDeferred.observe() always return deferred.
...
This makes it easier to use in an async/await world.
Also fixes a bug where cache descriptors would occaisonally return a raw
value rather than a deferred.
2019-10-30 11:35:46 +00:00
4697c0de0b
remove unneeded imports
2019-10-25 10:47:02 -04:00
4cf3a30a20
switch to using HomeserverTestCase
2019-10-25 10:42:07 -04:00
c40d7244f8
Merge branch 'develop' into cross-signing_federation
2019-10-24 22:31:25 -04:00
29a0bc5637
remove some unnecessary lines
2019-10-24 21:43:02 -04:00
848cd388d9
delete keys when deleting backups
2019-10-24 21:21:51 -04:00
3ca4c7c516
Use new EventPersistenceStore
2019-10-23 16:15:03 +01:00
3e3f9b684e
fix unit test
2019-10-22 22:26:30 -04:00
c66a06ac6b
Move storage classes into a main "data store".
...
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
...
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
a0d0ba7862
Fix MAU reaping where reserved users are specified. ( #6168 )
2019-10-11 09:38:26 +01:00
4908fb3b30
make storage layer in charge of interpreting the device key data
2019-10-10 19:15:30 -04:00
bc244627ac
Fix postgres unit tests
2019-10-10 15:37:53 +01:00
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
...
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
f44f1d2e83
Fix errors storing large retry intervals.
...
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.
To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
ce7a3e7e27
Fix fetching censored redactions from DB
...
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
034db2ba21
Fix dummy event insertion consent bug ( #6053 )
...
Fixes #5905
2019-09-26 11:47:53 +01:00
Erik Johnston