Commit Graph

92 Commits (41ad35b5235ad9ed8a1b8889287ae840ee3373bd)

Author SHA1 Message Date
Hubert Chathi cfdb84422d make black happy 2019-10-22 19:06:06 -04:00
Hubert Chathi a1aaf3eea6 don't crash if the user doesn't have cross-signing keys 2019-10-22 19:04:37 -04:00
Hubert Chathi 8d3542a64e implement federation parts of cross-signing 2019-10-22 19:04:35 -04:00
Hubert Chathi cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Erik Johnston b161786c14 Replace IN usage with helper funcs 2019-10-10 13:15:49 +01:00
Brendan Abolivier cef9f6753e Move devices's bg updates to a dedicated store 2019-10-07 17:34:20 +01:00
Hubert Chathi 8e86f5b65c Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-09-07 13:20:34 -04:00
Jorik Schellekens 5ade977d08 Opentracing context cannot be none 2019-09-05 15:06:13 +01:00
Hubert Chathi faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Jorik Schellekens a90d16dabc
Opentrace device lists (#5853)
Trace device list changes.
2019-09-03 10:21:30 +01:00
Andrew Morgan 4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Hubert Chathi e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
Hubert Chathi 72d296a7f3 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-08-28 17:33:44 -07:00
Jorik Schellekens 8767b63a82
Propagate opentracing contexts through EDUs (#5852)
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Hubert Chathi 7c3abc6572 apply PR review suggestions 2019-08-21 13:19:35 -07:00
Hubert Chathi f63ba7a795
Cross-signing [1/4] -- hidden devices (#5759)
* allow devices to be marked as "hidden"

This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
Hubert Chathi 336c546d6a Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-08-01 16:31:40 -04:00
Hubert Chathi d78a4fe156 don't need to return the hidden column any more 2019-08-01 02:16:09 -04:00
Hubert Chathi 73b26f827c really fix queries to work with Postgres
(by going back to not using SQL directly)
2019-07-31 18:37:05 -04:00
Hubert Chathi 430ea08186 PostgreSQL, Y U no like? 2019-07-31 15:38:11 -04:00
Hubert Chathi 185188be03 remove extra SQL query param 2019-07-31 15:18:15 -04:00
Hubert Chathi c1f0a5636c Merge branch 'develop' into cross-signing_hidden 2019-07-30 23:33:06 -04:00
Hubert Chathi bc95890822 Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-07-30 23:21:26 -04:00
Hubert Chathi 781ade836b apply changes from PR review 2019-07-30 23:09:50 -04:00
Hubert Chathi c659b9f94f allow uploading keys for cross-signing 2019-07-25 11:08:24 -04:00
Hubert Chathi d1c7c2a98a allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-07-24 23:21:52 -04:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Erik Johnston 729f5a4fb6 Review comments 2019-06-27 16:06:23 +01:00
Erik Johnston f335e77d53 Use batch_iter and correct docstring 2019-06-26 19:39:46 +01:00
Erik Johnston 806a06daf2 Rename get_users_whose_devices_changed 2019-06-26 19:39:19 +01:00
Erik Johnston a2f6d31a63 Refactor get_user_ids_changed to pull less from DB
When a client asks for users whose devices have changed since a token we
used to pull *all* users from the database since the token, which could
easily be thousands of rows for old tokens.

This PR changes this to only check for changes for users the client is
actually interested in.

Fixes #5553
2019-06-26 12:03:44 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Andrew Morgan 2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156)
fixes #5153
2019-06-06 23:54:00 +01:00
Amber Brown 7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Erik Johnston a84b8d56c2 Fixup slave stores 2019-03-04 18:04:57 +00:00
Richard van der Hoff 350f654e7b Add unique indexes to a couple of tables
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
2018-11-02 10:36:13 +00:00
Richard van der Hoff 50e328d1e7 Remove redundant database locks for device list updates
We can rely on the application-level per-user linearizer.
2018-11-02 10:36:13 +00:00
Erik Johnston 88e5ffe6fe Deduplicate device updates sent over replication
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
2018-10-29 17:34:34 +00:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff 03751a6420 Fix some looping_call calls which were broken in #3604
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Richard van der Hoff 371da42ae4 Wrap a number of things that run in the background
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Matthew Hodgson 9e40834f74 yes, we do need to invalidate the device_id_exists_cache when deleting a remote device 2018-07-19 11:15:10 +01:00
Matthew Hodgson 12ec58301f shift to using an explicit deleted flag on m.device_list_update EDUs
and generally make it work.
2018-07-12 11:39:43 +01:00
Matthew Hodgson 5797f5542b WIP to announce deleted devices over federation
Previously we queued up the poke correctly when the device was deleted,
but then the actual EDU wouldn't get sent, as the device was no longer known.
Instead, we now send EDUs for deleted devices too if there's a poke for them.
2018-07-12 01:32:39 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Adrian Tschira 933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Richard van der Hoff 6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston 64ed74c01e When pruning, delete from device_lists_outbound_last_success 2017-06-07 11:20:47 +01:00