637282bb50
Add additional type hints to the storage module. ( #8980 )
2020-12-30 08:09:53 -05:00
c619253db8
Stop sub-classing object ( #8249 )
2020-09-04 06:54:56 -04:00
37db6252b7
Convert additional databases to async/await part 3 ( #8201 )
2020-09-01 11:04:17 -04:00
30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) ( #8168 )
2020-08-27 07:41:01 -04:00
dbc630a628
Use the JSON encoder without whitespace in more places. ( #8124 )
2020-08-20 10:32:33 -04:00
894dae74fe
Convert misc database code to async ( #8087 )
2020-08-14 07:24:26 -04:00
a7bdf98d01
Rename database classes to make some sense ( #8033 )
2020-08-05 21:38:57 +01:00
f460da6031
Consistently use `db_to_json` to convert from database values to JSON objects. ( #7849 )
2020-07-16 11:32:19 -04:00
fcc2de7a0c
Update docstring per review comments
2020-04-03 10:51:32 +01:00
af47264b78
review comment
2020-04-02 12:04:55 +01:00
7b608cf468
Only run one background update at a time
2020-03-31 17:43:58 +01:00
b4c2234232
Make do_next_background_update return a bool
...
returning a None or an int that we don't use is confusing.
2020-03-31 17:43:58 +01:00
26d17b9bdc
Make `has_completed_background_updates` async
...
(Almost) everywhere that uses it is happy with an awaitable.
2020-03-31 17:43:58 +01:00
cfe8c8ab8e
Remove unused `start_background_update`
...
This was only used in a unit test, so let's just inline it in the test.
2020-03-31 17:24:06 +01:00
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 ( #6957 )
...
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
85db7f73be
Add a background update to clear tombstoned rooms from the directory ( #6648 )
...
* Add a background update to clear tombstoned rooms from the directory
* use the ABC metaclass
2020-01-07 14:18:43 +00:00
9824a39d80
Async/await for background updates ( #6647 )
...
so that bg update routines can be async
2020-01-07 14:12:42 +00:00
d537be1ebd
Pass Database into the data store
2019-12-06 15:49:44 +00:00
4a33a6dd19
Move background update handling out of store
2019-12-05 11:11:26 +00:00
756d4942f5
Move DB pool and helper functions into dedicated Database class
2019-12-05 10:46:37 +00:00
ee86abb2d6
Remove underscore from SQLBaseStore functions
2019-12-04 16:23:43 +00:00
14504ad573
Add CI for synapse_port_db ( #6140 )
...
This adds:
* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)
as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:
1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)
Step `2` is done via a new script located at `scripts-dev/update_database`.
The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
4fb3c129aa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2
2019-09-25 17:53:13 +01:00
39b50ad42a
Review comments
2019-09-25 17:22:33 +01:00
2135c198d1
Add has_completed_background_update
...
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
f02f14e09a
Fix logging
2019-09-24 14:39:07 +01:00
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
7efd1d87c2
Run black on the rest of the storage module ( #4996 )
2019-04-03 10:07:29 +01:00
282c97327f
Migrate the user directory initial population to a background task ( #4864 )
2019-03-19 04:50:24 +11:00
25d64a846a
Fix typos
2019-01-22 11:00:04 +00:00
667fba68f3
Run things as background processes
...
This fixes #3518 , and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
34194aaff7
Don't create event_search index on sqlite
...
... because the table is virtual
2017-05-11 12:46:55 +01:00
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
c84770b877
Fix bgupdate error if index already exists ( #2167 )
...
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135 .
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
994d7ae7c5
Remove broken use of clock.call_later
...
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.
We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
c94de0ab60
Add WHERE clause support to index creation
2016-09-12 16:55:01 +01:00
5beda10bbd
Reindex state_groups_state after pruning
2016-09-08 16:18:01 +01:00
955ef1f06c
fix: defer.returnValue takes one argument
2016-07-25 16:04:45 +01:00
2ee4c9ee02
background updates: fix assert again
2016-07-25 16:01:46 +01:00
Patrick Cloke