2c6a7dfcbf
Use attempt_to_set_autocommit everywhere. ( #16615 )
...
To avoid asserting the type of the database connection.
2023-11-09 16:19:42 -05:00
cfb6d38c47
Remove remaining usage of cursor_to_dict. ( #16564 )
2023-10-31 13:13:28 -04:00
12611bfcdd
Add support for pydantic v2 via pydantic.v1 compat module ( #16332 )
...
While maintaining support with pydantic v1.
2023-09-25 15:19:08 +00:00
aa483cb4c9
Update ruff config ( #16283 )
...
Enable additional checks & clean-up unneeded configuration.
2023-09-08 11:24:36 -04:00
6525fd65ee
Log the details of background update failures ( #16212 )
2023-09-01 12:41:56 +01:00
0377cb4fab
Override global statement timeout when creating indexes in Postgres ( #16085 )
2023-08-17 10:30:02 +01:00
e55a9b3e41
Fix downgrading to previous version of Synapse ( #15907 )
...
We do this by marking the constraint as deferrable.
2023-07-10 16:24:42 +01:00
95a96b21eb
Add foreign key constraint to `event_forward_extremities`. ( #15751 )
2023-07-05 09:43:19 +00:00
d162aecaac
Quick & dirty metric for background update status ( #15740 )
...
* Quick & dirty metric for background update status
* Changelog
* Remove debug
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
* Actually write to _aborted
---------
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
2023-06-07 17:12:23 +00:00
2ad91ec628
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} ( #15597 )
...
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.
For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.
Each table is updated as a separate schema delta to avoid
deadlocks between them.
For SQLite we simply rebuild the table & copy the data.
2023-05-26 13:16:08 -04:00
808105bd31
Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} ( #15437 )" ( #15580 )
...
This reverts commit a7b3e9ce65
2023-05-12 11:38:16 +01:00
a7b3e9ce65
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} ( #15437 )
...
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.
For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.
For SQLite we simply rebuild the table & copy the data.
2023-05-03 07:49:03 -04:00
373c485d8c
Handle half-created indices in receipts index background update ( #14650 )
...
When Synapse is terminated while running the background update to create
the `receipts_graph` or `receipts_linearized` indexes, the indexes may
be successfully created (or marked as invalid on postgres) while the
background update remains unfinished. When Synapse next starts up, the
background update will fail because the index already exists, or exists
but is invalid on postgres.
Use the existing code to create indices in background updates, since it
handles these edge cases.
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-12-09 23:02:11 +00:00
fff9b955fa
Generate separate snapshots for logical databases ( #13792 )
...
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
2022-09-20 14:14:12 +01:00
b2b0c85279
Support providing an index predicate for upserts. ( #13822 )
...
This is useful to upsert against a table which has a unique
partial index while avoiding conflicts.
2022-09-15 18:28:48 +00:00
f2d12ccabe
Use partial indices on SQLIte. ( #13802 )
...
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.
This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
2022-09-14 12:01:42 -04:00
53b77b203a
Replace noop background updates with DELETE. ( #12954 )
...
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
2022-06-13 14:06:27 -04:00
66a5f6c400
Add a unique index to `state_group_edges` to prevent duplicates being accidentally introduced and the consequential impact to performance. ( #12687 )
2022-05-19 14:16:49 +01:00
635f0d916b
Do not keep going if there are 5 back-to-back background update failures. ( #12781 )
2022-05-18 13:57:59 +00:00
aec69d2481
Another batch of type annotations ( #12726 )
2022-05-13 12:35:31 +01:00
ef3619e61d
Add config settings for background update parameters ( #11980 )
2022-03-11 10:46:45 -08:00
26211fec24
Fix a bug in background updates wherein background updates are never run using the default batch size ( #12157 )
2022-03-07 09:44:33 -08:00
d08ef6f155
Make background updates controllable via a plugin ( #11306 )
...
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-11-29 17:57:06 +01:00
7f9841bdec
Lower minumum batch size to 1 for background updates ( #11422 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-24 19:21:44 +00:00
ea20937084
Add an admin API to run background jobs. ( #11352 )
...
Instead of having admins poke into the database directly.
Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
4ee71b9637
Add some background update admin APIs ( #11263 )
...
Fixes #11259
2021-11-08 16:08:02 +00:00
bdfde6dca1
Use inline type hints in `http/federation/`, `storage/` and `util/` ( #10381 )
2021-07-15 12:46:54 -04:00
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
b7748d3c00
Import HomeServer from the proper module. ( #9665 )
2021-03-23 07:12:48 -04:00
0a00b7ff14
Update black, and run auto formatting over the codebase ( #9381 )
...
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md
2021-02-16 22:32:34 +00:00
637282bb50
Add additional type hints to the storage module. ( #8980 )
2020-12-30 08:09:53 -05:00
c619253db8
Stop sub-classing object ( #8249 )
2020-09-04 06:54:56 -04:00
37db6252b7
Convert additional databases to async/await part 3 ( #8201 )
2020-09-01 11:04:17 -04:00
30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) ( #8168 )
2020-08-27 07:41:01 -04:00
dbc630a628
Use the JSON encoder without whitespace in more places. ( #8124 )
2020-08-20 10:32:33 -04:00
894dae74fe
Convert misc database code to async ( #8087 )
2020-08-14 07:24:26 -04:00
a7bdf98d01
Rename database classes to make some sense ( #8033 )
2020-08-05 21:38:57 +01:00
f460da6031
Consistently use `db_to_json` to convert from database values to JSON objects. ( #7849 )
2020-07-16 11:32:19 -04:00
fcc2de7a0c
Update docstring per review comments
2020-04-03 10:51:32 +01:00
af47264b78
review comment
2020-04-02 12:04:55 +01:00
7b608cf468
Only run one background update at a time
2020-03-31 17:43:58 +01:00
b4c2234232
Make do_next_background_update return a bool
...
returning a None or an int that we don't use is confusing.
2020-03-31 17:43:58 +01:00
26d17b9bdc
Make `has_completed_background_updates` async
...
(Almost) everywhere that uses it is happy with an awaitable.
2020-03-31 17:43:58 +01:00
cfe8c8ab8e
Remove unused `start_background_update`
...
This was only used in a unit test, so let's just inline it in the test.
2020-03-31 17:24:06 +01:00
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 ( #6957 )
...
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
85db7f73be
Add a background update to clear tombstoned rooms from the directory ( #6648 )
...
* Add a background update to clear tombstoned rooms from the directory
* use the ABC metaclass
2020-01-07 14:18:43 +00:00
9824a39d80
Async/await for background updates ( #6647 )
...
so that bg update routines can be async
2020-01-07 14:12:42 +00:00
d537be1ebd
Pass Database into the data store
2019-12-06 15:49:44 +00:00
4a33a6dd19
Move background update handling out of store
2019-12-05 11:11:26 +00:00
756d4942f5
Move DB pool and helper functions into dedicated Database class
2019-12-05 10:46:37 +00:00
Patrick Cloke