3d12d97415
Track DB scheduling delay per-request
...
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
8615f19d20
rework runInteraction in terms of runConnection
...
... so that we can share the code
2018-01-16 17:08:29 +00:00
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
3241c7aac3
untested WIP but might actually work
2017-11-29 18:27:05 +00:00
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error
...
Fix error on sqlite 3.7
2017-11-27 12:25:48 +00:00
63ccaa5873
Avoid retrying forever on IntegrityError
2017-11-27 12:00:07 +00:00
6b48b3e277
fix sql fails
2017-11-22 18:06:24 +00:00
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
77a1227870
Fix broken ref to IntegrityError
2017-11-16 16:03:38 +00:00
10aaa1bc15
_simple_upsert: retry on IntegrityError
...
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
2017-11-16 15:30:15 +00:00
cdc9e50a5d
Cleanup in _simple_upsert_txn
...
Bail out early to reduce indentation
2017-11-16 15:29:10 +00:00
4dd1bfa8c1
Revert "Revert "move _state_group_cache to statestore""
...
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.
This reverts commit ab335edb02
2017-11-14 11:43:58 +00:00
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
ab335edb02
Revert "move _state_group_cache to statestore"
...
This reverts commit f5cf3638e9
2017-11-13 10:05:33 +00:00
f5cf3638e9
move _state_group_cache to statestore
...
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
97c544f91f
Add _simple_update
2017-08-25 11:11:37 +01:00
b5e8d529e6
Define CACHE_SIZE_FACTOR once
2017-07-04 09:56:44 +01:00
c62c480dc6
Merge pull request #2259 from matrix-org/erikj/fix_state_woes
...
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
197bd126f0
Fix bug where state_group tables got corrupted
...
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.
We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
21e255a8f1
Split the table in two
2017-06-01 14:50:46 +01:00
bfbc907cec
Prefill state caches
2017-05-15 15:11:13 +01:00
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
bbeeb97f75
Implement _simple_delete_many_txn, use it to delete devices
...
(But this doesn't implement the same for deleting access tokens or e2e keys.
Also respond to code review.
2017-03-13 17:53:23 +00:00
6ad71cc29d
Remove spurious SQL logging ( #1972 )
...
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
2017-03-08 18:00:44 +00:00
b84907bdbb
Intern table column names once
2017-02-28 14:38:16 +00:00
1a4f8022e6
Strip newlines from SQL queries
2017-02-23 11:15:31 +00:00
2849d3f29d
admin,storage: added more administrator functionalities
...
administrators can now:
- Set displayname of users
- Update user avatars
- Search for users by user_id
- Browse all users in a paginated API
- Reset user passwords
- Deactivate users
Helpers for doing paginated queries has also been added to storage
Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
d906206049
Increase state_group_cache_size
2017-01-17 11:31:08 +00:00
dd52d4de4c
Limit number of entries to prefill from cache
...
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.
In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
e8b1d2a452
Refactor test_filter to use real DataStore
...
* add tests for filter api errors
2016-10-18 12:17:38 -05:00
ba214a5e32
Remove lru option
2016-08-19 14:17:11 +01:00
949629291c
Do it in storage function
2016-08-16 17:05:34 +01:00
a2427981b7
Use cached get_user_by_access_token in slaves
2016-08-16 11:24:32 +01:00
89e786bd85
Doc get_next() context manager usage
2016-08-15 13:45:26 +01:00
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
99bbd90b0d
Always run txn.after_callbacks
2016-08-15 09:45:44 +01:00
053e83dafb
More doc-comments
...
Fix some more comments on some things
2016-07-20 16:40:28 +01:00
f328d95cef
Feature: Add deactivate account admin API
...
Allows server admins to "deactivate" accounts, which:
- Revokes all access tokens
- Removes all threepids
- Removes password
The API is a POST to `/admin/deactivate/<user_id>`
2016-06-30 15:40:58 +01:00
80f34d7b57
Fix setting the _clock in SQLBaseStore
2016-06-02 15:23:56 +01:00
d4503e25ed
Make deleting push actions more efficient
...
There's no index on received_ts, so manually binary search using the stream_ordering index, and only update it once an hour.
2016-05-20 17:56:10 +01:00
0c11c1be88
Spelling
2016-05-13 14:42:25 +01:00
13d37c3c56
Fixup add_pusher
2016-05-13 11:25:02 +01:00
7e2f971c08
Remove some unused functions ( #711 )
...
* Remove some unused functions
* get_room_events_stream is only used in tests
* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
a1e0d316ea
Move _get_cache_dict into the SQLBaseStore
2016-04-06 13:05:19 +01:00
acdfef7b14
Intern all the things
2016-03-23 16:25:54 +00:00
b591277620
Make stateGroupCache honour CACHE_SIZE_FACTOR
2016-03-22 10:32:50 +00:00
a1cf9e3bf3
Add a stream for push rule updates
2016-03-01 18:16:37 +00:00
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
aea5da0ef6
Guard against empty iterables
2016-01-25 15:59:29 +00:00
1ebf5e3d03
Correct docstring
2016-01-25 15:53:36 +00:00
ddd25def01
Implement a _simple_select_many_batch
2016-01-25 13:36:02 +00:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
6a5ff5f223
Track the time spent in the database per request.
...
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
c85c912562
Add basic full text search impl.
2015-10-09 15:48:31 +01:00
396834f1c0
synapse/storage/_base.py:_simple_max_id was unused
2015-09-23 10:30:38 +01:00
1d9036aff2
synapse/storage/_base.py:_simple_delete was unused
2015-09-23 10:30:25 +01:00
1ee3d26432
synapse/storage/_base.py:_simple_selectupdate_one was unused
2015-09-23 10:30:03 +01:00
7dd4f79c49
synapse/storage/_base.py:_execute_and_decode was unused
2015-09-22 18:37:07 +01:00
527d95dea0
synapse/storage/_base.py:Table was unused
2015-09-22 18:14:15 +01:00
b62c1395d6
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-09-01 13:11:55 +01:00
cd800ad99a
Lower size of 'stateGroupCache' now that we have data from matrix.org to support doing so
2015-09-01 10:09:03 +01:00
cecbd636e9
/tokenrefresh POST endpoint
...
This allows refresh tokens to be exchanged for (access_token,
refresh_token).
It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
2df8dd9b37
Move all the caches into their own package, synapse.util.caches
2015-08-11 18:00:59 +01:00
53a817518b
Comments
2015-08-11 11:40:40 +01:00
6eaa116867
Comment
2015-08-11 11:35:24 +01:00
4762c276cb
Docs
2015-08-11 11:33:41 +01:00
bb0a475c30
Comments
2015-08-10 14:27:38 +01:00
7ce71f2ffc
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 13:47:51 +01:00
86eaaa885b
Rename keyargs to args in CacheDescriptor
2015-08-10 13:44:44 +01:00
e0b6e49466
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 10:39:22 +01:00
2cd6cb9f65
Rename keyargs to args in Cache
2015-08-10 10:38:47 +01:00
7dec0b2bee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:28:39 +01:00
2352974aab
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:26:54 +01:00
9c5385b53a
s/observed/observer/
2015-08-07 19:26:38 +01:00
ffab798a38
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:18:47 +01:00
62126c996c
Propogate stale cache errors to calling functions
2015-08-07 19:17:58 +01:00
20addfa358
Change Cache to not use *args in its interface
2015-08-07 18:32:47 +01:00
0211890134
Implement a CacheListDescriptor
2015-08-07 18:14:49 +01:00
b8e386db59
Change Cache to not use *args in its interface
2015-08-07 11:52:21 +01:00
1d08bf7c17
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 14:03:15 +01:00
63b1eaf32c
Docs
2015-08-06 14:02:50 +01:00
b811c98574
Remove failed deferreds from cache
2015-08-06 14:01:27 +01:00
433314cc34
Re-implement DEBUG_CACHES flag
2015-08-06 14:01:05 +01:00
f596ff402e
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 13:37:56 +01:00
2efb93af52
Merge branch 'erikj/cached_keyword_args' into erikj/cache_deferreds
2015-08-06 13:35:28 +01:00
953dbd28a7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cached_keyword_args
2015-08-06 13:35:03 +01:00
7eea3e356f
Make @cached cache deferreds rather than the deferreds' values
2015-08-06 13:33:34 +01:00
3e1b77efc2
Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 16:45:56 +01:00
b52b4a84ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 15:41:20 +01:00
a89559d797
Use LRU cache by default
2015-08-05 15:39:47 +01:00
07507643cb
Use dictionary cache to do group -> state fetching
2015-08-05 15:11:42 +01:00
4d6cb8814e
Speed up event filtering (for ACL) logic
2015-08-04 09:32:23 +01:00
39e21ea51c
Add support for using keyword arguments with cached functions
2015-07-27 13:57:29 +01:00
80a61330ee
Add basic storage functions for handling of receipts
2015-07-01 17:19:12 +01:00
d8866d7277
Caches should be bound to instances.
...
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
2015-06-03 14:45:17 +01:00
4429e720ae
Merge branch 'master' of github.com:matrix-org/synapse into develop
2015-05-22 10:33:00 +01:00
53447e9cd3
Add caches for things requested by the pushers
2015-05-21 16:41:39 +01:00
d61ce3f670
Add a cache for get_current_state with state_key
2015-05-21 16:41:39 +01:00
65878a2319
Remove unused metric
2015-05-18 14:06:30 +01:00
c71176858b
Newline, remove debug logging
2015-05-18 10:11:14 +01:00
c3b37abdfd
PEP8
2015-05-15 16:59:58 +01:00
aa32bd38e4
Add a wait
2015-05-15 11:35:04 +01:00
a2c4f3f150
Fix daedlock
2015-05-15 10:54:04 +01:00
1d566edb81
Remove race condition
2015-05-14 16:54:35 +01:00
142934084a
Count and loop
2015-05-14 15:40:21 +01:00
7cd6a6f6cf
Awful idea for speeding up fetching of events
2015-05-14 15:34:02 +01:00
386b7330d2
Move from _base to events
2015-05-14 14:45:22 +01:00
7d6a1dae31
Jump out early
2015-05-14 14:27:58 +01:00
656223fbd3
Actually, we probably want to run this in a transaction
2015-05-14 14:26:35 +01:00
2f7f8e1c2b
Preemptively jump into a transaction if we ask for get_prev_content
2015-05-14 14:17:36 +01:00
e1e9f0c5b2
loop -> gatherResults
2015-05-14 13:58:49 +01:00
ab78a8926e
Err, we probably want a bigger limit
2015-05-14 13:47:16 +01:00
cdb3757942
Refactor _get_events
2015-05-14 13:31:55 +01:00
36ea26c5c0
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf
2015-05-14 12:01:38 +01:00
7c549dd557
Add ID generator for push_rules_enable to #resolve SYN-378
2015-05-14 11:44:03 +01:00
968b01a91a
Actually use async method
2015-05-13 17:02:46 +01:00
4071f29653
Fetch events from events_id in their own transactions
2015-05-13 16:59:41 +01:00
a988361aea
Typo
2015-05-13 15:44:15 +01:00
8888982db3
Don't insert None
2015-05-13 15:43:32 +01:00
cf706cc6ef
Don't return None
2015-05-13 15:31:25 +01:00
5971d240d4
Limit batch size
2015-05-13 15:26:49 +01:00
ca4f458787
Fetch events in bulk
2015-05-13 15:13:42 +01:00
6edff11a88
Don't fetch redaction and rejection stuff for each event, so we can use index only scan
2015-05-13 14:39:05 +01:00
409bcc76bd
Load events for state group seperately
2015-05-13 11:13:31 +01:00
da6a7bbdde
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context
2015-05-12 13:10:42 +01:00
cd525c0f5a
push_rules table expects an 'id' field
2015-05-11 11:24:01 +01:00
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
ed2584050f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany
2015-05-05 18:15:20 +01:00
31049c4d72
Merge pull request #139 from matrix-org/bugs/SYN-369
...
Fix race with cache invalidation. SYN-369
2015-05-05 17:46:13 +01:00
deb0237166
Add some doc-string
2015-05-05 17:45:11 +01:00
d18f37e026
Collect the invalidate callbacks on the transaction object rather than passing around a separate list
2015-05-05 17:32:21 +01:00
9951542393
Add a comment about the zip(*[zip(sorted(...),...)])
2015-05-05 17:06:55 +01:00
041b6cba61
SYN-369: Add comments to the sequence number logic in the cache
2015-05-05 16:32:44 +01:00
63075118a5
Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access
2015-05-05 16:24:04 +01:00
43c2e8deae
Add support for using executemany
2015-05-05 15:13:25 +01:00
1692dc019d
Don't call 'encode_parameter' no-op
2015-05-05 15:00:30 +01:00
261d809a47
Sequence the modifications to the cache so that selects don't race with inserts
2015-05-05 14:13:50 +01:00
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
fabb7acd45
Fix bug where we reconnected to the database on every query.
2015-05-01 10:24:24 +01:00
Richard van der Hoff