49d6aa1394
Retry on deadlock
2015-04-07 15:28:37 +01:00
0af5f5efaf
Don't use multiple UNIQUE constraints; it will cause deadlocks
2015-04-07 12:08:35 +01:00
c8d3f6486d
Implement or_ignore flag on inserts
2015-04-07 12:06:22 +01:00
304111afd0
Don't use AUTOINCREMENT, use an in memory version
2015-04-07 12:05:36 +01:00
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
a198894bf7
Appease pep8
2015-03-26 11:53:58 +00:00
953e40f9dc
Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one
2015-03-25 19:12:16 +00:00
f173d40a32
Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key
2015-03-25 19:06:05 +00:00
1b988b051b
Store the rejected reason in (Frozen)Event structs
2015-03-25 19:06:05 +00:00
033a517feb
Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file
2015-03-25 19:06:05 +00:00
9ba6487b3f
Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()
2015-03-25 19:05:34 +00:00
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
9e98f1022a
Don't order by rowid
2015-03-24 16:19:01 +00:00
6e7131f02f
Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.
2015-03-23 15:38:56 +00:00
0f86312c4c
Pull out the cache logic from the @cached wrapper into its own class we can reuse
2015-03-20 18:25:42 +00:00
b1022ed8b5
func(*EXPR) is valid Python syntax, really...
2015-03-20 17:47:45 +00:00
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
80cd08c190
PEP8
2015-03-20 16:03:25 +00:00
9517f4da4d
Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement
2015-03-20 16:02:47 +00:00
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
7e282a53a5
Tidy up _simple_... methods
2015-03-20 15:05:10 +00:00
91cb46191d
Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though
2015-03-20 14:59:45 +00:00
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
1489521ee5
Be polite and ensure we use @functools.wraps() when creating a function decorator
2015-03-17 17:19:22 +00:00
be170b1426
Add a metric for the scheduling latency of SQL queries
2015-03-16 17:21:59 +00:00
e75fa8bbbf
Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot
2015-03-12 16:24:51 +00:00
63cb7ece62
Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer
2015-03-12 16:24:51 +00:00
f1fbe3e09f
Rename TimerMetric to DistributionMetric; as it could count more than just time
2015-03-12 16:24:51 +00:00
642f725fd7
Pretend the 'getEvent' cache is just another cache in the set of all the others for metric
2015-03-12 16:24:51 +00:00
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
fa319a5786
Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely
2015-03-12 16:24:51 +00:00
25187ab674
Collect per-SQL-verb timer stats on query execution time
2015-03-12 16:24:50 +00:00
e1a7e3564f
Delete a couple of TODO markers of monitoring stats now done
2015-03-12 16:24:50 +00:00
8664599af7
Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache
2015-03-12 16:24:50 +00:00
b0cdf097f4
Sprinkle some CacheCounterMetrics around the synapse.storage layer
2015-03-12 16:24:50 +00:00
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
3d73383d18
Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users.
2015-03-02 10:16:24 +00:00
9640510de2
Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased
2015-02-23 18:41:58 +00:00
f53fcbce97
Use cache.pop() instead of a separate membership test + del []
2015-02-23 18:30:45 +00:00
27080698e7
Fix code style warning
2015-02-23 18:19:13 +00:00
a09e59a698
Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing
2015-02-23 16:55:57 +00:00
e76d485e29
Allow @cached-wrapped functions to have a prefill method for setting entries
2015-02-23 15:41:54 +00:00
55022d6ca5
Remove a TODO note
2015-02-19 18:38:09 +00:00
ebc3db295b
Take named arguments to @cached() decorator, add a 'max_entries' limit
2015-02-19 18:36:02 +00:00
077d200342
Move @cached decorator out into synapse.storage._base; add minimal docs
2015-02-19 17:29:39 +00:00
42bc56dad3
Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson
2015-02-11 17:01:38 +00:00
aff892ce79
Fix formatting
2015-02-11 15:02:35 +00:00
f5a70e0d2e
Add a cache for get_event
2015-02-11 15:01:15 +00:00
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
b085fac735
Code-style fixes
2015-02-10 16:30:48 +00:00
fda4422bc9
Fix pyflakes
2015-02-10 14:54:07 +00:00
d7c7efb691
Add performance counters for different stages of loading events
2015-02-10 14:50:53 +00:00
0c4536da8f
Use the transaction 'desc' rather than 'name', increment the txn_ids in
...
txn names
2015-02-09 18:06:31 +00:00
347b497db0
Formatting
2015-02-09 17:57:09 +00:00
3a5ad7dbd5
Performance counters for database transaction names
2015-02-09 17:55:56 +00:00
c4ee4ce93e
Fix typo
2015-02-09 15:00:37 +00:00
66fde49f07
Log database time every 10s and log as percentage
2015-02-09 14:45:15 +00:00
75656712e3
Time how long we're spending on the database thread
2015-02-09 14:22:52 +00:00
2f4cb04f45
Be more specific in naming columns in selects.
2015-01-30 14:48:11 +00:00
e0b7c521cb
Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage
...
Conflicts:
synapse/storage/__init__.py
synapse/storage/schema/delta/v12.sql
2015-01-30 14:08:28 +00:00
6d485dd1c7
unnecessary newlines
2015-01-28 14:48:42 +00:00
fb0928097a
More magic commas (including the place I copied it from...)
2015-01-28 14:48:07 +00:00
0cbb6b0f52
Google doc style
2015-01-28 14:44:41 +00:00
b1b85753d7
Add support for storing rejected events in EventContext and data stores
2015-01-22 15:50:17 +00:00
c06a9063e1
Merge branch 'develop' into pushers
2015-01-13 13:15:51 +00:00
36a2a877e2
Use time.time() instead of time.clock()
2015-01-06 16:34:41 +00:00
52b2c6c9c7
Don't include None's in _get_events_txn
2015-01-06 14:56:57 +00:00
5e23a19204
Merge pull request #28 from matrix-org/erikj-perf
...
Database performance improvements.
2015-01-06 13:33:40 +00:00
adb04b1e57
Update copyright notices
2015-01-06 13:21:39 +00:00
af1c7c7808
PEP8
2015-01-06 13:13:17 +00:00
52d8519008
Don't do batching when getting events.
2015-01-06 13:10:27 +00:00
98933e3db6
Only fetch prev_content when a client is streaming/paginating. Use transactions for event streams.
2015-01-06 13:03:23 +00:00
3e26720e05
Temporarily turn off 'redacted_because' and 'prev_content' keys
2015-01-06 11:26:58 +00:00
f4ea78e9e2
More debug logging
2015-01-06 11:24:18 +00:00
d7e8ea67b3
Reformat
2015-01-06 11:18:02 +00:00
b56730bb6e
Merge branch 'develop' into pushers
...
Conflicts:
synapse/api/errors.py
synapse/server.py
synapse/storage/__init__.py
2014-12-18 15:15:22 +00:00
9728c305a3
after a few rethinks, a working implementation of pushers.
2014-12-18 14:49:22 +00:00
f3788e3c78
Test some ideas that might help performance a bit
2014-12-17 23:37:08 +00:00
5b39cfff69
Don't assume an event exists
2014-12-16 18:25:24 +00:00
882dc8dcab
Persist internal_metadata
2014-12-16 13:17:09 +00:00
c8dd3314d6
Fix bug where we ignored event_edge_hashes table
2014-12-15 13:55:22 +00:00
75085bb4d1
Pyflakes
2014-12-12 14:34:34 +00:00
0b04369238
Fix public room joining by making sure replaces_state never points to itself.
2014-12-11 15:56:06 +00:00
9191292b0f
Fix prev_content
2014-12-11 15:16:55 +00:00
8cdebce470
Fix redactions. Fix 'age' key
2014-12-11 13:25:19 +00:00
02e4c18171
Remove dead code
2014-12-10 18:00:36 +00:00
aae8a37e63
Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor
2014-12-10 13:18:40 +00:00
aa3f66cf7f
Change the way we implement get_events to be less sucky
2014-12-09 13:35:26 +00:00
ba3d1e2fc0
Remove unused import
2014-12-08 12:01:25 +00:00
aed62a3583
track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover.
2014-12-07 02:26:07 +00:00
6630e1b579
Start making more things use EventContext rather than event.*
2014-12-05 16:20:48 +00:00
ec2b5d8c28
Store full JSON of events in db
2014-12-01 16:22:07 +00:00
a46e5ef621
SYN-163: Add an order by rowid to selects.
...
This should fix the bug where the edges of the graph get returned in a
different order than they were inserted in, and so no get_event no
longer returned the exact same JSON as was inserted. This meant that
signature checks failed.
2014-11-24 10:56:36 +00:00
db9ce032a4
Fix pep8 codestyle warnings
2014-11-20 17:26:36 +00:00
dfdda2c871
Use module loggers rather than the root logger. Exceptions caused by bad clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str'
2014-11-20 17:10:37 +00:00
32090aee16
Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts
2014-11-20 16:24:00 +00:00
cb4b6c844a
Merge PDUs and Events into one object
2014-11-14 21:25:02 +00:00
e903c941cb
Merge branch 'develop' into request_logging
...
Conflicts:
setup.py
synapse/storage/_base.py
synapse/util/async.py
2014-11-14 11:16:50 +00:00
f04b3d5042
Store all signatures on events rather than just dropping them
2014-11-12 17:02:34 +00:00
a8e565eca8
Add an EventValidator. Fix bugs in auth ++ storage
2014-11-10 18:25:42 +00:00
cdc1b5d629
Fix regression where we did not return redacted events.
2014-11-10 15:21:30 +00:00
6447db063a
Fix backfill to work. Add auth to backfill request
2014-11-10 11:59:51 +00:00
1c06806f90
Finish redaction algorithm.
2014-11-10 10:21:32 +00:00
bf6b72eb55
Start implementing auth chains
2014-11-07 10:53:38 +00:00
4317c8e583
Implement new replace_state and changed prev_state
...
`prev_state` is now a list of previous state ids, similiar to
prev_events. `replace_state` now points to what we think was replaced.
2014-11-06 15:10:55 +00:00
cc44ecc62f
Get correct prev_events
2014-11-05 13:23:35 +00:00
aa76bf39ab
Remove unused imports
2014-11-04 14:14:02 +00:00
b29517bd01
Add a request-id to each log line
2014-10-30 01:21:33 +00:00
e7858b6d7e
Start filling out and using new events tables
2014-10-29 16:59:24 +00:00
2d1dfb3b34
Begin implementing all the PDU storage stuff in Events land
2014-10-28 16:42:35 +00:00
da1dda3e1d
Add transaction level logging and timing information. Add a _simple_delete method
2014-10-28 11:18:04 +00:00
967ce43b59
Clean up LoggingTransaction
2014-10-28 10:53:11 +00:00
8e358ef35a
Add timer to LoggingTransaction
2014-10-28 10:34:05 +00:00
82c5820767
keep 'origin_server_ts' as 'ts' in the database to avoid needlessly updating schema
2014-10-17 17:31:48 +01:00
f5cf7ac25b
SPEC-7: Rename 'ts' to 'origin_server_ts'
2014-10-17 17:12:25 +01:00
07639c79d9
Respond with more helpful error messages for unsigned requests
2014-10-13 16:39:15 +01:00
70899d3ab2
Rename deletions to redactions
2014-09-24 15:27:59 +01:00
1e6c5b205c
Fix bug where we didn't correctly pull out the event_id of the deletion
2014-09-24 13:29:20 +01:00
efea61dc50
Rename 'pruned' to 'pruned_because'
2014-09-23 17:40:58 +01:00
78af6bbb98
Add m.room.deletion. If an event is deleted it will be returned to clients 'pruned', i.e. all client specified keys will be removed.
2014-09-23 15:28:32 +01:00
b0406b9ead
Merge remote-tracking branch 'origin/develop' into test-sqlite-memory
2014-09-15 14:15:10 +01:00
5bd9369a62
Correctly handle the 'age' key in events and pdus
2014-09-15 13:26:11 +01:00
1c51c8ab7d
Merge remote-tracking branch 'origin/develop' into test-sqlite-memory
...
Conflicts:
synapse/storage/pdu.py
2014-09-12 17:20:06 +01:00
39e3fc69e5
Make the state resolution use actual power levels rather than taking them from a Pdu key.
2014-09-12 17:11:09 +01:00
a840ff8f3f
Now don't need the other logger.debug() call in _execute
2014-09-12 14:38:27 +01:00
1c20249884
Logging of all SQL queries via the 'synapse.storage.SQL' logger
2014-09-12 14:37:55 +01:00
e53d77b501
Add a .runInteraction() method on SQLBaseStore itself to wrap the .db_pool
2014-09-12 14:28:07 +01:00
781ff713ba
When getting a state event also include the previous content
2014-09-06 02:23:36 +01:00
d703e712f7
Add support to _simple_insert() to do INSERT OR REPLACE
2014-09-03 18:18:41 +01:00
8a7c1d6a00
fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org hasn't been incorporated in time for launch.
2014-09-03 17:31:57 +01:00
4b63b06cad
Merge branch 'develop' into storage_transactions
...
Conflicts:
synapse/api/auth.py
synapse/handlers/room.py
synapse/storage/__init__.py
2014-08-26 16:07:05 +01:00
4b2ad549d5
Move the event storage into a single transaction
2014-08-26 14:36:03 +01:00
063e1b22e6
Stop internal keys from getting into SynapseEvents
2014-08-21 15:06:00 +01:00
d260a42ca2
PEP8 cleanups
2014-08-15 16:17:36 +01:00
114984a236
Start chagning the events stream to work with the new DB schema
2014-08-15 13:58:28 +01:00
5002efa31b
Reimplement the get public rooms api to work with new DB schema
2014-08-15 10:26:35 +01:00
2c46bb6208
Fix up typos and correct sql queries
2014-08-14 18:40:50 +01:00
661c711765
Start fixing places that use the data store.
2014-08-14 17:34:37 +01:00
2529f2bc01
Rename _execute_query
2014-08-14 16:58:51 +01:00
937c175029
Fix up RoomMemberStore to work with the new schema.
2014-08-14 16:02:10 +01:00
6d6a1c3454
Actually encode dicts as json in the DB
2014-08-14 14:30:25 +01:00
10294b6082
Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor
...
Conflicts:
synapse/storage/_base.py
2014-08-14 10:01:04 +01:00
d05aa651f8
An initial hack at storing presence state-change mtimes in database and presenting age durations to clients/federation events
2014-08-13 19:19:15 +01:00
baf04be5cf
Set datastore's .hs field in SQLBaseStore rather than in the toplevel DataStore mixed-in result class
2014-08-13 19:19:15 +01:00
336987bb8d
Initial stab at refactoring the SQL tables, including rejigging some of the storage layer.
2014-08-13 18:03:41 +01:00
f98e6380f1
add in whitespace after copyright statements to improve legibility
2014-08-13 03:14:34 +01:00
4f475c7697
Reference Matrix Home Server
2014-08-12 15:10:52 +01:00
Erik Johnston