Erik Johnston
ab78a8926e
Err, we probably want a bigger limit
2015-05-14 13:47:16 +01:00
Erik Johnston
cdb3757942
Refactor _get_events
2015-05-14 13:31:55 +01:00
Erik Johnston
36ea26c5c0
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf
2015-05-14 12:01:38 +01:00
David Baker
7c549dd557
Add ID generator for push_rules_enable to #resolve SYN-378
2015-05-14 11:44:03 +01:00
Erik Johnston
968b01a91a
Actually use async method
2015-05-13 17:02:46 +01:00
Erik Johnston
4071f29653
Fetch events from events_id in their own transactions
2015-05-13 16:59:41 +01:00
Erik Johnston
a988361aea
Typo
2015-05-13 15:44:15 +01:00
Erik Johnston
8888982db3
Don't insert None
2015-05-13 15:43:32 +01:00
Erik Johnston
cf706cc6ef
Don't return None
2015-05-13 15:31:25 +01:00
Erik Johnston
5971d240d4
Limit batch size
2015-05-13 15:26:49 +01:00
Erik Johnston
ca4f458787
Fetch events in bulk
2015-05-13 15:13:42 +01:00
Erik Johnston
6edff11a88
Don't fetch redaction and rejection stuff for each event, so we can use index only scan
2015-05-13 14:39:05 +01:00
Erik Johnston
409bcc76bd
Load events for state group seperately
2015-05-13 11:13:31 +01:00
Erik Johnston
da6a7bbdde
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context
2015-05-12 13:10:42 +01:00
Erik Johnston
cd525c0f5a
push_rules table expects an 'id' field
2015-05-11 11:24:01 +01:00
Erik Johnston
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
Erik Johnston
ed2584050f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany
2015-05-05 18:15:20 +01:00
Mark Haines
31049c4d72
Merge pull request #139 from matrix-org/bugs/SYN-369
...
Fix race with cache invalidation. SYN-369
2015-05-05 17:46:13 +01:00
Mark Haines
deb0237166
Add some doc-string
2015-05-05 17:45:11 +01:00
Mark Haines
d18f37e026
Collect the invalidate callbacks on the transaction object rather than passing around a separate list
2015-05-05 17:32:21 +01:00
Erik Johnston
9951542393
Add a comment about the zip(*[zip(sorted(...),...)])
2015-05-05 17:06:55 +01:00
Mark Haines
041b6cba61
SYN-369: Add comments to the sequence number logic in the cache
2015-05-05 16:32:44 +01:00
Mark Haines
63075118a5
Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access
2015-05-05 16:24:04 +01:00
Erik Johnston
43c2e8deae
Add support for using executemany
2015-05-05 15:13:25 +01:00
Erik Johnston
1692dc019d
Don't call 'encode_parameter' no-op
2015-05-05 15:00:30 +01:00
Mark Haines
261d809a47
Sequence the modifications to the cache so that selects don't race with inserts
2015-05-05 14:13:50 +01:00
Erik Johnston
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
Erik Johnston
fabb7acd45
Fix bug where we reconnected to the database on every query.
2015-05-01 10:24:24 +01:00
Erik Johnston
2732be83d9
Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.
2015-04-27 13:22:30 +01:00
Erik Johnston
e4c4664d73
Handle the fact that postgres databases can be restarted from under us
2015-04-27 12:40:49 +01:00
Erik Johnston
b8092fbc82
Go back to storing JSON in TEXT
2015-04-16 11:17:52 +01:00
Erik Johnston
5b31afcbd1
Remove debug logging
2015-04-15 16:27:04 +01:00
Erik Johnston
ed26e4012b
pushers table requires a unique id.
2015-04-15 16:24:14 +01:00
Erik Johnston
a5c72780e6
Don't pass in removed flag
2015-04-15 15:13:22 +01:00
Erik Johnston
4af32a2817
Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction
2015-04-15 14:51:21 +01:00
Erik Johnston
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
Erik Johnston
7ed2ec3061
Handle the fact that in sqlite binary data might be stored as unicode or bytes
2015-04-10 13:41:54 +01:00
Erik Johnston
8bf285e082
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-09 13:13:26 +01:00
Erik Johnston
8ad0f4912e
Stream ordering and out of order insertions.
...
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
2015-04-09 11:41:36 +01:00
Mark Haines
83f5125d52
Merge pull request #114 from matrix-org/improve_get_event_cache
...
Improve get event cache
2015-04-08 16:50:27 +01:00
Erik Johnston
3887350e47
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-08 14:11:07 +01:00
Erik Johnston
c1b34af441
Move database timer logging to seperate logger
2015-04-08 13:31:19 +01:00
Erik Johnston
9a05795619
Retry transaction, not SQL query
2015-04-08 13:11:28 +01:00
Erik Johnston
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
Erik Johnston
49d6aa1394
Retry on deadlock
2015-04-07 15:28:37 +01:00
Erik Johnston
0af5f5efaf
Don't use multiple UNIQUE constraints; it will cause deadlocks
2015-04-07 12:08:35 +01:00
Erik Johnston
c8d3f6486d
Implement or_ignore flag on inserts
2015-04-07 12:06:22 +01:00
Erik Johnston
304111afd0
Don't use AUTOINCREMENT, use an in memory version
2015-04-07 12:05:36 +01:00
Erik Johnston
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
Erik Johnston
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
Paul "LeoNerd" Evans
a198894bf7
Appease pep8
2015-03-26 11:53:58 +00:00
Paul "LeoNerd" Evans
953e40f9dc
Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one
2015-03-25 19:12:16 +00:00
Paul "LeoNerd" Evans
f173d40a32
Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key
2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
1b988b051b
Store the rejected reason in (Frozen)Event structs
2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
033a517feb
Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file
2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
9ba6487b3f
Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()
2015-03-25 19:05:34 +00:00
Erik Johnston
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
Erik Johnston
9e98f1022a
Don't order by rowid
2015-03-24 16:19:01 +00:00
Erik Johnston
6e7131f02f
Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.
2015-03-23 15:38:56 +00:00
Paul "LeoNerd" Evans
0f86312c4c
Pull out the cache logic from the @cached wrapper into its own class we can reuse
2015-03-20 18:25:42 +00:00
Paul "LeoNerd" Evans
b1022ed8b5
func(*EXPR) is valid Python syntax, really...
2015-03-20 17:47:45 +00:00
Erik Johnston
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
Erik Johnston
80cd08c190
PEP8
2015-03-20 16:03:25 +00:00
Erik Johnston
9517f4da4d
Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement
2015-03-20 16:02:47 +00:00
Erik Johnston
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
Erik Johnston
7e282a53a5
Tidy up _simple_... methods
2015-03-20 15:05:10 +00:00
Paul "LeoNerd" Evans
91cb46191d
Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though
2015-03-20 14:59:45 +00:00
Erik Johnston
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
Erik Johnston
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
Erik Johnston
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
Paul "LeoNerd" Evans
1489521ee5
Be polite and ensure we use @functools.wraps() when creating a function decorator
2015-03-17 17:19:22 +00:00
Paul "LeoNerd" Evans
be170b1426
Add a metric for the scheduling latency of SQL queries
2015-03-16 17:21:59 +00:00
Paul "LeoNerd" Evans
e75fa8bbbf
Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
63cb7ece62
Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f1fbe3e09f
Rename TimerMetric to DistributionMetric; as it could count more than just time
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
642f725fd7
Pretend the 'getEvent' cache is just another cache in the set of all the others for metric
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
fa319a5786
Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
25187ab674
Collect per-SQL-verb timer stats on query execution time
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
e1a7e3564f
Delete a couple of TODO markers of monitoring stats now done
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
8664599af7
Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
b0cdf097f4
Sprinkle some CacheCounterMetrics around the synapse.storage layer
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
Paul "LeoNerd" Evans
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
Kegan Dougal
3d73383d18
Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users.
2015-03-02 10:16:24 +00:00
Paul "LeoNerd" Evans
9640510de2
Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased
2015-02-23 18:41:58 +00:00
Paul "LeoNerd" Evans
f53fcbce97
Use cache.pop() instead of a separate membership test + del []
2015-02-23 18:30:45 +00:00
Mark Haines
27080698e7
Fix code style warning
2015-02-23 18:19:13 +00:00
Paul "LeoNerd" Evans
a09e59a698
Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing
2015-02-23 16:55:57 +00:00
Paul "LeoNerd" Evans
e76d485e29
Allow @cached-wrapped functions to have a prefill method for setting entries
2015-02-23 15:41:54 +00:00
Paul "LeoNerd" Evans
55022d6ca5
Remove a TODO note
2015-02-19 18:38:09 +00:00
Paul "LeoNerd" Evans
ebc3db295b
Take named arguments to @cached() decorator, add a 'max_entries' limit
2015-02-19 18:36:02 +00:00
Paul "LeoNerd" Evans
077d200342
Move @cached decorator out into synapse.storage._base; add minimal docs
2015-02-19 17:29:39 +00:00
Erik Johnston
42bc56dad3
Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson
2015-02-11 17:01:38 +00:00
Mark Haines
aff892ce79
Fix formatting
2015-02-11 15:02:35 +00:00
Mark Haines
f5a70e0d2e
Add a cache for get_event
2015-02-11 15:01:15 +00:00
Erik Johnston
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
Mark Haines
b085fac735
Code-style fixes
2015-02-10 16:30:48 +00:00
Mark Haines
fda4422bc9
Fix pyflakes
2015-02-10 14:54:07 +00:00
Mark Haines
d7c7efb691
Add performance counters for different stages of loading events
2015-02-10 14:50:53 +00:00
Mark Haines
0c4536da8f
Use the transaction 'desc' rather than 'name', increment the txn_ids in
...
txn names
2015-02-09 18:06:31 +00:00
Mark Haines
347b497db0
Formatting
2015-02-09 17:57:09 +00:00
Mark Haines
3a5ad7dbd5
Performance counters for database transaction names
2015-02-09 17:55:56 +00:00
Erik Johnston
c4ee4ce93e
Fix typo
2015-02-09 15:00:37 +00:00
Erik Johnston
66fde49f07
Log database time every 10s and log as percentage
2015-02-09 14:45:15 +00:00
Erik Johnston
75656712e3
Time how long we're spending on the database thread
2015-02-09 14:22:52 +00:00
Erik Johnston
2f4cb04f45
Be more specific in naming columns in selects.
2015-01-30 14:48:11 +00:00
Erik Johnston
e0b7c521cb
Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage
...
Conflicts:
synapse/storage/__init__.py
synapse/storage/schema/delta/v12.sql
2015-01-30 14:08:28 +00:00
David Baker
6d485dd1c7
unnecessary newlines
2015-01-28 14:48:42 +00:00
David Baker
fb0928097a
More magic commas (including the place I copied it from...)
2015-01-28 14:48:07 +00:00
David Baker
0cbb6b0f52
Google doc style
2015-01-28 14:44:41 +00:00
Erik Johnston
b1b85753d7
Add support for storing rejected events in EventContext and data stores
2015-01-22 15:50:17 +00:00
David Baker
c06a9063e1
Merge branch 'develop' into pushers
2015-01-13 13:15:51 +00:00
Erik Johnston
36a2a877e2
Use time.time() instead of time.clock()
2015-01-06 16:34:41 +00:00
Erik Johnston
52b2c6c9c7
Don't include None's in _get_events_txn
2015-01-06 14:56:57 +00:00
Mark Haines
5e23a19204
Merge pull request #28 from matrix-org/erikj-perf
...
Database performance improvements.
2015-01-06 13:33:40 +00:00
Mark Haines
adb04b1e57
Update copyright notices
2015-01-06 13:21:39 +00:00
Erik Johnston
af1c7c7808
PEP8
2015-01-06 13:13:17 +00:00
Erik Johnston
52d8519008
Don't do batching when getting events.
2015-01-06 13:10:27 +00:00
Erik Johnston
98933e3db6
Only fetch prev_content when a client is streaming/paginating. Use transactions for event streams.
2015-01-06 13:03:23 +00:00
Erik Johnston
3e26720e05
Temporarily turn off 'redacted_because' and 'prev_content' keys
2015-01-06 11:26:58 +00:00
Erik Johnston
f4ea78e9e2
More debug logging
2015-01-06 11:24:18 +00:00
Erik Johnston
d7e8ea67b3
Reformat
2015-01-06 11:18:02 +00:00
David Baker
b56730bb6e
Merge branch 'develop' into pushers
...
Conflicts:
synapse/api/errors.py
synapse/server.py
synapse/storage/__init__.py
2014-12-18 15:15:22 +00:00
David Baker
9728c305a3
after a few rethinks, a working implementation of pushers.
2014-12-18 14:49:22 +00:00
Erik Johnston
f3788e3c78
Test some ideas that might help performance a bit
2014-12-17 23:37:08 +00:00
Erik Johnston
5b39cfff69
Don't assume an event exists
2014-12-16 18:25:24 +00:00
Erik Johnston
882dc8dcab
Persist internal_metadata
2014-12-16 13:17:09 +00:00
Erik Johnston
c8dd3314d6
Fix bug where we ignored event_edge_hashes table
2014-12-15 13:55:22 +00:00
Erik Johnston
75085bb4d1
Pyflakes
2014-12-12 14:34:34 +00:00
Erik Johnston
0b04369238
Fix public room joining by making sure replaces_state never points to itself.
2014-12-11 15:56:06 +00:00
Erik Johnston
9191292b0f
Fix prev_content
2014-12-11 15:16:55 +00:00
Erik Johnston
8cdebce470
Fix redactions. Fix 'age' key
2014-12-11 13:25:19 +00:00
Erik Johnston
02e4c18171
Remove dead code
2014-12-10 18:00:36 +00:00
Erik Johnston
aae8a37e63
Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor
2014-12-10 13:18:40 +00:00
Erik Johnston
aa3f66cf7f
Change the way we implement get_events to be less sucky
2014-12-09 13:35:26 +00:00
Erik Johnston
ba3d1e2fc0
Remove unused import
2014-12-08 12:01:25 +00:00
Matthew Hodgson
aed62a3583
track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover.
2014-12-07 02:26:07 +00:00
Erik Johnston
6630e1b579
Start making more things use EventContext rather than event.*
2014-12-05 16:20:48 +00:00
Erik Johnston
ec2b5d8c28
Store full JSON of events in db
2014-12-01 16:22:07 +00:00
Erik Johnston
a46e5ef621
SYN-163: Add an order by rowid to selects.
...
This should fix the bug where the edges of the graph get returned in a
different order than they were inserted in, and so no get_event no
longer returned the exact same JSON as was inserted. This meant that
signature checks failed.
2014-11-24 10:56:36 +00:00
Mark Haines
db9ce032a4
Fix pep8 codestyle warnings
2014-11-20 17:26:36 +00:00
Mark Haines
dfdda2c871
Use module loggers rather than the root logger. Exceptions caused by bad clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str'
2014-11-20 17:10:37 +00:00
Mark Haines
32090aee16
Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts
2014-11-20 16:24:00 +00:00
Mark Haines
cb4b6c844a
Merge PDUs and Events into one object
2014-11-14 21:25:02 +00:00
Mark Haines
e903c941cb
Merge branch 'develop' into request_logging
...
Conflicts:
setup.py
synapse/storage/_base.py
synapse/util/async.py
2014-11-14 11:16:50 +00:00
Erik Johnston
f04b3d5042
Store all signatures on events rather than just dropping them
2014-11-12 17:02:34 +00:00
Erik Johnston
a8e565eca8
Add an EventValidator. Fix bugs in auth ++ storage
2014-11-10 18:25:42 +00:00
Erik Johnston
cdc1b5d629
Fix regression where we did not return redacted events.
2014-11-10 15:21:30 +00:00