61d05daab1
More join conditions
2015-04-30 18:41:05 +01:00
6ead27ddda
Add more conditions on JOINs to make postgres go a little faster.
2015-04-30 18:32:03 +01:00
f383d5a801
Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction
2015-04-30 17:12:52 +01:00
69d4063651
Add get_rooms_for_user cache
2015-04-30 16:47:51 +01:00
3c4c229788
Don't use sub queries, it makes postgres sad
2015-04-30 16:16:53 +01:00
29400b45b9
SYN-367: Use upsert rather than insert_or_replace
2015-04-30 15:21:31 +01:00
2366d28780
Don't needlessly join on state_events
2015-04-30 14:02:06 +01:00
1d7702833d
Make simple query rather than long one and then throw away half the results
2015-04-30 10:16:12 +01:00
b1ca784aca
Correctly decode, for sqlite and postgres, rows from pushers table
2015-04-29 19:41:14 +01:00
4a9dc5b2f5
pushkey' are also bytes.
2015-04-29 19:27:02 +01:00
0ade2712d1
Typo
2015-04-29 19:17:25 +01:00
50f96f256f
Also remove yield from within lock in the other generator
2015-04-29 19:17:00 +01:00
d2d61a8288
Fix deadlock in id_generators. No idea why this was an actual deadlock.
2015-04-29 19:15:23 +01:00
3864b3a8e6
Actually return rows
2015-04-29 18:07:36 +01:00
0618978238
Typo, args wrong way round
2015-04-29 18:04:35 +01:00
09177f4f2e
Decode buffers in same thread
2015-04-29 18:03:42 +01:00
472be88674
We store pusher data as bytes
2015-04-29 17:43:46 +01:00
12d381bd5d
Decode the data json in the storage layer (was moved but this part was missed)
2015-04-29 17:13:51 +01:00
f8c30faf25
Oops, update the contraint too
2015-04-29 16:58:42 +01:00
61cd5d9045
Be more postgressive
2015-04-29 16:57:14 +01:00
fb95035a65
Be postgressive
2015-04-29 16:53:41 +01:00
4669def000
Oops, forgot the schema delta file
2015-04-29 16:50:16 +01:00
0337eaf321
txn.execute doesn't return cursors
2015-04-29 16:43:39 +01:00
884fb88e28
txn.execute doesn't return cursors
2015-04-29 16:35:20 +01:00
d76c058eea
Fix invalid SQL to work in postgres land
2015-04-29 16:30:25 +01:00
c8ed9bd278
pushers table requires an access_token
2015-04-29 15:33:27 +01:00
970a9b9d2b
We can't use REPLACE when upgrading databases now we have postgres
2015-04-29 13:55:44 +01:00
64991b0c8b
Merge pull request #129 from matrix-org/key_distribution
...
Key distribution v2
2015-04-29 13:34:38 +01:00
e26a3d8d9e
bump database schema version
2015-04-29 13:32:32 +01:00
a9549fdce3
Use bytea rather than BLOB
2015-04-29 13:16:09 +01:00
4ad8b45155
Merge branch 'develop' into key_distribution
...
Conflicts:
synapse/config/homeserver.py
2015-04-29 13:15:14 +01:00
cd0864121b
Make postgres database error slightly more helpful
2015-04-29 12:12:25 +01:00
4932a7e2d9
Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__
2015-04-29 12:12:25 +01:00
a08bf11138
Appease PEP8
2015-04-29 11:44:48 +01:00
204132a998
Check that postgres database has correct charset set
2015-04-29 11:42:28 +01:00
327ca883ec
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-28 13:39:42 +01:00
18f8247701
Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit
2015-04-28 12:41:33 +01:00
9182f87664
Merge pull request #126 from matrix-org/csauth
...
Client / Server Auth Refactor
2015-04-28 11:00:27 +01:00
55e1bc8920
And don't bump the schema version unnecessarily
2015-04-28 10:54:15 +01:00
b96c133034
Add server_keys.sql to the current delta rather than creating a new delta
2015-04-28 10:50:00 +01:00
8558e1ec73
Make get_max_token into inlineCallbacks so that the lock works.
2015-04-27 15:19:44 +01:00
6f8e2d517e
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-27 14:41:40 +01:00
2732be83d9
Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.
2015-04-27 13:22:30 +01:00
e4c4664d73
Handle the fact that postgres databases can be restarted from under us
2015-04-27 12:40:49 +01:00
8a5be236e0
pep8
2015-04-27 11:49:18 +01:00
b02e1006b9
Run database check before daemonizing, at the cost of database hygiene.
2015-04-27 11:46:00 +01:00
f8152f2708
rename db method to be more informative
2015-04-27 10:16:26 +01:00
2f475bd5d5
pep8
2015-04-24 18:15:07 +01:00
a7b51f4539
Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266
2015-04-24 18:11:21 +01:00
eede182df7
Merge branch 'develop' into key_distribution
2015-04-24 10:35:49 +01:00
4e2f8b8722
Copyright notices
2015-04-24 10:35:29 +01:00
f30d47c876
Implement remote key lookup api
2015-04-22 14:21:08 +01:00
91c8f828e1
pep8
2015-04-17 19:56:04 +01:00
8db6832db8
Password reset, finally.
2015-04-17 19:53:47 +01:00
117f35ac4a
Add endpoint to get threepids from server
2015-04-17 17:20:18 +01:00
4eea5cf6c2
pep8
2015-04-17 16:46:45 +01:00
f96ab9d18d
make add3pid servlet work
2015-04-17 16:44:49 +01:00
e3417bbbe0
Revert needless change to storage.profile
2015-04-17 16:13:14 +01:00
cb03fafdf1
Merge branch 'develop' into csauth
2015-04-17 13:51:10 +01:00
0ec346d942
Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea
2015-04-17 11:10:20 +01:00
e1c0970c11
PEP8
2015-04-16 11:18:45 +01:00
b8092fbc82
Go back to storing JSON in TEXT
2015-04-16 11:17:52 +01:00
bc9e69e160
Move encoding and decoding of JSON into storage layer
2015-04-16 11:01:09 +01:00
ffad75bd62
Remove mysql/maria support
2015-04-15 17:00:50 +01:00
a429515bdd
Add methods for storing and retrieving the raw key json
2015-04-15 16:58:35 +01:00
cf04cedf21
Change full_schemas/11 to work with postgres
2015-04-15 16:53:47 +01:00
5b31afcbd1
Remove debug logging
2015-04-15 16:27:04 +01:00
6e91f14d09
Add missing yield
2015-04-15 16:25:07 +01:00
ed26e4012b
pushers table requires a unique id.
2015-04-15 16:24:14 +01:00
a5c72780e6
Don't pass in removed flag
2015-04-15 15:13:22 +01:00
4af32a2817
Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction
2015-04-15 14:51:21 +01:00
25d2914fba
Turn off persistance of sent_transactions.response_json
2015-04-15 14:09:35 +01:00
cce5d057d3
Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses
2015-04-15 11:43:25 +01:00
a971fa9d58
Use try..finally in contextlib.contextmanager
2015-04-15 10:25:43 +01:00
ded4128965
Use True for True rather than 1
2015-04-15 10:24:24 +01:00
f9e12f79ca
Add missing yield in storage func
2015-04-15 10:24:07 +01:00
c756dfeb14
Correctly identify deadlocks
2015-04-15 10:23:42 +01:00
32e14d8181
Return a sha256 fingerprint rather than the entire tls certificate
2015-04-14 19:10:09 +01:00
127fad17dd
Add postgres database engine
2015-04-14 14:50:29 +01:00
5a95cd4442
Rename user_ips.user -> user_id
2015-04-14 13:54:09 +01:00
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
3c741682e5
Correctly increment the _next_id initially
2015-04-14 09:54:44 +01:00
86fc9b617c
For backwards compat, make state_groups.id have a type of int, not varchar
2015-04-13 17:03:49 +01:00
93937b2b31
Remove duplicate rows
2015-04-10 18:47:03 +01:00
c5365dee56
Use case sensitive collations
2015-04-10 18:46:33 +01:00
4103b1c470
DROP indexes before recreating them
2015-04-10 18:46:09 +01:00
4d5b098626
Use LONGBLOB and TEXT for arbitary length rows
2015-04-10 18:45:09 +01:00
7ed2ec3061
Handle the fact that in sqlite binary data might be stored as unicode or bytes
2015-04-10 13:41:54 +01:00
ce797ad373
Bump schema version
2015-04-10 11:22:30 +01:00
7e863c51e6
Use unsigned bigint
2015-04-10 11:22:04 +01:00
0f12772e32
SQLite wants INTEGER and not BIGINT for primary keys
2015-04-10 11:16:09 +01:00
d5d4281647
Update full_schemas/16 to match delta files. Add delta/16 scripts
2015-04-10 10:59:46 +01:00
cda4a6f93f
Revert non-trivial changes to upgrade scripts
2015-04-10 10:19:50 +01:00
e2722f58ee
Fix schema again
2015-04-10 10:16:29 +01:00
a1665c5094
Revert non-trivial schema changes and move them to a new schema version.
2015-04-10 10:05:44 +01:00
2ded344620
Remove unused import
2015-04-09 13:46:06 +01:00
9707acfc40
Remove spurious spaces
2015-04-09 13:45:20 +01:00
8bf285e082
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-09 13:13:26 +01:00
8ad0f4912e
Stream ordering and out of order insertions.
...
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
2015-04-09 11:41:36 +01:00
279a547a8b
Use generic db exceptions rather than sqlite3 specific ones
2015-04-08 16:53:48 +01:00
83f5125d52
Merge pull request #114 from matrix-org/improve_get_event_cache
...
Improve get event cache
2015-04-08 16:50:27 +01:00
3887350e47
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-08 14:11:07 +01:00
c1b34af441
Move database timer logging to seperate logger
2015-04-08 13:31:19 +01:00
9a05795619
Retry transaction, not SQL query
2015-04-08 13:11:28 +01:00
24d8134ac1
Fix maria engine to correctly recognize deadlocks
2015-04-08 13:10:54 +01:00
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
49d6aa1394
Retry on deadlock
2015-04-07 15:28:37 +01:00
0bfa78b39b
PEP8
2015-04-07 12:16:05 +01:00
6bc9edd8b2
Fix prepare_sqlite3_database's convert_param_style
2015-04-07 12:13:58 +01:00
05a35d62b6
Bump database version
2015-04-07 12:10:15 +01:00
8574bf62dc
Add index to presence table
2015-04-07 12:09:36 +01:00
0af5f5efaf
Don't use multiple UNIQUE constraints; it will cause deadlocks
2015-04-07 12:08:35 +01:00
c8d3f6486d
Implement or_ignore flag on inserts
2015-04-07 12:06:22 +01:00
304111afd0
Don't use AUTOINCREMENT, use an in memory version
2015-04-07 12:05:36 +01:00
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
5e88a09a42
Add same user_id char checks as registration.
2015-03-31 14:00:25 +01:00
cf1fa59f4b
Use a sender localpart instead of a user ID.
...
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
2015-03-31 13:48:03 +01:00
c217504949
Edit SQL schema to use string IDs not ints. Use token as ID. Update tests.
2015-03-31 12:07:56 +01:00
b59aa74556
Fix tests and missing returns on deferreds.
2015-03-31 11:35:45 +01:00
d33ae65efc
Remove more reg/unreg methods. Read config not database for cache.
2015-03-31 11:00:00 +01:00
3e420aebd8
Revert "Add another @cached wrapper, this time on get_presence_state()"
...
This reverts commit ff1fa0fbf8
2015-03-27 16:16:58 +00:00
ff1fa0fbf8
Add another @cached wrapper, this time on get_presence_state()
2015-03-27 15:57:16 +00:00
a32e876ef4
Delete pushers when changing password
2015-03-26 13:40:16 +00:00
a198894bf7
Appease pep8
2015-03-26 11:53:58 +00:00
32206dde3f
Fixes from PR comments
2015-03-26 10:11:52 +00:00
4edcbcee3b
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
953e40f9dc
Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one
2015-03-25 19:12:16 +00:00
c1a256cc4c
Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API.
2015-03-25 19:06:22 +00:00
f173d40a32
Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key
2015-03-25 19:06:05 +00:00
1b988b051b
Store the rejected reason in (Frozen)Event structs
2015-03-25 19:06:05 +00:00
033a517feb
Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file
2015-03-25 19:06:05 +00:00
9ba6487b3f
Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()
2015-03-25 19:05:34 +00:00
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
c7023f2155
1) Pushers are now associated with an access token
...
2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
2015-03-24 17:24:15 +00:00
0ba393924a
Escape non printing ascii character
2015-03-24 16:31:52 +00:00
f488293d96
Don't reinsert into event_edges
2015-03-24 16:20:26 +00:00
1aa44939fc
Fix bugs in transactions storage
2015-03-24 16:20:05 +00:00
5a447098dd
Don't use room hosts table
2015-03-24 16:19:24 +00:00
9e98f1022a
Don't order by rowid
2015-03-24 16:19:01 +00:00
9115421ace
Use _simple_upsert
2015-03-24 16:17:39 +00:00
d19e79ecc9
Make deleting other access tokens when you change your password actually work
2015-03-24 15:33:48 +00:00
6e7131f02f
Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.
2015-03-23 15:38:56 +00:00
9a7f496298
Sanitize RoomMemberStore
2015-03-23 15:29:04 +00:00
d98660a60d
Implement password changing (finally) along with a start on making client/server auth more general.
2015-03-23 14:20:28 +00:00
d5272b1d2c
Use 'update or insert' rather than on 'conflict replace'
2015-03-23 14:02:34 +00:00
278149f533
Sanitize TransactionStore
2015-03-23 13:43:21 +00:00
72d8406409
Put a cache on get_aliases_for_room
2015-03-20 19:21:13 +00:00
a63b4f7101
Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale
2015-03-20 18:25:49 +00:00
0f86312c4c
Pull out the cache logic from the @cached wrapper into its own class we can reuse
2015-03-20 18:25:42 +00:00
b1022ed8b5
func(*EXPR) is valid Python syntax, really...
2015-03-20 17:47:45 +00:00
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
80cd08c190
PEP8
2015-03-20 16:03:25 +00:00
9517f4da4d
Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement
2015-03-20 16:02:47 +00:00
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
ceb61daa70
Add the tiniest of tiny one-element caches to get_room_events_max_id() as it's read every time someone hits eventstream
2015-03-20 15:44:06 +00:00
fce0114005
Start removing Tables
2015-03-20 15:05:44 +00:00
7e282a53a5
Tidy up _simple_... methods
2015-03-20 15:05:10 +00:00
91cb46191d
Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though
2015-03-20 14:59:45 +00:00
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
d288d273e1
Generate transaction id in code
2015-03-20 10:57:44 +00:00
d4f50f3ae5
decode_result takes an iterable
2015-03-20 10:57:26 +00:00
0d0610870d
Fix up schemas some more
2015-03-20 10:55:31 +00:00
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
58ed393235
Remove redundant key
2015-03-19 15:12:05 +00:00
fae059cc18
Fix up schemas to work with mariadb
2015-03-19 13:42:39 +00:00
bb24609158
Clean out event_forward_extremities table when the server rejoins the room
2015-03-18 11:19:47 +00:00
89036579ed
Update schema to work with mariadb
2015-03-18 11:18:49 +00:00
93978c5e2b
@cached() annotate get_user_by_token() - achieves a minor DB performance improvement
2015-03-17 17:24:51 +00:00
1489521ee5
Be polite and ensure we use @functools.wraps() when creating a function decorator
2015-03-17 17:19:22 +00:00
be170b1426
Add a metric for the scheduling latency of SQL queries
2015-03-16 17:21:59 +00:00
cd2539ab2a
Merge pull request #110 from matrix-org/fix_ban
...
Fix ban
2015-03-16 15:36:52 +00:00
f0d6f724a2
Set the service ID as soon as it is known.
2015-03-16 15:24:32 +00:00
f1d2b94e0b
Copy dict of context.current_state before changing it.
2015-03-16 15:13:05 +00:00
835e01fc70
Minor PR comment tweaks.
2015-03-16 10:16:59 +00:00
f9232c7917
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/appservice.py
2015-03-16 10:09:15 +00:00
e75fa8bbbf
Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot
2015-03-12 16:24:51 +00:00
63cb7ece62
Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer
2015-03-12 16:24:51 +00:00
f1fbe3e09f
Rename TimerMetric to DistributionMetric; as it could count more than just time
2015-03-12 16:24:51 +00:00
642f725fd7
Pretend the 'getEvent' cache is just another cache in the set of all the others for metric
2015-03-12 16:24:51 +00:00
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
fa319a5786
Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely
2015-03-12 16:24:51 +00:00
25187ab674
Collect per-SQL-verb timer stats on query execution time
2015-03-12 16:24:50 +00:00
e1a7e3564f
Delete a couple of TODO markers of monitoring stats now done
2015-03-12 16:24:50 +00:00
8664599af7
Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache
2015-03-12 16:24:50 +00:00
b0cdf097f4
Sprinkle some CacheCounterMetrics around the synapse.storage layer
2015-03-12 16:24:50 +00:00
a7a28f85ae
Appease pep8
2015-03-11 17:32:43 +00:00
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
04f8478aaa
Add the master push rule for the break-my-push button. Allow server default rules to be disabled by default.
2015-03-10 17:26:25 +00:00
8916acbc13
These aren't defined for redacted events so don't crash
2015-03-10 11:21:37 +00:00
db1fbc6c6f
Fix remaining scheduler bugs. Add more informative logging.
2015-03-10 10:04:20 +00:00
7e0bba555c
Remove unused import
2015-03-09 17:48:37 +00:00
04c9751f24
Bug fixes whilst putting it all together
2015-03-09 17:45:41 +00:00
b98cd03193
Use event IDs instead of dumping event content in the txns table.
2015-03-09 17:25:20 +00:00
21fd84dcb8
Use seconds; start gluing in the AS scheduler into the AS handler.
2015-03-09 17:01:19 +00:00
0a60bbf4fa
Finish appservice txn storage impl and tests.
2015-03-09 15:53:03 +00:00
1ead1caa18
Implement create_appservice_txn with tests.
2015-03-09 13:54:20 +00:00
Erik Johnston