ed2584050f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany
2015-05-05 18:15:20 +01:00
977338a7af
Use buffer(...) when inserting into bytea column
2015-05-05 18:12:53 +01:00
31049c4d72
Merge pull request #139 from matrix-org/bugs/SYN-369
...
Fix race with cache invalidation. SYN-369
2015-05-05 17:46:13 +01:00
deb0237166
Add some doc-string
2015-05-05 17:45:11 +01:00
d18f37e026
Collect the invalidate callbacks on the transaction object rather than passing around a separate list
2015-05-05 17:32:21 +01:00
9951542393
Add a comment about the zip(*[zip(sorted(...),...)])
2015-05-05 17:06:55 +01:00
041b6cba61
SYN-369: Add comments to the sequence number logic in the cache
2015-05-05 16:32:44 +01:00
63075118a5
Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access
2015-05-05 16:24:04 +01:00
531d7955fd
Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely
2015-05-05 16:12:28 +01:00
bfa4a7f8b0
Invalidate the room_member cache if the current state events updates
2015-05-05 15:43:49 +01:00
d0fece8d3c
Missing return for when the event was already persisted
2015-05-05 15:39:09 +01:00
bdcd7693c8
Fix indentation
2015-05-05 15:14:48 +01:00
43c2e8deae
Add support for using executemany
2015-05-05 15:13:25 +01:00
1692dc019d
Don't call 'encode_parameter' no-op
2015-05-05 15:00:30 +01:00
a9aea68fd5
Invalidate the caches from the correct thread
2015-05-05 14:57:08 +01:00
261d809a47
Sequence the modifications to the cache so that selects don't race with inserts
2015-05-05 14:13:50 +01:00
d9cc5de9e5
Correctly name transaction
2015-05-05 10:24:10 +01:00
b8940cd902
Remove some unused indexes
2015-05-01 16:14:25 +01:00
eb9bd2d949
user_id now in user_threepids
2015-05-01 15:04:37 +01:00
9eeb03c0dd
Don't use self.execute: it's designed for fetching stuff
2015-05-01 14:21:25 +01:00
7693f24792
No id field on user
2015-05-01 13:55:42 +01:00
92b20713d7
More missed get_user_by_id API changes
2015-05-01 13:45:54 +01:00
4011cf1c42
Cache latest_event_ids_in_room
2015-05-01 13:06:26 +01:00
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
fabb7acd45
Fix bug where we reconnected to the database on every query.
2015-05-01 10:24:24 +01:00
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
8be5284e91
Remove pointless join
2015-04-30 18:49:26 +01:00
503e4d3d52
Fix broken SQL
2015-04-30 18:44:47 +01:00
00718ae7a9
Need more yield
2015-04-30 18:43:39 +01:00
0465560c1a
Add missing param
2015-04-30 18:42:44 +01:00
61d05daab1
More join conditions
2015-04-30 18:41:05 +01:00
6ead27ddda
Add more conditions on JOINs to make postgres go a little faster.
2015-04-30 18:32:03 +01:00
f383d5a801
Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction
2015-04-30 17:12:52 +01:00
69d4063651
Add get_rooms_for_user cache
2015-04-30 16:47:51 +01:00
3c4c229788
Don't use sub queries, it makes postgres sad
2015-04-30 16:16:53 +01:00
29400b45b9
SYN-367: Use upsert rather than insert_or_replace
2015-04-30 15:21:31 +01:00
2366d28780
Don't needlessly join on state_events
2015-04-30 14:02:06 +01:00
1d7702833d
Make simple query rather than long one and then throw away half the results
2015-04-30 10:16:12 +01:00
b1ca784aca
Correctly decode, for sqlite and postgres, rows from pushers table
2015-04-29 19:41:14 +01:00
4a9dc5b2f5
pushkey' are also bytes.
2015-04-29 19:27:02 +01:00
0ade2712d1
Typo
2015-04-29 19:17:25 +01:00
50f96f256f
Also remove yield from within lock in the other generator
2015-04-29 19:17:00 +01:00
d2d61a8288
Fix deadlock in id_generators. No idea why this was an actual deadlock.
2015-04-29 19:15:23 +01:00
3864b3a8e6
Actually return rows
2015-04-29 18:07:36 +01:00
0618978238
Typo, args wrong way round
2015-04-29 18:04:35 +01:00
09177f4f2e
Decode buffers in same thread
2015-04-29 18:03:42 +01:00
472be88674
We store pusher data as bytes
2015-04-29 17:43:46 +01:00
12d381bd5d
Decode the data json in the storage layer (was moved but this part was missed)
2015-04-29 17:13:51 +01:00
f8c30faf25
Oops, update the contraint too
2015-04-29 16:58:42 +01:00
61cd5d9045
Be more postgressive
2015-04-29 16:57:14 +01:00
fb95035a65
Be postgressive
2015-04-29 16:53:41 +01:00
4669def000
Oops, forgot the schema delta file
2015-04-29 16:50:16 +01:00
0337eaf321
txn.execute doesn't return cursors
2015-04-29 16:43:39 +01:00
884fb88e28
txn.execute doesn't return cursors
2015-04-29 16:35:20 +01:00
d76c058eea
Fix invalid SQL to work in postgres land
2015-04-29 16:30:25 +01:00
c8ed9bd278
pushers table requires an access_token
2015-04-29 15:33:27 +01:00
970a9b9d2b
We can't use REPLACE when upgrading databases now we have postgres
2015-04-29 13:55:44 +01:00
64991b0c8b
Merge pull request #129 from matrix-org/key_distribution
...
Key distribution v2
2015-04-29 13:34:38 +01:00
e26a3d8d9e
bump database schema version
2015-04-29 13:32:32 +01:00
a9549fdce3
Use bytea rather than BLOB
2015-04-29 13:16:09 +01:00
4ad8b45155
Merge branch 'develop' into key_distribution
...
Conflicts:
synapse/config/homeserver.py
2015-04-29 13:15:14 +01:00
cd0864121b
Make postgres database error slightly more helpful
2015-04-29 12:12:25 +01:00
4932a7e2d9
Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__
2015-04-29 12:12:25 +01:00
a08bf11138
Appease PEP8
2015-04-29 11:44:48 +01:00
204132a998
Check that postgres database has correct charset set
2015-04-29 11:42:28 +01:00
327ca883ec
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-28 13:39:42 +01:00
18f8247701
Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit
2015-04-28 12:41:33 +01:00
9182f87664
Merge pull request #126 from matrix-org/csauth
...
Client / Server Auth Refactor
2015-04-28 11:00:27 +01:00
55e1bc8920
And don't bump the schema version unnecessarily
2015-04-28 10:54:15 +01:00
b96c133034
Add server_keys.sql to the current delta rather than creating a new delta
2015-04-28 10:50:00 +01:00
8558e1ec73
Make get_max_token into inlineCallbacks so that the lock works.
2015-04-27 15:19:44 +01:00
6f8e2d517e
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-27 14:41:40 +01:00
2732be83d9
Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.
2015-04-27 13:22:30 +01:00
e4c4664d73
Handle the fact that postgres databases can be restarted from under us
2015-04-27 12:40:49 +01:00
8a5be236e0
pep8
2015-04-27 11:49:18 +01:00
b02e1006b9
Run database check before daemonizing, at the cost of database hygiene.
2015-04-27 11:46:00 +01:00
f8152f2708
rename db method to be more informative
2015-04-27 10:16:26 +01:00
2f475bd5d5
pep8
2015-04-24 18:15:07 +01:00
a7b51f4539
Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266
2015-04-24 18:11:21 +01:00
eede182df7
Merge branch 'develop' into key_distribution
2015-04-24 10:35:49 +01:00
4e2f8b8722
Copyright notices
2015-04-24 10:35:29 +01:00
f30d47c876
Implement remote key lookup api
2015-04-22 14:21:08 +01:00
91c8f828e1
pep8
2015-04-17 19:56:04 +01:00
8db6832db8
Password reset, finally.
2015-04-17 19:53:47 +01:00
117f35ac4a
Add endpoint to get threepids from server
2015-04-17 17:20:18 +01:00
4eea5cf6c2
pep8
2015-04-17 16:46:45 +01:00
f96ab9d18d
make add3pid servlet work
2015-04-17 16:44:49 +01:00
e3417bbbe0
Revert needless change to storage.profile
2015-04-17 16:13:14 +01:00
cb03fafdf1
Merge branch 'develop' into csauth
2015-04-17 13:51:10 +01:00
0ec346d942
Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea
2015-04-17 11:10:20 +01:00
e1c0970c11
PEP8
2015-04-16 11:18:45 +01:00
b8092fbc82
Go back to storing JSON in TEXT
2015-04-16 11:17:52 +01:00
bc9e69e160
Move encoding and decoding of JSON into storage layer
2015-04-16 11:01:09 +01:00
ffad75bd62
Remove mysql/maria support
2015-04-15 17:00:50 +01:00
a429515bdd
Add methods for storing and retrieving the raw key json
2015-04-15 16:58:35 +01:00
cf04cedf21
Change full_schemas/11 to work with postgres
2015-04-15 16:53:47 +01:00
5b31afcbd1
Remove debug logging
2015-04-15 16:27:04 +01:00
6e91f14d09
Add missing yield
2015-04-15 16:25:07 +01:00
ed26e4012b
pushers table requires a unique id.
2015-04-15 16:24:14 +01:00
a5c72780e6
Don't pass in removed flag
2015-04-15 15:13:22 +01:00
4af32a2817
Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction
2015-04-15 14:51:21 +01:00
25d2914fba
Turn off persistance of sent_transactions.response_json
2015-04-15 14:09:35 +01:00
cce5d057d3
Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses
2015-04-15 11:43:25 +01:00
a971fa9d58
Use try..finally in contextlib.contextmanager
2015-04-15 10:25:43 +01:00
ded4128965
Use True for True rather than 1
2015-04-15 10:24:24 +01:00
f9e12f79ca
Add missing yield in storage func
2015-04-15 10:24:07 +01:00
c756dfeb14
Correctly identify deadlocks
2015-04-15 10:23:42 +01:00
32e14d8181
Return a sha256 fingerprint rather than the entire tls certificate
2015-04-14 19:10:09 +01:00
127fad17dd
Add postgres database engine
2015-04-14 14:50:29 +01:00
5a95cd4442
Rename user_ips.user -> user_id
2015-04-14 13:54:09 +01:00
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
3c741682e5
Correctly increment the _next_id initially
2015-04-14 09:54:44 +01:00
86fc9b617c
For backwards compat, make state_groups.id have a type of int, not varchar
2015-04-13 17:03:49 +01:00
93937b2b31
Remove duplicate rows
2015-04-10 18:47:03 +01:00
c5365dee56
Use case sensitive collations
2015-04-10 18:46:33 +01:00
4103b1c470
DROP indexes before recreating them
2015-04-10 18:46:09 +01:00
4d5b098626
Use LONGBLOB and TEXT for arbitary length rows
2015-04-10 18:45:09 +01:00
7ed2ec3061
Handle the fact that in sqlite binary data might be stored as unicode or bytes
2015-04-10 13:41:54 +01:00
ce797ad373
Bump schema version
2015-04-10 11:22:30 +01:00
7e863c51e6
Use unsigned bigint
2015-04-10 11:22:04 +01:00
0f12772e32
SQLite wants INTEGER and not BIGINT for primary keys
2015-04-10 11:16:09 +01:00
d5d4281647
Update full_schemas/16 to match delta files. Add delta/16 scripts
2015-04-10 10:59:46 +01:00
cda4a6f93f
Revert non-trivial changes to upgrade scripts
2015-04-10 10:19:50 +01:00
e2722f58ee
Fix schema again
2015-04-10 10:16:29 +01:00
a1665c5094
Revert non-trivial schema changes and move them to a new schema version.
2015-04-10 10:05:44 +01:00
2ded344620
Remove unused import
2015-04-09 13:46:06 +01:00
9707acfc40
Remove spurious spaces
2015-04-09 13:45:20 +01:00
8bf285e082
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-09 13:13:26 +01:00
8ad0f4912e
Stream ordering and out of order insertions.
...
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
2015-04-09 11:41:36 +01:00
279a547a8b
Use generic db exceptions rather than sqlite3 specific ones
2015-04-08 16:53:48 +01:00
83f5125d52
Merge pull request #114 from matrix-org/improve_get_event_cache
...
Improve get event cache
2015-04-08 16:50:27 +01:00
3887350e47
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-08 14:11:07 +01:00
c1b34af441
Move database timer logging to seperate logger
2015-04-08 13:31:19 +01:00
9a05795619
Retry transaction, not SQL query
2015-04-08 13:11:28 +01:00
24d8134ac1
Fix maria engine to correctly recognize deadlocks
2015-04-08 13:10:54 +01:00
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
49d6aa1394
Retry on deadlock
2015-04-07 15:28:37 +01:00
0bfa78b39b
PEP8
2015-04-07 12:16:05 +01:00
6bc9edd8b2
Fix prepare_sqlite3_database's convert_param_style
2015-04-07 12:13:58 +01:00
05a35d62b6
Bump database version
2015-04-07 12:10:15 +01:00
8574bf62dc
Add index to presence table
2015-04-07 12:09:36 +01:00
0af5f5efaf
Don't use multiple UNIQUE constraints; it will cause deadlocks
2015-04-07 12:08:35 +01:00
c8d3f6486d
Implement or_ignore flag on inserts
2015-04-07 12:06:22 +01:00
304111afd0
Don't use AUTOINCREMENT, use an in memory version
2015-04-07 12:05:36 +01:00
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
5e88a09a42
Add same user_id char checks as registration.
2015-03-31 14:00:25 +01:00
cf1fa59f4b
Use a sender localpart instead of a user ID.
...
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
2015-03-31 13:48:03 +01:00
c217504949
Edit SQL schema to use string IDs not ints. Use token as ID. Update tests.
2015-03-31 12:07:56 +01:00
b59aa74556
Fix tests and missing returns on deferreds.
2015-03-31 11:35:45 +01:00
d33ae65efc
Remove more reg/unreg methods. Read config not database for cache.
2015-03-31 11:00:00 +01:00
3e420aebd8
Revert "Add another @cached wrapper, this time on get_presence_state()"
...
This reverts commit ff1fa0fbf8
2015-03-27 16:16:58 +00:00
ff1fa0fbf8
Add another @cached wrapper, this time on get_presence_state()
2015-03-27 15:57:16 +00:00
a32e876ef4
Delete pushers when changing password
2015-03-26 13:40:16 +00:00
a198894bf7
Appease pep8
2015-03-26 11:53:58 +00:00
32206dde3f
Fixes from PR comments
2015-03-26 10:11:52 +00:00
4edcbcee3b
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
953e40f9dc
Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one
2015-03-25 19:12:16 +00:00
c1a256cc4c
Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API.
2015-03-25 19:06:22 +00:00
f173d40a32
Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key
2015-03-25 19:06:05 +00:00
1b988b051b
Store the rejected reason in (Frozen)Event structs
2015-03-25 19:06:05 +00:00
033a517feb
Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file
2015-03-25 19:06:05 +00:00
9ba6487b3f
Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()
2015-03-25 19:05:34 +00:00
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
c7023f2155
1) Pushers are now associated with an access token
...
2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
2015-03-24 17:24:15 +00:00
0ba393924a
Escape non printing ascii character
2015-03-24 16:31:52 +00:00
f488293d96
Don't reinsert into event_edges
2015-03-24 16:20:26 +00:00
1aa44939fc
Fix bugs in transactions storage
2015-03-24 16:20:05 +00:00
5a447098dd
Don't use room hosts table
2015-03-24 16:19:24 +00:00
9e98f1022a
Don't order by rowid
2015-03-24 16:19:01 +00:00
9115421ace
Use _simple_upsert
2015-03-24 16:17:39 +00:00
d19e79ecc9
Make deleting other access tokens when you change your password actually work
2015-03-24 15:33:48 +00:00
6e7131f02f
Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.
2015-03-23 15:38:56 +00:00
9a7f496298
Sanitize RoomMemberStore
2015-03-23 15:29:04 +00:00
d98660a60d
Implement password changing (finally) along with a start on making client/server auth more general.
2015-03-23 14:20:28 +00:00
d5272b1d2c
Use 'update or insert' rather than on 'conflict replace'
2015-03-23 14:02:34 +00:00
278149f533
Sanitize TransactionStore
2015-03-23 13:43:21 +00:00
72d8406409
Put a cache on get_aliases_for_room
2015-03-20 19:21:13 +00:00
a63b4f7101
Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale
2015-03-20 18:25:49 +00:00
0f86312c4c
Pull out the cache logic from the @cached wrapper into its own class we can reuse
2015-03-20 18:25:42 +00:00
b1022ed8b5
func(*EXPR) is valid Python syntax, really...
2015-03-20 17:47:45 +00:00
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
80cd08c190
PEP8
2015-03-20 16:03:25 +00:00
9517f4da4d
Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement
2015-03-20 16:02:47 +00:00
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
ceb61daa70
Add the tiniest of tiny one-element caches to get_room_events_max_id() as it's read every time someone hits eventstream
2015-03-20 15:44:06 +00:00
fce0114005
Start removing Tables
2015-03-20 15:05:44 +00:00
7e282a53a5
Tidy up _simple_... methods
2015-03-20 15:05:10 +00:00
91cb46191d
Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though
2015-03-20 14:59:45 +00:00
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
d288d273e1
Generate transaction id in code
2015-03-20 10:57:44 +00:00
d4f50f3ae5
decode_result takes an iterable
2015-03-20 10:57:26 +00:00
0d0610870d
Fix up schemas some more
2015-03-20 10:55:31 +00:00
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
58ed393235
Remove redundant key
2015-03-19 15:12:05 +00:00
fae059cc18
Fix up schemas to work with mariadb
2015-03-19 13:42:39 +00:00
bb24609158
Clean out event_forward_extremities table when the server rejoins the room
2015-03-18 11:19:47 +00:00
89036579ed
Update schema to work with mariadb
2015-03-18 11:18:49 +00:00
93978c5e2b
@cached() annotate get_user_by_token() - achieves a minor DB performance improvement
2015-03-17 17:24:51 +00:00
Erik Johnston