6e80c03d45
Merge branch 'develop' into dbkr/notifications_api
2016-08-20 00:16:18 +01:00
37adde32dc
Move defer.returnValue out of Measure
2016-08-19 18:23:44 +01:00
47dd8f02a1
Measure _get_event_from_row
2016-08-19 18:23:44 +01:00
e6784daf07
Merge pull request #1030 from matrix-org/erikj/cache_contexts
...
Add concept of cache contexts
2016-08-19 16:29:58 +01:00
c0d7d9d642
Rename to on_invalidate
2016-08-19 15:13:58 +01:00
dc76a3e909
Make cache_context an explicit option
2016-08-19 15:02:38 +01:00
f164fd9220
Move _bulk_get_push_rules_for_room to storage layer
2016-08-19 14:29:20 +01:00
ba214a5e32
Remove lru option
2016-08-19 14:17:11 +01:00
b770435389
Make get_new_events_for_appservice use indices
2016-08-19 10:28:42 +01:00
1e4217c90c
Explicit join
2016-08-18 17:53:44 +01:00
602c84cd9c
Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api
2016-08-18 17:15:26 +01:00
07229bbdae
Add appservice worker
2016-08-18 14:59:55 +01:00
9da84a9a1e
Make AppserviceHandler stream events from database
...
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
2016-08-18 11:54:41 +01:00
732cf72b86
Fix push_display_name_rename schema update
2016-08-17 18:12:21 +01:00
0d7cef0943
Merge pull request #1011 from matrix-org/dbkr/contains_display_name_override
...
Move display name rule
2016-08-17 16:15:45 +01:00
949629291c
Do it in storage function
2016-08-16 17:05:34 +01:00
ad42322257
Add migration script
...
To port existing rule actions & enable entries to the new name
2016-08-16 16:56:30 +01:00
25c2332071
Merge pull request #1010 from matrix-org/erikj/refactor_deletions
...
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
2016-08-16 11:37:53 +01:00
a2427981b7
Use cached get_user_by_access_token in slaves
2016-08-16 11:24:32 +01:00
6cbd1b495e
Merge branch 'fix_integrity_retry' of https://github.com/Ralith/synapse into Ralith-fix_integrity_retry
2016-08-16 10:50:24 +01:00
dc3a00f24f
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
2016-08-15 17:04:39 +01:00
89e786bd85
Doc get_next() context manager usage
2016-08-15 13:45:26 +01:00
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
99bbd90b0d
Always run txn.after_callbacks
2016-08-15 09:45:44 +01:00
8a57cc3123
Add missing database corruption recovery case
...
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
2016-08-14 11:50:22 -07:00
b4ecf0b886
Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api
2016-08-11 14:09:13 +01:00
a8b946decb
Raise 404 when couldn't find event
2016-08-05 15:31:02 +01:00
f0fa66f495
Delete more tables
2016-08-05 10:40:08 +01:00
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
8ad8490cff
Fix typo
2016-08-04 15:21:29 +01:00
59fa91fe88
Retry event persistence on IntegrityError
...
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.
This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
2016-08-04 15:02:15 +01:00
98385888b8
PEP8
2016-08-03 15:42:08 +01:00
68264d7404
Include device name in /keys/query response
...
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
530bc862dc
Merge branch 'rav/null_default_device_displayname' into develop
2016-08-03 14:30:32 +01:00
80ad710217
Remove other bit of deduplication
2016-08-03 13:25:59 +01:00
4fec5e57be
Default device_display_name to null
...
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
2016-08-03 11:53:00 +01:00
a8a32d2714
Ensure we only persist an event once at a time
2016-08-03 11:23:39 +01:00
5aa024e501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader
2016-07-29 11:24:56 +01:00
8dad08a950
Fix SQL to supply arguments in the same order
2016-07-29 09:57:13 +01:00
0a7d3cd00f
Create separate methods for getting messages to push
...
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.
The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.
The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
370135ad0b
Comment get_unread_push_actions_for_user_in_range function
2016-07-28 16:47:37 +01:00
6ede23ff1b
Add more key storage funcs into slave store
2016-07-28 15:41:26 +01:00
40e539683c
Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload
...
Make the device id on e2e key upload optional
2016-07-27 18:16:57 +01:00
26cb0efa88
SQL syntax fix
2016-07-27 12:30:22 +01:00
d47115ff8b
Delete e2e keys on device delete
2016-07-27 12:24:52 +01:00
2e3d90d67c
Make the device id on e2e key upload optional
...
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.
For active access_tokens without an associated device_id, we ought to register
the device in the devices table.
Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
2016-07-26 23:38:12 +01:00
87ffd21b29
Fix a couple of bugs in the transaction and keyring code
2016-07-26 19:19:08 +01:00
33d7776473
Fix typo
2016-07-26 13:32:15 +01:00
9ad8d9b17c
Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device
2016-07-26 13:29:46 +01:00
9c4cf83259
Merge pull request #948 from matrix-org/markjh/auth_fixes
...
Don't add rejections to the state_group, persist all rejections
2016-07-26 13:22:57 +01:00
8e02494166
Delete refresh tokens when deleting devices
2016-07-26 11:10:37 +01:00
a6f06ce3e2
Fix how push_actions are redacted.
2016-07-26 11:05:39 +01:00
efeb6176c1
Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically
2016-07-26 10:49:52 +01:00
012b4c1913
Implement updating devices
...
You can update the displayname of devices now.
2016-07-26 07:35:48 +01:00
436bffd15f
Implement deleting devices
2016-07-26 07:35:48 +01:00
1b3c3e6d68
Only update the events and event_json tables for rejected events
2016-07-25 18:44:30 +01:00
8f7f4cb92b
Don't add the events to forward extremities if the event is rejected
2016-07-25 17:13:37 +01:00
2623cec874
Don't add rejections to the state_group, persist all rejections
2016-07-25 16:12:16 +01:00
955ef1f06c
fix: defer.returnValue takes one argument
2016-07-25 16:04:45 +01:00
2ee4c9ee02
background updates: fix assert again
2016-07-25 16:01:46 +01:00
9dbd903f41
background updates: Fix assertion to do something
2016-07-25 14:05:23 +01:00
465117d7ca
Fix background_update tests
...
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00
363786845b
PEP8
2016-07-22 13:21:07 +01:00
ec5717caf5
Create index on user_ips in the background
...
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.
I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
2016-07-22 13:16:39 +01:00
aede7248ab
Split out a FederationReader process
2016-07-21 17:37:44 +01:00
c445f5fec7
storage/client_ips: remove some dead code
2016-07-21 11:58:47 +01:00
7314bf4682
Merge branch 'develop' into rav/get_devices_api
...
(pick up PR #938 in the hope of fixing the UTs)
2016-07-20 17:40:00 +01:00
bc8f265f0a
GET /devices endpoint
...
implement a GET /devices endpoint which lists all of the user's devices.
It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
ec041b335e
Record device_id in client_ips
...
Record the device_id when we add a client ip; it's somewhat redundant as we
could get it via the access_token, but it will make querying rather easier.
2016-07-20 16:41:03 +01:00
053e83dafb
More doc-comments
...
Fix some more comments on some things
2016-07-20 16:40:28 +01:00
e967bc86e7
Merge pull request #932 from matrix-org/rav/register_refactor
...
Further registration refactoring
2016-07-20 11:03:33 +01:00
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
f91faf09b3
Comment
2016-07-20 10:18:09 +01:00
40cbffb2d2
Further registration refactoring
...
* `RegistrationHandler.appservice_register` no longer issues an access token:
instead it is left for the caller to do it. (There are two of these, one in
`synapse/rest/client/v1/register.py`, which now simply calls
`AuthHandler.issue_access_token`, and the other in
`synapse/rest/client/v2_alpha/register.py`, which is covered below).
* In `synapse/rest/client/v2_alpha/register.py`, move the generation of
access_tokens into `_create_registration_details`. This means that the normal
flow no longer needs to call `AuthHandler.issue_access_token`; the
shared-secret flow can tell `RegistrationHandler.register` not to generate a
token; and the appservice flow continues to work despite the above change.
2016-07-19 18:46:19 +01:00
022b9176fe
schema fix
...
device_id should be text, not bigint.
2016-07-19 11:44:05 +01:00
f863a52cea
Add device_id support to /login
...
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.
Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.
Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
f52565de50
Fix /purge_history bug
...
This was caused by trying to insert duplicate backward extremeties
2016-07-15 14:23:15 +01:00
978fa53cc2
Pull out min stream_ordering from ex_outlier_stream
2016-07-15 10:22:30 +01:00
eec9609e96
event_backwards_extremeties may not be empty
2016-07-15 10:22:09 +01:00
d554ca5e1d
Add support for filters in paginate_room_events
2016-07-14 15:59:04 +01:00
b64aa6d687
Add sender and contains_url field to events table
2016-07-14 15:35:43 +01:00
dfde67a6fe
Add a comment explaining allow_none
2016-07-08 15:57:06 +01:00
10c843fcfb
Ensure that the guest user is in the database when upgrading accounts
2016-07-08 15:15:55 +01:00
58930da52b
Merge branch 'master' of github.com:matrix-org/synapse into develop
2016-07-08 14:11:37 +01:00
b92e7955be
Comment
2016-07-07 11:42:15 +01:00
c98e1479bd
Return 400 rather than 500
2016-07-07 11:41:07 +01:00
67f2c901ea
Add rest servlet. Fix SQL.
2016-07-06 15:56:59 +01:00
a17e7caeb7
Merge branch 'erikj/shared_secret' into erikj/test2
2016-07-06 14:46:31 +01:00
be3548f7e1
Remove spurious txn
2016-07-05 17:46:51 +01:00
4adf93e0f7
Fix for postgres
2016-07-05 17:34:25 +01:00
651faee698
Add an admin option to shared secret registration
2016-07-05 17:30:22 +01:00
8f8798bc0d
Add ReadWriteLock for pagination and history prune
2016-07-05 15:30:25 +01:00
dd2ccee27d
Fix typo
2016-07-05 14:06:07 +01:00
b6b0132ac7
Make get_events_around more efficient on sqlite3
2016-07-05 13:55:18 +01:00
d44d11d864
Use true/false for boolean parameter inclusive to avoid potential for sqli, and possibly make the code clearer
2016-07-05 10:39:13 +01:00
2d21d43c34
Add purge_history API
2016-07-05 10:28:51 +01:00
0fb76c71ac
Use different SQL for postgres and sqlite3 for when using multicolumn indexes
2016-07-04 19:44:55 +01:00
a67bf0b074
Add storage function to purge history for a room
2016-07-04 16:02:50 +01:00
f18d7546c6
Use a query that postgresql optimises better for get_events_around
2016-07-04 15:48:25 +01:00
f328d95cef
Feature: Add deactivate account admin API
...
Allows server admins to "deactivate" accounts, which:
- Revokes all access tokens
- Removes all threepids
- Removes password
The API is a POST to `/admin/deactivate/<user_id>`
2016-06-30 15:40:58 +01:00
6783534a0f
Merge pull request #886 from matrix-org/markjh/async_commit
...
Optionally make committing to postgres asynchronous.
2016-06-29 15:21:58 +01:00
a70688445d
Implement purge_media_cache admin API
2016-06-29 14:57:59 +01:00
314b146b2e
Track approximate last access time for remote media
2016-06-29 11:41:20 +01:00
aa3a4944d5
more pep8
2016-06-24 11:45:23 +01:00
870c45913e
Use similar naming we use in email notifs for push
...
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
b5fb7458d5
Actually we need to order these properly
...
otherwise we'll end up returning the wrong 20
2016-06-22 18:07:14 +01:00
f73fdb04a6
Style
2016-06-22 17:51:40 +01:00
3a4120e49a
Put most recent 20 messages in notif
...
Fixes https://github.com/vector-im/vector-web/issues/1648
2016-06-22 17:47:18 +01:00
d5fb561709
Optionally make committing to postgres asynchronous.
...
Useful when running tests when you don't care whether the server
will lose data that it claims that it has committed.
2016-06-20 17:53:38 +01:00
41e4b2efea
Add the create_profile method back since the tests use it
2016-06-17 19:20:47 +01:00
9f1800fba8
Remove registered_users from the distributor.
...
The only place that was observed was to set the profile. I've made it
so that the profile is set within store.register in the same transaction
that creates the user.
This required some slight changes to the registration code for upgrading
guest users, since it previously relied on the distributor swallowing errors
if the profile already existed.
2016-06-17 19:14:16 +01:00
1a815fb04f
Don't hit DB for noop replications queries
2016-06-08 11:33:30 +01:00
0b2158719c
Remove dead code.
...
Loading push rules now happens in the datastore, so we can remove
the methods that loaded them outside the datastore.
The ``waiting_for_join_list`` in federation handler is populated by
anything, so can be removed.
The ``_get_members_events_txn`` method isn't called from anywhere
so can be removed.
2016-06-07 15:07:11 +01:00
84379062f9
Fix AS retries, but with correct ordering
2016-06-07 10:24:50 +01:00
310197bab5
Fix AS retries
2016-06-07 09:34:50 +01:00
e4134c5e13
Merge pull request #841 from matrix-org/erikj/event_counter
...
Add metric counter for number of persisted events
2016-06-06 14:17:40 +01:00
7aa778fba9
Add metric counter for number of persisted events
2016-06-06 11:58:09 +01:00
70aee0717c
Add events to cache when we persist them
2016-06-06 11:34:53 +01:00
cffe46408f
Don't rely on options when inserting event into cache
2016-06-03 18:25:21 +01:00
10ea3f46ba
Change the way we cache events
2016-06-03 17:57:50 +01:00
05e01f21d7
Remove event fetching from DB threads
2016-06-03 17:22:13 +01:00
eef541a291
Move insert_client_ip to a separate class
2016-06-03 14:42:35 +01:00
d06f2a229e
Merge pull request #828 from matrix-org/erikj/joined_hosts_for_room
...
Make get_joined_hosts_for_room use get_users_in_room
2016-06-03 11:50:30 +01:00
722472b48c
Merge pull request #825 from matrix-org/erikj/cache_push_rules
...
Load push rules in storage layer so that they get cached
2016-06-03 11:44:32 +01:00
b09f348530
Merge pull request #824 from matrix-org/markjh/slaved_presence_store
...
Add a slaved store for presence
2016-06-03 11:26:33 +01:00
ccb56fc24b
Make get_joined_hosts_for_room use get_users_in_room
2016-06-03 11:20:23 +01:00
6a0afa582a
Load push rules in storage layer, so that they get cached
2016-06-03 11:10:00 +01:00
3ae915b27e
Add a slaved store for presence
2016-06-03 11:05:53 +01:00
9c26b390a2
Only get local users
2016-06-03 11:04:31 +01:00
065e739d6e
Merge pull request #811 from matrix-org/erikj/state_users_in_room
...
Use state to calculate get_users_in_room
2016-06-03 10:58:27 +01:00
696d7c5937
Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room
...
Add get_users_with_read_receipts_in_room cache
2016-06-03 10:58:24 +01:00
80f34d7b57
Fix setting the _clock in SQLBaseStore
2016-06-02 15:23:56 +01:00
a04a2d043c
Merge pull request #807 from matrix-org/erikj/push_rules_cache
...
Ensure we always return boolean in push rules
2016-06-01 18:07:48 +01:00
0f06b496d1
Merge pull request #806 from matrix-org/erikj/hash_cache
...
Cache get_event_reference_hashes
2016-06-01 18:07:42 +01:00
c8285564a3
Use state to calculate get_users_in_room
2016-06-01 15:25:25 +01:00
d60eed0710
Limit number of notifications in an email notification
2016-06-01 11:45:43 +01:00
43db0d9f6a
Add get_users_with_read_receipts_in_room cache
2016-06-01 10:54:32 +01:00
dea9f20f8c
Force boolean
2016-05-31 17:24:30 +01:00
c8c5bf950a
Fix synapse/storage/schema/delta/30/as_users.py
2016-05-31 17:10:40 +01:00
c9ca285d33
Merge pull request #805 from matrix-org/erikj/push_rules_cache
...
Fix GET /push_rules
2016-05-31 16:42:21 +01:00
1d4ee854e2
Fix typo
2016-05-31 15:45:53 +01:00
cca0093fa9
Change fix
2016-05-31 15:44:08 +01:00
aefd2d1cbc
Cache get_event_reference_hashes
2016-05-31 15:32:32 +01:00
10de8c2631
Merge pull request #804 from matrix-org/erikj/push_rules_cache
...
Add caches to bulk_get_push_rules*
2016-05-31 15:04:40 +01:00
e5b0bbcd33
Add caches to bulk_get_push_rules*
2016-05-31 13:46:58 +01:00
85b992f621
Fix to allow start with postgres
2016-05-27 10:44:44 +01:00
37b7e84620
Include the ts the notif was received at
2016-05-24 11:33:32 +01:00
a24bc5b2dc
Add GET /notifications API
2016-05-23 18:33:51 +01:00
Matthew Hodgson