95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer
...
Remove ReplicationLayer and user Client/Server directly
2018-03-13 15:45:08 +00:00
5d8476d8ff
Merge pull request #2981 from matrix-org/erikj/factor_remote_leave
...
Factor out _remote_reject_invite in RoomMember
2018-03-13 15:44:56 +00:00
47ce527f45
Add room_id to the response of `rooms/{roomId}/join`
...
Fixes #2349
2018-03-13 14:48:12 +01:00
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers
...
Don't build handlers on workers unnecessarily
2018-03-13 13:46:38 +00:00
cb9f8e527c
s/replication_client/federation_client/
2018-03-13 13:26:52 +00:00
cea462e285
s/replication_server/federation_server
2018-03-13 13:22:21 +00:00
bf8e97bd3c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave
2018-03-13 13:17:08 +00:00
ea3442c15c
Add docstring
2018-03-13 13:16:21 +00:00
16469a4f15
Merge pull request #2980 from matrix-org/erikj/rm_priv
...
Make RoomMemberHandler functions private that can be
2018-03-13 13:11:11 +00:00
c82111a55f
Merge pull request #2982 from matrix-org/erikj/fix_extra_users
...
extra_users is actually a list of UserIDs
2018-03-13 13:11:04 +00:00
da87791975
Merge pull request #2983 from matrix-org/erikj/rename_register_3pid
...
Refactor get_or_register_3pid_guest
2018-03-13 13:10:52 +00:00
99e9b4f26c
Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix
...
RoomMembershipRestServlet doesn't handle /forget
2018-03-13 13:10:44 +00:00
f5160d4a3e
RoomMembershipRestServlet doesn't handle /forget
...
Due to the order we register the REST handlers `/forget` was handled by
the correct handler.
2018-03-13 12:12:55 +00:00
8b3573a8b2
Refactor get_or_register_3pid_guest
2018-03-13 12:08:58 +00:00
299fd740c7
Merge pull request #2975 from matrix-org/rav/measure_persist_events
...
Add Measure block for persist_events
2018-03-13 12:06:25 +00:00
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props
...
Move property setting from ReplicationLayer to base classes
2018-03-13 11:45:25 +00:00
f43b6d6d9b
Fix docstring types
2018-03-13 11:29:35 +00:00
0f942f68c1
Factor out _remote_reject_invite in RoomMember
2018-03-13 11:22:45 +00:00
d0fcc48f9d
extra_users is actually a list of UserIDs
2018-03-13 11:20:06 +00:00
31becf4ac3
Make functions private that can be
2018-03-13 11:15:16 +00:00
d023ecb810
Don't build handlers on workers unnecessarily
2018-03-13 11:08:10 +00:00
ea7b3c4b1b
Remove unused ReplicationLayer
2018-03-13 11:00:04 +00:00
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
e05bf34117
Move property setting from ReplicationLayer to FederationBase
2018-03-13 10:51:30 +00:00
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
e48c7aac4d
Add transactional API to history purge
...
Make the purge request return quickly, and allow scripts to poll for updates.
2018-03-12 16:22:55 +00:00
1708412f56
Return an error when doing two purges on a room
...
Queuing up purges doesn't sound like a good thing.
2018-03-12 16:22:54 +00:00
b984dd0b73
Merge pull request #2961 from matrix-org/rav/run_in_background
...
Factor run_in_background out from preserve_fn
2018-03-12 16:19:13 +00:00
ba1d08bc4b
Merge pull request #2965 from matrix-org/rav/request_logging
...
Add a metric which increments when a request is received
2018-03-12 09:45:33 +00:00
58dd148c4f
Add some docstrings to help figure this out
2018-03-09 18:05:41 +00:00
88541f9009
Add a metric which increments when a request is received
...
It's useful to know when there are peaks in incoming requests - which isn't
quite the same as there being peaks in outgoing responses, due to the time
taken to handle requests.
2018-03-09 16:30:26 +00:00
dbe80a286b
refactor JsonResource
...
rephrase the OPTIONS and unrecognised request handling so that they look
similar to the common flow.
2018-03-09 16:22:16 +00:00
20f40348d4
Factor run_in_background out from preserve_fn
...
It annoys me that we create temporary function objects when there's really no
need for it. Let's factor the gubbins out of preserve_fn and start using it.
2018-03-08 11:50:11 +00:00
a56d54dcb7
Fix up log message
2018-03-07 11:55:31 +00:00
02a1296ad6
Fix typo
2018-03-07 11:55:31 +00:00
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
8ffaacbee3
Merge pull request #2949 from krombel/use_bcrypt_checkpw
...
use bcrypt.checkpw
2018-03-06 11:56:06 +00:00
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
ed9b5eced4
use bcrypt.checkpw
...
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago)
This makes use of that with enhancements which might get introduced
by that
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-05 18:02:59 +01:00
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
f394f5574d
Merge pull request #2929 from matrix-org/erikj/split_regististration_store
...
Split registration store
2018-03-05 11:33:07 +00:00
af7ed8e1ef
Return 401 for invalid access_token on logout
...
Signed-off-by: Duncan Klug <dklug@ucmerced.edu>
2018-03-02 22:01:27 -08:00
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
d960d23830
Add missing yield during 3pid signature checks
2018-03-02 11:03:18 +00:00
1a6c7cdf54
Merge pull request #2928 from matrix-org/erikj/read_marker_caches
...
Fix typo in getting replication account data processing
2018-03-01 17:56:14 +00:00
89b7232ff8
Fix typo in getting replication account data processing
2018-03-01 17:50:30 +00:00
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
65cf454fd1
Remove unused DataStore
2018-03-01 17:27:53 +00:00
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
483e8104db
Merge pull request #2926 from matrix-org/erikj/member_handler_move
...
Move RoomMemberHandler out of Handlers
2018-03-01 17:01:25 +00:00
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
529c026ac1
Move back to hs.is_mine
2018-03-01 16:49:12 +00:00
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
4bc4236faf
Merge pull request #2922 from matrix-org/erikj/split_room_store
...
Split up RoomStore
2018-03-01 15:55:01 +00:00
2324124a72
Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable
...
Rewrite make_deferred_yieldable avoiding inlineCallbacks
2018-03-01 15:39:10 +00:00
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
784f036306
Move RoomMemberHandler out of Handlers
2018-03-01 14:36:50 +00:00
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
3a75de923b
Rewrite make_deferred_yieldable avoiding inlineCallbacks
...
... because (a) it's actually simpler (b) it might be marginally more
performant?
2018-03-01 12:40:05 +00:00
126b9bf96f
Log in the correct places
2018-03-01 12:05:33 +00:00
157298f986
Don't do preserve_fn for every request
2018-03-01 11:59:45 +00:00
89f90d808a
Add some logging
2018-03-01 11:59:16 +00:00
8ded8ba2c7
Make repl send_event idempotent and retry on timeouts
...
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
2018-03-01 11:20:34 +00:00
f381d63813
Check event auth on the worker
2018-03-01 10:18:37 +00:00
6b8604239f
Correctly send ratelimit and extra_users params
2018-03-01 10:08:39 +00:00
f756f961ea
Fixup comments
2018-03-01 10:05:27 +00:00
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
199dba6c15
Merge pull request #2897 from matrix-org/erikj/split_account_data
...
Split AccountDataStore and TagStore
2018-02-23 11:23:11 +00:00
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
573712da6b
Update comments
2018-02-21 11:29:49 +00:00
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
27939cbb0e
Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed
...
Delete from push_actions_staging in federation too
2018-02-21 11:00:06 +00:00
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
d874d4f2d7
Delete from push_actions_staging in federation too
2018-02-20 11:37:52 +00:00
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
324c3e9399
Merge pull request #2868 from matrix-org/erikj/refactor_media_storage
...
Make store_file use store_into_file
2018-02-20 11:31:24 +00:00
3acd616979
Update pynacl dependency to 1.2.1 or higher
...
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
2018-02-19 10:45:22 +01:00
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
5fb347fc41
Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log
...
Don't log errors propogated from send_event
2018-02-15 12:31:49 +00:00
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
ef344b10e5
Don't log errors propogated from send_event
2018-02-15 11:03:49 +00:00
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
92c52df702
Make store_file use store_into_file
2018-02-14 17:55:18 +00:00
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
8fd1a32456
Fix typos in purge api & doc
...
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
2018-02-13 13:09:39 +00:00
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml
...
Allow use of higher versions of saml2
2018-02-13 12:24:08 +00:00
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store
...
Tell storage providers about new file so they can upload
2018-02-12 13:52:58 +00:00
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
5fa571a91b
Tell storage providers about new file so they can upload
2018-02-07 13:35:08 +00:00
50fe92cd26
Move presence handling into handle_new_client_event
...
As we want to have it run on the main synapse instance
2018-02-07 10:34:09 +00:00
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
e3624fad5f
Remove pointless ratelimit check
...
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
2018-02-07 10:30:25 +00:00
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation
...
Split event creation into a separate handler
2018-02-06 17:01:17 +00:00
3e1e69ccaf
Update copyright
2018-02-06 16:40:38 +00:00
770b2252ca
s/_create_new_client_event/create_new_client_event/
2018-02-06 16:40:30 +00:00
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
3fa362502c
Update places where we create events
2018-02-05 16:01:48 +00:00
5ff3d23564
Split event creation into a separate handler
2018-02-05 16:01:48 +00:00
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics
...
montoring metrics for number of cache evictions
2018-02-05 16:34:36 +01:00
bc496df192
report metrics on number of cache evictions
2018-02-05 15:34:01 +00:00
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor
...
Ensure media is in local cache before thumbnailing
2018-02-05 11:28:52 +00:00
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
9c9356512e
Merge pull request #2845 from matrix-org/rav/urlcache_error_handling
...
Handle url_previews with no content-type
2018-02-02 15:27:52 +01:00
18eae413af
Merge pull request #2842 from matrix-org/rav/state_resolution_handler
...
Factor out resolve_state_groups to a separate handler
2018-02-02 15:27:35 +01:00
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
9dcd667ac2
Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring
...
Docstring fixes
2018-02-02 15:26:49 +01:00
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
d5352cbba8
Handle url_previews with no content-type
...
avoid failing with an exception if the remote server doesn't give us a
Content-Type header.
Also, clean up the exception handling a bit.
2018-02-02 00:53:46 +00:00
14737ba495
doc arg types for _seperate
2018-02-01 12:41:34 +00:00
e15d4ea248
More docstring fixes
...
Fix a couple of errors in docstrings
2018-02-01 12:41:34 +00:00
a18828c129
Fix docstring for StateHandler.resolve_state_groups
...
The return type was a complete lie, so fix it
2018-02-01 12:41:34 +00:00
6da4c4d3bd
Factor out resolve_state_groups to a separate handler
...
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
2018-02-01 12:40:04 +00:00
0cbda53819
Rename resolve_state_groups -> resolve_state_groups_for_events
...
(to make way for a method that actually just does the state group resolution)
2018-02-01 12:40:00 +00:00
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
78e7e05188
Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail
...
Remove spurious log argument
2018-01-31 22:18:46 +01:00
ad48dfe73d
Merge pull request #2835 from matrix-org/rav/remove_event_type_param
...
Remove unused "event_type" param on state.get_current_state_ids
2018-01-31 22:17:57 +01:00
518a74586c
Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling
...
Improve exception handling in persist_event
2018-01-31 22:17:38 +01:00
d1fe4db882
Merge pull request #2833 from matrix-org/rav/pusher_hacks
...
Logging and metrics for the http pusher
2018-01-31 22:16:59 +01:00
421d68ca8c
Merge pull request #2817 from matrix-org/rav/http_conn_pool
...
Use a connection pool for the SimpleHttpClient
2018-01-31 22:14:22 +01:00
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
af19f5e9aa
Remove spurious log argument
...
... which would cause scary-looking and unhelpful errors in the log on dns fail
2018-01-30 17:52:03 +00:00
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
adfc0c9539
docstring for get_current_state_ids
2018-01-29 17:39:55 +00:00
d413a2ba98
Remove unused "event_type" param on state.get_current_state_ids
...
this param doesn't seem to be used, and is a bit pointless anyway because it
can easily be replicated by the caller. It is also horrible, because it changes
the return type of the method.
2018-01-29 17:06:57 +00:00
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
03dd745fe2
Better logging when pushes fail
2018-01-29 15:49:06 +00:00
e051abd20b
add appid/device_display_name to to pusher logging
2018-01-29 15:04:16 +00:00
02ba118f81
Increase http conn pool size
2018-01-29 14:30:15 +00:00
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
4b0f06e99c
Merge pull request #2830 from matrix-org/rav/factor_out_get_conn
...
Factor out get_db_conn to HomeServer base class
2018-01-27 17:56:25 +01:00
e98f0f9112
Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist
...
Fix server 500 on public rooms call when no rooms exist
2018-01-26 20:22:40 +00:00
25adde9a04
Factor out get_db_conn to HomeServer base class
...
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
2018-01-26 00:56:49 +00:00
73560237d6
add white space line
2018-01-26 00:15:10 +00:00
86c4f49a31
rather than try reconstruct the results object, better to guard against the xrange step argument being 0
2018-01-26 00:12:02 +00:00
f632083576
fix return type, should be a dict
2018-01-25 23:52:17 +00:00
6c6e197b0a
fix PEP8 violation
2018-01-25 23:47:46 +00:00
d02e43b15f
remove white space
2018-01-25 23:29:46 +00:00
349c739966
synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing
2018-01-25 23:28:44 +00:00
9a72b70630
fix thinko on 3pid whitelisting
2018-01-24 11:07:47 +01:00
25e2456ee7
Merge pull request #2816 from matrix-org/rav/metrics_comments
...
Add some comments about the reactor tick time metric
2018-01-23 19:39:26 +00:00
d32385336f
add ?ts massaging for ASes ( #2754 )
...
blindly implement ?ts for AS. untested
2018-01-23 09:59:06 +01:00
4528dd2443
Fix logging and add user_id
2018-01-22 20:15:42 +00:00
93efd7eb04
logging and debug for http pusher
2018-01-22 18:14:10 +00:00
ab9f844aaf
Add federation_domain_whitelist option ( #2820 )
...
Add federation_domain_whitelist
gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
d84f65255e
Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid
...
add registrations_require_3pid and allow_local_3pids
2018-01-22 13:57:22 +00:00
a94d9b6b82
Appease the linter
...
These are ids anyways, not mxc uris.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
5552ed9a7f
Add an admin route to get all the media in a room
...
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00
2c8526cac7
Use a connection pool for the SimpleHttpClient
...
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
2018-01-20 00:55:44 +00:00
87b7d72760
Add some comments about the reactor tick time metric
2018-01-19 23:51:04 +00:00
49fce04624
fix typo (thanks sytest)
2018-01-19 19:55:38 +00:00
b0d9e633ee
Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs
...
Use the right path for url_preview thumbnails
2018-01-19 18:57:15 +00:00
ad7ec63d08
Use the right path for url_preview thumbnails
...
This was introduced by #2627 : we were overwriting the original media for url
previews with the thumbnails :/
(fixes https://github.com/vector-im/riot-web/issues/6012 , hopefully)
2018-01-19 18:29:39 +00:00
62d7d66ae5
oops, check all login types
2018-01-19 18:23:56 +00:00
8fe253f19b
fix PR nitpicking
2018-01-19 18:23:45 +00:00
293380bef7
trailing commas
2018-01-19 15:38:53 +00:00
447f4f0d5f
rewrite based on PR feedback:
...
* [ ] split config options into allowed_local_3pids and registrations_require_3pid
* [ ] simplify and comment logic for picking registration flows
* [ ] fix docstring and move check_3pid_allowed into a new util module
* [ ] use check_3pid_allowed everywhere
@erikjohnston PTAL
2018-01-19 15:33:55 +00:00
9d332e0f79
fix up v1, and improve errors
2018-01-19 00:53:58 +00:00
0af58f14ee
fix pep8
2018-01-19 00:33:51 +00:00
28a6ccb49c
add registrations_require_3pid
...
lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with.
Typically useful for stopping people from registering with non-work emails
2018-01-19 00:19:58 +00:00
cd871a3057
Fix storage provider bug introduced when renamed to store_local
2018-01-18 18:37:59 +00:00
8ff6726c0d
Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config
...
Make storage providers configurable
2018-01-18 18:33:57 +00:00
d69768348f
Fix passing wrong config to provider constructor
2018-01-18 17:14:05 +00:00
8e85220373
Remove duplicate directory test
2018-01-18 17:12:35 +00:00
3fe2bae857
Missing staticmethod
2018-01-18 17:11:45 +00:00
aae77da73f
Fixup comments
2018-01-18 17:11:29 +00:00
b6dc7044a9
Merge pull request #2804 from matrix-org/erikj/file_consumer
...
Add decent impl of a FileConsumer
2018-01-18 16:31:33 +00:00
9a89dae8c5
Fix typo in thumbnail resource causing access times to be incorrect
2018-01-18 15:06:24 +00:00
0af5dc63a8
Make storage providers more configurable
2018-01-18 14:07:21 +00:00
d57765fc8a
Fix bugs in block metrics
...
... which I introduced in #2785
2018-01-18 12:24:42 +00:00
2cf6a7bc20
Use better file consumer
2018-01-18 12:00:46 +00:00
4a53f3a3e8
Ensure media is in local cache before thumbnailing
2018-01-18 12:00:46 +00:00
be0dfcd4a2
Do logcontexts correctly
2018-01-18 11:57:57 +00:00
1432f7ccd5
Move test stuff to tests
2018-01-18 11:57:57 +00:00
2f18a2647b
Make all fields private
2018-01-18 11:57:54 +00:00
d6af5512bb
Merge pull request #2809 from matrix-org/rav/metrics_errors
...
better exception logging in callbackmetrics
2018-01-18 11:46:37 +00:00
ce236f8ac8
better exception logging in callbackmetrics
...
when we fail to render a metric, give a clue as to which metric it was
2018-01-18 11:30:49 +00:00
dc519602ac
Ensure we registerProducer isn't called twice
2018-01-18 11:07:17 +00:00
17b54389fe
Fix _notify_empty typo
2018-01-18 11:05:34 +00:00
28b338ed9b
Move definition of paused_producer to __init__
2018-01-18 11:04:41 +00:00
a177325b49
Fix comments
2018-01-18 11:02:43 +00:00
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
...
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
1224612a79
Log room when doing state resolution
...
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
bc67e7d260
Add decent impl of a FileConsumer
...
Twisted core doesn't have a general purpose one, so we need to write one
ourselves.
Features:
- All writing happens in background thread
- Supports both push and pull producers
- Push producers get paused if the consumer falls behind
2018-01-17 16:43:03 +00:00
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed
...
Keep track of last access time for local media
2018-01-17 16:39:02 +00:00
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql
...
fix SQL when searching all users
2018-01-17 16:27:54 +00:00
2d9ab533f9
fix SQL when searching all users
2018-01-17 15:58:52 +00:00
390093d45e
Split resolve_events into two functions
...
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
a7e4ff9cca
Merge pull request #2795 from matrix-org/rav/track_db_scheduling
...
Track DB scheduling delay per-request
2018-01-17 14:29:37 +00:00
f884cfffb9
Merge pull request #2797 from matrix-org/rav/user_id_checking
...
Sanity checking for user ids
2018-01-17 14:29:12 +00:00
a5213df1f7
Sanity checking for user ids
...
Check the user_id passed to a couple of APIs for validity, to avoid
"IndexError: list index out of range" exception which looks scary and results
in a 500 rather than a more useful error.
Fixes #1432 , among other things
2018-01-17 14:28:54 +00:00
e8f7541d3f
Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling
2018-01-17 14:01:57 +00:00
fb6563b4be
Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis
...
Track db txn time in millisecs
2018-01-17 13:52:42 +00:00
1954e867b4
Merge pull request #2796 from matrix-org/rav/fix_closed_connection_errors
...
Fix 'NoneType' object has no attribute 'writeHeaders'
2018-01-17 13:52:24 +00:00
11ab2f56f5
Merge branch 'develop' into rav/fix_closed_connection_errors
2018-01-17 11:25:11 +00:00
0486a7814a
Merge branch 'develop' into rav/rework_run_interaction
2018-01-17 11:24:38 +00:00
90c14da992
Merge branch 'develop' into rav/db_txn_time_in_millis
2018-01-17 11:19:55 +00:00
38506773eb
Merge remote-tracking branch 'origin/develop' into rav/optimise_logging_context
2018-01-17 10:57:13 +00:00
300edc2348
Update last access time when thumbnails are viewed
2018-01-17 10:24:43 +00:00
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
3cb2dabaad
Merge pull request #2789 from matrix-org/erikj/fix_thumbnails
...
Fix thumbnailing remote files
2018-01-17 10:22:20 +00:00
d728c47142
Add docstring
2018-01-17 10:06:14 +00:00
936482d507
Fix 'NoneType' object has no attribute 'writeHeaders'
...
Avoid throwing a (harmless) exception when we try to write an error response to
an http request where the client has disconnected.
This comes up as a CRITICAL error in the logs which tends to mislead people
into thinking there's an actual problem
2018-01-16 17:58:16 +00:00
Erik Johnston