4ecb4bdac9
wip attempt at caching
2018-08-02 22:41:05 +01:00
74b1d46ad9
do mau checks based on monthly_active_users table
2018-08-02 16:57:35 +01:00
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
14fa9d4d92
Avoid extra db lookups
...
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
4a6725d9d1
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-02 11:04:18 +01:00
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
...
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
08281fe6b7
self.db_conn unused
2018-08-01 23:26:24 +01:00
c21d82bab3
normalise reaping query
2018-08-01 23:24:38 +01:00
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
...
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
7ff44d9215
improve clarity
2018-08-01 16:17:00 +01:00
da7785147d
Python 3: Convert some unicode/bytes uses ( #3569 )
2018-08-02 00:54:06 +10:00
303f1c851f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits
2018-08-01 13:42:50 +01:00
a6d7b74915
update docs
2018-08-01 13:39:14 +01:00
4e5ac901dd
clean up
2018-08-01 12:03:57 +01:00
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
6023cdd227
remove errant print
2018-08-01 10:27:17 +01:00
7931393495
make count_monthly_users async synapse/handlers/auth.py
2018-08-01 10:21:56 +01:00
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
...
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
0bc9b9e397
reinstate explicit include of EventsWorkerStore
2018-07-31 13:11:04 +01:00
fef7e58ac6
actually close conn
2018-07-30 22:29:44 +01:00
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
251e6c1210
limit register and sign in on number of monthly users
2018-07-30 15:55:57 +01:00
143f1a2532
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store
2018-07-30 09:56:18 +01:00
e9b2d047f6
make /context lazyload & filter aware ( #3567 )
...
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
...
Stop populating events.content
2018-07-26 22:54:08 +01:00
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
...
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
...
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
bd4b25f4d0
Remove some redundant joins on event_edges.room_id
...
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.
event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
21e878ebb6
Make EventStore inherit from EventFederationStore
...
(since it uses methods therein)
Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
bc7944e6d2
switch missing_types to be a bool
2018-07-25 23:36:31 +01:00
2565804030
Merge branch 'develop' into matthew/filter_members
2018-07-25 17:27:49 +01:00
7d9fb88617
incorporate more review.
2018-07-25 16:33:50 +01:00
78a691d005
Split out DB writes in federation handler
...
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
...
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
...
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
...
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
...
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
454f59b7ad
Merge branch 'develop' into matthew/filter_members
2018-07-24 14:03:37 +01:00
223341205e
Don't require to_delete to have event_ids
2018-07-24 14:02:40 +01:00
e22700c3dd
consider non-filter_type types as wildcards, thus missing from the state-group-cache
2018-07-24 13:59:07 +01:00
d19fba3655
Merge branch 'develop' into matthew/filter_members
2018-07-24 12:39:54 +01:00
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
30bfed5aa5
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-24 11:46:09 +01:00
97acd385a3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta
2018-07-24 11:32:13 +01:00
0fa73e4a63
Remove unnecessary if
2018-07-24 11:19:23 +01:00
ff5426f6b8
Speed up _calculate_state_delta
2018-07-24 10:55:11 +01:00
a678145010
Merge branch 'develop' into rav/logcontext_fixes
2018-07-24 10:43:30 +01:00
8dff6e0322
Logcontext fixes
...
Fix some random logcontext leaks.
2018-07-24 00:37:17 +01:00
69fb5dbdab
fix idiocy
2018-07-24 00:04:44 +01:00
1938cffaea
Add some measure blocks to persist_events
...
... to help us figure out where 40% of CPU is going
2018-07-23 23:48:19 +01:00
efcdacad7d
handle case where types is [] on postgres correctly
2018-07-23 22:41:05 +01:00
adfe29ec0b
Merge branch 'develop' into matthew/filter_members
2018-07-23 19:21:37 +01:00
254fb430d1
incorporate review
2018-07-23 19:21:20 +01:00
8b9f164fff
Comments
2018-07-23 17:43:01 +01:00
50c60e5fad
Only get cached state from context in persist_event
...
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
2018-07-23 17:21:40 +01:00
4f5cc8e4e7
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-23 17:15:12 +01:00
dae6dc1e77
Remove redundant checks on room forgottenness
...
Fixes #3550
2018-07-23 17:13:34 +01:00
e42510ba63
Use new getters
2018-07-23 13:17:22 +01:00
9b34f3ea3a
Merge branch 'develop' into matthew/sync_deleted_devices
2018-07-23 10:03:28 +01:00
c1bf2b587e
add trailing comma
2018-07-23 09:56:23 +01:00
2f558300cc
fix thinkos; unbreak tests
2018-07-19 19:22:27 +01:00
bcaec2915a
incorporate review
2018-07-19 19:03:50 +01:00
924eb34d94
add a filtered_types param to limit filtering to specific types
2018-07-19 18:32:02 +01:00
be3adfc331
merge develop pydoc for _get_state_for_groups
2018-07-19 11:26:04 +01:00
9e40834f74
yes, we do need to invalidate the device_id_exists_cache when deleting a remote device
2018-07-19 11:15:10 +01:00
f1a15ea206
revert 00bc979
...
... we've fixed the things that caused the warnings, so we should reinstate the
warning.
2018-07-19 11:14:20 +01:00
1ffb7bec20
Merge remote-tracking branch 'origin/release-v0.33.0' into develop
2018-07-19 11:12:33 +01:00
c0685f67c0
spell out that include_deleted_devices requires include_all_devices
2018-07-19 10:59:02 +01:00
00bc979137
Disable logcontext warning
...
Temporary workaround to #3518 while we release 0.33.0.
2018-07-19 10:51:15 +01:00
667fba68f3
Run things as background processes
...
This fixes #3518 , and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
12ec58301f
shift to using an explicit deleted flag on m.device_list_update EDUs
...
and generally make it work.
2018-07-12 11:39:43 +01:00
0456e05977
Merge pull request #3505 from matrix-org/erikj/receipts_cahce
...
Use stream cache in get_linearized_receipts_for_room
2018-07-12 09:46:29 +01:00
aff1dfdf3d
Update return value docstring
2018-07-12 09:45:37 +01:00
5797f5542b
WIP to announce deleted devices over federation
...
Previously we queued up the poke correctly when the device was deleted,
but then the actual EDU wouldn't get sent, as the device was no longer known.
Instead, we now send EDUs for deleted devices too if there's a poke for them.
2018-07-12 01:32:39 +01:00
6ccefef07a
Use 'is not None' and add comments
2018-07-10 18:12:39 +01:00
ea752bdd99
s/becuase/because/g
2018-07-10 17:58:18 +01:00
05f5dabc10
Use stream cache in get_linearized_receipts_for_room
...
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
c3c29aa196
Attempt to include db threads in cpu usage stats ( #3496 )
...
Let's try to include time spent in the DB threads in the per-request/block cpu
usage metrics.
2018-07-10 16:12:36 +01:00
e31e5dee38
Add CPU metrics for _fetch_event_list
...
add a Measure block on _fetch_event_list, in the hope that we can better
measure CPU usage here.
2018-07-09 18:15:54 +01:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
3905c693c5
Invalidate cache on correct thread
2018-07-02 11:36:44 +01:00
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
028490afd4
Fix error on deleting users pending deactivation
...
Use simple_delete instead of simple_delete_one as commented
2018-06-26 10:52:52 +01:00
244484bf3c
Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
...
This reverts commit 1d009013b3
2018-06-25 13:42:55 +01:00
1d009013b3
Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
...
This reverts commit ce0d911156b4a5d767a9
2018-06-22 16:35:10 +01:00
9850f66abe
Deleting from event_push_actions needs to use an index
2018-06-22 15:54:48 +01:00
200e11c5bf
Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable
...
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:18:51 +01:00
f8272813a9
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:12:26 +01:00
1d7ad11747
Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation
...
Make push actions rotation configurable
2018-06-22 15:07:31 +01:00
ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility
...
Support hiding events from deleted users
2018-06-22 15:06:44 +01:00
f79abda87f
Merge pull request #3427 from matrix-org/erikj/remove_filters
...
remove dead filter_events_for_clients
2018-06-22 14:45:42 +01:00
75dc3ddeab
Make push actions rotation configurable
2018-06-22 14:44:37 +01:00
43e02c409d
Disable partial state group caching for wildcard lookups
...
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
2018-06-22 11:52:07 +01:00
240f192523
Merge pull request #3382 from matrix-org/rav/optimise_state_groups
...
Optimise state_group_cache update
2018-06-22 11:20:20 +01:00
70e6501913
Merge pull request #3419 from matrix-org/rav/events_per_request
...
Log number of events fetched from DB
2018-06-22 11:17:56 +01:00
0495fe0035
Indirect evt_count updates via method call
...
so that we can stub it for the sentinel and not have a billion failing UTs
2018-06-22 10:42:28 +01:00
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
b088aafcae
Log number of events fetched from DB
...
When we finish processing a request, log the number of events we fetched from
the database to handle it.
[I'm trying to figure out which requests are responsible for large amounts of
event cache churn. It may turn out to be more helpful to add counts to the
prometheus per-request/block metrics, but that is an extension to this code
anyway.]
2018-06-21 06:15:03 +01:00
3ff8a619f5
UserErasureStore
...
to store which users have been erased
2018-06-12 09:52:22 +01:00
bd348f0af6
remove dead filter_events_for_clients
...
This is only used by filter_events_for_client, so we can simplify the whole
thing by just doing one user at a time, and removing a dead storage function to
boot.
2018-06-12 09:51:31 +01:00
eb32b2ca20
Optimise state_group_cache update
...
(1) matrix-org-hotfixes has removed the intern calls; let's do the same here.
(2) remove redundant iteritems() so we can used an optimised db update.
2018-06-11 22:56:11 +01:00
187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind
...
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
c96d882a02
Merge branch 'develop' into matthew/filter_members
2018-06-10 12:26:14 +03:00
d3a8c9c55e
Fix sql error in _get_state_groups_from_groups
...
If this was called with a `(type, None)` entry in types (which is supposed to
return all state of type `type`), it would explode with a sql error.
2018-06-06 14:19:01 +01:00
f29b41fde9
Merge pull request #3324 from matrix-org/rav/remove_dead_method
...
Remove was_forgotten_at
2018-06-04 18:18:08 +01:00
042eedfa2b
Add hacky cache factor override system
2018-06-04 15:39:28 +01:00
28f09fcdd5
Merge branch 'develop' into matthew/filter_members
2018-06-04 00:09:17 +03:00
c1f4118bb6
Remove was_forgotten_at
...
This is unused. IT MUST DIE!!!1
̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢
̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *
2018-06-01 18:21:49 +01:00
71477f3317
Merge pull request #3264 from matrix-org/neil/sign-up-stats
...
daily user type phone home stats
2018-06-01 13:42:01 +00:00
9f797a24a4
Handle RRs which arrive before their events
2018-06-01 14:01:43 +01:00
857e6fd8b6
Ignore depth when updating read-receipts
...
Order read receipts by stream ordering instead of depth
2018-06-01 12:18:11 +01:00
4986b084f8
remove unnecessary INSERT
2018-06-01 10:50:40 +01:00
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy ( #3307 )
2018-05-31 19:03:47 +10:00
872cf43516
Merge pull request #3303 from NotAFile/py3-memoryview
...
use memoryview in py3
2018-05-30 12:51:42 +10:00
debff7ae09
Merge pull request #3281 from NotAFile/py3-six-isinstance
...
remaining isintance fixes
2018-05-30 12:44:46 +10:00
4a9cbdbc15
Exempt AS-registered users from doing gdpr
2018-05-29 19:54:32 +01:00
ab0ef31dc7
create users index on creation_ts
2018-05-29 17:51:08 +01:00
558f3d376a
create index in background
2018-05-29 17:47:55 +01:00
c379acd4fd
bump version
2018-05-29 17:47:28 +01:00
4b9d0cde97
add remaining memoryview changes
2018-05-29 17:42:43 +02:00
1afafb3497
use memoryview in py3
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-29 17:14:34 +02:00
a6c8f7c875
add pydoc
2018-05-29 01:09:55 +01:00
7a6df013cc
merge develop
2018-05-29 00:25:22 +01:00
a2eb5db4a0
update metrics to be in seconds
2018-05-28 19:10:27 +10:00
754826a830
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-28 18:57:23 +10:00
cc1349c06a
Merge pull request #3279 from NotAFile/py3-more-iteritems
...
more six iteritems
2018-05-24 14:23:13 -05:00
dd068ca979
remaining isintance fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:55:08 +02:00
17a70cf6e9
Misc. py3 fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
6c16a4ec1b
more iteritems
2018-05-24 20:19:06 +02:00
7ea07c7305
Merge pull request #3278 from NotAFile/py3-storage-base
...
Py3 storage/_base.py
2018-05-24 13:08:09 -05:00
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
...
replace some iteritems with six
2018-05-24 13:04:07 -05:00
36501068d8
Merge pull request #3247 from NotAFile/py3-misc
...
Misc Python3 fixes
2018-05-24 12:58:37 -05:00
095292304f
Py3 storage/_base.py
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 18:24:12 +02:00
77a23e2e05
Merge remote-tracking branch 'origin/develop' into dbkr/unbind
2018-05-24 16:20:53 +01:00
2c7866d664
Hit the 3pid unbind endpoint on deactivation
2018-05-23 14:38:56 +01:00
53cc2cde1f
cleanup
2018-05-22 17:32:57 -05:00
a17e901f4d
Remove unused string formatting param
2018-05-22 18:24:32 +01:00
5494c1d71e
Don't support limitless pagination
...
The pagination storage function supported not specifiying a limit on the
number of events returned. This was triggered when using the search or
context API with a limit of zero, which the storage function took to
mean not being limited.
2018-05-22 18:15:21 +01:00
d8cb7225d2
daily user type phone home stats
2018-05-22 18:09:09 +01:00
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
fcc525b0b7
rest of the changes
2018-05-21 19:48:57 -05:00
933bf2dd35
replace some iteritems with six
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
dcc235b47d
use stand-in value if maxint is not available
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:44 +02:00
644aac5f73
Tighter filtering for user_daily_visits
2018-05-18 17:10:35 +01:00
ef466b3a13
fix psql compatability bug
2018-05-18 15:51:21 +01:00
ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred
...
Fix error in handling receipts
2018-05-18 11:23:11 +01:00
fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token
...
Make purge_history operate on tokens
2018-05-18 10:35:23 +01:00
8b1c856d81
Fix error in handling receipts
...
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):
> TypeError: argument of type 'ObservableDeferred' is not iterable
fixes #3234
2018-05-18 09:15:35 +01:00
f7906203f6
Merge pull request #3212 from matrix-org/erikj/epa_stream
...
Use stream rather depth ordering for push actions
2018-05-17 12:01:21 +01:00
680530cc7f
Clarify comment
2018-05-16 11:47:29 +01:00
43e6e82c4d
Comments
2018-05-16 11:13:31 +01:00
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
c945af8799
Move and rename variable
2018-05-16 10:52:06 +01:00
be11a02c4f
remove empty line
2018-05-16 10:45:40 +01:00
31c2502ca8
style and further contraining query
2018-05-16 09:46:43 +01:00
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
05ac15ae82
Limit query load of generate_user_daily_visits
...
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
5f27ed75ad
Make purge_history operate on tokens
...
As we're soon going to change how topological_ordering works
2018-05-15 16:23:50 +01:00
37dbee6490
Use events_to_purge table rather than token
2018-05-15 16:23:47 +01:00
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
f077e97914
instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day
2018-05-14 13:50:58 +01:00
8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate
...
Part user from rooms on account deactivate
2018-05-14 11:31:48 +01:00
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
6406b70aeb
Use stream rather depth ordering for push actions
...
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
217bc53c98
Many docstrings
2018-05-10 12:20:40 +01:00
bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync
...
Refactor sync APIs to reuse pagination API
2018-05-09 16:16:56 +01:00
5adb75bcba
Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination
...
Refactor get_recent_events_for_room return type
2018-05-09 16:07:14 +01:00
7ce98804ff
Fix up comment
2018-05-09 15:42:39 +01:00
cddf91c8b9
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change
2018-05-09 15:32:07 +01:00
9896dab8f6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination
2018-05-09 15:31:33 +01:00
75552d2148
Update comments
2018-05-09 15:15:38 +01:00
46df23f581
Add the schema file
2018-05-09 15:07:54 +01:00
7e8726b8fb
Part deactivated users in the background
...
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
2018-05-09 14:54:28 +01:00
c0e08dc45b
Remove unused code path from member change DB func
...
The function is never called without a from_key, so we can remove all
the handling for that scenario.
2018-05-09 14:31:32 +01:00
e2accd7f1d
Refactor sync APIs to reuse pagination API
...
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
2018-05-09 13:43:39 +01:00
e5ab9cd24b
Don't unnecessarily require token to be stream token
...
This allows calling the `get_recent_event_ids_for_room` function in more
situations.
2018-05-09 11:58:35 +01:00
c4af4c24ca
Refactor get_recent_events_for_room return type
...
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
2018-05-09 11:55:34 +01:00
05e0a2462c
Refactor pagination DB API to return concrete type
...
This makes it easier to document what is being returned by the storage
functions and what some functions expect as arguments.
2018-05-09 11:34:24 +01:00
7dd13415db
Remove unused from_token param
2018-05-09 10:58:16 +01:00
27cf170558
Refactor recent events func to use pagination func
...
This also removes a cache that is unlikely to ever get hit.
2018-05-09 10:55:55 +01:00
23ec51c94c
Fix up comments and make function private
2018-05-09 09:55:19 +01:00
696f532453
Reuse existing pagination code for context API
2018-05-08 16:20:19 +01:00
3e6d306e94
Parse tokens before calling DB function
2018-05-08 16:18:58 +01:00
274b8c6025
Only fetch required fields from database
2018-05-08 16:15:25 +01:00
06c0d0ed08
Split paginate_room_events storage function
2018-05-08 16:14:26 +01:00
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
...
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
31c7c29d43
Fix up grammar
2018-05-03 10:38:58 +01:00
dd1a832419
remove user agent from data model, will just join on user_ips
2018-05-01 12:13:49 +01:00
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
6e10eed28e
Refactor event storage to not require state
...
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.
We also add a check to see if the state groups of the old extremities
are the same as the new ones.
2018-04-27 11:38:02 +01:00
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
7ec8e798b4
Fix media admin APIs
2018-04-26 11:31:22 +01:00
fb6015d0a6
pep8
2018-04-25 17:56:11 +01:00
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
22881b3d69
Also fix reindexing of search
2018-04-25 15:32:04 +01:00
ba3166743c
Fix quarantine media admin API
2018-04-25 15:11:18 +01:00
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
...
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
878995e660
Replace Queue with six.moves.queue
...
and a six.range change which I missed the last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
415aeefd89
Format docstring
2018-04-12 12:07:09 +01:00
23a7f9d7f4
Doc we raise on unknown event
2018-04-12 11:20:51 +01:00
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
Neil Johnson