49ef8ec399
Fix a cache-invalidation bug for worker-based deployments ( #5920 )
...
Some of the caches on worker processes were not being correctly invalidated
when a room's state was changed in a way that did not affect the membership
list of the room.
We need to make sure we send out cache invalidations even when no memberships
are changing.
2019-08-28 10:18:16 +01:00
322ccac33f
Allow schema deltas to be engine-specific ( #5911 )
...
* Allow schema deltas to be engine-specific
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Code style (Black)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 11:53:21 +01:00
1a7e6eb633
Add Admin API capability to set adminship of a user ( #5878 )
...
Admin API: Set adminship of a user
2019-08-27 10:14:00 +01:00
d1e0b91083
Code style (Black)
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:39:11 +01:00
aefa76f5cd
Allow schema deltas to be engine-specific
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:14:00 +01:00
ae38e0569f
Ignore consent for support users
2019-08-23 09:15:10 +01:00
8767b63a82
Propagate opentracing contexts through EDUs ( #5852 )
...
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
9a6f2be572
Opentrace e2e keys ( #5855 )
...
Add opentracing tags and logs for e2e keys
2019-08-22 11:28:12 +01:00
c9f11d09fc
Add missing index on users_in_public_rooms. ( #5894 )
2019-08-22 10:43:13 +01:00
119aa31b10
Servlet to purge old rooms ( #5845 )
2019-08-22 10:42:59 +01:00
4dab867288
Drop some unused tables. ( #5893 )
...
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
8a5f6ed130
Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index
...
add the version field to the index for e2e_room_keys
2019-08-16 17:45:50 -07:00
c058aeb88d
update set_e2e_room_key to agree with fixed index
2019-08-14 18:02:58 -07:00
b7f7cc7ace
add the version field to the index for e2e_room_keys
2019-08-14 17:14:40 -07:00
af187805b3
Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop
...
Handle pusher being deleted during processing.
2019-08-13 14:08:29 +01:00
fb3469f53a
Clarify docstring
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-13 10:17:23 +01:00
2546f32b90
Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses
...
Don't unnecessarily block notifying of new events.
2019-08-13 09:36:25 +01:00
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions
...
Fix handling of redactions of redactions
2019-08-12 12:25:19 +01:00
bf4db42920
Don't unnecessarily block notifying of new events.
...
When persisting events we calculate new stream orderings up front.
Before we notify about an event all events with lower stream orderings
must have finished being persisted.
This PR moves the assignment of stream ordering till *after* calculated
the new current state and split the batch of events into separate chunks
for persistence. This means that if it takes a long time to calculate
new current state then it will not block events in other rooms being
notified about.
This should help reduce some global pauses in the events stream which
can last for tens of seconds (if not longer), caused by some
particularly expensive state resolutions.
2019-08-06 13:32:02 +01:00
5d018d23f0
Have ClientReaderSlavedStore inherit RegistrationStore ( #5806 )
...
Fixes #5803
2019-08-01 13:54:56 +01:00
d02e41dcb2
Handle pusher being deleted during processing.
...
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
2019-08-01 13:44:12 +01:00
123c04daa7
Don't recreate column
2019-07-30 16:42:48 +01:00
15056ca208
Fix current_state_events membership background update.
...
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
2019-07-30 14:51:41 +01:00
4e97eb89e5
Handle loops in redaction events
2019-07-30 00:00:34 +01:00
448bcfd0f9
recursively fetch redactions
2019-07-30 00:00:34 +01:00
e6a6c4fbab
split _get_events_from_db out of _enqueue_events
2019-07-29 23:15:15 +01:00
c9964ba600
Return dicts from _fetch_event_list
2019-07-29 23:15:15 +01:00
7c8c3b8437
Merge pull request #5774 from matrix-org/erikj/fix_rejected_membership
...
Fix room summary when rejected events are in state
2019-07-29 17:15:15 +01:00
3e013b7c8e
Merge pull request #5752 from matrix-org/erikj/forgotten_user
...
Remove some more joins on room_memberships
2019-07-29 17:15:01 +01:00
df3a5db629
Expand comment
2019-07-29 16:40:25 +01:00
105e7f6ed3
Remove lost comment
2019-07-29 16:09:48 +01:00
45df38e61b
Fix current_state bg update to work on old SQLite
2019-07-29 13:04:10 +01:00
d74595e2ca
Merge branch 'master' into develop
2019-07-26 12:39:33 +01:00
14c24c9037
Fix room summary when rejected events are in state
...
Annoyingly, `current_state_events` table can include rejected events,
in which case the membership column will be null. To work around this
lets just always filter out null membership for now.
2019-07-26 10:11:36 +01:00
0f2ecb961e
Fix DoS when there is a cycle in redaction events
...
Make sure that synapse doesn't explode when a redaction redacts itself, or
there is a larger cycle.
2019-07-26 06:36:48 +00:00
26d742fed6
Merge pull request #5767 from matrix-org/rav/redactions/cross_room_id
...
log when a redaction attempts to redact an event in a different room
2019-07-25 18:49:56 +01:00
b1605cdd23
log when a redaction attempts to redact an event in a different room
2019-07-25 18:26:20 +01:00
f30a71a67b
Stop trying to fetch events with event_id=None. ( #5753 )
...
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.
I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
2019-07-24 13:16:18 +01:00
0c4a99607e
Remove join when calculating room summaries.
2019-07-24 11:49:15 +01:00
62921fb53e
Remove join on room_memberships when fetching rooms for user.
2019-07-24 11:45:58 +01:00
32768e96d4
Add function to get all forgotten rooms for user
...
This will allow us to efficiently filter out rooms that have been
forgotten in other queries without having to join against the
`room_memberships` table.
2019-07-24 11:44:23 +01:00
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
b2a629ef49
Speed up current state background update.
...
Turns out that storing huge JSON arrays in the progress JSON isn't
something that postgres particularly likes.
2019-07-23 10:01:30 +01:00
66f5ff72fd
Add `user_type` to returned fields in admin API user list endpoints
...
Mostly user type will be empty (normal user) but there is also the
"support" user type.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-07-22 15:29:18 +03:00
7704873cb8
Merge pull request #5720 from matrix-org/erikj/transactions_upsert
...
Use upsert when updating destination retry interval
2019-07-19 16:51:16 +01:00
d7bd9651bc
Merge pull request #5713 from matrix-org/erikj/use_cache_for_filtered_state
...
Delegate to cached version when using get_filtered_current_state_ids
2019-07-19 16:30:49 +01:00
5c07c97c09
Merge pull request #5706 from matrix-org/erikj/add_memberships_to_current_state
...
Add membership column to current_state_events table
2019-07-19 16:30:33 +01:00
2410335507
Use upsert when updating destination retry interval
2019-07-19 13:40:24 +01:00
bd2e1a2aa8
LoggingTransaction accepts None for callback lists.
...
Its a bit disingenuousto give LoggingTransaction lists to append
callbacks to if we're not going to run the callbacks.
2019-07-19 13:36:04 +01:00
ebc5ed1296
Update comment for new column
2019-07-19 13:29:02 +01:00
10523241d8
Delegate to cached version when using get_filtered_current_state_ids
...
In the case where it gets called with `StateFilter.all()`
2019-07-18 15:17:39 +01:00
8e1ada9e6f
Use the current_state_events.membership column
2019-07-18 14:16:01 +01:00
059d8c1a4e
Track if current_state_events.membership is up to date
2019-07-18 14:16:01 +01:00
c618a5d348
Add background update for current_state_events.membership column
2019-07-18 14:16:01 +01:00
6de09e07a6
Add membership column to current_state_events table.
...
It turns out that doing a join is surprisingly expensive for the DB to
do when room_membership table is larger than the disk cache.
2019-07-18 14:15:57 +01:00
9c70a02a9c
Ignore redactions of m.room.create events ( #5701 )
2019-07-17 19:08:02 +01:00
2091c91fde
More refactoring in `get_events_as_list` ( #5707 )
...
We can now use `_get_events_from_cache_or_db` rather than going right back to
the database, which means that (a) we can benefit from caching, and (b) it
opens the way forward to more extensive checks on the original event.
We now always require the original event to exist before we will serve up a
redaction.
2019-07-17 17:34:13 +01:00
375162b3c3
Fix redaction authentication ( #5700 )
...
Ensures that redactions are correctly authenticated for recent room versions.
There are a few things going on here:
* `_fetch_event_rows` is updated to return a dict rather than a list of rows.
* Rather than returning multiple copies of an event which was redacted
multiple times, it returns the redactions as a list within the dict.
* It also returns the actual rejection reason, rather than merely the fact
that it was rejected, so that we don't have to query the table again in
`_get_event_from_row`.
* The redaction handling is factored out of `_get_event_from_row`, and now
checks if any of the redactions are valid.
2019-07-17 16:52:02 +01:00
65c5592b8e
Refactor `get_events_as_list` ( #5699 )
...
A couple of changes here:
* get rid of a redundant `allow_rejected` condition - we should already have filtered out any rejected
events before we get to that point in the code, and the redundancy is confusing. Instead, let's stick in
an assertion just to make double-sure we aren't leaking rejected events by mistake.
* factor out a `_get_events_from_cache_or_db` method, which is going to be important for a
forthcoming fix to redactions.
2019-07-17 16:49:19 +01:00
d86321300a
Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data
...
Add basic function to get all data for a user out of synapse
2019-07-15 10:04:02 +01:00
5f158ec039
Implement access token expiry ( #5660 )
...
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
db0a50bc40
Fixup docstrings
2019-07-12 16:59:59 +01:00
24aa0e0a5b
fix typo: backgroud -> background
2019-07-12 15:29:40 +01:00
78a1cd36b5
small typo fix ( #5655 )
2019-07-11 13:33:23 +01:00
953dbb7980
Remove access-token support from RegistrationStore.register ( #5642 )
...
The 'token' param is no longer used anywhere except the tests, so let's kill
that off too.
2019-07-10 16:26:49 +01:00
d88421ab03
Include the original event in /relations ( #5626 )
...
When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.
2019-07-09 13:43:08 +01:00
80cc82a445
Remove support for invite_3pid_guest. ( #5625 )
...
This has never been documented, and I'm not sure it's ever been used outside
sytest.
It's quite a lot of poorly-maintained code, so I'd like to get rid of it.
For now I haven't removed the database table; I suggest we leave that for a
future clearout.
2019-07-05 16:47:58 +01:00
7f0d8e4288
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data
2019-07-05 14:08:21 +01:00
9ccea16d45
Assume key existence. Update docstrings
2019-07-05 14:07:56 +01:00
a6a776f3d8
remove dead transaction persist code ( #5622 )
...
this hasn't done anything for years
2019-07-05 12:59:42 +01:00
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
91753cae59
Fix a number of "Starting txn from sentinel context" warnings ( #5605 )
...
Fixes #5602 , #5603
2019-07-03 09:31:27 +01:00
8ee69f299c
Add basic function to get all data for a user out of synapse
2019-07-02 12:09:04 +01:00
729f5a4fb6
Review comments
2019-06-27 16:06:23 +01:00
f335e77d53
Use batch_iter and correct docstring
2019-06-26 19:39:46 +01:00
806a06daf2
Rename get_users_whose_devices_changed
2019-06-26 19:39:19 +01:00
a2f6d31a63
Refactor get_user_ids_changed to pull less from DB
...
When a client asks for users whose devices have changed since a token we
used to pull *all* users from the database since the token, which could
easily be thousands of rows for old tokens.
This PR changes this to only check for changes for users the client is
actually interested in.
Fixes #5553
2019-06-26 12:03:44 +01:00
e59a8cd2e5
Merge pull request #5499 from matrix-org/rav/cleanup_metrics
...
Cleanups and sanity-checking in cpu and db metrics
2019-06-24 17:12:54 +01:00
1793de6c6d
black
2019-06-24 11:16:13 +01:00
5097aee740
Merge branch 'develop' into rav/cleanup_metrics
2019-06-24 10:00:13 +01:00
c753c098dd
Merge pull request #5498 from matrix-org/rav/fix_clock_reversal
...
Use monotonic clock where possible for metrics
2019-06-24 09:55:12 +01:00
0e8b35f7b0
Fix "Unexpected entry in 'full_schemas'" log warning ( #5509 )
...
There is a README.txt which always sets off this warning, which is a bit
alarming when you first start synapse. I don't think we need to warn about
this.
2019-06-21 15:11:57 +01:00
f8bd30af2a
Black
2019-06-20 13:10:06 +01:00
45f28a9d2f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/histogram_extremities
2019-06-20 11:59:14 +01:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
f682af052d
Simplify PerformanceCounters.update interface
...
we already have the duration for the update, so may as well use it rather than
passing extra params around and recalculating it.
2019-06-19 21:18:38 +01:00
68128d5626
Remove unused _get_event_counters
...
This has been redundant since cdb3757942
2019-06-19 21:14:25 +01:00
15bf32e062
Use monotonic clock where possible for metrics
...
Fixes intermittent errors observed on Apple hardware which were caused by
time.clock() appearing to go backwards when called from different threads.
Also fixes a bug where database activity times were logged as 1/1000 of their
correct ratio due to confusion between milliseconds and seconds.
2019-06-19 21:09:43 +01:00
e0be8d7016
Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events
...
Add experimental option to reduce extremities.
2019-06-19 13:19:18 +01:00
65787b0f7c
Add descriptions and remove redundant set(..)
2019-06-19 11:49:39 +01:00
092e36457a
Fix typo in deactivation background job
2019-06-19 11:09:41 +01:00
b42f90470f
Add experimental option to reduce extremities.
...
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.
THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
499d4a32cd
Add metrics for len of new extremities persisted.
...
Of new events being persisted add metrics for total size of forward
extremities and number of unchanged, "stale" extremities.
2019-06-17 17:02:48 +01:00
eba7caf09f
Remove Postgres 9.4 support ( #5448 )
2019-06-18 00:59:00 +10:00
9ca4ae7131
Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive
...
Prometheus histograms are cumalative
2019-06-14 18:21:42 +01:00
e0b77b004d
Fix background job for deactivated flag
2019-06-14 16:00:45 +01:00
6d56a694f4
Don't send renewal emails to deactivated users
2019-06-14 15:05:56 +01:00
3ed595e327
Prometheus histograms are cumalative
2019-06-14 14:07:32 +01:00
d0530382ee
Track deactivated accounts in the database ( #5378 )
2019-06-14 13:18:24 +01:00
6312d6cc7c
Expose statistics on extrems to prometheus ( #5384 )
2019-06-13 22:40:52 +10:00
Richard van der Hoff