Commit Graph

446 Commits (db2b105d69fa331bb3f050df82266314f61577ea)

Author SHA1 Message Date
Patrick Cloke 2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
reivilibre 4c2096599c
Make the `get_global_account_data_by_type_for_user` cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Patrick Cloke 68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Patrick Cloke 0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
David Robertson c9eb678b73
Remove debug logging for #4422 (#11693)
as per
https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
2022-01-06 12:44:36 +00:00
Patrick Cloke b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Richard van der Hoff 2215954147
Various opentracing enhancements (#11619)
* Wrap `auth.get_user_by_req` in an opentracing span

give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.

This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.

* Emit opentracing span for encoding json responses

This can be a significant time sink.

* Rename all sync spans with a prefix

* Write an opentracing span for encoding sync response

* opentracing span to group generate_room_entries

* opentracing spans within sync.encode_response

* changelog

* Use the `trace` decorator instead of context managers
2021-12-21 11:10:36 +00:00
David Robertson fd2dadb815
Adjust _get_rooms_changed comments (#11550)
C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
2021-12-10 19:19:48 +00:00
Patrick Cloke d93362d87f
Add a constant for receipt types (m.read). (#11531)
And expand some type hints in the receipts storage module.
2021-12-08 12:26:29 -05:00
David Robertson 14d593f72d
Refactors in `_generate_sync_entry_for_rooms` (#11515)
* Move sync_token up to the top
* Pull out _get_ignored_users
* Try to signpost the body of `_generate_sync_entry_for_rooms`
* Pull out _calculate_user_changes

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-07 12:42:05 +00:00
David Robertson 9c55dedc8c
Correctly ignore invites from ignored users (#11511) 2021-12-07 11:24:31 +00:00
David Robertson d26808dd85
Comments on the /sync tentacles (#11494)
This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-02 20:58:32 +00:00
Patrick Cloke a19d01c3d9
Support filtering by relations per MSC3440 (#11236)
Adds experimental support for `relation_types` and `relation_senders`
fields for filters.
2021-11-09 08:10:58 -05:00
Patrick Cloke 4054dfa409
Add type hints for event streams. (#10856) 2021-09-21 13:34:26 -04:00
Patrick Cloke b3590614da
Require type hints in the handlers module. (#10831)
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.

This also removes the unused construct_auth_difference method
from the FederationHandler.
2021-09-20 08:56:23 -04:00
David Robertson b4c1af8cea Merge branch 'release-v1.43' into develop 2021-09-17 12:15:17 +01:00
Erik Johnston 9a6f4a684f
Cleanup opentracing logging for syncs (#10828)
We added a bunch of spans in #10704, but this ended up adding a lot of
redundant spans for rooms where nothing changed, so instead we only
start the span if there might be something interesting going on.
2021-09-15 17:14:25 +01:00
Patrick Cloke 8c7a531e27
Use direct references for some configuration variables (part 2) (#10812) 2021-09-15 08:34:52 -04:00
Azrenbeth f1c6b76418
Add logging to help debug #9424 (#10704) 2021-09-06 16:08:25 +01:00
Patrick Cloke bd7d398b05
Additional type hints for the sync REST servlet. (#10666) 2021-08-23 08:14:42 -04:00
Patrick Cloke 000aa89be6
Do not include rooms with an unknown room version in a sync response. (#10644)
A user will still see this room if it is in a local cache, but it will
not reappear if clearing the cache and reloading.
2021-08-19 11:12:55 -04:00
Patrick Cloke bec01c0758
Convert room member storage tuples to attrs. (#10629)
Instead of using namedtuples. This helps with asserting type hints
and code completion.
2021-08-18 09:22:07 -04:00
Patrick Cloke 6a5f8fbcda
Use auto-attribs for attrs classes for sync. (#10630) 2021-08-18 07:27:32 -04:00
Richard van der Hoff 4b10880da3
Make sync response cache time configurable. (#10513) 2021-08-03 14:45:04 +01:00
Andrew Morgan 74d09a43d9
Always communicate device OTK counts to clients (#10485)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-07-27 14:36:38 +01:00
Jonathan de Jong 95e47b2e78
[pyupgrade] `synapse/` (#10348)
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of #9744
2021-07-19 15:28:05 +01:00
Jonathan de Jong 98aec1cc9d
Use inline type hints in `handlers/` and `rest/`. (#10382) 2021-07-16 18:22:36 +01:00
Richard van der Hoff fcf3c7032b
Ensure that we do not cache empty sync responses after a timeout (#10158)
Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
2021-06-17 16:23:11 +01:00
Sorunome d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Richard van der Hoff fa1db8f156
Delete completes to-device messages earlier in /sync (#10124)
I hope this will improve
https://github.com/matrix-org/synapse/issues/9564.
2021-06-07 09:19:06 +01:00
Erik Johnston 4deaebfe00
Make /sync do less state res (#10102) 2021-06-02 15:48:17 +01:00
Erik Johnston d0aee697ac
Use get_current_users_in_room from store and not StateHandler (#9910) 2021-05-05 16:49:34 +01:00
Richard van der Hoff 294c675033
Remove `synapse.types.Collection` (#9856)
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Jonathan de Jong 2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Patrick Cloke 44bb881096
Add type hints to expiring cache. (#9730) 2021-04-06 08:58:18 -04:00
Erik Johnston 33548f37aa
Improve tracing for to device messages (#9686) 2021-04-01 17:08:21 +01:00
Patrick Cloke 7e8dc9934e
Add a type hints for service notices to the HomeServer object. (#9675) 2021-03-24 06:48:46 -04:00
Erik Johnston ad721fc559
Fix bad naming of storage function (#9637)
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
2021-03-17 13:20:08 +00:00
Jonathan de Jong d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Patrick Cloke aee10768d8 Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"
This reverts commit f5c93fc993.

This is being backed out due to a regression (#9507) and additional
review feedback being provided.
2021-03-02 09:43:34 -05:00
Jonathan de Jong f5c93fc993
Fix #8518 (sync requests being cached wrongly on timeout) (#9358)
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false".

This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-02-24 13:57:00 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke 637282bb50
Add additional type hints to the storage module. (#8980) 2020-12-30 08:09:53 -05:00
Erik Johnston f737368a26
Add admin API for logging in as a user (#8617) 2020-11-17 10:51:25 +00:00
Patrick Cloke 34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Will Hunt c276bd9969
Send some ephemeral events to appservices (#8437)
Optionally sends typing, presence, and read receipt information to appservices.
2020-10-15 12:33:28 -04:00
Patrick Cloke 1781bbe319
Add type hints to response cache. (#8507) 2020-10-09 11:35:11 -04:00
Erik Johnston 9ca6341969
Fix returning incorrect prev_batch token in incremental sync (#8486) 2020-10-07 13:49:40 +01:00
Hubert Chathi 3cd78bbe9e
Add support for MSC2732: olm fallback keys (#8312) 2020-10-06 13:26:29 -04:00
Patrick Cloke c5251c6fbd
Do not assume that account data is of the correct form. (#8454)
This fixes a bug where `m.ignored_user_list` was assumed to be a dict,
leading to odd behavior for users who set it to something else.
2020-10-05 09:28:05 -04:00
Patrick Cloke 4ff0201e62
Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
Erik Johnston ea70f1c362
Various clean ups to room stream tokens. (#8423) 2020-09-29 21:48:33 +01:00
Erik Johnston ac11fcbbb8
Add EventStreamPosition type (#8388)
The idea is to remove some of the places we pass around `int`, where it can represent one of two things:

1. the position of an event in the stream; or
2. a token that partitions the stream, used as part of the stream tokens.

The valid operations are then:

1. did a position happen before or after a token;
2. get all events that happened before or after a token; and
3. get all events between two tokens.

(Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
2020-09-24 13:24:17 +01:00
Patrick Cloke aec294ee0d
Use slots in attrs classes where possible (#8296)
slots use less memory (and attribute access is faster) while slightly
limiting the flexibility of the class attributes. This focuses on objects
which are instantiated "often" and for short periods of time.
2020-09-14 12:50:06 -04:00
Erik Johnston fe8ed1b46f
Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281) 2020-09-11 12:22:55 +01:00
Erik Johnston 63c0e9e195
Add types to StreamToken and RoomStreamToken (#8279)
The intention here is to change `StreamToken.room_key` to be a `RoomStreamToken` in a future PR, but that is a big enough change without this refactoring too.
2020-09-08 16:48:15 +01:00
Patrick Cloke c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Erik Johnston 5bfc79486d
Fix typing for SyncHandler (#8237) 2020-09-03 12:54:10 +01:00
Brendan Abolivier 5a1dd297c3
Re-implement unread counts (again) (#8059) 2020-09-02 17:19:37 +01:00
Patrick Cloke d5e73cb6aa
Define StateMap as immutable and add a MutableStateMap type. (#8183) 2020-08-28 07:28:53 -04:00
Brendan Abolivier 2ffd6783c7
Revert #7736 (#8039) 2020-08-06 17:15:35 +01:00
Patrick Cloke e19de43eb5
Convert streams to async. (#8014) 2020-08-04 07:21:47 -04:00
Brendan Abolivier 8dff4a1242
Re-implement unread counts (#7736) 2020-07-29 18:26:55 +01:00
Patrick Cloke e739b20588
Fix up types and comments that refer to Deferreds. (#7945) 2020-07-24 10:53:25 -04:00
Richard van der Hoff b74919c72e
Add debugging to sync response generation (#7929) 2020-07-22 13:43:10 +01:00
Brendan Abolivier 74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Brendan Abolivier 6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Patrick Cloke bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Brendan Abolivier 9dbd006607
Appease mypy 2020-06-10 20:44:24 +01:00
Brendan Abolivier 243f0ba6ce
Lint 2020-06-10 20:35:35 +01:00
Brendan Abolivier df3323a7cf
Use temporary prefixes as per the MSC 2020-06-10 20:32:01 +01:00
Brendan Abolivier ef345c5a7b
Add a new unread_counter to sync responses 2020-06-10 16:21:16 +01:00
Patrick Cloke f1e61ef85c Typo fixes. 2020-06-05 08:43:21 -04:00
Erik Johnston f9073893af Speed up fetching device lists changes in sync.
Currently we copy `users_who_share_room` needlessly about three times,
which is expensive when the set is large (which it can easily be).
2020-05-05 17:40:29 +01:00
Matthew Hodgson 118b58f0c9 typos 2020-04-11 20:55:18 +01:00
Andrew Morgan 104844c1e1 Add explanatory comment 2020-03-30 14:00:11 +01:00
David Baker 9fc588e6dc Just add own user ID to the list we track device changes for 2020-03-30 10:11:26 +01:00
David Baker 09cc058a4c Always send the user updates to their own device list
This will allow clients to notify users about new devices even if
the user isn't in any rooms (yet).
2020-03-27 12:26:47 +00:00
Richard van der Hoff 39230d2171
Clean up some LoggingContext stuff (#7120)
* Pull Sentinel out of LoggingContext

... and drop a few unnecessary references to it

* Factor out LoggingContext.current_context

move `current_context` and `set_context` out to top-level functions.

Mostly this means that I can more easily trace what's actually referring to
LoggingContext, but I think it's generally neater.

* move copy-to-parent into `stop`

this really just makes `start` and `stop` more symetric. It also means that it
behaves correctly if you manually `set_log_context` rather than using the
context manager.

* Replace `LoggingContext.alive` with `finished`

Turn `alive` into `finished` and make it a bit better defined.
2020-03-24 14:45:33 +00:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Richard van der Hoff 605cd089f7 Merge branch 'release-v1.10.0' into develop 2020-02-11 10:43:47 +00:00
Matthew Hodgson 01209382fb
filter out m.room.aliases from /sync state blocks (#6884)
We forgot to filter out aliases from /sync state blocks as well as the timeline.
2020-02-10 18:07:35 +00:00
Erik Johnston ed630ea17c
Reduce amount of logging at INFO level. (#6862)
A lot of the things we log at INFO are now a bit superfluous, so lets
make them DEBUG logs to reduce the amount we log by default.

Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2020-02-06 13:31:05 +00:00
Erik Johnston b9391c9575
Add typing to SyncHandler (#6821)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-02-03 18:05:44 +00:00
Erik Johnston 28c98e51ff
Add `local_current_membership` table (#6655)
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
2020-01-15 14:59:33 +00:00
Erik Johnston b3a4e35ca8 Fixup functions to consistently return deferreds 2019-12-06 10:40:05 +00:00
Erik Johnston 8437e2383e Port SyncHandler to async/await 2019-12-05 17:58:25 +00:00
Amber Brown 020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston 69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Hubert Chathi faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Andrew Morgan 2a44782666
Remove double return statements (#5962)
Remove all the "double return" statements which were a result of us removing all the instances of

```
defer.returnValue(...)
return
```

statements when we switched to python3 fully.
2019-09-03 11:42:45 +01:00
Andrew Morgan 4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Hubert Chathi e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
Erik Johnston 748aa38378 Remove logging for #5407 and update comments 2019-08-15 12:02:18 +01:00
Erik Johnston 6881f21f3e Handle TimelineBatch being limited and empty.
This hopefully addresses #5407 by gracefully handling an empty but
limited TimelineBatch. We also add some logging to figure out how this
is happening.
2019-08-06 12:59:00 +01:00
Hubert Chathi 336c546d6a Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-08-01 16:31:40 -04:00
Hubert Chathi c659b9f94f allow uploading keys for cross-signing 2019-07-25 11:08:24 -04:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00