Commit Graph

174 Commits (7d99414edf2c5c7e602a88c72245add665e6afb4)

Author SHA1 Message Date
Erik Johnston 0d1d3e0708
Speed up `get_unread_event_push_actions_by_room` (#13005)
Fixes #11887 hopefully.

The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller.

This needs two major changes:
1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it
2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`.

In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
2022-06-15 15:17:14 +00:00
Sean Quah 3c1c40d843
Clean up the test code for client disconnections (#12929)
* Reword failure message about `await_result=False`
* Use `reactor.advance()` instead of `reactor.pump()`
* Raise `AssertionError`s ourselves
* Un-instance method `_test_disconnect`
* Replace `ThreadedMemoryReactorClock` with `MemoryReactorClock`
2022-06-07 18:17:32 +01:00
Erik Johnston 1e453053cb
Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
reivilibre 177b884ad7
Lay some foundation work to allow workers to only subscribe to some kinds of messages, reducing replication traffic. (#12672) 2022-05-19 16:29:08 +01:00
Sean Quah a5c26750b5
Fix room upgrades creating an empty room when auth fails (#12696)
Signed-off-by: Sean Quah <seanq@element.io>
2022-05-16 14:06:04 +01:00
Sean Quah a559c8b0d9
Respect the `@cancellable` flag for `ReplicationEndpoint`s (#12700)
While `ReplicationEndpoint`s register themselves via `JsonResource`,
they pass a method that calls the handler, instead of the handler itself,
to `register_paths`. As a result, `JsonResource` will not correctly pick
up the `@cancellable` flag and we have to apply it ourselves.

Signed-off-by: Sean Quah <seanq@element.io>
2022-05-11 12:25:39 +01:00
Patrick Cloke 7fbf42499d
Use `getClientAddress` instead of `getClientIP`. (#12599)
getClientIP was deprecated in Twisted 18.4.0, which also added
getClientAddress. The Synapse minimum version for Twisted is
currently 18.9.0, so all supported versions have the new API.
2022-05-04 14:11:21 -04:00
Šimon Brandner 116a4c8340
Implement changes to MSC2285 (hidden read receipts) (#12168)
* Changes hidden read receipts to be a separate receipt type
  (instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
2022-05-04 11:59:22 -04:00
Šimon Brandner 0d9eaa19fd Use constants for receipt types in tests. (#12582) 2022-04-28 13:34:33 -04:00
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Richard van der Hoff 33ebee47e4
Remove redundant `get_success` calls in test code (#12346)
There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
2022-04-01 16:10:31 +01:00
David Robertson a2b00a4486
Bump `black` and `click` versions (#12320) 2022-03-29 10:41:19 +00:00
Patrick Cloke 3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. (#12192)
Since the object it returns is a ReplicationCommandHandler.

This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
Patrick Cloke 02d708568b
Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Patrick Cloke d0e78af35e
Add missing type hints to synapse.replication. (#11938) 2022-02-08 11:03:08 -05:00
reivilibre 4c2096599c
Make the `get_global_account_data_by_type_for_user` cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Patrick Cloke cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
Patrick Cloke b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Sean Quah ffd858aa68
Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)
Also refactor the stream ID trackers/generators a bit and try to
document them better.
2021-11-26 18:41:31 +00:00
David Robertson 4c96ce396e
Misc typing fixes for `tests`, part 1 of N (#11323)
* Annotate HomeserverTestCase.servlets
* Correct annotation of federation_auth_origin
* Use AnyStr custom_headers instead of a Union

This allows (str, str) and (bytes, bytes).
This disallows (str, bytes) and (bytes, str)

* DomainSpecificString.SIGIL is a ClassVar
2021-11-12 15:50:54 +00:00
David Robertson e46ac85d67
type-hint `HomeserverTestcase.setup_test_homeserver` (#10961)
* type-hint `HomeserverTestcase.setup_test_homeserver`

For better IDE completion. A small drive-by.
2021-10-01 12:22:47 +01:00
David Robertson 29364145b2
Pass str to twisted's IReactorTCP (#10895)
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.

Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
2021-09-30 12:51:47 +01:00
Erik Johnston 50022cff96
Add reactor to `SynapseRequest` and fix up types. (#10868) 2021-09-24 11:01:25 +01:00
Patrick Cloke 47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Patrick Cloke bfb4b858a9
Create a constant for a small png image in tests. (#10834)
To avoid duplicating it between a few tests.
2021-09-16 12:01:14 -04:00
Richard van der Hoff 1800aabfc2
Split `FederationHandler` in half (#10692)
The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
2021-08-26 21:41:44 +01:00
Patrick Cloke 000aa89be6
Do not include rooms with an unknown room version in a sync response. (#10644)
A user will still see this room if it is in a local cache, but it will
not reappear if clearing the cache and reloading.
2021-08-19 11:12:55 -04:00
Patrick Cloke bec01c0758
Convert room member storage tuples to attrs. (#10629)
Instead of using namedtuples. This helps with asserting type hints
and code completion.
2021-08-18 09:22:07 -04:00
reivilibre 642a42edde
Flatten the synapse.rest.client package (#10600) 2021-08-17 11:57:58 +00:00
Jonathan de Jong 93729719b8
Use inline type hints in `tests/` (#10350)
This PR is tantamount to running:

    python3.8 -m com2ann -v 6 tests/

(com2ann requires python 3.8 to run)
2021-07-13 11:52:58 +01:00
Jonathan de Jong 89cfc3dd98
[pyupgrade] `tests/` (#10347) 2021-07-13 11:43:15 +01:00
Richard van der Hoff 6e8fb42be7
Improve validation for `send_{join,leave,knock}` (#10225)
The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
2021-06-24 15:30:49 +01:00
Eric Eastwood 96f6293de5
Add endpoints for backfilling history (MSC2716) (#9247)
Work on https://github.com/matrix-org/matrix-doc/pull/2716
2021-06-22 10:02:53 +01:00
Andrew Morgan 4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Richard van der Hoff 3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Richard van der Hoff 84936e2264
Kill off `_PushHTTPChannel`. (#9878)
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.

Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
2021-04-23 18:40:57 +01:00
Richard van der Hoff 59d24c5bef
pass a reactor into SynapseSite (#9874) 2021-04-23 17:06:47 +01:00
Jonathan de Jong 495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Erik Johnston 00a6db9676
Move some replication processing out of generic_worker (#9796)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Dan Callahan 1d5f0e3529
Bump black configuration to target py36 (#9781)
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
Patrick Cloke 0b3112123d
Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00
Jonathan de Jong 2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Andrew Morgan 5fdff97719
Fix CI by ignore type for None module import (#9709) 2021-03-29 14:42:38 +01:00
Erik Johnston b5efcb577e
Make it possible to use dmypy (#9692)
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Patrick Cloke d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. (#9608) 2021-03-15 11:14:39 -04:00
Patrick Cloke 55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Patrick Cloke 7fdc6cefb3
Fix additional type hints. (#9543)
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09 07:41:32 -05:00
Erik Johnston 2927921942
Clean up `ShardedWorkerHandlingConfig` (#9466)
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00