Commit Graph

181 Commits (release-v1.69)

Author SHA1 Message Date
Patrick Cloke efd108b45d
Accept & store thread IDs for receipts (implement MSC3771). (#13782)
Updates the `/receipts` endpoint and receipt EDU handler to parse a
`thread_id` from the body and insert it in the database.
2022-09-23 14:33:28 +00:00
Brendan Abolivier 8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) (#13799)
Partial implementation of MSC3881
2022-09-21 14:39:01 +00:00
Mathieu Velten 6bd8763804
Add cache invalidation across workers to module API (#13667)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-21 15:32:01 +02:00
Patrick Cloke 666ae87729
Update event push action and receipt tables to support threads. (#13753)
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`,
and `event_push_summary` tables. This will notifications to be segmented by the thread
in a future pull request. The `thread_id` column stores the root event ID or the special
value `"main"`.

The `thread_id` column for `event_push_actions` and `event_push_summary` is
backfilled with `"main"` for all existing rows. New entries into `event_push_actions`
and `event_push_actions_staging` will get the proper thread ID.

`receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar,
except `NULL` is a special value meaning the receipt is "unthreaded".

See MSC3771 and MSC3773 for where this data will be useful.
2022-09-14 17:11:16 +00:00
reivilibre 7bc110a19e
Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods. (#13662) 2022-08-31 11:16:05 +00:00
Patrick Cloke f3fba4914d
Reduce the number of tests using TCP replication. (#13543)
Uses Redis replication in additional test cases (instead of
TCP replication). A small step towards dropping TCP replication.
2022-08-19 08:25:24 -04:00
Nick Mills-Barrett 86e366a46e
Remove old empty/redundant slaved stores. (#13349) 2022-07-21 17:56:45 +00:00
Erik Johnston 0d1d3e0708
Speed up `get_unread_event_push_actions_by_room` (#13005)
Fixes #11887 hopefully.

The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller.

This needs two major changes:
1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it
2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`.

In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
2022-06-15 15:17:14 +00:00
Sean Quah 3c1c40d843
Clean up the test code for client disconnections (#12929)
* Reword failure message about `await_result=False`
* Use `reactor.advance()` instead of `reactor.pump()`
* Raise `AssertionError`s ourselves
* Un-instance method `_test_disconnect`
* Replace `ThreadedMemoryReactorClock` with `MemoryReactorClock`
2022-06-07 18:17:32 +01:00
Erik Johnston 1e453053cb
Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
reivilibre 177b884ad7
Lay some foundation work to allow workers to only subscribe to some kinds of messages, reducing replication traffic. (#12672) 2022-05-19 16:29:08 +01:00
Sean Quah a5c26750b5
Fix room upgrades creating an empty room when auth fails (#12696)
Signed-off-by: Sean Quah <seanq@element.io>
2022-05-16 14:06:04 +01:00
Sean Quah a559c8b0d9
Respect the `@cancellable` flag for `ReplicationEndpoint`s (#12700)
While `ReplicationEndpoint`s register themselves via `JsonResource`,
they pass a method that calls the handler, instead of the handler itself,
to `register_paths`. As a result, `JsonResource` will not correctly pick
up the `@cancellable` flag and we have to apply it ourselves.

Signed-off-by: Sean Quah <seanq@element.io>
2022-05-11 12:25:39 +01:00
Patrick Cloke 7fbf42499d
Use `getClientAddress` instead of `getClientIP`. (#12599)
getClientIP was deprecated in Twisted 18.4.0, which also added
getClientAddress. The Synapse minimum version for Twisted is
currently 18.9.0, so all supported versions have the new API.
2022-05-04 14:11:21 -04:00
Šimon Brandner 116a4c8340
Implement changes to MSC2285 (hidden read receipts) (#12168)
* Changes hidden read receipts to be a separate receipt type
  (instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
2022-05-04 11:59:22 -04:00
Šimon Brandner 0d9eaa19fd Use constants for receipt types in tests. (#12582) 2022-04-28 13:34:33 -04:00
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Richard van der Hoff 33ebee47e4
Remove redundant `get_success` calls in test code (#12346)
There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
2022-04-01 16:10:31 +01:00
David Robertson a2b00a4486
Bump `black` and `click` versions (#12320) 2022-03-29 10:41:19 +00:00
Patrick Cloke 3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. (#12192)
Since the object it returns is a ReplicationCommandHandler.

This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
Patrick Cloke 02d708568b
Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Patrick Cloke d0e78af35e
Add missing type hints to synapse.replication. (#11938) 2022-02-08 11:03:08 -05:00
reivilibre 4c2096599c
Make the `get_global_account_data_by_type_for_user` cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Patrick Cloke cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
Patrick Cloke b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Sean Quah ffd858aa68
Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)
Also refactor the stream ID trackers/generators a bit and try to
document them better.
2021-11-26 18:41:31 +00:00
David Robertson 4c96ce396e
Misc typing fixes for `tests`, part 1 of N (#11323)
* Annotate HomeserverTestCase.servlets
* Correct annotation of federation_auth_origin
* Use AnyStr custom_headers instead of a Union

This allows (str, str) and (bytes, bytes).
This disallows (str, bytes) and (bytes, str)

* DomainSpecificString.SIGIL is a ClassVar
2021-11-12 15:50:54 +00:00
David Robertson e46ac85d67
type-hint `HomeserverTestcase.setup_test_homeserver` (#10961)
* type-hint `HomeserverTestcase.setup_test_homeserver`

For better IDE completion. A small drive-by.
2021-10-01 12:22:47 +01:00
David Robertson 29364145b2
Pass str to twisted's IReactorTCP (#10895)
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.

Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
2021-09-30 12:51:47 +01:00
Erik Johnston 50022cff96
Add reactor to `SynapseRequest` and fix up types. (#10868) 2021-09-24 11:01:25 +01:00
Patrick Cloke 47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Patrick Cloke bfb4b858a9
Create a constant for a small png image in tests. (#10834)
To avoid duplicating it between a few tests.
2021-09-16 12:01:14 -04:00
Richard van der Hoff 1800aabfc2
Split `FederationHandler` in half (#10692)
The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
2021-08-26 21:41:44 +01:00
Patrick Cloke 000aa89be6
Do not include rooms with an unknown room version in a sync response. (#10644)
A user will still see this room if it is in a local cache, but it will
not reappear if clearing the cache and reloading.
2021-08-19 11:12:55 -04:00
Patrick Cloke bec01c0758
Convert room member storage tuples to attrs. (#10629)
Instead of using namedtuples. This helps with asserting type hints
and code completion.
2021-08-18 09:22:07 -04:00
reivilibre 642a42edde
Flatten the synapse.rest.client package (#10600) 2021-08-17 11:57:58 +00:00
Jonathan de Jong 93729719b8
Use inline type hints in `tests/` (#10350)
This PR is tantamount to running:

    python3.8 -m com2ann -v 6 tests/

(com2ann requires python 3.8 to run)
2021-07-13 11:52:58 +01:00
Jonathan de Jong 89cfc3dd98
[pyupgrade] `tests/` (#10347) 2021-07-13 11:43:15 +01:00
Richard van der Hoff 6e8fb42be7
Improve validation for `send_{join,leave,knock}` (#10225)
The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
2021-06-24 15:30:49 +01:00
Eric Eastwood 96f6293de5
Add endpoints for backfilling history (MSC2716) (#9247)
Work on https://github.com/matrix-org/matrix-doc/pull/2716
2021-06-22 10:02:53 +01:00
Andrew Morgan 4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Richard van der Hoff 3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Richard van der Hoff 84936e2264
Kill off `_PushHTTPChannel`. (#9878)
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.

Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
2021-04-23 18:40:57 +01:00
Richard van der Hoff 59d24c5bef
pass a reactor into SynapseSite (#9874) 2021-04-23 17:06:47 +01:00
Jonathan de Jong 495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Erik Johnston 00a6db9676
Move some replication processing out of generic_worker (#9796)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Dan Callahan 1d5f0e3529
Bump black configuration to target py36 (#9781)
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
Patrick Cloke 0b3112123d
Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00