Commit Graph

3289 Commits (63d87c08c8e9acedb64ef6ee135acda34e4e370d)

Author SHA1 Message Date
David Robertson a1c9869394
Merge branch 'release-v1.79' into develop 2023-03-13 18:35:21 +00:00
David Robertson c071cd5a0e
Ensure fed-sender catchup does not block for full state (#15248)
* Reproduce bad scenario in test
* Avoid catchup optimisation for partial state rooms
2023-03-13 12:31:19 +00:00
David Robertson 4bb26c95a9
Refactor `filter_events_for_server` (#15240)
* Tweak docstring and type hint

* Flip logic and provide better name

* Separate decision from action

* Track a set of strings, not EventBases

* Require explicit boolean options from callers

* Add explicit option for partial state rooms

* Changelog

* Rename param
2023-03-10 15:31:25 +00:00
Andrew Morgan e157c63f68
Fix missing conditional for registering `on_remove_user_third_party_identifier` module api callbacks (#15227 2023-03-10 10:35:18 +00:00
Patrick Cloke e7c3832ba6
Pull in netaddr type hints. (#15231)
And fix any issues from having those type hints.
2023-03-09 07:09:49 -05:00
Patrick Cloke 20ed8c926b
Stabilize support for MSC3873: disambuguated event push keys. (#15190)
This removes the experimental configuration option and
always escapes the push rule condition keys.

Also escapes any (experimental) push rule condition keys
in the base rules which contain dot in a field name.
2023-03-07 11:27:57 -05:00
Quentin Gliech 47bc84dd53
Pass the Requester down to the HttpTransactionCache. (#15200) 2023-03-07 16:05:22 +00:00
Patrick Cloke 820f02b70b
Stabilize support for MSC3966: event_property_contains push condition. (#15187)
This removes the configuration flag & updates the identifiers to
use the stable version.
2023-03-07 10:06:02 -05:00
Patrick Cloke 05e0a4089a
Stop applying edits to event contents (MSC3925). (#15193)
Enables MSC3925 support by default, which:

* Includes the full edit event in the bundled aggregations of an
  edited event.
* Stops modifying the original event's content to return the new
  content from the edit event.

This is a backwards-incompatible change that is considered to be
"correct" by the spec.
2023-03-06 09:43:01 -05:00
Patrick Cloke fd9cadcf53
Stabilize support for MSC3758: event_property_is push condition (#15185)
This removes the configuration flag & updates the identifiers to
use the stable version.
2023-03-06 08:38:01 -05:00
6543 6b6e91e610
Fix ICU tests on alpine / macOS. (#15177)
The word boundary behaviour is slightly different, consider it
acceptable for the tests.
2023-03-03 14:22:06 +00:00
Patrick Cloke 8ef324ea6f
Update intentional mentions (MSC3952) to depend on `exact_event_property_contains` (MSC3966). (#15051)
This replaces the specific `is_user_mention` push rule condition
used in MSC3952 with the generic `exact_event_property_contains`
push rule condition from MSC3966.
2023-03-02 08:30:51 -05:00
Hugh Nimmo-Smith 916b8061d2
Implementation of MSC3967: Don't require UIA for initial upload of cross signing keys (#15077) 2023-03-02 10:34:59 +00:00
Richard van der Hoff 2b78981736
Remove support for aggregating reactions (#15172)
It turns out that no clients rely on server-side aggregation of `m.annotation`
relationships: it's just not very useful as currently implemented.

It's also non-trivial to calculate.

I want to remove it from MSC2677, so to keep the implementation in line, let's
remove it here.
2023-02-28 18:49:28 +00:00
H. Shay b2fd03d075 Merge branch 'master' into develop 2023-02-28 10:14:20 -08:00
reivilibre d62cd940cb
Fix a long-standing bug where an initial sync would not respond to changes to the list of ignored users if there was an initial sync cached. (#15163) 2023-02-28 17:11:26 +00:00
Patrick Cloke e746f80b4f
Do not accept pattern_type from user input in push rules. (#15088)
Internally the push rules module uses a `pattern_type` property for `event_match`
conditions (and `related_event_match`) to mark the condition as matching the
current user's Matrix ID or localpart.

This is leaky to the Client-Server API where a user can successfully set a condition
which provides `pattern_type` instead of `pattern` (note that there's no benefit to
doing this -- the user can just use their own Matrix ID or localpart instead). When
serializing back to the client the `pattern_type` property is converted into a proper
`pattern`.

The following changes are made to avoid this:

* Separate the `KnownCondition::EventMatch` enum value into `EventMatch`
  and `EventMatchType`, each with their own expected properties. (Note that a
  similar change is made for `RelatedEventMatch`.)
* Make it such that the `pattern_type` variants serialize to the same condition kind,
  but cannot be deserialized (since they're only provided by base rules).
* As a final tweak, convert `user_id` vs. `user_localpart` values into an enum.
2023-02-28 10:11:20 -05:00
Dirk Klimpel 93f7955eba
Admin API endpoint to delete a reported event (#15116)
* Admin api to delete event report

* lint +  tests

* newsfile

* Apply suggestions from code review

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* revert changes - move to WorkerStore

* update unit test

* Note that timestamp is in millseconds

---------

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2023-02-28 12:09:10 +00:00
Andrew Morgan b40657314e
Add module API callbacks for adding and deleting local 3PID associations (#15044 2023-02-27 14:19:19 +00:00
Patrick Cloke 4fc8875876
Refactor media modules. (#15146)
* Removes the `v1` directory from `test.rest.media.v1`.
* Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`.
* Flatten the `v1` directory from `synapse.rest.media`,  but leave compatiblity
  with 3rd party media repositories and spam checkers.
2023-02-27 08:26:05 -05:00
Shay 1c95ddd09b
Batch up storing state groups when creating new room (#14918) 2023-02-24 13:15:29 -08:00
Erik Johnston b2357a898c
Fix bug where 5s delays would occasionally happen. (#15150)
This only affects deployments using workers.
2023-02-24 14:39:50 +00:00
Sean Quah 335f52d595
Improve handling of non-ASCII characters in user directory search (#15143)
* Fix a long-standing bug where non-ASCII characters in search terms,
  including accented letters, would not match characters in a different
  case.
* Fix a long-standing bug where search terms using combining accents
  would not match display names using precomposed accents and vice
  versa.

To fully take effect, the user directory must be rebuilt after this
change.

Fixes #14630.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-24 13:39:45 +00:00
Patrick Cloke 682151a464
Do not fail completely if oEmbed autodiscovery fails. (#15092)
Previously if an autodiscovered oEmbed request failed (e.g. the
oEmbed endpoint is down or does not exist) then the entire URL
preview would fail. Instead we now return everything we can, even
if this additional request fails.
2023-02-23 16:08:53 -05:00
Patrick Cloke ec79870f14
Fix a typo in MSC3873 config option. (#15138)
Previously the experimental configuration option referred to the wrong
MSC number.
2023-02-23 16:06:42 -05:00
Dirk Klimpel a068ad7dd4
Add information on uploaded media to user export command. (#15107) 2023-02-23 13:14:17 -05:00
dependabot[bot] 9bb2eac719
Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
David Robertson 647ff3ef65
Remove unused `room_alias` field from `/createRoom` response (#15093)
* Change `create_room` return type

* Don't return room alias from /createRoom

* Update other callsites

* Fix up mypy complaints

It looks like new_room_user_id is None iff new_room_id is None. It's a
shame we haven't expressed this in a way that mypy can understand.

* Changelog
2023-02-22 11:07:28 +00:00
reivilibre 1cbc3f197c
Fix a bug introduced in Synapse v1.74.0 where searching with colons when using ICU for search term tokenisation would fail with an error. (#15079)
Co-authored-by: David Robertson <davidr@element.io>
2023-02-20 12:00:18 +00:00
Patrick Cloke c9b9143655
Fix-up type hints in tests/server.py. (#15084)
This file was being ignored by mypy, we remove that
and add the missing type hints & deal with any fallout.
2023-02-17 18:19:38 +00:00
Dirk Klimpel 61bfcd669a
Add account data to export command (#14969)
* Add account data to to export command

* newsfile

* remove not needed function

* update newsfile

* adopt #14973
2023-02-17 13:54:55 +00:00
David Robertson ffc2ee521d
Use mypy 1.0 (#15052)
* Update mypy and mypy-zope
* Remove unused ignores

These used to suppress

```
synapse/storage/engines/__init__.py:28: error: "__new__" must return a
class instance (got "NoReturn")  [misc]
```

and

```
synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons"  [attr-defined]
```

(note that we check `hasattr(e, "reasons")` above)

* Avoid empty body warnings, sometimes by marking methods as abstract

E.g.

```
tests/handlers/test_register.py:58: error: Missing return statement  [empty-body]
tests/handlers/test_register.py:108: error: Missing return statement  [empty-body]
```

* Suppress false positive about `JaegerConfig`

Complaint was

```
synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context  [truthy-function]
```

* Fix not calling `is_state()`

Oops!

```
tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context  [truthy-function]
```

* Suppress false positives from ParamSpecs

````
synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
````

* Drive-by improvement to `wrapping_logic` annotation

* Workaround false "unreachable" positives

See https://github.com/Shoobx/mypy-zope/issues/91

```
tests/http/test_proxyagent.py:626: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:762: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:826: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:838: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:845: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:60: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:93: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:127: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:152: error: Statement is unreachable  [unreachable]
```

* Changelog

* Tweak DBAPI2 Protocol to be accepted by mypy 1.0

Some extra context in:
- https://github.com/matrix-org/python-canonicaljson/pull/57
- https://github.com/python/mypy/issues/6002
- https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected

* Pull in updated canonicaljson lib

so the protocol check just works

* Improve comments in opentracing

I tried to workaround the ignores but found it too much trouble.

I think the corresponding issue is
https://github.com/python/mypy/issues/12909. The mypy repo has a PR
claiming to fix this (https://github.com/python/mypy/pull/14677) which
might mean this gets resolved soon?

* Better annotation for INTERACTIVE_AUTH_CHECKERS

* Drive-by AUTH_TYPE annotation, to remove an ignore
2023-02-16 16:09:11 +00:00
Patrick Cloke 979f237b28
Update intentional mentions (MSC3952) to depend on `exact_event_match` (MSC3758). (#15037)
This replaces the specific `is_room_mention` push rule condition
used in MSC3952 with the generic `exact_event_match` push rule
condition from MSC3758.

No functionality changes due to this.
2023-02-16 09:51:22 -05:00
Patrick Cloke 42aea0d8af
Add final type hint to tests.unittest. (#15072)
Adds a return type to HomeServerTestCase.make_homeserver and deal
with any variables which are no longer Any.
2023-02-14 14:03:35 -05:00
Patrick Cloke 119e0795a5
Implement MSC3966: Add a push rule condition to search for a value in an array. (#15045)
The `exact_event_property_contains` condition can be used to
search for a value inside of an array.
2023-02-14 14:02:19 -05:00
reivilibre e9b1ff9f31
Prevent clients from reporting nonexistent events. (#13779) 2023-02-14 15:50:59 +00:00
Mathieu Velten 6cddf24e36
Faster joins: don't stall when a user joins during a fast join (#14606)
Fixes #12801.
Complement tests are at
https://github.com/matrix-org/complement/pull/567.

Avoid blocking on full state when handling a subsequent join into a
partial state room.

Also always perform a remote join into partial state rooms, since we do
not know whether the joining user has been banned and want to avoid
leaking history to banned users.

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:31:05 +00:00
Sean Quah d0c713cc85
Return read-only collections from `@cached` methods (#13755)
It's important that collections returned from `@cached` methods are not
modified, otherwise future retrievals from the cache will return the
modified collection.

This applies to the return values from `@cached` methods and the values
inside the dictionaries returned by `@cachedList` methods. It's not
necessary for the dictionaries returned by `@cachedList` methods
themselves to be read-only.

Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:29:00 +00:00
Patrick Cloke 14be78d492
Support for MSC3758: exact_event_match push condition (#14964)
This specifies to search for an exact value match, instead of
string globbing. It only works across non-compound JSON values
(null, boolean, integer, and strings).
2023-02-10 12:37:07 -05:00
Shay 03bccd542b
Add a class UnpersistedEventContext to allow for the batching up of storing state groups (#14675)
* add class UnpersistedEventContext

* modify create new client event to create unpersistedeventcontexts

* persist event contexts after creation

* fix tests to persist unpersisted event contexts

* cleanup

* misc lints + cleanup

* changelog + fix comments

* lints

* fix batch insertion?

* reduce redundant calculation

* add unpersisted event classes

* rework compute_event_context, split into function that returns unpersisted event context and then persists it

* use calculate_context_info to create unpersisted event contexts

* update typing

* $%#^&*

* black

* fix comments and consolidate classes, use attr.s for class

* requested changes

* lint

* requested changes

* requested changes

* refactor to be stupidly explicit

* clearer renaming and flow

* make partial state non-optional

* update docstrings

---------

Co-authored-by: Erik Johnston <erik@matrix.org>
2023-02-09 13:05:02 -08:00
Patrick Cloke d22c1c862c
Respond correctly to unknown methods on known endpoints (#14605)
Respond with a 405 error if a request is received on a known endpoint,
but to an unknown method, per MSC3743.
2023-02-09 13:04:24 -05:00
David Robertson 7081bb56e2
Proper types for `tests.module_api` (#15031)
* -> None for test methods

* A first batch of type fixes

* Introduce common parent test case

* Fixup that big test method

* tests.module_api passes mypy

* Changelog
2023-02-09 00:23:35 +00:00
Patrick Cloke 30509a1010
Add more missing type hints to tests. (#15028) 2023-02-08 16:29:49 -05:00
Patrick Cloke 4eed7b2ede
Add missing type hints to tests. (#15027) 2023-02-08 19:52:37 +00:00
Patrick Cloke c951fbedcb
MSC3873: Escape keys when flattening dicts. (#15004)
This disambiguates keys which attempt to match fields
with a dot in them (e.g. m.relates_to).

Disabled by default behind an experimental configuration flag.
2023-02-08 13:09:41 -05:00
David Robertson 2dff93099b
Typecheck tests.rest.media.v1.test_media_storage (#15008)
* Fix MediaStorage type hint

* Typecheck tests.rest.media.v1.test_media_storage

* Changelog

* Remove assert and make the comment succinct

* Fix syntax for olddeps
2023-02-07 15:24:44 +00:00
David Robertson 4dd2b6165c
Proper types for tests.test_terms_auth (#15007)
* Proper types for tests.test_terms_auth

* Changelog
2023-02-07 12:03:39 +00:00
Patrick Cloke 5b55c32d61
Add tests for using _flatten_dict with an event. (#15002) 2023-02-07 06:56:09 -05:00
David Robertson d0fed7a37b
Properly typecheck types.http (#14988)
* Tweak http types in Synapse

AFACIS these are correct, and they make mypy happier on tests.http.

* Type hints for test_proxyagent

* type hints for test_srv_resolver

* test_matrix_federation_agent

* tests.http.server._base

* tests.http.__init__

* tests.http.test_additional_resource

* tests.http.test_client

* tests.http.test_endpoint

* tests.http.test_matrixfederationclient

* tests.http.test_servlet

* tests.http.test_simple_client

* tests.http.test_site

* One fixup in tests.server

* Untyped defs

* Changelog

* Fixup syntax for Python 3.7

* Fix olddeps syntax

* Use a twisted IPv4 addr for dummy_address

* Fix typo, thanks Sean

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Remove redundant `Optional`

---------

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-02-07 00:20:04 +00:00
Patrick Cloke d0fa217cd9
Add missing types to test_state. (#14985) 2023-02-06 16:11:09 +00:00