Commit Graph

15254 Commits (157c571f3e9d3d09cd763405b6a9eb967f2807e7)

Author SHA1 Message Date
reivilibre e9b1ff9f31
Prevent clients from reporting nonexistent events. (#13779) 2023-02-14 15:50:59 +00:00
Sean Quah 463c19ac36
Faster joins: Omit device list updates from partial state rooms in /sync (#15069)
...when lazy loading of members is not enabled. It's weird to notify
a client that another user's device list has changed when the client
doesn't think that they share a room.

Note that when a room is un-partial stated, device list updates are
emitted for every member in that room over /sync.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-14 12:32:19 +00:00
Erik Johnston cb262713b7
Fix clashing DB txn name (#15070)
* Fix clashing DB txn name

* Newsfile
2023-02-14 11:20:25 +00:00
Erik Johnston f09db5c991
Skip calculating unread push actions in `/sync` when `enable_push` is false. (#14980) 2023-02-14 11:10:29 +00:00
Harishankar Kumar db2b105d69
Change collection[str] to StrCollection in event_auth code (#14929)
Signed-off-by: Harishankar Kumar <hari01584@gmail.com>
2023-02-14 09:37:08 +00:00
reivilibre 3d7aead5d6
Tweak comment on `_is_local_room_accessible` as part of room visibility in `/hierarchy` to clarify the condition for a room being visible. (#14834) 2023-02-13 16:30:58 +00:00
Andrew Morgan bdccfd2477
Refactor arguments of `try_unbind_threepid(_with_id_server)` from dict to separate args (#15053) 2023-02-13 12:12:48 +00:00
David Robertson c10e131250
Apply logging from hotfixes branch to develop (#15054)
* Apply logging from hotfixes branch to develop

Part of #4826.

Originally added in #11882.

* Changelog
2023-02-13 11:49:20 +00:00
Mathieu Velten 6cddf24e36
Faster joins: don't stall when a user joins during a fast join (#14606)
Fixes #12801.
Complement tests are at
https://github.com/matrix-org/complement/pull/567.

Avoid blocking on full state when handling a subsequent join into a
partial state room.

Also always perform a remote join into partial state rooms, since we do
not know whether the joining user has been banned and want to avoid
leaking history to banned users.

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:31:05 +00:00
Sean Quah d0c713cc85
Return read-only collections from `@cached` methods (#13755)
It's important that collections returned from `@cached` methods are not
modified, otherwise future retrievals from the cache will return the
modified collection.

This applies to the return values from `@cached` methods and the values
inside the dictionaries returned by `@cachedList` methods. It's not
necessary for the dictionaries returned by `@cachedList` methods
themselves to be read-only.

Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:29:00 +00:00
Patrick Cloke 14be78d492
Support for MSC3758: exact_event_match push condition (#14964)
This specifies to search for an exact value match, instead of
string globbing. It only works across non-compound JSON values
(null, boolean, integer, and strings).
2023-02-10 12:37:07 -05:00
Patrick Cloke cf5233b783
Avoid fetching unused account data in sync. (#14973)
The per-room account data is no longer unconditionally
fetched, even if all rooms will be filtered out.

Global account data will not be fetched if it will all be
filtered out.
2023-02-10 14:22:16 +00:00
David Robertson d793fcd241
Merge branch 'release-v1.77' into develop 2023-02-10 13:43:18 +00:00
Sean Quah b95407908d
Avoid mutating cached values in `_generate_sync_entry_for_account_data` (#15047) 2023-02-10 08:11:20 -05:00
Patrick Cloke a481fb9f98
Refactor get_user_devices_from_cache to avoid mutating cached values. (#15040)
The previous version of the code could mutate a cached value,
but only if the input requested all devices of a user *and* a specific
device.

To avoid this nonsensical situation we no longer fetch a specific
device ID if all of a user's devices are returned.
2023-02-10 08:09:47 -05:00
Erik Johnston fd296b7343
Fix exception on start up about device lists (#15041)
Fixes #15010.
2023-02-10 09:52:35 +00:00
David Robertson a5a799722d
Tag federation request spans with the worker name (#15042)
* Systematically include worker name as process info

* Changelog

* don't bother with inner setdefault
2023-02-09 22:33:39 +00:00
Shay 03bccd542b
Add a class UnpersistedEventContext to allow for the batching up of storing state groups (#14675)
* add class UnpersistedEventContext

* modify create new client event to create unpersistedeventcontexts

* persist event contexts after creation

* fix tests to persist unpersisted event contexts

* cleanup

* misc lints + cleanup

* changelog + fix comments

* lints

* fix batch insertion?

* reduce redundant calculation

* add unpersisted event classes

* rework compute_event_context, split into function that returns unpersisted event context and then persists it

* use calculate_context_info to create unpersisted event contexts

* update typing

* $%#^&*

* black

* fix comments and consolidate classes, use attr.s for class

* requested changes

* lint

* requested changes

* requested changes

* refactor to be stupidly explicit

* clearer renaming and flow

* make partial state non-optional

* update docstrings

---------

Co-authored-by: Erik Johnston <erik@matrix.org>
2023-02-09 13:05:02 -08:00
Andrew Morgan c1d2ce2901
Do not always start a db txn on Postgres (#14840) 2023-02-09 19:57:01 +00:00
Patrick Cloke d22c1c862c
Respond correctly to unknown methods on known endpoints (#14605)
Respond with a 405 error if a request is received on a known endpoint,
but to an unknown method, per MSC3743.
2023-02-09 13:04:24 -05:00
Patrick Cloke 8a6e043488
Avoid mutating cached room aliases. (#15038)
This might cause incorrect data in other callers which
are not expecting the canonical alias to be added into
the response.
2023-02-09 15:56:02 +00:00
David Robertson cd2484dc2e
Bump schema version (#15036)
* Bump schema version

This should have been included in
f10caa73ee (and #14979).

* Changelog
2023-02-09 15:28:26 +00:00
Patrick Cloke 733531ee3e
Add final type hint to synapse.server. (#15035) 2023-02-09 09:49:04 -05:00
Shay 55e4d27b36
Limit concurrent event creation for a room to avoid state resolution when sending bursts of events to a local room (#14977) 2023-02-08 11:25:11 -08:00
Patrick Cloke c951fbedcb
MSC3873: Escape keys when flattening dicts. (#15004)
This disambiguates keys which attempt to match fields
with a dot in them (e.g. m.relates_to).

Disabled by default behind an experimental configuration flag.
2023-02-08 13:09:41 -05:00
Erik Johnston c78c67c5a9
Fix bug in replication where response is cached (#15024) 2023-02-08 16:41:55 +00:00
David Robertson dccae64083
Merge branch 'release-v1.77' into develop 2023-02-08 12:45:46 +00:00
David Robertson f10caa73ee
Disambiguate `get_ex_outlier_stream_rows` query
A backwards-compatible piece of #14979 that's safe to land now.
2023-02-07 15:33:33 +00:00
David Robertson 9cd7610f86
Revert "Add `event_stream_ordering` column to membership state tables (#14979)"
This reverts commit 5fdc12f482.
2023-02-07 15:26:55 +00:00
David Robertson 2dff93099b
Typecheck tests.rest.media.v1.test_media_storage (#15008)
* Fix MediaStorage type hint

* Typecheck tests.rest.media.v1.test_media_storage

* Changelog

* Remove assert and make the comment succinct

* Fix syntax for olddeps
2023-02-07 15:24:44 +00:00
Patrick Cloke 5b55c32d61
Add tests for using _flatten_dict with an event. (#15002) 2023-02-07 06:56:09 -05:00
David Robertson d0fed7a37b
Properly typecheck types.http (#14988)
* Tweak http types in Synapse

AFACIS these are correct, and they make mypy happier on tests.http.

* Type hints for test_proxyagent

* type hints for test_srv_resolver

* test_matrix_federation_agent

* tests.http.server._base

* tests.http.__init__

* tests.http.test_additional_resource

* tests.http.test_client

* tests.http.test_endpoint

* tests.http.test_matrixfederationclient

* tests.http.test_servlet

* tests.http.test_simple_client

* tests.http.test_site

* One fixup in tests.server

* Untyped defs

* Changelog

* Fixup syntax for Python 3.7

* Fix olddeps syntax

* Use a twisted IPv4 addr for dummy_address

* Fix typo, thanks Sean

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Remove redundant `Optional`

---------

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-02-07 00:20:04 +00:00
Nick Mills-Barrett 5fdc12f482
Add `event_stream_ordering` column to membership state tables (#14979)
This adds an `event_stream_ordering` column to `current_state_events`,
`local_current_membership` and `room_memberships`. Each of these tables
is regularly joined with the `events` table to get the stream ordering
and denormalising this into each table will yield significant query
performance improvements once used. Includes a background job to
populate these values from the `events` table.

Same idea as https://github.com/matrix-org/synapse/pull/13703.

Signed off by Nick @ Beeper (@fizzadar).
2023-02-07 00:10:54 +00:00
David Robertson e8269ed391
Type hints for tests.appservice (#14990)
* Accept a Sequence of events in synapse.appservice

This avoids some casts/ignores in the tests I'm about to fixup. It seems
that `List[Mock]` is not a subtype of `List[EventBase]`, but
`Sequence[Mock]` is a subtype of `Sequence[EventBase]`. So presumably
`Mock` is considered a subtype of anything, much like `Any`.

* make tests.appservice.test_scheduler pass mypy

* Extra hints in tests.appservice.test_scheduler

* Extra hints in tests.appservice.test_api

* Extra hints in tests.appservice.test_appservice

* Disallow untyped defs

* Changelog
2023-02-06 12:49:06 +00:00
David Robertson b3bf58a8a5
Only notify the target of a membership event (#14971)
* Only notify the target of a membership event

Naughty, but should be a big speedup in large rooms
2023-02-06 11:29:51 +00:00
David Robertson 6e6edea6c1
Properly typecheck tests.api (#14983) 2023-02-03 20:03:23 +00:00
Patrick Cloke b2d97bac09
Implement MSC3958: suppress notifications from edits (#14960)
Co-authored-by: Brad Murray <brad@beeper.com>
Co-authored-by: Nick Barrett <nick@beeper.com>

Copy the suppress_edits push rule from Beeper to implement MSC3958.

9415a1284b/rust/src/push/base_rules.rs (L98-L114)
2023-02-03 14:31:14 -05:00
Patrick Cloke f0cae26d58
Add a docstring & tests for _flatten_dict. (#14981) 2023-02-03 16:48:13 +00:00
Patrick Cloke 52700a0bcf
Support the backwards compatibility features in MSC3952. (#14958)
If the feature is enabled and the event has a `m.mentions` property,
skip processing of the legacy mentions rules.
2023-02-03 16:28:20 +00:00
Sean Quah 0a686d1d13
Faster joins: Refactor handling of servers in room (#14954)
Ensure that the list of servers in a partial state room always contains
the server we joined off.

Also refactor `get_partial_state_servers_at_join` to return `None` when
the given room is no longer partial stated, to explicitly indicate when
the room has partial state. Otherwise it's not clear whether an empty
list means that the room has full state, or the room is partial stated,
but the server we joined off told us that there are no servers in the
room.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-03 15:39:59 +00:00
Patrick Cloke 8e9fc28c6a
Reload the pyo3-log config when the Python logging config changes. (#14976)
Since pyo3-log is initialized very early in the Python start-up
it caches the state of the loggers before they're fully initialized
(and thus are essentially disabled). Whenever we reload the
logging configuration we now also tell pyo3-log to discard
any cached logging configuration it has; it will refetch the
current logging configuration from Python at the next point
it logs.

This fixes Rust log lines not appearing in the homeserver logs.
2023-02-03 08:27:31 -05:00
Patrick Cloke da05b70af5
Skip unused calculations in sync handler. (#14908)
If a sync request does not need to calculate per-room entries &
is not generating presence & is not generating device list data
(e.g. during initial sync) avoid the expensive calculation of room
specific data.

This is a micro-optimisation for clients syncing simply to receive
to-device information.
2023-02-02 13:45:12 -05:00
Patrick Cloke f36da501be
Do not calculate presence or ephemeral events when they are filtered out (#14970)
This expands the previous optimisation from being only for initial
sync to being for all sync requests.

It also inverts some of the logic to be inclusive instead of exclusive.
2023-02-02 11:58:20 -05:00
David Robertson 2186ebed6c
Fetch fewer events when getting hosts in room (#14962) 2023-02-02 16:49:14 +00:00
realtyem 58214dbb9b
Allow enabling the asyncio reactor in complement (#14858)
Signed-off-by: Jason Little realtyem@gmail.com
2023-02-01 23:42:45 +00:00
Patrick Cloke 1182ae5063
Add helper to parse an enum from query args & use it. (#14956)
The `parse_enum` helper pulls an enum value from the query string
(by delegating down to the parse_string helper with values generated
from the enum).

This is used to pull out "f" and "b" in most places and then we thread
the resulting Direction enum throughout more code.
2023-02-01 21:35:24 +00:00
Patrick Cloke 230a831c73
Attempt to delete more duplicate rows in receipts_linearized table. (#14915)
The previous assumption was that the stream_id column was unique
(for a room ID, receipt type, user ID tuple), but this turned out to be
incorrect.

Now find the max stream ID, then map this back to a database-specific
row identifier and delete other rows which match the (room ID, receipt type,
user ID) tuple, but *not* the row ID.
2023-02-01 15:45:10 -05:00
Dirk Klimpel bf82b56bab
Add more user information to export-data command. (#14894)
* The user's profile information.
* The user's devices.
* The user's connections / IP address information.
2023-02-01 15:45:19 +00:00
David Robertson 3b8574b4f2
Tag /send_join responses to detect faster joins (#14950)
* Tag /send_join responses to detect faster joins

* Changelog

* Define a proper SynapseTag

* isort
2023-01-31 12:43:20 +00:00
Sean Quah 805b641fb6
Fix "Re-starting finished log context" spam when creating events (#14947)
`run_in_background` calls re-use the current logging context. When they
are not awaited, they can complete after the current logging context has
been marked as finished, which leads to log spam. Use
`run_as_background_process` instead.

Fixes one of the instances of #13090.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-01-31 11:31:52 +00:00