6a751ff5e0
Allow sending a membership event to unban a user ( #10807 )
...
* Allow membership event to unban user
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-09-21 10:23:34 +00:00
b3590614da
Require type hints in the handlers module. ( #10831 )
...
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.
This also removes the unused construct_auth_difference method
from the FederationHandler.
2021-09-20 08:56:23 -04:00
437961744c
Fix remove_stale_pushers job on SQLite. ( #10843 )
2021-09-20 10:26:13 +01:00
b4c1af8cea
Merge branch 'release-v1.43' into develop
2021-09-17 12:15:17 +01:00
daac1e645c
1.43.0rc2
2021-09-17 10:43:51 +01:00
9a6f4a684f
Cleanup opentracing logging for syncs ( #10828 )
...
We added a bunch of spans in #10704 , but this ended up adding a lot of
redundant spans for rooms where nothing changed, so instead we only
start the span if there might be something interesting going on.
2021-09-15 17:14:25 +01:00
3eba047d38
Add type hints to state database module. ( #10823 )
2021-09-15 09:54:13 -04:00
b93259082c
Add missing type hints to non-client REST servlets. ( #10817 )
...
Including admin, consent, key, synapse, and media. All REST servlets
(the synapse.rest module) now require typed method definitions.
2021-09-15 08:45:32 -04:00
8c7a531e27
Use direct references for some configuration variables (part 2) ( #10812 )
2021-09-15 08:34:52 -04:00
145c006ef7
Verify `?chunk_id` actually corresponds to an insertion event that exists (MSC2716) ( #10776 )
2021-09-15 09:34:30 +01:00
1c555527b3
Split out `/batch_send` meta events to their own fields (MSC2716) ( #10777 )
2021-09-15 09:30:58 +01:00
8eb7cb2e0d
Make StateFilter frozen so we can hash it ( #10816 )
...
Also enables Mypy for related tests.
2021-09-14 16:35:53 +01:00
14b8c0476f
Prevent logging context going missing on federation request timeout ( #10810 )
...
In `MatrixFederationHttpClient._send_request()`, we make a HTTP request
using an `Agent`, wrap that request in a timeout and await the resulting
`Deferred`. On its own, the `Agent` performing the HTTP request
correctly stashes and restores the logging context while waiting.
The addition of the timeout introduces a path where the logging context
is not restored when execution resumes.
To address this, we wrap the timeout `Deferred` in a
`make_deferred_yieldable()` to stash the logging context and restore it
on completion of the `await`. However this is not sufficient, since by
the time we construct the timeout `Deferred`, the `Agent` has already
stashed and cleared the logging context when using
`make_deferred_yieldable()` to produce its `Deferred` for the request.
Hence, we wrap the `Agent` request in a `run_in_background()` to "fork"
and preserve the logging context so that we can stash and restore it
when `await`ing the timeout `Deferred`.
This approach is similar to the one used with `defer.gatherResults`.
Note that the code is still not fully correct. When a timeout occurs,
the request remains running in the background (existing behavior which
is nothing to do with the new call to `run_in_background`) and may
re-start the logging context after it has finished.
2021-09-14 13:01:30 +01:00
51e1b96d04
Synapse 1.43.0rc1 (2021-09-14)
...
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566 ))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643 ))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712 ))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714 ), [\#10759](https://github.com/matrix-org/synapse/issues/10759 ))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375 ) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244 ) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772 ))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658 ))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730 ))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743 ))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757 ))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733 ))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10760](https://github.com/matrix-org/synapse/issues/10760 ))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774 ))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783 ))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556 ))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648 ))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735 ))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758 ))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html ). ([\#10795](https://github.com/matrix-org/synapse/issues/10795 ))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804 ))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693 ))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424 ](https://github.com/matrix-org/synapse/issues/9424 )). ([\#10704](https://github.com/matrix-org/synapse/issues/10704 ))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601 ))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697 ))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707 ), [\#10728](https://github.com/matrix-org/synapse/issues/10728 ), [\#10736](https://github.com/matrix-org/synapse/issues/10736 ))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727 ))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738 ))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744 ), [\#10745](https://github.com/matrix-org/synapse/issues/10745 ), [\#10746](https://github.com/matrix-org/synapse/issues/10746 ), [\#10771](https://github.com/matrix-org/synapse/issues/10771 ), [\#10773](https://github.com/matrix-org/synapse/issues/10773 ), [\#10781](https://github.com/matrix-org/synapse/issues/10781 ))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748 ))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750 ))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752 ))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753 ))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754 ), [\#10755](https://github.com/matrix-org/synapse/issues/10755 ), [\#10756](https://github.com/matrix-org/synapse/issues/10756 ), [\#10780](https://github.com/matrix-org/synapse/issues/10780 ), [\#10784](https://github.com/matrix-org/synapse/issues/10784 ))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775 ))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778 ))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779 ))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788 ))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789 ))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798 ))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmFAgvcTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9M6REACpRcTVbsp9Z3c6cajrShHWZr7o4z9y
9gjs38mELM7aysP1rcePsYWgsJtG66YCvmErEjuBGBKOSpuQCTRs+2fDC0m5bpia
Av+LzLyWM8lK2vaeg93xP5aBK7Nspkst3XW7Ststkcu25rZFkR/6gU1dE0+YXVbH
wlohqRGA085cZy2xGy+PalwuY8YWcMI2Vw2T+4pCxM7Awly6uK5ieghppKPZ4Xns
Z6rklnkrsf2yaCHzMZYq6TkCiZaJeXyGAxiBEPCrvOvBgTcfX53ZMa2YESOROgil
b9U2YUFa6d0V2vl1KZoSri/fafhY3LC1H+GJvxNeUFFOo1spKKOluNWPaxvZXQDk
+7MKdIzYp2UHyqn2F9rAf213RmGFvh3QDMxXkjUqlJby/mcxzMYsshTBPExcOR44
/XOOhrQRuOAbYttikXRldEta5R6oST9TKJ6ft0HZyYu0VFS2n5RM/HjPXfCZDPBk
0R6l1GjRf3YYYNu+9XZ1o1nZ49RgOHe3p+Ar5istzgbCL3/7nVo7lnCljldFz6ol
xQUdPCnGA0fOWins7o84gQw1fzvcbEHDlL/45VvwR6y5r3ZvbxFa17Vw1s1eKYhT
AglZvUhy8nXzYrasSagQ/a3xWMOcdpiyHyLCEHvpYdYWEbblYNmihivOrzffJsqd
DaRSoFyMrMI53A==
=fq25
-----END PGP SIGNATURE-----
Merge tag 'v1.43.0rc1' into develop
Synapse 1.43.0rc1 (2021-09-14)
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566 ))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643 ))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712 ))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714 ), [\#10759](https://github.com/matrix-org/synapse/issues/10759 ))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375 ) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244 ) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772 ))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658 ))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730 ))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743 ))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757 ))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733 ))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10760](https://github.com/matrix-org/synapse/issues/10760 ))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774 ))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783 ))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556 ))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648 ))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735 ))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758 ))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html ). ([\#10795](https://github.com/matrix-org/synapse/issues/10795 ))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804 ))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693 ))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424 ](https://github.com/matrix-org/synapse/issues/9424 )). ([\#10704](https://github.com/matrix-org/synapse/issues/10704 ))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601 ))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697 ))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707 ), [\#10728](https://github.com/matrix-org/synapse/issues/10728 ), [\#10736](https://github.com/matrix-org/synapse/issues/10736 ))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727 ))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738 ))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744 ), [\#10745](https://github.com/matrix-org/synapse/issues/10745 ), [\#10746](https://github.com/matrix-org/synapse/issues/10746 ), [\#10771](https://github.com/matrix-org/synapse/issues/10771 ), [\#10773](https://github.com/matrix-org/synapse/issues/10773 ), [\#10781](https://github.com/matrix-org/synapse/issues/10781 ))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748 ))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750 ))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752 ))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753 ))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754 ), [\#10755](https://github.com/matrix-org/synapse/issues/10755 ), [\#10756](https://github.com/matrix-org/synapse/issues/10756 ), [\#10780](https://github.com/matrix-org/synapse/issues/10780 ), [\#10784](https://github.com/matrix-org/synapse/issues/10784 ))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775 ))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778 ))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779 ))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788 ))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789 ))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798 ))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799 ))
2021-09-14 12:59:53 +01:00
b996782df5
Convert media repo's FileInfo to attrs. ( #10785 )
...
This is mostly an internal change, but improves type hints in the
media code.
2021-09-14 07:09:38 -04:00
d725e0956f
1.43.0rc1
2021-09-14 11:47:11 +01:00
319b8b6bef
Name the type of token in "Invalid token" messages ( #10815 )
...
I had one of these error messages yesterday and assumed it was an
invalid auth token (because that was an HTTP query parameter in the
test) I was working on. In fact, it was an invalid next batch token for
syncing.
2021-09-14 11:25:05 +01:00
01c88a09cd
Use direct references for some configuration variables ( #10798 )
...
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
9f111075e8
Fix copy-paste error in the password section of the sample-config. ( #10804 )
2021-09-13 08:58:34 -04:00
003846d68a
Use the BaseReporter super-class for _WrappedRustReporter. ( #10799 )
...
This fixes mypy errors with jaeger-client >= 4.7.0 and should be a no-op
for versions before that.
2021-09-13 08:54:01 -04:00
524b8ead77
Add types to synapse.util. ( #10601 )
2021-09-10 17:03:18 +01:00
63f28e4a0c
Handle room upgrades for spaces ( #10774 )
...
By copying the `room_type` field of the create event and
migrating any non-empty `m.space.child` events to the
new room that is created.
2021-09-10 07:30:05 -04:00
318162f5de
Easy refactors of the user directory ( #10789 )
...
No functional changes here. This came out as I was working to tackle #5677
2021-09-10 10:54:38 +01:00
0c0da36a68
Ask consent on SSO registration with default mxid ( #10733 )
...
Fixes #10732 : consent flow skipped during SSO user registration if username is left at default
Signed-off-by: Andrew Ferrazzutti fair@miscworks.net
2021-09-10 10:36:45 +01:00
7f0565e029
Don't needlessly batch in `add_event_to_cache` ( #10784 )
...
We've already batched up the events previously, and assume in other
places in the events.py file that we have. Removing this makes it easier
to adjust the batch sizes in one place.
2021-09-10 10:16:52 +01:00
273b6861f2
Remove unstable MSC2858 API, including `experimental.msc2858_enabled` config option ( #10693 )
...
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-09 17:59:59 +01:00
a621ba0259
Prefer room v9 for restricted rooms. ( #10772 )
...
Hint to clients via the room capabilities API (MSC3244) that
room version 9 should be preferred for creating a room with
restricted join rules (instead of room version 8).
2021-09-09 07:37:33 -04:00
abedf7d77f
Get rid of `_auth_and_persist_event` ( #10781 )
...
This is only called in two places, and the code seems much clearer without it.
2021-09-08 19:03:08 +01:00
1fdf2cf8e8
Document that /spaces & /hierarchy can be routed to workers. ( #10648 )
...
Also refactors some of the registration of endpoints on workers.
2021-09-08 11:02:31 -04:00
74f01e11c9
Skip handling of push actions for outlier events ( #10780 )
...
Outlier events don't ever have push actions associated with them, so we
can skip some expensive queries during event persistence.
2021-09-08 15:18:35 +01:00
0288e6033b
Add a constant for m.federate. ( #10775 )
2021-09-08 10:00:43 -04:00
66d72b7e17
Change logging of puppeted requests to better differentiate users ( #10779 )
...
This used to be a comma and got accidentally changed to a period in #9654 , but a pipe character is more easier to parse visually.
2021-09-08 12:59:15 +00:00
580a15e039
Request JSON for oEmbed requests (and ignore XML only providers). ( #10759 )
...
This adds the format to the request arguments / URL to
ensure that JSON data is returned (which is all that
Synapse supports).
This also adds additional error checking / filtering to the
configuration file to ignore XML-only providers.
2021-09-08 07:17:52 -04:00
aacdce8fc0
Add some assertions about outliers ( #10773 )
...
I think I have finally teased apart the codepaths which handle outliers, and those that handle non-outliers.
Let's add some assertions to demonstrate my newfound knowledge.
2021-09-08 10:41:13 +01:00
5724883ac2
Persist auth events before the events that rely on them ( #10771 )
...
If we're persisting an event E which has auth_events A1, A2, then we ought to make sure that we correctly auth
and persist A1 and A2, before we blindly accept E.
This PR does part of that - it persists the auth events first - but it does not fully solve the problem, because we
still don't check that the auth events weren't rejected.
2021-09-08 10:37:50 +01:00
857b000996
Merge branch 'master' into develop
2021-09-07 17:19:32 +01:00
6631321687
1.42.0
2021-09-07 16:20:03 +01:00
89ba834818
Use attrs internally for the URL preview code & add documentation. ( #10753 )
2021-09-07 13:10:34 +00:00
a23f3abb9b
Return stripped m.space.child events via the space summary. ( #10760 )
...
The full event content cannot be trusted from this API (as no auth
chain, etc.) is processed over federation. Returning the full event
content was a bug as MSC2946 specifies that only the stripped
state should be returned.
This also avoids calculating aggregations / annotations which go
unused.
2021-09-07 08:43:54 -04:00
f30c9745ab
Underscore-prefix private fields in `FederationEventHandler` ( #10746 )
2021-09-07 11:15:51 +01:00
287108fb2e
Merge tag 'v1.42.0rc2' into develop
...
Synapse 1.42.0rc2 (2021-09-06)
==============================
This version of Synapse removes deprecated room-management admin APIs, removes out-of-date
email pushers, and improves error handling for fallback templates for user-interactive
authentication. For more information on these points, server administrators are
encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375 ). ([\#10747](https://github.com/matrix-org/synapse/issues/10747 ))
Internal Changes
----------------
- Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768 ))
2021-09-06 16:29:59 +01:00
f1c6b76418
Add logging to help debug #9424 ( #10704 )
2021-09-06 16:08:25 +01:00
6e895366ea
Add config option to use non-default manhole password and keys ( #10643 )
2021-09-06 16:08:03 +01:00
e9958d908d
1.42.0rc2
2021-09-06 15:25:23 +01:00
8c9e723fe0
Add a warning when using deprecated template_dir settings ( #10768 )
...
The deprecation itself happened in #10596 which shipped with Synapse v1.41.0. However, it doesn't seem fair to suddenly drop support for these settings in ~4-6w without being more vocal about said deprecation.
2021-09-06 16:23:50 +02:00
b298de780a
Stop using BaseHandler in `FederationEventHandler` ( #10745 )
...
It's now only used in a couple of places, so we can drop it altogether.
2021-09-06 14:49:33 +01:00
40a1fddd1b
Allow `room_alias_name` parameter to be handled by /createRoom calls on workers ( #10757 )
2021-09-06 14:37:15 +01:00
7bb3673f37
Ease searching for M_TOO_LARGE-related error codes ( #10750 )
2021-09-06 14:35:56 +01:00
56e2a30634
Move `maybe_kick_guest_users` out of `BaseHandler` ( #10744 )
...
This is part of my ongoing war against BaseHandler. I've moved kick_guest_users into RoomMemberHandler (since it calls out to that handler anyway), and split maybe_kick_guest_users into the two places it is called.
2021-09-06 12:17:16 +01:00
2ca0d64854
Speed up persisting redacted events ( #10756 )
2021-09-06 10:14:07 +01:00
1ca70fd312
Allow room creator to send MSC2716 related events in existing room versions ( #10566 )
...
* Allow room creator to send MSC2716 related events in existing room versions
Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869
Restoring `get_create_event_for_room_txn` from,
44bb3f0cf5https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org ), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`).
>
> https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642
* Remove and switch away from get_create_event_for_room_txn
* Fix no create event being found because no state events persisted yet
* Fix and add tests for rooms creator bg update
* Populate rooms.creator field for easy lookup
Part of https://github.com/matrix-org/synapse/pull/10566
- Fill in creator whenever we insert into the rooms table
- Add background update to backfill any missing creator values
* Add changelog
* Fix usage
* Remove extra delta already included in #10697
* Don't worry about setting creator for invite
* Only iterate over rows missing the creator
See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898
* Use constant to fetch room creator field
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029
* More protection from other random types
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853
* Move new background update to end of list
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181
* Fix query casing
* Fix ambiguity iterating over cursor instead of list
Fix `psycopg2.ProgrammingError: no results to fetch` error
when tests run with Postgres.
```
SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room
```
---
We use `txn.fetchall` because it will return the results as a
list or an empty list when there are no results.
Docs:
> `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used:
>
> https://www.psycopg.org/docs/cursor.html#cursor-iterable
And I'm guessing iterating over a raw cursor does something weird when there are no results.
---
Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530
```
tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[FAIL]
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column
self.get_success(
File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success
return self.successResultOf(d)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf
self.fail(
twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead:
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback
self._startRunCallbacks(fail)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
self._runCallbacks()
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
result = current_context.run(
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update
await self._do_background_update(desired_duration_ms)
File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update
items_updated = await update_handler(progress, batch_size)
File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column
end = await self.db_pool.runInteraction(
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction
result = await self.runWithConnection(
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection
return await make_deferred_yieldable(
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda>
d.addCallback(lambda x: function(*args, **kwargs))
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
return function(*args, **kwargs)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
raise exception.with_traceback(traceback)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
result = func(conn, *args, **kw)
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func
return func(db_conn, *args, **kwargs)
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction
r = func(cursor, *args, **kwargs)
File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn
for room_id, event_json in txn:
psycopg2.ProgrammingError: no results to fetch
```
* Move code not under the MSC2716 room version underneath an experimental config option
See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909
* Add ordering to rooms creator background update
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277
* Add comment to better document constant
See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458
* Use constant field
2021-09-04 00:58:49 -05:00
92b6ac31b2
Speed up MultiWriterIdGenerator when lots of IDs are in flight. ( #10755 )
2021-09-03 18:23:46 +01:00
ae3c16318b
Support MSC3375: room version 9. ( #10747 )
2021-09-03 12:51:15 -04:00
924276f482
Add a partial index to `presence_stream` to speed up startups ( #10748 )
...
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-03 17:16:56 +01:00
0eae330a26
Use `execute_values` more in PostgreSQL ( #10754 )
...
`execute_values` is a faster version of `execute_batch`.
2021-09-03 15:35:49 +00:00
2cb85bdf75
Raise an error if an unknown preset is used to create a room. ( #10738 )
...
Raises a 400 error instead of a 500 if an unknown preset is passed
from a client to create a room.
2021-09-03 13:46:18 +00:00
ecbfa4fe4f
Additional type hints for client REST servlets (part 5) ( #10736 )
...
Additionally this enforce type hints on all function signatures inside
of the synapse.rest.client package.
2021-09-03 09:22:22 -04:00
f58d202e3f
Fix bug with reusing 'txn' when persisting event. ( #10743 )
...
This will only happen when a server has multiple out of band membership
events in a single room.
2021-09-03 10:59:25 +01:00
c586d6803a
Ignore rooms with unknown room versions in the spaces summary. ( #10727 )
...
This avoids breaking the entire endpoint if a room with
an unsupported room version is encountered.
2021-09-01 17:01:08 +00:00
6258730ebe
Consider the `origin_server_ts` of the `m.space.child` event when ordering rooms. ( #10730 )
...
This updates the ordering of the returned events from the spaces
summary API to that defined in MSC2946 (which updates MSC1772).
Previously a step was skipped causing ordering to be inconsistent with
clients.
2021-09-01 12:59:52 -04:00
d1f1b46c2c
Additional type hints for client REST servlets (part 4) ( #10728 )
2021-09-01 11:59:32 -04:00
dc75fb7f05
Populate `rooms.creator` field for easy lookup ( #10697 )
...
Part of https://github.com/matrix-org/synapse/pull/10566
- Fill in creator whenever we insert into the rooms table
- Add background update to backfill any missing creator values
2021-09-01 16:27:58 +01:00
e059094119
Synapse 1.42.0rc1 (2021-09-01)
...
==============================
Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142 ))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283 ): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452 ))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524 ))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10613](https://github.com/matrix-org/synapse/issues/10613 ), [\#10725](https://github.com/matrix-org/synapse/issues/10725 ))
Bugfixes
--------
- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232 ))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561 ))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581 ), [\#10734](https://github.com/matrix-org/synapse/issues/10734 ))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593 ))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644 ))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654 ))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677 ))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679 ))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684 ))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703 ))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713 ))
Improved Documentation
----------------------
- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192 ))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595 ))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639 ))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708 ))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711 ))
Deprecations and Removals
-------------------------
- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830 ))
Internal Changes
----------------
- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608 ))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614 ), [\#10615](https://github.com/matrix-org/synapse/issues/10615 ), [\#10624](https://github.com/matrix-org/synapse/issues/10624 ), [\#10640](https://github.com/matrix-org/synapse/issues/10640 ))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621 ))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627 ))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629 ), [\#10642](https://github.com/matrix-org/synapse/issues/10642 ))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630 ))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645 ))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 )) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647 ))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651 ), [\#10672](https://github.com/matrix-org/synapse/issues/10672 ))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662 ))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664 ))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665 ), [\#10666](https://github.com/matrix-org/synapse/issues/10666 ), [\#10674](https://github.com/matrix-org/synapse/issues/10674 ))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667 ))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686 ))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692 ))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmEvd3cACgkQLS76LzL7
4EcF0hAAvOjDeH2hn0eI4YX03/BTCesJ5pCTbEJeC9Hsui3SbWamyYarH0Ae0gM6
BP9x5/XkCdsZhKT2OJedtZQUQ0cUws5FdJP87NnE1m7KBW9djbTxmkue3JKaHNTT
ajvEoLe8UjOFbjvyMc+188uWVgtKQ6KLcetwllTpYY3FJ3+yk9CQP1ydYzwfHcHe
hXW3b20WoRdAmH0B27mj1oQzCeXclbj1kCthcBcxH5RytA8bBc1itBtpAP2FFaeS
Ml6paoXSs0wZaXXL5T1MhQtcVOOG5Yh7EPsemdyfqkwhIk9QtV3nmGfC7K5iEeq0
DldTgBNE0wLO+ycdgEZ2oCvitCLT4WDoKkVqKPA91wpuvCMeK5lnDdqbFUSN3hfd
l8Oz+ydhjsCx7+pBBopnsyGrPIHNaN/vySYGF3lPa3ANKS2J0IVlAaJ59bZqzg2C
vzdyMBCnn0fhnXMkADPLnIYqtebvC09JlPBRGo5BKmwmlM8Ea1dLQ+Lcy5vXalcw
+GaYw3U805c03EvsWQxWchjwq6IwMeP06YpK/1ruMjlrCWYJp3V7f/4hXjlTdYm5
OEWc/kHOnlLfH2cA4aS0sV5WsnEuftotQIzBXuXWw5hSMdFmH7eAviPg0vXgfeA/
i0FOkOgEYR8O3RMh4Cip8+h44t8SPNjYUFJK9ip3DmNateeUCH0=
=XxMc
-----END PGP SIGNATURE-----
Merge tag 'v1.42.0rc1' into develop
Synapse 1.42.0rc1 (2021-09-01)
==============================
Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142 ))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283 ): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452 ))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524 ))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10613](https://github.com/matrix-org/synapse/issues/10613 ), [\#10725](https://github.com/matrix-org/synapse/issues/10725 ))
Bugfixes
--------
- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232 ))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561 ))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581 ), [\#10734](https://github.com/matrix-org/synapse/issues/10734 ))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593 ))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644 ))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654 ))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677 ))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679 ))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684 ))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703 ))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713 ))
Improved Documentation
----------------------
- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192 ))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595 ))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639 ))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708 ))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711 ))
Deprecations and Removals
-------------------------
- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830 ))
Internal Changes
----------------
- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608 ))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614 ), [\#10615](https://github.com/matrix-org/synapse/issues/10615 ), [\#10624](https://github.com/matrix-org/synapse/issues/10624 ), [\#10640](https://github.com/matrix-org/synapse/issues/10640 ))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621 ))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627 ))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629 ), [\#10642](https://github.com/matrix-org/synapse/issues/10642 ))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630 ))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645 ))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 )) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647 ))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651 ), [\#10672](https://github.com/matrix-org/synapse/issues/10672 ))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662 ))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664 ))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665 ), [\#10666](https://github.com/matrix-org/synapse/issues/10666 ), [\#10674](https://github.com/matrix-org/synapse/issues/10674 ))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667 ))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686 ))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692 ))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706 ))
2021-09-01 14:58:14 +01:00
d9069388f3
Correctly include room avatars in email notifications ( #10658 )
...
Judging by the template, this was intended ages ago, but we never
actually passed an avatar URL to the template. So let's provide one.
Closes #1546 .
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-09-01 13:48:41 +01:00
f8bf83b811
Skip the final GC on shutdown to improve restart times ( #10712 )
...
Use `gc.freeze()` on exit to exclude all existing objects from the final GC.
In testing, this sped up shutdown by up to a few seconds.
`gc.freeze()` runs in constant time, so there is little chance of performance
regression.
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-01 11:55:31 +01:00
6b2aca473a
1.42.0rc1
2021-09-01 11:47:24 +01:00
3693ea61f5
Fix iteration in _remove_deleted_email_pushers background job. ( #10734 )
2021-09-01 09:13:01 +00:00
e2481dbe93
Allow configuration of the oEmbed URLs. ( #10714 )
...
This adds configuration options (under an `oembed` section) to
configure which URLs are matched to use oEmbed for URL
previews.
2021-08-31 18:37:07 -04:00
287918e2d4
Additional type hints for the client REST servlets (part 3). ( #10707 )
2021-08-31 17:22:29 +00:00
78e590d473
Move the sessions delta to the latest schema version. ( #10725 )
...
This was erroneously put under schema version 62 instead of 63.
2021-08-31 16:38:43 +00:00
5d9e7e0c71
Merge branch 'master' into develop
2021-08-31 14:09:48 +01:00
a4c8a2f08b
1.41.1
2021-08-31 13:43:28 +01:00
cb35df940a
Merge pull request from GHSA-jj53-8fmw-f2w2
2021-08-31 11:24:09 +01:00
52c7a51cfc
Merge pull request from GHSA-3x4c-pq33-4w3q
...
* Add some tests to characterise the problem
Some failing. Current states:
RoomsMemberListTestCase
test_get_member_list ...
[OK]
test_get_member_list_mixed_memberships ...
[OK]
test_get_member_list_no_permission ...
[OK]
test_get_member_list_no_permission_former_member ...
[OK]
test_get_member_list_no_permission_former_member_with_at_token ...
[FAIL]
test_get_member_list_no_room ...
[OK]
test_get_member_list_no_permission_with_at_token ...
[FAIL]
* Correct the tests
* Check user is/was member before divulging room membership
* Pull out only the 1 membership event we want.
* Update tests/rest/client/v1/test_rooms.py
Co-authored-by: Erik Johnston <erik@matrix.org>
* Fixup tests (following apply review suggestion)
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-08-31 10:09:58 +01:00
e3abc0a5cc
Merge remote-tracking branch 'origin/release-v1.41' into develop
2021-08-27 16:35:02 +01:00
8f98260552
Fix incompatibility with Twisted < 21. ( #10713 )
...
Turns out that the functionality added in #10546 to skip TLS was incompatible
with older Twisted versions, so we need to be a bit more inventive.
Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is
really hard.
2021-08-27 16:33:41 +01:00
029b7ad7b9
Remove unused `compare_digest` function. ( #10706 )
2021-08-27 07:08:02 -04:00
e62cdbef1a
Improve ServerNoticeServlet to avoid duplicate requests ( #10679 )
...
Fixes : #9544
2021-08-27 09:16:40 +00:00
c4fa4f37cb
Fix perf of fetching the same events many times. ( #10703 )
...
The code to deduplicate repeated fetches of the same set of events was
N^2 (over the number of events requested), which could lead to a process
being completely wedged.
The main fix is to deduplicate the returned deferreds so we only await
on a deferred once rather than many times. Seperately, when handling the
returned events from the defrered we only add the events we care about
to the event map to be returned (so that we don't pay the price of
inserting extraneous events into the dict).
2021-08-27 09:15:50 +00:00
1800aabfc2
Split `FederationHandler` in half ( #10692 )
...
The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
2021-08-26 21:41:44 +01:00
96715d7633
Make `backfill` and `get_missing_events` use the same codepath ( #10645 )
...
Given that backfill and get_missing_events are basically the same thing, it's somewhat crazy that we have entirely separate code paths for them. This makes backfill use the existing get_missing_events code, and then clears up all the unused code.
2021-08-26 18:34:57 +01:00
40f619eaa5
Validate new m.room.power_levels events ( #10232 )
...
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-08-26 17:07:58 +01:00
ad17fbd20e
Remove pushers when deleting 3pid from account ( #10581 )
...
When a user deletes an email from their account it will
now also remove all pushers for that email and that user
(even if these pushers were created by a different client)
2021-08-26 13:53:57 +01:00
1aa0dad021
Additional type hints for REST servlets (part 2). ( #10674 )
...
Applies the changes from #10665 to additional modules.
2021-08-26 11:53:52 +00:00
5548fe0978
Cache the result of fetching the room hierarchy over federation. ( #10647 )
2021-08-26 07:16:53 -04:00
b45cc1530b
Make a note to leave a summary when one is bumping the schema version ( #10621 )
...
I found this easy to miss (and evidently, it looks like it was missed for schema version 62).
2021-08-25 17:00:44 +01:00
7367473f96
Fix error when selecting between thumbnails with the same quality ( #10684 )
...
Fixes #10318
2021-08-25 09:51:08 +00:00
cd22fb568a
Merge branch 'master' into develop
2021-08-24 17:13:29 +01:00
6f77a3d433
1.41.0
2021-08-24 15:31:55 +01:00
d12ba52f17
Persist room hierarchy pagination sessions to the database. ( #10613 )
2021-08-24 08:14:03 -04:00
15db8b7c7f
Correctly initialise the `synapse_user_logins` metric. ( #10677 )
...
Fix a bug where the prometheus metrics for SSO logins wouldn't be initialised
until the first user logged in with a given auth provider.
2021-08-24 09:17:51 +00:00
0c1d6f65d7
Enforce the max length for per-room display names / avatar URLs. ( #10654 )
...
To match the maximum lengths allowed for profile data.
2021-08-23 11:25:33 -04:00
bd7d398b05
Additional type hints for the sync REST servlet. ( #10666 )
2021-08-23 08:14:42 -04:00
2af6d31b78
Addtional type hints for the REST servlets. ( #10665 )
2021-08-23 08:14:17 -04:00
31dac7ffee
Do not include stack traces for known exceptions when trying multiple federation destinations. ( #10662 )
2021-08-23 08:00:25 -04:00
947dbbdfd1
Implement MSC3231: Token authenticated registration ( #10142 )
...
Signed-off-by: Callum Brown <callum@calcuode.com>
This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ).
2021-08-21 22:14:43 +01:00
ee3b2ac59a
Validate device_keys for C-S /keys/query requests ( #10593 )
...
* Validate device_keys for C-S /keys/query requests
Closes #10354
A small, not particularly critical fix. I'm interested in seeing if we
can find a more systematic approach though. #8445 is the place for any discussion.
2021-08-20 15:47:03 +01:00
5cda75fede
Set room version 8 as preferred for restricted rooms. ( #10571 )
2021-08-20 07:17:50 -04:00
e81d62009e
Split `on_receive_pdu` in half ( #10640 )
...
Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.
2021-08-19 17:05:12 +00:00
50af1efe4b
Extract `_resolve_state_at_missing_prevs` ( #10624 )
...
This is a follow-up to #10615 : it takes the code that constructs the state at a backwards extremity, and extracts it to a separate method.
2021-08-19 17:31:40 +01:00
000aa89be6
Do not include rooms with an unknown room version in a sync response. ( #10644 )
...
A user will still see this room if it is in a local cache, but it will
not reappear if clearing the cache and reloading.
2021-08-19 11:12:55 -04:00
b5fef6054a
Support MSC3283: Expose `enable_set_displayname` in capabilities ( #10452 )
2021-08-19 09:40:40 +00:00
220f901229
Remove not needed database updates in modify user admin API ( #10627 )
2021-08-19 10:25:05 +01:00
0c3565da4c
Additional type hints for the proxy agent and SRV resolver modules. ( #10608 )
2021-08-18 13:53:20 -04:00
78a70a2e0b
Merge branch 'release-v1.41' into develop
2021-08-18 17:02:47 +01:00
d9856d9150
Fix weakref_slot parameter for room member storage attrs. ( #10642 )
...
Follow-up to #10629 which set it to true, not false.
2021-08-18 15:00:37 +00:00
49cb7eae97
1.41.0rc1
2021-08-18 15:52:11 +01:00
bec01c0758
Convert room member storage tuples to attrs. ( #10629 )
...
Instead of using namedtuples. This helps with asserting type hints
and code completion.
2021-08-18 09:22:07 -04:00
3692f7fd33
Mount /_synapse/admin/v1/users/{userId}/media admin API on media workers only ( #10628 )
...
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-08-18 13:25:12 +01:00
6e613a10d0
Display an error page during failure of fallback UIA. ( #10561 )
2021-08-18 08:13:35 -04:00
964f29cb6f
Refactor `on_receive_pdu` code ( #10615 )
...
* drop room pdu linearizer sooner
No point holding onto it while we recheck the db
* move out `missing_prevs` calculation
we're going to need `missing_prevs` whatever we do, so we may as well calculate
it eagerly and just update it if it gets outdated.
* Add another `if missing_prevs` condition
this should be a no-op, since all the code inside the block already checks `if
missing_prevs`
* reorder if conditions
This shouldn't change the logic at all.
* Push down `min_depth` read
No point reading it from the database unless we're going to use it.
* Collect the sent_to_us_directly code together
Move the remaining `sent_to_us_directly` code inside the `if
sent_to_us_directly` block.
* Properly separate the `not sent_to_us_directly` branch
Since the only way this second block is now reachable is if we
*didn't* go into the `sent_to_us_directly` branch, we can replace it with a
simple `else`.
* changelog
2021-08-18 12:36:22 +01:00
6a5f8fbcda
Use auto-attribs for attrs classes for sync. ( #10630 )
2021-08-18 07:27:32 -04:00
5581dd7bf7
Allow modules to run looping call on all instances ( #10638 )
...
By default the calls only ran on the worker configured to run background
tasks.
2021-08-18 10:21:11 +00:00
430241a1e9
Remove deprecated Shutdown Room and Purge Room Admin API ( #8830 )
2021-08-17 21:19:13 +01:00
703e3a9e85
Allow /createRoom to be run on workers ( #10564 )
...
Fixes https://github.com/matrix-org/synapse/issues/7867
2021-08-17 14:33:16 +01:00
1a9f531c79
Port the PresenceRouter module interface to the new generic interface ( #10524 )
...
Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
2021-08-17 13:22:45 +00:00
84469bdac7
Remove the unused public_room_list_stream ( #10565 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-08-17 14:02:50 +01:00
5639759980
Centralise the custom template directory ( #10596 )
...
Several configuration sections are using separate settings for custom template directories, which can be confusing. This PR adds a new top-level configuration for a custom template directory which is then used for every module. The only exception is the consent templates, since the consent template directory require a specific hierarchy, so it's probably better that it stays separate from everything else.
2021-08-17 14:45:24 +02:00
c4cf0c0473
Attempt to pull from the legacy spaces summary API over federation. ( #10583 )
...
If the new /hierarchy API does not exist on all destinations,
fallback to querying the /spaces API and translating the results.
This is a backwards compatibility hack since not all of the
federated homeservers will update at the same time.
2021-08-17 08:19:12 -04:00
272b89d547
Stop setting the outlier flag for things that aren't ( #10614 )
...
Marking things as outliers to inhibit pushes is a sledgehammer to crack a
nut. Move the test further down the stack so that we just inhibit the thing we
want.
2021-08-17 13:13:42 +01:00
5f7b1e1f27
Make `PeriodicallyFlushingMemoryHandler` the default logging handler. ( #10518 )
2021-08-17 13:13:11 +01:00
642a42edde
Flatten the synapse.rest.client package ( #10600 )
2021-08-17 11:57:58 +00:00
b62eba7705
Always list fallback key types in /sync ( #10623 )
2021-08-17 12:32:25 +01:00
3bcd525b46
Allow to edit `external_ids` by Edit User admin API ( #10598 )
...
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-17 11:56:11 +01:00
58f0d97275
update links to schema doc ( #10620 )
2021-08-17 10:45:35 +00:00
ae2714c1f3
Allow using several custom template directories ( #10587 )
...
Allow using several directories in read_templates.
2021-08-17 10:23:14 +00:00
a933c2c7d8
Add an admin API to check if a username is available ( #10578 )
...
This adds a new API GET /_synapse/admin/v1/username_available?username=foo to check if a username is available. It is the counterpart to https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available , except that it works even if registration is disabled.
2021-08-17 10:52:38 +01:00
19e51b14d2
Manhole: wrap coroutines in `defer.ensureDeferred` automatically ( #10602 )
2021-08-16 18:11:48 +01:00
5af83efe8d
Validate the max_rooms_per_space parameter to ensure it is non-negative. ( #10611 )
2021-08-16 12:01:30 -04:00
0ace38b7b3
Experimental support for MSC3266 Room Summary API. ( #10394 )
2021-08-16 14:49:12 +00:00
87b62f8bb2
Split `synapse.federation.transport.server` into multiple files. ( #10590 )
2021-08-16 10:14:31 -04:00
2d9ca4ca77
Clean up some logging in the federation event handler ( #10591 )
...
* Include outlier status in `str(event)`
In places where we log event objects, knowing whether or not you're dealing
with an outlier is super useful.
* Remove duplicated logging in get_missing_events
When we process events received from get_missing_events, we log them twice
(once in `_get_missing_events_for_pdu`, and once in `on_receive_pdu`). Reduce
the duplication by removing the logging in `on_receive_pdu`, and ensuring the
call sites do sensible logging.
* log in `on_receive_pdu` when we already have the event
* Log which prev_events we are missing
* changelog
2021-08-16 13:19:02 +01:00
7de445161f
Support federation in the new spaces summary API (MSC2946). ( #10569 )
2021-08-16 08:06:17 -04:00
a3a7514570
Handle string read receipt data ( #10606 )
...
* Handle string read receipt data
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Test that we handle string read receipt data
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Add changelog for #10606
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Add docs
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Ignore malformed RRs
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Only surround hidden = ...
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Remove unnecessary argument
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Update changelog.d/10606.bugfix
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-08-16 12:22:38 +01:00
c8d54be44c
Move /batch_send to /v2_alpha directory (MSC2716) ( #10576 )
...
* Move /batch_send to /v2_alpha directory
As pointed out by @erikjohnston,
https://github.com/matrix-org/synapse/pull/10552#discussion_r685836624
2021-08-13 14:37:24 -05:00
c12b5577f2
Fix a harmless exception when the staged events queue is empty. ( #10592 )
2021-08-13 11:49:06 +00:00
98a3355d9a
Update the pagination parameter name based on MSC2946 review. ( #10579 )
2021-08-11 15:44:45 -04:00
915b37e5ef
Admin API to delete media for a specific user ( #10558 )
2021-08-11 19:29:59 +00:00
3ebb6694f0
Allow requesting the summary of a space which is joinable. ( #10580 )
...
As opposed to only allowing the summary of spaces which the user is
already in or has world-readable visibility.
This makes the logic consistent with whether a space/room is returned
as part of a space and whether a space summary can start at a space.
2021-08-11 15:04:51 -04:00
5acd8b5a96
Expire old spaces summary pagination sessions. ( #10574 )
2021-08-11 18:52:09 +00:00
2ae2a04616
Clarify error message when joining a restricted room. ( #10572 )
2021-08-11 14:31:39 -04:00
339c3918e1
support federation queries through http connect proxy ( #10475 )
...
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-11 15:34:59 +01:00
8c654b7309
Only return state events that the AS passed in via `state_events_at_start` (MSC2716) ( #10552 )
...
* Only return state events that the AS passed in via state_events_at_start
As discovered by @Half-Shot in
https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684158448
Part of MSC2716
* Add changelog
* Fix changelog extension
2021-08-10 18:10:40 -05:00
fe1d0c8618
Add local support for the new spaces summary endpoint (MSC2946) ( #10549 )
...
This adds support for the /hierarchy endpoint, which is an
update to MSC2946. Currently this only supports rooms known
locally to the homeserver.
2021-08-10 13:08:17 -04:00
691593bf71
Fix an edge-case with invited rooms over federation in the spaces summary. ( #10560 )
...
If a room which the requesting user was invited to was queried over
federation it will now properly appear in the spaces summary (instead
of being stripped out by the requesting server).
2021-08-10 14:56:54 +00:00
b5de77cf86
Merge branch 'master' into develop
2021-08-10 14:23:57 +01:00
9f7c038272
1.40.0
2021-08-10 13:50:58 +01:00
7afb615839
When redacting, keep event fields around that maintain the historical event structure intact (MSC2716) ( #10538 )
...
* Keep event fields that maintain the historical event structure intact
Fix https://github.com/matrix-org/synapse/issues/10521
* Add changelog
* Bump room version
* Better changelog text
* Fix up room version after develop merge
2021-08-09 20:23:31 -05:00
6b61debf5c
Do not remove `status_msg` when user going offline ( #10550 )
...
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-09 16:21:04 +00:00
f8e86b7d2e
Merge branch 'release-v1.40' into develop
2021-08-09 14:44:28 +01:00
Aaron Raimist