OpenCloud
/
MatrixSynapse
mirror of https://github.com/matrix-org/synapse
1
0
Fork 0
Code Issues Releases Wiki Activity

Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes

erikj/disable_catchup_to_hq
Richard van der Hoff 2021-03-24 12:45:54 +00:00
parent 05ec9e8d37 c73cc2c2ad
commit ea74189a90
197 changed files with 2098 additions and 531 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

86
CHANGES.md
View File

@ -1,8 +1,86 @@
Removal warning Synapse 1.30.0 (2021-03-22)
--------------- ===========================
Note that this release deprecates the ability for appservices to
call `POST /_matrix/client/r0/register` without the body parameter `type`. Appservice
developers should use a `type` value of `m.login.application_service` as
per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions).
In future releases, calling this endpoint with an access token - but without a `m.login.application_service`
type - will fail.
No significant changes.
Synapse 1.30.0rc1 (2021-03-16)
==============================
Features
--------
- Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](https://github.com/matrix-org/synapse/issues/9510), [\#9511](https://github.com/matrix-org/synapse/issues/9511), [\#9573](https://github.com/matrix-org/synapse/issues/9573))
- Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](https://github.com/matrix-org/synapse/issues/9540))
- Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](https://github.com/matrix-org/synapse/issues/9549))
- Optimise handling of incomplete room history for incoming federation. ([\#9601](https://github.com/matrix-org/synapse/issues/9601))
- Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](https://github.com/matrix-org/matrix-doc/pull/2858)). ([\#9617](https://github.com/matrix-org/synapse/issues/9617))
- Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](https://github.com/matrix-org/synapse/issues/9626))
Bugfixes
--------
- Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](https://github.com/matrix-org/synapse/issues/9473))
- Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](https://github.com/matrix-org/synapse/issues/9542), [\#9583](https://github.com/matrix-org/synapse/issues/9583))
- Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](https://github.com/matrix-org/synapse/issues/9567))
- Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](https://github.com/matrix-org/synapse/issues/9587))
- Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](https://github.com/matrix-org/synapse/issues/9597))
- Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](https://github.com/matrix-org/synapse/issues/9620))
- Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](https://github.com/matrix-org/synapse/issues/9623))
Updates to the Docker image
---------------------------
- Make use of an improved malloc implementation (`jemalloc`) in the docker image. ([\#8553](https://github.com/matrix-org/synapse/issues/8553))
Improved Documentation
----------------------
- Add relayd entry to reverse proxy example configurations. ([\#9508](https://github.com/matrix-org/synapse/issues/9508))
- Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](https://github.com/matrix-org/synapse/issues/9550))
- Link to the "List user's media" admin API from the media admin API docs. ([\#9571](https://github.com/matrix-org/synapse/issues/9571))
- Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](https://github.com/matrix-org/synapse/issues/9580))
- Clarify the sample configuration for `stats` settings. ([\#9604](https://github.com/matrix-org/synapse/issues/9604))
Deprecations and Removals
-------------------------
- The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](https://github.com/matrix-org/synapse/issues/9540))
- Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](https://github.com/matrix-org/synapse/issues/9559))
Internal Changes
----------------
- Add tests to ResponseCache. ([\#9458](https://github.com/matrix-org/synapse/issues/9458))
- Add type hints to purge room and server notice admin API. ([\#9520](https://github.com/matrix-org/synapse/issues/9520))
- Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](https://github.com/matrix-org/synapse/issues/9523))
- Fix incorrect type hints. ([\#9528](https://github.com/matrix-org/synapse/issues/9528), [\#9543](https://github.com/matrix-org/synapse/issues/9543), [\#9591](https://github.com/matrix-org/synapse/issues/9591), [\#9608](https://github.com/matrix-org/synapse/issues/9608), [\#9618](https://github.com/matrix-org/synapse/issues/9618))
- Add an additional test for purging a room. ([\#9541](https://github.com/matrix-org/synapse/issues/9541))
- Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](https://github.com/matrix-org/synapse/issues/9560))
- Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](https://github.com/matrix-org/synapse/issues/9561))
- Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](https://github.com/matrix-org/synapse/issues/9562))
- Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](https://github.com/matrix-org/synapse/issues/9563))
- Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](https://github.com/matrix-org/synapse/issues/9568))
- Improve efficiency of calculating the auth chain in large rooms. ([\#9576](https://github.com/matrix-org/synapse/issues/9576))
- Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](https://github.com/matrix-org/synapse/issues/9586))
- Add logging for redis connection setup. ([\#9590](https://github.com/matrix-org/synapse/issues/9590))
- Improve logging when processing incoming transactions. ([\#9596](https://github.com/matrix-org/synapse/issues/9596))
- Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](https://github.com/matrix-org/synapse/issues/9604))
- Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](https://github.com/matrix-org/synapse/issues/9619))
Note that this release deprecates the ability for appservices to call `POST /_matrix/client/r0/register` without the body parameter `type`. Appservice developers should use a `type` value of `m.login.application_service` as per the spec. In future releases, calling this endpoint with an access token but
without a valid type will fail.
Synapse 1.29.0 (2021-03-08) Synapse 1.29.0 (2021-03-08)
=========================== ===========================

1
changelog.d/8553.docker
View File

@ -1 +0,0 @@
Use jemalloc if available in docker.

1
changelog.d/9411.misc Normal file
View File

@ -0,0 +1 @@
Preparatory steps for removing redundant `outlier` data from `event_json.internal_metadata` column.

1
changelog.d/9458.misc
View File

@ -1 +0,0 @@
Add tests to ResponseCache.

1
changelog.d/9473.bugfix
View File

@ -1 +0,0 @@
Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`.

1
changelog.d/9499.misc Normal file
View File

@ -0,0 +1 @@
Introduce bugbear to the test suite and fix some of it's lint violations.

1
changelog.d/9508.doc
View File

@ -1 +0,0 @@
Add relayd entry to reverse proxy example configurations.

1
changelog.d/9510.feature
View File

@ -1 +0,0 @@
Add prometheus metrics for number of users successfully registering and logging in.

1
changelog.d/9511.feature
View File

@ -1 +0,0 @@
Add prometheus metrics for number of users successfully registering and logging in.

1
changelog.d/9520.misc
View File

@ -1 +0,0 @@
Add type hints to purge room and server notice admin API.

1
changelog.d/9523.misc
View File

@ -1 +0,0 @@
Add extra logging to ObservableDeferred when callbacks throw exceptions.

1
changelog.d/9528.misc
View File

@ -1 +0,0 @@
Fix incorrect type hints.

1
changelog.d/9540.feature
View File

@ -1 +0,0 @@
Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers.

1
changelog.d/9540.removal
View File

@ -1 +0,0 @@
The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`.

1
changelog.d/9541.misc
View File

@ -1 +0,0 @@
Add an additional test for purging a room.

1
changelog.d/9542.bugfix
View File

@ -1 +0,0 @@
Purge chain cover indexes for events that were purged prior to Synapse v1.29.0.

1
changelog.d/9543.misc
View File

@ -1 +0,0 @@
Fix incorrect type hints.

1
changelog.d/9549.feature
View File

@ -1 +0,0 @@
Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets.

1
changelog.d/9550.doc
View File

@ -1 +0,0 @@
Improve the SAML2 upgrade notes for 1.27.0.

1
changelog.d/9559.removal
View File

@ -1 +0,0 @@
Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release.

1
changelog.d/9560.misc
View File

@ -1 +0,0 @@
Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting.

1
changelog.d/9561.misc
View File

@ -1 +0,0 @@
Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle.

1
changelog.d/9562.misc
View File

@ -1 +0,0 @@
Fix spurious errors reported by the `config-lint.sh` script.

1
changelog.d/9563.misc
View File

@ -1 +0,0 @@
Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper.

1
changelog.d/9567.bugfix
View File

@ -1 +0,0 @@
Fix bug where federation requests were not correctly retried on 5xx responses.

1
changelog.d/9568.misc
View File

@ -1 +0,0 @@
Do not have mypy ignore type hints from unpaddedbase64.

1
changelog.d/9571.doc
View File

@ -1 +0,0 @@
Link to the "List user's media" admin API from the media admin API docs.

1
changelog.d/9573.feature
View File

@ -1 +0,0 @@
Add prometheus metrics for number of users successfully registering and logging in.

1
changelog.d/9576.misc
View File

@ -1 +0,0 @@
Improve efficiency of calculating the auth chain in large rooms.

1
changelog.d/9580.doc
View File

@ -1 +0,0 @@
Clarify the spam checker modules documentation example to mention that `parse_config` is a required method.

1
changelog.d/9583.bugfix
View File

@ -1 +0,0 @@
Purge chain cover indexes for events that were purged prior to Synapse v1.29.0.

1
changelog.d/9585.bugfix Normal file
View File

@ -0,0 +1 @@
Fix a longstanding bug that could cause issues when editing a reply to a message.

1
changelog.d/9586.misc
View File

@ -1 +0,0 @@
Convert `synapse.types.Requester` to an `attrs` class.

1
changelog.d/9587.bugfix
View File

@ -1 +0,0 @@
Re-Activating account with admin API when local passwords are disabled.

1
changelog.d/9588.bugfix Normal file
View File

@ -0,0 +1 @@
Fix the `/capabilities` endpoint to return `m.change_password` as disabled if the local password database is not used for authentication. Contributed by @dklimpel.

1
changelog.d/9590.misc
View File

@ -1 +0,0 @@
Add logging for redis connection setup.

1
changelog.d/9591.misc
View File

@ -1 +0,0 @@
Fix incorrect type hints.

1
changelog.d/9596.misc
View File

@ -1 +0,0 @@
Improve logging when processing incoming transactions.

1
changelog.d/9597.bugfix
View File

@ -1 +0,0 @@
Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages.

1
changelog.d/9601.feature
View File

@ -1 +0,0 @@
Optimise handling of incomplete room history for incoming federation.

1
changelog.d/9604.doc
View File

@ -1 +0,0 @@
Clarify the sample configuration for `stats` settings.

1
changelog.d/9604.misc
View File

@ -1 +0,0 @@
Remove unused `stats.retention` setting, and emit a warning if stats are disabled.

1
changelog.d/9608.misc
View File

@ -1 +0,0 @@
Fix incorrect type hints.

1
changelog.d/9609.feature Normal file
View File

@ -0,0 +1 @@
Logins using OpenID Connect can require attributes on the `userinfo` response in order to login. Contributed by Hubbe King.

1
changelog.d/9612.docker Normal file
View File

@ -0,0 +1 @@
Include [opencontainers labels](https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys) in the Docker image.

1
changelog.d/9617.feature
View File

@ -1 +0,0 @@
Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](https://github.com/matrix-org/matrix-doc/pull/2858)).

1
changelog.d/9618.misc
View File

@ -1 +0,0 @@
Fix incorrect type hints.

1
changelog.d/9619.misc
View File

@ -1 +0,0 @@
Prevent attempting to bundle aggregations for state events in /context APIs.

1
changelog.d/9620.bugfix
View File

@ -1 +0,0 @@
Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`.

1
changelog.d/9623.bugfix
View File

@ -1 +0,0 @@
Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request.

1
changelog.d/9626.feature
View File

@ -1 +0,0 @@
Tell spam checker modules about the SSO IdP a user registered through if one was used.

1
changelog.d/9631.misc Normal file
View File

@ -0,0 +1 @@
Add additional type hints to the Homeserver object.

1
changelog.d/9634.misc Normal file
View File

@ -0,0 +1 @@
Only save remote cross-signing and device keys if they're different from the current ones.

1
changelog.d/9636.bugfix Normal file
View File

@ -0,0 +1 @@
Checks if passwords are allowed before setting it for the user.

1
changelog.d/9637.misc Normal file
View File

@ -0,0 +1 @@
Rename storage function to fix spelling and not conflict with another functions name.

1
changelog.d/9638.misc Normal file
View File

@ -0,0 +1 @@
Add additional type hints to the Homeserver object.

1
changelog.d/9639.bugfix Normal file
View File

@ -0,0 +1 @@
Fix bug where federation sending can stall due to `concurrent access` database exceptions when it falls behind.

1
changelog.d/9640.misc Normal file
View File

@ -0,0 +1 @@
Improve performance of federation catch up by sending events the latest events in the room to the remote, rather than just the last event sent by the local server.

1
changelog.d/9643.feature Normal file
View File

@ -0,0 +1 @@
Add initial experimental support for a "space summary" API.

1
changelog.d/9644.feature Normal file
View File

@ -0,0 +1 @@
Implement the busy presence state as described in [MSC3026](https://github.com/matrix-org/matrix-doc/pull/3026).

1
changelog.d/9645.misc Normal file
View File

@ -0,0 +1 @@
In the `federation_client` commandline client, stop automatically adding the URL prefix, so that servlets on other prefixes can be tested.

1
changelog.d/9647.misc Normal file
View File

@ -0,0 +1 @@
In the `federation_client` commandline client, handle inline `signing_key`s in `homeserver.yaml`.

1
changelog.d/9649.misc Normal file
View File

@ -0,0 +1 @@
Fixed some antipattern issues to improve code quality.

1
changelog.d/9652.feature Normal file
View File

@ -0,0 +1 @@
Add initial experimental support for a "space summary" API.

1
changelog.d/9653.feature Normal file
View File

@ -0,0 +1 @@
Add initial experimental support for a "space summary" API.

1
changelog.d/9657.feature Normal file
View File

@ -0,0 +1 @@
Add support for credentials for proxy authentication in the `HTTPS_PROXY` environment variable.

1
changelog.d/9665.misc Normal file
View File

@ -0,0 +1 @@
Import `HomeServer` from the proper module.

1
changelog.d/9674.misc Normal file
View File

@ -0,0 +1 @@
Increase default join ratelimiting burst rate.

1
changelog.d/9675.misc Normal file
View File

@ -0,0 +1 @@
Add additional type hints to the Homeserver object.

1
changelog.d/9676.misc Normal file
View File

@ -0,0 +1 @@
Add type hints to third party event rules and visibility modules.

1
changelog.d/9678.misc Normal file
View File

@ -0,0 +1 @@
Bump mypy-zope to 0.2.13 to fix "Cannot determine consistent method resolution order (MRO)" errors when running mypy a second time.

1
changelog.d/9679.doc Normal file
View File

@ -0,0 +1 @@
Improve worker documentation for fallback/web auth endpoints.

6
debian/changelog vendored
View File

@ -1,3 +1,9 @@
matrix-synapse-py3 (1.30.0) stable; urgency=medium
* New synapse release 1.30.0.
-- Synapse Packaging team <packages@matrix.org> Mon, 22 Mar 2021 13:15:34 +0000
matrix-synapse-py3 (1.29.0) stable; urgency=medium matrix-synapse-py3 (1.29.0) stable; urgency=medium
[ Jonathan de Jong ] [ Jonathan de Jong ]

5
docker/Dockerfile
View File

@ -18,6 +18,11 @@ ARG PYTHON_VERSION=3.8
### ###
FROM docker.io/python:${PYTHON_VERSION}-slim as builder FROM docker.io/python:${PYTHON_VERSION}-slim as builder
LABEL org.opencontainers.image.url='https://matrix.org/docs/projects/server/synapse'
LABEL org.opencontainers.image.documentation='https://github.com/matrix-org/synapse/blob/master/docker/README.md'
LABEL org.opencontainers.image.source='https://github.com/matrix-org/synapse.git'
LABEL org.opencontainers.image.licenses='Apache-2.0'
# install the OS build deps # install the OS build deps
RUN apt-get update && apt-get install -y \ RUN apt-get update && apt-get install -y \
build-essential \ build-essential \

28
docs/sample_config.yaml
View File

@ -869,10 +869,10 @@ log_config: "CONFDIR/SERVERNAME.log.config"
#rc_joins: #rc_joins:
# local: # local:
# per_second: 0.1 # per_second: 0.1
# burst_count: 3 # burst_count: 10
# remote: # remote:
# per_second: 0.01 # per_second: 0.01
# burst_count: 3 # burst_count: 10
# #
#rc_3pid_validation: #rc_3pid_validation:
# per_second: 0.003 # per_second: 0.003
@ -1873,6 +1873,24 @@ saml2_config:
# which is set to the claims returned by the UserInfo Endpoint and/or # which is set to the claims returned by the UserInfo Endpoint and/or
# in the ID Token. # in the ID Token.
# #
# It is possible to configure Synapse to only allow logins if certain attributes
# match particular values in the OIDC userinfo. The requirements can be listed under
# `attribute_requirements` as shown below. All of the listed attributes must
# match for the login to be permitted. Additional attributes can be added to
# userinfo by expanding the `scopes` section of the OIDC config to retrieve
# additional information from the OIDC provider.
#
# If the OIDC claim is a list, then the attribute must match any value in the list.
# Otherwise, it must exactly match the value of the claim. Using the example
# below, the `family_name` claim MUST be "Stephensson", but the `groups`
# claim MUST contain "admin".
#
# attribute_requirements:
# - attribute: family_name
# value: "Stephensson"
# - attribute: groups
# value: "admin"
#
# See https://github.com/matrix-org/synapse/blob/master/docs/openid.md # See https://github.com/matrix-org/synapse/blob/master/docs/openid.md
# for information on how to configure these options. # for information on how to configure these options.
# #
@ -1905,6 +1923,9 @@ oidc_providers:
# localpart_template: "{{ user.login }}" # localpart_template: "{{ user.login }}"
# display_name_template: "{{ user.name }}" # display_name_template: "{{ user.name }}"
# email_template: "{{ user.email }}" # email_template: "{{ user.email }}"
# attribute_requirements:
# - attribute: userGroup
# value: "synapseUsers"
# For use with Keycloak # For use with Keycloak
# #
@ -1914,6 +1935,9 @@ oidc_providers:
# client_id: "synapse" # client_id: "synapse"
# client_secret: "copy secret generated in Keycloak UI" # client_secret: "copy secret generated in Keycloak UI"
# scopes: ["openid", "profile"] # scopes: ["openid", "profile"]
# attribute_requirements:
# - attribute: groups
# value: "admin"
# For use with Github # For use with Github
# #

1
docs/workers.md
View File

@ -232,7 +232,6 @@ expressions:
# Registration/login requests # Registration/login requests
^/_matrix/client/(api/v1|r0|unstable)/login$ ^/_matrix/client/(api/v1|r0|unstable)/login$
^/_matrix/client/(r0|unstable)/register$ ^/_matrix/client/(r0|unstable)/register$
^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$
# Event sending requests # Event sending requests
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact

5
mypy.ini
View File

@ -20,8 +20,9 @@ files =
synapse/crypto, synapse/crypto,
synapse/event_auth.py, synapse/event_auth.py,
synapse/events/builder.py, synapse/events/builder.py,
synapse/events/validator.py,
synapse/events/spamcheck.py, synapse/events/spamcheck.py,
synapse/events/third_party_rules.py,
synapse/events/validator.py,
synapse/federation, synapse/federation,
synapse/groups, synapse/groups,
synapse/handlers, synapse/handlers,
@ -38,6 +39,7 @@ files =
synapse/push, synapse/push,
synapse/replication, synapse/replication,
synapse/rest, synapse/rest,
synapse/secrets.py,
synapse/server.py, synapse/server.py,
synapse/server_notices, synapse/server_notices,
synapse/spam_checker_api, synapse/spam_checker_api,
@ -71,6 +73,7 @@ files =
synapse/util/metrics.py, synapse/util/metrics.py,
synapse/util/macaroons.py, synapse/util/macaroons.py,
synapse/util/stringutils.py, synapse/util/stringutils.py,
synapse/visibility.py,
tests/replication, tests/replication,
tests/test_utils, tests/test_utils,
tests/handlers/test_password_providers.py, tests/handlers/test_password_providers.py,

75
scripts-dev/federation_client.py
View File

@ -22,8 +22,8 @@ import sys
from typing import Any, Optional from typing import Any, Optional
from urllib import parse as urlparse from urllib import parse as urlparse
import nacl.signing
import requests import requests
import signedjson.key
import signedjson.types import signedjson.types
import srvlookup import srvlookup
import yaml import yaml
@ -44,18 +44,6 @@ def encode_base64(input_bytes):
return output_string return output_string
def decode_base64(input_string):
"""Decode a base64 string to bytes inferring padding from the length of the
string."""
input_bytes = input_string.encode("ascii")
input_len = len(input_bytes)
padding = b"=" * (3 - ((input_len + 3) % 4))
output_len = 3 * ((input_len + 2) // 4) + (input_len + 2) % 4 - 2
output_bytes = base64.b64decode(input_bytes + padding)
return output_bytes[:output_len]
def encode_canonical_json(value): def encode_canonical_json(value):
return json.dumps( return json.dumps(
value, value,
@ -88,42 +76,6 @@ def sign_json(
return json_object return json_object
NACL_ED25519 = "ed25519"
def decode_signing_key_base64(algorithm, version, key_base64):
"""Decode a base64 encoded signing key
Args:
algorithm (str): The algorithm the key is for (currently "ed25519").
version (str): Identifies this key out of the keys for this entity.
key_base64 (str): Base64 encoded bytes of the key.
Returns:
A SigningKey object.
"""
if algorithm == NACL_ED25519:
key_bytes = decode_base64(key_base64)
key = nacl.signing.SigningKey(key_bytes)
key.version = version
key.alg = NACL_ED25519
return key
else:
raise ValueError("Unsupported algorithm %s" % (algorithm,))
def read_signing_keys(stream):
"""Reads a list of keys from a stream
Args:
stream : A stream to iterate for keys.
Returns:
list of SigningKey objects.
"""
keys = []
for line in stream:
algorithm, version, key_base64 = line.split()
keys.append(decode_signing_key_base64(algorithm, version, key_base64))
return keys
def request( def request(
method: Optional[str], method: Optional[str],
origin_name: str, origin_name: str,
@ -223,23 +175,28 @@ def main():
parser.add_argument("--body", help="Data to send as the body of the HTTP request") parser.add_argument("--body", help="Data to send as the body of the HTTP request")
parser.add_argument( parser.add_argument(
"path", help="request path. We will add '/_matrix/federation/v1/' to this." "path", help="request path, including the '/_matrix/federation/...' prefix."
) )
args = parser.parse_args() args = parser.parse_args()
if not args.server_name or not args.signing_key_path: args.signing_key = None
if args.signing_key_path:
with open(args.signing_key_path) as f:
args.signing_key = f.readline()
if not args.server_name or not args.signing_key:
read_args_from_config(args) read_args_from_config(args)
with open(args.signing_key_path) as f: algorithm, version, key_base64 = args.signing_key.split()
key = read_signing_keys(f)[0] key = signedjson.key.decode_signing_key_base64(algorithm, version, key_base64)
result = request( result = request(
args.method, args.method,
args.server_name, args.server_name,
key, key,
args.destination, args.destination,
"/_matrix/federation/v1/" + args.path, args.path,
content=args.body, content=args.body,
) )
@ -255,10 +212,16 @@ def main():
def read_args_from_config(args): def read_args_from_config(args):
with open(args.config, "r") as fh: with open(args.config, "r") as fh:
config = yaml.safe_load(fh) config = yaml.safe_load(fh)
if not args.server_name: if not args.server_name:
args.server_name = config["server_name"] args.server_name = config["server_name"]
if not args.signing_key_path:
args.signing_key_path = config["signing_key_path"] if not args.signing_key:
if "signing_key" in config:
args.signing_key = config["signing_key"]
else:
with open(config["signing_key_path"]) as f:
args.signing_key = f.readline()
class MatrixConnectionAdapter(HTTPAdapter): class MatrixConnectionAdapter(HTTPAdapter):

2
scripts/move_remote_media_to_new_store.py
View File

@ -51,7 +51,7 @@ def main(src_repo, dest_repo):
parts = line.split("|") parts = line.split("|")
if len(parts) != 2: if len(parts) != 2:
print("Unable to parse input line %s" % line, file=sys.stderr) print("Unable to parse input line %s" % line, file=sys.stderr)
exit(1) sys.exit(1)
move_media(parts[0], parts[1], src_paths, dest_paths) move_media(parts[0], parts[1], src_paths, dest_paths)

3
setup.cfg
View File

@ -18,7 +18,8 @@ ignore =
# E203: whitespace before ':' (which is contrary to pep8?) # E203: whitespace before ':' (which is contrary to pep8?)
# E731: do not assign a lambda expression, use a def # E731: do not assign a lambda expression, use a def
# E501: Line too long (black enforces this for us) # E501: Line too long (black enforces this for us)
ignore=W503,W504,E203,E731,E501 # B00: Subsection of the bugbear suite (TODO: add in remaining fixes)
ignore=W503,W504,E203,E731,E501,B00
[isort] [isort]
line_length = 88 line_length = 88

3
setup.py
View File

@ -99,10 +99,11 @@ CONDITIONAL_REQUIREMENTS["lint"] = [
"isort==5.7.0", "isort==5.7.0",
"black==20.8b1", "black==20.8b1",
"flake8-comprehensions", "flake8-comprehensions",
"flake8-bugbear",
"flake8", "flake8",
] ]
CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.11"] CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.13"]
# Dependencies which are exclusively required by unit test code. This is # Dependencies which are exclusively required by unit test code. This is
# NOT a list of all modules that are necessary to run the unit tests. # NOT a list of all modules that are necessary to run the unit tests.

2
synapse/__init__.py
View File

@ -48,7 +48,7 @@ try:
except ImportError: except ImportError:
pass pass
__version__ = "1.29.0" __version__ = "1.30.0"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when # We import here so that we don't have to install a bunch of deps when

7
synapse/api/constants.py
View File

@ -51,6 +51,7 @@ class PresenceState:
OFFLINE = "offline" OFFLINE = "offline"
UNAVAILABLE = "unavailable" UNAVAILABLE = "unavailable"
ONLINE = "online" ONLINE = "online"
BUSY = "org.matrix.msc3026.busy"
class JoinRules: class JoinRules:
@ -100,6 +101,9 @@ class EventTypes:
Dummy = "org.matrix.dummy_event" Dummy = "org.matrix.dummy_event"
MSC1772_SPACE_CHILD = "org.matrix.msc1772.space.child"
MSC1772_SPACE_PARENT = "org.matrix.msc1772.space.parent"
class EduTypes: class EduTypes:
Presence = "m.presence" Presence = "m.presence"
@ -160,6 +164,9 @@ class EventContentFields:
# cf https://github.com/matrix-org/matrix-doc/pull/2228 # cf https://github.com/matrix-org/matrix-doc/pull/2228
SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after" SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after"
# cf https://github.com/matrix-org/matrix-doc/pull/1772
MSC1772_ROOM_TYPE = "org.matrix.msc1772.type"
class RoomEncryptionAlgorithms: class RoomEncryptionAlgorithms:
MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2" MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2"

4
synapse/app/__init__.py
View File

@ -22,7 +22,9 @@ logger = logging.getLogger(__name__)
try: try:
python_dependencies.check_requirements() python_dependencies.check_requirements()
except python_dependencies.DependencyException as e: except python_dependencies.DependencyException as e:
sys.stderr.writelines(e.message) sys.stderr.writelines(
e.message # noqa: B306, DependencyException.message is a property
)
sys.exit(1) sys.exit(1)

8
synapse/app/generic_worker.py
View File

@ -302,6 +302,8 @@ class GenericWorkerPresence(BasePresenceHandler):
self.send_stop_syncing, UPDATE_SYNCING_USERS_MS self.send_stop_syncing, UPDATE_SYNCING_USERS_MS
) )
self._busy_presence_enabled = hs.config.experimental.msc3026_enabled
hs.get_reactor().addSystemEventTrigger( hs.get_reactor().addSystemEventTrigger(
"before", "before",
"shutdown", "shutdown",
@ -439,8 +441,12 @@ class GenericWorkerPresence(BasePresenceHandler):
PresenceState.ONLINE, PresenceState.ONLINE,
PresenceState.UNAVAILABLE, PresenceState.UNAVAILABLE,
PresenceState.OFFLINE, PresenceState.OFFLINE,
PresenceState.BUSY,
) )
if presence not in valid_presence:
if presence not in valid_presence or (
presence == PresenceState.BUSY and not self._busy_presence_enabled
):
raise SynapseError(400, "Invalid presence state") raise SynapseError(400, "Invalid presence state")
user_id = target_user.to_string() user_id = target_user.to_string()

4
synapse/config/experimental.py
View File

@ -27,3 +27,7 @@ class ExperimentalConfig(Config):
# MSC2858 (multiple SSO identity providers) # MSC2858 (multiple SSO identity providers)
self.msc2858_enabled = experimental.get("msc2858_enabled", False) # type: bool self.msc2858_enabled = experimental.get("msc2858_enabled", False) # type: bool
# Spaces (MSC1772, MSC2946, etc)
self.spaces_enabled = experimental.get("spaces_enabled", False) # type: bool
# MSC3026 (busy presence state)
self.msc3026_enabled = experimental.get("msc3026_enabled", False) # type: bool

6
synapse/config/key.py
View File

@ -404,7 +404,11 @@ def _parse_key_servers(key_servers, federation_verify_certificates):
try: try:
jsonschema.validate(key_servers, TRUSTED_KEY_SERVERS_SCHEMA) jsonschema.validate(key_servers, TRUSTED_KEY_SERVERS_SCHEMA)
except jsonschema.ValidationError as e: except jsonschema.ValidationError as e:
raise ConfigError("Unable to parse 'trusted_key_servers': " + e.message) raise ConfigError(
"Unable to parse 'trusted_key_servers': {}".format(
e.message # noqa: B306, jsonschema.ValidationError.message is a valid attribute
)
)
for server in key_servers: for server in key_servers:
server_name = server["server_name"] server_name = server["server_name"]

4
synapse/config/metrics.py
View File

@ -56,7 +56,9 @@ class MetricsConfig(Config):
try: try:
check_requirements("sentry") check_requirements("sentry")
except DependencyException as e: except DependencyException as e:
raise ConfigError(e.message) raise ConfigError(
e.message # noqa: B306, DependencyException.message is a property
)
self.sentry_dsn = config["sentry"].get("dsn") self.sentry_dsn = config["sentry"].get("dsn")
if not self.sentry_dsn: if not self.sentry_dsn:

44
synapse/config/oidc_config.py
View File

@ -15,11 +15,12 @@
# limitations under the License. # limitations under the License.
from collections import Counter from collections import Counter
from typing import Iterable, Mapping, Optional, Tuple, Type from typing import Iterable, List, Mapping, Optional, Tuple, Type
import attr import attr
from synapse.config._util import validate_config from synapse.config._util import validate_config
from synapse.config.sso import SsoAttributeRequirement
from synapse.python_dependencies import DependencyException, check_requirements from synapse.python_dependencies import DependencyException, check_requirements
from synapse.types import Collection, JsonDict from synapse.types import Collection, JsonDict
from synapse.util.module_loader import load_module from synapse.util.module_loader import load_module
@ -41,7 +42,9 @@ class OIDCConfig(Config):
try: try:
check_requirements("oidc") check_requirements("oidc")
except DependencyException as e: except DependencyException as e:
raise ConfigError(e.message) from e raise ConfigError(
e.message # noqa: B306, DependencyException.message is a property
) from e
# check we don't have any duplicate idp_ids now. (The SSO handler will also # check we don't have any duplicate idp_ids now. (The SSO handler will also
# check for duplicates when the REST listeners get registered, but that happens # check for duplicates when the REST listeners get registered, but that happens
@ -191,6 +194,24 @@ class OIDCConfig(Config):
# which is set to the claims returned by the UserInfo Endpoint and/or # which is set to the claims returned by the UserInfo Endpoint and/or
# in the ID Token. # in the ID Token.
# #
# It is possible to configure Synapse to only allow logins if certain attributes
# match particular values in the OIDC userinfo. The requirements can be listed under
# `attribute_requirements` as shown below. All of the listed attributes must
# match for the login to be permitted. Additional attributes can be added to
# userinfo by expanding the `scopes` section of the OIDC config to retrieve
# additional information from the OIDC provider.
#
# If the OIDC claim is a list, then the attribute must match any value in the list.
# Otherwise, it must exactly match the value of the claim. Using the example
# below, the `family_name` claim MUST be "Stephensson", but the `groups`
# claim MUST contain "admin".
#
# attribute_requirements:
# - attribute: family_name
# value: "Stephensson"
# - attribute: groups
# value: "admin"
#
# See https://github.com/matrix-org/synapse/blob/master/docs/openid.md # See https://github.com/matrix-org/synapse/blob/master/docs/openid.md
# for information on how to configure these options. # for information on how to configure these options.
# #
@ -223,6 +244,9 @@ class OIDCConfig(Config):
# localpart_template: "{{{{ user.login }}}}" # localpart_template: "{{{{ user.login }}}}"
# display_name_template: "{{{{ user.name }}}}" # display_name_template: "{{{{ user.name }}}}"
# email_template: "{{{{ user.email }}}}" # email_template: "{{{{ user.email }}}}"
# attribute_requirements:
# - attribute: userGroup
# value: "synapseUsers"
# For use with Keycloak # For use with Keycloak
# #
@ -232,6 +256,9 @@ class OIDCConfig(Config):
# client_id: "synapse" # client_id: "synapse"
# client_secret: "copy secret generated in Keycloak UI" # client_secret: "copy secret generated in Keycloak UI"
# scopes: ["openid", "profile"] # scopes: ["openid", "profile"]
# attribute_requirements:
# - attribute: groups
# value: "admin"
# For use with Github # For use with Github
# #
@ -329,6 +356,10 @@ OIDC_PROVIDER_CONFIG_SCHEMA = {
}, },
"allow_existing_users": {"type": "boolean"}, "allow_existing_users": {"type": "boolean"},
"user_mapping_provider": {"type": ["object", "null"]}, "user_mapping_provider": {"type": ["object", "null"]},
"attribute_requirements": {
"type": "array",
"items": SsoAttributeRequirement.JSON_SCHEMA,
},
}, },
} }
@ -465,6 +496,11 @@ def _parse_oidc_config_dict(
jwt_header=client_secret_jwt_key_config["jwt_header"], jwt_header=client_secret_jwt_key_config["jwt_header"],
jwt_payload=client_secret_jwt_key_config.get("jwt_payload", {}), jwt_payload=client_secret_jwt_key_config.get("jwt_payload", {}),
) )
# parse attribute_requirements from config (list of dicts) into a list of SsoAttributeRequirement
attribute_requirements = [
SsoAttributeRequirement(**x)
for x in oidc_config.get("attribute_requirements", [])
]
return OidcProviderConfig( return OidcProviderConfig(
idp_id=idp_id, idp_id=idp_id,
@ -488,6 +524,7 @@ def _parse_oidc_config_dict(
allow_existing_users=oidc_config.get("allow_existing_users", False), allow_existing_users=oidc_config.get("allow_existing_users", False),
user_mapping_provider_class=user_mapping_provider_class, user_mapping_provider_class=user_mapping_provider_class,
user_mapping_provider_config=user_mapping_provider_config, user_mapping_provider_config=user_mapping_provider_config,
attribute_requirements=attribute_requirements,
) )
@ -577,3 +614,6 @@ class OidcProviderConfig:
# the config of the user mapping provider # the config of the user mapping provider
user_mapping_provider_config = attr.ib() user_mapping_provider_config = attr.ib()
# required attributes to require in userinfo to allow login/registration
attribute_requirements = attr.ib(type=List[SsoAttributeRequirement])

8
synapse/config/ratelimiting.py
View File

@ -95,11 +95,11 @@ class RatelimitConfig(Config):
self.rc_joins_local = RateLimitConfig( self.rc_joins_local = RateLimitConfig(
config.get("rc_joins", {}).get("local", {}), config.get("rc_joins", {}).get("local", {}),
defaults={"per_second": 0.1, "burst_count": 3}, defaults={"per_second": 0.1, "burst_count": 10},
) )
self.rc_joins_remote = RateLimitConfig( self.rc_joins_remote = RateLimitConfig(
config.get("rc_joins", {}).get("remote", {}), config.get("rc_joins", {}).get("remote", {}),
defaults={"per_second": 0.01, "burst_count": 3}, defaults={"per_second": 0.01, "burst_count": 10},
) )
# Ratelimit cross-user key requests: # Ratelimit cross-user key requests:
@ -187,10 +187,10 @@ class RatelimitConfig(Config):
#rc_joins: #rc_joins:
# local: # local:
# per_second: 0.1 # per_second: 0.1
# burst_count: 3 # burst_count: 10
# remote: # remote:
# per_second: 0.01 # per_second: 0.01
# burst_count: 3 # burst_count: 10
# #
#rc_3pid_validation: #rc_3pid_validation:
# per_second: 0.003 # per_second: 0.003

4
synapse/config/repository.py
View File

@ -176,7 +176,9 @@ class ContentRepositoryConfig(Config):
check_requirements("url_preview") check_requirements("url_preview")
except DependencyException as e: except DependencyException as e:
raise ConfigError(e.message) raise ConfigError(
e.message # noqa: B306, DependencyException.message is a property
)
if "url_preview_ip_range_blacklist" not in config: if "url_preview_ip_range_blacklist" not in config:
raise ConfigError( raise ConfigError(

4
synapse/config/saml2_config.py
View File

@ -76,7 +76,9 @@ class SAML2Config(Config):
try: try:
check_requirements("saml2") check_requirements("saml2")
except DependencyException as e: except DependencyException as e:
raise ConfigError(e.message) raise ConfigError(
e.message # noqa: B306, DependencyException.message is a property
)
self.saml2_enabled = True self.saml2_enabled = True

4
synapse/config/tracer.py
View File

@ -39,7 +39,9 @@ class TracerConfig(Config):
try: try:
check_requirements("opentracing") check_requirements("opentracing")
except DependencyException as e: except DependencyException as e:
raise ConfigError(e.message) raise ConfigError(
e.message # noqa: B306, DependencyException.message is a property
)
# The tracer is enabled so sanitize the config # The tracer is enabled so sanitize the config

2
synapse/crypto/context_factory.py
View File

@ -219,7 +219,7 @@ class SSLClientConnectionCreator:
# ... and we also gut-wrench a '_synapse_tls_verifier' attribute into the # ... and we also gut-wrench a '_synapse_tls_verifier' attribute into the
# tls_protocol so that the SSL context's info callback has something to # tls_protocol so that the SSL context's info callback has something to
# call to do the cert verification. # call to do the cert verification.
setattr(tls_protocol, "_synapse_tls_verifier", self._verifier) tls_protocol._synapse_tls_verifier = self._verifier
return connection return connection

2
synapse/crypto/keyring.py
View File

@ -57,7 +57,7 @@ from synapse.util.metrics import Measure
from synapse.util.retryutils import NotRetryingDestination from synapse.util.retryutils import NotRetryingDestination
if TYPE_CHECKING: if TYPE_CHECKING:
from synapse.app.homeserver import HomeServer from synapse.server import HomeServer
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)

9
synapse/events/__init__.py
View File

@ -98,7 +98,7 @@ class DefaultDictProperty(DictProperty):
class _EventInternalMetadata: class _EventInternalMetadata:
__slots__ = ["_dict", "stream_ordering"] __slots__ = ["_dict", "stream_ordering", "outlier"]
def __init__(self, internal_metadata_dict: JsonDict): def __init__(self, internal_metadata_dict: JsonDict):
# we have to copy the dict, because it turns out that the same dict is # we have to copy the dict, because it turns out that the same dict is
@ -108,7 +108,10 @@ class _EventInternalMetadata:
# the stream ordering of this event. None, until it has been persisted. # the stream ordering of this event. None, until it has been persisted.
self.stream_ordering = None # type: Optional[int] self.stream_ordering = None # type: Optional[int]
outlier = DictProperty("outlier") # type: bool # whether this event is an outlier (ie, whether we have the state at that point
# in the DAG)
self.outlier = False
out_of_band_membership = DictProperty("out_of_band_membership") # type: bool out_of_band_membership = DictProperty("out_of_band_membership") # type: bool
send_on_behalf_of = DictProperty("send_on_behalf_of") # type: str send_on_behalf_of = DictProperty("send_on_behalf_of") # type: str
recheck_redaction = DictProperty("recheck_redaction") # type: bool recheck_redaction = DictProperty("recheck_redaction") # type: bool
@ -129,7 +132,7 @@ class _EventInternalMetadata:
return dict(self._dict) return dict(self._dict)
def is_outlier(self) -> bool: def is_outlier(self) -> bool:
return self._dict.get("outlier", False) return self.outlier
def is_out_of_band_membership(self) -> bool: def is_out_of_band_membership(self) -> bool:
"""Whether this is an out of band membership, like an invite or an invite """Whether this is an out of band membership, like an invite or an invite

15
synapse/events/third_party_rules.py
View File

@ -13,12 +13,15 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from typing import Callable, Union from typing import TYPE_CHECKING, Union
from synapse.events import EventBase from synapse.events import EventBase
from synapse.events.snapshot import EventContext from synapse.events.snapshot import EventContext
from synapse.types import Requester, StateMap from synapse.types import Requester, StateMap
if TYPE_CHECKING:
from synapse.server import HomeServer
class ThirdPartyEventRules: class ThirdPartyEventRules:
"""Allows server admins to provide a Python module implementing an extra """Allows server admins to provide a Python module implementing an extra
@ -28,7 +31,7 @@ class ThirdPartyEventRules:
behaviours. behaviours.
""" """
def __init__(self, hs): def __init__(self, hs: "HomeServer"):
self.third_party_rules = None self.third_party_rules = None
self.store = hs.get_datastore() self.store = hs.get_datastore()
@ -95,10 +98,9 @@ class ThirdPartyEventRules:
if self.third_party_rules is None: if self.third_party_rules is None:
return True return True
ret = await self.third_party_rules.on_create_room( return await self.third_party_rules.on_create_room(
requester, config, is_requester_admin requester, config, is_requester_admin
) )
return ret
async def check_threepid_can_be_invited( async def check_threepid_can_be_invited(
self, medium: str, address: str, room_id: str self, medium: str, address: str, room_id: str
@ -119,10 +121,9 @@ class ThirdPartyEventRules:
state_events = await self._get_state_map_for_room(room_id) state_events = await self._get_state_map_for_room(room_id)
ret = await self.third_party_rules.check_threepid_can_be_invited( return await self.third_party_rules.check_threepid_can_be_invited(
medium, address, state_events medium, address, state_events
) )
return ret
async def check_visibility_can_be_modified( async def check_visibility_can_be_modified(
self, room_id: str, new_visibility: str self, room_id: str, new_visibility: str
@ -143,7 +144,7 @@ class ThirdPartyEventRules:
check_func = getattr( check_func = getattr(
self.third_party_rules, "check_visibility_can_be_modified", None self.third_party_rules, "check_visibility_can_be_modified", None
) )
if not check_func or not isinstance(check_func, Callable): if not check_func or not callable(check_func):
return True return True
state_events = await self._get_state_map_for_room(room_id) state_events = await self._get_state_map_for_room(room_id)

16
synapse/events/utils.py
View File

@ -22,6 +22,7 @@ from synapse.api.constants import EventTypes, RelationTypes
from synapse.api.errors import Codes, SynapseError from synapse.api.errors import Codes, SynapseError
from synapse.api.room_versions import RoomVersion from synapse.api.room_versions import RoomVersion
from synapse.util.async_helpers import yieldable_gather_results from synapse.util.async_helpers import yieldable_gather_results
from synapse.util.frozenutils import unfreeze
from . import EventBase from . import EventBase
@ -54,6 +55,8 @@ def prune_event(event: EventBase) -> EventBase:
event.internal_metadata.stream_ordering event.internal_metadata.stream_ordering
) )
pruned_event.internal_metadata.outlier = event.internal_metadata.outlier
# Mark the event as redacted # Mark the event as redacted
pruned_event.internal_metadata.redacted = True pruned_event.internal_metadata.redacted = True
@ -400,10 +403,19 @@ class EventClientSerializer:
# If there is an edit replace the content, preserving existing # If there is an edit replace the content, preserving existing
# relations. # relations.
# Ensure we take copies of the edit content, otherwise we risk modifying
# the original event.
edit_content = edit.content.copy()
# Unfreeze the event content if necessary, so that we may modify it below
edit_content = unfreeze(edit_content)
serialized_event["content"] = edit_content.get("m.new_content", {})
# Check for existing relations
relations = event.content.get("m.relates_to") relations = event.content.get("m.relates_to")
serialized_event["content"] = edit.content.get("m.new_content", {})
if relations: if relations:
serialized_event["content"]["m.relates_to"] = relations # Keep the relations, ensuring we use a dict copy of the original
serialized_event["content"]["m.relates_to"] = relations.copy()
else: else:
serialized_event["content"].pop("m.relates_to", None) serialized_event["content"].pop("m.relates_to", None)

168
synapse/federation/federation_client.py
View File

@ -27,11 +27,13 @@ from typing import (
List, List,
Mapping, Mapping,
Optional, Optional,
Sequence,
Tuple, Tuple,
TypeVar, TypeVar,
Union, Union,
) )
import attr
from prometheus_client import Counter from prometheus_client import Counter
from twisted.internet import defer from twisted.internet import defer
@ -62,7 +64,7 @@ from synapse.util.caches.expiringcache import ExpiringCache
from synapse.util.retryutils import NotRetryingDestination from synapse.util.retryutils import NotRetryingDestination
if TYPE_CHECKING: if TYPE_CHECKING:
from synapse.app.homeserver import HomeServer from synapse.server import HomeServer
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -455,6 +457,7 @@ class FederationClient(FederationBase):
description: str, description: str,
destinations: Iterable[str], destinations: Iterable[str],
callback: Callable[[str], Awaitable[T]], callback: Callable[[str], Awaitable[T]],
failover_on_unknown_endpoint: bool = False,
) -> T: ) -> T:
"""Try an operation on a series of servers, until it succeeds """Try an operation on a series of servers, until it succeeds
@ -474,6 +477,10 @@ class FederationClient(FederationBase):
next server tried. Normally the stacktrace is logged but this is next server tried. Normally the stacktrace is logged but this is
suppressed if the exception is an InvalidResponseError. suppressed if the exception is an InvalidResponseError.
failover_on_unknown_endpoint: if True, we will try other servers if it looks
like a server doesn't support the endpoint. This is typically useful
if the endpoint in question is new or experimental.
Returns: Returns:
The result of callback, if it succeeds The result of callback, if it succeeds
@ -493,9 +500,24 @@ class FederationClient(FederationBase):
except UnsupportedRoomVersionError: except UnsupportedRoomVersionError:
raise raise
except HttpResponseException as e: except HttpResponseException as e:
if not 500 <= e.code < 600: synapse_error = e.to_synapse_error()
raise e.to_synapse_error() failover = False
else:
if 500 <= e.code < 600:
failover = True
elif failover_on_unknown_endpoint:
# there is no good way to detect an "unknown" endpoint. Dendrite
# returns a 404 (with no body); synapse returns a 400
# with M_UNRECOGNISED.
if e.code == 404 or (
e.code == 400 and synapse_error.errcode == Codes.UNRECOGNIZED
):
failover = True
if not failover:
raise synapse_error from e
logger.warning( logger.warning(
"Failed to %s via %s: %i %s", "Failed to %s via %s: %i %s",
description, description,
@ -1042,3 +1064,141 @@ class FederationClient(FederationBase):
# If we don't manage to find it, return None. It's not an error if a # If we don't manage to find it, return None. It's not an error if a
# server doesn't give it to us. # server doesn't give it to us.
return None return None
async def get_space_summary(
self,
destinations: Iterable[str],
room_id: str,
suggested_only: bool,
max_rooms_per_space: Optional[int],
exclude_rooms: List[str],
) -> "FederationSpaceSummaryResult":
"""
Call other servers to get a summary of the given space
Args:
destinations: The remote servers. We will try them in turn, omitting any
that have been blacklisted.
room_id: ID of the space to be queried
suggested_only: If true, ask the remote server to only return children
with the "suggested" flag set
max_rooms_per_space: A limit on the number of children to return for each
space
exclude_rooms: A list of room IDs to tell the remote server to skip
Returns:
a parsed FederationSpaceSummaryResult
Raises:
SynapseError if we were unable to get a valid summary from any of the
remote servers
"""
async def send_request(destination: str) -> FederationSpaceSummaryResult:
res = await self.transport_layer.get_space_summary(
destination=destination,
room_id=room_id,
suggested_only=suggested_only,
max_rooms_per_space=max_rooms_per_space,
exclude_rooms=exclude_rooms,
)
try:
return FederationSpaceSummaryResult.from_json_dict(res)
except ValueError as e:
raise InvalidResponseError(str(e))
return await self._try_destination_list(
"fetch space summary",
destinations,
send_request,
failover_on_unknown_endpoint=True,
)
@attr.s(frozen=True, slots=True)
class FederationSpaceSummaryEventResult:
"""Represents a single event in the result of a successful get_space_summary call.
It's essentially just a serialised event object, but we do a bit of parsing and
validation in `from_json_dict` and store some of the validated properties in
object attributes.
"""
event_type = attr.ib(type=str)
state_key = attr.ib(type=str)
via = attr.ib(type=Sequence[str])
# the raw data, including the above keys
data = attr.ib(type=JsonDict)
@classmethod
def from_json_dict(cls, d: JsonDict) -> "FederationSpaceSummaryEventResult":
"""Parse an event within the result of a /spaces/ request
Args:
d: json object to be parsed
Raises:
ValueError if d is not a valid event
"""
event_type = d.get("type")
if not isinstance(event_type, str):
raise ValueError("Invalid event: 'event_type' must be a str")
state_key = d.get("state_key")
if not isinstance(state_key, str):
raise ValueError("Invalid event: 'state_key' must be a str")
content = d.get("content")
if not isinstance(content, dict):
raise ValueError("Invalid event: 'content' must be a dict")
via = content.get("via")
if not isinstance(via, Sequence):
raise ValueError("Invalid event: 'via' must be a list")
if any(not isinstance(v, str) for v in via):
raise ValueError("Invalid event: 'via' must be a list of strings")
return cls(event_type, state_key, via, d)
@attr.s(frozen=True, slots=True)
class FederationSpaceSummaryResult:
"""Represents the data returned by a successful get_space_summary call."""
rooms = attr.ib(type=Sequence[JsonDict])
events = attr.ib(type=Sequence[FederationSpaceSummaryEventResult])
@classmethod
def from_json_dict(cls, d: JsonDict) -> "FederationSpaceSummaryResult":
"""Parse the result of a /spaces/ request
Args:
d: json object to be parsed
Raises:
ValueError if d is not a valid /spaces/ response
"""
rooms = d.get("rooms")
if not isinstance(rooms, Sequence):
raise ValueError("'rooms' must be a list")
if any(not isinstance(r, dict) for r in rooms):
raise ValueError("Invalid room in 'rooms' list")
events = d.get("events")
if not isinstance(events, Sequence):
raise ValueError("'events' must be a list")
if any(not isinstance(e, dict) for e in events):
raise ValueError("Invalid event in 'events' list")
parsed_events = [
FederationSpaceSummaryEventResult.from_json_dict(e) for e in events
]
return cls(rooms, parsed_events)

25
synapse/federation/federation_server.py
View File

@ -35,7 +35,7 @@ from twisted.internet import defer
from twisted.internet.abstract import isIPAddress from twisted.internet.abstract import isIPAddress
from twisted.python import failure from twisted.python import failure
from synapse.api.constants import EduTypes, EventTypes, Membership from synapse.api.constants import EduTypes, EventTypes
from synapse.api.errors import ( from synapse.api.errors import (
AuthError, AuthError,
Codes, Codes,
@ -63,7 +63,7 @@ from synapse.replication.http.federation import (
ReplicationFederationSendEduRestServlet, ReplicationFederationSendEduRestServlet,
ReplicationGetQueryRestServlet, ReplicationGetQueryRestServlet,
) )
from synapse.types import JsonDict, get_domain_from_id from synapse.types import JsonDict
from synapse.util import glob_to_regex, json_decoder, unwrapFirstError from synapse.util import glob_to_regex, json_decoder, unwrapFirstError
from synapse.util.async_helpers import Linearizer, concurrently_execute from synapse.util.async_helpers import Linearizer, concurrently_execute
from synapse.util.caches.response_cache import ResponseCache from synapse.util.caches.response_cache import ResponseCache
@ -727,27 +727,6 @@ class FederationServer(FederationBase):
if the event was unacceptable for any other reason (eg, too large, if the event was unacceptable for any other reason (eg, too large,
too many prev_events, couldn't find the prev_events) too many prev_events, couldn't find the prev_events)
""" """
# check that it's actually being sent from a valid destination to
# workaround bug #1753 in 0.18.5 and 0.18.6
if origin != get_domain_from_id(pdu.sender):
# We continue to accept join events from any server; this is
# necessary for the federation join dance to work correctly.
# (When we join over federation, the "helper" server is
# responsible for sending out the join event, rather than the
# origin. See bug #1893. This is also true for some third party
# invites).
if not (
pdu.type == "m.room.member"
and pdu.content
and pdu.content.get("membership", None)
in (Membership.JOIN, Membership.INVITE)
):
logger.info(
"Discarding PDU %s from invalid origin %s", pdu.event_id, origin
)
return
else:
logger.info("Accepting join PDU %s from %s", pdu.event_id, origin)
# We've already checked that we know the room version by this point # We've already checked that we know the room version by this point
room_version = await self.store.get_room_version(pdu.room_id) room_version = await self.store.get_room_version(pdu.room_id)

Some files were not shown because too many files have changed in this diff Show More