From 013f3f5e44278c7894e48ccd179ad4bca63bd857 Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Tue, 29 Mar 2022 14:12:41 +0100 Subject: [PATCH 1/6] Move warning of open registration to v1.56 upgrade notes (#12322) --- CHANGES.md | 13 ++++++++++++- changelog.d/12322.misc | 1 + docs/upgrade.md | 11 +++++------ 3 files changed, 18 insertions(+), 7 deletions(-) create mode 100644 changelog.d/12322.misc diff --git a/CHANGES.md b/CHANGES.md index ee1eb9a582..a9ac4a5d3d 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,6 +1,17 @@ Synapse 1.56.0rc1 (2022-03-29) ============================== +Synapse will now refuse to start up if open registration is enabled, in order to help mitigate +abuse across the federation. If you would like +to provide registration to anyone, consider adding [email](https://github.com/matrix-org/synapse/blob/8a519f8abc6de772167c2cca101d22ee2052fafc/docs/sample_config.yaml#L1285), +[recaptcha](https://matrix-org.github.io/synapse/v1.56/CAPTCHA_SETUP.html) +or [token-based](https://matrix-org.github.io/synapse/v1.56/usage/administration/admin_api/registration_tokens.html) verification +in order to prevent automated registration from bad actors. + +This check can be disabled by setting the `enable_registration_without_verification` option in your +homeserver configuration file to `true`. More details are available in the +[upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade.html#open-registration-without-verification-is-now-disabled-by-default). + Features -------- @@ -33,7 +44,7 @@ Internal Changes - Rename `shared_rooms` to `mutual_rooms` ([MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)), as per proposal changes. ([\#12036](https://github.com/matrix-org/synapse/issues/12036)) - Remove check on `update_user_directory` for shared rooms handler ([MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)), and update/expand documentation. ([\#12038](https://github.com/matrix-org/synapse/issues/12038)) - Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) instead of abusing `auth_event_ids`. ([\#12083](https://github.com/matrix-org/synapse/issues/12083), [\#12304](https://github.com/matrix-org/synapse/issues/12304)) -- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set. ([\#12091](https://github.com/matrix-org/synapse/issues/12091)) +- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set to `true`. ([\#12091](https://github.com/matrix-org/synapse/issues/12091)) - Add tests for database transaction callbacks. ([\#12198](https://github.com/matrix-org/synapse/issues/12198)) - Handle cancellation in `DatabasePool.runInteraction`. ([\#12199](https://github.com/matrix-org/synapse/issues/12199)) - Add missing type hints for cache storage. ([\#12216](https://github.com/matrix-org/synapse/issues/12216)) diff --git a/changelog.d/12322.misc b/changelog.d/12322.misc new file mode 100644 index 0000000000..fd595df81c --- /dev/null +++ b/changelog.d/12322.misc @@ -0,0 +1 @@ +Refuse to start if registration is enabled without email, captcha, or token-based verification unless new config flag `enable_registration_without_verification` is set to `true`. \ No newline at end of file diff --git a/docs/upgrade.md b/docs/upgrade.md index 062e823333..10d81b949c 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -87,6 +87,11 @@ process, for example: # Upgrading to v1.56.0 +## Open registration without verification is now disabled by default + +Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config +flag `enable_registration_without_verification` is set to "true". + ## Groups/communities feature has been deprecated The non-standard groups/communities feature in Synapse has been deprecated and will @@ -108,12 +113,6 @@ for more information and instructions on how to fix a database with incorrect va # Upgrading to v1.55.0 -## Open registration without verification is now disabled by default - -Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config -flag `enable_registration_without_verification` is set to "true". - - ## `synctl` script has been moved The `synctl` script From 4d693f9b7964b8be055e83f2154d357ecac6b5f3 Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Tue, 29 Mar 2022 14:32:50 +0100 Subject: [PATCH 2/6] Add note in changelog of non-C type locale causing Synapse to refuse to start (#12323) --- CHANGES.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index a9ac4a5d3d..de40921bff 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -12,6 +12,11 @@ This check can be disabled by setting the `enable_registration_without_verificat homeserver configuration file to `true`. More details are available in the [upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade.html#open-registration-without-verification-is-now-disabled-by-default). +Synapse will additionally now refuse to start when using PostgreSQL with a non-`C` values for `COLLATE` and `CTYPE`, unless +the config flag `allow_unsafe_locale`, found in the database section of the configuration file, is set to `true`. See the +[upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade#change-in-behaviour-for-postgresql-databases-with-unsafe-locale) +for details. + Features -------- From a7293ef16fb3ca9f1234877f1496b30cdeeb01cd Mon Sep 17 00:00:00 2001 From: David Robertson Date: Tue, 29 Mar 2022 11:41:19 +0100 Subject: [PATCH 3/6] Bump `black` and `click` versions (#12320) --- changelog.d/12320.misc | 1 + contrib/jitsimeetbridge/jitsimeetbridge.py | 15 +++++++++------ setup.py | 4 ++-- synapse/api/constants.py | 2 +- synapse/appservice/scheduler.py | 2 +- synapse/crypto/keyring.py | 2 +- synapse/events/utils.py | 2 +- synapse/http/proxyagent.py | 2 +- synapse/replication/http/_base.py | 2 +- synapse/rest/media/v1/media_storage.py | 2 +- synapse/rest/media/v1/preview_html.py | 4 ++-- synapse/storage/database.py | 2 +- synapse/util/patch_inline_callbacks.py | 17 ++++++++++------- synapse/util/retryutils.py | 2 +- tests/handlers/test_federation.py | 6 +++--- tests/replication/_base.py | 2 +- 16 files changed, 37 insertions(+), 30 deletions(-) create mode 100644 changelog.d/12320.misc diff --git a/changelog.d/12320.misc b/changelog.d/12320.misc new file mode 100644 index 0000000000..7b4748d230 --- /dev/null +++ b/changelog.d/12320.misc @@ -0,0 +1 @@ +Bump the version of `black` for compatibility with the latest `click` release. diff --git a/contrib/jitsimeetbridge/jitsimeetbridge.py b/contrib/jitsimeetbridge/jitsimeetbridge.py index 495fd4e10a..b3de468687 100644 --- a/contrib/jitsimeetbridge/jitsimeetbridge.py +++ b/contrib/jitsimeetbridge/jitsimeetbridge.py @@ -193,12 +193,15 @@ class TrivialXmppClient: time.sleep(7) print("SSRC spammer started") while self.running: - ssrcMsg = "%(nick)s" % { - "tojid": "%s@%s/%s" % (ROOMNAME, ROOMDOMAIN, self.shortJid), - "nick": self.userId, - "assrc": self.ssrcs["audio"], - "vssrc": self.ssrcs["video"], - } + ssrcMsg = ( + "%(nick)s" + % { + "tojid": "%s@%s/%s" % (ROOMNAME, ROOMDOMAIN, self.shortJid), + "nick": self.userId, + "assrc": self.ssrcs["audio"], + "vssrc": self.ssrcs["video"], + } + ) res = self.sendIq(ssrcMsg) print("reply from ssrc announce: ", res) time.sleep(10) diff --git a/setup.py b/setup.py index 63da71ad7b..48bd418bb8 100755 --- a/setup.py +++ b/setup.py @@ -95,7 +95,7 @@ CONDITIONAL_REQUIREMENTS["all"] = list(ALL_OPTIONAL_REQUIREMENTS) # We pin black so that our tests don't start failing on new releases. CONDITIONAL_REQUIREMENTS["lint"] = [ "isort==5.7.0", - "black==21.12b0", + "black==22.3.0", "flake8-comprehensions", "flake8-bugbear==21.3.2", "flake8", @@ -128,7 +128,7 @@ CONDITIONAL_REQUIREMENTS["dev"] = ( + CONDITIONAL_REQUIREMENTS["test"] + [ # The following are used by the release script - "click==7.1.2", + "click==8.1.0", "redbaron==0.9.2", "GitPython==3.1.14", "commonmark==0.9.1", diff --git a/synapse/api/constants.py b/synapse/api/constants.py index b0c08a074d..92907415e6 100644 --- a/synapse/api/constants.py +++ b/synapse/api/constants.py @@ -23,7 +23,7 @@ from typing_extensions import Final MAX_PDU_SIZE = 65536 # the "depth" field on events is limited to 2**63 - 1 -MAX_DEPTH = 2 ** 63 - 1 +MAX_DEPTH = 2**63 - 1 # the maximum length for a room alias is 255 characters MAX_ALIAS_LENGTH = 255 diff --git a/synapse/appservice/scheduler.py b/synapse/appservice/scheduler.py index 72417151ba..a6084b9c35 100644 --- a/synapse/appservice/scheduler.py +++ b/synapse/appservice/scheduler.py @@ -428,7 +428,7 @@ class _Recoverer: "as-recoverer-%s" % (self.service.id,), self.retry ) - delay = 2 ** self.backoff_counter + delay = 2**self.backoff_counter logger.info("Scheduling retries on %s in %fs", self.service.id, delay) self.clock.call_later(delay, _retry) diff --git a/synapse/crypto/keyring.py b/synapse/crypto/keyring.py index 93d56c077a..6cf384f6a1 100644 --- a/synapse/crypto/keyring.py +++ b/synapse/crypto/keyring.py @@ -182,7 +182,7 @@ class Keyring: vk = get_verify_key(hs.signing_key) self._local_verify_keys[f"{vk.alg}:{vk.version}"] = FetchKeyResult( verify_key=vk, - valid_until_ts=2 ** 63, # fake future timestamp + valid_until_ts=2**63, # fake future timestamp ) async def verify_json_for_server( diff --git a/synapse/events/utils.py b/synapse/events/utils.py index 7120062127..918e87ed9c 100644 --- a/synapse/events/utils.py +++ b/synapse/events/utils.py @@ -49,7 +49,7 @@ if TYPE_CHECKING: # the literal fields "foo\" and "bar" but will instead be treated as "foo\\.bar" SPLIT_FIELD_REGEX = re.compile(r"(? cls.RETRY_ON_CONNECT_ERROR_ATTEMPTS: raise - delay = 2 ** attempts + delay = 2**attempts logger.warning( "%s request connection failed; retrying in %ds: %r", cls.NAME, diff --git a/synapse/rest/media/v1/media_storage.py b/synapse/rest/media/v1/media_storage.py index 9f6c251caf..604f18bf52 100644 --- a/synapse/rest/media/v1/media_storage.py +++ b/synapse/rest/media/v1/media_storage.py @@ -352,7 +352,7 @@ class ReadableFileWrapper: `IConsumer`. """ - CHUNK_SIZE = 2 ** 14 + CHUNK_SIZE = 2**14 clock: Clock path: str diff --git a/synapse/rest/media/v1/preview_html.py b/synapse/rest/media/v1/preview_html.py index 4cc9c66fbe..ca73965fc2 100644 --- a/synapse/rest/media/v1/preview_html.py +++ b/synapse/rest/media/v1/preview_html.py @@ -23,10 +23,10 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) _charset_match = re.compile( - br'<\s*meta[^>]*charset\s*=\s*"?([a-z0-9_-]+)"?', flags=re.I + rb'<\s*meta[^>]*charset\s*=\s*"?([a-z0-9_-]+)"?', flags=re.I ) _xml_encoding_match = re.compile( - br'\s*<\s*\?\s*xml[^>]*encoding="([a-z0-9_-]+)"', flags=re.I + rb'\s*<\s*\?\s*xml[^>]*encoding="([a-z0-9_-]+)"', flags=re.I ) _content_type_match = re.compile(r'.*; *charset="?(.*?)"?(;|$)', flags=re.I) diff --git a/synapse/storage/database.py b/synapse/storage/database.py index 72fef1533f..3ef2bdd74b 100644 --- a/synapse/storage/database.py +++ b/synapse/storage/database.py @@ -63,7 +63,7 @@ if TYPE_CHECKING: from synapse.server import HomeServer # python 3 does not have a maximum int value -MAX_TXN_ID = 2 ** 63 - 1 +MAX_TXN_ID = 2**63 - 1 logger = logging.getLogger(__name__) diff --git a/synapse/util/patch_inline_callbacks.py b/synapse/util/patch_inline_callbacks.py index 6d4b0b7c5a..dace68666c 100644 --- a/synapse/util/patch_inline_callbacks.py +++ b/synapse/util/patch_inline_callbacks.py @@ -217,13 +217,16 @@ def _check_yield_points( # We don't raise here as its perfectly valid for contexts to # change in a function, as long as it sets the correct context # on resolving (which is checked separately). - err = "%s changed context from %s to %s, happened between lines %d and %d in %s" % ( - frame.f_code.co_name, - expected_context, - current_context(), - last_yield_line_no, - frame.f_lineno, - frame.f_code.co_filename, + err = ( + "%s changed context from %s to %s, happened between lines %d and %d in %s" + % ( + frame.f_code.co_name, + expected_context, + current_context(), + last_yield_line_no, + frame.f_lineno, + frame.f_code.co_filename, + ) ) changes.append(err) diff --git a/synapse/util/retryutils.py b/synapse/util/retryutils.py index 648d9a95a7..d81f2527d7 100644 --- a/synapse/util/retryutils.py +++ b/synapse/util/retryutils.py @@ -30,7 +30,7 @@ MIN_RETRY_INTERVAL = 10 * 60 * 1000 RETRY_MULTIPLIER = 5 # a cap on the backoff. (Essentially none) -MAX_RETRY_INTERVAL = 2 ** 62 +MAX_RETRY_INTERVAL = 2**62 class NotRetryingDestination(Exception): diff --git a/tests/handlers/test_federation.py b/tests/handlers/test_federation.py index 89078fc637..4d65639a1e 100644 --- a/tests/handlers/test_federation.py +++ b/tests/handlers/test_federation.py @@ -496,8 +496,8 @@ class EventFromPduTestCase(TestCase): def test_invalid_numbers(self) -> None: """Invalid values for an integer should be rejected, all floats should be rejected.""" for value in [ - -(2 ** 53), - 2 ** 53, + -(2**53), + 2**53, 1.0, float("inf"), float("-inf"), @@ -524,7 +524,7 @@ class EventFromPduTestCase(TestCase): event_from_pdu_json( { "type": EventTypes.Message, - "content": {"foo": [{"bar": 2 ** 56}]}, + "content": {"foo": [{"bar": 2**56}]}, "room_id": "!room:test", "sender": "@user:test", "depth": 1, diff --git a/tests/replication/_base.py b/tests/replication/_base.py index 9c5df266bd..a0589b6d6a 100644 --- a/tests/replication/_base.py +++ b/tests/replication/_base.py @@ -206,7 +206,7 @@ class BaseStreamTestCase(unittest.HomeserverTestCase): path: bytes = request.path # type: ignore self.assertRegex( path, - br"^/_synapse/replication/get_repl_stream_updates/%s/[^/]+$" + rb"^/_synapse/replication/get_repl_stream_updates/%s/[^/]+$" % (stream_name.encode("ascii"),), ) From ca7e34cb57a93ea48e7bb00b89c63b6f14989736 Mon Sep 17 00:00:00 2001 From: Sean Quah Date: Tue, 5 Apr 2022 12:13:44 +0100 Subject: [PATCH 4/6] Pin signedjson to <= 1.1.1 as a temporary workaround for #12324 To be reverted after the Synapse 1.56 release. Signed-off-by: Sean Quah --- synapse/python_dependencies.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py index 8419ab3aca..d02cca0bbb 100644 --- a/synapse/python_dependencies.py +++ b/synapse/python_dependencies.py @@ -48,7 +48,7 @@ REQUIREMENTS = [ "unpaddedbase64>=1.1.0", "canonicaljson>=1.4.0", # we use the type definitions added in signedjson 1.1. - "signedjson>=1.1.0", + "signedjson>=1.1.0,<=1.1.1", "pynacl>=1.2.1", "idna>=2.5", # validating SSL certs for IP addresses requires service_identity 18.1. From 7a95e80418d7cc79a0fe4800a279b71fa84279f1 Mon Sep 17 00:00:00 2001 From: Sean Quah Date: Tue, 5 Apr 2022 12:44:00 +0100 Subject: [PATCH 5/6] 1.56.0 --- CHANGES.md | 16 +++++++++++++--- changelog.d/12320.misc | 1 - changelog.d/12322.misc | 1 - debian/changelog | 6 ++++++ synapse/__init__.py | 2 +- 5 files changed, 20 insertions(+), 6 deletions(-) delete mode 100644 changelog.d/12320.misc delete mode 100644 changelog.d/12322.misc diff --git a/CHANGES.md b/CHANGES.md index de40921bff..4cbe804d04 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,5 @@ -Synapse 1.56.0rc1 (2022-03-29) -============================== +Synapse 1.56.0 (2022-04-05) +=========================== Synapse will now refuse to start up if open registration is enabled, in order to help mitigate abuse across the federation. If you would like @@ -17,6 +17,16 @@ the config flag `allow_unsafe_locale`, found in the database section of the conf [upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade#change-in-behaviour-for-postgresql-databases-with-unsafe-locale) for details. +Internal Changes +---------------- + +- Bump the version of `black` for compatibility with the latest `click` release. ([\#12320](https://github.com/matrix-org/synapse/issues/12320)) +- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set to `true`. ([\#12322](https://github.com/matrix-org/synapse/issues/12322)) + + +Synapse 1.56.0rc1 (2022-03-29) +============================== + Features -------- @@ -6457,4 +6467,4 @@ Internal Changes - Don't run CI build checks until sample config check has passed. ([\#5370](https://github.com/matrix-org/synapse/issues/5370)) - Automatically retry buildkite builds (max twice) when an agent is lost. ([\#5380](https://github.com/matrix-org/synapse/issues/5380)) -**Changelogs for versions older than 1.0.0 can be found [here](CHANGES-pre-1.0.md).** \ No newline at end of file +**Changelogs for versions older than 1.0.0 can be found [here](CHANGES-pre-1.0.md).** diff --git a/changelog.d/12320.misc b/changelog.d/12320.misc deleted file mode 100644 index 7b4748d230..0000000000 --- a/changelog.d/12320.misc +++ /dev/null @@ -1 +0,0 @@ -Bump the version of `black` for compatibility with the latest `click` release. diff --git a/changelog.d/12322.misc b/changelog.d/12322.misc deleted file mode 100644 index fd595df81c..0000000000 --- a/changelog.d/12322.misc +++ /dev/null @@ -1 +0,0 @@ -Refuse to start if registration is enabled without email, captcha, or token-based verification unless new config flag `enable_registration_without_verification` is set to `true`. \ No newline at end of file diff --git a/debian/changelog b/debian/changelog index d04954457d..903d98af02 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +matrix-synapse-py3 (1.56.0) stable; urgency=medium + + * New synapse release 1.56.0. + + -- Synapse Packaging team Tue, 05 Apr 2022 12:38:39 +0100 + matrix-synapse-py3 (1.56.0~rc1) stable; urgency=medium * New synapse release 1.56.0~rc1. diff --git a/synapse/__init__.py b/synapse/__init__.py index 0960305d79..2e6510537f 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -68,7 +68,7 @@ try: except ImportError: pass -__version__ = "1.56.0rc1" +__version__ = "1.56.0" if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): # We import here so that we don't have to install a bunch of deps when From ac80bfba4252c3bb8138cc3711271be63182eed8 Mon Sep 17 00:00:00 2001 From: Sean Quah Date: Tue, 5 Apr 2022 12:55:15 +0100 Subject: [PATCH 6/6] Tweak CHANGES.md --- CHANGES.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 4cbe804d04..518781f931 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -7,7 +7,6 @@ to provide registration to anyone, consider adding [email](https://github.com/ma [recaptcha](https://matrix-org.github.io/synapse/v1.56/CAPTCHA_SETUP.html) or [token-based](https://matrix-org.github.io/synapse/v1.56/usage/administration/admin_api/registration_tokens.html) verification in order to prevent automated registration from bad actors. - This check can be disabled by setting the `enable_registration_without_verification` option in your homeserver configuration file to `true`. More details are available in the [upgrade notes](https://matrix-org.github.io/synapse/v1.56/upgrade.html#open-registration-without-verification-is-now-disabled-by-default). @@ -21,7 +20,6 @@ Internal Changes ---------------- - Bump the version of `black` for compatibility with the latest `click` release. ([\#12320](https://github.com/matrix-org/synapse/issues/12320)) -- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set to `true`. ([\#12322](https://github.com/matrix-org/synapse/issues/12322)) Synapse 1.56.0rc1 (2022-03-29) @@ -59,7 +57,7 @@ Internal Changes - Rename `shared_rooms` to `mutual_rooms` ([MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)), as per proposal changes. ([\#12036](https://github.com/matrix-org/synapse/issues/12036)) - Remove check on `update_user_directory` for shared rooms handler ([MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)), and update/expand documentation. ([\#12038](https://github.com/matrix-org/synapse/issues/12038)) - Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) instead of abusing `auth_event_ids`. ([\#12083](https://github.com/matrix-org/synapse/issues/12083), [\#12304](https://github.com/matrix-org/synapse/issues/12304)) -- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set to `true`. ([\#12091](https://github.com/matrix-org/synapse/issues/12091)) +- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set to `true`. ([\#12091](https://github.com/matrix-org/synapse/issues/12091), [\#12322](https://github.com/matrix-org/synapse/issues/12322)) - Add tests for database transaction callbacks. ([\#12198](https://github.com/matrix-org/synapse/issues/12198)) - Handle cancellation in `DatabasePool.runInteraction`. ([\#12199](https://github.com/matrix-org/synapse/issues/12199)) - Add missing type hints for cache storage. ([\#12216](https://github.com/matrix-org/synapse/issues/12216))