Merge pull request #8858 from matrix-org/rav/sso_uia

UIA: offer only available auth flows
remove unused FakeResponse (#8864 )
2020-12-02 20:06:53 +00:00 · 2020-12-02 18:58:25 +00:00 · 2020-12-02 18:54:15 +00:00 · 2020-12-02 18:54:15 +00:00 · 2020-12-02 18:30:29 +00:00 · 2020-12-02 16:30:01 +00:00
119 changed files with 854 additions and 272 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -1,3 +1,75 @@
+Synapse 1.24.0rc1 (2020-12-02)
+==============================
+
+Features
+--------
+
+- Add admin API for logging in as a user. ([\#8617](https://github.com/matrix-org/synapse/issues/8617))
+- Allow specification of the SAML IdP if the metadata returns multiple IdPs. ([\#8630](https://github.com/matrix-org/synapse/issues/8630))
+- Add support for re-trying generation of a localpart for OpenID Connect mapping providers. ([\#8801](https://github.com/matrix-org/synapse/issues/8801), [\#8855](https://github.com/matrix-org/synapse/issues/8855))
+- Allow the `Date` header through CORS. Contributed by Nicolas Chamo. ([\#8804](https://github.com/matrix-org/synapse/issues/8804))
+- Add a config option, `push.group_by_unread_count`, which controls whether unread message counts in push notifications are defined as "the number of rooms with unread messages" or "total unread messages". ([\#8820](https://github.com/matrix-org/synapse/issues/8820))
+- Add `force_purge` option to delete-room admin api. ([\#8843](https://github.com/matrix-org/synapse/issues/8843))
+
+
+Bugfixes
+--------
+
+- Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0. ([\#8744](https://github.com/matrix-org/synapse/issues/8744))
+- Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body. ([\#8776](https://github.com/matrix-org/synapse/issues/8776))
+- Fix a bug where synctl could spawn duplicate copies of a worker. Contributed by Waylon Cude. ([\#8798](https://github.com/matrix-org/synapse/issues/8798))
+- Allow per-room profiles to be used for the server notice user. ([\#8799](https://github.com/matrix-org/synapse/issues/8799))
+- Fix a bug where logging could break after a call to SIGHUP. ([\#8817](https://github.com/matrix-org/synapse/issues/8817))
+- Fix `register_new_matrix_user` failing with "Bad Request" when trailing slash is included in server URL. Contributed by @angdraug. ([\#8823](https://github.com/matrix-org/synapse/issues/8823))
+- Fix a minor long-standing bug in login, where we would offer the `password` login type if a custom auth provider supported it, even if password login was disabled. ([\#8835](https://github.com/matrix-org/synapse/issues/8835))
+- Fix a long-standing bug which caused Synapse to require unspecified parameters during user-interactive authentication. ([\#8848](https://github.com/matrix-org/synapse/issues/8848))
+- Fix a bug introduced in v1.20.0 where the user-agent and IP address reported during user registration for CAS, OpenID Connect, and SAML were of the wrong form. ([\#8784](https://github.com/matrix-org/synapse/issues/8784))
+
+
+Improved Documentation
+----------------------
+
+- Clarify the usecase for a msisdn delegate. Contributed by Adrian Wannenmacher. ([\#8734](https://github.com/matrix-org/synapse/issues/8734))
+- Remove extraneous comma from JSON example in User Admin API docs. ([\#8771](https://github.com/matrix-org/synapse/issues/8771))
+- Update `turn-howto.md` with troubleshooting notes. ([\#8779](https://github.com/matrix-org/synapse/issues/8779))
+- Fix the example on how to set the `Content-Type` header in nginx for the Client Well-Known URI. ([\#8793](https://github.com/matrix-org/synapse/issues/8793))
+- Improve the documentation for the admin API to list all media in a room with respect to encrypted events. ([\#8795](https://github.com/matrix-org/synapse/issues/8795))
+- Update the formatting of the `push` section of the homeserver config file to better align with the [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). ([\#8818](https://github.com/matrix-org/synapse/issues/8818))
+- Improve documentation how to configure prometheus for workers. ([\#8822](https://github.com/matrix-org/synapse/issues/8822))
+- Update example prometheus console. ([\#8824](https://github.com/matrix-org/synapse/issues/8824))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove old `/_matrix/client/*/admin` endpoints which were deprecated since Synapse 1.20.0. ([\#8785](https://github.com/matrix-org/synapse/issues/8785))
+- Disable pretty printing JSON responses for curl. Users who want pretty-printed output should use [jq](https://stedolan.github.io/jq/) in combination with curl. Contributed by @tulir. ([\#8833](https://github.com/matrix-org/synapse/issues/8833))
+
+
+Internal Changes
+----------------
+
+- Simplify the way the `HomeServer` object caches its internal attributes. ([\#8565](https://github.com/matrix-org/synapse/issues/8565), [\#8851](https://github.com/matrix-org/synapse/issues/8851))
+- Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru. ([\#8731](https://github.com/matrix-org/synapse/issues/8731))
+- Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite. ([\#8751](https://github.com/matrix-org/synapse/issues/8751))
+- Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events. ([\#8754](https://github.com/matrix-org/synapse/issues/8754))
+- Refactor test utilities for injecting HTTP requests. ([\#8757](https://github.com/matrix-org/synapse/issues/8757), [\#8758](https://github.com/matrix-org/synapse/issues/8758), [\#8759](https://github.com/matrix-org/synapse/issues/8759), [\#8760](https://github.com/matrix-org/synapse/issues/8760), [\#8761](https://github.com/matrix-org/synapse/issues/8761), [\#8777](https://github.com/matrix-org/synapse/issues/8777))
+- Consolidate logic between the OpenID Connect and SAML code. ([\#8765](https://github.com/matrix-org/synapse/issues/8765))
+- Use `TYPE_CHECKING` instead of magic `MYPY` variable. ([\#8770](https://github.com/matrix-org/synapse/issues/8770))
+- Add a commandline script to sign arbitrary json objects. ([\#8772](https://github.com/matrix-org/synapse/issues/8772))
+- Minor log line improvements for the SSO mapping code used to generate Matrix IDs from SSO IDs. ([\#8773](https://github.com/matrix-org/synapse/issues/8773))
+- Add additional error checking for OpenID Connect and SAML mapping providers. ([\#8774](https://github.com/matrix-org/synapse/issues/8774), [\#8800](https://github.com/matrix-org/synapse/issues/8800))
+- Add type hints to HTTP abstractions. ([\#8806](https://github.com/matrix-org/synapse/issues/8806), [\#8812](https://github.com/matrix-org/synapse/issues/8812))
+- Remove unnecessary function arguments and add typing to several membership replication classes. ([\#8809](https://github.com/matrix-org/synapse/issues/8809))
+- Optimise the lookup for an invite from another homeserver when trying to reject it. ([\#8815](https://github.com/matrix-org/synapse/issues/8815))
+- Add tests for `password_auth_provider`s. ([\#8819](https://github.com/matrix-org/synapse/issues/8819))
+- Drop redundant database index on `event_json`. ([\#8845](https://github.com/matrix-org/synapse/issues/8845))
+- Simplify `uk.half-shot.msc2778.login.application_service` login handler. ([\#8847](https://github.com/matrix-org/synapse/issues/8847))
+- Refactor `password_auth_provider` support code. ([\#8849](https://github.com/matrix-org/synapse/issues/8849))
+- Add missing `ordering` to background database updates. ([\#8850](https://github.com/matrix-org/synapse/issues/8850))
+- Allow for specifying a room version when creating a room in unit tests via `RestHelper.create_room_as`. ([\#8854](https://github.com/matrix-org/synapse/issues/8854))
+
+
 Synapse 1.23.0 (2020-11-18)
 ===========================

--- a/changelog.d/8565.misc
+++ b/changelog.d/8565.misc
@ -1 +0,0 @@
-Simplify the way the `HomeServer` object caches its internal attributes.
--- a/changelog.d/8617.feature
+++ b/changelog.d/8617.feature
@ -1 +0,0 @@
-Add admin API for logging in as a user.
--- a/changelog.d/8630.feature
+++ b/changelog.d/8630.feature
@ -1 +0,0 @@
-Allow specification of the SAML IdP if the metadata returns multiple IdPs.
--- a/changelog.d/8731.misc
+++ b/changelog.d/8731.misc
@ -1 +0,0 @@
-Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru.
--- a/changelog.d/8734.doc
+++ b/changelog.d/8734.doc
@ -1 +0,0 @@
-Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher.
--- a/changelog.d/8744.bugfix
+++ b/changelog.d/8744.bugfix
@ -1 +0,0 @@
-Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0. 
--- a/changelog.d/8751.misc
+++ b/changelog.d/8751.misc
@ -1 +0,0 @@
-Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite.
--- a/changelog.d/8754.misc
+++ b/changelog.d/8754.misc
@ -1 +0,0 @@
-Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events.
--- a/changelog.d/8757.misc
+++ b/changelog.d/8757.misc
@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
--- a/changelog.d/8758.misc
+++ b/changelog.d/8758.misc
@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
--- a/changelog.d/8759.misc
+++ b/changelog.d/8759.misc
@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
--- a/changelog.d/8760.misc
+++ b/changelog.d/8760.misc
@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
--- a/changelog.d/8761.misc
+++ b/changelog.d/8761.misc
@ -1 +0,0 @@
- Refactor test utilities for injecting HTTP requests.
--- a/changelog.d/8765.misc
+++ b/changelog.d/8765.misc
@ -1 +0,0 @@
-Consolidate logic between the OpenID Connect and SAML code.
--- a/changelog.d/8770.misc
+++ b/changelog.d/8770.misc
@ -1 +0,0 @@
-Use `TYPE_CHECKING` instead of magic `MYPY` variable.
--- a/changelog.d/8771.doc
+++ b/changelog.d/8771.doc
@ -1 +0,0 @@
-Remove extraneous comma from JSON example in User Admin API docs.
--- a/changelog.d/8772.misc
+++ b/changelog.d/8772.misc
@ -1 +0,0 @@
-Add a commandline script to sign arbitrary json objects.
--- a/changelog.d/8773.misc
+++ b/changelog.d/8773.misc
@ -1 +0,0 @@
-Minor log line improvements for the SSO mapping code used to generate Matrix IDs from SSO IDs.
--- a/changelog.d/8774.misc
+++ b/changelog.d/8774.misc
@ -1 +0,0 @@
-Add additional error checking for OpenID Connect and SAML mapping providers.
--- a/changelog.d/8776.bugfix
+++ b/changelog.d/8776.bugfix
@ -1 +0,0 @@
-Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body.
--- a/changelog.d/8777.misc
+++ b/changelog.d/8777.misc
@ -1 +0,0 @@
- Refactor test utilities for injecting HTTP requests.
--- a/changelog.d/8779.doc
+++ b/changelog.d/8779.doc
@ -1 +0,0 @@
-Update `turn-howto.md` with troubleshooting notes.
--- a/changelog.d/8784.misc
+++ b/changelog.d/8784.misc
@ -1 +0,0 @@
-Fix a bug introduced in v1.20.0 where the user-agent and IP address reported during user registration for CAS, OpenID Connect, and SAML were of the wrong form.
--- a/changelog.d/8785.removal
+++ b/changelog.d/8785.removal
@ -1 +0,0 @@
-Remove old `/_matrix/client/*/admin` endpoints which was deprecated since Synapse 1.20.0.
--- a/changelog.d/8793.doc
+++ b/changelog.d/8793.doc
@ -1 +0,0 @@
-Fix the example on how to set the `Content-Type` header in nginx for the Client Well-Known URI.
--- a/changelog.d/8795.doc
+++ b/changelog.d/8795.doc
@ -1 +0,0 @@
-Improve the documentation for the admin API to list all media in a room with respect to encrypted events.
--- a/changelog.d/8798.bugfix
+++ b/changelog.d/8798.bugfix
@ -1 +0,0 @@
-Fix a bug where synctl could spawn duplicate copies of a worker. Contributed by Waylon Cude.
--- a/changelog.d/8799.bugfix
+++ b/changelog.d/8799.bugfix
@ -1 +0,0 @@
-Allow per-room profiles to be used for the server notice user.
--- a/changelog.d/8800.misc
+++ b/changelog.d/8800.misc
@ -1 +0,0 @@
-Add additional error checking for OpenID Connect and SAML mapping providers.
--- a/changelog.d/8801.feature
+++ b/changelog.d/8801.feature
@ -1 +0,0 @@
-Add support for re-trying generation of a localpart for OpenID Connect mapping providers.
--- a/changelog.d/8802.doc
+++ b/changelog.d/8802.doc
@ -0,0 +1 @@
+Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules.
--- a/changelog.d/8804.feature
+++ b/changelog.d/8804.feature
@ -1 +0,0 @@
-Allow Date header through CORS. Contributed by Nicolas Chamo.
--- a/changelog.d/8806.misc
+++ b/changelog.d/8806.misc
@ -1 +0,0 @@
-Add type hints to HTTP abstractions.
--- a/changelog.d/8809.misc
+++ b/changelog.d/8809.misc
@ -1 +0,0 @@
-Remove unnecessary function arguments and add typing to several membership replication classes.
--- a/changelog.d/8812.misc
+++ b/changelog.d/8812.misc
@ -1 +0,0 @@
-Add type hints to HTTP abstractions.
--- a/changelog.d/8815.misc
+++ b/changelog.d/8815.misc
@ -1 +0,0 @@
-Optimise the lookup for an invite from another homeserver when trying to reject it.
--- a/changelog.d/8817.bugfix
+++ b/changelog.d/8817.bugfix
@ -1 +0,0 @@
-Fix bug where logging could break after a call to SIGHUP.
--- a/changelog.d/8818.doc
+++ b/changelog.d/8818.doc
@ -1 +0,0 @@
-Update the formatting of the `push` section of the homeserver config file to better align with the [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).
--- a/changelog.d/8819.misc
+++ b/changelog.d/8819.misc
@ -1 +0,0 @@
-Add tests for `password_auth_provider`s.
--- a/changelog.d/8820.feature
+++ b/changelog.d/8820.feature
@ -1 +0,0 @@
-Add a config option, `push.group_by_unread_count`, which controls whether unread message counts in push notifications are defined as "the number of rooms with unread messages" or "total unread messages".
--- a/changelog.d/8821.bugfix
+++ b/changelog.d/8821.bugfix
@ -0,0 +1 @@
+Apply the `federation_ip_range_blacklist` to push and key revocation requests.
--- a/changelog.d/8822.doc
+++ b/changelog.d/8822.doc
@ -1 +0,0 @@
-Improve documentation how to configure prometheus for workers.
--- a/changelog.d/8823.bugfix
+++ b/changelog.d/8823.bugfix
@ -1 +0,0 @@
-Fix `register_new_matrix_user` failing with "Bad Request" when trailing slash is included in server URL. Contributed by @angdraug.
--- a/changelog.d/8824.doc
+++ b/changelog.d/8824.doc
@ -1 +0,0 @@
-Update example prometheus console.
--- a/changelog.d/8827.bugfix
+++ b/changelog.d/8827.bugfix
@ -0,0 +1 @@
+Fix bug where we might not correctly calculate the current state for rooms with multiple extremities.
--- a/changelog.d/8833.removal
+++ b/changelog.d/8833.removal
@ -1 +0,0 @@
-Disable pretty printing JSON responses for curl. Users who want pretty-printed output should use [jq](https://stedolan.github.io/jq/) in combination with curl. Contributed by @tulir.
--- a/changelog.d/8835.bugfix
+++ b/changelog.d/8835.bugfix
@ -1 +0,0 @@
-Fix minor long-standing bug in login, where we would offer the `password` login type if a custom auth provider supported it, even if password login was disabled.
--- a/changelog.d/8837.bugfix
+++ b/changelog.d/8837.bugfix
@ -0,0 +1 @@
+Fix a long standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix.
--- a/changelog.d/8843.feature
+++ b/changelog.d/8843.feature
@ -1 +0,0 @@
-Add `force_purge` option to delete-room admin api.
--- a/changelog.d/8845.misc
+++ b/changelog.d/8845.misc
@ -1 +0,0 @@
-Drop redundant database index on `event_json`.
--- a/changelog.d/8847.misc
+++ b/changelog.d/8847.misc
@ -1 +0,0 @@
-Simplify `uk.half-shot.msc2778.login.application_service` login handler.
--- a/changelog.d/8848.bugfix
+++ b/changelog.d/8848.bugfix
@ -1 +0,0 @@
-Fix a long-standing bug which caused Synapse to require unspecified parameters during user-interactive authentication.
--- a/changelog.d/8849.misc
+++ b/changelog.d/8849.misc
@ -1 +0,0 @@
-Refactor `password_auth_provider` support code.
--- a/changelog.d/8850.misc
+++ b/changelog.d/8850.misc
@ -1 +0,0 @@
-Add missing `ordering` to background database updates.
--- a/changelog.d/8851.misc
+++ b/changelog.d/8851.misc
@ -1 +0,0 @@
-Simplify the way the `HomeServer` object caches its internal attributes.
--- a/changelog.d/8854.misc
+++ b/changelog.d/8854.misc
@ -1 +0,0 @@
-Allow for specifying a room version when creating a room in unit tests via `RestHelper.create_room_as`.
--- a/changelog.d/8855.feature
+++ b/changelog.d/8855.feature
@ -1 +0,0 @@
-Add support for re-trying generation of a localpart for OpenID Connect mapping providers.
--- a/changelog.d/8858.bugfix
+++ b/changelog.d/8858.bugfix
@ -0,0 +1 @@
+Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password.
--- a/changelog.d/8864.misc
+++ b/changelog.d/8864.misc
@ -0,0 +1 @@
+Remove unused `FakeResponse` class from unit tests.
--- a/contrib/prometheus/synapse-v2.rules
+++ b/contrib/prometheus/synapse-v2.rules
@ -58,3 +58,21 @@ groups:
    labels:
      type: "PDU"
    expr: 'synapse_federation_transaction_queue_pending_pdus + 0'
+
+  - record: synapse_storage_events_persisted_by_source_type
+    expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_type="remote"})
+    labels:
+      type: remote
+  - record: synapse_storage_events_persisted_by_source_type
+    expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_entity="*client*",origin_type="local"})
+    labels:
+      type: local
+  - record: synapse_storage_events_persisted_by_source_type
+    expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_entity!="*client*",origin_type="local"})
+    labels:
+      type: bridges
+  - record: synapse_storage_events_persisted_by_event_type
+    expr: sum without(origin_entity, origin_type) (synapse_storage_events_persisted_events_sep)
+  - record: synapse_storage_events_persisted_by_origin
+    expr: sum without(type) (synapse_storage_events_persisted_events_sep)
+
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@ -642,17 +642,19 @@ acme:
 #  - nyc.example.com
 #  - syd.example.com

-# Prevent federation requests from being sent to the following
-# blacklist IP address CIDR ranges. If this option is not specified, or
-# specified with an empty list, no ip range blacklist will be enforced.
+# Prevent outgoing requests from being sent to the following blacklisted IP address
+# CIDR ranges. If this option is not specified, or specified with an empty list,
+# no IP range blacklist will be enforced.
 #
-# As of Synapse v1.4.0 this option also affects any outbound requests to identity
-# servers provided by user input.
+# The blacklist applies to the outbound requests for federation, identity servers,
+# push servers, and for checking key validitity for third-party invite events.
 #
 # (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
 # listed here, since they correspond to unroutable addresses.)
 #
-federation_ip_range_blacklist:
+# This option replaces federation_ip_range_blacklist in Synapse v1.24.0.
+#
+ip_range_blacklist:
  - '127.0.0.0/8'
  - '10.0.0.0/8'
  - '172.16.0.0/12'
--- a/synapse/init.py
+++ b/synapse/init.py
@ -48,7 +48,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.23.0"
+__version__ = "1.24.0rc1"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@ -266,7 +266,6 @@ class GenericWorkerPresence(BasePresenceHandler):
        super().__init__(hs)
        self.hs = hs
        self.is_mine_id = hs.is_mine_id
-        self.http_client = hs.get_simple_http_client()

        self._presence_enabled = hs.config.use_presence

--- a/synapse/config/federation.py
+++ b/synapse/config/federation.py
@ -36,22 +36,30 @@ class FederationConfig(Config):
            for domain in federation_domain_whitelist:
                self.federation_domain_whitelist[domain] = True

-        self.federation_ip_range_blacklist = config.get(
-            "federation_ip_range_blacklist", []
-        )
+        ip_range_blacklist = config.get("ip_range_blacklist", [])

        # Attempt to create an IPSet from the given ranges
        try:
-            self.federation_ip_range_blacklist = IPSet(
-                self.federation_ip_range_blacklist
-            )
+            self.ip_range_blacklist = IPSet(ip_range_blacklist)
+        except Exception as e:
+            raise ConfigError("Invalid range(s) provided in ip_range_blacklist: %s" % e)
+        # Always blacklist 0.0.0.0, ::
+        self.ip_range_blacklist.update(["0.0.0.0", "::"])

-            # Always blacklist 0.0.0.0, ::
-            self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])
+        # The federation_ip_range_blacklist is used for backwards-compatibility
+        # and only applies to federation and identity servers. If it is not given,
+        # default to ip_range_blacklist.
+        federation_ip_range_blacklist = config.get(
+            "federation_ip_range_blacklist", ip_range_blacklist
+        )
+        try:
+            self.federation_ip_range_blacklist = IPSet(federation_ip_range_blacklist)
        except Exception as e:
            raise ConfigError(
                "Invalid range(s) provided in federation_ip_range_blacklist: %s" % e
            )
+        # Always blacklist 0.0.0.0, ::
+        self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])

        federation_metrics_domains = config.get("federation_metrics_domains") or []
        validate_config(
@ -76,17 +84,19 @@ class FederationConfig(Config):
        #  - nyc.example.com
        #  - syd.example.com

-        # Prevent federation requests from being sent to the following
-        # blacklist IP address CIDR ranges. If this option is not specified, or
-        # specified with an empty list, no ip range blacklist will be enforced.
+        # Prevent outgoing requests from being sent to the following blacklisted IP address
+        # CIDR ranges. If this option is not specified, or specified with an empty list,
+        # no IP range blacklist will be enforced.
        #
-        # As of Synapse v1.4.0 this option also affects any outbound requests to identity
-        # servers provided by user input.
+        # The blacklist applies to the outbound requests for federation, identity servers,
+        # push servers, and for checking key validitity for third-party invite events.
        #
        # (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
        # listed here, since they correspond to unroutable addresses.)
        #
-        federation_ip_range_blacklist:
+        # This option replaces federation_ip_range_blacklist in Synapse v1.24.0.
+        #
+        ip_range_blacklist:
          - '127.0.0.0/8'
          - '10.0.0.0/8'
          - '172.16.0.0/12'
--- a/synapse/crypto/keyring.py
+++ b/synapse/crypto/keyring.py
@ -578,7 +578,7 @@ class PerspectivesKeyFetcher(BaseV2KeyFetcher):
    def __init__(self, hs):
        super().__init__(hs)
        self.clock = hs.get_clock()
-        self.client = hs.get_http_client()
+        self.client = hs.get_federation_http_client()
        self.key_servers = self.config.key_servers

    async def get_keys(self, keys_to_fetch):
@ -748,7 +748,7 @@ class ServerKeyFetcher(BaseV2KeyFetcher):
    def __init__(self, hs):
        super().__init__(hs)
        self.clock = hs.get_clock()
-        self.client = hs.get_http_client()
+        self.client = hs.get_federation_http_client()

    async def get_keys(self, keys_to_fetch):
        """
--- a/synapse/federation/federation_server.py
+++ b/synapse/federation/federation_server.py
@ -845,7 +845,6 @@ class FederationHandlerRegistry:

    def __init__(self, hs: "HomeServer"):
        self.config = hs.config
-        self.http_client = hs.get_simple_http_client()
        self.clock = hs.get_clock()
        self._instance_name = hs.get_instance_name()

--- a/synapse/federation/transport/client.py
+++ b/synapse/federation/transport/client.py
@ -35,7 +35,7 @@ class TransportLayerClient:

    def __init__(self, hs):
        self.server_name = hs.hostname
-        self.client = hs.get_http_client()
+        self.client = hs.get_federation_http_client()

    @log_function
    def get_room_state_ids(self, destination, room_id, event_id):
--- a/synapse/federation/transport/server.py
+++ b/synapse/federation/transport/server.py
@ -1462,7 +1462,7 @@ def register_servlets(hs, resource, authenticator, ratelimiter, servlet_groups=N

    Args:
        hs (synapse.server.HomeServer): homeserver
-        resource (TransportLayerServer): resource class to register to
+        resource (JsonResource): resource class to register to
        authenticator (Authenticator): authenticator to use
        ratelimiter (util.ratelimitutils.FederationRateLimiter): ratelimiter to use
        servlet_groups (list[str], optional): List of servlet groups to register.
--- a/synapse/handlers/auth.py
+++ b/synapse/handlers/auth.py
@ -193,9 +193,7 @@ class AuthHandler(BaseHandler):
        self.hs = hs  # FIXME better possibility to access registrationHandler later?
        self.macaroon_gen = hs.get_macaroon_generator()
        self._password_enabled = hs.config.password_enabled
-        self._sso_enabled = (
-            hs.config.cas_enabled or hs.config.saml2_enabled or hs.config.oidc_enabled
-        )
+        self._password_localdb_enabled = hs.config.password_localdb_enabled

        # we keep this as a list despite the O(N^2) implication so that we can
        # keep PASSWORD first and avoid confusing clients which pick the first
@ -205,7 +203,7 @@ class AuthHandler(BaseHandler):

        # start out by assuming PASSWORD is enabled; we will remove it later if not.
        login_types = []
-        if hs.config.password_localdb_enabled:
+        if self._password_localdb_enabled:
            login_types.append(LoginType.PASSWORD)

        for provider in self.password_providers:
@ -219,14 +217,6 @@ class AuthHandler(BaseHandler):

        self._supported_login_types = login_types

-        # Login types and UI Auth types have a heavy overlap, but are not
-        # necessarily identical. Login types have SSO (and other login types)
-        # added in the rest layer, see synapse.rest.client.v1.login.LoginRestServerlet.on_GET.
-        ui_auth_types = login_types.copy()
-        if self._sso_enabled:
-            ui_auth_types.append(LoginType.SSO)
-        self._supported_ui_auth_types = ui_auth_types
-
        # Ratelimiter for failed auth during UIA. Uses same ratelimit config
        # as per `rc_login.failed_attempts`.
        self._failed_uia_attempts_ratelimiter = Ratelimiter(
@ -339,7 +329,10 @@ class AuthHandler(BaseHandler):
        self._failed_uia_attempts_ratelimiter.ratelimit(user_id, update=False)

        # build a list of supported flows
-        flows = [[login_type] for login_type in self._supported_ui_auth_types]
+        supported_ui_auth_types = await self._get_available_ui_auth_types(
+            requester.user
+        )
+        flows = [[login_type] for login_type in supported_ui_auth_types]

        try:
            result, params, session_id = await self.check_ui_auth(
@ -351,7 +344,7 @@ class AuthHandler(BaseHandler):
            raise

        # find the completed login type
-        for login_type in self._supported_ui_auth_types:
+        for login_type in supported_ui_auth_types:
            if login_type not in result:
                continue

@ -367,6 +360,41 @@ class AuthHandler(BaseHandler):

        return params, session_id

+    async def _get_available_ui_auth_types(self, user: UserID) -> Iterable[str]:
+        """Get a list of the authentication types this user can use
+        """
+
+        ui_auth_types = set()
+
+        # if the HS supports password auth, and the user has a non-null password, we
+        # support password auth
+        if self._password_localdb_enabled and self._password_enabled:
+            lookupres = await self._find_user_id_and_pwd_hash(user.to_string())
+            if lookupres:
+                _, password_hash = lookupres
+                if password_hash:
+                    ui_auth_types.add(LoginType.PASSWORD)
+
+        # also allow auth from password providers
+        for provider in self.password_providers:
+            for t in provider.get_supported_login_types().keys():
+                if t == LoginType.PASSWORD and not self._password_enabled:
+                    continue
+                ui_auth_types.add(t)
+
+        # if sso is enabled, allow the user to log in via SSO iff they have a mapping
+        # from sso to mxid.
+        if self.hs.config.saml2.saml2_enabled or self.hs.config.oidc.oidc_enabled:
+            if await self.store.get_external_ids_by_user(user.to_string()):
+                ui_auth_types.add(LoginType.SSO)
+
+        # Our CAS impl does not (yet) correctly register users in user_external_ids,
+        # so always offer that if it's available.
+        if self.hs.config.cas.cas_enabled:
+            ui_auth_types.add(LoginType.SSO)
+
+        return ui_auth_types
+
    def get_enabled_auth_types(self):
        """Return the enabled user-interactive authentication types

@ -1029,7 +1057,7 @@ class AuthHandler(BaseHandler):
            if result:
                return result

-        if login_type == LoginType.PASSWORD and self.hs.config.password_localdb_enabled:
+        if login_type == LoginType.PASSWORD and self._password_localdb_enabled:
            known_login_type = True

            # we've already checked that there is a (valid) password field
--- a/synapse/handlers/federation.py
+++ b/synapse/handlers/federation.py
@ -140,7 +140,7 @@ class FederationHandler(BaseHandler):
        self._message_handler = hs.get_message_handler()
        self._server_notices_mxid = hs.config.server_notices_mxid
        self.config = hs.config
-        self.http_client = hs.get_simple_http_client()
+        self.http_client = hs.get_proxied_blacklisted_http_client()
        self._instance_name = hs.get_instance_name()
        self._replication = hs.get_replication_data_handler()

--- a/synapse/handlers/identity.py
+++ b/synapse/handlers/identity.py
@ -46,13 +46,13 @@ class IdentityHandler(BaseHandler):
    def __init__(self, hs):
        super().__init__(hs)

+        # An HTTP client for contacting trusted URLs.
        self.http_client = SimpleHttpClient(hs)
-        # We create a blacklisting instance of SimpleHttpClient for contacting identity
-        # servers specified by clients
+        # An HTTP client for contacting identity servers specified by clients.
        self.blacklisting_http_client = SimpleHttpClient(
            hs, ip_blacklist=hs.config.federation_ip_range_blacklist
        )
-        self.federation_http_client = hs.get_http_client()
+        self.federation_http_client = hs.get_federation_http_client()
        self.hs = hs

    async def threepid_from_creds(
--- a/synapse/http/client.py
+++ b/synapse/http/client.py
@ -125,7 +125,7 @@ def _make_scheduler(reactor):
    return _scheduler


-class IPBlacklistingResolver:
+class _IPBlacklistingResolver:
    """
    A proxy for reactor.nameResolver which only produces non-blacklisted IP
    addresses, preventing DNS rebinding attacks on URL preview.
@ -199,6 +199,35 @@ class IPBlacklistingResolver:
        return r


+@implementer(IReactorPluggableNameResolver)
+class BlacklistingReactorWrapper:
+    """
+    A Reactor wrapper which will prevent DNS resolution to blacklisted IP
+    addresses, to prevent DNS rebinding.
+    """
+
+    def __init__(
+        self,
+        reactor: IReactorPluggableNameResolver,
+        ip_whitelist: Optional[IPSet],
+        ip_blacklist: IPSet,
+    ):
+        self._reactor = reactor
+
+        # We need to use a DNS resolver which filters out blacklisted IP
+        # addresses, to prevent DNS rebinding.
+        self._nameResolver = _IPBlacklistingResolver(
+            self._reactor, ip_whitelist, ip_blacklist
+        )
+
+    def __getattr__(self, attr: str) -> Any:
+        # Passthrough to the real reactor except for the DNS resolver.
+        if attr == "nameResolver":
+            return self._nameResolver
+        else:
+            return getattr(self._reactor, attr)
+
+
 class BlacklistingAgentWrapper(Agent):
    """
    An Agent wrapper which will prevent access to IP addresses being accessed
@ -292,22 +321,11 @@ class SimpleHttpClient:
        self.user_agent = self.user_agent.encode("ascii")

        if self._ip_blacklist:
-            real_reactor = hs.get_reactor()
            # If we have an IP blacklist, we need to use a DNS resolver which
            # filters out blacklisted IP addresses, to prevent DNS rebinding.
-            nameResolver = IPBlacklistingResolver(
-                real_reactor, self._ip_whitelist, self._ip_blacklist
+            self.reactor = BlacklistingReactorWrapper(
+                hs.get_reactor(), self._ip_whitelist, self._ip_blacklist
            )
-
-            @implementer(IReactorPluggableNameResolver)
-            class Reactor:
-                def __getattr__(_self, attr):
-                    if attr == "nameResolver":
-                        return nameResolver
-                    else:
-                        return getattr(real_reactor, attr)
-
-            self.reactor = Reactor()
        else:
            self.reactor = hs.get_reactor()

--- a/synapse/http/federation/matrix_federation_agent.py
+++ b/synapse/http/federation/matrix_federation_agent.py
@ -16,7 +16,7 @@ import logging
 import urllib.parse
 from typing import List, Optional

-from netaddr import AddrFormatError, IPAddress
+from netaddr import AddrFormatError, IPAddress, IPSet
 from zope.interface import implementer

 from twisted.internet import defer
@ -31,6 +31,7 @@ from twisted.web.http_headers import Headers
 from twisted.web.iweb import IAgent, IAgentEndpointFactory, IBodyProducer

 from synapse.crypto.context_factory import FederationPolicyForHTTPS
+from synapse.http.client import BlacklistingAgentWrapper
 from synapse.http.federation.srv_resolver import Server, SrvResolver
 from synapse.http.federation.well_known_resolver import WellKnownResolver
 from synapse.logging.context import make_deferred_yieldable, run_in_background
@ -70,6 +71,7 @@ class MatrixFederationAgent:
        reactor: IReactorCore,
        tls_client_options_factory: Optional[FederationPolicyForHTTPS],
        user_agent: bytes,
+        ip_blacklist: IPSet,
        _srv_resolver: Optional[SrvResolver] = None,
        _well_known_resolver: Optional[WellKnownResolver] = None,
    ):
@ -90,12 +92,18 @@ class MatrixFederationAgent:
        self.user_agent = user_agent

        if _well_known_resolver is None:
+            # Note that the name resolver has already been wrapped in a
+            # IPBlacklistingResolver by MatrixFederationHttpClient.
            _well_known_resolver = WellKnownResolver(
                self._reactor,
-                agent=Agent(
+                agent=BlacklistingAgentWrapper(
+                    Agent(
+                        self._reactor,
+                        pool=self._pool,
+                        contextFactory=tls_client_options_factory,
+                    ),
                    self._reactor,
-                    pool=self._pool,
-                    contextFactory=tls_client_options_factory,
+                    ip_blacklist=ip_blacklist,
                ),
                user_agent=self.user_agent,
            )
--- a/synapse/http/matrixfederationclient.py
+++ b/synapse/http/matrixfederationclient.py
@ -26,11 +26,10 @@ import treq
 from canonicaljson import encode_canonical_json
 from prometheus_client import Counter
 from signedjson.sign import sign_json
-from zope.interface import implementer

 from twisted.internet import defer
 from twisted.internet.error import DNSLookupError
-from twisted.internet.interfaces import IReactorPluggableNameResolver, IReactorTime
+from twisted.internet.interfaces import IReactorTime
 from twisted.internet.task import _EPSILON, Cooperator
 from twisted.web.http_headers import Headers
 from twisted.web.iweb import IBodyProducer, IResponse
@ -45,7 +44,7 @@ from synapse.api.errors import (
 from synapse.http import QuieterFileBodyProducer
 from synapse.http.client import (
    BlacklistingAgentWrapper,
-    IPBlacklistingResolver,
+    BlacklistingReactorWrapper,
    encode_query_args,
    readBodyToFile,
 )
@ -221,31 +220,22 @@ class MatrixFederationHttpClient:
        self.signing_key = hs.signing_key
        self.server_name = hs.hostname

-        real_reactor = hs.get_reactor()
-
        # We need to use a DNS resolver which filters out blacklisted IP
        # addresses, to prevent DNS rebinding.
-        nameResolver = IPBlacklistingResolver(
-            real_reactor, None, hs.config.federation_ip_range_blacklist
+        self.reactor = BlacklistingReactorWrapper(
+            hs.get_reactor(), None, hs.config.federation_ip_range_blacklist
        )

-        @implementer(IReactorPluggableNameResolver)
-        class Reactor:
-            def __getattr__(_self, attr):
-                if attr == "nameResolver":
-                    return nameResolver
-                else:
-                    return getattr(real_reactor, attr)
-
-        self.reactor = Reactor()
-
        user_agent = hs.version_string
        if hs.config.user_agent_suffix:
            user_agent = "%s %s" % (user_agent, hs.config.user_agent_suffix)
        user_agent = user_agent.encode("ascii")

        self.agent = MatrixFederationAgent(
-            self.reactor, tls_client_options_factory, user_agent
+            self.reactor,
+            tls_client_options_factory,
+            user_agent,
+            hs.config.federation_ip_range_blacklist,
        )

        # Use a BlacklistingAgentWrapper to prevent circumventing the IP
--- a/synapse/push/httppusher.py
+++ b/synapse/push/httppusher.py
@ -100,7 +100,7 @@ class HttpPusher:
        if "url" not in self.data:
            raise PusherConfigException("'url' required in data for HTTP pusher")
        self.url = self.data["url"]
-        self.http_client = hs.get_proxied_http_client()
+        self.http_client = hs.get_proxied_blacklisted_http_client()
        self.data_minus_url = {}
        self.data_minus_url.update(self.data)
        del self.data_minus_url["url"]
--- a/synapse/rest/admin/users.py
+++ b/synapse/rest/admin/users.py
@ -420,6 +420,9 @@ class UserRegisterServlet(RestServlet):
        if user_type is not None and user_type not in UserTypes.ALL_USER_TYPES:
            raise SynapseError(400, "Invalid user type")

+        if "mac" not in body:
+            raise SynapseError(400, "mac must be specified", errcode=Codes.BAD_JSON)
+
        got_mac = body["mac"]

        want_mac_builder = hmac.new(
--- a/synapse/rest/media/v1/media_repository.py
+++ b/synapse/rest/media/v1/media_repository.py
@ -66,7 +66,7 @@ class MediaRepository:
    def __init__(self, hs):
        self.hs = hs
        self.auth = hs.get_auth()
-        self.client = hs.get_http_client()
+        self.client = hs.get_federation_http_client()
        self.clock = hs.get_clock()
        self.server_name = hs.hostname
        self.store = hs.get_datastore()
--- a/synapse/server.py
+++ b/synapse/server.py
@ -350,16 +350,45 @@ class HomeServer(metaclass=abc.ABCMeta):

    @cache_in_self
    def get_simple_http_client(self) -> SimpleHttpClient:
+        """
+        An HTTP client with no special configuration.
+        """
        return SimpleHttpClient(self)

    @cache_in_self
    def get_proxied_http_client(self) -> SimpleHttpClient:
+        """
+        An HTTP client that uses configured HTTP(S) proxies.
+        """
        return SimpleHttpClient(
            self,
            http_proxy=os.getenvb(b"http_proxy"),
            https_proxy=os.getenvb(b"HTTPS_PROXY"),
        )

+    @cache_in_self
+    def get_proxied_blacklisted_http_client(self) -> SimpleHttpClient:
+        """
+        An HTTP client that uses configured HTTP(S) proxies and blacklists IPs
+        based on the IP range blacklist.
+        """
+        return SimpleHttpClient(
+            self,
+            ip_blacklist=self.config.ip_range_blacklist,
+            http_proxy=os.getenvb(b"http_proxy"),
+            https_proxy=os.getenvb(b"HTTPS_PROXY"),
+        )
+
+    @cache_in_self
+    def get_federation_http_client(self) -> MatrixFederationHttpClient:
+        """
+        An HTTP client for federation.
+        """
+        tls_client_options_factory = context_factory.FederationPolicyForHTTPS(
+            self.config
+        )
+        return MatrixFederationHttpClient(self, tls_client_options_factory)
+
    @cache_in_self
    def get_room_creation_handler(self) -> RoomCreationHandler:
        return RoomCreationHandler(self)
@ -514,13 +543,6 @@ class HomeServer(metaclass=abc.ABCMeta):
    def get_pusherpool(self) -> PusherPool:
        return PusherPool(self)

-    @cache_in_self
-    def get_http_client(self) -> MatrixFederationHttpClient:
-        tls_client_options_factory = context_factory.FederationPolicyForHTTPS(
-            self.config
-        )
-        return MatrixFederationHttpClient(self, tls_client_options_factory)
-
    @cache_in_self
    def get_media_repository_resource(self) -> MediaRepositoryResource:
        # build the media repo resource. This indirects through the HomeServer
--- a/synapse/state/v2.py
+++ b/synapse/state/v2.py
@ -38,7 +38,7 @@ from synapse.api.constants import EventTypes
 from synapse.api.errors import AuthError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.events import EventBase
-from synapse.types import MutableStateMap, StateMap
+from synapse.types import Collection, MutableStateMap, StateMap
 from synapse.util import Clock

 logger = logging.getLogger(__name__)
@ -252,9 +252,88 @@ async def _get_auth_chain_difference(
        Set of event IDs
    """

-    difference = await state_res_store.get_auth_chain_difference(
-        [set(state_set.values()) for state_set in state_sets]
-    )
+    # The `StateResolutionStore.get_auth_chain_difference` function assumes that
+    # all events passed to it (and their auth chains) have been persisted
+    # previously. This is not the case for any events in the `event_map`, and so
+    # we need to manually handle those events.
+    #
+    # We do this by:
+    #   1. calculating the auth chain difference for the state sets based on the
+    #      events in `event_map` alone
+    #   2. replacing any events in the state_sets that are also in `event_map`
+    #      with their auth events (recursively), and then calling
+    #      `store.get_auth_chain_difference` as normal
+    #   3. adding the results of 1 and 2 together.
+
+    # Map from event ID in `event_map` to their auth event IDs, and their auth
+    # event IDs if they appear in the `event_map`. This is the intersection of
+    # the event's auth chain with the events in the `event_map` *plus* their
+    # auth event IDs.
+    events_to_auth_chain = {}  # type: Dict[str, Set[str]]
+    for event in event_map.values():
+        chain = {event.event_id}
+        events_to_auth_chain[event.event_id] = chain
+
+        to_search = [event]
+        while to_search:
+            for auth_id in to_search.pop().auth_event_ids():
+                chain.add(auth_id)
+                auth_event = event_map.get(auth_id)
+                if auth_event:
+                    to_search.append(auth_event)
+
+    # We now a) calculate the auth chain difference for the unpersisted events
+    # and b) work out the state sets to pass to the store.
+    #
+    # Note: If the `event_map` is empty (which is the common case), we can do a
+    # much simpler calculation.
+    if event_map:
+        # The list of state sets to pass to the store, where each state set is a set
+        # of the event ids making up the state. This is similar to `state_sets`,
+        # except that (a) we only have event ids, not the complete
+        # ((type, state_key)->event_id) mappings; and (b) we have stripped out
+        # unpersisted events and replaced them with the persisted events in
+        # their auth chain.
+        state_sets_ids = []  # type: List[Set[str]]
+
+        # For each state set, the unpersisted event IDs reachable (by their auth
+        # chain) from the events in that set.
+        unpersisted_set_ids = []  # type: List[Set[str]]
+
+        for state_set in state_sets:
+            set_ids = set()  # type: Set[str]
+            state_sets_ids.append(set_ids)
+
+            unpersisted_ids = set()  # type: Set[str]
+            unpersisted_set_ids.append(unpersisted_ids)
+
+            for event_id in state_set.values():
+                event_chain = events_to_auth_chain.get(event_id)
+                if event_chain is not None:
+                    # We have an event in `event_map`. We add all the auth
+                    # events that it references (that aren't also in `event_map`).
+                    set_ids.update(e for e in event_chain if e not in event_map)
+
+                    # We also add the full chain of unpersisted event IDs
+                    # referenced by this state set, so that we can work out the
+                    # auth chain difference of the unpersisted events.
+                    unpersisted_ids.update(e for e in event_chain if e in event_map)
+                else:
+                    set_ids.add(event_id)
+
+        # The auth chain difference of the unpersisted events of the state sets
+        # is calculated by taking the difference between the union and
+        # intersections.
+        union = unpersisted_set_ids[0].union(*unpersisted_set_ids[1:])
+        intersection = unpersisted_set_ids[0].intersection(*unpersisted_set_ids[1:])
+
+        difference_from_event_map = union - intersection  # type: Collection[str]
+    else:
+        difference_from_event_map = ()
+        state_sets_ids = [set(state_set.values()) for state_set in state_sets]
+
+    difference = await state_res_store.get_auth_chain_difference(state_sets_ids)
+    difference.update(difference_from_event_map)

    return difference

--- a/synapse/storage/databases/main/registration.py
+++ b/synapse/storage/databases/main/registration.py
@ -463,6 +463,23 @@ class RegistrationWorkerStore(CacheInvalidationWorkerStore):
            desc="get_user_by_external_id",
        )

+    async def get_external_ids_by_user(self, mxid: str) -> List[Tuple[str, str]]:
+        """Look up external ids for the given user
+
+        Args:
+            mxid: the MXID to be looked up
+
+        Returns:
+            Tuples of (auth_provider, external_id)
+        """
+        res = await self.db_pool.simple_select_list(
+            table="user_external_ids",
+            keyvalues={"user_id": mxid},
+            retcols=("auth_provider", "external_id"),
+            desc="get_external_ids_by_user",
+        )
+        return [(r["auth_provider"], r["external_id"]) for r in res]
+
    async def count_all_users(self):
        """Counts all users registered on the homeserver."""

@ -963,6 +980,14 @@ class RegistrationBackgroundUpdateStore(RegistrationWorkerStore):
            "users_set_deactivated_flag", self._background_update_set_deactivated_flag
        )

+        self.db_pool.updates.register_background_index_update(
+            "user_external_ids_user_id_idx",
+            index_name="user_external_ids_user_id_idx",
+            table="user_external_ids",
+            columns=["user_id"],
+            unique=False,
+        )
+
    async def _background_update_set_deactivated_flag(self, progress, batch_size):
        """Retrieves a list of all deactivated users and sets the 'deactivated' flag to 1
        for each of them.
--- a/synapse/storage/databases/main/schema/delta/58/25user_external_ids_user_id_idx.sql
+++ b/synapse/storage/databases/main/schema/delta/58/25user_external_ids_user_id_idx.sql
@ -0,0 +1,17 @@
+/* Copyright 2020 The Matrix.org Foundation C.I.C
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+INSERT INTO background_updates (ordering, update_name, progress_json) VALUES
+  (5825, 'user_external_ids_user_id_idx', '{}');
--- a/tests/api/test_filtering.py
+++ b/tests/api/test_filtering.py
@ -50,7 +50,9 @@ class FilteringTestCase(unittest.TestCase):
        self.mock_http_client.put_json = DeferredMockCallable()

        hs = yield setup_test_homeserver(
-            self.addCleanup, http_client=self.mock_http_client, keyring=Mock(),
+            self.addCleanup,
+            federation_http_client=self.mock_http_client,
+            keyring=Mock(),
        )

        self.filtering = hs.get_filtering()
--- a/tests/app/test_frontend_proxy.py
+++ b/tests/app/test_frontend_proxy.py
@ -23,7 +23,7 @@ class FrontendProxyTests(HomeserverTestCase):
    def make_homeserver(self, reactor, clock):

        hs = self.setup_test_homeserver(
-            http_client=None, homeserver_to_use=GenericWorkerServer
+            federation_http_client=None, homeserver_to_use=GenericWorkerServer
        )

        return hs
--- a/tests/app/test_openid_listener.py
+++ b/tests/app/test_openid_listener.py
@ -27,7 +27,7 @@ from tests.unittest import HomeserverTestCase
 class FederationReaderOpenIDListenerTests(HomeserverTestCase):
    def make_homeserver(self, reactor, clock):
        hs = self.setup_test_homeserver(
-            http_client=None, homeserver_to_use=GenericWorkerServer
+            federation_http_client=None, homeserver_to_use=GenericWorkerServer
        )
        return hs

@ -84,7 +84,7 @@ class FederationReaderOpenIDListenerTests(HomeserverTestCase):
 class SynapseHomeserverOpenIDListenerTests(HomeserverTestCase):
    def make_homeserver(self, reactor, clock):
        hs = self.setup_test_homeserver(
-            http_client=None, homeserver_to_use=SynapseHomeServer
+            federation_http_client=None, homeserver_to_use=SynapseHomeServer
        )
        return hs

--- a/tests/crypto/test_keyring.py
+++ b/tests/crypto/test_keyring.py
@ -315,7 +315,7 @@ class KeyringTestCase(unittest.HomeserverTestCase):
 class ServerKeyFetcherTestCase(unittest.HomeserverTestCase):
    def make_homeserver(self, reactor, clock):
        self.http_client = Mock()
-        hs = self.setup_test_homeserver(http_client=self.http_client)
+        hs = self.setup_test_homeserver(federation_http_client=self.http_client)
        return hs

    def test_get_keys_from_server(self):
@ -395,7 +395,9 @@ class PerspectivesKeyFetcherTestCase(unittest.HomeserverTestCase):
            }
        ]

-        return self.setup_test_homeserver(http_client=self.http_client, config=config)
+        return self.setup_test_homeserver(
+            federation_http_client=self.http_client, config=config
+        )

    def build_perspectives_response(
        self, server_name: str, signing_key: SigningKey, valid_until_ts: int,
--- a/tests/handlers/test_device.py
+++ b/tests/handlers/test_device.py
@ -27,7 +27,7 @@ user2 = "@theresa:bbb"

 class DeviceTestCase(unittest.HomeserverTestCase):
    def make_homeserver(self, reactor, clock):
-        hs = self.setup_test_homeserver("server", http_client=None)
+        hs = self.setup_test_homeserver("server", federation_http_client=None)
        self.handler = hs.get_device_handler()
        self.store = hs.get_datastore()
        return hs
@ -229,7 +229,7 @@ class DeviceTestCase(unittest.HomeserverTestCase):

 class DehydrationTestCase(unittest.HomeserverTestCase):
    def make_homeserver(self, reactor, clock):
-        hs = self.setup_test_homeserver("server", http_client=None)
+        hs = self.setup_test_homeserver("server", federation_http_client=None)
        self.handler = hs.get_device_handler()
        self.registration = hs.get_registration_handler()
        self.auth = hs.get_auth()
--- a/tests/handlers/test_directory.py
+++ b/tests/handlers/test_directory.py
@ -42,7 +42,7 @@ class DirectoryTestCase(unittest.HomeserverTestCase):
        self.mock_registry.register_query_handler = register_query_handler

        hs = self.setup_test_homeserver(
-            http_client=None,
+            federation_http_client=None,
            resource_for_federation=Mock(),
            federation_client=self.mock_federation,
            federation_registry=self.mock_registry,
--- a/tests/handlers/test_federation.py
+++ b/tests/handlers/test_federation.py
@ -37,7 +37,7 @@ class FederationTestCase(unittest.HomeserverTestCase):
    ]

    def make_homeserver(self, reactor, clock):
-        hs = self.setup_test_homeserver(http_client=None)
+        hs = self.setup_test_homeserver(federation_http_client=None)
        self.handler = hs.get_federation_handler()
        self.store = hs.get_datastore()
        return hs
--- a/tests/handlers/test_oidc.py
+++ b/tests/handlers/test_oidc.py
@ -17,30 +17,15 @@ from urllib.parse import parse_qs, urlparse

 from mock import Mock, patch

-import attr
 import pymacaroons

-from twisted.python.failure import Failure
-from twisted.web._newclient import ResponseDone
-
 from synapse.handlers.oidc_handler import OidcError, OidcMappingProvider
 from synapse.handlers.sso import MappingException
 from synapse.types import UserID

+from tests.test_utils import FakeResponse
 from tests.unittest import HomeserverTestCase, override_config

-
-@attr.s
-class FakeResponse:
-    code = attr.ib()
-    body = attr.ib()
-    phrase = attr.ib()
-
-    def deliverBody(self, protocol):
-        protocol.dataReceived(self.body)
-        protocol.connectionLost(Failure(ResponseDone()))
-
-
 # These are a few constants that are used as config parameters in the tests.
 ISSUER = "https://issuer/"
 CLIENT_ID = "test-client-id"
--- a/tests/handlers/test_presence.py
+++ b/tests/handlers/test_presence.py
@ -463,7 +463,7 @@ class PresenceJoinTestCase(unittest.HomeserverTestCase):

    def make_homeserver(self, reactor, clock):
        hs = self.setup_test_homeserver(
-            "server", http_client=None, federation_sender=Mock()
+            "server", federation_http_client=None, federation_sender=Mock()
        )
        return hs

--- a/tests/handlers/test_profile.py
+++ b/tests/handlers/test_profile.py
@ -44,7 +44,7 @@ class ProfileTestCase(unittest.TestCase):

        hs = yield setup_test_homeserver(
            self.addCleanup,
-            http_client=None,
+            federation_http_client=None,
            resource_for_federation=Mock(),
            federation_client=self.mock_federation,
            federation_server=Mock(),
--- a/tests/handlers/test_typing.py
+++ b/tests/handlers/test_typing.py
@ -15,18 +15,20 @@


 import json
+from typing import Dict

 from mock import ANY, Mock, call

 from twisted.internet import defer
+from twisted.web.resource import Resource

 from synapse.api.errors import AuthError
+from synapse.federation.transport.server import TransportLayerServer
 from synapse.types import UserID, create_requester

 from tests import unittest
 from tests.test_utils import make_awaitable
 from tests.unittest import override_config
-from tests.utils import register_federation_servlets

 # Some local users to test with
 U_APPLE = UserID.from_string("@apple:test")
@ -53,8 +55,6 @@ def _make_edu_transaction_json(edu_type, content):


 class TypingNotificationsTestCase(unittest.HomeserverTestCase):
-    servlets = [register_federation_servlets]
-
    def make_homeserver(self, reactor, clock):
        # we mock out the keyring so as to skip the authentication check on the
        # federation API call.
@ -70,13 +70,18 @@ class TypingNotificationsTestCase(unittest.HomeserverTestCase):

        hs = self.setup_test_homeserver(
            notifier=Mock(),
-            http_client=mock_federation_client,
+            federation_http_client=mock_federation_client,
            keyring=mock_keyring,
            replication_streams={},
        )

        return hs

+    def create_resource_dict(self) -> Dict[str, Resource]:
+        d = super().create_resource_dict()
+        d["/_matrix/federation"] = TransportLayerServer(self.hs)
+        return d
+
    def prepare(self, reactor, clock, hs):
        mock_notifier = hs.get_notifier()
        self.on_new_event = mock_notifier.on_new_event
@ -192,7 +197,7 @@ class TypingNotificationsTestCase(unittest.HomeserverTestCase):
            )
        )

-        put_json = self.hs.get_http_client().put_json
+        put_json = self.hs.get_federation_http_client().put_json
        put_json.assert_called_once_with(
            "farm",
            path="/_matrix/federation/v1/send/1000000",
@ -270,7 +275,7 @@ class TypingNotificationsTestCase(unittest.HomeserverTestCase):

        self.on_new_event.assert_has_calls([call("typing_key", 1, rooms=[ROOM_ID])])

-        put_json = self.hs.get_http_client().put_json
+        put_json = self.hs.get_federation_http_client().put_json
        put_json.assert_called_once_with(
            "farm",
            path="/_matrix/federation/v1/send/1000000",
--- a/tests/http/federation/test_matrix_federation_agent.py
+++ b/tests/http/federation/test_matrix_federation_agent.py
@ -17,6 +17,7 @@ import logging
 from mock import Mock

 import treq
+from netaddr import IPSet
 from service_identity import VerificationError
 from zope.interface import implementer

@ -103,6 +104,7 @@ class MatrixFederationAgentTests(unittest.TestCase):
            reactor=self.reactor,
            tls_client_options_factory=self.tls_factory,
            user_agent="test-agent",  # Note that this is unused since _well_known_resolver is provided.
+            ip_blacklist=IPSet(),
            _srv_resolver=self.mock_resolver,
            _well_known_resolver=self.well_known_resolver,
        )
@ -736,6 +738,7 @@ class MatrixFederationAgentTests(unittest.TestCase):
            reactor=self.reactor,
            tls_client_options_factory=tls_factory,
            user_agent=b"test-agent",  # This is unused since _well_known_resolver is passed below.
+            ip_blacklist=IPSet(),
            _srv_resolver=self.mock_resolver,
            _well_known_resolver=WellKnownResolver(
                self.reactor,
--- a/tests/push/test_http.py
+++ b/tests/push/test_http.py
@ -49,7 +49,9 @@ class HTTPPusherTests(HomeserverTestCase):
        config = self.default_config()
        config["start_pushers"] = True

-        hs = self.setup_test_homeserver(config=config, proxied_http_client=m)
+        hs = self.setup_test_homeserver(
+            config=config, proxied_blacklisted_http_client=m
+        )

        return hs

--- a/tests/replication/_base.py
+++ b/tests/replication/_base.py
@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import Any, Callable, List, Optional, Tuple
+from typing import Any, Callable, Dict, List, Optional, Tuple

 import attr

@ -21,6 +21,7 @@ from twisted.internet.interfaces import IConsumer, IPullProducer, IReactorTime
 from twisted.internet.protocol import Protocol
 from twisted.internet.task import LoopingCall
 from twisted.web.http import HTTPChannel
+from twisted.web.resource import Resource

 from synapse.app.generic_worker import (
    GenericWorkerReplicationHandler,
@ -28,7 +29,7 @@ from synapse.app.generic_worker import (
 )
 from synapse.http.server import JsonResource
 from synapse.http.site import SynapseRequest, SynapseSite
-from synapse.replication.http import ReplicationRestResource, streams
+from synapse.replication.http import ReplicationRestResource
 from synapse.replication.tcp.handler import ReplicationCommandHandler
 from synapse.replication.tcp.protocol import ClientReplicationStreamProtocol
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
@ -54,10 +55,6 @@ class BaseStreamTestCase(unittest.HomeserverTestCase):
    if not hiredis:
        skip = "Requires hiredis"

-    servlets = [
-        streams.register_servlets,
-    ]
-
    def prepare(self, reactor, clock, hs):
        # build a replication server
        server_factory = ReplicationStreamProtocolFactory(hs)
@ -67,7 +64,7 @@ class BaseStreamTestCase(unittest.HomeserverTestCase):
        # Make a new HomeServer object for the worker
        self.reactor.lookups["testserv"] = "1.2.3.4"
        self.worker_hs = self.setup_test_homeserver(
-            http_client=None,
+            federation_http_client=None,
            homeserver_to_use=GenericWorkerServer,
            config=self._get_worker_hs_config(),
            reactor=self.reactor,
@ -88,6 +85,11 @@ class BaseStreamTestCase(unittest.HomeserverTestCase):
        self._client_transport = None
        self._server_transport = None

+    def create_resource_dict(self) -> Dict[str, Resource]:
+        d = super().create_resource_dict()
+        d["/_synapse/replication"] = ReplicationRestResource(self.hs)
+        return d
+
    def _get_worker_hs_config(self) -> dict:
        config = self.default_config()
        config["worker_app"] = "synapse.app.generic_worker"
@ -264,7 +266,7 @@ class BaseMultiWorkerStreamTestCase(unittest.HomeserverTestCase):
            worker_app: Type of worker, e.g. `synapse.app.federation_sender`.
            extra_config: Any extra config to use for this instances.
            **kwargs: Options that get passed to `self.setup_test_homeserver`,
-                useful to e.g. pass some mocks for things like `http_client`
+                useful to e.g. pass some mocks for things like `federation_http_client`

        Returns:
            The new worker HomeServer instance.
--- a/tests/replication/test_federation_sender_shard.py
+++ b/tests/replication/test_federation_sender_shard.py
@ -50,7 +50,7 @@ class FederationSenderTestCase(BaseMultiWorkerStreamTestCase):
        self.make_worker_hs(
            "synapse.app.federation_sender",
            {"send_federation": True},
-            http_client=mock_client,
+            federation_http_client=mock_client,
        )

        user = self.register_user("user", "pass")
@ -81,7 +81,7 @@ class FederationSenderTestCase(BaseMultiWorkerStreamTestCase):
                "worker_name": "sender1",
                "federation_sender_instances": ["sender1", "sender2"],
            },
-            http_client=mock_client1,
+            federation_http_client=mock_client1,
        )

        mock_client2 = Mock(spec=["put_json"])
@ -93,7 +93,7 @@ class FederationSenderTestCase(BaseMultiWorkerStreamTestCase):
                "worker_name": "sender2",
                "federation_sender_instances": ["sender1", "sender2"],
            },
-            http_client=mock_client2,
+            federation_http_client=mock_client2,
        )

        user = self.register_user("user2", "pass")
@ -144,7 +144,7 @@ class FederationSenderTestCase(BaseMultiWorkerStreamTestCase):
                "worker_name": "sender1",
                "federation_sender_instances": ["sender1", "sender2"],
            },
-            http_client=mock_client1,
+            federation_http_client=mock_client1,
        )

        mock_client2 = Mock(spec=["put_json"])
@ -156,7 +156,7 @@ class FederationSenderTestCase(BaseMultiWorkerStreamTestCase):
                "worker_name": "sender2",
                "federation_sender_instances": ["sender1", "sender2"],
            },
-            http_client=mock_client2,
+            federation_http_client=mock_client2,
        )

        user = self.register_user("user3", "pass")
--- a/tests/replication/test_pusher_shard.py
+++ b/tests/replication/test_pusher_shard.py
@ -98,7 +98,7 @@ class PusherShardTestCase(BaseMultiWorkerStreamTestCase):
        self.make_worker_hs(
            "synapse.app.pusher",
            {"start_pushers": True},
-            proxied_http_client=http_client_mock,
+            proxied_blacklisted_http_client=http_client_mock,
        )

        event_id = self._create_pusher_and_send_msg("user")
@ -133,7 +133,7 @@ class PusherShardTestCase(BaseMultiWorkerStreamTestCase):
                "worker_name": "pusher1",
                "pusher_instances": ["pusher1", "pusher2"],
            },
-            proxied_http_client=http_client_mock1,
+            proxied_blacklisted_http_client=http_client_mock1,
        )

        http_client_mock2 = Mock(spec_set=["post_json_get_json"])
@ -148,7 +148,7 @@ class PusherShardTestCase(BaseMultiWorkerStreamTestCase):
                "worker_name": "pusher2",
                "pusher_instances": ["pusher1", "pusher2"],
            },
-            proxied_http_client=http_client_mock2,
+            proxied_blacklisted_http_client=http_client_mock2,
        )

        # We choose a user name that we know should go to pusher1.
--- a/tests/rest/admin/test_admin.py
+++ b/tests/rest/admin/test_admin.py
@ -210,7 +210,7 @@ class QuarantineMediaTestCase(unittest.HomeserverTestCase):
        }
        config["media_storage_providers"] = [provider_config]

-        hs = self.setup_test_homeserver(config=config, http_client=client)
+        hs = self.setup_test_homeserver(config=config, federation_http_client=client)

        return hs

--- a/tests/rest/client/v1/test_presence.py
+++ b/tests/rest/client/v1/test_presence.py
@ -38,7 +38,7 @@ class PresenceTestCase(unittest.HomeserverTestCase):

        hs = self.setup_test_homeserver(
            "red",
-            http_client=None,
+            federation_http_client=None,
            federation_client=Mock(),
            presence_handler=presence_handler,
        )
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Richard van der Hoff	ed5172852a	Merge pull request #8858 from matrix-org/rav/sso_uia UIA: offer only available auth flows	2020-12-02 20:06:53 +00:00
Richard van der Hoff	f347f0cd58	remove unused FakeResponse (#8864 )	2020-12-02 18:58:25 +00:00
Richard van der Hoff	935732768c	newsfile	2020-12-02 18:54:15 +00:00
Richard van der Hoff	0bac276890	UIA: offer only available auth flows During user-interactive auth, do not offer password auth to users with no password, nor SSO auth to users with no SSO. Fixes #7559.	2020-12-02 18:54:15 +00:00
Richard van der Hoff	76469898ee	Factor out FakeResponse from test_oidc	2020-12-02 18:30:29 +00:00
Richard van der Hoff	7ea85302f3	fix up various test cases A few test cases were relying on being able to mount non-client servlets on the test resource. it's better to give them their own Resources.	2020-12-02 16:30:01 +00:00
Patrick Cloke	30fba62108	Apply an IP range blacklist to push and key revocation requests. (#8821 ) Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).	2020-12-02 11:09:24 -05:00
Erik Johnston	c5b6abd53d	Correctly handle unpersisted events when calculating auth chain difference. (#8827 ) We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as: 1. the next time a state event happens in the room the current state should correct itself; 2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and 3. we mostly use the state at an event to do important operations, which isn't affected by this.	2020-12-02 15:22:37 +00:00
Richard van der Hoff	693516e756	Add `create_resource_dict` method to HomeserverTestCase Rather than using a single JsonResource, construct a resource tree, as we do in the prod code, and allow testcases to add extra resources by overriding `create_resource_dict`.	2020-12-02 15:21:00 +00:00
Johanna Dorothea Reichmann	0fed46ebe5	Add missing prometheus rules for persisted events (#8802 ) The official dashboard uses data from these rules, but they were never added to the synapse-v2.rules. They are mentioned in this issue: https://github.com/matrix-org/synapse/issues/7917#issuecomment-661330409, but never got added to the rules. Adding them results in all graphs in the "Event persist rate" section to function as intended. Signed-off-by: Johanna Dorothea Reichmann <transcaffeine@finallycoffee.eu>	2020-12-02 15:18:41 +00:00
David Florness	c4675e1b24	Add additional validation for the admin register endpoint. (#8837 ) Raise a proper 400 error if the `mac` field is missing.	2020-12-02 10:01:15 -05:00
Patrick Cloke	e41720d85f	Minor changes to the CHANGES doc.	2020-12-02 09:17:42 -05:00
Patrick Cloke	c67af840aa	Minor fixes to changelog.	2020-12-02 09:03:12 -05:00
Patrick Cloke	53b12688dd	1.24.0rc1	2020-12-02 08:57:51 -05:00
				`@ -1 +0,0 @@`
				Simplify the way the `HomeServer` object caches its internal attributes.
				`@ -1 +0,0 @@`
				`Allow specification of the SAML IdP if the metadata returns multiple IdPs.`
				`@ -1 +0,0 @@`
				`Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru.`
				`@ -1 +0,0 @@`
				`Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher.`
				`@ -1 +0,0 @@`
				`Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0.`
				`@ -1 +0,0 @@`
				Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite.
				`@ -1 +0,0 @@`
				Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events.
				`@ -1 +0,0 @@`
				`Refactor test utilities for injecting HTTP requests.`
				`@ -1 +0,0 @@`
				`Consolidate logic between the OpenID Connect and SAML code.`