Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes

2019-11-20 15:17:10 +00:00 · 2019-11-20 15:17:10 +00:00 · 2c59eb368c
parent 6d1a3e2bdd 41e4566682
commit 2c59eb368c
66 changed files with 824 additions and 686 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -1,3 +1,77 @@
 Synapse 1.6.0rc1 (2019-11-20)
 =============================
 Features
 --------
 - Add federation support for cross-signing. ([\#5727](https://github.com/matrix-org/synapse/issues/5727))
 - Increase default room version from 4 to 5, thereby enforcing server key validity period checks. ([\#6220](https://github.com/matrix-org/synapse/issues/6220))
 - Add support for outbound http proxying via http_proxy/HTTPS_PROXY env vars. ([\#6238](https://github.com/matrix-org/synapse/issues/6238))
 - Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)). ([\#6301](https://github.com/matrix-org/synapse/issues/6301), [\#6310](https://github.com/matrix-org/synapse/issues/6310), [\#6340](https://github.com/matrix-org/synapse/issues/6340))
 Bugfixes
 --------
 - Fix LruCache callback deduplication for Python 3.8. Contributed by @V02460. ([\#6213](https://github.com/matrix-org/synapse/issues/6213))
 - Remove a room from a server's public rooms list on room upgrade. ([\#6232](https://github.com/matrix-org/synapse/issues/6232), [\#6235](https://github.com/matrix-org/synapse/issues/6235))
 - Delete keys from key backup when deleting backup versions. ([\#6253](https://github.com/matrix-org/synapse/issues/6253))
 - Make notification of cross-signing signatures work with workers. ([\#6254](https://github.com/matrix-org/synapse/issues/6254))
 - Fix exception when remote servers attempt to join a room that they're not allowed to join. ([\#6278](https://github.com/matrix-org/synapse/issues/6278))
 - Prevent errors from appearing on Synapse startup if `git` is not installed. ([\#6284](https://github.com/matrix-org/synapse/issues/6284))
 - Appservice requests will no longer contain a double slash prefix when the appservice url provided ends in a slash. ([\#6306](https://github.com/matrix-org/synapse/issues/6306))
 - Fix `/purge_room` admin API. ([\#6307](https://github.com/matrix-org/synapse/issues/6307))
 - Fix the `hidden` field in the `devices` table for SQLite versions prior to 3.23.0. ([\#6313](https://github.com/matrix-org/synapse/issues/6313))
 - Fix bug which casued rejected events to be persisted with the wrong room state. ([\#6320](https://github.com/matrix-org/synapse/issues/6320))
 - Fix bug where `rc_login` ratelimiting would prematurely kick in. ([\#6335](https://github.com/matrix-org/synapse/issues/6335))
 - Prevent the server taking a long time to start up when guest registration is enabled. ([\#6338](https://github.com/matrix-org/synapse/issues/6338))
 - Fix bug where upgrading a guest account to a full user would fail when account validity is enabled. ([\#6359](https://github.com/matrix-org/synapse/issues/6359))
 - Fix `to_device` stream ID getting reset every time Synapse restarts, which had the potential to cause unable to decrypt errors. ([\#6363](https://github.com/matrix-org/synapse/issues/6363))
 - Fix permission denied error when trying to generate a config file with the docker image. ([\#6389](https://github.com/matrix-org/synapse/issues/6389))
 Improved Documentation
 ----------------------
 - Contributor documentation now mentions script to run linters. ([\#6164](https://github.com/matrix-org/synapse/issues/6164))
 - Modify CAPTCHA_SETUP.md to update the terms `private key` and `public key` to `secret key` and `site key` respectively. Contributed by Yash Jipkate. ([\#6257](https://github.com/matrix-org/synapse/issues/6257))
 - Update `INSTALL.md` Email section to talk about `account_threepid_delegates`. ([\#6272](https://github.com/matrix-org/synapse/issues/6272))
 - Fix a small typo in `account_threepid_delegates` configuration option. ([\#6273](https://github.com/matrix-org/synapse/issues/6273))
 Internal Changes
 ----------------
 - Add a CI job to test the `synapse_port_db` script. ([\#6140](https://github.com/matrix-org/synapse/issues/6140), [\#6276](https://github.com/matrix-org/synapse/issues/6276))
 - Convert EventContext to an attrs. ([\#6218](https://github.com/matrix-org/synapse/issues/6218))
 - Move `persist_events` out from main data store. ([\#6240](https://github.com/matrix-org/synapse/issues/6240), [\#6300](https://github.com/matrix-org/synapse/issues/6300))
 - Reduce verbosity of user/room stats. ([\#6250](https://github.com/matrix-org/synapse/issues/6250))
 - Reduce impact of debug logging. ([\#6251](https://github.com/matrix-org/synapse/issues/6251))
 - Expose some homeserver functionality to spam checkers. ([\#6259](https://github.com/matrix-org/synapse/issues/6259))
 - Change cache descriptors to always return deferreds. ([\#6263](https://github.com/matrix-org/synapse/issues/6263), [\#6291](https://github.com/matrix-org/synapse/issues/6291))
 - Fix incorrect comment regarding the functionality of an `if` statement. ([\#6269](https://github.com/matrix-org/synapse/issues/6269))
 - Update CI to run `isort` over the `scripts` and `scripts-dev` directories. ([\#6270](https://github.com/matrix-org/synapse/issues/6270))
 - Replace every instance of `logger.warn` method with `logger.warning` as the former is deprecated. ([\#6271](https://github.com/matrix-org/synapse/issues/6271), [\#6314](https://github.com/matrix-org/synapse/issues/6314))
 - Port replication http server endpoints to async/await. ([\#6274](https://github.com/matrix-org/synapse/issues/6274))
 - Port room rest handlers to async/await. ([\#6275](https://github.com/matrix-org/synapse/issues/6275))
 - Remove redundant CLI parameters on CI's `flake8` step. ([\#6277](https://github.com/matrix-org/synapse/issues/6277))
 - Port `federation_server.py` to async/await. ([\#6279](https://github.com/matrix-org/synapse/issues/6279))
 - Port receipt and read markers to async/wait. ([\#6280](https://github.com/matrix-org/synapse/issues/6280))
 - Split out state storage into separate data store. ([\#6294](https://github.com/matrix-org/synapse/issues/6294), [\#6295](https://github.com/matrix-org/synapse/issues/6295))
 - Refactor EventContext for clarity. ([\#6298](https://github.com/matrix-org/synapse/issues/6298))
 - Update the version of black used to 19.10b0. ([\#6304](https://github.com/matrix-org/synapse/issues/6304))
 - Add some documentation about worker replication. ([\#6305](https://github.com/matrix-org/synapse/issues/6305))
 - Move admin endpoints into separate files. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6308](https://github.com/matrix-org/synapse/issues/6308))
 - Document the use of `lint.sh` for code style enforcement & extend it to run on specified paths only. ([\#6312](https://github.com/matrix-org/synapse/issues/6312))
 - Add optional python dependencies and dependant binary libraries to snapcraft packaging. ([\#6317](https://github.com/matrix-org/synapse/issues/6317))
 - Remove the dependency on psutil and replace functionality with the stdlib `resource` module. ([\#6318](https://github.com/matrix-org/synapse/issues/6318), [\#6336](https://github.com/matrix-org/synapse/issues/6336))
 - Improve documentation for EventContext fields. ([\#6319](https://github.com/matrix-org/synapse/issues/6319))
 - Add some checks that we aren't using state from rejected events. ([\#6330](https://github.com/matrix-org/synapse/issues/6330))
 - Add continuous integration for python 3.8. ([\#6341](https://github.com/matrix-org/synapse/issues/6341))
 - Correct spacing/case of various instances of the word "homeserver". ([\#6357](https://github.com/matrix-org/synapse/issues/6357))
 - Temporarily blacklist the failing unit test PurgeRoomTestCase.test_purge_room. ([\#6361](https://github.com/matrix-org/synapse/issues/6361))
 Synapse 1.5.1 (2019-11-06)
 ==========================
--- a/changelog.d/5727.feature
+++ b/changelog.d/5727.feature
@ -1 +0,0 @@
 Add federation support for cross-signing.
--- a/changelog.d/6140.misc
+++ b/changelog.d/6140.misc
@ -1 +0,0 @@
 Add a CI job to test the `synapse_port_db` script.
--- a/changelog.d/6164.doc
+++ b/changelog.d/6164.doc
@ -1 +0,0 @@
 Contributor documentation now mentions script to run linters.
--- a/changelog.d/6213.bugfix
+++ b/changelog.d/6213.bugfix
@ -1 +0,0 @@
 Fix LruCache callback deduplication for Python 3.8. Contributed by @V02460.
--- a/changelog.d/6218.misc
+++ b/changelog.d/6218.misc
@ -1 +0,0 @@
 Convert EventContext to an attrs.
--- a/changelog.d/6220.feature
+++ b/changelog.d/6220.feature
@ -1 +0,0 @@
 Increase default room version from 4 to 5, thereby enforcing server key validity period checks.
--- a/changelog.d/6232.bugfix
+++ b/changelog.d/6232.bugfix
@ -1 +0,0 @@
 Remove a room from a server's public rooms list on room upgrade.
--- a/changelog.d/6235.bugfix
+++ b/changelog.d/6235.bugfix
@ -1 +0,0 @@
 Remove a room from a server's public rooms list on room upgrade.
--- a/changelog.d/6238.feature
+++ b/changelog.d/6238.feature
@ -1 +0,0 @@
 Add support for outbound http proxying via http_proxy/HTTPS_PROXY env vars.
--- a/changelog.d/6240.misc
+++ b/changelog.d/6240.misc
@ -1 +0,0 @@
 Move `persist_events` out from main data store.
--- a/changelog.d/6250.misc
+++ b/changelog.d/6250.misc
@ -1 +0,0 @@
 Reduce verbosity of user/room stats.
--- a/changelog.d/6251.misc
+++ b/changelog.d/6251.misc
@ -1 +0,0 @@
 Reduce impact of debug logging.
--- a/changelog.d/6253.bugfix
+++ b/changelog.d/6253.bugfix
@ -1 +0,0 @@
 Delete keys from key backup when deleting backup versions.
--- a/changelog.d/6254.bugfix
+++ b/changelog.d/6254.bugfix
@ -1 +0,0 @@
 Make notification of cross-signing signatures work with workers.
--- a/changelog.d/6257.doc
+++ b/changelog.d/6257.doc
@ -1 +0,0 @@
 Modify CAPTCHA_SETUP.md to update the terms `private key` and `public key` to `secret key` and `site key` respectively. Contributed by Yash Jipkate.
--- a/changelog.d/6259.misc
+++ b/changelog.d/6259.misc
@ -1 +0,0 @@
 Expose some homeserver functionality to spam checkers.
--- a/changelog.d/6263.misc
+++ b/changelog.d/6263.misc
@ -1 +0,0 @@
 Change cache descriptors to always return deferreds.
--- a/changelog.d/6269.misc
+++ b/changelog.d/6269.misc
@ -1 +0,0 @@
 Fix incorrect comment regarding the functionality of an `if` statement.
--- a/changelog.d/6270.misc
+++ b/changelog.d/6270.misc
@ -1 +0,0 @@
 Update CI to run `isort` over the `scripts` and `scripts-dev` directories.
--- a/changelog.d/6271.misc
+++ b/changelog.d/6271.misc
@ -1 +0,0 @@
 Replace every instance of `logger.warn` method with `logger.warning` as the former is deprecated.
--- a/changelog.d/6272.doc
+++ b/changelog.d/6272.doc
@ -1 +0,0 @@
 Update `INSTALL.md` Email section to talk about `account_threepid_delegates`.
--- a/changelog.d/6273.doc
+++ b/changelog.d/6273.doc
@ -1 +0,0 @@
 Fix a small typo in `account_threepid_delegates` configuration option.
--- a/changelog.d/6274.misc
+++ b/changelog.d/6274.misc
@ -1 +0,0 @@
 Port replication http server endpoints to async/await.
--- a/changelog.d/6275.misc
+++ b/changelog.d/6275.misc
@ -1 +0,0 @@
 Port room rest handlers to async/await.
--- a/changelog.d/6276.misc
+++ b/changelog.d/6276.misc
@ -1 +0,0 @@
 Add a CI job to test the `synapse_port_db` script.
--- a/changelog.d/6277.misc
+++ b/changelog.d/6277.misc
@ -1 +0,0 @@
 Remove redundant CLI parameters on CI's `flake8` step.
--- a/changelog.d/6278.bugfix
+++ b/changelog.d/6278.bugfix
@ -1 +0,0 @@
 Fix exception when remote servers attempt to join a room that they're not allowed to join.
--- a/changelog.d/6279.misc
+++ b/changelog.d/6279.misc
@ -1 +0,0 @@
 Port `federation_server.py` to async/await.
--- a/changelog.d/6280.misc
+++ b/changelog.d/6280.misc
@ -1 +0,0 @@
 Port receipt and read markers to async/wait.
--- a/changelog.d/6284.bugfix
+++ b/changelog.d/6284.bugfix
@ -1 +0,0 @@
 Prevent errors from appearing on Synapse startup if `git` is not installed.
--- a/changelog.d/6291.misc
+++ b/changelog.d/6291.misc
@ -1 +0,0 @@
 Change cache descriptors to always return deferreds.
--- a/changelog.d/6294.misc
+++ b/changelog.d/6294.misc
@ -1 +0,0 @@
 Split out state storage into separate data store.
--- a/changelog.d/6295.misc
+++ b/changelog.d/6295.misc
@ -1 +0,0 @@
 Split out state storage into separate data store.
--- a/changelog.d/6298.misc
+++ b/changelog.d/6298.misc
@ -1 +0,0 @@
 Refactor EventContext for clarity.
--- a/changelog.d/6300.misc
+++ b/changelog.d/6300.misc
@ -1 +0,0 @@
 Move `persist_events` out from main data store.
--- a/changelog.d/6301.feature
+++ b/changelog.d/6301.feature
@ -1 +0,0 @@
 Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)).
--- a/changelog.d/6304.misc
+++ b/changelog.d/6304.misc
@ -1 +0,0 @@
 Update the version of black used to 19.10b0.
--- a/changelog.d/6305.misc
+++ b/changelog.d/6305.misc
@ -1 +0,0 @@
 Add some documentation about worker replication.
--- a/changelog.d/6306.bugfix
+++ b/changelog.d/6306.bugfix
@ -1 +0,0 @@
 Appservice requests will no longer contain a double slash prefix when the appservice url provided ends in a slash.
--- a/changelog.d/6307.bugfix
+++ b/changelog.d/6307.bugfix
@ -1 +0,0 @@
 Fix `/purge_room` admin API.
--- a/changelog.d/6310.feature
+++ b/changelog.d/6310.feature
@ -1 +0,0 @@
 Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)).
--- a/changelog.d/6312.misc
+++ b/changelog.d/6312.misc
@ -1 +0,0 @@
 Document the use of `lint.sh` for code style enforcement & extend it to run on specified paths only.
--- a/changelog.d/6313.bugfix
+++ b/changelog.d/6313.bugfix
@ -1 +0,0 @@
 Fix the `hidden` field in the `devices` table for SQLite versions prior to 3.23.0.
--- a/changelog.d/6314.misc
+++ b/changelog.d/6314.misc
@ -1 +0,0 @@
 Replace every instance of `logger.warn` method with `logger.warning` as the former is deprecated.
--- a/changelog.d/6317.misc
+++ b/changelog.d/6317.misc
@ -1 +0,0 @@
 Add optional python dependencies and dependant binary libraries to snapcraft packaging.
--- a/changelog.d/6318.misc
+++ b/changelog.d/6318.misc
@ -1 +0,0 @@
 Remove the dependency on psutil and replace functionality with the stdlib `resource` module.
--- a/changelog.d/6319.misc
+++ b/changelog.d/6319.misc
@ -1 +0,0 @@
 Improve documentation for EventContext fields.
--- a/changelog.d/6320.bugfix
+++ b/changelog.d/6320.bugfix
@ -1 +0,0 @@
 Fix bug which casued rejected events to be persisted with the wrong room state.
--- a/changelog.d/6330.misc
+++ b/changelog.d/6330.misc
@ -1 +0,0 @@
 Add some checks that we aren't using state from rejected events.
--- a/changelog.d/6336.misc
+++ b/changelog.d/6336.misc
@ -1 +0,0 @@
 Remove the dependency on psutil and replace functionality with the stdlib `resource` module.
--- a/changelog.d/6338.bugfix
+++ b/changelog.d/6338.bugfix
@ -1 +0,0 @@
 Prevent the server taking a long time to start up when guest registration is enabled.
--- a/changelog.d/6340.feature
+++ b/changelog.d/6340.feature
@ -1 +0,0 @@
 Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)).
--- a/changelog.d/6341.misc
+++ b/changelog.d/6341.misc
@ -1 +0,0 @@
 Add continuous integration for python 3.8.
--- a/changelog.d/6357.misc
+++ b/changelog.d/6357.misc
@ -1 +0,0 @@
 Correct spacing/case of various instances of the word "homeserver".
--- a/changelog.d/6359.bugfix
+++ b/changelog.d/6359.bugfix
@ -1 +0,0 @@
 Fix bug where upgrading a guest account to a full user would fail when account validity is enabled.
--- a/changelog.d/6361.misc
+++ b/changelog.d/6361.misc
@ -1 +0,0 @@
 Temporarily blacklist the failing unit test PurgeRoomTestCase.test_purge_room.
--- a/changelog.d/6363.bugfix
+++ b/changelog.d/6363.bugfix
@ -1 +0,0 @@
 Fix `to_device` stream ID getting reset every time Synapse restarts, which had the potential to cause unable to decrypt errors.
--- a/docker/start.py
+++ b/docker/start.py
@ -169,11 +169,11 @@ def run_generate_config(environ, ownership):
    # log("running %s" % (args, ))
    if ownership is not None:
        args = ["su-exec", ownership] + args
        os.execv("/sbin/su-exec", args)
        # make sure that synapse has perms to write to the data dir.
        subprocess.check_output(["chown", ownership, data_dir])
        args = ["su-exec", ownership] + args
        os.execv("/sbin/su-exec", args)
    else:
        os.execv("/usr/local/bin/python", args)
--- a/synapse/init.py
+++ b/synapse/init.py
@ -36,7 +36,7 @@ try:
 except ImportError:
    pass
-__version__ = "1.5.1"
+__version__ = "1.6.0rc1"
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
--- a/synapse/handlers/auth.py
+++ b/synapse/handlers/auth.py
@ -102,8 +102,9 @@ class AuthHandler(BaseHandler):
                        login_types.append(t)
        self._supported_login_types = login_types
-        self._account_ratelimiter = Ratelimiter()
+        # Ratelimiter for failed auth during UIA. Uses same ratelimit config
-        self._failed_attempts_ratelimiter = Ratelimiter()
+        # as per `rc_login.failed_attempts`.
        self._failed_uia_attempts_ratelimiter = Ratelimiter()
        self._clock = self.hs.get_clock()
@ -133,12 +134,38 @@ class AuthHandler(BaseHandler):
            AuthError if the client has completed a login flow, and it gives
                a different user to `requester`
            LimitExceededError if the ratelimiter's failed request count for this
                user is too high to proceed
        """
        user_id = requester.user.to_string()
        # Check if we should be ratelimited due to too many previous failed attempts
        self._failed_uia_attempts_ratelimiter.ratelimit(
            user_id,
            time_now_s=self._clock.time(),
            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
            update=False,
        )
        # build a list of supported flows
        flows = [[login_type] for login_type in self._supported_login_types]
        try:
            result, params, _ = yield self.check_auth(flows, request_body, clientip)
        except LoginError:
            # Update the ratelimite to say we failed (`can_do_action` doesn't raise).
            self._failed_uia_attempts_ratelimiter.can_do_action(
                user_id,
                time_now_s=self._clock.time(),
                rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
                burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
                update=True,
            )
            raise
        # find the completed login type
        for login_type in self._supported_login_types:
@ -501,11 +528,8 @@ class AuthHandler(BaseHandler):
            multiple matches
        Raises:
            LimitExceededError if the ratelimiter's login requests count for this
                user is too high too proceed.
            UserDeactivatedError if a user is found but is deactivated.
        """
        self.ratelimit_login_per_account(user_id)
        res = yield self._find_user_id_and_pwd_hash(user_id)
        if res is not None:
            return res[0]
@ -572,8 +596,6 @@ class AuthHandler(BaseHandler):
            StoreError if there was a problem accessing the database
            SynapseError if there was a problem with the request
            LoginError if there was an authentication problem.
            LimitExceededError if the ratelimiter's login requests count for this
                user is too high too proceed.
        """
        if username.startswith("@"):
@ -581,8 +603,6 @@ class AuthHandler(BaseHandler):
        else:
            qualified_user_id = UserID(username, self.hs.hostname).to_string()
        self.ratelimit_login_per_account(qualified_user_id)
        login_type = login_submission.get("type")
        known_login_type = False
@ -650,15 +670,6 @@ class AuthHandler(BaseHandler):
        if not known_login_type:
            raise SynapseError(400, "Unknown login type %s" % login_type)
        # unknown username or invalid password.
        self._failed_attempts_ratelimiter.ratelimit(
            qualified_user_id.lower(),
            time_now_s=self._clock.time(),
            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
            update=True,
        )
        # We raise a 403 here, but note that if we're doing user-interactive
        # login, it turns all LoginErrors into a 401 anyway.
        raise LoginError(403, "Invalid password", errcode=Codes.FORBIDDEN)
@ -710,10 +721,6 @@ class AuthHandler(BaseHandler):
        Returns:
            Deferred[unicode] the canonical_user_id, or Deferred[None] if
                unknown user/bad password
        Raises:
            LimitExceededError if the ratelimiter's login requests count for this
                user is too high too proceed.
        """
        lookupres = yield self._find_user_id_and_pwd_hash(user_id)
        if not lookupres:
@ -742,7 +749,7 @@ class AuthHandler(BaseHandler):
            auth_api.validate_macaroon(macaroon, "login", user_id)
        except Exception:
            raise AuthError(403, "Invalid token", errcode=Codes.FORBIDDEN)
-        self.ratelimit_login_per_account(user_id)
+
        yield self.auth.check_auth_blocking(user_id)
        return user_id
@ -912,35 +919,6 @@ class AuthHandler(BaseHandler):
        else:
            return defer.succeed(False)
    def ratelimit_login_per_account(self, user_id):
        """Checks whether the process must be stopped because of ratelimiting.
        Checks against two ratelimiters: the generic one for login attempts per
        account and the one specific to failed attempts.
        Args:
            user_id (unicode): complete @user:id
        Raises:
            LimitExceededError if one of the ratelimiters' login requests count
                for this user is too high too proceed.
        """
        self._failed_attempts_ratelimiter.ratelimit(
            user_id.lower(),
            time_now_s=self._clock.time(),
            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
            update=False,
        )
        self._account_ratelimiter.ratelimit(
            user_id.lower(),
            time_now_s=self._clock.time(),
            rate_hz=self.hs.config.rc_login_account.per_second,
            burst_count=self.hs.config.rc_login_account.burst_count,
            update=True,
        )
@attr.s
 class MacaroonGenerator(object):
--- a/synapse/rest/admin/init.py
+++ b/synapse/rest/admin/init.py
@ -14,62 +14,39 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import hashlib
 import hmac
 import logging
 import platform
 import re
 from six import text_type
 from six.moves import http_client
 import synapse
 from synapse.api.constants import Membership, UserTypes
 from synapse.api.errors import Codes, NotFoundError, SynapseError
 from synapse.http.server import JsonResource
-from synapse.http.servlet import (
+from synapse.http.servlet import RestServlet, parse_json_object_from_request
    RestServlet,
    assert_params_in_dict,
    parse_integer,
    parse_json_object_from_request,
    parse_string,
 )
 from synapse.rest.admin._base import (
    assert_requester_is_admin,
    assert_user_is_admin,
    historical_admin_path_patterns,
 )
 from synapse.rest.admin.groups import DeleteGroupAdminRestServlet
 from synapse.rest.admin.media import ListMediaInRoom, register_servlets_for_media_repo
 from synapse.rest.admin.purge_room_servlet import PurgeRoomServlet
 from synapse.rest.admin.rooms import ShutdownRoomRestServlet
 from synapse.rest.admin.server_notice_servlet import SendServerNoticeServlet
-from synapse.rest.admin.users import UserAdminServlet
+from synapse.rest.admin.users import (
-from synapse.types import UserID, create_requester
+    AccountValidityRenewServlet,
-from synapse.util.async_helpers import maybe_awaitable
+    DeactivateAccountRestServlet,
    GetUsersPaginatedRestServlet,
    ResetPasswordRestServlet,
    SearchUsersRestServlet,
    UserAdminServlet,
    UserRegisterServlet,
    UsersRestServlet,
    WhoisRestServlet,
 )
 from synapse.util.versionstring import get_version_string
 logger = logging.getLogger(__name__)
 class UsersRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/users/(?P<user_id>[^/]*)$")
    def __init__(self, hs):
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, user_id):
        target_user = UserID.from_string(user_id)
        await assert_requester_is_admin(self.auth, request)
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only users a local user")
        ret = await self.handlers.admin_handler.get_users()
        return 200, ret
 class VersionServlet(RestServlet):
    PATTERNS = (re.compile("^/_synapse/admin/v1/server_version$"),)
@ -83,159 +60,6 @@ class VersionServlet(RestServlet):
        return 200, self.res
 class UserRegisterServlet(RestServlet):
    """
    Attributes:
         NONCE_TIMEOUT (int): Seconds until a generated nonce won't be accepted
         nonces (dict[str, int]): The nonces that we will accept. A dict of
             nonce to the time it was generated, in int seconds.
    """
    PATTERNS = historical_admin_path_patterns("/register")
    NONCE_TIMEOUT = 60
    def __init__(self, hs):
        self.handlers = hs.get_handlers()
        self.reactor = hs.get_reactor()
        self.nonces = {}
        self.hs = hs
    def _clear_old_nonces(self):
        """
        Clear out old nonces that are older than NONCE_TIMEOUT.
        """
        now = int(self.reactor.seconds())
        for k, v in list(self.nonces.items()):
            if now - v > self.NONCE_TIMEOUT:
                del self.nonces[k]
    def on_GET(self, request):
        """
        Generate a new nonce.
        """
        self._clear_old_nonces()
        nonce = self.hs.get_secrets().token_hex(64)
        self.nonces[nonce] = int(self.reactor.seconds())
        return 200, {"nonce": nonce}
    async def on_POST(self, request):
        self._clear_old_nonces()
        if not self.hs.config.registration_shared_secret:
            raise SynapseError(400, "Shared secret registration is not enabled")
        body = parse_json_object_from_request(request)
        if "nonce" not in body:
            raise SynapseError(400, "nonce must be specified", errcode=Codes.BAD_JSON)
        nonce = body["nonce"]
        if nonce not in self.nonces:
            raise SynapseError(400, "unrecognised nonce")
        # Delete the nonce, so it can't be reused, even if it's invalid
        del self.nonces[nonce]
        if "username" not in body:
            raise SynapseError(
                400, "username must be specified", errcode=Codes.BAD_JSON
            )
        else:
            if (
                not isinstance(body["username"], text_type)
                or len(body["username"]) > 512
            ):
                raise SynapseError(400, "Invalid username")
            username = body["username"].encode("utf-8")
            if b"\x00" in username:
                raise SynapseError(400, "Invalid username")
        if "password" not in body:
            raise SynapseError(
                400, "password must be specified", errcode=Codes.BAD_JSON
            )
        else:
            if (
                not isinstance(body["password"], text_type)
                or len(body["password"]) > 512
            ):
                raise SynapseError(400, "Invalid password")
            password = body["password"].encode("utf-8")
            if b"\x00" in password:
                raise SynapseError(400, "Invalid password")
        admin = body.get("admin", None)
        user_type = body.get("user_type", None)
        if user_type is not None and user_type not in UserTypes.ALL_USER_TYPES:
            raise SynapseError(400, "Invalid user type")
        got_mac = body["mac"]
        want_mac = hmac.new(
            key=self.hs.config.registration_shared_secret.encode(),
            digestmod=hashlib.sha1,
        )
        want_mac.update(nonce.encode("utf8"))
        want_mac.update(b"\x00")
        want_mac.update(username)
        want_mac.update(b"\x00")
        want_mac.update(password)
        want_mac.update(b"\x00")
        want_mac.update(b"admin" if admin else b"notadmin")
        if user_type:
            want_mac.update(b"\x00")
            want_mac.update(user_type.encode("utf8"))
        want_mac = want_mac.hexdigest()
        if not hmac.compare_digest(want_mac.encode("ascii"), got_mac.encode("ascii")):
            raise SynapseError(403, "HMAC incorrect")
        # Reuse the parts of RegisterRestServlet to reduce code duplication
        from synapse.rest.client.v2_alpha.register import RegisterRestServlet
        register = RegisterRestServlet(self.hs)
        user_id = await register.registration_handler.register_user(
            localpart=body["username"].lower(),
            password=body["password"],
            admin=bool(admin),
            user_type=user_type,
        )
        result = await register._create_registration_details(user_id, body)
        return 200, result
 class WhoisRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/whois/(?P<user_id>[^/]*)")
    def __init__(self, hs):
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, user_id):
        target_user = UserID.from_string(user_id)
        requester = await self.auth.get_user_by_req(request)
        auth_user = requester.user
        if target_user != auth_user:
            await assert_user_is_admin(self.auth, auth_user)
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only whois a local user")
        ret = await self.handlers.admin_handler.get_whois(target_user)
        return 200, ret
 class PurgeHistoryRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns(
        "/purge_history/(?P<room_id>[^/]*)(/(?P<event_id>[^/]+))?"
@ -342,369 +166,6 @@ class PurgeHistoryStatusRestServlet(RestServlet):
        return 200, purge_status.asdict()
 class DeactivateAccountRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/deactivate/(?P<target_user_id>[^/]*)")
    def __init__(self, hs):
        self._deactivate_account_handler = hs.get_deactivate_account_handler()
        self.auth = hs.get_auth()
    async def on_POST(self, request, target_user_id):
        await assert_requester_is_admin(self.auth, request)
        body = parse_json_object_from_request(request, allow_empty_body=True)
        erase = body.get("erase", False)
        if not isinstance(erase, bool):
            raise SynapseError(
                http_client.BAD_REQUEST,
                "Param 'erase' must be a boolean, if given",
                Codes.BAD_JSON,
            )
        UserID.from_string(target_user_id)
        result = await self._deactivate_account_handler.deactivate_account(
            target_user_id, erase
        )
        if result:
            id_server_unbind_result = "success"
        else:
            id_server_unbind_result = "no-support"
        return 200, {"id_server_unbind_result": id_server_unbind_result}
 class ShutdownRoomRestServlet(RestServlet):
    """Shuts down a room by removing all local users from the room and blocking
    all future invites and joins to the room. Any local aliases will be repointed
    to a new room created by `new_room_user_id` and kicked users will be auto
    joined to the new room.
    """
    PATTERNS = historical_admin_path_patterns("/shutdown_room/(?P<room_id>[^/]+)")
    DEFAULT_MESSAGE = (
        "Sharing illegal content on this server is not permitted and rooms in"
        " violation will be blocked."
    )
    def __init__(self, hs):
        self.hs = hs
        self.store = hs.get_datastore()
        self.state = hs.get_state_handler()
        self._room_creation_handler = hs.get_room_creation_handler()
        self.event_creation_handler = hs.get_event_creation_handler()
        self.room_member_handler = hs.get_room_member_handler()
        self.auth = hs.get_auth()
    async def on_POST(self, request, room_id):
        requester = await self.auth.get_user_by_req(request)
        await assert_user_is_admin(self.auth, requester.user)
        content = parse_json_object_from_request(request)
        assert_params_in_dict(content, ["new_room_user_id"])
        new_room_user_id = content["new_room_user_id"]
        room_creator_requester = create_requester(new_room_user_id)
        message = content.get("message", self.DEFAULT_MESSAGE)
        room_name = content.get("room_name", "Content Violation Notification")
        info = await self._room_creation_handler.create_room(
            room_creator_requester,
            config={
                "preset": "public_chat",
                "name": room_name,
                "power_level_content_override": {"users_default": -10},
            },
            ratelimit=False,
        )
        new_room_id = info["room_id"]
        requester_user_id = requester.user.to_string()
        logger.info(
            "Shutting down room %r, joining to new room: %r", room_id, new_room_id
        )
        # This will work even if the room is already blocked, but that is
        # desirable in case the first attempt at blocking the room failed below.
        await self.store.block_room(room_id, requester_user_id)
        users = await self.state.get_current_users_in_room(room_id)
        kicked_users = []
        failed_to_kick_users = []
        for user_id in users:
            if not self.hs.is_mine_id(user_id):
                continue
            logger.info("Kicking %r from %r...", user_id, room_id)
            try:
                target_requester = create_requester(user_id)
                await self.room_member_handler.update_membership(
                    requester=target_requester,
                    target=target_requester.user,
                    room_id=room_id,
                    action=Membership.LEAVE,
                    content={},
                    ratelimit=False,
                    require_consent=False,
                )
                await self.room_member_handler.forget(target_requester.user, room_id)
                await self.room_member_handler.update_membership(
                    requester=target_requester,
                    target=target_requester.user,
                    room_id=new_room_id,
                    action=Membership.JOIN,
                    content={},
                    ratelimit=False,
                    require_consent=False,
                )
                kicked_users.append(user_id)
            except Exception:
                logger.exception(
                    "Failed to leave old room and join new room for %r", user_id
                )
                failed_to_kick_users.append(user_id)
        await self.event_creation_handler.create_and_send_nonmember_event(
            room_creator_requester,
            {
                "type": "m.room.message",
                "content": {"body": message, "msgtype": "m.text"},
                "room_id": new_room_id,
                "sender": new_room_user_id,
            },
            ratelimit=False,
        )
        aliases_for_room = await maybe_awaitable(
            self.store.get_aliases_for_room(room_id)
        )
        await self.store.update_aliases_for_room(
            room_id, new_room_id, requester_user_id
        )
        return (
            200,
            {
                "kicked_users": kicked_users,
                "failed_to_kick_users": failed_to_kick_users,
                "local_aliases": aliases_for_room,
                "new_room_id": new_room_id,
            },
        )
 class ResetPasswordRestServlet(RestServlet):
    """Post request to allow an administrator reset password for a user.
    This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/reset_password/
            @user:to_reset_password?access_token=admin_access_token
        JsonBodyToSend:
            {
                "new_password": "secret"
            }
        Returns:
            200 OK with empty object if success otherwise an error.
        """
    PATTERNS = historical_admin_path_patterns(
        "/reset_password/(?P<target_user_id>[^/]*)"
    )
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self.hs = hs
        self.auth = hs.get_auth()
        self._set_password_handler = hs.get_set_password_handler()
    async def on_POST(self, request, target_user_id):
        """Post request to allow an administrator reset password for a user.
        This needs user to have administrator access in Synapse.
        """
        requester = await self.auth.get_user_by_req(request)
        await assert_user_is_admin(self.auth, requester.user)
        UserID.from_string(target_user_id)
        params = parse_json_object_from_request(request)
        assert_params_in_dict(params, ["new_password"])
        new_password = params["new_password"]
        await self._set_password_handler.set_password(
            target_user_id, new_password, requester
        )
        return 200, {}
 class GetUsersPaginatedRestServlet(RestServlet):
    """Get request to get specific number of users from Synapse.
    This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/users_paginate/
            @admin:user?access_token=admin_access_token&start=0&limit=10
        Returns:
            200 OK with json object {list[dict[str, Any]], count} or empty object.
        """
    PATTERNS = historical_admin_path_patterns(
        "/users_paginate/(?P<target_user_id>[^/]*)"
    )
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, target_user_id):
        """Get request to get specific number of users from Synapse.
        This needs user to have administrator access in Synapse.
        """
        await assert_requester_is_admin(self.auth, request)
        target_user = UserID.from_string(target_user_id)
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only users a local user")
        order = "name"  # order by name in user table
        start = parse_integer(request, "start", required=True)
        limit = parse_integer(request, "limit", required=True)
        logger.info("limit: %s, start: %s", limit, start)
        ret = await self.handlers.admin_handler.get_users_paginate(order, start, limit)
        return 200, ret
    async def on_POST(self, request, target_user_id):
        """Post request to get specific number of users from Synapse..
        This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/users_paginate/
            @admin:user?access_token=admin_access_token
        JsonBodyToSend:
            {
                "start": "0",
                "limit": "10
            }
        Returns:
            200 OK with json object {list[dict[str, Any]], count} or empty object.
        """
        await assert_requester_is_admin(self.auth, request)
        UserID.from_string(target_user_id)
        order = "name"  # order by name in user table
        params = parse_json_object_from_request(request)
        assert_params_in_dict(params, ["limit", "start"])
        limit = params["limit"]
        start = params["start"]
        logger.info("limit: %s, start: %s", limit, start)
        ret = await self.handlers.admin_handler.get_users_paginate(order, start, limit)
        return 200, ret
 class SearchUsersRestServlet(RestServlet):
    """Get request to search user table for specific users according to
    search term.
    This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/search_users/
            @admin:user?access_token=admin_access_token&term=alice
        Returns:
            200 OK with json object {list[dict[str, Any]], count} or empty object.
    """
    PATTERNS = historical_admin_path_patterns("/search_users/(?P<target_user_id>[^/]*)")
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, target_user_id):
        """Get request to search user table for specific users according to
        search term.
        This needs user to have a administrator access in Synapse.
        """
        await assert_requester_is_admin(self.auth, request)
        target_user = UserID.from_string(target_user_id)
        # To allow all users to get the users list
        # if not is_admin and target_user != auth_user:
        #     raise AuthError(403, "You are not a server admin")
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only users a local user")
        term = parse_string(request, "term", required=True)
        logger.info("term: %s ", term)
        ret = await self.handlers.admin_handler.search_users(term)
        return 200, ret
 class DeleteGroupAdminRestServlet(RestServlet):
    """Allows deleting of local groups
    """
    PATTERNS = historical_admin_path_patterns("/delete_group/(?P<group_id>[^/]*)")
    def __init__(self, hs):
        self.group_server = hs.get_groups_server_handler()
        self.is_mine_id = hs.is_mine_id
        self.auth = hs.get_auth()
    async def on_POST(self, request, group_id):
        requester = await self.auth.get_user_by_req(request)
        await assert_user_is_admin(self.auth, requester.user)
        if not self.is_mine_id(group_id):
            raise SynapseError(400, "Can only delete local groups")
        await self.group_server.delete_group(group_id, requester.user.to_string())
        return 200, {}
 class AccountValidityRenewServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/account_validity/validity$")
    def __init__(self, hs):
        """
        Args:
            hs (synapse.server.HomeServer): server
        """
        self.hs = hs
        self.account_activity_handler = hs.get_account_validity_handler()
        self.auth = hs.get_auth()
    async def on_POST(self, request):
        await assert_requester_is_admin(self.auth, request)
        body = parse_json_object_from_request(request)
        if "user_id" not in body:
            raise SynapseError(400, "Missing property 'user_id' in the request body")
        expiration_ts = await self.account_activity_handler.renew_account_for_user(
            body["user_id"],
            body.get("expiration_ts"),
            not body.get("enable_renewal_emails", True),
        )
        res = {"expiration_ts": expiration_ts}
        return 200, res
 ########################################################################################
 #
 # please don't add more servlets here: this file is already long and unwieldy. Put
--- a/synapse/rest/admin/groups.py
+++ b/synapse/rest/admin/groups.py
@ -0,0 +1,46 @@
 # -*- coding: utf-8 -*-
 # Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
 from synapse.api.errors import SynapseError
 from synapse.http.servlet import RestServlet
 from synapse.rest.admin._base import (
    assert_user_is_admin,
    historical_admin_path_patterns,
 )
 logger = logging.getLogger(__name__)
 class DeleteGroupAdminRestServlet(RestServlet):
    """Allows deleting of local groups
    """
    PATTERNS = historical_admin_path_patterns("/delete_group/(?P<group_id>[^/]*)")
    def __init__(self, hs):
        self.group_server = hs.get_groups_server_handler()
        self.is_mine_id = hs.is_mine_id
        self.auth = hs.get_auth()
    async def on_POST(self, request, group_id):
        requester = await self.auth.get_user_by_req(request)
        await assert_user_is_admin(self.auth, requester.user)
        if not self.is_mine_id(group_id):
            raise SynapseError(400, "Can only delete local groups")
        await self.group_server.delete_group(group_id, requester.user.to_string())
        return 200, {}
--- a/synapse/rest/admin/rooms.py
+++ b/synapse/rest/admin/rooms.py
@ -0,0 +1,157 @@
 # -*- coding: utf-8 -*-
 # Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
 from synapse.api.constants import Membership
 from synapse.http.servlet import (
    RestServlet,
    assert_params_in_dict,
    parse_json_object_from_request,
 )
 from synapse.rest.admin._base import (
    assert_user_is_admin,
    historical_admin_path_patterns,
 )
 from synapse.types import create_requester
 from synapse.util.async_helpers import maybe_awaitable
 logger = logging.getLogger(__name__)
 class ShutdownRoomRestServlet(RestServlet):
    """Shuts down a room by removing all local users from the room and blocking
    all future invites and joins to the room. Any local aliases will be repointed
    to a new room created by `new_room_user_id` and kicked users will be auto
    joined to the new room.
    """
    PATTERNS = historical_admin_path_patterns("/shutdown_room/(?P<room_id>[^/]+)")
    DEFAULT_MESSAGE = (
        "Sharing illegal content on this server is not permitted and rooms in"
        " violation will be blocked."
    )
    def __init__(self, hs):
        self.hs = hs
        self.store = hs.get_datastore()
        self.state = hs.get_state_handler()
        self._room_creation_handler = hs.get_room_creation_handler()
        self.event_creation_handler = hs.get_event_creation_handler()
        self.room_member_handler = hs.get_room_member_handler()
        self.auth = hs.get_auth()
    async def on_POST(self, request, room_id):
        requester = await self.auth.get_user_by_req(request)
        await assert_user_is_admin(self.auth, requester.user)
        content = parse_json_object_from_request(request)
        assert_params_in_dict(content, ["new_room_user_id"])
        new_room_user_id = content["new_room_user_id"]
        room_creator_requester = create_requester(new_room_user_id)
        message = content.get("message", self.DEFAULT_MESSAGE)
        room_name = content.get("room_name", "Content Violation Notification")
        info = await self._room_creation_handler.create_room(
            room_creator_requester,
            config={
                "preset": "public_chat",
                "name": room_name,
                "power_level_content_override": {"users_default": -10},
            },
            ratelimit=False,
        )
        new_room_id = info["room_id"]
        requester_user_id = requester.user.to_string()
        logger.info(
            "Shutting down room %r, joining to new room: %r", room_id, new_room_id
        )
        # This will work even if the room is already blocked, but that is
        # desirable in case the first attempt at blocking the room failed below.
        await self.store.block_room(room_id, requester_user_id)
        users = await self.state.get_current_users_in_room(room_id)
        kicked_users = []
        failed_to_kick_users = []
        for user_id in users:
            if not self.hs.is_mine_id(user_id):
                continue
            logger.info("Kicking %r from %r...", user_id, room_id)
            try:
                target_requester = create_requester(user_id)
                await self.room_member_handler.update_membership(
                    requester=target_requester,
                    target=target_requester.user,
                    room_id=room_id,
                    action=Membership.LEAVE,
                    content={},
                    ratelimit=False,
                    require_consent=False,
                )
                await self.room_member_handler.forget(target_requester.user, room_id)
                await self.room_member_handler.update_membership(
                    requester=target_requester,
                    target=target_requester.user,
                    room_id=new_room_id,
                    action=Membership.JOIN,
                    content={},
                    ratelimit=False,
                    require_consent=False,
                )
                kicked_users.append(user_id)
            except Exception:
                logger.exception(
                    "Failed to leave old room and join new room for %r", user_id
                )
                failed_to_kick_users.append(user_id)
        await self.event_creation_handler.create_and_send_nonmember_event(
            room_creator_requester,
            {
                "type": "m.room.message",
                "content": {"body": message, "msgtype": "m.text"},
                "room_id": new_room_id,
                "sender": new_room_user_id,
            },
            ratelimit=False,
        )
        aliases_for_room = await maybe_awaitable(
            self.store.get_aliases_for_room(room_id)
        )
        await self.store.update_aliases_for_room(
            room_id, new_room_id, requester_user_id
        )
        return (
            200,
            {
                "kicked_users": kicked_users,
                "failed_to_kick_users": failed_to_kick_users,
                "local_aliases": aliases_for_room,
                "new_room_id": new_room_id,
            },
        )
--- a/synapse/rest/admin/users.py
+++ b/synapse/rest/admin/users.py
@ -12,17 +12,419 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import hashlib
 import hmac
 import logging
 import re
-from synapse.api.errors import SynapseError
+from six import text_type
 from six.moves import http_client
 from synapse.api.constants import UserTypes
 from synapse.api.errors import Codes, SynapseError
 from synapse.http.servlet import (
    RestServlet,
    assert_params_in_dict,
    parse_integer,
    parse_json_object_from_request,
    parse_string,
 )
 from synapse.rest.admin._base import (
    assert_requester_is_admin,
    assert_user_is_admin,
    historical_admin_path_patterns,
 )
 from synapse.rest.admin import assert_requester_is_admin, assert_user_is_admin
 from synapse.types import UserID
 logger = logging.getLogger(__name__)
 class UsersRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/users/(?P<user_id>[^/]*)$")
    def __init__(self, hs):
        self.hs = hs
        self.auth = hs.get_auth()
        self.admin_handler = hs.get_handlers().admin_handler
    async def on_GET(self, request, user_id):
        target_user = UserID.from_string(user_id)
        await assert_requester_is_admin(self.auth, request)
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only users a local user")
        ret = await self.admin_handler.get_users()
        return 200, ret
 class GetUsersPaginatedRestServlet(RestServlet):
    """Get request to get specific number of users from Synapse.
    This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/users_paginate/
            @admin:user?access_token=admin_access_token&start=0&limit=10
        Returns:
            200 OK with json object {list[dict[str, Any]], count} or empty object.
        """
    PATTERNS = historical_admin_path_patterns(
        "/users_paginate/(?P<target_user_id>[^/]*)"
    )
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, target_user_id):
        """Get request to get specific number of users from Synapse.
        This needs user to have administrator access in Synapse.
        """
        await assert_requester_is_admin(self.auth, request)
        target_user = UserID.from_string(target_user_id)
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only users a local user")
        order = "name"  # order by name in user table
        start = parse_integer(request, "start", required=True)
        limit = parse_integer(request, "limit", required=True)
        logger.info("limit: %s, start: %s", limit, start)
        ret = await self.handlers.admin_handler.get_users_paginate(order, start, limit)
        return 200, ret
    async def on_POST(self, request, target_user_id):
        """Post request to get specific number of users from Synapse..
        This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/users_paginate/
            @admin:user?access_token=admin_access_token
        JsonBodyToSend:
            {
                "start": "0",
                "limit": "10
            }
        Returns:
            200 OK with json object {list[dict[str, Any]], count} or empty object.
        """
        await assert_requester_is_admin(self.auth, request)
        UserID.from_string(target_user_id)
        order = "name"  # order by name in user table
        params = parse_json_object_from_request(request)
        assert_params_in_dict(params, ["limit", "start"])
        limit = params["limit"]
        start = params["start"]
        logger.info("limit: %s, start: %s", limit, start)
        ret = await self.handlers.admin_handler.get_users_paginate(order, start, limit)
        return 200, ret
 class UserRegisterServlet(RestServlet):
    """
    Attributes:
         NONCE_TIMEOUT (int): Seconds until a generated nonce won't be accepted
         nonces (dict[str, int]): The nonces that we will accept. A dict of
             nonce to the time it was generated, in int seconds.
    """
    PATTERNS = historical_admin_path_patterns("/register")
    NONCE_TIMEOUT = 60
    def __init__(self, hs):
        self.handlers = hs.get_handlers()
        self.reactor = hs.get_reactor()
        self.nonces = {}
        self.hs = hs
    def _clear_old_nonces(self):
        """
        Clear out old nonces that are older than NONCE_TIMEOUT.
        """
        now = int(self.reactor.seconds())
        for k, v in list(self.nonces.items()):
            if now - v > self.NONCE_TIMEOUT:
                del self.nonces[k]
    def on_GET(self, request):
        """
        Generate a new nonce.
        """
        self._clear_old_nonces()
        nonce = self.hs.get_secrets().token_hex(64)
        self.nonces[nonce] = int(self.reactor.seconds())
        return 200, {"nonce": nonce}
    async def on_POST(self, request):
        self._clear_old_nonces()
        if not self.hs.config.registration_shared_secret:
            raise SynapseError(400, "Shared secret registration is not enabled")
        body = parse_json_object_from_request(request)
        if "nonce" not in body:
            raise SynapseError(400, "nonce must be specified", errcode=Codes.BAD_JSON)
        nonce = body["nonce"]
        if nonce not in self.nonces:
            raise SynapseError(400, "unrecognised nonce")
        # Delete the nonce, so it can't be reused, even if it's invalid
        del self.nonces[nonce]
        if "username" not in body:
            raise SynapseError(
                400, "username must be specified", errcode=Codes.BAD_JSON
            )
        else:
            if (
                not isinstance(body["username"], text_type)
                or len(body["username"]) > 512
            ):
                raise SynapseError(400, "Invalid username")
            username = body["username"].encode("utf-8")
            if b"\x00" in username:
                raise SynapseError(400, "Invalid username")
        if "password" not in body:
            raise SynapseError(
                400, "password must be specified", errcode=Codes.BAD_JSON
            )
        else:
            if (
                not isinstance(body["password"], text_type)
                or len(body["password"]) > 512
            ):
                raise SynapseError(400, "Invalid password")
            password = body["password"].encode("utf-8")
            if b"\x00" in password:
                raise SynapseError(400, "Invalid password")
        admin = body.get("admin", None)
        user_type = body.get("user_type", None)
        if user_type is not None and user_type not in UserTypes.ALL_USER_TYPES:
            raise SynapseError(400, "Invalid user type")
        got_mac = body["mac"]
        want_mac = hmac.new(
            key=self.hs.config.registration_shared_secret.encode(),
            digestmod=hashlib.sha1,
        )
        want_mac.update(nonce.encode("utf8"))
        want_mac.update(b"\x00")
        want_mac.update(username)
        want_mac.update(b"\x00")
        want_mac.update(password)
        want_mac.update(b"\x00")
        want_mac.update(b"admin" if admin else b"notadmin")
        if user_type:
            want_mac.update(b"\x00")
            want_mac.update(user_type.encode("utf8"))
        want_mac = want_mac.hexdigest()
        if not hmac.compare_digest(want_mac.encode("ascii"), got_mac.encode("ascii")):
            raise SynapseError(403, "HMAC incorrect")
        # Reuse the parts of RegisterRestServlet to reduce code duplication
        from synapse.rest.client.v2_alpha.register import RegisterRestServlet
        register = RegisterRestServlet(self.hs)
        user_id = await register.registration_handler.register_user(
            localpart=body["username"].lower(),
            password=body["password"],
            admin=bool(admin),
            user_type=user_type,
        )
        result = await register._create_registration_details(user_id, body)
        return 200, result
 class WhoisRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/whois/(?P<user_id>[^/]*)")
    def __init__(self, hs):
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, user_id):
        target_user = UserID.from_string(user_id)
        requester = await self.auth.get_user_by_req(request)
        auth_user = requester.user
        if target_user != auth_user:
            await assert_user_is_admin(self.auth, auth_user)
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only whois a local user")
        ret = await self.handlers.admin_handler.get_whois(target_user)
        return 200, ret
 class DeactivateAccountRestServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/deactivate/(?P<target_user_id>[^/]*)")
    def __init__(self, hs):
        self._deactivate_account_handler = hs.get_deactivate_account_handler()
        self.auth = hs.get_auth()
    async def on_POST(self, request, target_user_id):
        await assert_requester_is_admin(self.auth, request)
        body = parse_json_object_from_request(request, allow_empty_body=True)
        erase = body.get("erase", False)
        if not isinstance(erase, bool):
            raise SynapseError(
                http_client.BAD_REQUEST,
                "Param 'erase' must be a boolean, if given",
                Codes.BAD_JSON,
            )
        UserID.from_string(target_user_id)
        result = await self._deactivate_account_handler.deactivate_account(
            target_user_id, erase
        )
        if result:
            id_server_unbind_result = "success"
        else:
            id_server_unbind_result = "no-support"
        return 200, {"id_server_unbind_result": id_server_unbind_result}
 class AccountValidityRenewServlet(RestServlet):
    PATTERNS = historical_admin_path_patterns("/account_validity/validity$")
    def __init__(self, hs):
        """
        Args:
            hs (synapse.server.HomeServer): server
        """
        self.hs = hs
        self.account_activity_handler = hs.get_account_validity_handler()
        self.auth = hs.get_auth()
    async def on_POST(self, request):
        await assert_requester_is_admin(self.auth, request)
        body = parse_json_object_from_request(request)
        if "user_id" not in body:
            raise SynapseError(400, "Missing property 'user_id' in the request body")
        expiration_ts = await self.account_activity_handler.renew_account_for_user(
            body["user_id"],
            body.get("expiration_ts"),
            not body.get("enable_renewal_emails", True),
        )
        res = {"expiration_ts": expiration_ts}
        return 200, res
 class ResetPasswordRestServlet(RestServlet):
    """Post request to allow an administrator reset password for a user.
    This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/reset_password/
            @user:to_reset_password?access_token=admin_access_token
        JsonBodyToSend:
            {
                "new_password": "secret"
            }
        Returns:
            200 OK with empty object if success otherwise an error.
        """
    PATTERNS = historical_admin_path_patterns(
        "/reset_password/(?P<target_user_id>[^/]*)"
    )
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self.hs = hs
        self.auth = hs.get_auth()
        self._set_password_handler = hs.get_set_password_handler()
    async def on_POST(self, request, target_user_id):
        """Post request to allow an administrator reset password for a user.
        This needs user to have administrator access in Synapse.
        """
        requester = await self.auth.get_user_by_req(request)
        await assert_user_is_admin(self.auth, requester.user)
        UserID.from_string(target_user_id)
        params = parse_json_object_from_request(request)
        assert_params_in_dict(params, ["new_password"])
        new_password = params["new_password"]
        await self._set_password_handler.set_password(
            target_user_id, new_password, requester
        )
        return 200, {}
 class SearchUsersRestServlet(RestServlet):
    """Get request to search user table for specific users according to
    search term.
    This needs user to have administrator access in Synapse.
        Example:
            http://localhost:8008/_synapse/admin/v1/search_users/
            @admin:user?access_token=admin_access_token&term=alice
        Returns:
            200 OK with json object {list[dict[str, Any]], count} or empty object.
    """
    PATTERNS = historical_admin_path_patterns("/search_users/(?P<target_user_id>[^/]*)")
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self.hs = hs
        self.auth = hs.get_auth()
        self.handlers = hs.get_handlers()
    async def on_GET(self, request, target_user_id):
        """Get request to search user table for specific users according to
        search term.
        This needs user to have a administrator access in Synapse.
        """
        await assert_requester_is_admin(self.auth, request)
        target_user = UserID.from_string(target_user_id)
        # To allow all users to get the users list
        # if not is_admin and target_user != auth_user:
        #     raise AuthError(403, "You are not a server admin")
        if not self.hs.is_mine(target_user):
            raise SynapseError(400, "Can only users a local user")
        term = parse_string(request, "term", required=True)
        logger.info("term: %s ", term)
        ret = await self.handlers.admin_handler.search_users(term)
        return 200, ret
 class UserAdminServlet(RestServlet):
    """
--- a/synapse/rest/client/v1/login.py
+++ b/synapse/rest/client/v1/login.py
@ -92,8 +92,11 @@ class LoginRestServlet(RestServlet):
        self.auth_handler = self.hs.get_auth_handler()
        self.registration_handler = hs.get_registration_handler()
        self.handlers = hs.get_handlers()
        self._clock = hs.get_clock()
        self._well_known_builder = WellKnownBuilder(hs)
        self._address_ratelimiter = Ratelimiter()
        self._account_ratelimiter = Ratelimiter()
        self._failed_attempts_ratelimiter = Ratelimiter()
    def on_GET(self, request):
        flows = []
@ -202,6 +205,16 @@ class LoginRestServlet(RestServlet):
                # (See add_threepid in synapse/handlers/auth.py)
                address = address.lower()
            # We also apply account rate limiting using the 3PID as a key, as
            # otherwise using 3PID bypasses the ratelimiting based on user ID.
            self._failed_attempts_ratelimiter.ratelimit(
                (medium, address),
                time_now_s=self._clock.time(),
                rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
                burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
                update=False,
            )
            # Check for login providers that support 3pid login types
            (
                canonical_user_id,
@ -211,7 +224,8 @@ class LoginRestServlet(RestServlet):
            )
            if canonical_user_id:
                # Authentication through password provider and 3pid succeeded
-                result = yield self._register_device_with_callback(
+
                result = yield self._complete_login(
                    canonical_user_id, login_submission, callback_3pid
                )
                return result
@ -225,6 +239,21 @@ class LoginRestServlet(RestServlet):
                logger.warning(
                    "unknown 3pid identifier medium %s, address %r", medium, address
                )
                # We mark that we've failed to log in here, as
                # `check_password_provider_3pid` might have returned `None` due
                # to an incorrect password, rather than the account not
                # existing.
                #
                # If it returned None but the 3PID was bound then we won't hit
                # this code path, which is fine as then the per-user ratelimit
                # will kick in below.
                self._failed_attempts_ratelimiter.can_do_action(
                    (medium, address),
                    time_now_s=self._clock.time(),
                    rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
                    burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
                    update=True,
                )
                raise LoginError(403, "", errcode=Codes.FORBIDDEN)
            identifier = {"type": "m.id.user", "user": user_id}
@ -236,29 +265,84 @@ class LoginRestServlet(RestServlet):
        if "user" not in identifier:
            raise SynapseError(400, "User identifier is missing 'user' key")
        if identifier["user"].startswith("@"):
            qualified_user_id = identifier["user"]
        else:
            qualified_user_id = UserID(identifier["user"], self.hs.hostname).to_string()
        # Check if we've hit the failed ratelimit (but don't update it)
        self._failed_attempts_ratelimiter.ratelimit(
            qualified_user_id.lower(),
            time_now_s=self._clock.time(),
            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
            update=False,
        )
        try:
            canonical_user_id, callback = yield self.auth_handler.validate_login(
                identifier["user"], login_submission
            )
        except LoginError:
            # The user has failed to log in, so we need to update the rate
            # limiter. Using `can_do_action` avoids us raising a ratelimit
            # exception and masking the LoginError. The actual ratelimiting
            # should have happened above.
            self._failed_attempts_ratelimiter.can_do_action(
                qualified_user_id.lower(),
                time_now_s=self._clock.time(),
                rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
                burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
                update=True,
            )
            raise
-        result = yield self._register_device_with_callback(
+        result = yield self._complete_login(
            canonical_user_id, login_submission, callback
        )
        return result
    @defer.inlineCallbacks
-    def _register_device_with_callback(self, user_id, login_submission, callback=None):
+    def _complete_login(
-        """ Registers a device with a given user_id. Optionally run a callback
+        self, user_id, login_submission, callback=None, create_non_existant_users=False
-        function after registration has completed.
+    ):
        """Called when we've successfully authed the user and now need to
        actually login them in (e.g. create devices). This gets called on
        all succesful logins.
        Applies the ratelimiting for succesful login attempts against an
        account.
        Args:
            user_id (str): ID of the user to register.
            login_submission (dict): Dictionary of login information.
            callback (func|None): Callback function to run after registration.
            create_non_existant_users (bool): Whether to create the user if
                they don't exist. Defaults to False.
        Returns:
            result (Dict[str,str]): Dictionary of account information after
                successful registration.
        """
        # Before we actually log them in we check if they've already logged in
        # too often. This happens here rather than before as we don't
        # necessarily know the user before now.
        self._account_ratelimiter.ratelimit(
            user_id.lower(),
            time_now_s=self._clock.time(),
            rate_hz=self.hs.config.rc_login_account.per_second,
            burst_count=self.hs.config.rc_login_account.burst_count,
            update=True,
        )
        if create_non_existant_users:
            user_id = yield self.auth_handler.check_user_exists(user_id)
            if not user_id:
                user_id = yield self.registration_handler.register_user(
                    localpart=UserID.from_string(user_id).localpart
                )
        device_id = login_submission.get("device_id")
        initial_display_name = login_submission.get("initial_device_display_name")
        device_id, access_token = yield self.registration_handler.register_device(
@ -285,7 +369,7 @@ class LoginRestServlet(RestServlet):
            token
        )
-        result = yield self._register_device_with_callback(user_id, login_submission)
+        result = yield self._complete_login(user_id, login_submission)
        return result
    @defer.inlineCallbacks
@ -313,15 +397,8 @@ class LoginRestServlet(RestServlet):
            raise LoginError(401, "Invalid JWT", errcode=Codes.UNAUTHORIZED)
        user_id = UserID(user, self.hs.hostname).to_string()
-
+        result = yield self._complete_login(
-        registered_user_id = yield self.auth_handler.check_user_exists(user_id)
+            user_id, login_submission, create_non_existant_users=True
        if not registered_user_id:
            registered_user_id = yield self.registration_handler.register_user(
                localpart=user
            )
        result = yield self._register_device_with_callback(
            registered_user_id, login_submission
        )
        return result
		`@ -1 +0,0 @@`
			Add a CI job to test the `synapse_port_db` script.
		`@ -1 +0,0 @@`
			`Contributor documentation now mentions script to run linters.`
		`@ -1 +0,0 @@`
			`Fix LruCache callback deduplication for Python 3.8. Contributed by @V02460.`
		`@ -1 +0,0 @@`
			`Increase default room version from 4 to 5, thereby enforcing server key validity period checks.`
		`@ -1 +0,0 @@`
			`Remove a room from a server's public rooms list on room upgrade.`
		`@ -1 +0,0 @@`
			`Add support for outbound http proxying via http_proxy/HTTPS_PROXY env vars.`
		`@ -1 +0,0 @@`
			Move `persist_events` out from main data store.
		`@ -1 +0,0 @@`
			`Delete keys from key backup when deleting backup versions.`
		`@ -1 +0,0 @@`
			`Make notification of cross-signing signatures work with workers.`
		`@ -1 +0,0 @@`
			Modify CAPTCHA_SETUP.md to update the terms `private key` and `public key` to `secret key` and `site key` respectively. Contributed by Yash Jipkate.