Merge branch 'rav/fix_retry_reset' into matrix-org-hotfixes

michaelkaye/matrix_org_hotfixes_increase_replication_timeout
Richard van der Hoff 2019-09-20 12:14:19 +01:00
commit 8569f3cdef
26 changed files with 167 additions and 86 deletions

View File

@ -2,52 +2,78 @@ Upgrading Synapse
=================
Before upgrading check if any special steps are required to upgrade from the
what you currently have installed to current version of synapse. The extra
what you currently have installed to current version of Synapse. The extra
instructions that may be required are listed later in this document.
1. If synapse was installed in a virtualenv then activate that virtualenv before
upgrading. If synapse is installed in a virtualenv in ``~/synapse/env`` then
run:
* If Synapse was installed using `prebuilt packages
<INSTALL.md#prebuilt-packages>`_, you will need to follow the normal process
for upgrading those packages.
.. code:: bash
* If Synapse was installed from source, then:
1. Activate the virtualenv before upgrading. For example, if Synapse is
installed in a virtualenv in ``~/synapse/env`` then run:
.. code:: bash
source ~/synapse/env/bin/activate
2. If synapse was installed using pip then upgrade to the latest version by
running:
2. If Synapse was installed using pip then upgrade to the latest version by
running:
.. code:: bash
.. code:: bash
pip install --upgrade matrix-synapse[all]
pip install --upgrade matrix-synapse
# restart synapse
synctl restart
If Synapse was installed using git then upgrade to the latest version by
running:
.. code:: bash
If synapse was installed using git then upgrade to the latest version by
running:
.. code:: bash
# Pull the latest version of the master branch.
git pull
pip install --upgrade .
# Update synapse and its python dependencies.
pip install --upgrade .[all]
3. Restart Synapse:
.. code:: bash
# restart synapse
./synctl restart
To check whether your update was successful, you can check the Server header
returned by the Client-Server API:
To check whether your update was successful, you can check the running server
version with:
.. code:: bash
# replace <host.name> with the hostname of your synapse homeserver.
# You may need to specify a port (eg, :8448) if your server is not
# configured on port 443.
curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
# you may need to replace 'localhost:8008' if synapse is not configured
# to listen on port 8008.
curl http://localhost:8008/_synapse/admin/v1/server_version
Rolling back to older versions
------------------------------
Rolling back to previous releases can be difficult, due to database schema
changes between releases. Where we have been able to test the rollback process,
this will be noted below.
In general, you will need to undo any changes made during the upgrade process,
for example:
* pip:
.. code:: bash
source env/bin/activate
# replace `1.3.0` accordingly:
pip install matrix-synapse==1.3.0
* Debian:
.. code:: bash
# replace `1.3.0` and `stretch` accordingly:
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
Upgrading to v1.4.0
===================
@ -99,6 +125,31 @@ Synapse will expect these files to exist inside the configured template director
default templates, see `synapse/res/templates
<https://github.com/matrix-org/synapse/tree/master/synapse/res/templates>`_.
Rolling back to v1.3.1
----------------------
If you encounter problems with v1.4.0, it should be possible to roll back to
v1.3.1, subject to the following:
* The 'room statistics' engine was heavily reworked in this release (see
`#5971 <https://github.com/matrix-org/synapse/pull/5971>`_), including
significant changes to the database schema, which are not easily
reverted. This will cause the room statistics engine to stop updating when
you downgrade.
The room statistics are essentially unused in v1.3.1 (in future versions of
Synapse, they will be used to populate the room directory), so there should
be no loss of functionality. However, the statistics engine will write errors
to the logs, which can be avoided by setting the following in `homeserver.yaml`:
.. code:: yaml
stats:
enabled: false
Don't forget to re-enable it when you upgrade again, in preparation for its
use in the room directory!
Upgrading to v1.2.0
===================

View File

@ -1 +1 @@
Drop some unused tables.
Stop populating some unused tables.

View File

@ -1 +1 @@
Add POST /_matrix/client/r0/account/3pid/unbind endpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account.
Add POST /_matrix/client/unstable/account/3pid/unbind endpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account.

1
changelog.d/5992.feature Normal file
View File

@ -0,0 +1 @@
Give appropriate exit codes when synctl fails.

1
changelog.d/6028.feature Normal file
View File

@ -0,0 +1 @@
Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`.

2
changelog.d/6047.misc Normal file
View File

@ -0,0 +1,2 @@
Stop populating some unused tables.

1
changelog.d/6049.doc Normal file
View File

@ -0,0 +1 @@
Add some notes on rolling back to v1.3.1.

1
changelog.d/6050.doc Normal file
View File

@ -0,0 +1 @@
Update the upgrade notes.

1
changelog.d/6056.bugfix Normal file
View File

@ -0,0 +1 @@
Remove POST method from password reset submit_token endpoint until we implement submit_url functionality.

1
changelog.d/6058.docker Normal file
View File

@ -0,0 +1 @@
Provide SYNAPSE_WORKER envvar to specify python module.

1
changelog.d/6059.bugfix Normal file
View File

@ -0,0 +1 @@
Fix logcontext spam on non-Linux platforms.

1
changelog.d/6062.bugfix Normal file
View File

@ -0,0 +1 @@
Add POST /_matrix/client/unstable/account/3pid/unbind endpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account.

1
changelog.d/6063.bugfix Normal file
View File

@ -0,0 +1 @@
Ensure query parameters in email validation links are URL-encoded.

1
changelog.d/6067.feature Normal file
View File

@ -0,0 +1 @@
Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/).

1
changelog.d/6072.misc Normal file
View File

@ -0,0 +1 @@
Add a 'failure_ts' column to the 'destinations' database table.

View File

@ -89,6 +89,8 @@ The following environment variables are supported in run mode:
`/data`.
* `SYNAPSE_CONFIG_PATH`: path to the config file. Defaults to
`<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
* `SYNAPSE_WORKER`: module to execute, used when running synapse with workers.
Defaults to `synapse.app.homeserver`, which is suitable for non-worker mode.
* `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`.
* `TZ`: the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) the container will run with. Defaults to `UTC`.

View File

@ -182,6 +182,7 @@ def main(args, environ):
mode = args[1] if len(args) > 1 else None
desired_uid = int(environ.get("UID", "991"))
desired_gid = int(environ.get("GID", "991"))
synapse_worker = environ.get("SYNAPSE_WORKER", "synapse.app.homeserver")
if (desired_uid == os.getuid()) and (desired_gid == os.getgid()):
ownership = None
else:
@ -245,7 +246,7 @@ def main(args, environ):
log("Starting synapse with config file " + config_path)
args = ["python", "-m", "synapse.app.homeserver", "--config-path", config_path]
args = ["python", "-m", synapse_worker, "--config-path", config_path]
if ownership is not None:
args = ["su-exec", ownership] + args
os.execv("/sbin/su-exec", args)

View File

@ -938,7 +938,7 @@ uploads_path: "DATADIR/uploads"
# https://matrix.org/docs/spec/identity_service/latest
#
account_threepid_delegates:
#email: https://example.com # Delegate email sending to matrix.org
#email: https://example.com # Delegate email sending to example.org
#msisdn: http://localhost:8090 # Delegate SMS sending to this local process
# Users who register on this homeserver will automatically be joined

View File

@ -294,7 +294,7 @@ class RegistrationConfig(Config):
# https://matrix.org/docs/spec/identity_service/latest
#
account_threepid_delegates:
#email: https://example.com # Delegate email sending to matrix.org
#email: https://example.com # Delegate email sending to example.org
#msisdn: http://localhost:8090 # Delegate SMS sending to this local process
# Users who register on this homeserver will automatically be joined

View File

@ -165,7 +165,7 @@ class Authenticator(object):
async def _reset_retry_timings(self, origin):
try:
logger.info("Marking origin %r as up", origin)
await self.store.set_destination_retry_timings(origin, 0, 0)
await self.store.set_destination_retry_timings(origin, None, 0, 0)
except Exception:
logger.exception("Error resetting retry timings on %s", origin)

View File

@ -1,4 +1,5 @@
# Copyright 2014-2016 OpenMarket Ltd
# Copyright 2019 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -42,13 +43,17 @@ try:
# exception.
resource.getrusage(RUSAGE_THREAD)
is_thread_resource_usage_supported = True
def get_thread_resource_usage():
return resource.getrusage(RUSAGE_THREAD)
except Exception:
# If the system doesn't support resource.getrusage(RUSAGE_THREAD) then we
# won't track resource usage by returning None.
# won't track resource usage.
is_thread_resource_usage_supported = False
def get_thread_resource_usage():
return None
@ -359,7 +364,11 @@ class LoggingContext(object):
# When we stop, let's record the cpu used since we started
if not self.usage_start:
logger.warning("Called stop on logcontext %s without calling start", self)
# Log a warning on platforms that support thread usage tracking
if is_thread_resource_usage_supported:
logger.warning(
"Called stop on logcontext %s without calling start", self
)
return
utime_delta, stime_delta = self._get_cputime()

View File

@ -136,10 +136,11 @@ class Mailer(object):
group together multiple email sending attempts
sid (str): The generated session ID
"""
params = {"token": token, "client_secret": client_secret, "sid": sid}
link = (
self.hs.config.public_baseurl
+ "_matrix/client/unstable/password_reset/email/submit_token"
"?token=%s&client_secret=%s&sid=%s" % (token, client_secret, sid)
+ "_matrix/client/unstable/password_reset/email/submit_token?%s"
% urllib.parse.urlencode(params)
)
template_vars = {"link": link}
@ -163,10 +164,11 @@ class Mailer(object):
group together multiple email sending attempts
sid (str): The generated session ID
"""
params = {"token": token, "client_secret": client_secret, "sid": sid}
link = (
self.hs.config.public_baseurl
+ "_matrix/client/unstable/registration/email/submit_token"
"?token=%s&client_secret=%s&sid=%s" % (token, client_secret, sid)
+ "_matrix/client/unstable/registration/email/submit_token?%s"
% urllib.parse.urlencode(params)
)
template_vars = {"link": link}

View File

@ -272,23 +272,6 @@ class PasswordResetSubmitTokenServlet(RestServlet):
request.write(html.encode("utf-8"))
finish_request(request)
@defer.inlineCallbacks
def on_POST(self, request, medium):
if medium != "email":
raise SynapseError(
400, "This medium is currently not supported for password resets"
)
body = parse_json_object_from_request(request)
assert_params_in_dict(body, ["sid", "client_secret", "token"])
valid, _ = yield self.store.validate_threepid_session(
body["sid"], body["client_secret"], body["token"], self.clock.time_msec()
)
response_code = 200 if valid else 400
return response_code, {"success": valid}
class PasswordRestServlet(RestServlet):
PATTERNS = client_patterns("/account/password$")
@ -538,15 +521,11 @@ class ThreepidRestServlet(RestServlet):
user_id, threepid["medium"], threepid["address"], threepid["validated_at"]
)
if "bind" in body and body["bind"]:
logger.debug("Binding threepid %s to %s", threepid, user_id)
yield self.identity_handler.bind_threepid(threepid_creds, user_id)
return 200, {}
class ThreepidUnbindRestServlet(RestServlet):
PATTERNS = client_patterns("/account/3pid/unbind$")
PATTERNS = client_patterns("/account/3pid/unbind$", releases=(), unstable=True)
def __init__(self, hs):
super(ThreepidUnbindRestServlet, self).__init__()

View File

@ -1,20 +0,0 @@
/* Copyright 2019 The Matrix.org Foundation C.I.C.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-- these tables are never used.
DROP TABLE IF EXISTS room_names;
DROP TABLE IF EXISTS topics;
DROP TABLE IF EXISTS history_visibility;
DROP TABLE IF EXISTS guest_access;

43
synctl
View File

@ -71,7 +71,20 @@ def abort(message, colour=RED, stream=sys.stderr):
sys.exit(1)
def start(configfile, daemonize=True):
def start(configfile: str, daemonize: bool = True) -> bool:
"""Attempts to start synapse.
Args:
configfile: path to a yaml synapse config file
daemonize: whether to daemonize synapse or keep it attached to the current
session
Returns:
True if the process started successfully
False if there was an error starting the process
If deamonize is False it will only return once synapse exits.
"""
write("Starting ...")
args = SYNAPSE
@ -83,25 +96,40 @@ def start(configfile, daemonize=True):
try:
subprocess.check_call(args)
write("started synapse.app.homeserver(%r)" % (configfile,), colour=GREEN)
return True
except subprocess.CalledProcessError as e:
write(
"error starting (exit code: %d); see above for logs" % e.returncode,
colour=RED,
)
return False
def start_worker(app, configfile, worker_configfile):
def start_worker(app: str, configfile: str, worker_configfile: str) -> bool:
"""Attempts to start a synapse worker.
Args:
app: name of the worker's appservice
configfile: path to a yaml synapse config file
worker_configfile: path to worker specific yaml synapse file
Returns:
True if the process started successfully
False if there was an error starting the process
"""
args = [sys.executable, "-B", "-m", app, "-c", configfile, "-c", worker_configfile]
try:
subprocess.check_call(args)
write("started %s(%r)" % (app, worker_configfile), colour=GREEN)
return True
except subprocess.CalledProcessError as e:
write(
"error starting %s(%r) (exit code: %d); see above for logs"
% (app, worker_configfile, e.returncode),
colour=RED,
)
return False
def stop(pidfile, app):
@ -292,11 +320,14 @@ def main():
write("All processes exited; now restarting...")
if action == "start" or action == "restart":
error = False
if start_stop_synapse:
# Check if synapse is already running
if os.path.exists(pidfile) and pid_running(int(open(pidfile).read())):
abort("synapse.app.homeserver already running")
start(configfile, bool(options.daemonize))
if not start(configfile, bool(options.daemonize)):
error = True
for worker in workers:
env = os.environ.copy()
@ -307,12 +338,16 @@ def main():
for cache_name, factor in iteritems(worker.cache_factors):
os.environ["SYNAPSE_CACHE_FACTOR_" + cache_name.upper()] = str(factor)
start_worker(worker.app, configfile, worker.configfile)
if not start_worker(worker.app, configfile, worker.configfile):
error = True
# Reset env back to the original
os.environ.clear()
os.environ.update(env)
if error:
exit(1)
if __name__ == "__main__":
main()

View File

@ -29,3 +29,12 @@ Enabling an unknown default rule fails with 404
# Blacklisted due to https://github.com/matrix-org/synapse/issues/1663
New federated private chats get full presence information (SYN-115)
# Blacklisted temporarily due to https://github.com/matrix-org/matrix-doc/pull/2290
# These sytests need to be updated with new endpoints, which will come in a later PR
# That PR will also remove this blacklist
Can bind 3PID via home server
Can bind and unbind 3PID via homeserver
3PIDs are unbound after account deactivation
Can bind and unbind 3PID via /unbind by specifying the identity server
Can bind and unbind 3PID via /unbind without specifying the identity server