OpenCloud
/
MatrixSynapse
mirror of https://github.com/matrix-org/synapse
1
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'develop' into matrix-org-hotfixes

pull/8675/head
Richard van der Hoff 2020-07-24 09:57:49 +01:00
parent 25880bd441 1ec688bf21
commit be777e325d
277 changed files with 5677 additions and 2756 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

64
CHANGES.md
View File

@ -1,3 +1,67 @@
Synapse 1.17.0 (2020-07-13)
===========================
Synapse 1.17.0 is identical to 1.17.0rc1, with the addition of the fix that was included in 1.16.1.
Synapse 1.16.1 (2020-07-10)
===========================
In some distributions of Synapse 1.16.0, we incorrectly included a database migration which added a new, unused table. This release removes the redundant table.
Bugfixes
--------
- Drop table `local_rejections_stream` which was incorrectly added in Synapse 1.16.0. ([\#7816](https://github.com/matrix-org/synapse/issues/7816), [b1beb3ff5](https://github.com/matrix-org/synapse/commit/b1beb3ff5))
Synapse 1.17.0rc1 (2020-07-09)
==============================
Bugfixes
--------
- Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel. ([\#7021](https://github.com/matrix-org/synapse/issues/7021))
- Fix "Tried to close a non-active scope!" error messages when opentracing is enabled. ([\#7732](https://github.com/matrix-org/synapse/issues/7732))
- Fix incorrect error message when database CTYPE was set incorrectly. ([\#7760](https://github.com/matrix-org/synapse/issues/7760))
- Fix to not ignore `set_tweak` actions in Push Rules that have no `value`, as permitted by the specification. ([\#7766](https://github.com/matrix-org/synapse/issues/7766))
- Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian. ([\#7779](https://github.com/matrix-org/synapse/issues/7779))
- Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent. ([\#7797](https://github.com/matrix-org/synapse/issues/7797))
- Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation. ([\#7804](https://github.com/matrix-org/synapse/issues/7804), [\#7809](https://github.com/matrix-org/synapse/issues/7809), [\#7810](https://github.com/matrix-org/synapse/issues/7810))
Updates to the Docker image
---------------------------
- Include libwebp in the Docker file to properly handle webp image uploads. ([\#7791](https://github.com/matrix-org/synapse/issues/7791))
Improved Documentation
----------------------
- Improve the documentation of the non-standard JSON web token login type. ([\#7776](https://github.com/matrix-org/synapse/issues/7776))
- Update doc links for caddy. Contributed by Nicolai Søborg. ([\#7789](https://github.com/matrix-org/synapse/issues/7789))
Internal Changes
----------------
- Refactor getting replication updates from database. ([\#7740](https://github.com/matrix-org/synapse/issues/7740))
- Send push notifications with a high or low priority depending upon whether they may generate user-observable effects. ([\#7765](https://github.com/matrix-org/synapse/issues/7765))
- Use symbolic names for replication stream names. ([\#7768](https://github.com/matrix-org/synapse/issues/7768))
- Add early returns to `_check_for_soft_fail`. ([\#7769](https://github.com/matrix-org/synapse/issues/7769))
- Fix up `synapse.handlers.federation` to pass mypy. ([\#7770](https://github.com/matrix-org/synapse/issues/7770))
- Convert the appserver handler to async/await. ([\#7775](https://github.com/matrix-org/synapse/issues/7775))
- Allow to use higher versions of prometheus_client <0.9.0 which are expected to introduce no breaking changes. Contributed by Oliver Kurz. ([\#7780](https://github.com/matrix-org/synapse/issues/7780))
- Update linting scripts and codebase to be compatible with `isort` v5. ([\#7786](https://github.com/matrix-org/synapse/issues/7786))
- Stop populating unused table `local_invites`. ([\#7793](https://github.com/matrix-org/synapse/issues/7793))
- Ensure that strings (not bytes) are passed into JSON serialization. ([\#7799](https://github.com/matrix-org/synapse/issues/7799))
- Switch from simplejson to the standard library json. ([\#7800](https://github.com/matrix-org/synapse/issues/7800))
- Add `signing_key` property to `HomeServer` to save code duplication. ([\#7805](https://github.com/matrix-org/synapse/issues/7805))
- Improve stacktraces from exceptions in background processes. ([\#7808](https://github.com/matrix-org/synapse/issues/7808))
- Fix various spelling errors in comments and log lines. ([\#7811](https://github.com/matrix-org/synapse/issues/7811))
Synapse 1.16.0 (2020-07-08)
===========================

12
INSTALL.md
View File

@ -405,13 +405,11 @@ so, you will need to edit `homeserver.yaml`, as follows:
```
* You will also need to uncomment the `tls_certificate_path` and
`tls_private_key_path` lines under the `TLS` section. You can either
point these settings at an existing certificate and key, or you can
enable Synapse's built-in ACME (Let's Encrypt) support. Instructions
for having Synapse automatically provision and renew federation
certificates through ACME can be found at [ACME.md](docs/ACME.md).
Note that, as pointed out in that document, this feature will not
work with installs set up after November 2019.
`tls_private_key_path` lines under the `TLS` section. You will need to manage
provisioning of these certificates yourself — Synapse had built-in ACME
support, but the ACMEv1 protocol Synapse implements is deprecated, not
allowed by LetsEncrypt for new sites, and will break for existing sites in
late 2020. See [ACME.md](docs/ACME.md).
If you are using your own certificate, be sure to use a `.pem` file that
includes the full certificate chain including any intermediate certificates

10
README.rst
View File

@ -188,12 +188,8 @@ Using PostgreSQL
================
Synapse offers two database engines:
* `SQLite <https://sqlite.org/>`_
* `PostgreSQL <https://www.postgresql.org>`_
By default Synapse uses SQLite in and doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
light workloads.
* `SQLite <https://sqlite.org/>`_
Almost all installations should opt to use PostgreSQL. Advantages include:
@ -207,6 +203,10 @@ Almost all installations should opt to use PostgreSQL. Advantages include:
For information on how to install and use PostgreSQL, please see
`docs/postgres.md <docs/postgres.md>`_.
By default Synapse uses SQLite and in doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
light workloads.
.. _reverse-proxy:
Using a reverse proxy with Synapse

1
changelog.d/6455.feature Normal file
View File

@ -0,0 +1 @@
Include room states on invite events that are sent to application services. Contributed by @Sorunome.

1
changelog.d/7021.bugfix
View File

@ -1 +0,0 @@
Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel.

1
changelog.d/7613.feature Normal file
View File

@ -0,0 +1 @@
Add delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel.

1
changelog.d/7732.bugfix
View File

@ -1 +0,0 @@
Fix "Tried to close a non-active scope!" error messages when opentracing is enabled.

1
changelog.d/7740.misc
View File

@ -1 +0,0 @@
Refactor getting replication updates from database.

1
changelog.d/7760.bugfix
View File

@ -1 +0,0 @@
Fix incorrect error message when database CTYPE was set incorrectly.

1
changelog.d/7765.misc
View File

@ -1 +0,0 @@
Send push notifications with a high or low priority depending upon whether they may generate user-observable effects.

1
changelog.d/7766.bugfix
View File

@ -1 +0,0 @@
Fix to not ignore `set_tweak` actions in Push Rules that have no `value`, as permitted by the specification.

1
changelog.d/7768.misc
View File

@ -1 +0,0 @@
Use symbolic names for replication stream names.

1
changelog.d/7769.misc
View File

@ -1 +0,0 @@
Add early returns to `_check_for_soft_fail`.

1
changelog.d/7770.misc
View File

@ -1 +0,0 @@
Fix up `synapse.handlers.federation` to pass mypy.

1
changelog.d/7775.misc
View File

@ -1 +0,0 @@
Convert the appserver handler to async/await.

1
changelog.d/7776.doc
View File

@ -1 +0,0 @@
Improve the documentation of the non-standard JSON web token login type.

1
changelog.d/7779.bugfix
View File

@ -1 +0,0 @@
Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian.

1
changelog.d/7780.misc
View File

@ -1 +0,0 @@
Allow to use higher versions of prometheus_client <0.9.0 which are expected to introduce no breaking changes. Contributed by Oliver Kurz.

1
changelog.d/7786.misc
View File

@ -1 +0,0 @@
Update linting scripts and codebase to be compatible with `isort` v5.

1
changelog.d/7789.doc
View File

@ -1 +0,0 @@
Update doc links for caddy. Contributed by Nicolai Søborg.

1
changelog.d/7791.docker
View File

@ -1 +0,0 @@
Include libwebp in the Docker file to properly handle webp image uploads.

1
changelog.d/7793.misc
View File

@ -1 +0,0 @@
Stop populating unused table `local_invites`.

1
changelog.d/7797.bugfix
View File

@ -1 +0,0 @@
Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent.

1
changelog.d/7798.feature Normal file
View File

@ -0,0 +1 @@
Add experimental support for running multiple federation sender processes.

1
changelog.d/7799.misc
View File

@ -1 +0,0 @@
Ensure that strings (not bytes) are passed into JSON serialization.

1
changelog.d/7800.misc
View File

@ -1 +0,0 @@
Switch from simplejson to the standard library json.

1
changelog.d/7802.misc Normal file
View File

@ -0,0 +1 @@
Switch from simplejson to the standard library json.

1
changelog.d/7804.bugfix
View File

@ -1 +0,0 @@
Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation.

1
changelog.d/7805.misc
View File

@ -1 +0,0 @@
Add `signing_key` property to `HomeServer` to save code duplication.

1
changelog.d/7810.bugfix
View File

@ -1 +0,0 @@
Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation.

1
changelog.d/7813.misc Normal file
View File

@ -0,0 +1 @@
Add type hints to the http server code and remove an unused parameter.

1
changelog.d/7815.bugfix Normal file
View File

@ -0,0 +1 @@
Fix detection of out of sync remote device lists when receiving events from remote users.

1
changelog.d/7817.bugfix Normal file
View File

@ -0,0 +1 @@
Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain.

1
changelog.d/7820.misc Normal file
View File

@ -0,0 +1 @@
Add type hints to synapse.api.errors module.

1
changelog.d/7822.bugfix Normal file
View File

@ -0,0 +1 @@
Fix a bug causing Synapse to misinterpret the value `off` for `encryption_enabled_by_default_for_room_type` in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0.

1
changelog.d/7827.feature Normal file
View File

@ -0,0 +1 @@
Add the option to validate the `iss` and `aud` claims for JWT logins.

1
changelog.d/7829.bugfix Normal file
View File

@ -0,0 +1 @@
Fix bug where we did not always pass in `app_name` or `server_name` to email templates, including e.g. for registration emails.

1
changelog.d/7830.feature Normal file
View File

@ -0,0 +1 @@
Add support for handling registration requests across multiple client reader workers.

1
changelog.d/7836.misc Normal file
View File

@ -0,0 +1 @@
Ensure that calls to `json.dumps` are compatible with the standard library json.

1
changelog.d/7839.docker Normal file
View File

@ -0,0 +1 @@
Base docker image on Debian Buster rather than Alpine Linux. Contributed by @maquis196.

1
changelog.d/7842.feature Normal file
View File

@ -0,0 +1 @@
Add an admin API to list the users in a room. Contributed by Awesome Technologies Innovationslabor GmbH.

1
changelog.d/7844.bugfix Normal file
View File

@ -0,0 +1 @@
Errors which occur while using the non-standard JWT login now return the proper error: `403 Forbidden` with an error code of `M_FORBIDDEN`.

1
changelog.d/7846.feature Normal file
View File

@ -0,0 +1 @@
Allow email subjects to be customised through Synapse's configuration.

1
changelog.d/7847.feature Normal file
View File

@ -0,0 +1 @@
Add the ability to re-activate an account from the admin API.

1
changelog.d/7848.misc Normal file
View File

@ -0,0 +1 @@
Remove redundant `retry_on_integrity_error` wrapper for event persistence code.

1
changelog.d/7849.misc Normal file
View File

@ -0,0 +1 @@
Consistently use `db_to_json` to convert from database values to JSON objects.

1
changelog.d/7850.bugfix Normal file
View File

@ -0,0 +1 @@
Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0.

1
changelog.d/7851.misc Normal file
View File

@ -0,0 +1 @@
Convert E2E keys and room keys handlers to async/await.

1
changelog.d/7853.misc Normal file
View File

@ -0,0 +1 @@
Add support for handling registration requests across multiple client reader workers.

1
changelog.d/7854.bugfix Normal file
View File

@ -0,0 +1 @@
Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation.

1
changelog.d/7855.feature Normal file
View File

@ -0,0 +1 @@
Add experimental support for running multiple pusher workers.

1
changelog.d/7856.misc Normal file
View File

@ -0,0 +1 @@
Small performance improvement in typing processing.

1
changelog.d/7858.misc Normal file
View File

@ -0,0 +1 @@
The default value of `filter_timeline_limit` was changed from -1 (no limit) to 100.

1
changelog.d/7859.bugfix Normal file
View File

@ -0,0 +1 @@
Fix a bug which allowed empty rooms to be rejoined over federation.

1
changelog.d/7860.misc Normal file
View File

@ -0,0 +1 @@
Convert _base, profile, and _receipts handlers to async/await.

1
changelog.d/7861.misc Normal file
View File

@ -0,0 +1 @@
Optimise queueing of inbound replication commands.

1
changelog.d/7866.bugfix Normal file
View File

@ -0,0 +1 @@
Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers.

1
changelog.d/7868.misc Normal file
View File

@ -0,0 +1 @@
Convert synapse.app and federation client to async/await.

1
changelog.d/7869.feature Normal file
View File

@ -0,0 +1 @@
Add experimental support for moving typing off master.

1
changelog.d/7870.misc Normal file
View File

@ -0,0 +1 @@
Add some type annotations to `HomeServer` and `BaseHandler`.

1
changelog.d/7871.misc Normal file
View File

@ -0,0 +1 @@
Convert device handler to async/await.

1
changelog.d/7872.bugfix Normal file
View File

@ -0,0 +1 @@
Fix a long standing bug where the tracing of async functions with opentracing was broken.

1
changelog.d/7874.misc Normal file
View File

@ -0,0 +1 @@
Convert the federation agent and related code to async/await.

1
changelog.d/7877.misc Normal file
View File

@ -0,0 +1 @@
Clean up `PreserveLoggingContext`.

1
changelog.d/7878.removal Normal file
View File

@ -0,0 +1 @@
Remove unused `synapse_replication_tcp_resource_invalidate_cache` prometheus metric.

1
changelog.d/7879.feature Normal file
View File

@ -0,0 +1 @@
Report CPU metrics to prometheus for time spent processing replication commands.

1
changelog.d/7880.bugfix Normal file
View File

@ -0,0 +1 @@
Fix "TypeError in `synapse.notifier`" exceptions.

1
changelog.d/7881.misc Normal file
View File

@ -0,0 +1 @@
Change "unknown room version" logging from 'error' to 'warning'.

1
changelog.d/7882.misc Normal file
View File

@ -0,0 +1 @@
Stop using `device_max_stream_id` table and just use `device_inbox.stream_id`.

1
changelog.d/7884.misc Normal file
View File

@ -0,0 +1 @@
Convert the message handler to async/await.

1
changelog.d/7885.doc Normal file
View File

@ -0,0 +1 @@
Provide instructions on using `register_new_matrix_user` via docker.

1
changelog.d/7888.misc Normal file
View File

@ -0,0 +1 @@
Remove Ubuntu Eoan from the list of `.deb` packages that we build as it is now end-of-life. Contributed by @gary-kim.

1
changelog.d/7889.doc Normal file
View File

@ -0,0 +1 @@
Change the sample config postgres user section to use `synapse_user` instead of `synapse` to align with the documentation.

1
changelog.d/7890.misc Normal file
View File

@ -0,0 +1 @@
Fix typo in generated config file. Contributed by @ThiefMaster.

1
changelog.d/7892.misc Normal file
View File

@ -0,0 +1 @@
Import ABC from `collections.abc` for Python 3.10 compatibility.

1
changelog.d/7895.bugfix Normal file
View File

@ -0,0 +1 @@
Fix deprecation warning due to invalid escape sequences.

2
changelog.d/7897.misc Normal file
View File

@ -0,0 +1,2 @@
Remove unused functions `time_function`, `trace_function`, `get_previous_frames`
and `get_previous_frame` from `synapse.logging.utils` module.

1
changelog.d/7908.feature Normal file
View File

@ -0,0 +1 @@
Add the ability to re-activate an account from the admin API.

1
changelog.d/7912.misc Normal file
View File

@ -0,0 +1 @@
Convert `RoomListHandler` to async/await.

1
changelog.d/7914.misc Normal file
View File

@ -0,0 +1 @@
Lint the `contrib/` directory in CI and linting scripts, add `synctl` to the linting script for consistency with CI.

1
changelog.d/7919.misc Normal file
View File

@ -0,0 +1 @@
Use Element CSS and logo in notification emails when app name is Element.

1
changelog.d/7927.misc Normal file
View File

@ -0,0 +1 @@
Optimisation to /sync handling: skip serializing the response if the client has already disconnected.

1
changelog.d/7928.misc Normal file
View File

@ -0,0 +1 @@
When a client disconnects, don't log it as 'Error processing request'.

1
changelog.d/7929.misc Normal file
View File

@ -0,0 +1 @@
Add debugging to `/sync` response generation (disabled by default).

1
changelog.d/7930.feature Normal file
View File

@ -0,0 +1 @@
Abort federation requests where the client disconnects before the ratelimiter expires.

1
changelog.d/7931.feature Normal file
View File

@ -0,0 +1 @@
Cache responses to `/_matrix/federation/v1/state_ids` to reduce duplicated work.

1
changelog.d/7933.doc Normal file
View File

@ -0,0 +1 @@
Reorder database paragraphs to promote postgres over sqlite.

1
changelog.d/7934.doc Normal file
View File

@ -0,0 +1 @@
Update the dates of ACME v1's end of life in [`ACME.md`](https://github.com/matrix-org/synapse/blob/master/docs/ACME.md).

1
changelog.d/7935.misc Normal file
View File

@ -0,0 +1 @@
Convert the auth providers to be async/await.

1
changelog.d/7939.misc Normal file
View File

@ -0,0 +1 @@
Convert presence handler helpers to async/await.

21
contrib/cmdclient/console.py
View File

@ -17,9 +17,6 @@
""" Starts a synapse client console. """
from __future__ import print_function
from twisted.internet import reactor, defer, threads
from http import TwistedHttpClient
import argparse
import cmd
import getpass
@ -28,12 +25,14 @@ import shlex
import sys
import time
import urllib
import urlparse
from http import TwistedHttpClient
import nacl.signing
import nacl.encoding
import nacl.signing
import urlparse
from signedjson.sign import SignatureVerifyException, verify_signed_json
from signedjson.sign import verify_signed_json, SignatureVerifyException
from twisted.internet import defer, reactor, threads
CONFIG_JSON = "cmdclient_config.json"
@ -493,7 +492,7 @@ class SynapseCmd(cmd.Cmd):
"list messages <roomid> from=END&to=START&limit=3"
"""
args = self._parse(line, ["type", "roomid", "qp"])
if not "type" in args or not "roomid" in args:
if "type" not in args or "roomid" not in args:
print("Must specify type and room ID.")
return
if args["type"] not in ["members", "messages"]:
@ -508,7 +507,7 @@ class SynapseCmd(cmd.Cmd):
try:
key_value = key_value_str.split("=")
qp[key_value[0]] = key_value[1]
except:
except Exception:
print("Bad query param: %s" % key_value)
return
@ -585,7 +584,7 @@ class SynapseCmd(cmd.Cmd):
parsed_url = urlparse.urlparse(args["path"])
qp.update(urlparse.parse_qs(parsed_url.query))
args["path"] = parsed_url.path
except:
except Exception:
pass
reactor.callFromThread(
@ -772,10 +771,10 @@ def main(server_url, identity_server_url, username, token, config_path):
syn_cmd.config = json.load(config)
try:
http_client.verbose = "on" == syn_cmd.config["verbose"]
except:
except Exception:
pass
print("Loaded config from %s" % config_path)
except:
except Exception:
pass
# Twisted-specific: Runs the command processor in Twisted's event loop

10
contrib/cmdclient/http.py
View File

@ -14,14 +14,14 @@
# limitations under the License.
from __future__ import print_function
from twisted.web.client import Agent, readBody
from twisted.web.http_headers import Headers
from twisted.internet import defer, reactor
from pprint import pformat
import json
import urllib
from pprint import pformat
from twisted.internet import defer, reactor
from twisted.web.client import Agent, readBody
from twisted.web.http_headers import Headers
class HttpClient(object):

55
contrib/experiments/test_messaging.py
View File

@ -28,27 +28,24 @@ Currently assumes the local address is localhost:<port>
"""
from synapse.federation import ReplicationHandler
from synapse.federation.units import Pdu
from synapse.util import origin_from_ucid
from synapse.app.homeserver import SynapseHomeServer
# from synapse.logging.utils import log_function
from twisted.internet import reactor, defer
from twisted.python import log
import argparse
import curses.wrapper
import json
import logging
import os
import re
import cursesio
import curses.wrapper
from twisted.internet import defer, reactor
from twisted.python import log
from synapse.app.homeserver import SynapseHomeServer
from synapse.federation import ReplicationHandler
from synapse.federation.units import Pdu
from synapse.util import origin_from_ucid
# from synapse.logging.utils import log_function
logger = logging.getLogger("example")
@ -75,7 +72,7 @@ class InputOutput(object):
"""
try:
m = re.match("^join (\S+)$", line)
m = re.match(r"^join (\S+)$", line)
if m:
# The `sender` wants to join a room.
(room_name,) = m.groups()
@ -84,7 +81,7 @@ class InputOutput(object):
# self.print_line("OK.")
return
m = re.match("^invite (\S+) (\S+)$", line)
m = re.match(r"^invite (\S+) (\S+)$", line)
if m:
# `sender` wants to invite someone to a room
room_name, invitee = m.groups()
@ -93,7 +90,7 @@ class InputOutput(object):
# self.print_line("OK.")
return
m = re.match("^send (\S+) (.*)$", line)
m = re.match(r"^send (\S+) (.*)$", line)
if m:
# `sender` wants to message a room
room_name, body = m.groups()
@ -102,7 +99,7 @@ class InputOutput(object):
# self.print_line("OK.")
return
m = re.match("^backfill (\S+)$", line)
m = re.match(r"^backfill (\S+)$", line)
if m:
# we want to backfill a room
(room_name,) = m.groups()
@ -201,16 +198,6 @@ class HomeServer(ReplicationHandler):
% (pdu.context, pdu.pdu_type, json.dumps(pdu.content))
)
# def on_state_change(self, pdu):
##self.output.print_line("#%s (state) %s *** %s" %
##(pdu.context, pdu.state_key, pdu.pdu_type)
##)
# if "joinee" in pdu.content:
# self._on_join(pdu.context, pdu.content["joinee"])
# elif "invitee" in pdu.content:
# self._on_invite(pdu.origin, pdu.context, pdu.content["invitee"])
def _on_message(self, pdu):
""" We received a message
"""
@ -314,7 +301,7 @@ class HomeServer(ReplicationHandler):
return self.replication_layer.backfill(dest, room_name, limit)
def _get_room_remote_servers(self, room_name):
return [i for i in self.joined_rooms.setdefault(room_name).servers]
return list(self.joined_rooms.setdefault(room_name).servers)
def _get_or_create_room(self, room_name):
return self.joined_rooms.setdefault(room_name, Room(room_name))
@ -334,7 +321,7 @@ def main(stdscr):
user = args.user
server_name = origin_from_ucid(user)
## Set up logging ##
# Set up logging
root_logger = logging.getLogger()
@ -354,7 +341,7 @@ def main(stdscr):
observer = log.PythonLoggingObserver()
observer.start()
## Set up synapse server
# Set up synapse server
curses_stdio = cursesio.CursesStdIO(stdscr)
input_output = InputOutput(curses_stdio, user)
@ -368,16 +355,16 @@ def main(stdscr):
input_output.set_home_server(hs)
## Add input_output logger
# Add input_output logger
io_logger = IOLoggerHandler(input_output)
io_logger.setFormatter(formatter)
root_logger.addHandler(io_logger)
## Start! ##
# Start!
try:
port = int(server_name.split(":")[1])
except:
except Exception:
port = 12345
app_hs.get_http_server().start_listening(port)

299
contrib/grafana/synapse.json
View File

@ -1,7 +1,44 @@
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "6.7.4"
},
{
"type": "panel",
"id": "graph",
"name": "Graph",
"version": ""
},
{
"type": "panel",
"id": "heatmap",
"name": "Heatmap",
"version": ""
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
}
],
"annotations": {
"list": [
{
"$$hashKey": "object:76",
"builtIn": 1,
"datasource": "$datasource",
"enable": false,
@ -17,8 +54,8 @@
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 1,
"iteration": 1591098104645,
"id": null,
"iteration": 1594646317221,
"links": [
{
"asDropdown": true,
@ -34,7 +71,7 @@
"panels": [
{
"collapsed": false,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -269,7 +306,6 @@
"show": false
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@ -559,7 +595,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -1423,7 +1459,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -1795,7 +1831,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -2531,7 +2567,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -2823,7 +2859,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -2844,7 +2880,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 33
"y": 6
},
"hiddenSeries": false,
"id": 79,
@ -2940,7 +2976,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 33
"y": 6
},
"hiddenSeries": false,
"id": 83,
@ -3038,7 +3074,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 42
"y": 15
},
"hiddenSeries": false,
"id": 109,
@ -3137,7 +3173,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 42
"y": 15
},
"hiddenSeries": false,
"id": 111,
@ -3223,14 +3259,14 @@
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"description": "",
"description": "Number of events queued up on the master process for processing by the federation sender",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 51
"y": 24
},
"hiddenSeries": false,
"id": 140,
@ -3354,6 +3390,103 @@
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "${DS_PROMETHEUS}",
"description": "The number of events in the in-memory queues ",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"hiddenSeries": false,
"id": 142,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "synapse_federation_transaction_queue_pending_pdus{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"interval": "",
"legendFormat": "pending PDUs {{job}}-{{index}}",
"refId": "A"
},
{
"expr": "synapse_federation_transaction_queue_pending_edus{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"interval": "",
"legendFormat": "pending EDUs {{job}}-{{index}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "In-memory federation transmission queues",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"$$hashKey": "object:317",
"format": "short",
"label": "events",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"$$hashKey": "object:318",
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Federation",
@ -3361,7 +3494,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -3567,7 +3700,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -3588,7 +3721,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 52
"y": 79
},
"hiddenSeries": false,
"id": 48,
@ -3682,7 +3815,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 52
"y": 79
},
"hiddenSeries": false,
"id": 104,
@ -3802,7 +3935,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 59
"y": 86
},
"hiddenSeries": false,
"id": 10,
@ -3898,7 +4031,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 59
"y": 86
},
"hiddenSeries": false,
"id": 11,
@ -3987,7 +4120,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -4011,7 +4144,7 @@
"h": 13,
"w": 12,
"x": 0,
"y": 67
"y": 80
},
"hiddenSeries": false,
"id": 12,
@ -4106,7 +4239,7 @@
"h": 13,
"w": 12,
"x": 12,
"y": 67
"y": 80
},
"hiddenSeries": false,
"id": 26,
@ -4201,7 +4334,7 @@
"h": 13,
"w": 12,
"x": 0,
"y": 80
"y": 93
},
"hiddenSeries": false,
"id": 13,
@ -4297,7 +4430,7 @@
"h": 13,
"w": 12,
"x": 12,
"y": 80
"y": 93
},
"hiddenSeries": false,
"id": 27,
@ -4392,7 +4525,7 @@
"h": 13,
"w": 12,
"x": 0,
"y": 93
"y": 106
},
"hiddenSeries": false,
"id": 28,
@ -4486,7 +4619,7 @@
"h": 13,
"w": 12,
"x": 12,
"y": 93
"y": 106
},
"hiddenSeries": false,
"id": 25,
@ -4572,7 +4705,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -5062,7 +5195,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -5083,7 +5216,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 66
"y": 121
},
"hiddenSeries": false,
"id": 91,
@ -5179,7 +5312,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 66
"y": 121
},
"hiddenSeries": false,
"id": 21,
@ -5271,7 +5404,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 75
"y": 130
},
"hiddenSeries": false,
"id": 89,
@ -5369,7 +5502,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 75
"y": 130
},
"hiddenSeries": false,
"id": 93,
@ -5459,7 +5592,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 84
"y": 139
},
"hiddenSeries": false,
"id": 95,
@ -5552,12 +5685,12 @@
"mode": "spectrum"
},
"dataFormat": "tsbuckets",
"datasource": "Prometheus",
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 84
"y": 139
},
"heatmap": {},
"hideZeroBuckets": true,
@ -5567,7 +5700,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@ -5609,7 +5741,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -5630,7 +5762,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 39
"y": 66
},
"hiddenSeries": false,
"id": 2,
@ -5754,7 +5886,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 39
"y": 66
},
"hiddenSeries": false,
"id": 41,
@ -5847,7 +5979,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 46
"y": 73
},
"hiddenSeries": false,
"id": 42,
@ -5939,7 +6071,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 46
"y": 73
},
"hiddenSeries": false,
"id": 43,
@ -6031,7 +6163,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 53
"y": 80
},
"hiddenSeries": false,
"id": 113,
@ -6129,7 +6261,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 53
"y": 80
},
"hiddenSeries": false,
"id": 115,
@ -6215,7 +6347,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -6236,7 +6368,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 58
"y": 40
},
"hiddenSeries": false,
"id": 67,
@ -6267,7 +6399,7 @@
"steppedLine": false,
"targets": [
{
"expr": " synapse_event_persisted_position{instance=\"$instance\",job=\"synapse\"} - ignoring(index, job, name) group_right() synapse_event_processing_positions{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"expr": "max(synapse_event_persisted_position{instance=\"$instance\"}) - ignoring(instance,index, job, name) group_right() synapse_event_processing_positions{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
@ -6328,7 +6460,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 58
"y": 40
},
"hiddenSeries": false,
"id": 71,
@ -6362,6 +6494,7 @@
"expr": "time()*1000-synapse_event_processing_last_ts{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{job}}-{{index}} {{name}}",
"refId": "B"
@ -6420,7 +6553,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 67
"y": 49
},
"hiddenSeries": false,
"id": 121,
@ -6509,7 +6642,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@ -6539,7 +6672,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 41
"y": 86
},
"heatmap": {},
"hideZeroBuckets": true,
@ -6549,7 +6682,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@ -6599,7 +6731,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 41
"y": 86
},
"hiddenSeries": false,
"id": 124,
@ -6700,7 +6832,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 49
"y": 94
},
"heatmap": {},
"hideZeroBuckets": true,
@ -6710,7 +6842,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@ -6760,7 +6891,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 49
"y": 94
},
"hiddenSeries": false,
"id": 128,
@ -6879,7 +7010,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 57
"y": 102
},
"heatmap": {},
"hideZeroBuckets": true,
@ -6889,7 +7020,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@ -6939,7 +7069,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 57
"y": 102
},
"hiddenSeries": false,
"id": 130,
@ -7058,7 +7188,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 65
"y": 110
},
"heatmap": {},
"hideZeroBuckets": true,
@ -7068,12 +7198,12 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
"expr": "rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0)",
"expr": "rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size])",
"format": "heatmap",
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{le}}",
"refId": "A"
@ -7118,7 +7248,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 65
"y": 110
},
"hiddenSeries": false,
"id": 132,
@ -7149,29 +7279,33 @@
"steppedLine": false,
"targets": [
{
"expr": "histogram_quantile(0.5, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0)) ",
"expr": "histogram_quantile(0.5, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "50%",
"refId": "A"
},
{
"expr": "histogram_quantile(0.75, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
"expr": "histogram_quantile(0.75, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "75%",
"refId": "B"
},
{
"expr": "histogram_quantile(0.90, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
"expr": "histogram_quantile(0.90, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "90%",
"refId": "C"
},
{
"expr": "histogram_quantile(0.99, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
"expr": "histogram_quantile(0.99, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "99%",
"refId": "D"
@ -7181,7 +7315,7 @@
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Number of state resolution performed, by number of state groups involved (quantiles)",
"title": "Number of state resolutions performed, by number of state groups involved (quantiles)",
"tooltip": {
"shared": true,
"sort": 0,
@ -7233,6 +7367,7 @@
"list": [
{
"current": {
"selected": false,
"text": "Prometheus",
"value": "Prometheus"
},
@ -7309,14 +7444,12 @@
},
{
"allValue": null,
"current": {
"text": "matrix.org",
"value": "matrix.org"
},
"current": {},
"datasource": "$datasource",
"definition": "",
"hide": 0,
"includeAll": false,
"index": -1,
"label": null,
"multi": false,
"name": "instance",
@ -7335,17 +7468,13 @@
{
"allFormat": "regex wildcard",
"allValue": "",
"current": {
"text": "synapse",
"value": [
"synapse"
]
},
"current": {},
"datasource": "$datasource",
"definition": "",
"hide": 0,
"hideLabel": false,
"includeAll": true,
"index": -1,
"label": "Job",
"multi": true,
"multiFormat": "regex values",
@ -7366,16 +7495,13 @@
{
"allFormat": "regex wildcard",
"allValue": ".*",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"current": {},
"datasource": "$datasource",
"definition": "",
"hide": 0,
"hideLabel": false,
"includeAll": true,
"index": -1,
"label": "",
"multi": true,
"multiFormat": "regex values",
@ -7428,5 +7554,8 @@
"timezone": "",
"title": "Synapse",
"uid": "000000012",
"version": 29
"variables": {
"list": []
},
"version": 32
}

21
contrib/graph/graph.py
View File

@ -1,5 +1,13 @@
from __future__ import print_function
import argparse
import cgi
import datetime
import json
import pydot
import urllib2
# Copyright 2014-2016 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
@ -15,15 +23,6 @@ from __future__ import print_function
# limitations under the License.
import sqlite3
import pydot
import cgi
import json
import datetime
import argparse
import urllib2
def make_name(pdu_id, origin):
return "%s@%s" % (pdu_id, origin)
@ -33,7 +32,7 @@ def make_graph(pdus, room, filename_prefix):
node_map = {}
origins = set()
colors = set(("red", "green", "blue", "yellow", "purple"))
colors = {"red", "green", "blue", "yellow", "purple"}
for pdu in pdus:
origins.add(pdu.get("origin"))
@ -49,7 +48,7 @@ def make_graph(pdus, room, filename_prefix):
try:
c = colors.pop()
color_map[o] = c
except:
except Exception:
print("Run out of colours!")
color_map[o] = "black"

13
contrib/graph/graph2.py
View File

@ -13,12 +13,13 @@
# limitations under the License.
import sqlite3
import pydot
import cgi
import json
import datetime
import argparse
import cgi
import datetime
import json
import sqlite3
import pydot
from synapse.events import FrozenEvent
from synapse.util.frozenutils import unfreeze
@ -98,7 +99,7 @@ def make_graph(db_name, room_id, file_prefix, limit):
for prev_id, _ in event.prev_events:
try:
end_node = node_map[prev_id]
except:
except Exception:
end_node = pydot.Node(name=prev_id, label="<<b>%s</b>>" % (prev_id,))
node_map[prev_id] = end_node

22
contrib/graph/graph3.py
View File

@ -1,5 +1,15 @@
from __future__ import print_function
import argparse
import cgi
import datetime
import pydot
import simplejson as json
from synapse.events import FrozenEvent
from synapse.util.frozenutils import unfreeze
# Copyright 2016 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
@ -15,16 +25,6 @@ from __future__ import print_function
# limitations under the License.
import pydot
import cgi
import simplejson as json
import datetime
import argparse
from synapse.events import FrozenEvent
from synapse.util.frozenutils import unfreeze
def make_graph(file_name, room_id, file_prefix, limit):
print("Reading lines")
with open(file_name) as f:
@ -106,7 +106,7 @@ def make_graph(file_name, room_id, file_prefix, limit):
for prev_id, _ in event.prev_events:
try:
end_node = node_map[prev_id]
except:
except Exception:
end_node = pydot.Node(name=prev_id, label="<<b>%s</b>>" % (prev_id,))
node_map[prev_id] = end_node

10
contrib/jitsimeetbridge/jitsimeetbridge.py
View File

@ -12,15 +12,15 @@ npm install jquery jsdom
"""
from __future__ import print_function
import gevent
import grequests
from BeautifulSoup import BeautifulSoup
import json
import urllib
import subprocess
import time
# ACCESS_TOKEN="" #
import gevent
import grequests
from BeautifulSoup import BeautifulSoup
ACCESS_TOKEN = ""
MATRIXBASE = "https://matrix.org/_matrix/client/api/v1/"
MYUSERNAME = "@davetest:matrix.org"

6
contrib/scripts/kick_users.py
View File

@ -1,10 +1,12 @@
#!/usr/bin/env python
from __future__ import print_function
from argparse import ArgumentParser
import json
import requests
import sys
import urllib
from argparse import ArgumentParser
import requests
try:
raw_input

18
debian/changelog vendored
View File

@ -1,3 +1,21 @@
matrix-synapse-py3 (1.17.0) stable; urgency=medium
* New synapse release 1.17.0.
-- Synapse Packaging team <packages@matrix.org> Mon, 13 Jul 2020 10:20:31 +0100
matrix-synapse-py3 (1.16.1) stable; urgency=medium
* New synapse release 1.16.1.
-- Synapse Packaging team <packages@matrix.org> Fri, 10 Jul 2020 12:09:24 +0100
matrix-synapse-py3 (1.17.0rc1) stable; urgency=medium
* New synapse release 1.17.0rc1.
-- Synapse Packaging team <packages@matrix.org> Thu, 09 Jul 2020 16:53:12 +0100
matrix-synapse-py3 (1.16.0) stable; urgency=medium
* New synapse release 1.16.0.

Some files were not shown because too many files have changed in this diff Show More