Merge branch 'develop' into cross-signing_hidden

pull/5759/head
Hubert Chathi 2019-07-30 23:33:06 -04:00
commit c1f0a5636c
298 changed files with 3402 additions and 2125 deletions

View File

@ -49,14 +49,15 @@ steps:
- command:
- "python -m pip install tox"
- "apt-get update && apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev zlib1g-dev"
- "python3.5 -m pip install tox"
- "tox -e py35-old,codecov"
label: ":python: 3.5 / SQLite / Old Deps"
env:
TRIAL_FLAGS: "-j 2"
plugins:
- docker#v3.0.1:
image: "python:3.5"
image: "ubuntu:xenial" # We use xenail to get an old sqlite and python
propagate-environment: true
retry:
automatic:
@ -220,8 +221,10 @@ steps:
env:
POSTGRES: "1"
WORKERS: "1"
BLACKLIST: "synapse-blacklist-with-workers"
command:
- "bash .buildkite/merge_base_branch.sh"
- "bash -c 'cat /src/sytest-blacklist /src/.buildkite/worker-blacklist > /src/synapse-blacklist-with-workers'"
- "bash /synapse_sytest.sh"
plugins:
- docker#v3.0.1:
@ -229,7 +232,6 @@ steps:
propagate-environment: true
always-pull: true
workdir: "/src"
soft_fail: true
retry:
automatic:
- exit_status: -1

View File

@ -0,0 +1,34 @@
# This file serves as a blacklist for SyTest tests that we expect will fail in
# Synapse when run under worker mode. For more details, see sytest-blacklist.
Message history can be paginated
m.room.history_visibility == "world_readable" allows/forbids appropriately for Guest users
m.room.history_visibility == "world_readable" allows/forbids appropriately for Real users
Can re-join room if re-invited
/upgrade creates a new room
The only membership state included in an initial sync is for all the senders in the timeline
Local device key changes get to remote servers
If remote user leaves room we no longer receive device updates
Forgotten room messages cannot be paginated
Inbound federation can get public room list
Members from the gap are included in gappy incr LL sync
Leaves are present in non-gapped incremental syncs
Old leaves are present in gapped incremental syncs
User sees updates to presence from other users in the incremental sync.
Gapped incremental syncs include all state changes
Old members are included in gappy incr LL sync if they start speaking

View File

@ -1,3 +1,125 @@
Synapse 1.2.1 (2019-07-26)
==========================
Security update
---------------
This release includes *four* security fixes:
- Prevent an attack where a federated server could send redactions for arbitrary events in v1 and v2 rooms. ([\#5767](https://github.com/matrix-org/synapse/issues/5767))
- Prevent a denial-of-service attack where cycles of redaction events would make Synapse spin infinitely. Thanks to `@lrizika:matrix.org` for identifying and responsibly disclosing this issue. ([0f2ecb961](https://github.com/matrix-org/synapse/commit/0f2ecb961))
- Prevent an attack where users could be joined or parted from public rooms without their consent. Thanks to @dylangerdaly for identifying and responsibly disclosing this issue. ([\#5744](https://github.com/matrix-org/synapse/issues/5744))
- Fix a vulnerability where a federated server could spoof read-receipts from
users on other servers. Thanks to @dylangerdaly for identifying this issue too. ([\#5743](https://github.com/matrix-org/synapse/issues/5743))
Additionally, the following fix was in Synapse **1.2.0**, but was not correctly
identified during the original release:
- It was possible for a room moderator to send a redaction for an `m.room.create` event, which would downgrade the room to version 1. Thanks to `/dev/ponies` for identifying and responsibly disclosing this issue! ([\#5701](https://github.com/matrix-org/synapse/issues/5701))
Synapse 1.2.0 (2019-07-25)
==========================
No significant changes.
Synapse 1.2.0rc2 (2019-07-24)
=============================
Bugfixes
--------
- Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
Synapse 1.2.0rc1 (2019-07-22)
=============================
Security fixes
--------------
This update included a security fix which was initially incorrectly flagged as
a regular bug fix.
- It was possible for a room moderator to send a redaction for an `m.room.create` event, which would downgrade the room to version 1. Thanks to `/dev/ponies` for identifying and responsibly disclosing this issue! ([\#5701](https://github.com/matrix-org/synapse/issues/5701))
Features
--------
- Add support for opentracing. ([\#5544](https://github.com/matrix-org/synapse/issues/5544), [\#5712](https://github.com/matrix-org/synapse/issues/5712))
- Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](https://github.com/matrix-org/synapse/issues/5589))
- Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](https://github.com/matrix-org/synapse/issues/5597))
- Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](https://github.com/matrix-org/synapse/issues/5613))
- Add default push rule to ignore reactions. ([\#5623](https://github.com/matrix-org/synapse/issues/5623))
- Include the original event when asking for its relations. ([\#5626](https://github.com/matrix-org/synapse/issues/5626))
- Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](https://github.com/matrix-org/synapse/issues/5660))
- Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](https://github.com/matrix-org/synapse/issues/5674))
- Enable aggregations support by default ([\#5714](https://github.com/matrix-org/synapse/issues/5714))
Bugfixes
--------
- Fix 'utime went backwards' errors on daemonization. ([\#5609](https://github.com/matrix-org/synapse/issues/5609))
- Various minor fixes to the federation request rate limiter. ([\#5621](https://github.com/matrix-org/synapse/issues/5621))
- Forbid viewing relations on an event once it has been redacted. ([\#5629](https://github.com/matrix-org/synapse/issues/5629))
- Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](https://github.com/matrix-org/synapse/issues/5638))
- Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](https://github.com/matrix-org/synapse/issues/5644))
- Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](https://github.com/matrix-org/synapse/issues/5654))
- Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](https://github.com/matrix-org/synapse/issues/5658))
- Fix some problems with authenticating redactions in recent room versions. ([\#5699](https://github.com/matrix-org/synapse/issues/5699), [\#5700](https://github.com/matrix-org/synapse/issues/5700), [\#5707](https://github.com/matrix-org/synapse/issues/5707))
Updates to the Docker image
---------------------------
- Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](https://github.com/matrix-org/synapse/issues/5619))
- Add missing space in default logging file format generated by the Docker image. ([\#5620](https://github.com/matrix-org/synapse/issues/5620))
Improved Documentation
----------------------
- Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](https://github.com/matrix-org/synapse/issues/5397))
- --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](https://github.com/matrix-org/synapse/issues/5651))
- Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](https://github.com/matrix-org/synapse/issues/5661))
- Minor tweaks to postgres documentation. ([\#5675](https://github.com/matrix-org/synapse/issues/5675))
Deprecations and Removals
-------------------------
- Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](https://github.com/matrix-org/synapse/issues/5625))
Internal Changes
----------------
- Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](https://github.com/matrix-org/synapse/issues/5606), [\#5617](https://github.com/matrix-org/synapse/issues/5617))
- Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](https://github.com/matrix-org/synapse/issues/5611))
- Make runtime errors surrounding password reset emails much clearer. ([\#5616](https://github.com/matrix-org/synapse/issues/5616))
- Remove dead code for persiting outgoing federation transactions. ([\#5622](https://github.com/matrix-org/synapse/issues/5622))
- Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](https://github.com/matrix-org/synapse/issues/5627))
- Move RegistrationHandler.get_or_create_user to test code. ([\#5628](https://github.com/matrix-org/synapse/issues/5628))
- Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](https://github.com/matrix-org/synapse/issues/5630))
- Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](https://github.com/matrix-org/synapse/issues/5636))
- Unblacklist some user_directory sytests. ([\#5637](https://github.com/matrix-org/synapse/issues/5637))
- Factor out some redundant code in the login implementation. ([\#5639](https://github.com/matrix-org/synapse/issues/5639))
- Update ModuleApi to avoid register(generate_token=True). ([\#5640](https://github.com/matrix-org/synapse/issues/5640))
- Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](https://github.com/matrix-org/synapse/issues/5641))
- Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](https://github.com/matrix-org/synapse/issues/5642))
- Improve logging for auto-join when a new user is created. ([\#5643](https://github.com/matrix-org/synapse/issues/5643))
- Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](https://github.com/matrix-org/synapse/issues/5645))
- Fix a small typo in a code comment. ([\#5655](https://github.com/matrix-org/synapse/issues/5655))
- Clean up exception handling around client access tokens. ([\#5656](https://github.com/matrix-org/synapse/issues/5656))
- Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](https://github.com/matrix-org/synapse/issues/5657))
- Inline issue_access_token. ([\#5659](https://github.com/matrix-org/synapse/issues/5659))
- Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](https://github.com/matrix-org/synapse/issues/5664))
- Add a `docker` type to the towncrier configuration. ([\#5673](https://github.com/matrix-org/synapse/issues/5673))
- Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](https://github.com/matrix-org/synapse/issues/5689))
- Documentation for opentracing. ([\#5703](https://github.com/matrix-org/synapse/issues/5703))
Synapse 1.1.0 (2019-07-04)
==========================

View File

@ -7,7 +7,6 @@ include demo/README
include demo/demo.tls.dh
include demo/*.py
include demo/*.sh
include sytest-blacklist
recursive-include synapse/storage/schema *.sql
recursive-include synapse/storage/schema *.sql.postgres
@ -34,6 +33,7 @@ exclude Dockerfile
exclude .dockerignore
exclude test_postgresql.sh
exclude .editorconfig
exclude sytest-blacklist
include pyproject.toml
recursive-include changelog.d *

View File

@ -1 +0,0 @@
Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks!

View File

@ -1,2 +0,0 @@
Add support for opentracing.

View File

@ -1 +0,0 @@
Add ability to pull all locally stored events out of synapse that a particular user can see.

View File

@ -1 +0,0 @@
Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance.

View File

@ -1 +0,0 @@
Move logging code out of `synapse.util` and into `synapse.logging`.

View File

@ -1 +0,0 @@
Fix 'utime went backwards' errors on daemonization.

View File

@ -1 +0,0 @@
Add a blacklist file to the repo to blacklist certain sytests from failing CI.

View File

@ -1 +0,0 @@
Add `sender` and `origin_server_ts` fields to `m.replace`.

View File

@ -1 +0,0 @@
Make runtime errors surrounding password reset emails much clearer.

View File

@ -1 +0,0 @@
Move logging code out of `synapse.util` and into `synapse.logging`.

View File

@ -1 +0,0 @@
Base Docker image on a newer Alpine Linux version (3.8 -> 3.10).

View File

@ -1 +0,0 @@
Add missing space in default logging file format generated by the Docker image.

View File

@ -1 +0,0 @@
Various minor fixes to the federation request rate limiter.

View File

@ -1 +0,0 @@
Remove dead code for persiting outgoing federation transactions.

View File

@ -1 +0,0 @@
Add default push rule to ignore reactions.

View File

@ -1 +0,0 @@
Remove support for the `invite_3pid_guest` configuration setting.

View File

@ -1 +0,0 @@
Include the original event when asking for its relations.

View File

@ -1 +0,0 @@
Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI.

View File

@ -1 +0,0 @@
Move RegistrationHandler.get_or_create_user to test code.

View File

@ -1 +0,0 @@
Forbid viewing relations on an event once it has been redacted.

View File

@ -1 +0,0 @@
Add some more common python virtual-environment paths to the black exclusion list.

View File

@ -1 +0,0 @@
Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details.

View File

@ -1 +0,0 @@
Unblacklist some user_directory sytests.

View File

@ -1 +0,0 @@
Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format.

View File

@ -1 +0,0 @@
Factor out some redundant code in the login implementation.

View File

@ -1 +0,0 @@
Update ModuleApi to avoid register(generate_token=True).

View File

@ -1 +0,0 @@
Remove access-token support from RegistrationHandler.register, and rename it.

View File

@ -1 +0,0 @@
Remove access-token support from `RegistrationStore.register`, and rename it.

View File

@ -1 +0,0 @@
Improve logging for auto-join when a new user is created.

View File

@ -1 +0,0 @@
Fix newly-registered users not being able to lookup their own profile without joining a room.

View File

@ -1 +0,0 @@
Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure.

View File

@ -1 +0,0 @@
--no-pep517 should be --no-use-pep517 in the documentation to setup the development environment.

View File

@ -1 +0,0 @@
Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`.

View File

@ -1 +0,0 @@
Fix a small typo in a code comment.

View File

@ -1 +0,0 @@
Clean up exception handling around client access tokens.

View File

@ -1 +0,0 @@
Add a mechanism for per-test homeserver configuration in the unit tests.

View File

@ -1 +0,0 @@
Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated.

View File

@ -1 +0,0 @@
Inline issue_access_token.

View File

@ -1 +0,0 @@
Implement `session_lifetime` configuration option, after which access tokens will expire.

View File

@ -1 +0,0 @@
Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks!

View File

@ -1 +0,0 @@
Update the sytest BuildKite configuration to checkout Synapse in `/src`.

View File

@ -1 +0,0 @@
Add a `docker` type to the towncrier configuration.

View File

@ -1 +0,0 @@
Return "This account has been deactivated" when a deactivated user tries to login.

View File

@ -1 +0,0 @@
Minor tweaks to postgres documentation.

View File

@ -1 +0,0 @@
Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces.

1
changelog.d/5693.bugfix Normal file
View File

@ -0,0 +1 @@
Fix UISIs during homeserver outage.

1
changelog.d/5694.misc Normal file
View File

@ -0,0 +1 @@
Make Jaeger fully configurable.

View File

@ -1 +0,0 @@
Fix some problems with authenticating redactions in recent room versions.

View File

@ -1,2 +0,0 @@
Fix some problems with authenticating redactions in recent room versions.

View File

@ -1 +0,0 @@
Ignore redactions of m.room.create events.

View File

@ -1 +0,0 @@
Fix some problems with authenticating redactions in recent room versions.

View File

@ -1,2 +0,0 @@
Add support for opentracing.

View File

@ -1 +0,0 @@
Enable aggregations support by default

1
changelog.d/5722.misc Normal file
View File

@ -0,0 +1 @@
Add a set of opentracing utils.

1
changelog.d/5724.bugfix Normal file
View File

@ -0,0 +1 @@
Fix stack overflow in server key lookup code.

1
changelog.d/5725.bugfix Normal file
View File

@ -0,0 +1 @@
start.sh no longer uses deprecated cli option.

1
changelog.d/5729.removal Normal file
View File

@ -0,0 +1 @@
Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration.

1
changelog.d/5730.misc Normal file
View File

@ -0,0 +1 @@
Cache result of get_version_string to reduce overhead of `/version` federation requests.

1
changelog.d/5731.misc Normal file
View File

@ -0,0 +1 @@
Return 'user_type' in admin API user endpoints results.

1
changelog.d/5732.feature Normal file
View File

@ -0,0 +1 @@
Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify.

1
changelog.d/5733.misc Normal file
View File

@ -0,0 +1 @@
Don't package the sytest test blacklist file.

1
changelog.d/5736.misc Normal file
View File

@ -0,0 +1 @@
Replace uses of returnValue with plain return, as returnValue is not needed on Python 3.

1
changelog.d/5738.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

1
changelog.d/5740.misc Normal file
View File

@ -0,0 +1 @@
Blacklist some flakey tests in worker mode.

1
changelog.d/5743.bugfix Normal file
View File

@ -0,0 +1 @@
Log when we receive an event receipt from an unexpected origin.

1
changelog.d/5746.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

1
changelog.d/5749.misc Normal file
View File

@ -0,0 +1 @@
Fix some error cases in the caching layer.

1
changelog.d/5750.misc Normal file
View File

@ -0,0 +1 @@
Add a prometheus metric for pending cache lookups.

1
changelog.d/5752.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

1
changelog.d/5753.misc Normal file
View File

@ -0,0 +1 @@
Stop trying to fetch events with event_id=None.

1
changelog.d/5768.misc Normal file
View File

@ -0,0 +1 @@
Convert RedactionTestCase to modern test style.

1
changelog.d/5770.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

1
changelog.d/5774.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

1
changelog.d/5775.bugfix Normal file
View File

@ -0,0 +1 @@
Fix debian packaging scripts to correctly build sid packages.

1
changelog.d/5780.misc Normal file
View File

@ -0,0 +1 @@
Allow looping calls to be given arguments.

1
changelog.d/5782.removal Normal file
View File

@ -0,0 +1 @@
Remove non-functional 'expire_access_token' setting.

1
changelog.d/5783.feature Normal file
View File

@ -0,0 +1 @@
Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers.

1
changelog.d/5785.misc Normal file
View File

@ -0,0 +1 @@
Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO.

1
changelog.d/5789.bugfix Normal file
View File

@ -0,0 +1 @@
Fix UISIs during homeserver outage.

1
changelog.d/5792.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

1
changelog.d/5793.misc Normal file
View File

@ -0,0 +1 @@
Reduce database IO usage by optimising queries for current membership.

View File

@ -4,7 +4,8 @@ After=matrix-synapse.service
BindsTo=matrix-synapse.service
[Service]
Type=simple
Type=notify
NotifyAccess=main
User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=/etc/default/matrix-synapse

View File

@ -2,7 +2,8 @@
Description=Synapse Matrix Homeserver
[Service]
Type=simple
Type=notify
NotifyAccess=main
User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=/etc/default/matrix-synapse

View File

@ -14,7 +14,9 @@
Description=Synapse Matrix homeserver
[Service]
Type=simple
Type=notify
NotifyAccess=main
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-abort
User=synapse

13
debian/changelog vendored
View File

@ -1,4 +1,10 @@
matrix-synapse-py3 (1.1.0-1) UNRELEASED; urgency=medium
matrix-synapse-py3 (1.2.1) stable; urgency=medium
* New synapse release 1.2.1.
-- Synapse Packaging team <packages@matrix.org> Fri, 26 Jul 2019 11:32:47 +0100
matrix-synapse-py3 (1.2.0) stable; urgency=medium
[ Amber Brown ]
* Update logging config defaults to match API changes in Synapse.
@ -6,7 +12,10 @@ matrix-synapse-py3 (1.1.0-1) UNRELEASED; urgency=medium
[ Richard van der Hoff ]
* Add Recommends and Depends for some libraries which you probably want.
-- Erik Johnston <erikj@rae> Thu, 04 Jul 2019 13:59:02 +0100
[ Synapse Packaging team ]
* New synapse release 1.2.0.
-- Synapse Packaging team <packages@matrix.org> Thu, 25 Jul 2019 14:10:07 +0100
matrix-synapse-py3 (1.1.0) stable; urgency=medium

View File

@ -29,7 +29,7 @@ for port in 8080 8081 8082; do
if ! grep -F "Customisation made by demo/start.sh" -q $DIR/etc/$port.config; then
printf '\n\n# Customisation made by demo/start.sh\n' >> $DIR/etc/$port.config
echo 'enable_registration: true' >> $DIR/etc/$port.config
# Warning, this heredoc depends on the interaction of tabs and spaces. Please don't
@ -43,7 +43,7 @@ for port in 8080 8081 8082; do
tls: true
resources:
- names: [client, federation]
- port: $port
tls: false
bind_addresses: ['::1', '127.0.0.1']
@ -68,7 +68,7 @@ for port in 8080 8081 8082; do
# Generate tls keys
openssl req -x509 -newkey rsa:4096 -keyout $DIR/etc/localhost\:$https_port.tls.key -out $DIR/etc/localhost\:$https_port.tls.crt -days 365 -nodes -subj "/O=matrix"
# Ignore keys from the trusted keys server
echo '# Ignore keys from the trusted keys server' >> $DIR/etc/$port.config
echo 'trusted_key_servers:' >> $DIR/etc/$port.config
@ -120,7 +120,6 @@ for port in 8080 8081 8082; do
python3 -m synapse.app.homeserver \
--config-path "$DIR/etc/$port.config" \
-D \
-vv \
popd
done

View File

@ -42,6 +42,11 @@ RUN cd dh-virtualenv-1.1 && dpkg-buildpackage -us -uc -b
###
FROM ${distro}
# Get the distro we want to pull from as a dynamic build variable
# (We need to define it in each build stage)
ARG distro=""
ENV distro ${distro}
# Install the build dependencies
#
# NB: keep this list in sync with the list of build-deps in debian/control

View File

@ -4,7 +4,8 @@
set -ex
DIST=`lsb_release -c -s`
# Get the codename from distro env
DIST=`cut -d ':' -f2 <<< $distro`
# we get a read-only copy of the source: make a writeable copy
cp -aT /synapse/source /synapse/build

View File

@ -148,7 +148,7 @@ call any other functions.
d = more_stuff()
result = yield d # also fine, of course
defer.returnValue(result)
return result
def nonInlineCallbacksFun():
logger.debug("just a wrapper really")

100
docs/opentracing.rst Normal file
View File

@ -0,0 +1,100 @@
===========
OpenTracing
===========
Background
----------
OpenTracing is a semi-standard being adopted by a number of distributed tracing
platforms. It is a common api for facilitating vendor-agnostic tracing
instrumentation. That is, we can use the OpenTracing api and select one of a
number of tracer implementations to do the heavy lifting in the background.
Our current selected implementation is Jaeger.
OpenTracing is a tool which gives an insight into the causal relationship of
work done in and between servers. The servers each track events and report them
to a centralised server - in Synapse's case: Jaeger. The basic unit used to
represent events is the span. The span roughly represents a single piece of work
that was done and the time at which it occurred. A span can have child spans,
meaning that the work of the child had to be completed for the parent span to
complete, or it can have follow-on spans which represent work that is undertaken
as a result of the parent but is not depended on by the parent to in order to
finish.
Since this is undertaken in a distributed environment a request to another
server, such as an RPC or a simple GET, can be considered a span (a unit or
work) for the local server. This causal link is what OpenTracing aims to
capture and visualise. In order to do this metadata about the local server's
span, i.e the 'span context', needs to be included with the request to the
remote.
It is up to the remote server to decide what it does with the spans
it creates. This is called the sampling policy and it can be configured
through Jaeger's settings.
For OpenTracing concepts see
https://opentracing.io/docs/overview/what-is-tracing/.
For more information about Jaeger's implementation see
https://www.jaegertracing.io/docs/
=====================
Seting up OpenTracing
=====================
To receive OpenTracing spans, start up a Jaeger server. This can be done
using docker like so:
.. code-block:: bash
docker run -d --name jaeger
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
jaegertracing/all-in-one:1.13
Latest documentation is probably at
https://www.jaegertracing.io/docs/1.13/getting-started/
Enable OpenTracing in Synapse
-----------------------------
OpenTracing is not enabled by default. It must be enabled in the homeserver
config by uncommenting the config options under ``opentracing`` as shown in
the `sample config <./sample_config.yaml>`_. For example:
.. code-block:: yaml
opentracing:
tracer_enabled: true
homeserver_whitelist:
- "mytrustedhomeserver.org"
- "*.myotherhomeservers.com"
Homeserver whitelisting
-----------------------
The homeserver whitelist is configured using regular expressions. A list of regular
expressions can be given and their union will be compared when propagating any
spans contexts to another homeserver.
Though it's mostly safe to send and receive span contexts to and from
untrusted users since span contexts are usually opaque ids it can lead to
two problems, namely:
- If the span context is marked as sampled by the sending homeserver the receiver will
sample it. Therefore two homeservers with wildly different sampling policies
could incur higher sampling counts than intended.
- Sending servers can attach arbitrary data to spans, known as 'baggage'. For safety this has been disabled in Synapse
but that doesn't prevent another server sending you baggage which will be logged
to OpenTracing's logs.
==================
Configuring Jaeger
==================
Sampling strategies can be set as in this document:
https://www.jaegertracing.io/docs/1.13/sampling/

View File

@ -278,6 +278,23 @@ listeners:
# Used by phonehome stats to group together related servers.
#server_context: context
# Resource-constrained Homeserver Settings
#
# If limit_remote_rooms.enabled is True, the room complexity will be
# checked before a user joins a new remote room. If it is above
# limit_remote_rooms.complexity, it will disallow joining or
# instantly leave.
#
# limit_remote_rooms.complexity_error can be set to customise the text
# displayed to the user when a room above the complexity threshold has
# its join cancelled.
#
# Uncomment the below lines to enable:
#limit_remote_rooms:
# enabled: True
# complexity: 1.0
# complexity_error: "This room is too complex."
# Whether to require a user to be in the room to add an alias to it.
# Defaults to 'true'.
#
@ -925,10 +942,6 @@ uploads_path: "DATADIR/uploads"
#
# macaroon_secret_key: <PRIVATE STRING>
# Used to enable access token expiration.
#
#expire_access_token: False
# a secret which is used to calculate HMACs for form values, to stop
# falsification of values. Must be specified for the User Consent
# forms to work.
@ -1422,21 +1435,27 @@ opentracing:
#enabled: true
# The list of homeservers we wish to send and receive span contexts and span baggage.
#
# Though it's mostly safe to send and receive span contexts to and from
# untrusted users since span contexts are usually opaque ids it can lead to
# two problems, namely:
# - If the span context is marked as sampled by the sending homeserver the receiver will
# sample it. Therefore two homeservers with wildly disparaging sampling policies
# could incur higher sampling counts than intended.
# - Span baggage can be arbitrary data. For safety this has been disabled in synapse
# but that doesn't prevent another server sending you baggage which will be logged
# to opentracing logs.
#
# This a list of regexes which are matched against the server_name of the
# See docs/opentracing.rst
# This is a list of regexes which are matched against the server_name of the
# homeserver.
#
# By defult, it is empty, so no servers are matched.
#
#homeserver_whitelist:
# - ".*"
# Jaeger can be configured to sample traces at different rates.
# All configuration options provided by Jaeger can be set here.
# Jaeger's configuration mostly related to trace sampling which
# is documented here:
# https://www.jaegertracing.io/docs/1.13/sampling/.
#
#jaeger_config:
# sampler:
# type: const
# param: 1
# Logging whether spans were started and reported
#
# logging:
# false

View File

@ -35,4 +35,4 @@ try:
except ImportError:
pass
__version__ = "1.1.0"
__version__ = "1.2.1"

View File

@ -128,7 +128,7 @@ class Auth(object):
)
self._check_joined_room(member, user_id, room_id)
defer.returnValue(member)
return member
@defer.inlineCallbacks
def check_user_was_in_room(self, room_id, user_id):
@ -156,13 +156,13 @@ class Auth(object):
if forgot:
raise AuthError(403, "User %s not in room %s" % (user_id, room_id))
defer.returnValue(member)
return member
@defer.inlineCallbacks
def check_host_in_room(self, room_id, host):
with Measure(self.clock, "check_host_in_room"):
latest_event_ids = yield self.store.is_host_joined(room_id, host)
defer.returnValue(latest_event_ids)
return latest_event_ids
def _check_joined_room(self, member, user_id, room_id):
if not member or member.membership != Membership.JOIN:
@ -219,9 +219,7 @@ class Auth(object):
device_id="dummy-device", # stubbed
)
defer.returnValue(
synapse.types.create_requester(user_id, app_service=app_service)
)
return synapse.types.create_requester(user_id, app_service=app_service)
user_info = yield self.get_user_by_access_token(access_token, rights)
user = user_info["user"]
@ -262,10 +260,8 @@ class Auth(object):
request.authenticated_entity = user.to_string()
defer.returnValue(
synapse.types.create_requester(
user, token_id, is_guest, device_id, app_service=app_service
)
return synapse.types.create_requester(
user, token_id, is_guest, device_id, app_service=app_service
)
except KeyError:
raise MissingClientTokenError()
@ -276,25 +272,25 @@ class Auth(object):
self.get_access_token_from_request(request)
)
if app_service is None:
defer.returnValue((None, None))
return (None, None)
if app_service.ip_range_whitelist:
ip_address = IPAddress(self.hs.get_ip_from_request(request))
if ip_address not in app_service.ip_range_whitelist:
defer.returnValue((None, None))
return (None, None)
if b"user_id" not in request.args:
defer.returnValue((app_service.sender, app_service))
return (app_service.sender, app_service)
user_id = request.args[b"user_id"][0].decode("utf8")
if app_service.sender == user_id:
defer.returnValue((app_service.sender, app_service))
return (app_service.sender, app_service)
if not app_service.is_interested_in_user(user_id):
raise AuthError(403, "Application service cannot masquerade as this user.")
if not (yield self.store.get_user_by_id(user_id)):
raise AuthError(403, "Application service has not registered this user")
defer.returnValue((user_id, app_service))
return (user_id, app_service)
@defer.inlineCallbacks
def get_user_by_access_token(self, token, rights="access"):
@ -330,7 +326,7 @@ class Auth(object):
msg="Access token has expired", soft_logout=True
)
defer.returnValue(r)
return r
# otherwise it needs to be a valid macaroon
try:
@ -378,7 +374,7 @@ class Auth(object):
}
else:
raise RuntimeError("Unknown rights setting %s", rights)
defer.returnValue(ret)
return ret
except (
_InvalidMacaroonException,
pymacaroons.exceptions.MacaroonException,
@ -414,21 +410,16 @@ class Auth(object):
try:
user_id = self.get_user_id_from_macaroon(macaroon)
has_expiry = False
guest = False
for caveat in macaroon.caveats:
if caveat.caveat_id.startswith("time "):
has_expiry = True
elif caveat.caveat_id == "guest = true":
if caveat.caveat_id == "guest = true":
guest = True
self.validate_macaroon(
macaroon, rights, self.hs.config.expire_access_token, user_id=user_id
)
self.validate_macaroon(macaroon, rights, user_id=user_id)
except (pymacaroons.exceptions.MacaroonException, TypeError, ValueError):
raise InvalidClientTokenError("Invalid macaroon passed.")
if not has_expiry and rights == "access":
if rights == "access":
self.token_cache[token] = (user_id, guest)
return user_id, guest
@ -454,7 +445,7 @@ class Auth(object):
return caveat.caveat_id[len(user_prefix) :]
raise InvalidClientTokenError("No user caveat in macaroon")
def validate_macaroon(self, macaroon, type_string, verify_expiry, user_id):
def validate_macaroon(self, macaroon, type_string, user_id):
"""
validate that a Macaroon is understood by and was signed by this server.
@ -462,7 +453,6 @@ class Auth(object):
macaroon(pymacaroons.Macaroon): The macaroon to validate
type_string(str): The kind of token required (e.g. "access",
"delete_pusher")
verify_expiry(bool): Whether to verify whether the macaroon has expired.
user_id (str): The user_id required
"""
v = pymacaroons.Verifier()
@ -475,19 +465,7 @@ class Auth(object):
v.satisfy_exact("type = " + type_string)
v.satisfy_exact("user_id = %s" % user_id)
v.satisfy_exact("guest = true")
# verify_expiry should really always be True, but there exist access
# tokens in the wild which expire when they should not, so we can't
# enforce expiry yet (so we have to allow any caveat starting with
# 'time < ' in access tokens).
#
# On the other hand, short-term login tokens (as used by CAS login, for
# example) have an expiry time which we do want to enforce.
if verify_expiry:
v.satisfy_general(self._verify_expiry)
else:
v.satisfy_general(lambda c: c.startswith("time < "))
v.satisfy_general(self._verify_expiry)
# access_tokens include a nonce for uniqueness: any value is acceptable
v.satisfy_general(lambda c: c.startswith("nonce = "))
@ -506,7 +484,7 @@ class Auth(object):
def _look_up_user_by_access_token(self, token):
ret = yield self.store.get_user_by_access_token(token)
if not ret:
defer.returnValue(None)
return None
# we use ret.get() below because *lots* of unit tests stub out
# get_user_by_access_token in a way where it only returns a couple of
@ -518,7 +496,7 @@ class Auth(object):
"device_id": ret.get("device_id"),
"valid_until_ms": ret.get("valid_until_ms"),
}
defer.returnValue(user_info)
return user_info
def get_appservice_by_req(self, request):
token = self.get_access_token_from_request(request)
@ -543,7 +521,7 @@ class Auth(object):
@defer.inlineCallbacks
def compute_auth_events(self, event, current_state_ids, for_verification=False):
if event.type == EventTypes.Create:
defer.returnValue([])
return []
auth_ids = []
@ -604,7 +582,7 @@ class Auth(object):
if member_event.content["membership"] == Membership.JOIN:
auth_ids.append(member_event.event_id)
defer.returnValue(auth_ids)
return auth_ids
@defer.inlineCallbacks
def check_can_change_room_list(self, room_id, user):
@ -618,7 +596,7 @@ class Auth(object):
is_admin = yield self.is_server_admin(user)
if is_admin:
defer.returnValue(True)
return True
user_id = user.to_string()
yield self.check_joined_room(room_id, user_id)
@ -712,7 +690,7 @@ class Auth(object):
# * The user is a guest user, and has joined the room
# else it will throw.
member_event = yield self.check_user_was_in_room(room_id, user_id)
defer.returnValue((member_event.membership, member_event.event_id))
return (member_event.membership, member_event.event_id)
except AuthError:
visibility = yield self.state.get_current_state(
room_id, EventTypes.RoomHistoryVisibility, ""
@ -721,7 +699,7 @@ class Auth(object):
visibility
and visibility.content["history_visibility"] == "world_readable"
):
defer.returnValue((Membership.JOIN, None))
return (Membership.JOIN, None)
return
raise AuthError(
403, "Guest access not allowed", errcode=Codes.GUEST_ACCESS_FORBIDDEN

View File

@ -132,7 +132,7 @@ class Filtering(object):
@defer.inlineCallbacks
def get_user_filter(self, user_localpart, filter_id):
result = yield self.store.get_user_filter(user_localpart, filter_id)
defer.returnValue(FilterCollection(result))
return FilterCollection(result)
def add_user_filter(self, user_localpart, user_filter):
self.check_valid_filter(user_filter)

View File

@ -15,10 +15,12 @@
import gc
import logging
import os
import signal
import sys
import traceback
import sdnotify
from daemonize import Daemonize
from twisted.internet import defer, error, reactor
@ -242,9 +244,16 @@ def start(hs, listeners=None):
if hasattr(signal, "SIGHUP"):
def handle_sighup(*args, **kwargs):
# Tell systemd our state, if we're using it. This will silently fail if
# we're not using systemd.
sd_channel = sdnotify.SystemdNotifier()
sd_channel.notify("RELOADING=1")
for i in _sighup_callbacks:
i(hs)
sd_channel.notify("READY=1")
signal.signal(signal.SIGHUP, handle_sighup)
register_sighup(refresh_certificate)
@ -260,6 +269,7 @@ def start(hs, listeners=None):
hs.get_datastore().start_profiling()
setup_sentry(hs)
setup_sdnotify(hs)
except Exception:
traceback.print_exc(file=sys.stderr)
reactor = hs.get_reactor()
@ -292,6 +302,25 @@ def setup_sentry(hs):
scope.set_tag("worker_name", name)
def setup_sdnotify(hs):
"""Adds process state hooks to tell systemd what we are up to.
"""
# Tell systemd our state, if we're using it. This will silently fail if
# we're not using systemd.
sd_channel = sdnotify.SystemdNotifier()
hs.get_reactor().addSystemEventTrigger(
"after",
"startup",
lambda: sd_channel.notify("READY=1\nMAINPID=%s" % (os.getpid())),
)
hs.get_reactor().addSystemEventTrigger(
"before", "shutdown", lambda: sd_channel.notify("STOPPING=1")
)
def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
"""Replaces the resolver with one that limits the number of in flight DNS
requests.

View File

@ -168,7 +168,9 @@ def start(config_options):
)
ps.setup()
reactor.callWhenRunning(_base.start, ps, config.worker_listeners)
reactor.addSystemEventTrigger(
"before", "startup", _base.start, ps, config.worker_listeners
)
_base.start_worker_reactor("synapse-appservice", config)

Some files were not shown because too many files have changed in this diff Show More