Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes
Erik Johnston
commit
fa0fc4acce
|
|
@ -69,7 +69,7 @@ with open('pyproject.toml', 'w') as f:
|
|||
"
|
||||
python3 -c "$REMOVE_DEV_DEPENDENCIES"
|
||||
|
||||
pipx install poetry==1.1.12
|
||||
pipx install poetry==1.1.14
|
||||
~/.local/bin/poetry lock
|
||||
|
||||
echo "::group::Patched pyproject.toml"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,16 @@
|
|||
# Commits in this file will be removed from GitHub blame results.
|
||||
#
|
||||
# To use this file locally, use:
|
||||
# git blame --ignore-revs-file="path/to/.git-blame-ignore-revs" <files>
|
||||
#
|
||||
# or configure the `blame.ignoreRevsFile` option in your git config.
|
||||
#
|
||||
# If ignoring a pull request that was not squash merged, only the merge
|
||||
# commit needs to be put here. Child commits will be resolved from it.
|
||||
|
||||
# Run black (#3679).
|
||||
8b3d9b6b199abb87246f982d5db356f1966db925
|
||||
|
||||
# Black reformatting (#5482).
|
||||
32e7c9e7f20b57dd081023ac42d6931a8da9b3a3
|
||||
|
||||
|
|
|
|||
|
|
@ -328,9 +328,6 @@ jobs:
|
|||
- arrangement: monolith
|
||||
database: Postgres
|
||||
|
||||
- arrangement: workers
|
||||
database: Postgres
|
||||
|
||||
steps:
|
||||
- name: Run actions/checkout@v2 for synapse
|
||||
uses: actions/checkout@v2
|
||||
|
|
@ -346,6 +343,30 @@ jobs:
|
|||
shell: bash
|
||||
name: Run Complement Tests
|
||||
|
||||
# XXX When complement with workers is stable, move this back into the standard
|
||||
# "complement" matrix above.
|
||||
#
|
||||
# See https://github.com/matrix-org/synapse/issues/13161
|
||||
complement-workers:
|
||||
if: "${{ !failure() && !cancelled() }}"
|
||||
needs: linting-done
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Run actions/checkout@v2 for synapse
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
path: synapse
|
||||
|
||||
- name: Prepare Complement's Prerequisites
|
||||
run: synapse/.ci/scripts/setup_complement_prerequisites.sh
|
||||
|
||||
- run: |
|
||||
set -o pipefail
|
||||
POSTGRES=1 WORKERS=1 COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | gotestfmt
|
||||
shell: bash
|
||||
name: Run Complement Tests
|
||||
|
||||
# a job which marks all the other jobs as complete, thus allowing PRs to be merged.
|
||||
tests-done:
|
||||
if: ${{ always() }}
|
||||
|
|
|
|||
|
|
@ -127,12 +127,12 @@ jobs:
|
|||
run: |
|
||||
set -x
|
||||
DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx
|
||||
pipx install poetry==1.1.12
|
||||
pipx install poetry==1.1.14
|
||||
|
||||
poetry remove -n twisted
|
||||
poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk
|
||||
poetry lock --no-update
|
||||
# NOT IN 1.1.12 poetry lock --check
|
||||
# NOT IN 1.1.14 poetry lock --check
|
||||
working-directory: synapse
|
||||
|
||||
- run: |
|
||||
|
|
|
|||
32
CHANGES.md
32
CHANGES.md
|
|
@ -1,3 +1,27 @@
|
|||
Synapse vNext
|
||||
=============
|
||||
|
||||
As of this release, Synapse no longer allows the tasks of verifying email address ownership, and password reset confirmation, to be delegated to an identity server. For more information, see the [upgrade notes](https://matrix-org.github.io/synapse/v1.64/upgrade.html#upgrading-to-v1640).
|
||||
|
||||
|
||||
Synapse 1.63.1 (2022-07-20)
|
||||
===========================
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix a bug introduced in Synapse 1.63.0 where push actions were incorrectly calculated for appservice users. This caused performance issues on servers with large numbers of appservices. ([\#13332](https://github.com/matrix-org/synapse/issues/13332))
|
||||
|
||||
|
||||
Synapse 1.63.0 (2022-07-19)
|
||||
===========================
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Clarify that homeserver server names are included in the reported data when the `report_stats` config option is enabled. ([\#13321](https://github.com/matrix-org/synapse/issues/13321))
|
||||
|
||||
|
||||
Synapse 1.63.0rc1 (2022-07-12)
|
||||
==============================
|
||||
|
||||
|
|
@ -6,7 +30,7 @@ Features
|
|||
|
||||
- Add a rate limit for local users sending invites. ([\#13125](https://github.com/matrix-org/synapse/issues/13125))
|
||||
- Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of `/publicRooms` by room type. ([\#13031](https://github.com/matrix-org/synapse/issues/13031))
|
||||
- Improve validation logic in Synapse's REST endpoints. ([\#13148](https://github.com/matrix-org/synapse/issues/13148))
|
||||
- Improve validation logic in the account data REST endpoints. ([\#13148](https://github.com/matrix-org/synapse/issues/13148))
|
||||
|
||||
|
||||
Bugfixes
|
||||
|
|
@ -34,7 +58,7 @@ Improved Documentation
|
|||
- Add an explanation of the `--report-stats` argument to the docs. ([\#13029](https://github.com/matrix-org/synapse/issues/13029))
|
||||
- Add a helpful example bash script to the contrib directory for creating multiple worker configuration files of the same type. Contributed by @villepeh. ([\#13032](https://github.com/matrix-org/synapse/issues/13032))
|
||||
- Add missing links to config options. ([\#13166](https://github.com/matrix-org/synapse/issues/13166))
|
||||
- Add documentation for anonymised homeserver statistics collection. ([\#13086](https://github.com/matrix-org/synapse/issues/13086))
|
||||
- Add documentation for homeserver usage statistics collection. ([\#13086](https://github.com/matrix-org/synapse/issues/13086))
|
||||
- Add documentation for the existing `databases` option in the homeserver configuration manual. ([\#13212](https://github.com/matrix-org/synapse/issues/13212))
|
||||
- Clean up references to sample configuration and redirect users to the configuration manual instead. ([\#13077](https://github.com/matrix-org/synapse/issues/13077), [\#13139](https://github.com/matrix-org/synapse/issues/13139))
|
||||
- Document how the Synapse team does reviews. ([\#13132](https://github.com/matrix-org/synapse/issues/13132))
|
||||
|
|
@ -73,7 +97,6 @@ Internal Changes
|
|||
- More aggressively rotate push actions. ([\#13211](https://github.com/matrix-org/synapse/issues/13211))
|
||||
- Add `max_line_length` setting for Python files to the `.editorconfig`. Contributed by @sumnerevans @ Beeper. ([\#13228](https://github.com/matrix-org/synapse/issues/13228))
|
||||
|
||||
|
||||
Synapse 1.62.0 (2022-07-05)
|
||||
===========================
|
||||
|
||||
|
|
@ -81,7 +104,6 @@ No significant changes since 1.62.0rc3.
|
|||
|
||||
Authors of spam-checker plugins should consult the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.62/docs/upgrade.md#upgrading-to-v1620) to learn about the enriched signatures for spam checker callbacks, which are supported with this release of Synapse.
|
||||
|
||||
|
||||
Synapse 1.62.0rc3 (2022-07-04)
|
||||
==============================
|
||||
|
||||
|
|
@ -121,7 +143,7 @@ Bugfixes
|
|||
- Update [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786) implementation to check `state_key`. ([\#12939](https://github.com/matrix-org/synapse/issues/12939))
|
||||
- Fix a bug introduced in Synapse 1.58 where Synapse would not report full version information when installed from a git checkout. This is a best-effort affair and not guaranteed to be stable. ([\#12973](https://github.com/matrix-org/synapse/issues/12973))
|
||||
- Fix a bug introduced in Synapse 1.60 where Synapse would fail to start if the `sqlite3` module was not available. ([\#12979](https://github.com/matrix-org/synapse/issues/12979))
|
||||
- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
|
||||
- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
|
||||
in Synapse v1.41.0. ([\#12991](https://github.com/matrix-org/synapse/issues/12991))
|
||||
- Fix a long-standing bug which meant that rate limiting was not restrictive enough in some cases. ([\#13018](https://github.com/matrix-org/synapse/issues/13018))
|
||||
- Fix a bug introduced in Synapse 1.58 where profile requests for a malformed user ID would ccause an internal error. Synapse now returns 400 Bad Request in this situation. ([\#13041](https://github.com/matrix-org/synapse/issues/13041))
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
Use lower isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Drop tables used for groups/communities.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Provide more info why we don't have any thumbnails to serve.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Make the AS login method call `Auth.get_user_by_req` for checking the AS token.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Always use a version of canonicaljson that supports the C implementation of frozendict.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Drop support for delegating email verification to an external server.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Refactor receipts servlet logic to avoid duplicated code.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add a `room_type` field in the responses for the list room and room details admin API. Contributed by @andrewdoh.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Remove unused database table `event_reference_hashes`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add support for room version 10.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@fizzadar).
|
||||
|
|
@ -0,0 +1 @@
|
|||
Provide an example of using the Admin API. Contributed by @jejo86.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Move the documentation for how URL previews work to the URL preview module.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Call the v2 identity service `/3pid/unbind` endpoint, rather than v1.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).
|
||||
|
|
@ -0,0 +1 @@
|
|||
Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@fizzadar).
|
||||
|
|
@ -0,0 +1 @@
|
|||
Preparatory work for a per-room rate limiter on joins.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Preparatory work for a per-room rate limiter on joins.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Preparatory work for a per-room rate limiter on joins.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Log the stack when waiting for an entire room to be un-partial stated.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Fix spurious warning when fetching state after a missing prev event.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Clean-up tests for notifications.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Move the documentation for how URL previews work to the URL preview module.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the "enable_email_notifs" and "email_notifs_for_new_users" options were enabled. Contributed by @thomasweston12.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Do not fail build if complement with workers fails.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Don't pull out state in `compute_event_context` for unconflicted state.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Fix a bug introduced in Synapse 1.40 where a user invited to a restricted room would be briefly unable to join.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add another `contrib` script to help set up worker processes. Contributed by @villepeh.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Don't pull out state in `compute_event_context` for unconflicted state.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttle additional joins if that rate grows too large.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Fix long-standing bug where in rare instances Synapse could store the incorrect state for a room after a state resolution.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Reduce the rebuild time for the complement-synapse docker image.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Don't pull out the full state when creating an event.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Update locked version of `frozendict` to 2.3.2, which has a fix for a memory leak.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Upgrade from Poetry 1.1.14 to 1.1.12, to fix bugs when locking packages.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Use `HTTPStatus` constants in place of literals in tests.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Improve performance of query `_get_subset_users_in_room_with_profiles`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Remove unnecessary `json.dumps` from tests.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Don't pull out the full state when creating an event.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).
|
||||
|
|
@ -0,0 +1 @@
|
|||
Reduce memory usage of sending dummy events.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Prevent formatting changes of [#3679](https://github.com/matrix-org/synapse/pull/3679) from appearing in `git blame`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add notes when config options where changed. Contributed by @behrmann.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Validate federation destinations and log an error if a destination is invalid.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Reduce memory usage of state caches.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Reduce the amount of state we store in the `state_cache`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Stop builindg `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add missing type hints to open tracing module.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@fizzadar).
|
||||
|
|
@ -0,0 +1 @@
|
|||
Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Mention that BuildKit is needed when building Docker images for tests.
|
||||
|
|
@ -0,0 +1 @@
|
|||
When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Add missing type hints to open tracing module.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@fizzadar).
|
||||
|
|
@ -0,0 +1 @@
|
|||
Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known.
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
# Creating multiple workers with a bash script
|
||||
# Creating multiple generic workers with a bash script
|
||||
|
||||
Setting up multiple worker configuration files manually can be time-consuming.
|
||||
You can alternatively create multiple worker configuration files with a simple `bash` script. For example:
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
# Creating multiple stream writers with a bash script
|
||||
|
||||
This script creates multiple [stream writer](https://github.com/matrix-org/synapse/blob/develop/docs/workers.md#stream-writers) workers.
|
||||
|
||||
Stream writers require both replication and HTTP listeners.
|
||||
|
||||
It also prints out the example lines for Synapse main configuration file.
|
||||
|
||||
Remember to route necessary endpoints directly to a worker associated with it.
|
||||
|
||||
If you run the script as-is, it will create workers with the replication listener starting from port 8034 and another, regular http listener starting from 8044. If you don't need all of the stream writers listed in the script, just remove them from the ```STREAM_WRITERS``` array.
|
||||
|
||||
```sh
|
||||
#!/bin/bash
|
||||
|
||||
# Start with these replication and http ports.
|
||||
# The script loop starts with the exact port and then increments it by one.
|
||||
REP_START_PORT=8034
|
||||
HTTP_START_PORT=8044
|
||||
|
||||
# Stream writer workers to generate. Feel free to add or remove them as you wish.
|
||||
# Event persister ("events") isn't included here as it does not require its
|
||||
# own HTTP listener.
|
||||
|
||||
STREAM_WRITERS+=( "presence" "typing" "receipts" "to_device" "account_data" )
|
||||
|
||||
NUM_WRITERS=$(expr ${#STREAM_WRITERS[@]})
|
||||
|
||||
i=0
|
||||
|
||||
while [ $i -lt "$NUM_WRITERS" ]
|
||||
do
|
||||
cat << EOF > ${STREAM_WRITERS[$i]}_stream_writer.yaml
|
||||
worker_app: synapse.app.generic_worker
|
||||
worker_name: ${STREAM_WRITERS[$i]}_stream_writer
|
||||
|
||||
# The replication listener on the main synapse process.
|
||||
worker_replication_host: 127.0.0.1
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: $(expr $REP_START_PORT + $i)
|
||||
resources:
|
||||
- names: [replication]
|
||||
|
||||
- type: http
|
||||
port: $(expr $HTTP_START_PORT + $i)
|
||||
resources:
|
||||
- names: [client]
|
||||
|
||||
worker_log_config: /etc/matrix-synapse/stream-writer-log.yaml
|
||||
EOF
|
||||
HOMESERVER_YAML_INSTANCE_MAP+=$" ${STREAM_WRITERS[$i]}_stream_writer:
|
||||
host: 127.0.0.1
|
||||
port: $(expr $REP_START_PORT + $i)
|
||||
"
|
||||
|
||||
HOMESERVER_YAML_STREAM_WRITERS+=$" ${STREAM_WRITERS[$i]}: ${STREAM_WRITERS[$i]}_stream_writer
|
||||
"
|
||||
|
||||
((i++))
|
||||
done
|
||||
|
||||
cat << EXAMPLECONFIG
|
||||
# Add these lines to your homeserver.yaml.
|
||||
# Don't forget to configure your reverse proxy and
|
||||
# necessary endpoints to their respective worker.
|
||||
|
||||
# See https://github.com/matrix-org/synapse/blob/develop/docs/workers.md
|
||||
# for more information.
|
||||
|
||||
# Remember: Under NO circumstances should the replication
|
||||
# listener be exposed to the public internet;
|
||||
# it has no authentication and is unencrypted.
|
||||
|
||||
instance_map:
|
||||
$HOMESERVER_YAML_INSTANCE_MAP
|
||||
stream_writers:
|
||||
$HOMESERVER_YAML_STREAM_WRITERS
|
||||
EXAMPLECONFIG
|
||||
```
|
||||
|
||||
Copy the code above save it to a file ```create_stream_writers.sh``` (for example).
|
||||
|
||||
Make the script executable by running ```chmod +x create_stream_writers.sh```.
|
||||
|
||||
## Run the script to create workers and print out a sample configuration
|
||||
|
||||
Simply run the script to create YAML files in the current folder and print out the required configuration for ```homeserver.yaml```.
|
||||
|
||||
```console
|
||||
$ ./create_stream_writers.sh
|
||||
|
||||
# Add these lines to your homeserver.yaml.
|
||||
# Don't forget to configure your reverse proxy and
|
||||
# necessary endpoints to their respective worker.
|
||||
|
||||
# See https://github.com/matrix-org/synapse/blob/develop/docs/workers.md
|
||||
# for more information
|
||||
|
||||
# Remember: Under NO circumstances should the replication
|
||||
# listener be exposed to the public internet;
|
||||
# it has no authentication and is unencrypted.
|
||||
|
||||
instance_map:
|
||||
presence_stream_writer:
|
||||
host: 127.0.0.1
|
||||
port: 8034
|
||||
typing_stream_writer:
|
||||
host: 127.0.0.1
|
||||
port: 8035
|
||||
receipts_stream_writer:
|
||||
host: 127.0.0.1
|
||||
port: 8036
|
||||
to_device_stream_writer:
|
||||
host: 127.0.0.1
|
||||
port: 8037
|
||||
account_data_stream_writer:
|
||||
host: 127.0.0.1
|
||||
port: 8038
|
||||
|
||||
stream_writers:
|
||||
presence: presence_stream_writer
|
||||
typing: typing_stream_writer
|
||||
receipts: receipts_stream_writer
|
||||
to_device: to_device_stream_writer
|
||||
account_data: account_data_stream_writer
|
||||
```
|
||||
|
||||
Simply copy-and-paste the output to an appropriate place in your Synapse main configuration file.
|
||||
|
||||
## Write directly to Synapse configuration file
|
||||
|
||||
You could also write the output directly to homeserver main configuration file. **This, however, is not recommended** as even a small typo (such as replacing >> with >) can erase the entire ```homeserver.yaml```.
|
||||
|
||||
If you do this, back up your original configuration file first:
|
||||
|
||||
```console
|
||||
# Back up homeserver.yaml first
|
||||
cp /etc/matrix-synapse/homeserver.yaml /etc/matrix-synapse/homeserver.yaml.bak
|
||||
|
||||
# Create workers and write output to your homeserver.yaml
|
||||
./create_stream_writers.sh >> /etc/matrix-synapse/homeserver.yaml
|
||||
```
|
||||
|
|
@ -1,3 +1,17 @@
|
|||
matrix-synapse-py3 (1.63.1) stable; urgency=medium
|
||||
|
||||
* New Synapse release 1.63.1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Wed, 20 Jul 2022 13:36:52 +0100
|
||||
|
||||
matrix-synapse-py3 (1.63.0) stable; urgency=medium
|
||||
|
||||
* Clarify that homeserver server names are included in the data reported
|
||||
by opt-in server stats reporting (`report_stats` homeserver config option).
|
||||
* New Synapse release 1.63.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 19 Jul 2022 14:42:24 +0200
|
||||
|
||||
matrix-synapse-py3 (1.63.0~rc1) stable; urgency=medium
|
||||
|
||||
* New Synapse release 1.63.0rc1.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ EOF
|
|||
# This file is autogenerated, and will be recreated on upgrade if it is deleted.
|
||||
# Any changes you make will be preserved.
|
||||
|
||||
# Whether to report anonymized homeserver usage statistics.
|
||||
# Whether to report homeserver usage statistics.
|
||||
report_stats: false
|
||||
EOF
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ msgstr ""
|
|||
#. Type: boolean
|
||||
#. Description
|
||||
#: ../templates:2001
|
||||
msgid "Report anonymous statistics?"
|
||||
msgid "Report homeserver usage statistics?"
|
||||
msgstr ""
|
||||
|
||||
#. Type: boolean
|
||||
|
|
@ -45,11 +45,11 @@ msgstr ""
|
|||
#: ../templates:2001
|
||||
msgid ""
|
||||
"Developers of Matrix and Synapse really appreciate helping the project out "
|
||||
"by reporting anonymized usage statistics from this homeserver. Only very "
|
||||
"basic aggregate data (e.g. number of users) will be reported, but it helps "
|
||||
"track the growth of the Matrix community, and helps in making Matrix a "
|
||||
"success, as well as to convince other networks that they should peer with "
|
||||
"Matrix."
|
||||
"by reporting homeserver usage statistics from this homeserver. Your "
|
||||
"homeserver's server name, along with very basic aggregate data (e.g. "
|
||||
"number of users) will be reported. But it helps track the growth of the "
|
||||
"Matrix community, and helps in making Matrix a success, as well as to "
|
||||
"convince other networks that they should peer with Matrix."
|
||||
msgstr ""
|
||||
|
||||
#. Type: boolean
|
||||
|
|
|
|||
|
|
@ -10,12 +10,13 @@ _Description: Name of the server:
|
|||
Template: matrix-synapse/report-stats
|
||||
Type: boolean
|
||||
Default: false
|
||||
_Description: Report anonymous statistics?
|
||||
_Description: Report homeserver usage statistics?
|
||||
Developers of Matrix and Synapse really appreciate helping the
|
||||
project out by reporting anonymized usage statistics from this
|
||||
homeserver. Only very basic aggregate data (e.g. number of users)
|
||||
will be reported, but it helps track the growth of the Matrix
|
||||
community, and helps in making Matrix a success, as well as to
|
||||
convince other networks that they should peer with Matrix.
|
||||
project out by reporting homeserver usage statistics from this
|
||||
homeserver. Your homeserver's server name, along with very basic
|
||||
aggregate data (e.g. number of users) will be reported. But it
|
||||
helps track the growth of the Matrix community, and helps in
|
||||
making Matrix a success, as well as to convince other networks
|
||||
that they should peer with Matrix.
|
||||
.
|
||||
Thank you.
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ RUN \
|
|||
|
||||
# We install poetry in its own build stage to avoid its dependencies conflicting with
|
||||
# synapse's dependencies.
|
||||
# We use a specific commit from poetry's master branch instead of our usual 1.1.12,
|
||||
# We use a specific commit from poetry's master branch instead of our usual 1.1.14,
|
||||
# to incorporate fixes to some bugs in `poetry export`. This commit corresponds to
|
||||
# https://github.com/python-poetry/poetry/pull/5156 and
|
||||
# https://github.com/python-poetry/poetry/issues/5141 ;
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
# syntax=docker/dockerfile:1
|
||||
# Inherit from the official Synapse docker image
|
||||
ARG SYNAPSE_VERSION=latest
|
||||
FROM matrixdotorg/synapse:$SYNAPSE_VERSION
|
||||
|
|
|
|||
|
|
@ -22,6 +22,10 @@ Consult the [contributing guide][guideComplementSh] for instructions on how to u
|
|||
Under some circumstances, you may wish to build the images manually.
|
||||
The instructions below will lead you to doing that.
|
||||
|
||||
Note that these images can only be built using [BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/),
|
||||
therefore BuildKit needs to be enabled when calling `docker build`. This can be done by
|
||||
setting `DOCKER_BUILDKIT=1` in your environment.
|
||||
|
||||
Start by building the base Synapse docker image. If you wish to run tests with the latest
|
||||
release of Synapse, instead of your current checkout, you can skip this step. From the
|
||||
root of the repository:
|
||||
|
|
|
|||
|
|
@ -1,45 +1,62 @@
|
|||
# syntax=docker/dockerfile:1
|
||||
# This dockerfile builds on top of 'docker/Dockerfile-workers' in matrix-org/synapse
|
||||
# by including a built-in postgres instance, as well as setting up the homeserver so
|
||||
# that it is ready for testing via Complement.
|
||||
#
|
||||
# Instructions for building this image from those it depends on is detailed in this guide:
|
||||
# https://github.com/matrix-org/synapse/blob/develop/docker/README-testing.md#testing-with-postgresql-and-single-or-multi-process-synapse
|
||||
|
||||
ARG SYNAPSE_VERSION=latest
|
||||
|
||||
# first of all, we create a base image with a postgres server and database,
|
||||
# which we can copy into the target image. For repeated rebuilds, this is
|
||||
# much faster than apt installing postgres each time.
|
||||
#
|
||||
# This trick only works because (a) the Synapse image happens to have all the
|
||||
# shared libraries that postgres wants, (b) we use a postgres image based on
|
||||
# the same debian version as Synapse's docker image (so the versions of the
|
||||
# shared libraries match).
|
||||
|
||||
FROM postgres:13-bullseye AS postgres_base
|
||||
# initialise the database cluster in /var/lib/postgresql
|
||||
RUN gosu postgres initdb --locale=C --encoding=UTF-8 --auth-host password
|
||||
|
||||
# Configure a password and create a database for Synapse
|
||||
RUN echo "ALTER USER postgres PASSWORD 'somesecret'" | gosu postgres postgres --single
|
||||
RUN echo "CREATE DATABASE synapse" | gosu postgres postgres --single
|
||||
|
||||
# now build the final image, based on the Synapse image.
|
||||
|
||||
FROM matrixdotorg/synapse-workers:$SYNAPSE_VERSION
|
||||
# copy the postgres installation over from the image we built above
|
||||
RUN adduser --system --uid 999 postgres --home /var/lib/postgresql
|
||||
COPY --from=postgres_base /var/lib/postgresql /var/lib/postgresql
|
||||
COPY --from=postgres_base /usr/lib/postgresql /usr/lib/postgresql
|
||||
COPY --from=postgres_base /usr/share/postgresql /usr/share/postgresql
|
||||
RUN mkdir /var/run/postgresql && chown postgres /var/run/postgresql
|
||||
ENV PATH="${PATH}:/usr/lib/postgresql/13/bin"
|
||||
ENV PGDATA=/var/lib/postgresql/data
|
||||
|
||||
# Install postgresql
|
||||
RUN apt-get update && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yqq postgresql-13
|
||||
# Extend the shared homeserver config to disable rate-limiting,
|
||||
# set Complement's static shared secret, enable registration, amongst other
|
||||
# tweaks to get Synapse ready for testing.
|
||||
# To do this, we copy the old template out of the way and then include it
|
||||
# with Jinja2.
|
||||
RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2
|
||||
COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2
|
||||
|
||||
# Configure a user and create a database for Synapse
|
||||
RUN pg_ctlcluster 13 main start && su postgres -c "echo \
|
||||
\"ALTER USER postgres PASSWORD 'somesecret'; \
|
||||
CREATE DATABASE synapse \
|
||||
ENCODING 'UTF8' \
|
||||
LC_COLLATE='C' \
|
||||
LC_CTYPE='C' \
|
||||
template=template0;\" | psql" && pg_ctlcluster 13 main stop
|
||||
WORKDIR /data
|
||||
|
||||
# Extend the shared homeserver config to disable rate-limiting,
|
||||
# set Complement's static shared secret, enable registration, amongst other
|
||||
# tweaks to get Synapse ready for testing.
|
||||
# To do this, we copy the old template out of the way and then include it
|
||||
# with Jinja2.
|
||||
RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2
|
||||
COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2
|
||||
COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf
|
||||
|
||||
WORKDIR /data
|
||||
# Copy the entrypoint
|
||||
COPY conf/start_for_complement.sh /
|
||||
|
||||
COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf
|
||||
# Expose nginx's listener ports
|
||||
EXPOSE 8008 8448
|
||||
|
||||
# Copy the entrypoint
|
||||
COPY conf/start_for_complement.sh /
|
||||
ENTRYPOINT ["/start_for_complement.sh"]
|
||||
|
||||
# Expose nginx's listener ports
|
||||
EXPOSE 8008 8448
|
||||
|
||||
ENTRYPOINT ["/start_for_complement.sh"]
|
||||
|
||||
# Update the healthcheck to have a shorter check interval
|
||||
HEALTHCHECK --start-period=5s --interval=1s --timeout=1s \
|
||||
CMD /bin/sh /healthcheck.sh
|
||||
# Update the healthcheck to have a shorter check interval
|
||||
HEALTHCHECK --start-period=5s --interval=1s --timeout=1s \
|
||||
CMD /bin/sh /healthcheck.sh
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
[program:postgres]
|
||||
command=/usr/local/bin/prefix-log /usr/bin/pg_ctlcluster 13 main start --foreground
|
||||
command=/usr/local/bin/prefix-log gosu postgres postgres
|
||||
|
||||
# Only start if START_POSTGRES=1
|
||||
autostart=%(ENV_START_POSTGRES)s
|
||||
|
|
|
|||
|
|
@ -67,6 +67,10 @@ rc_joins:
|
|||
per_second: 9999
|
||||
burst_count: 9999
|
||||
|
||||
rc_joins_per_room:
|
||||
per_second: 9999
|
||||
burst_count: 9999
|
||||
|
||||
rc_3pid_validation:
|
||||
per_second: 1000
|
||||
burst_count: 1000
|
||||
|
|
|
|||
|
|
@ -35,7 +35,6 @@
|
|||
- [Application Services](application_services.md)
|
||||
- [Server Notices](server_notices.md)
|
||||
- [Consent Tracking](consent_tracking.md)
|
||||
- [URL Previews](development/url_previews.md)
|
||||
- [User Directory](user_directory.md)
|
||||
- [Message Retention Policies](message_retention_policies.md)
|
||||
- [Pluggable Modules](modules/index.md)
|
||||
|
|
@ -69,7 +68,7 @@
|
|||
- [Federation](usage/administration/admin_api/federation.md)
|
||||
- [Manhole](manhole.md)
|
||||
- [Monitoring](metrics-howto.md)
|
||||
- [Reporting Anonymised Statistics](usage/administration/monitoring/reporting_anonymised_statistics.md)
|
||||
- [Reporting Homeserver Usage Statistics](usage/administration/monitoring/reporting_homeserver_usage_statistics.md)
|
||||
- [Understanding Synapse Through Grafana Graphs](usage/administration/understanding_synapse_through_grafana_graphs.md)
|
||||
- [Useful SQL for Admins](usage/administration/useful_sql_for_admins.md)
|
||||
- [Database Maintenance Tools](usage/administration/database_maintenance_tools.md)
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ The following fields are possible in the JSON response body:
|
|||
- `guest_access` - Whether guests can join the room. One of: ["can_join", "forbidden"].
|
||||
- `history_visibility` - Who can see the room history. One of: ["invited", "joined", "shared", "world_readable"].
|
||||
- `state_events` - Total number of state_events of a room. Complexity of the room.
|
||||
- `room_type` - The type of the room taken from the room's creation event; for example "m.space" if the room is a space. If the room does not define a type, the value will be `null`.
|
||||
* `offset` - The current pagination offset in rooms. This parameter should be
|
||||
used instead of `next_token` for room offset as `next_token` is
|
||||
not intended to be parsed.
|
||||
|
|
@ -101,7 +102,8 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 93534
|
||||
"state_events": 93534,
|
||||
"room_type": "m.space"
|
||||
},
|
||||
... (8 hidden items) ...
|
||||
{
|
||||
|
|
@ -118,7 +120,8 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 8345
|
||||
"state_events": 8345,
|
||||
"room_type": null
|
||||
}
|
||||
],
|
||||
"offset": 0,
|
||||
|
|
@ -151,7 +154,8 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 8
|
||||
"state_events": 8,
|
||||
"room_type": null
|
||||
}
|
||||
],
|
||||
"offset": 0,
|
||||
|
|
@ -184,7 +188,8 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 93534
|
||||
"state_events": 93534,
|
||||
"room_type": null
|
||||
},
|
||||
... (98 hidden items) ...
|
||||
{
|
||||
|
|
@ -201,7 +206,8 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 8345
|
||||
"state_events": 8345,
|
||||
"room_type": "m.space"
|
||||
}
|
||||
],
|
||||
"offset": 0,
|
||||
|
|
@ -238,7 +244,9 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 93534
|
||||
"state_events": 93534,
|
||||
"room_type": "m.space"
|
||||
|
||||
},
|
||||
... (48 hidden items) ...
|
||||
{
|
||||
|
|
@ -255,7 +263,9 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 8345
|
||||
"state_events": 8345,
|
||||
"room_type": null
|
||||
|
||||
}
|
||||
],
|
||||
"offset": 100,
|
||||
|
|
@ -290,6 +300,8 @@ The following fields are possible in the JSON response body:
|
|||
* `guest_access` - Whether guests can join the room. One of: ["can_join", "forbidden"].
|
||||
* `history_visibility` - Who can see the room history. One of: ["invited", "joined", "shared", "world_readable"].
|
||||
* `state_events` - Total number of state_events of a room. Complexity of the room.
|
||||
* `room_type` - The type of the room taken from the room's creation event; for example "m.space" if the room is a space.
|
||||
If the room does not define a type, the value will be `null`.
|
||||
|
||||
The API is:
|
||||
|
||||
|
|
@ -317,7 +329,8 @@ A response body like the following is returned:
|
|||
"join_rules": "invite",
|
||||
"guest_access": null,
|
||||
"history_visibility": "shared",
|
||||
"state_events": 93534
|
||||
"state_events": 93534,
|
||||
"room_type": "m.space"
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -544,7 +544,7 @@ Gets a list of all local media that a specific `user_id` has created.
|
|||
These are media that the user has uploaded themselves
|
||||
([local media](../media_repository.md#local-media)), as well as
|
||||
[URL preview images](../media_repository.md#url-previews) requested by the user if the
|
||||
[feature is enabled](../development/url_previews.md).
|
||||
[feature is enabled](../usage/configuration/config_documentation.md#url_preview_enabled).
|
||||
|
||||
By default, the response is ordered by descending creation date and ascending media ID.
|
||||
The newest media is on top. You can change the order with parameters
|
||||
|
|
|
|||
|
|
@ -237,3 +237,28 @@ poetry run pip install build && poetry run python -m build
|
|||
because [`build`](https://github.com/pypa/build) is a standardish tool which
|
||||
doesn't require poetry. (It's what we use in CI too). However, you could try
|
||||
`poetry build` too.
|
||||
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
## Check the version of poetry with `poetry --version`.
|
||||
|
||||
At the time of writing, the 1.2 series is beta only. We have seen some examples
|
||||
where the lockfiles generated by 1.2 prereleasese aren't interpreted correctly
|
||||
by poetry 1.1.x. For now, use poetry 1.1.14, which includes a critical
|
||||
[change](https://github.com/python-poetry/poetry/pull/5973) needed to remain
|
||||
[compatible with PyPI](https://github.com/pypi/warehouse/pull/11775).
|
||||
|
||||
It can also be useful to check the version of `poetry-core` in use. If you've
|
||||
installed `poetry` with `pipx`, try `pipx runpip poetry list | grep poetry-core`.
|
||||
|
||||
## Clear caches: `poetry cache clear --all pypi`.
|
||||
|
||||
Poetry caches a bunch of information about packages that isn't readily available
|
||||
from PyPI. (This is what makes poetry seem slow when doing the first
|
||||
`poetry install`.) Try `poetry cache list` and `poetry cache clear --all
|
||||
<name of cache>` to see if that fixes things.
|
||||
|
||||
## Try `--verbose` or `--dry-run` arguments.
|
||||
|
||||
Sometimes useful to see what poetry's internal logic is.
|
||||
|
|
|
|||
|
|
@ -1,61 +0,0 @@
|
|||
URL Previews
|
||||
============
|
||||
|
||||
The `GET /_matrix/media/r0/preview_url` endpoint provides a generic preview API
|
||||
for URLs which outputs [Open Graph](https://ogp.me/) responses (with some Matrix
|
||||
specific additions).
|
||||
|
||||
This does have trade-offs compared to other designs:
|
||||
|
||||
* Pros:
|
||||
* Simple and flexible; can be used by any clients at any point
|
||||
* Cons:
|
||||
* If each homeserver provides one of these independently, all the HSes in a
|
||||
room may needlessly DoS the target URI
|
||||
* The URL metadata must be stored somewhere, rather than just using Matrix
|
||||
itself to store the media.
|
||||
* Matrix cannot be used to distribute the metadata between homeservers.
|
||||
|
||||
When Synapse is asked to preview a URL it does the following:
|
||||
|
||||
1. Checks against a URL blacklist (defined as `url_preview_url_blacklist` in the
|
||||
config).
|
||||
2. Checks the in-memory cache by URLs and returns the result if it exists. (This
|
||||
is also used to de-duplicate processing of multiple in-flight requests at once.)
|
||||
3. Kicks off a background process to generate a preview:
|
||||
1. Checks the database cache by URL and timestamp and returns the result if it
|
||||
has not expired and was successful (a 2xx return code).
|
||||
2. Checks if the URL matches an [oEmbed](https://oembed.com/) pattern. If it
|
||||
does, update the URL to download.
|
||||
3. Downloads the URL and stores it into a file via the media storage provider
|
||||
and saves the local media metadata.
|
||||
4. If the media is an image:
|
||||
1. Generates thumbnails.
|
||||
2. Generates an Open Graph response based on image properties.
|
||||
5. If the media is HTML:
|
||||
1. Decodes the HTML via the stored file.
|
||||
2. Generates an Open Graph response from the HTML.
|
||||
3. If a JSON oEmbed URL was found in the HTML via autodiscovery:
|
||||
1. Downloads the URL and stores it into a file via the media storage provider
|
||||
and saves the local media metadata.
|
||||
2. Convert the oEmbed response to an Open Graph response.
|
||||
3. Override any Open Graph data from the HTML with data from oEmbed.
|
||||
4. If an image exists in the Open Graph response:
|
||||
1. Downloads the URL and stores it into a file via the media storage
|
||||
provider and saves the local media metadata.
|
||||
2. Generates thumbnails.
|
||||
3. Updates the Open Graph response based on image properties.
|
||||
6. If the media is JSON and an oEmbed URL was found:
|
||||
1. Convert the oEmbed response to an Open Graph response.
|
||||
2. If a thumbnail or image is in the oEmbed response:
|
||||
1. Downloads the URL and stores it into a file via the media storage
|
||||
provider and saves the local media metadata.
|
||||
2. Generates thumbnails.
|
||||
3. Updates the Open Graph response based on image properties.
|
||||
7. Stores the result in the database cache.
|
||||
4. Returns the result.
|
||||
|
||||
The in-memory cache expires after 1 hour.
|
||||
|
||||
Expired entries in the database cache (and their associated media files) are
|
||||
deleted every 10 seconds. The default expiration time is 1 hour from download.
|
||||
|
|
@ -7,8 +7,7 @@ The media repository
|
|||
users.
|
||||
* caches avatars, attachments and their thumbnails for media uploaded by remote
|
||||
users.
|
||||
* caches resources and thumbnails used for
|
||||
[URL previews](development/url_previews.md).
|
||||
* caches resources and thumbnails used for URL previews.
|
||||
|
||||
All media in Matrix can be identified by a unique
|
||||
[MXC URI](https://spec.matrix.org/latest/client-server-api/#matrix-content-mxc-uris),
|
||||
|
|
@ -59,8 +58,6 @@ remote_thumbnail/matrix.org/aa/bb/cccccccccccccccccccc/128-96-image-jpeg
|
|||
Note that `remote_thumbnail/` does not have an `s`.
|
||||
|
||||
## URL Previews
|
||||
See [URL Previews](development/url_previews.md) for documentation on the URL preview
|
||||
process.
|
||||
|
||||
When generating previews for URLs, Synapse may download and cache various
|
||||
resources, including images. These resources are assigned temporary media IDs
|
||||
|
|
|
|||
|
|
@ -89,6 +89,40 @@ process, for example:
|
|||
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
```
|
||||
|
||||
# Upgrading to v1.64.0
|
||||
|
||||
## Delegation of email validation no longer supported
|
||||
|
||||
As of this version, Synapse no longer allows the tasks of verifying email address
|
||||
ownership, and password reset confirmation, to be delegated to an identity server.
|
||||
|
||||
To continue to allow users to add email addresses to their homeserver accounts,
|
||||
and perform password resets, make sure that Synapse is configured with a
|
||||
working email server in the `email` configuration section (including, at a
|
||||
minimum, a `notif_from` setting.)
|
||||
|
||||
Specifying an `email` setting under `account_threepid_delegates` will now cause
|
||||
an error at startup.
|
||||
|
||||
## Changes to the event replication streams
|
||||
|
||||
Synapse now includes a flag indicating if an event is an outlier when
|
||||
replicating it to other workers. This is a forwards- and backwards-incompatible
|
||||
change: v1.63 and workers cannot process events replicated by v1.64 workers, and
|
||||
vice versa.
|
||||
|
||||
Once all workers are upgraded to v1.64 (or downgraded to v1.63), event
|
||||
replication will resume as normal.
|
||||
|
||||
## frozendict release
|
||||
|
||||
[frozendict 2.3.3](https://github.com/Marco-Sulla/python-frozendict/releases/tag/v2.3.3)
|
||||
has recently been released, which fixes a memory leak that occurs during `/sync`
|
||||
requests. We advise server administrators who installed Synapse via pip to upgrade
|
||||
frozendict with `pip install --upgrade frozendict`. The Docker image
|
||||
`matrixdotorg/synapse` and the Debian packages from `packages.matrix.org` already
|
||||
include the updated library.
|
||||
|
||||
# Upgrading to v1.62.0
|
||||
|
||||
## New signatures for spam checker callbacks
|
||||
|
|
|
|||
|
|
@ -18,6 +18,11 @@ already on your `$PATH` depending on how Synapse was installed.
|
|||
Finding your user's `access_token` is client-dependent, but will usually be shown in the client's settings.
|
||||
|
||||
## Making an Admin API request
|
||||
For security reasons, we [recommend](reverse_proxy.md#synapse-administration-endpoints)
|
||||
that the Admin API (`/_synapse/admin/...`) should be hidden from public view using a
|
||||
reverse proxy. This means you should typically query the Admin API from a terminal on
|
||||
the machine which runs Synapse.
|
||||
|
||||
Once you have your `access_token`, you will need to authenticate each request to an Admin API endpoint by
|
||||
providing the token as either a query parameter or a request header. To add it as a request header in cURL:
|
||||
|
||||
|
|
@ -25,5 +30,17 @@ providing the token as either a query parameter or a request header. To add it a
|
|||
curl --header "Authorization: Bearer <access_token>" <the_rest_of_your_API_request>
|
||||
```
|
||||
|
||||
For example, suppose we want to
|
||||
[query the account](user_admin_api.md#query-user-account) of the user
|
||||
`@foo:bar.com`. We need an admin access token (e.g.
|
||||
`syt_AjfVef2_L33JNpafeif_0feKJfeaf0CQpoZk`), and we need to know which port
|
||||
Synapse's [`client` listener](config_documentation.md#listeners) is listening
|
||||
on (e.g. `8008`). Then we can use the following command to request the account
|
||||
information from the Admin API.
|
||||
|
||||
```sh
|
||||
curl --header "Authorization: Bearer syt_AjfVef2_L33JNpafeif_0feKJfeaf0CQpoZk" -X GET http://127.0.0.1:8008/_synapse/admin/v2/users/@foo:bar.com
|
||||
```
|
||||
|
||||
For more details on access tokens in Matrix, please refer to the complete
|
||||
[matrix spec documentation](https://matrix.org/docs/spec/client_server/r0.6.1#using-access-tokens).
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
# Reporting Anonymised Statistics
|
||||
# Reporting Homeserver Usage Statistics
|
||||
|
||||
When generating your Synapse configuration file, you are asked whether you
|
||||
would like to report anonymised statistics to Matrix.org. These statistics
|
||||
would like to report usage statistics to Matrix.org. These statistics
|
||||
provide the foundation a glimpse into the number of Synapse homeservers
|
||||
participating in the network, as well as statistics such as the number of
|
||||
rooms being created and messages being sent. This feature is sometimes
|
||||
affectionately called "phone-home" stats. Reporting
|
||||
affectionately called "phone home" stats. Reporting
|
||||
[is optional](../../configuration/config_documentation.md#report_stats)
|
||||
and the reporting endpoint
|
||||
[can be configured](../../configuration/config_documentation.md#report_stats_endpoint),
|
||||
|
|
@ -21,9 +21,9 @@ The following statistics are sent to the configured reporting endpoint:
|
|||
|
||||
| Statistic Name | Type | Description |
|
||||
|----------------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `homeserver` | string | The homeserver's server name. |
|
||||
| `memory_rss` | int | The memory usage of the process (in kilobytes on Unix-based systems, bytes on MacOS). |
|
||||
| `cpu_average` | int | CPU time in % of a single core (not % of all cores). |
|
||||
| `homeserver` | string | The homeserver's server name. |
|
||||
| `server_context` | string | An arbitrary string used to group statistics from a set of homeservers. |
|
||||
| `timestamp` | int | The current time, represented as the number of seconds since the epoch. |
|
||||
| `uptime_seconds` | int | The number of seconds since the homeserver was last started. |
|
||||
|
|
@ -239,6 +239,8 @@ If this option is provided, it parses the given yaml to json and
|
|||
serves it on `/.well-known/matrix/client` endpoint
|
||||
alongside the standard properties.
|
||||
|
||||
*Added in Synapse 1.62.0.*
|
||||
|
||||
Example configuration:
|
||||
```yaml
|
||||
extra_well_known_client_content :
|
||||
|
|
@ -1155,6 +1157,9 @@ Caching can be configured through the following sub-options:
|
|||
with intermittent connections, at the cost of higher memory usage.
|
||||
A value of zero means that sync responses are not cached.
|
||||
Defaults to 2m.
|
||||
|
||||
*Changed in Synapse 1.62.0*: The default was changed from 0 to 2m.
|
||||
|
||||
* `cache_autotuning` and its sub-options `max_cache_memory_usage`, `target_cache_memory_usage`, and
|
||||
`min_cache_ttl` work in conjunction with each other to maintain a balance between cache memory
|
||||
usage and cache entry availability. You must be using [jemalloc](https://github.com/matrix-org/synapse#help-synapse-is-slow-and-eats-all-my-ramcpu)
|
||||
|
|
@ -1471,6 +1476,25 @@ rc_joins:
|
|||
per_second: 0.03
|
||||
burst_count: 12
|
||||
```
|
||||
---
|
||||
### `rc_joins_per_room`
|
||||
|
||||
This option allows admins to ratelimit joins to a room based on the number of recent
|
||||
joins (local or remote) to that room. It is intended to mitigate mass-join spam
|
||||
waves which target multiple homeservers.
|
||||
|
||||
By default, one join is permitted to a room every second, with an accumulating
|
||||
buffer of up to ten instantaneous joins.
|
||||
|
||||
Example configuration (default values):
|
||||
```yaml
|
||||
rc_joins_per_room:
|
||||
per_second: 1
|
||||
burst_count: 10
|
||||
```
|
||||
|
||||
_Added in Synapse 1.64.0._
|
||||
|
||||
---
|
||||
### `rc_3pid_validation`
|
||||
|
||||
|
|
@ -1504,6 +1528,8 @@ The `rc_invites.per_user` limit applies to the *receiver* of the invite, rather
|
|||
sender, meaning that a `rc_invite.per_user.burst_count` of 5 mandates that a single user
|
||||
cannot *receive* more than a burst of 5 invites at a time.
|
||||
|
||||
In contrast, the `rc_invites.per_issuer` limit applies to the *issuer* of the invite, meaning that a `rc_invite.per_issuer.burst_count` of 5 mandates that single user cannot *send* more than a burst of 5 invites at a time.
|
||||
|
||||
Example configuration:
|
||||
```yaml
|
||||
rc_invites:
|
||||
|
|
@ -1513,7 +1539,13 @@ rc_invites:
|
|||
per_user:
|
||||
per_second: 0.004
|
||||
burst_count: 3
|
||||
per_issuer:
|
||||
per_second: 0.5
|
||||
burst_count: 5
|
||||
```
|
||||
|
||||
_Changed in version 1.63:_ added the `per_issuer` limit.
|
||||
|
||||
---
|
||||
### `rc_third_party_invite`
|
||||
|
||||
|
|
@ -2168,30 +2200,26 @@ default_identity_server: https://matrix.org
|
|||
---
|
||||
### `account_threepid_delegates`
|
||||
|
||||
Handle threepid (email/phone etc) registration and password resets through a set of
|
||||
*trusted* identity servers. Note that this allows the configured identity server to
|
||||
reset passwords for accounts!
|
||||
Delegate verification of phone numbers to an identity server.
|
||||
|
||||
Be aware that if `email` is not set, and SMTP options have not been
|
||||
configured in the email config block, registration and user password resets via
|
||||
email will be globally disabled.
|
||||
When a user wishes to add a phone number to their account, we need to verify that they
|
||||
actually own that phone number, which requires sending them a text message (SMS).
|
||||
Currently Synapse does not support sending those texts itself and instead delegates the
|
||||
task to an identity server. The base URI for the identity server to be used is
|
||||
specified by the `account_threepid_delegates.msisdn` option.
|
||||
|
||||
Additionally, if `msisdn` is not set, registration and password resets via msisdn
|
||||
will be disabled regardless, and users will not be able to associate an msisdn
|
||||
identifier to their account. This is due to Synapse currently not supporting
|
||||
any method of sending SMS messages on its own.
|
||||
If this is left unspecified, Synapse will not allow users to add phone numbers to
|
||||
their account.
|
||||
|
||||
To enable using an identity server for operations regarding a particular third-party
|
||||
identifier type, set the value to the URL of that identity server as shown in the
|
||||
examples below.
|
||||
(Servers handling the these requests must answer the `/requestToken` endpoints defined
|
||||
by the Matrix Identity Service API
|
||||
[specification](https://matrix.org/docs/spec/identity_service/latest).)
|
||||
|
||||
Servers handling the these requests must answer the `/requestToken` endpoints defined
|
||||
by the Matrix Identity Service API [specification](https://matrix.org/docs/spec/identity_service/latest).
|
||||
*Updated in Synapse 1.64.0*: No longer accepts an `email` option.
|
||||
|
||||
Example configuration:
|
||||
```yaml
|
||||
account_threepid_delegates:
|
||||
email: https://example.com # Delegate email sending to example.com
|
||||
msisdn: http://localhost:8090 # Delegate SMS sending to this local process
|
||||
```
|
||||
---
|
||||
|
|
@ -2409,9 +2437,14 @@ metrics_flags:
|
|||
---
|
||||
### `report_stats`
|
||||
|
||||
Whether or not to report anonymized homeserver usage statistics. This is originally
|
||||
Whether or not to report homeserver usage statistics. This is originally
|
||||
set when generating the config. Set this option to true or false to change the current
|
||||
behavior.
|
||||
behavior. See
|
||||
[Reporting Homeserver Usage Statistics](../administration/monitoring/reporting_homeserver_usage_statistics.md)
|
||||
for information on what data is reported.
|
||||
|
||||
Statistics will be reported 5 minutes after Synapse starts, and then every 3 hours
|
||||
after that.
|
||||
|
||||
Example configuration:
|
||||
```yaml
|
||||
|
|
@ -2420,7 +2453,7 @@ report_stats: true
|
|||
---
|
||||
### `report_stats_endpoint`
|
||||
|
||||
The endpoint to report the anonymized homeserver usage statistics to.
|
||||
The endpoint to report homeserver usage statistics to.
|
||||
Defaults to https://matrix.org/report-usage-stats/push
|
||||
|
||||
Example configuration:
|
||||
|
|
|
|||
3
mypy.ini
3
mypy.ini
|
|
@ -84,9 +84,6 @@ disallow_untyped_defs = False
|
|||
[mypy-synapse.http.matrixfederationclient]
|
||||
disallow_untyped_defs = False
|
||||
|
||||
[mypy-synapse.logging.opentracing]
|
||||
disallow_untyped_defs = False
|
||||
|
||||
[mypy-synapse.metrics._reactor_metrics]
|
||||
disallow_untyped_defs = False
|
||||
# This module imports select.epoll. That exists on Linux, but doesn't on macOS.
|
||||
|
|
|
|||
|
|
@ -290,7 +290,7 @@ importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}
|
|||
|
||||
[[package]]
|
||||
name = "frozendict"
|
||||
version = "2.3.0"
|
||||
version = "2.3.3"
|
||||
description = "A simple immutable dictionary"
|
||||
category = "main"
|
||||
optional = false
|
||||
|
|
@ -1563,7 +1563,7 @@ url_preview = ["lxml"]
|
|||
[metadata]
|
||||
lock-version = "1.1"
|
||||
python-versions = "^3.7.1"
|
||||
content-hash = "e96625923122e29b6ea5964379828e321b6cede2b020fc32c6f86c09d86d1ae8"
|
||||
content-hash = "c24bbcee7e86dbbe7cdbf49f91a25b310bf21095452641e7440129f59b077f78"
|
||||
|
||||
[metadata.files]
|
||||
attrs = [
|
||||
|
|
@ -1753,23 +1753,23 @@ flake8-comprehensions = [
|
|||
{file = "flake8_comprehensions-3.8.0-py3-none-any.whl", hash = "sha256:9406314803abe1193c064544ab14fdc43c58424c0882f6ff8a581eb73fc9bb58"},
|
||||
]
|
||||
frozendict = [
|
||||
{file = "frozendict-2.3.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e18e2abd144a9433b0a8334582843b2aa0d3b9ac8b209aaa912ad365115fe2e1"},
|
||||
{file = "frozendict-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:96dc7a02e78da5725e5e642269bb7ae792e0c9f13f10f2e02689175ebbfedb35"},
|
||||
{file = "frozendict-2.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:752a6dcfaf9bb20a7ecab24980e4dbe041f154509c989207caf185522ef85461"},
|
||||
{file = "frozendict-2.3.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:5346d9fc1c936c76d33975a9a9f1a067342963105d9a403a99e787c939cc2bb2"},
|
||||
{file = "frozendict-2.3.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:60dd2253f1bacb63a7c486ec541a968af4f985ffb06602ee8954a3d39ec6bd2e"},
|
||||
{file = "frozendict-2.3.0-cp36-cp36m-win_amd64.whl", hash = "sha256:b2e044602ce17e5cd86724add46660fb9d80169545164e763300a3b839cb1b79"},
|
||||
{file = "frozendict-2.3.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a27a69b1ac3591e4258325108aee62b53c0eeb6ad0a993ae68d3c7eaea980420"},
|
||||
{file = "frozendict-2.3.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f45ef5f6b184d84744fff97b61f6b9a855e24d36b713ea2352fc723a047afa5"},
|
||||
{file = "frozendict-2.3.0-cp37-cp37m-win_amd64.whl", hash = "sha256:2d3f5016650c0e9a192f5024e68fb4d63f670d0ee58b099ed3f5b4c62ea30ecb"},
|
||||
{file = "frozendict-2.3.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6cf605916f50aabaaba5624c81eb270200f6c2c466c46960237a125ec8fe3ae0"},
|
||||
{file = "frozendict-2.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f6da06e44904beae4412199d7e49be4f85c6cc168ab06b77c735ea7da5ce3454"},
|
||||
{file = "frozendict-2.3.0-cp38-cp38-win_amd64.whl", hash = "sha256:1f34793fb409c4fa70ffd25bea87b01f3bd305fb1c6b09e7dff085b126302206"},
|
||||
{file = "frozendict-2.3.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:fd72494a559bdcd28aa71f4aa81860269cd0b7c45fff3e2614a0a053ecfd2a13"},
|
||||
{file = "frozendict-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00ea9166aa68cc5feed05986206fdbf35e838a09cb3feef998cf35978ff8a803"},
|
||||
{file = "frozendict-2.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:9ffaf440648b44e0bc694c1a4701801941378ba3ba6541e17750ae4b4aeeb116"},
|
||||
{file = "frozendict-2.3.0-py3-none-any.whl", hash = "sha256:8578fe06815fcdcc672bd5603eebc98361a5317c1c3a13b28c6c810f6ea3b323"},
|
||||
{file = "frozendict-2.3.0.tar.gz", hash = "sha256:da4231adefc5928e7810da2732269d3ad7b5616295b3e693746392a8205ea0b5"},
|
||||
{file = "frozendict-2.3.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:39942914c1217a5a49c7551495a103b3dbd216e19413687e003b859c6b0ebc12"},
|
||||
{file = "frozendict-2.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5589256058b31f2b91419fa30b8dc62dbdefe7710e688a3fd5b43849161eecc9"},
|
||||
{file = "frozendict-2.3.3-cp310-cp310-win_amd64.whl", hash = "sha256:35eb7e59e287c41f4f712d4d3d2333354175b155d217b97c99c201d2d8920790"},
|
||||
{file = "frozendict-2.3.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:310aaf81793abf4f471895e6fe65e0e74a28a2aaf7b25c2ba6ccd4e35af06842"},
|
||||
{file = "frozendict-2.3.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c353c11010a986566a0cb37f9a783c560ffff7d67d5e7fd52221fb03757cdc43"},
|
||||
{file = "frozendict-2.3.3-cp36-cp36m-win_amd64.whl", hash = "sha256:15b5f82aad108125336593cec1b6420c638bf45f449c57e50949fc7654ea5a41"},
|
||||
{file = "frozendict-2.3.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a4737e5257756bd6b877504ff50185b705db577b5330d53040a6cf6417bb3cdb"},
|
||||
{file = "frozendict-2.3.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:80a14c11e33e8b0bc09e07bba3732c77a502c39edb8c3959fd9a0e490e031158"},
|
||||
{file = "frozendict-2.3.3-cp37-cp37m-win_amd64.whl", hash = "sha256:027952d1698ac9c766ef43711226b178cdd49d2acbdff396936639ad1d2a5615"},
|
||||
{file = "frozendict-2.3.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:ef818d66c85098a37cf42509545a4ba7dd0c4c679d6262123a8dc14cc474bab7"},
|
||||
{file = "frozendict-2.3.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:812279f2b270c980112dc4e367b168054f937108f8044eced4199e0ab2945a37"},
|
||||
{file = "frozendict-2.3.3-cp38-cp38-win_amd64.whl", hash = "sha256:c1fb7efbfebc2075f781be3d9774e4ba6ce4fc399148b02097f68d4b3c4bc00a"},
|
||||
{file = "frozendict-2.3.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:a0b46d4bf95bce843c0151959d54c3e5b8d0ce29cb44794e820b3ec980d63eee"},
|
||||
{file = "frozendict-2.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:38c4660f37fcc70a32ff997fe58e40b3fcc60b2017b286e33828efaa16b01308"},
|
||||
{file = "frozendict-2.3.3-cp39-cp39-win_amd64.whl", hash = "sha256:919e3609844fece11ab18bcbf28a3ed20f8108ad4149d7927d413687f281c6c9"},
|
||||
{file = "frozendict-2.3.3-py3-none-any.whl", hash = "sha256:f988b482d08972a196664718167a993a61c9e9f6fe7b0ca2443570b5f20ca44a"},
|
||||
{file = "frozendict-2.3.3.tar.gz", hash = "sha256:398539c52af3c647d103185bbaa1291679f0507ad035fe3bab2a8b0366d52cf1"},
|
||||
]
|
||||
gitdb = [
|
||||
{file = "gitdb-4.0.9-py3-none-any.whl", hash = "sha256:8033ad4e853066ba6ca92050b9df2f89301b8fc8bf7e9324d412a63f8bf1a8fd"},
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ skip_gitignore = true
|
|||
|
||||
[tool.poetry]
|
||||
name = "matrix-synapse"
|
||||
version = "1.63.0rc1"
|
||||
version = "1.63.1"
|
||||
description = "Homeserver for the Matrix decentralised comms protocol"
|
||||
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
|
||||
license = "Apache-2.0"
|
||||
|
|
@ -110,7 +110,9 @@ jsonschema = ">=3.0.0"
|
|||
frozendict = ">=1,!=2.1.2"
|
||||
# We require 2.1.0 or higher for type hints. Previous guard was >= 1.1.0
|
||||
unpaddedbase64 = ">=2.1.0"
|
||||
canonicaljson = "^1.4.0"
|
||||
# We require 1.5.0 to work around an issue when running against the C implementation of
|
||||
# frozendict: https://github.com/matrix-org/python-canonicaljson/issues/36
|
||||
canonicaljson = "^1.5.0"
|
||||
# we use the type definitions added in signedjson 1.1.
|
||||
signedjson = "^1.1.0"
|
||||
# validating SSL certs for IP addresses requires service_identity 18.1.
|
||||
|
|
|
|||
|
|
@ -26,7 +26,6 @@ DISTS = (
|
|||
"debian:bookworm",
|
||||
"debian:sid",
|
||||
"ubuntu:focal", # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
|
||||
"ubuntu:impish", # 21.10 (EOL 2022-07)
|
||||
"ubuntu:jammy", # 22.04 LTS (EOL 2027-04)
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ def main() -> None:
|
|||
parser.add_argument(
|
||||
"--report-stats",
|
||||
action="store",
|
||||
help="Whether the generated config reports anonymized usage statistics",
|
||||
help="Whether the generated config reports homeserver usage statistics",
|
||||
choices=["yes", "no"],
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -166,22 +166,6 @@ IGNORED_TABLES = {
|
|||
"ui_auth_sessions",
|
||||
"ui_auth_sessions_credentials",
|
||||
"ui_auth_sessions_ips",
|
||||
# Groups/communities is no longer supported.
|
||||
"group_attestations_remote",
|
||||
"group_attestations_renewals",
|
||||
"group_invites",
|
||||
"group_roles",
|
||||
"group_room_categories",
|
||||
"group_rooms",
|
||||
"group_summary_roles",
|
||||
"group_summary_room_categories",
|
||||
"group_summary_rooms",
|
||||
"group_summary_users",
|
||||
"group_users",
|
||||
"groups",
|
||||
"local_group_membership",
|
||||
"local_group_updates",
|
||||
"remote_profile_cache",
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -27,6 +27,33 @@ class Ratelimiter:
|
|||
"""
|
||||
Ratelimit actions marked by arbitrary keys.
|
||||
|
||||
(Note that the source code speaks of "actions" and "burst_count" rather than
|
||||
"tokens" and a "bucket_size".)
|
||||
|
||||
This is a "leaky bucket as a meter". For each key to be tracked there is a bucket
|
||||
containing some number 0 <= T <= `burst_count` of tokens corresponding to previously
|
||||
permitted requests for that key. Each bucket starts empty, and gradually leaks
|
||||
tokens at a rate of `rate_hz`.
|
||||
|
||||
Upon an incoming request, we must determine:
|
||||
- the key that this request falls under (which bucket to inspect), and
|
||||
- the cost C of this request in tokens.
|
||||
Then, if there is room in the bucket for C tokens (T + C <= `burst_count`),
|
||||
the request is permitted and `cost` tokens are added to the bucket.
|
||||
Otherwise the request is denied, and the bucket continues to hold T tokens.
|
||||
|
||||
This means that the limiter enforces an average request frequency of `rate_hz`,
|
||||
while accumulating a buffer of up to `burst_count` requests which can be consumed
|
||||
instantaneously.
|
||||
|
||||
The tricky bit is the leaking. We do not want to have a periodic process which
|
||||
leaks every bucket! Instead, we track
|
||||
- the time point when the bucket was last completely empty, and
|
||||
- how many tokens have added to the bucket permitted since then.
|
||||
Then for each incoming request, we can calculate how many tokens have leaked
|
||||
since this time point, and use that to decide if we should accept or reject the
|
||||
request.
|
||||
|
||||
Args:
|
||||
clock: A homeserver clock, for retrieving the current time
|
||||
rate_hz: The long term number of actions that can be performed in a second.
|
||||
|
|
@ -41,14 +68,30 @@ class Ratelimiter:
|
|||
self.burst_count = burst_count
|
||||
self.store = store
|
||||
|
||||
# A ordered dictionary keeping track of actions, when they were last
|
||||
# performed and how often. Each entry is a mapping from a key of arbitrary type
|
||||
# to a tuple representing:
|
||||
# * How many times an action has occurred since a point in time
|
||||
# * The point in time
|
||||
# * The rate_hz of this particular entry. This can vary per request
|
||||
# An ordered dictionary representing the token buckets tracked by this rate
|
||||
# limiter. Each entry maps a key of arbitrary type to a tuple representing:
|
||||
# * The number of tokens currently in the bucket,
|
||||
# * The time point when the bucket was last completely empty, and
|
||||
# * The rate_hz (leak rate) of this particular bucket.
|
||||
self.actions: OrderedDict[Hashable, Tuple[float, float, float]] = OrderedDict()
|
||||
|
||||
def _get_key(
|
||||
self, requester: Optional[Requester], key: Optional[Hashable]
|
||||
) -> Hashable:
|
||||
"""Use the requester's MXID as a fallback key if no key is provided."""
|
||||
if key is None:
|
||||
if not requester:
|
||||
raise ValueError("Must supply at least one of `requester` or `key`")
|
||||
|
||||
key = requester.user.to_string()
|
||||
return key
|
||||
|
||||
def _get_action_counts(
|
||||
self, key: Hashable, time_now_s: float
|
||||
) -> Tuple[float, float, float]:
|
||||
"""Retrieve the action counts, with a fallback representing an empty bucket."""
|
||||
return self.actions.get(key, (0.0, time_now_s, 0.0))
|
||||
|
||||
async def can_do_action(
|
||||
self,
|
||||
requester: Optional[Requester],
|
||||
|
|
@ -88,11 +131,7 @@ class Ratelimiter:
|
|||
* The reactor timestamp for when the action can be performed next.
|
||||
-1 if rate_hz is less than or equal to zero
|
||||
"""
|
||||
if key is None:
|
||||
if not requester:
|
||||
raise ValueError("Must supply at least one of `requester` or `key`")
|
||||
|
||||
key = requester.user.to_string()
|
||||
key = self._get_key(requester, key)
|
||||
|
||||
if requester:
|
||||
# Disable rate limiting of users belonging to any AS that is configured
|
||||
|
|
@ -121,7 +160,7 @@ class Ratelimiter:
|
|||
self._prune_message_counts(time_now_s)
|
||||
|
||||
# Check if there is an existing count entry for this key
|
||||
action_count, time_start, _ = self.actions.get(key, (0.0, time_now_s, 0.0))
|
||||
action_count, time_start, _ = self._get_action_counts(key, time_now_s)
|
||||
|
||||
# Check whether performing another action is allowed
|
||||
time_delta = time_now_s - time_start
|
||||
|
|
@ -164,6 +203,37 @@ class Ratelimiter:
|
|||
|
||||
return allowed, time_allowed
|
||||
|
||||
def record_action(
|
||||
self,
|
||||
requester: Optional[Requester],
|
||||
key: Optional[Hashable] = None,
|
||||
n_actions: int = 1,
|
||||
_time_now_s: Optional[float] = None,
|
||||
) -> None:
|
||||
"""Record that an action(s) took place, even if they violate the rate limit.
|
||||
|
||||
This is useful for tracking the frequency of events that happen across
|
||||
federation which we still want to impose local rate limits on. For instance, if
|
||||
we are alice.com monitoring a particular room, we cannot prevent bob.com
|
||||
from joining users to that room. However, we can track the number of recent
|
||||
joins in the room and refuse to serve new joins ourselves if there have been too
|
||||
many in the room across both homeservers.
|
||||
|
||||
Args:
|
||||
requester: The requester that is doing the action, if any.
|
||||
key: An arbitrary key used to classify an action. Defaults to the
|
||||
requester's user ID.
|
||||
n_actions: The number of times the user wants to do this action. If the user
|
||||
cannot do all of the actions, the user's action count is not incremented
|
||||
at all.
|
||||
_time_now_s: The current time. Optional, defaults to the current time according
|
||||
to self.clock. Only used by tests.
|
||||
"""
|
||||
key = self._get_key(requester, key)
|
||||
time_now_s = _time_now_s if _time_now_s is not None else self.clock.time()
|
||||
action_count, time_start, rate_hz = self._get_action_counts(key, time_now_s)
|
||||
self.actions[key] = (action_count + n_actions, time_start, rate_hz)
|
||||
|
||||
def _prune_message_counts(self, time_now_s: float) -> None:
|
||||
"""Remove message count entries that have not exceeded their defined
|
||||
rate_hz limit
|
||||
|
|
|
|||
|
|
@ -84,6 +84,8 @@ class RoomVersion:
|
|||
# MSC3787: Adds support for a `knock_restricted` join rule, mixing concepts of
|
||||
# knocks and restricted join rules into the same join condition.
|
||||
msc3787_knock_restricted_join_rule: bool
|
||||
# MSC3667: Enforce integer power levels
|
||||
msc3667_int_only_power_levels: bool
|
||||
|
||||
|
||||
class RoomVersions:
|
||||
|
|
@ -103,6 +105,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V2 = RoomVersion(
|
||||
"2",
|
||||
|
|
@ -120,6 +123,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V3 = RoomVersion(
|
||||
"3",
|
||||
|
|
@ -137,6 +141,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V4 = RoomVersion(
|
||||
"4",
|
||||
|
|
@ -154,6 +159,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V5 = RoomVersion(
|
||||
"5",
|
||||
|
|
@ -171,6 +177,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V6 = RoomVersion(
|
||||
"6",
|
||||
|
|
@ -188,6 +195,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
MSC2176 = RoomVersion(
|
||||
"org.matrix.msc2176",
|
||||
|
|
@ -205,6 +213,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V7 = RoomVersion(
|
||||
"7",
|
||||
|
|
@ -222,6 +231,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V8 = RoomVersion(
|
||||
"8",
|
||||
|
|
@ -239,6 +249,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V9 = RoomVersion(
|
||||
"9",
|
||||
|
|
@ -256,6 +267,7 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
MSC2716v3 = RoomVersion(
|
||||
"org.matrix.msc2716v3",
|
||||
|
|
@ -273,6 +285,7 @@ class RoomVersions:
|
|||
msc2716_historical=True,
|
||||
msc2716_redactions=True,
|
||||
msc3787_knock_restricted_join_rule=False,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
MSC3787 = RoomVersion(
|
||||
"org.matrix.msc3787",
|
||||
|
|
@ -290,6 +303,25 @@ class RoomVersions:
|
|||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=True,
|
||||
msc3667_int_only_power_levels=False,
|
||||
)
|
||||
V10 = RoomVersion(
|
||||
"10",
|
||||
RoomDisposition.STABLE,
|
||||
EventFormatVersions.V3,
|
||||
StateResolutionVersions.V2,
|
||||
enforce_key_validity=True,
|
||||
special_case_aliases_auth=False,
|
||||
strict_canonicaljson=True,
|
||||
limit_notifications_power_levels=True,
|
||||
msc2176_redaction_rules=False,
|
||||
msc3083_join_rules=True,
|
||||
msc3375_redaction_rules=True,
|
||||
msc2403_knocking=True,
|
||||
msc2716_historical=False,
|
||||
msc2716_redactions=False,
|
||||
msc3787_knock_restricted_join_rule=True,
|
||||
msc3667_int_only_power_levels=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -308,6 +340,7 @@ KNOWN_ROOM_VERSIONS: Dict[str, RoomVersion] = {
|
|||
RoomVersions.V9,
|
||||
RoomVersions.MSC2716v3,
|
||||
RoomVersions.MSC3787,
|
||||
RoomVersions.V10,
|
||||
)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -28,19 +28,22 @@ from synapse.config.homeserver import HomeServerConfig
|
|||
from synapse.config.logger import setup_logging
|
||||
from synapse.events import EventBase
|
||||
from synapse.handlers.admin import ExfiltrationWriter
|
||||
from synapse.replication.slave.storage._base import BaseSlavedStore
|
||||
from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
|
||||
from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
|
||||
from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
|
||||
from synapse.replication.slave.storage.devices import SlavedDeviceStore
|
||||
from synapse.replication.slave.storage.events import SlavedEventStore
|
||||
from synapse.replication.slave.storage.filtering import SlavedFilteringStore
|
||||
from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
|
||||
from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
|
||||
from synapse.replication.slave.storage.registration import SlavedRegistrationStore
|
||||
from synapse.server import HomeServer
|
||||
from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
|
||||
from synapse.storage.databases.main.account_data import AccountDataWorkerStore
|
||||
from synapse.storage.databases.main.appservice import (
|
||||
ApplicationServiceTransactionWorkerStore,
|
||||
ApplicationServiceWorkerStore,
|
||||
)
|
||||
from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
|
||||
from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
|
||||
from synapse.storage.databases.main.registration import RegistrationWorkerStore
|
||||
from synapse.storage.databases.main.room import RoomWorkerStore
|
||||
from synapse.storage.databases.main.tags import TagsWorkerStore
|
||||
from synapse.types import StateMap
|
||||
from synapse.util import SYNAPSE_VERSION
|
||||
from synapse.util.logcontext import LoggingContext
|
||||
|
|
@ -49,16 +52,17 @@ logger = logging.getLogger("synapse.app.admin_cmd")
|
|||
|
||||
|
||||
class AdminCmdSlavedStore(
|
||||
SlavedReceiptsStore,
|
||||
SlavedAccountDataStore,
|
||||
SlavedApplicationServiceStore,
|
||||
SlavedRegistrationStore,
|
||||
SlavedFilteringStore,
|
||||
SlavedDeviceInboxStore,
|
||||
SlavedDeviceStore,
|
||||
SlavedPushRuleStore,
|
||||
SlavedEventStore,
|
||||
BaseSlavedStore,
|
||||
TagsWorkerStore,
|
||||
DeviceInboxWorkerStore,
|
||||
AccountDataWorkerStore,
|
||||
ApplicationServiceTransactionWorkerStore,
|
||||
ApplicationServiceWorkerStore,
|
||||
RegistrationWorkerStore,
|
||||
ReceiptsWorkerStore,
|
||||
RoomWorkerStore,
|
||||
):
|
||||
def __init__(
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue