Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes

michaelkaye/matrix_org_hotfixes_increase_replication_timeout
Erik Johnston 2020-01-31 15:06:16 +00:00
commit c854d255e5
158 changed files with 1736 additions and 565 deletions

View File

@ -1,3 +1,96 @@
Synapse 1.9.1 (2020-01-28)
==========================
Bugfixes
--------
- Fix bug where setting `mau_limit_reserved_threepids` config would cause Synapse to refuse to start. ([\#6793](https://github.com/matrix-org/synapse/issues/6793))
Synapse 1.9.0 (2020-01-23)
==========================
**WARNING**: As of this release, Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions [here](docs/postgres.md)).
If your Synapse deployment uses workers, note that the reverse-proxy configurations for the `synapse.app.media_repository`, `synapse.app.federation_reader` and `synapse.app.event_creator` workers have changed, with the addition of a few paths (see the updated configurations [here](docs/workers.md#available-worker-applications)). Existing configurations will continue to work.
Improved Documentation
----------------------
- Fix endpoint documentation for the List Rooms admin API. ([\#6770](https://github.com/matrix-org/synapse/issues/6770))
Synapse 1.9.0rc1 (2020-01-22)
=============================
Features
--------
- Allow admin to create or modify a user. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5742](https://github.com/matrix-org/synapse/issues/5742))
- Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media. ([\#6681](https://github.com/matrix-org/synapse/issues/6681), [\#6756](https://github.com/matrix-org/synapse/issues/6756))
- Add `org.matrix.e2e_cross_signing` to `unstable_features` in `/versions` as per [MSC1756](https://github.com/matrix-org/matrix-doc/pull/1756). ([\#6712](https://github.com/matrix-org/synapse/issues/6712))
- Add a new admin API to list and filter rooms on the server. ([\#6720](https://github.com/matrix-org/synapse/issues/6720))
Bugfixes
--------
- Correctly proxy HTTP errors due to API calls to remote group servers. ([\#6654](https://github.com/matrix-org/synapse/issues/6654))
- Fix media repo admin APIs when using a media worker. ([\#6664](https://github.com/matrix-org/synapse/issues/6664))
- Fix "CRITICAL" errors being logged when a request is received for a uri containing non-ascii characters. ([\#6682](https://github.com/matrix-org/synapse/issues/6682))
- Fix a bug where we would assign a numeric user ID if somebody tried registering with an empty username. ([\#6690](https://github.com/matrix-org/synapse/issues/6690))
- Fix `purge_room` admin API. ([\#6711](https://github.com/matrix-org/synapse/issues/6711))
- Fix a bug causing Synapse to not always purge quiet rooms with a low `max_lifetime` in their message retention policies when running the automated purge jobs. ([\#6714](https://github.com/matrix-org/synapse/issues/6714))
- Fix the `synapse_port_db` not correctly running background updates. Thanks @tadzik for reporting. ([\#6718](https://github.com/matrix-org/synapse/issues/6718))
- Fix changing password via user admin API. ([\#6730](https://github.com/matrix-org/synapse/issues/6730))
- Fix `/events/:event_id` deprecated API. ([\#6731](https://github.com/matrix-org/synapse/issues/6731))
- Fix monthly active user limiting support for worker mode, fixes [#4639](https://github.com/matrix-org/synapse/issues/4639). ([\#6742](https://github.com/matrix-org/synapse/issues/6742))
- Fix bug when setting `account_validity` to an empty block in the config. Thanks to @Sorunome for reporting. ([\#6747](https://github.com/matrix-org/synapse/issues/6747))
- Fix `AttributeError: 'NoneType' object has no attribute 'get'` in `hash_password` when configuration has an empty `password_config`. Contributed by @ivilata. ([\#6753](https://github.com/matrix-org/synapse/issues/6753))
- Fix the `docker-compose.yaml` overriding the entire `/etc` folder of the container. Contributed by Fabian Meyer. ([\#6656](https://github.com/matrix-org/synapse/issues/6656))
Improved Documentation
----------------------
- Fix a typo in the configuration example for purge jobs in the sample configuration file. ([\#6621](https://github.com/matrix-org/synapse/issues/6621))
- Add complete documentation of the message retention policies support. ([\#6624](https://github.com/matrix-org/synapse/issues/6624), [\#6665](https://github.com/matrix-org/synapse/issues/6665))
- Add some helpful tips about changelog entries to the GitHub pull request template. ([\#6663](https://github.com/matrix-org/synapse/issues/6663))
- Clarify the `account_validity` and `email` sections of the sample configuration. ([\#6685](https://github.com/matrix-org/synapse/issues/6685))
- Add more endpoints to the documentation for Synapse workers. ([\#6698](https://github.com/matrix-org/synapse/issues/6698))
Deprecations and Removals
-------------------------
- Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions [here](docs/postgres.md)). ([\#6675](https://github.com/matrix-org/synapse/issues/6675))
Internal Changes
----------------
- Add `local_current_membership` table for tracking local user membership state in rooms. ([\#6655](https://github.com/matrix-org/synapse/issues/6655), [\#6728](https://github.com/matrix-org/synapse/issues/6728))
- Port `synapse.replication.tcp` to async/await. ([\#6666](https://github.com/matrix-org/synapse/issues/6666))
- Fixup `synapse.replication` to pass mypy checks. ([\#6667](https://github.com/matrix-org/synapse/issues/6667))
- Allow `additional_resources` to implement `IResource` directly. ([\#6686](https://github.com/matrix-org/synapse/issues/6686))
- Allow REST endpoint implementations to raise a `RedirectException`, which will redirect the user's browser to a given location. ([\#6687](https://github.com/matrix-org/synapse/issues/6687))
- Updates and extensions to the module API. ([\#6688](https://github.com/matrix-org/synapse/issues/6688))
- Updates to the SAML mapping provider API. ([\#6689](https://github.com/matrix-org/synapse/issues/6689), [\#6723](https://github.com/matrix-org/synapse/issues/6723))
- Remove redundant `RegistrationError` class. ([\#6691](https://github.com/matrix-org/synapse/issues/6691))
- Don't block processing of incoming EDUs behind processing PDUs in the same transaction. ([\#6697](https://github.com/matrix-org/synapse/issues/6697))
- Remove duplicate check for the `session` query parameter on the `/auth/xxx/fallback/web` Client-Server endpoint. ([\#6702](https://github.com/matrix-org/synapse/issues/6702))
- Attempt to retry sending a transaction when we detect a remote server has come back online, rather than waiting for a transaction to be triggered by new data. ([\#6706](https://github.com/matrix-org/synapse/issues/6706))
- Add `StateMap` type alias to simplify types. ([\#6715](https://github.com/matrix-org/synapse/issues/6715))
- Add a `DeltaState` to track changes to be made to current state during event persistence. ([\#6716](https://github.com/matrix-org/synapse/issues/6716))
- Add more logging around message retention policies support. ([\#6717](https://github.com/matrix-org/synapse/issues/6717))
- When processing a SAML response, log the assertions for easier configuration. ([\#6724](https://github.com/matrix-org/synapse/issues/6724))
- Fixup `synapse.rest` to pass mypy. ([\#6732](https://github.com/matrix-org/synapse/issues/6732), [\#6764](https://github.com/matrix-org/synapse/issues/6764))
- Fixup `synapse.api` to pass mypy. ([\#6733](https://github.com/matrix-org/synapse/issues/6733))
- Allow streaming cache 'invalidate all' to workers. ([\#6749](https://github.com/matrix-org/synapse/issues/6749))
- Remove unused CI docker compose files. ([\#6754](https://github.com/matrix-org/synapse/issues/6754))
Synapse 1.8.0 (2020-01-09) Synapse 1.8.0 (2020-01-09)
========================== ==========================

View File

@ -76,6 +76,15 @@ for example:
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
Upgrading to **<NEXT_VERSION>**
===============================
Synapse will now log a warning on start up if used with a PostgreSQL database
that has a non-recommended locale set.
See [docs/postgres.md](docs/postgres.md) for details.
Upgrading to v1.8.0 Upgrading to v1.8.0
=================== ===================

View File

@ -1 +0,0 @@
Allow admin to create or modify a user. Contributed by Awesome Technologies Innovationslabor GmbH.

View File

@ -1 +0,0 @@
Fix a typo in the configuration example for purge jobs in the sample configuration file.

View File

@ -1 +0,0 @@
Add complete documentation of the message retention policies support.

View File

@ -1 +0,0 @@
Correctly proxy HTTP errors due to API calls to remote group servers.

View File

@ -1 +0,0 @@
Add `local_current_membership` table for tracking local user membership state in rooms.

View File

@ -1 +0,0 @@
No more overriding the entire /etc folder of the container in docker-compose.yaml. Contributed by Fabian Meyer.

View File

@ -1 +0,0 @@
Add some helpful tips about changelog entries to the github pull request template.

View File

@ -1 +0,0 @@
Fix media repo admin APIs when using a media worker.

View File

@ -1 +0,0 @@
Add complete documentation of the message retention policies support.

View File

@ -1 +0,0 @@
Port `synapse.replication.tcp` to async/await.

View File

@ -1 +0,0 @@
Fixup `synapse.replication` to pass mypy checks.

View File

@ -1 +0,0 @@
Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions [here](docs/postgres.md)).

View File

@ -1 +0,0 @@
Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media.

View File

@ -1,2 +0,0 @@
Fix "CRITICAL" errors being logged when a request is received for a uri containing non-ascii characters.

View File

@ -1 +0,0 @@
Clarify the `account_validity` and `email` sections of the sample configuration.

View File

@ -1 +0,0 @@
Allow additional_resources to implement IResource directly.

View File

@ -1 +0,0 @@
Allow REST endpoint implementations to raise a RedirectException, which will redirect the user's browser to a given location.

View File

@ -1 +0,0 @@
Updates and extensions to the module API.

View File

@ -1 +0,0 @@
Updates to the SAML mapping provider API.

View File

@ -1 +0,0 @@
Fix a bug where we would assign a numeric userid if somebody tried registering with an empty username.

View File

@ -1 +0,0 @@
Remove redundant RegistrationError class.

View File

@ -1 +0,0 @@
Don't block processing of incoming EDUs behind processing PDUs in the same transaction.

View File

@ -1 +0,0 @@
Add more endpoints to the documentation for Synapse workers.

View File

@ -1 +0,0 @@
Remove duplicate check for the `session` query parameter on the `/auth/xxx/fallback/web` Client-Server endpoint.

View File

@ -1 +0,0 @@
Attempt to retry sending a transaction when we detect a remote server has come back online, rather than waiting for a transaction to be triggered by new data.

View File

@ -1 +0,0 @@
Fix `purge_room` admin API.

View File

@ -1 +0,0 @@
Add org.matrix.e2e_cross_signing to unstable_features in /versions as per [MSC1756](https://github.com/matrix-org/matrix-doc/pull/1756).

View File

@ -1 +0,0 @@
Fix a bug causing Synapse to not always purge quiet rooms with a low `max_lifetime` in their message retention policies when running the automated purge jobs.

View File

@ -1 +0,0 @@
Add StateMap type alias to simplify types.

View File

@ -1 +0,0 @@
Add a `DeltaState` to track changes to be made to current state during event persistence.

View File

@ -1 +0,0 @@
Add more logging around message retention policies support.

View File

@ -1 +0,0 @@
Fix a bug causing the `synapse_port_db` script to return 0 in a specific error case.

View File

@ -1 +0,0 @@
Add a new admin API to list and filter rooms on the server.

View File

@ -1 +0,0 @@
Updates to the SAML mapping provider API.

View File

@ -1 +0,0 @@
When processing a SAML response, log the assertions for easier configuration.

View File

@ -1 +0,0 @@
Add `local_current_membership` table for tracking local user membership state in rooms.

1
changelog.d/6729.misc Normal file
View File

@ -0,0 +1 @@
Record room versions in the `rooms` table.

View File

@ -1 +0,0 @@
Fix changing password via user admin API.

View File

@ -1 +0,0 @@
Fix `/events/:event_id` deprecated API.

View File

@ -1 +0,0 @@
Fixup `synapse.rest` to pass mypy.

View File

@ -1 +0,0 @@
Fixup synapse.api to pass mypy.

1
changelog.d/6734.bugfix Normal file
View File

@ -0,0 +1 @@
Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS).

View File

@ -1 +0,0 @@
Fix monthly active user limiting support for worker mode, fixes [#4639](https://github.com/matrix-org/synapse/issues/4639).

View File

@ -1 +0,0 @@
Fix bug when setting `account_validity` to an empty block in the config. Thanks to @Sorunome for reporting.

1
changelog.d/6748.misc Normal file
View File

@ -0,0 +1 @@
Propagate cache invalidates from workers to other workers.

View File

@ -1 +0,0 @@
Allow streaming cache 'invalidate all' to workers.

1
changelog.d/6751.misc Normal file
View File

@ -0,0 +1 @@
Remove some unnecessary admin handler abstraction methods.

View File

@ -1 +0,0 @@
Fix `AttributeError: 'NoneType' object has no attribute 'get'` in `hash_password` when configuration has an empty `password_config`. Contributed by @ivilata.

View File

@ -1 +0,0 @@
Remove unused CI docker compose files.

View File

@ -1 +0,0 @@
Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media.

1
changelog.d/6757.misc Normal file
View File

@ -0,0 +1 @@
Add some debugging for media storage providers.

1
changelog.d/6761.bugfix Normal file
View File

@ -0,0 +1 @@
Minor fixes to `PUT /_synapse/admin/v2/users` admin api.

View File

@ -1 +0,0 @@
Fixup `synapse.rest` to pass mypy.

1
changelog.d/6767.bugfix Normal file
View File

@ -0,0 +1 @@
Validate `client_secret` parameter using the regex provided by the Client-Server API, temporarily allowing `:` characters for older clients. The `:` character will be removed in a future release.

1
changelog.d/6771.bugfix Normal file
View File

@ -0,0 +1 @@
Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key).

1
changelog.d/6775.doc Normal file
View File

@ -0,0 +1 @@
Clarify documentation related to `user_dir` and `federation_reader` workers.

1
changelog.d/6776.misc Normal file
View File

@ -0,0 +1 @@
Detect unknown remote devices and mark cache as stale.

1
changelog.d/6786.misc Normal file
View File

@ -0,0 +1 @@
Attempt to resync remote users' devices when detected as stale.

1
changelog.d/6787.feature Normal file
View File

@ -0,0 +1 @@
Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260).

1
changelog.d/6788.misc Normal file
View File

@ -0,0 +1 @@
Record room versions in the `rooms` table.

1
changelog.d/6790.feature Normal file
View File

@ -0,0 +1 @@
Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260).

1
changelog.d/6792.misc Normal file
View File

@ -0,0 +1 @@
Delete current state from the database when server leaves a room.

1
changelog.d/6794.feature Normal file
View File

@ -0,0 +1 @@
Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260).

1
changelog.d/6795.bugfix Normal file
View File

@ -0,0 +1 @@
Fix outbound federation request metrics.

1
changelog.d/6796.bugfix Normal file
View File

@ -0,0 +1 @@
Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device.

1
changelog.d/6797.misc Normal file
View File

@ -0,0 +1 @@
When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale.

1
changelog.d/6799.bugfix Normal file
View File

@ -0,0 +1 @@
Fix race in federation sender worker that delayed sending of device updates.

1
changelog.d/6800.bugfix Normal file
View File

@ -0,0 +1 @@
Fix race in federation sender worker that delayed sending of device updates.

1
changelog.d/6801.bugfix Normal file
View File

@ -0,0 +1 @@
Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room.

1
changelog.d/6802.misc Normal file
View File

@ -0,0 +1 @@
Add background update to clean out left rooms from current state.

1
changelog.d/6803.misc Normal file
View File

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6805.misc Normal file
View File

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6806.misc Normal file
View File

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6807.misc Normal file
View File

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6810.misc Normal file
View File

@ -0,0 +1 @@
Record room versions in the `rooms` table.

1
changelog.d/6811.bugfix Normal file
View File

@ -0,0 +1 @@
Fix waking up other workers when remote server is detected to have come back online.

1
changelog.d/6816.misc Normal file
View File

@ -0,0 +1 @@
Add background update to clean out left rooms from current state.

1
changelog.d/6819.misc Normal file
View File

@ -0,0 +1 @@
Detect unknown remote devices and mark cache as stale.

1
changelog.d/6820.misc Normal file
View File

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

12
debian/changelog vendored
View File

@ -1,3 +1,15 @@
matrix-synapse-py3 (1.9.1) stable; urgency=medium
* New synapse release 1.9.1.
-- Synapse Packaging team <packages@matrix.org> Tue, 28 Jan 2020 13:09:23 +0000
matrix-synapse-py3 (1.9.0) stable; urgency=medium
* New synapse release 1.9.0.
-- Synapse Packaging team <packages@matrix.org> Thu, 23 Jan 2020 12:56:31 +0000
matrix-synapse-py3 (1.8.0) stable; urgency=medium matrix-synapse-py3 (1.8.0) stable; urgency=medium
[ Richard van der Hoff ] [ Richard van der Hoff ]

View File

@ -46,7 +46,7 @@ The following fields are possible in the JSON response body:
A standard request with no filtering: A standard request with no filtering:
``` ```
GET /_synapse/admin/rooms GET /_synapse/admin/v1/rooms
{} {}
``` ```
@ -78,7 +78,7 @@ Response:
Filtering by room name: Filtering by room name:
``` ```
GET /_synapse/admin/rooms?search_term=TWIM GET /_synapse/admin/v1/rooms?search_term=TWIM
{} {}
``` ```
@ -103,7 +103,7 @@ Response:
Paginating through a list of rooms: Paginating through a list of rooms:
``` ```
GET /_synapse/admin/rooms?order_by=size GET /_synapse/admin/v1/rooms?order_by=size
{} {}
``` ```
@ -139,7 +139,7 @@ To get the next batch of room results, we repeat our request, setting the `from`
parameter to the value of `next_token`. parameter to the value of `next_token`.
``` ```
GET /_synapse/admin/rooms?order_by=size&from=100 GET /_synapse/admin/v1/rooms?order_by=size&from=100
{} {}
``` ```

View File

@ -32,7 +32,7 @@ Assuming your PostgreSQL database user is called `postgres`, first authenticate
su - postgres su - postgres
# Or, if your system uses sudo to get administrative rights # Or, if your system uses sudo to get administrative rights
sudo -u postgres bash sudo -u postgres bash
Then, create a user ``synapse_user`` with: Then, create a user ``synapse_user`` with:
createuser --pwprompt synapse_user createuser --pwprompt synapse_user
@ -63,6 +63,24 @@ You may need to enable password authentication so `synapse_user` can
connect to the database. See connect to the database. See
<https://www.postgresql.org/docs/11/auth-pg-hba-conf.html>. <https://www.postgresql.org/docs/11/auth-pg-hba-conf.html>.
### Fixing incorrect `COLLATE` or `CTYPE`
Synapse will refuse to set up a new database if it has the wrong values of
`COLLATE` and `CTYPE` set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to take a dump and recreate the database with
the correct `COLLATE` and `CTYPE` parameters (as per
[docs/postgres.md](docs/postgres.md)). It is also possible to change the
parameters on a live database and run a `REINDEX` on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
## Tuning Postgres ## Tuning Postgres
The default settings should be fine for most deployments. For larger The default settings should be fine for most deployments. For larger

View File

@ -185,6 +185,19 @@ reverse-proxy configuration.
The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single
instance. instance.
Note that `federation` must be added to the listener resources in the worker config:
```yaml
worker_app: synapse.app.federation_reader
...
worker_listeners:
- type: http
port: <port>
resources:
- names:
- federation
```
### `synapse.app.federation_sender` ### `synapse.app.federation_sender`
Handles sending federation traffic to other servers. Doesn't handle any Handles sending federation traffic to other servers. Doesn't handle any
@ -265,6 +278,10 @@ the following regular expressions:
^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$ ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
When using this worker you must also set `update_user_directory: False` in the
shared configuration file to stop the main synapse running background
jobs related to updating the user directory.
### `synapse.app.frontend_proxy` ### `synapse.app.frontend_proxy`
Proxies some frequently-requested client endpoints to add caching and remove Proxies some frequently-requested client endpoints to add caching and remove

View File

@ -36,7 +36,7 @@ try:
except ImportError: except ImportError:
pass pass
__version__ = "1.9.0.dev2" __version__ = "1.9.1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when # We import here so that we don't have to install a bunch of deps when

View File

@ -33,6 +33,7 @@ from synapse.api.errors import (
MissingClientTokenError, MissingClientTokenError,
ResourceLimitError, ResourceLimitError,
) )
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
from synapse.config.server import is_threepid_reserved from synapse.config.server import is_threepid_reserved
from synapse.types import StateMap, UserID from synapse.types import StateMap, UserID
from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache
@ -77,15 +78,17 @@ class Auth(object):
self._account_validity = hs.config.account_validity self._account_validity = hs.config.account_validity
@defer.inlineCallbacks @defer.inlineCallbacks
def check_from_context(self, room_version, event, context, do_sig_check=True): def check_from_context(self, room_version: str, event, context, do_sig_check=True):
prev_state_ids = yield context.get_prev_state_ids() prev_state_ids = yield context.get_prev_state_ids()
auth_events_ids = yield self.compute_auth_events( auth_events_ids = yield self.compute_auth_events(
event, prev_state_ids, for_verification=True event, prev_state_ids, for_verification=True
) )
auth_events = yield self.store.get_events(auth_events_ids) auth_events = yield self.store.get_events(auth_events_ids)
auth_events = {(e.type, e.state_key): e for e in itervalues(auth_events)} auth_events = {(e.type, e.state_key): e for e in itervalues(auth_events)}
room_version_obj = KNOWN_ROOM_VERSIONS[room_version]
event_auth.check( event_auth.check(
room_version, event, auth_events=auth_events, do_sig_check=do_sig_check room_version_obj, event, auth_events=auth_events, do_sig_check=do_sig_check
) )
@defer.inlineCallbacks @defer.inlineCallbacks

View File

@ -402,11 +402,9 @@ class UnsupportedRoomVersionError(SynapseError):
"""The client's request to create a room used a room version that the server does """The client's request to create a room used a room version that the server does
not support.""" not support."""
def __init__(self): def __init__(self, msg="Homeserver does not support this room version"):
super(UnsupportedRoomVersionError, self).__init__( super(UnsupportedRoomVersionError, self).__init__(
code=400, code=400, msg=msg, errcode=Codes.UNSUPPORTED_ROOM_VERSION,
msg="Homeserver does not support this room version",
errcode=Codes.UNSUPPORTED_ROOM_VERSION,
) )

View File

@ -57,6 +57,9 @@ class RoomVersion(object):
state_res = attr.ib() # int; one of the StateResolutionVersions state_res = attr.ib() # int; one of the StateResolutionVersions
enforce_key_validity = attr.ib() # bool enforce_key_validity = attr.ib() # bool
# bool: before MSC2260, anyone was allowed to send an aliases event
special_case_aliases_auth = attr.ib(type=bool, default=False)
class RoomVersions(object): class RoomVersions(object):
V1 = RoomVersion( V1 = RoomVersion(
@ -65,6 +68,7 @@ class RoomVersions(object):
EventFormatVersions.V1, EventFormatVersions.V1,
StateResolutionVersions.V1, StateResolutionVersions.V1,
enforce_key_validity=False, enforce_key_validity=False,
special_case_aliases_auth=True,
) )
V2 = RoomVersion( V2 = RoomVersion(
"2", "2",
@ -72,6 +76,7 @@ class RoomVersions(object):
EventFormatVersions.V1, EventFormatVersions.V1,
StateResolutionVersions.V2, StateResolutionVersions.V2,
enforce_key_validity=False, enforce_key_validity=False,
special_case_aliases_auth=True,
) )
V3 = RoomVersion( V3 = RoomVersion(
"3", "3",
@ -79,6 +84,7 @@ class RoomVersions(object):
EventFormatVersions.V2, EventFormatVersions.V2,
StateResolutionVersions.V2, StateResolutionVersions.V2,
enforce_key_validity=False, enforce_key_validity=False,
special_case_aliases_auth=True,
) )
V4 = RoomVersion( V4 = RoomVersion(
"4", "4",
@ -86,6 +92,7 @@ class RoomVersions(object):
EventFormatVersions.V3, EventFormatVersions.V3,
StateResolutionVersions.V2, StateResolutionVersions.V2,
enforce_key_validity=False, enforce_key_validity=False,
special_case_aliases_auth=True,
) )
V5 = RoomVersion( V5 = RoomVersion(
"5", "5",
@ -93,6 +100,14 @@ class RoomVersions(object):
EventFormatVersions.V3, EventFormatVersions.V3,
StateResolutionVersions.V2, StateResolutionVersions.V2,
enforce_key_validity=True, enforce_key_validity=True,
special_case_aliases_auth=True,
)
MSC2260_DEV = RoomVersion(
"org.matrix.msc2260",
RoomDisposition.UNSTABLE,
EventFormatVersions.V3,
StateResolutionVersions.V2,
enforce_key_validity=True,
) )
@ -104,5 +119,6 @@ KNOWN_ROOM_VERSIONS = {
RoomVersions.V3, RoomVersions.V3,
RoomVersions.V4, RoomVersions.V4,
RoomVersions.V5, RoomVersions.V5,
RoomVersions.MSC2260_DEV,
) )
} # type: Dict[str, RoomVersion] } # type: Dict[str, RoomVersion]

View File

@ -38,7 +38,11 @@ from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
from synapse.replication.slave.storage.registration import SlavedRegistrationStore from synapse.replication.slave.storage.registration import SlavedRegistrationStore
from synapse.replication.slave.storage.transactions import SlavedTransactionStore from synapse.replication.slave.storage.transactions import SlavedTransactionStore
from synapse.replication.tcp.client import ReplicationClientHandler from synapse.replication.tcp.client import ReplicationClientHandler
from synapse.replication.tcp.streams._base import ReceiptsStream from synapse.replication.tcp.streams._base import (
DeviceListsStream,
ReceiptsStream,
ToDeviceStream,
)
from synapse.server import HomeServer from synapse.server import HomeServer
from synapse.storage.database import Database from synapse.storage.database import Database
from synapse.types import ReadReceipt from synapse.types import ReadReceipt
@ -256,6 +260,20 @@ class FederationSenderHandler(object):
"process_receipts_for_federation", self._on_new_receipts, rows "process_receipts_for_federation", self._on_new_receipts, rows
) )
# ... as well as device updates and messages
elif stream_name == DeviceListsStream.NAME:
hosts = set(row.destination for row in rows)
for host in hosts:
self.federation_sender.send_device_messages(host)
elif stream_name == ToDeviceStream.NAME:
# The to_device stream includes stuff to be pushed to both local
# clients and remote servers, so we ignore entities that start with
# '@' (since they'll be local users rather than destinations).
hosts = set(row.entity for row in rows if not row.entity.startswith("@"))
for host in hosts:
self.federation_sender.send_device_messages(host)
@defer.inlineCallbacks @defer.inlineCallbacks
def _on_new_receipts(self, rows): def _on_new_receipts(self, rows):
""" """

View File

@ -1,6 +1,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
#
# Copyright 2014-2016 OpenMarket Ltd # Copyright 2014-2016 OpenMarket Ltd
# Copyright 2020 The Matrix.org Foundation C.I.C.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -17,13 +18,17 @@
import collections.abc import collections.abc
import hashlib import hashlib
import logging import logging
from typing import Dict
from canonicaljson import encode_canonical_json from canonicaljson import encode_canonical_json
from signedjson.sign import sign_json from signedjson.sign import sign_json
from signedjson.types import SigningKey
from unpaddedbase64 import decode_base64, encode_base64 from unpaddedbase64 import decode_base64, encode_base64
from synapse.api.errors import Codes, SynapseError from synapse.api.errors import Codes, SynapseError
from synapse.api.room_versions import RoomVersion
from synapse.events.utils import prune_event, prune_event_dict from synapse.events.utils import prune_event, prune_event_dict
from synapse.types import JsonDict
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -112,18 +117,28 @@ def compute_event_reference_hash(event, hash_algorithm=hashlib.sha256):
return hashed.name, hashed.digest() return hashed.name, hashed.digest()
def compute_event_signature(event_dict, signature_name, signing_key): def compute_event_signature(
room_version: RoomVersion,
event_dict: JsonDict,
signature_name: str,
signing_key: SigningKey,
) -> Dict[str, Dict[str, str]]:
"""Compute the signature of the event for the given name and key. """Compute the signature of the event for the given name and key.
Args: Args:
event_dict (dict): The event as a dict room_version: the version of the room that this event is in.
signature_name (str): The name of the entity signing the event (the room version determines the redaction algorithm and hence the
json to be signed)
event_dict: The event as a dict
signature_name: The name of the entity signing the event
(typically the server's hostname). (typically the server's hostname).
signing_key (syutil.crypto.SigningKey): The key to sign with
signing_key: The key to sign with
Returns: Returns:
dict[str, dict[str, str]]: Returns a dictionary in the same format of a dictionary in the same format of an event's signatures field.
an event's signatures field.
""" """
redact_json = prune_event_dict(event_dict) redact_json = prune_event_dict(event_dict)
redact_json.pop("age_ts", None) redact_json.pop("age_ts", None)
@ -137,23 +152,26 @@ def compute_event_signature(event_dict, signature_name, signing_key):
def add_hashes_and_signatures( def add_hashes_and_signatures(
event_dict, signature_name, signing_key, hash_algorithm=hashlib.sha256 room_version: RoomVersion,
event_dict: JsonDict,
signature_name: str,
signing_key: SigningKey,
): ):
"""Add content hash and sign the event """Add content hash and sign the event
Args: Args:
event_dict (dict): The event to add hashes to and sign room_version: the version of the room this event is in
signature_name (str): The name of the entity signing the event
event_dict: The event to add hashes to and sign
signature_name: The name of the entity signing the event
(typically the server's hostname). (typically the server's hostname).
signing_key (syutil.crypto.SigningKey): The key to sign with signing_key: The key to sign with
hash_algorithm: A hasher from `hashlib`, e.g. hashlib.sha256, to use
to hash the event
""" """
name, digest = compute_content_hash(event_dict, hash_algorithm=hash_algorithm) name, digest = compute_content_hash(event_dict, hash_algorithm=hashlib.sha256)
event_dict.setdefault("hashes", {})[name] = encode_base64(digest) event_dict.setdefault("hashes", {})[name] = encode_base64(digest)
event_dict["signatures"] = compute_event_signature( event_dict["signatures"] = compute_event_signature(
event_dict, signature_name=signature_name, signing_key=signing_key room_version, event_dict, signature_name=signature_name, signing_key=signing_key
) )

View File

@ -1,5 +1,6 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# Copyright 2014 - 2016 OpenMarket Ltd # Copyright 2014 - 2016 OpenMarket Ltd
# Copyright 2020 The Matrix.org Foundation C.I.C.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -23,17 +24,27 @@ from unpaddedbase64 import decode_base64
from synapse.api.constants import EventTypes, JoinRules, Membership from synapse.api.constants import EventTypes, JoinRules, Membership
from synapse.api.errors import AuthError, EventSizeError, SynapseError from synapse.api.errors import AuthError, EventSizeError, SynapseError
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, EventFormatVersions from synapse.api.room_versions import (
KNOWN_ROOM_VERSIONS,
EventFormatVersions,
RoomVersion,
)
from synapse.types import UserID, get_domain_from_id from synapse.types import UserID, get_domain_from_id
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def check(room_version, event, auth_events, do_sig_check=True, do_size_check=True): def check(
room_version_obj: RoomVersion,
event,
auth_events,
do_sig_check=True,
do_size_check=True,
):
""" Checks if this event is correctly authed. """ Checks if this event is correctly authed.
Args: Args:
room_version (str): the version of the room room_version_obj: the version of the room
event: the event being checked. event: the event being checked.
auth_events (dict: event-key -> event): the existing room state. auth_events (dict: event-key -> event): the existing room state.
@ -89,7 +100,12 @@ def check(room_version, event, auth_events, do_sig_check=True, do_size_check=Tru
if not event.signatures.get(event_id_domain): if not event.signatures.get(event_id_domain):
raise AuthError(403, "Event not signed by sending server") raise AuthError(403, "Event not signed by sending server")
# Implementation of https://matrix.org/docs/spec/rooms/v1#authorization-rules
#
# 1. If type is m.room.create:
if event.type == EventTypes.Create: if event.type == EventTypes.Create:
# 1b. If the domain of the room_id does not match the domain of the sender,
# reject.
sender_domain = get_domain_from_id(event.sender) sender_domain = get_domain_from_id(event.sender)
room_id_domain = get_domain_from_id(event.room_id) room_id_domain = get_domain_from_id(event.room_id)
if room_id_domain != sender_domain: if room_id_domain != sender_domain:
@ -97,39 +113,49 @@ def check(room_version, event, auth_events, do_sig_check=True, do_size_check=Tru
403, "Creation event's room_id domain does not match sender's" 403, "Creation event's room_id domain does not match sender's"
) )
room_version = event.content.get("room_version", "1") # 1c. If content.room_version is present and is not a recognised version, reject
if room_version not in KNOWN_ROOM_VERSIONS: room_version_prop = event.content.get("room_version", "1")
if room_version_prop not in KNOWN_ROOM_VERSIONS:
raise AuthError( raise AuthError(
403, "room appears to have unsupported version %s" % (room_version,) 403,
"room appears to have unsupported version %s" % (room_version_prop,),
) )
# FIXME
logger.debug("Allowing! %s", event) logger.debug("Allowing! %s", event)
return return
# 3. If event does not have a m.room.create in its auth_events, reject.
creation_event = auth_events.get((EventTypes.Create, ""), None) creation_event = auth_events.get((EventTypes.Create, ""), None)
if not creation_event: if not creation_event:
raise AuthError(403, "No create event in auth events") raise AuthError(403, "No create event in auth events")
# additional check for m.federate
creating_domain = get_domain_from_id(event.room_id) creating_domain = get_domain_from_id(event.room_id)
originating_domain = get_domain_from_id(event.sender) originating_domain = get_domain_from_id(event.sender)
if creating_domain != originating_domain: if creating_domain != originating_domain:
if not _can_federate(event, auth_events): if not _can_federate(event, auth_events):
raise AuthError(403, "This room has been marked as unfederatable.") raise AuthError(403, "This room has been marked as unfederatable.")
# FIXME: Temp hack # 4. If type is m.room.aliases
if event.type == EventTypes.Aliases: if event.type == EventTypes.Aliases:
# 4a. If event has no state_key, reject
if not event.is_state(): if not event.is_state():
raise AuthError(403, "Alias event must be a state event") raise AuthError(403, "Alias event must be a state event")
if not event.state_key: if not event.state_key:
raise AuthError(403, "Alias event must have non-empty state_key") raise AuthError(403, "Alias event must have non-empty state_key")
# 4b. If sender's domain doesn't matches [sic] state_key, reject
sender_domain = get_domain_from_id(event.sender) sender_domain = get_domain_from_id(event.sender)
if event.state_key != sender_domain: if event.state_key != sender_domain:
raise AuthError( raise AuthError(
403, "Alias event's state_key does not match sender's domain" 403, "Alias event's state_key does not match sender's domain"
) )
logger.debug("Allowing! %s", event)
return # 4c. Otherwise, allow.
# This is removed by https://github.com/matrix-org/matrix-doc/pull/2260
if room_version_obj.special_case_aliases_auth:
logger.debug("Allowing! %s", event)
return
if logger.isEnabledFor(logging.DEBUG): if logger.isEnabledFor(logging.DEBUG):
logger.debug("Auth events: %s", [a.event_id for a in auth_events.values()]) logger.debug("Auth events: %s", [a.event_id for a in auth_events.values()])
@ -160,7 +186,7 @@ def check(room_version, event, auth_events, do_sig_check=True, do_size_check=Tru
_check_power_levels(event, auth_events) _check_power_levels(event, auth_events)
if event.type == EventTypes.Redaction: if event.type == EventTypes.Redaction:
check_redaction(room_version, event, auth_events) check_redaction(room_version_obj, event, auth_events)
logger.debug("Allowing! %s", event) logger.debug("Allowing! %s", event)
@ -386,7 +412,7 @@ def _can_send_event(event, auth_events):
return True return True
def check_redaction(room_version, event, auth_events): def check_redaction(room_version_obj: RoomVersion, event, auth_events):
"""Check whether the event sender is allowed to redact the target event. """Check whether the event sender is allowed to redact the target event.
Returns: Returns:
@ -406,11 +432,7 @@ def check_redaction(room_version, event, auth_events):
if user_level >= redact_level: if user_level >= redact_level:
return False return False
v = KNOWN_ROOM_VERSIONS.get(room_version) if room_version_obj.event_format == EventFormatVersions.V1:
if not v:
raise RuntimeError("Unrecognized room version %r" % (room_version,))
if v.event_format == EventFormatVersions.V1:
redacter_domain = get_domain_from_id(event.event_id) redacter_domain = get_domain_from_id(event.event_id)
redactee_domain = get_domain_from_id(event.redacts) redactee_domain = get_domain_from_id(event.redacts)
if redacter_domain == redactee_domain: if redacter_domain == redactee_domain:

View File

@ -23,6 +23,7 @@ from unpaddedbase64 import encode_base64
from synapse.api.errors import UnsupportedRoomVersionError from synapse.api.errors import UnsupportedRoomVersionError
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, EventFormatVersions from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, EventFormatVersions
from synapse.types import JsonDict
from synapse.util.caches import intern_dict from synapse.util.caches import intern_dict
from synapse.util.frozenutils import freeze from synapse.util.frozenutils import freeze
@ -116,16 +117,32 @@ class _EventInternalMetadata(object):
return getattr(self, "redacted", False) return getattr(self, "redacted", False)
def _event_dict_property(key): _SENTINEL = object()
def _event_dict_property(key, default=_SENTINEL):
"""Creates a new property for the given key that delegates access to
`self._event_dict`.
The default is used if the key is missing from the `_event_dict`, if given,
otherwise an AttributeError will be raised.
Note: If a default is given then `hasattr` will always return true.
"""
# We want to be able to use hasattr with the event dict properties. # We want to be able to use hasattr with the event dict properties.
# However, (on python3) hasattr expects AttributeError to be raised. Hence, # However, (on python3) hasattr expects AttributeError to be raised. Hence,
# we need to transform the KeyError into an AttributeError # we need to transform the KeyError into an AttributeError
def getter(self):
def getter_raises(self):
try: try:
return self._event_dict[key] return self._event_dict[key]
except KeyError: except KeyError:
raise AttributeError(key) raise AttributeError(key)
def getter_default(self):
return self._event_dict.get(key, default)
def setter(self, v): def setter(self, v):
try: try:
self._event_dict[key] = v self._event_dict[key] = v
@ -138,7 +155,11 @@ def _event_dict_property(key):
except KeyError: except KeyError:
raise AttributeError(key) raise AttributeError(key)
return property(getter, setter, delete) if default is _SENTINEL:
# No default given, so use the getter that raises
return property(getter_raises, setter, delete)
else:
return property(getter_default, setter, delete)
class EventBase(object): class EventBase(object):
@ -165,7 +186,7 @@ class EventBase(object):
origin = _event_dict_property("origin") origin = _event_dict_property("origin")
origin_server_ts = _event_dict_property("origin_server_ts") origin_server_ts = _event_dict_property("origin_server_ts")
prev_events = _event_dict_property("prev_events") prev_events = _event_dict_property("prev_events")
redacts = _event_dict_property("redacts") redacts = _event_dict_property("redacts", None)
room_id = _event_dict_property("room_id") room_id = _event_dict_property("room_id")
sender = _event_dict_property("sender") sender = _event_dict_property("sender")
user_id = _event_dict_property("sender") user_id = _event_dict_property("sender")
@ -177,7 +198,7 @@ class EventBase(object):
def is_state(self): def is_state(self):
return hasattr(self, "state_key") and self.state_key is not None return hasattr(self, "state_key") and self.state_key is not None
def get_dict(self): def get_dict(self) -> JsonDict:
d = dict(self._event_dict) d = dict(self._event_dict)
d.update({"signatures": self.signatures, "unsigned": dict(self.unsigned)}) d.update({"signatures": self.signatures, "unsigned": dict(self.unsigned)})
@ -189,7 +210,7 @@ class EventBase(object):
def get_internal_metadata_dict(self): def get_internal_metadata_dict(self):
return self.internal_metadata.get_dict() return self.internal_metadata.get_dict()
def get_pdu_json(self, time_now=None): def get_pdu_json(self, time_now=None) -> JsonDict:
pdu_json = self.get_dict() pdu_json = self.get_dict()
if time_now is not None and "age_ts" in pdu_json["unsigned"]: if time_now is not None and "age_ts" in pdu_json["unsigned"]:

View File

@ -12,8 +12,10 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from typing import Optional
import attr import attr
from nacl.signing import SigningKey
from twisted.internet import defer from twisted.internet import defer
@ -23,13 +25,18 @@ from synapse.api.room_versions import (
KNOWN_EVENT_FORMAT_VERSIONS, KNOWN_EVENT_FORMAT_VERSIONS,
KNOWN_ROOM_VERSIONS, KNOWN_ROOM_VERSIONS,
EventFormatVersions, EventFormatVersions,
RoomVersion,
) )
from synapse.crypto.event_signing import add_hashes_and_signatures from synapse.crypto.event_signing import add_hashes_and_signatures
from synapse.types import EventID from synapse.events import (
EventBase,
_EventInternalMetadata,
event_type_from_format_version,
)
from synapse.types import EventID, JsonDict
from synapse.util import Clock
from synapse.util.stringutils import random_string from synapse.util.stringutils import random_string
from . import _EventInternalMetadata, event_type_from_format_version
@attr.s(slots=True, cmp=False, frozen=True) @attr.s(slots=True, cmp=False, frozen=True)
class EventBuilder(object): class EventBuilder(object):
@ -40,7 +47,7 @@ class EventBuilder(object):
content/unsigned/internal_metadata fields are still mutable) content/unsigned/internal_metadata fields are still mutable)
Attributes: Attributes:
format_version (int): Event format version room_version: Version of the target room
room_id (str) room_id (str)
type (str) type (str)
sender (str) sender (str)
@ -63,7 +70,7 @@ class EventBuilder(object):
_hostname = attr.ib() _hostname = attr.ib()
_signing_key = attr.ib() _signing_key = attr.ib()
format_version = attr.ib() room_version = attr.ib(type=RoomVersion)
room_id = attr.ib() room_id = attr.ib()
type = attr.ib() type = attr.ib()
@ -108,7 +115,8 @@ class EventBuilder(object):
) )
auth_ids = yield self._auth.compute_auth_events(self, state_ids) auth_ids = yield self._auth.compute_auth_events(self, state_ids)
if self.format_version == EventFormatVersions.V1: format_version = self.room_version.event_format
if format_version == EventFormatVersions.V1:
auth_events = yield self._store.add_event_hashes(auth_ids) auth_events = yield self._store.add_event_hashes(auth_ids)
prev_events = yield self._store.add_event_hashes(prev_event_ids) prev_events = yield self._store.add_event_hashes(prev_event_ids)
else: else:
@ -148,7 +156,7 @@ class EventBuilder(object):
clock=self._clock, clock=self._clock,
hostname=self._hostname, hostname=self._hostname,
signing_key=self._signing_key, signing_key=self._signing_key,
format_version=self.format_version, room_version=self.room_version,
event_dict=event_dict, event_dict=event_dict,
internal_metadata_dict=self.internal_metadata.get_dict(), internal_metadata_dict=self.internal_metadata.get_dict(),
) )
@ -201,7 +209,7 @@ class EventBuilderFactory(object):
clock=self.clock, clock=self.clock,
hostname=self.hostname, hostname=self.hostname,
signing_key=self.signing_key, signing_key=self.signing_key,
format_version=room_version.event_format, room_version=room_version,
type=key_values["type"], type=key_values["type"],
state_key=key_values.get("state_key"), state_key=key_values.get("state_key"),
room_id=key_values["room_id"], room_id=key_values["room_id"],
@ -214,29 +222,19 @@ class EventBuilderFactory(object):
def create_local_event_from_event_dict( def create_local_event_from_event_dict(
clock, clock: Clock,
hostname, hostname: str,
signing_key, signing_key: SigningKey,
format_version, room_version: RoomVersion,
event_dict, event_dict: JsonDict,
internal_metadata_dict=None, internal_metadata_dict: Optional[JsonDict] = None,
): ) -> EventBase:
"""Takes a fully formed event dict, ensuring that fields like `origin` """Takes a fully formed event dict, ensuring that fields like `origin`
and `origin_server_ts` have correct values for a locally produced event, and `origin_server_ts` have correct values for a locally produced event,
then signs and hashes it. then signs and hashes it.
Args:
clock (Clock)
hostname (str)
signing_key
format_version (int)
event_dict (dict)
internal_metadata_dict (dict|None)
Returns:
FrozenEvent
""" """
format_version = room_version.event_format
if format_version not in KNOWN_EVENT_FORMAT_VERSIONS: if format_version not in KNOWN_EVENT_FORMAT_VERSIONS:
raise Exception("No event format defined for version %r" % (format_version,)) raise Exception("No event format defined for version %r" % (format_version,))
@ -257,7 +255,7 @@ def create_local_event_from_event_dict(
event_dict.setdefault("signatures", {}) event_dict.setdefault("signatures", {})
add_hashes_and_signatures(event_dict, hostname, signing_key) add_hashes_and_signatures(room_version, event_dict, hostname, signing_key)
return event_type_from_format_version(format_version)( return event_type_from_format_version(format_version)(
event_dict, internal_metadata_dict=internal_metadata_dict event_dict, internal_metadata_dict=internal_metadata_dict
) )

View File

@ -12,8 +12,9 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import collections
import re import re
from typing import Mapping, Union
from six import string_types from six import string_types
@ -422,3 +423,37 @@ class EventClientSerializer(object):
return yieldable_gather_results( return yieldable_gather_results(
self.serialize_event, events, time_now=time_now, **kwargs self.serialize_event, events, time_now=time_now, **kwargs
) )
def copy_power_levels_contents(
old_power_levels: Mapping[str, Union[int, Mapping[str, int]]]
):
"""Copy the content of a power_levels event, unfreezing frozendicts along the way
Raises:
TypeError if the input does not look like a valid power levels event content
"""
if not isinstance(old_power_levels, collections.Mapping):
raise TypeError("Not a valid power-levels content: %r" % (old_power_levels,))
power_levels = {}
for k, v in old_power_levels.items():
if isinstance(v, int):
power_levels[k] = v
continue
if isinstance(v, collections.Mapping):
power_levels[k] = h = {}
for k1, v1 in v.items():
# we should only have one level of nesting
if not isinstance(v1, int):
raise TypeError(
"Invalid power_levels value for %s.%s: %r" % (k, k1, v1)
)
h[k1] = v1
continue
raise TypeError("Invalid power_levels value for %s: %r" % (k, v))
return power_levels

View File

@ -17,6 +17,7 @@
import copy import copy
import itertools import itertools
import logging import logging
from typing import Dict, Iterable
from prometheus_client import Counter from prometheus_client import Counter
@ -29,6 +30,7 @@ from synapse.api.errors import (
FederationDeniedError, FederationDeniedError,
HttpResponseException, HttpResponseException,
SynapseError, SynapseError,
UnsupportedRoomVersionError,
) )
from synapse.api.room_versions import ( from synapse.api.room_versions import (
KNOWN_ROOM_VERSIONS, KNOWN_ROOM_VERSIONS,
@ -196,7 +198,7 @@ class FederationClient(FederationBase):
logger.debug("backfill transaction_data=%r", transaction_data) logger.debug("backfill transaction_data=%r", transaction_data)
room_version = yield self.store.get_room_version(room_id) room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
pdus = [ pdus = [
@ -334,7 +336,7 @@ class FederationClient(FederationBase):
def get_event_auth(self, destination, room_id, event_id): def get_event_auth(self, destination, room_id, event_id):
res = yield self.transport_layer.get_event_auth(destination, room_id, event_id) res = yield self.transport_layer.get_event_auth(destination, room_id, event_id)
room_version = yield self.store.get_room_version(room_id) room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
auth_chain = [ auth_chain = [
@ -385,6 +387,8 @@ class FederationClient(FederationBase):
return res return res
except InvalidResponseError as e: except InvalidResponseError as e:
logger.warning("Failed to %s via %s: %s", description, destination, e) logger.warning("Failed to %s via %s: %s", description, destination, e)
except UnsupportedRoomVersionError:
raise
except HttpResponseException as e: except HttpResponseException as e:
if not 500 <= e.code < 600: if not 500 <= e.code < 600:
raise e.to_synapse_error() raise e.to_synapse_error()
@ -404,7 +408,13 @@ class FederationClient(FederationBase):
raise SynapseError(502, "Failed to %s via any server" % (description,)) raise SynapseError(502, "Failed to %s via any server" % (description,))
def make_membership_event( def make_membership_event(
self, destinations, room_id, user_id, membership, content, params self,
destinations: Iterable[str],
room_id: str,
user_id: str,
membership: str,
content: dict,
params: Dict[str, str],
): ):
""" """
Creates an m.room.member event, with context, without participating in the room. Creates an m.room.member event, with context, without participating in the room.
@ -417,21 +427,23 @@ class FederationClient(FederationBase):
Note that this does not append any events to any graphs. Note that this does not append any events to any graphs.
Args: Args:
destinations (Iterable[str]): Candidate homeservers which are probably destinations: Candidate homeservers which are probably
participating in the room. participating in the room.
room_id (str): The room in which the event will happen. room_id: The room in which the event will happen.
user_id (str): The user whose membership is being evented. user_id: The user whose membership is being evented.
membership (str): The "membership" property of the event. Must be membership: The "membership" property of the event. Must be one of
one of "join" or "leave". "join" or "leave".
content (dict): Any additional data to put into the content field content: Any additional data to put into the content field of the
of the event. event.
params (dict[str, str|Iterable[str]]): Query parameters to include in the params: Query parameters to include in the request.
request.
Return: Return:
Deferred[tuple[str, FrozenEvent, int]]: resolves to a tuple of Deferred[Tuple[str, FrozenEvent, RoomVersion]]: resolves to a tuple of
`(origin, event, event_format)` where origin is the remote `(origin, event, room_version)` where origin is the remote
homeserver which generated the event, and event_format is one of homeserver which generated the event, and room_version is the
`synapse.api.room_versions.EventFormatVersions`. version of the room.
Fails with a `UnsupportedRoomVersionError` if remote responds with
a room version we don't understand.
Fails with a ``SynapseError`` if the chosen remote server Fails with a ``SynapseError`` if the chosen remote server
returns a 300/400 code. returns a 300/400 code.
@ -453,8 +465,10 @@ class FederationClient(FederationBase):
# Note: If not supplied, the room version may be either v1 or v2, # Note: If not supplied, the room version may be either v1 or v2,
# however either way the event format version will be v1. # however either way the event format version will be v1.
room_version = ret.get("room_version", RoomVersions.V1.identifier) room_version_id = ret.get("room_version", RoomVersions.V1.identifier)
event_format = room_version_to_event_format(room_version) room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)
if not room_version:
raise UnsupportedRoomVersionError()
pdu_dict = ret.get("event", None) pdu_dict = ret.get("event", None)
if not isinstance(pdu_dict, dict): if not isinstance(pdu_dict, dict):
@ -474,11 +488,11 @@ class FederationClient(FederationBase):
self._clock, self._clock,
self.hostname, self.hostname,
self.signing_key, self.signing_key,
format_version=event_format, room_version=room_version,
event_dict=pdu_dict, event_dict=pdu_dict,
) )
return (destination, ev, event_format) return (destination, ev, room_version)
return self._try_destination_list( return self._try_destination_list(
"make_" + membership, destinations, send_request "make_" + membership, destinations, send_request
@ -633,7 +647,7 @@ class FederationClient(FederationBase):
@defer.inlineCallbacks @defer.inlineCallbacks
def send_invite(self, destination, room_id, event_id, pdu): def send_invite(self, destination, room_id, event_id, pdu):
room_version = yield self.store.get_room_version(room_id) room_version = yield self.store.get_room_version_id(room_id)
content = yield self._do_send_invite(destination, pdu, room_version) content = yield self._do_send_invite(destination, pdu, room_version)
@ -641,7 +655,7 @@ class FederationClient(FederationBase):
logger.debug("Got response to send_invite: %s", pdu_dict) logger.debug("Got response to send_invite: %s", pdu_dict)
room_version = yield self.store.get_room_version(room_id) room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
pdu = event_from_pdu_json(pdu_dict, format_ver) pdu = event_from_pdu_json(pdu_dict, format_ver)
@ -843,7 +857,7 @@ class FederationClient(FederationBase):
timeout=timeout, timeout=timeout,
) )
room_version = yield self.store.get_room_version(room_id) room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
events = [ events = [

View File

@ -234,7 +234,7 @@ class FederationServer(FederationBase):
continue continue
try: try:
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
except NotFoundError: except NotFoundError:
logger.info("Ignoring PDU for unknown room_id: %s", room_id) logger.info("Ignoring PDU for unknown room_id: %s", room_id)
continue continue
@ -334,7 +334,7 @@ class FederationServer(FederationBase):
) )
) )
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
resp["room_version"] = room_version resp["room_version"] = room_version
return 200, resp return 200, resp
@ -385,7 +385,7 @@ class FederationServer(FederationBase):
origin_host, _ = parse_server_name(origin) origin_host, _ = parse_server_name(origin)
await self.check_server_matches_acl(origin_host, room_id) await self.check_server_matches_acl(origin_host, room_id)
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
if room_version not in supported_versions: if room_version not in supported_versions:
logger.warning( logger.warning(
"Room version %s not in %s", room_version, supported_versions "Room version %s not in %s", room_version, supported_versions
@ -410,14 +410,14 @@ class FederationServer(FederationBase):
origin_host, _ = parse_server_name(origin) origin_host, _ = parse_server_name(origin)
await self.check_server_matches_acl(origin_host, pdu.room_id) await self.check_server_matches_acl(origin_host, pdu.room_id)
pdu = await self._check_sigs_and_hash(room_version, pdu) pdu = await self._check_sigs_and_hash(room_version, pdu)
ret_pdu = await self.handler.on_invite_request(origin, pdu) ret_pdu = await self.handler.on_invite_request(origin, pdu, room_version)
time_now = self._clock.time_msec() time_now = self._clock.time_msec()
return {"event": ret_pdu.get_pdu_json(time_now)} return {"event": ret_pdu.get_pdu_json(time_now)}
async def on_send_join_request(self, origin, content, room_id): async def on_send_join_request(self, origin, content, room_id):
logger.debug("on_send_join_request: content: %s", content) logger.debug("on_send_join_request: content: %s", content)
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
pdu = event_from_pdu_json(content, format_ver) pdu = event_from_pdu_json(content, format_ver)
@ -440,7 +440,7 @@ class FederationServer(FederationBase):
await self.check_server_matches_acl(origin_host, room_id) await self.check_server_matches_acl(origin_host, room_id)
pdu = await self.handler.on_make_leave_request(origin, room_id, user_id) pdu = await self.handler.on_make_leave_request(origin, room_id, user_id)
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
time_now = self._clock.time_msec() time_now = self._clock.time_msec()
return {"event": pdu.get_pdu_json(time_now), "room_version": room_version} return {"event": pdu.get_pdu_json(time_now), "room_version": room_version}
@ -448,7 +448,7 @@ class FederationServer(FederationBase):
async def on_send_leave_request(self, origin, content, room_id): async def on_send_leave_request(self, origin, content, room_id):
logger.debug("on_send_leave_request: content: %s", content) logger.debug("on_send_leave_request: content: %s", content)
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
pdu = event_from_pdu_json(content, format_ver) pdu = event_from_pdu_json(content, format_ver)
@ -495,7 +495,7 @@ class FederationServer(FederationBase):
origin_host, _ = parse_server_name(origin) origin_host, _ = parse_server_name(origin)
await self.check_server_matches_acl(origin_host, room_id) await self.check_server_matches_acl(origin_host, room_id)
room_version = await self.store.get_room_version(room_id) room_version = await self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version) format_ver = room_version_to_event_format(room_version)
auth_chain = [ auth_chain = [
@ -664,7 +664,7 @@ class FederationServer(FederationBase):
logger.info("Accepting join PDU %s from %s", pdu.event_id, origin) logger.info("Accepting join PDU %s from %s", pdu.event_id, origin)
# We've already checked that we know the room version by this point # We've already checked that we know the room version by this point
room_version = await self.store.get_room_version(pdu.room_id) room_version = await self.store.get_room_version_id(pdu.room_id)
# Check signature. # Check signature.
try: try:

View File

@ -69,8 +69,6 @@ class FederationRemoteSendQueue(object):
self.edus = SortedDict() # stream position -> Edu self.edus = SortedDict() # stream position -> Edu
self.device_messages = SortedDict() # stream position -> destination
self.pos = 1 self.pos = 1
self.pos_time = SortedDict() self.pos_time = SortedDict()
@ -92,7 +90,6 @@ class FederationRemoteSendQueue(object):
"keyed_edu", "keyed_edu",
"keyed_edu_changed", "keyed_edu_changed",
"edus", "edus",
"device_messages",
"pos_time", "pos_time",
"presence_destinations", "presence_destinations",
]: ]:
@ -171,12 +168,6 @@ class FederationRemoteSendQueue(object):
for key in keys[:i]: for key in keys[:i]:
del self.edus[key] del self.edus[key]
# Delete things out of device map
keys = self.device_messages.keys()
i = self.device_messages.bisect_left(position_to_delete)
for key in keys[:i]:
del self.device_messages[key]
def notify_new_events(self, current_id): def notify_new_events(self, current_id):
"""As per FederationSender""" """As per FederationSender"""
# We don't need to replicate this as it gets sent down a different # We don't need to replicate this as it gets sent down a different
@ -249,9 +240,8 @@ class FederationRemoteSendQueue(object):
def send_device_messages(self, destination): def send_device_messages(self, destination):
"""As per FederationSender""" """As per FederationSender"""
pos = self._next_pos() # We don't need to replicate this as it gets sent down a different
self.device_messages[pos] = destination # stream.
self.notifier.on_new_replication_data()
def get_current_token(self): def get_current_token(self):
return self.pos - 1 return self.pos - 1
@ -339,14 +329,6 @@ class FederationRemoteSendQueue(object):
for (pos, edu) in edus: for (pos, edu) in edus:
rows.append((pos, EduRow(edu))) rows.append((pos, EduRow(edu)))
# Fetch changed device messages
i = self.device_messages.bisect_right(from_token)
j = self.device_messages.bisect_right(to_token) + 1
device_messages = {v: k for k, v in self.device_messages.items()[i:j]}
for (destination, pos) in iteritems(device_messages):
rows.append((pos, DeviceRow(destination=destination)))
# Sort rows based on pos # Sort rows based on pos
rows.sort() rows.sort()
@ -472,28 +454,9 @@ class EduRow(BaseFederationRow, namedtuple("EduRow", ("edu",))): # Edu
buff.edus.setdefault(self.edu.destination, []).append(self.edu) buff.edus.setdefault(self.edu.destination, []).append(self.edu)
class DeviceRow(BaseFederationRow, namedtuple("DeviceRow", ("destination",))): # str
"""Streams the fact that either a) there is pending to device messages for
users on the remote, or b) a local users device has changed and needs to
be sent to the remote.
"""
TypeId = "d"
@staticmethod
def from_data(data):
return DeviceRow(destination=data["destination"])
def to_data(self):
return {"destination": self.destination}
def add_to_buffer(self, buff):
buff.device_destinations.add(self.destination)
TypeToRow = { TypeToRow = {
Row.TypeId: Row Row.TypeId: Row
for Row in (PresenceRow, PresenceDestinationsRow, KeyedEduRow, EduRow, DeviceRow) for Row in (PresenceRow, PresenceDestinationsRow, KeyedEduRow, EduRow,)
} }
@ -504,7 +467,6 @@ ParsedFederationStreamData = namedtuple(
"presence_destinations", # list of tuples of UserPresenceState and destinations "presence_destinations", # list of tuples of UserPresenceState and destinations
"keyed_edus", # dict of destination -> { key -> Edu } "keyed_edus", # dict of destination -> { key -> Edu }
"edus", # dict of destination -> [Edu] "edus", # dict of destination -> [Edu]
"device_destinations", # set of destinations
), ),
) )
@ -523,11 +485,7 @@ def process_rows_for_federation(transaction_queue, rows):
# them into the appropriate collection and then send them off. # them into the appropriate collection and then send them off.
buff = ParsedFederationStreamData( buff = ParsedFederationStreamData(
presence=[], presence=[], presence_destinations=[], keyed_edus={}, edus={},
presence_destinations=[],
keyed_edus={},
edus={},
device_destinations=set(),
) )
# Parse the rows in the stream and add to the buffer # Parse the rows in the stream and add to the buffer
@ -555,6 +513,3 @@ def process_rows_for_federation(transaction_queue, rows):
for destination, edu_list in iteritems(buff.edus): for destination, edu_list in iteritems(buff.edus):
for edu in edu_list: for edu in edu_list:
transaction_queue.send_edu(edu, None) transaction_queue.send_edu(edu, None)
for destination in buff.device_destinations:
transaction_queue.send_device_messages(destination)

View File

@ -15,6 +15,7 @@
# limitations under the License. # limitations under the License.
import logging import logging
from typing import Any, Dict
from six.moves import urllib from six.moves import urllib
@ -352,7 +353,9 @@ class TransportLayerClient(object):
else: else:
path = _create_v1_path("/publicRooms") path = _create_v1_path("/publicRooms")
args = {"include_all_networks": "true" if include_all_networks else "false"} args = {
"include_all_networks": "true" if include_all_networks else "false"
} # type: Dict[str, Any]
if third_party_instance_id: if third_party_instance_id:
args["third_party_instance_id"] = (third_party_instance_id,) args["third_party_instance_id"] = (third_party_instance_id,)
if limit: if limit:

View File

@ -18,6 +18,7 @@
import functools import functools
import logging import logging
import re import re
from typing import Optional, Tuple, Type
from twisted.internet.defer import maybeDeferred from twisted.internet.defer import maybeDeferred
@ -267,6 +268,8 @@ class BaseFederationServlet(object):
returned. returned.
""" """
PATH = "" # Overridden in subclasses, the regex to match against the path.
REQUIRE_AUTH = True REQUIRE_AUTH = True
PREFIX = FEDERATION_V1_PREFIX # Allows specifying the API version PREFIX = FEDERATION_V1_PREFIX # Allows specifying the API version
@ -347,9 +350,6 @@ class BaseFederationServlet(object):
return response return response
# Extra logic that functools.wraps() doesn't finish
new_func.__self__ = func.__self__
return new_func return new_func
def register(self, server): def register(self, server):
@ -824,7 +824,7 @@ class PublicRoomList(BaseFederationServlet):
if not self.allow_access: if not self.allow_access:
raise FederationDeniedError(origin) raise FederationDeniedError(origin)
limit = int(content.get("limit", 100)) limit = int(content.get("limit", 100)) # type: Optional[int]
since_token = content.get("since", None) since_token = content.get("since", None)
search_filter = content.get("filter", None) search_filter = content.get("filter", None)
@ -971,7 +971,7 @@ class FederationGroupsAddRoomsConfigServlet(BaseFederationServlet):
if get_domain_from_id(requester_user_id) != origin: if get_domain_from_id(requester_user_id) != origin:
raise SynapseError(403, "requester_user_id doesn't match origin") raise SynapseError(403, "requester_user_id doesn't match origin")
result = await self.groups_handler.update_room_in_group( result = await self.handler.update_room_in_group(
group_id, requester_user_id, room_id, config_key, content group_id, requester_user_id, room_id, config_key, content
) )
@ -1422,11 +1422,13 @@ FEDERATION_SERVLET_CLASSES = (
On3pidBindServlet, On3pidBindServlet,
FederationVersionServlet, FederationVersionServlet,
RoomComplexityServlet, RoomComplexityServlet,
) ) # type: Tuple[Type[BaseFederationServlet], ...]
OPENID_SERVLET_CLASSES = (OpenIdUserInfo,) OPENID_SERVLET_CLASSES = (
OpenIdUserInfo,
) # type: Tuple[Type[BaseFederationServlet], ...]
ROOM_LIST_CLASSES = (PublicRoomList,) ROOM_LIST_CLASSES = (PublicRoomList,) # type: Tuple[Type[PublicRoomList], ...]
GROUP_SERVER_SERVLET_CLASSES = ( GROUP_SERVER_SERVLET_CLASSES = (
FederationGroupsProfileServlet, FederationGroupsProfileServlet,
@ -1447,17 +1449,19 @@ GROUP_SERVER_SERVLET_CLASSES = (
FederationGroupsAddRoomsServlet, FederationGroupsAddRoomsServlet,
FederationGroupsAddRoomsConfigServlet, FederationGroupsAddRoomsConfigServlet,
FederationGroupsSettingJoinPolicyServlet, FederationGroupsSettingJoinPolicyServlet,
) ) # type: Tuple[Type[BaseFederationServlet], ...]
GROUP_LOCAL_SERVLET_CLASSES = ( GROUP_LOCAL_SERVLET_CLASSES = (
FederationGroupsLocalInviteServlet, FederationGroupsLocalInviteServlet,
FederationGroupsRemoveLocalUserServlet, FederationGroupsRemoveLocalUserServlet,
FederationGroupsBulkPublicisedServlet, FederationGroupsBulkPublicisedServlet,
) ) # type: Tuple[Type[BaseFederationServlet], ...]
GROUP_ATTESTATION_SERVLET_CLASSES = (FederationGroupsRenewAttestaionServlet,) GROUP_ATTESTATION_SERVLET_CLASSES = (
FederationGroupsRenewAttestaionServlet,
) # type: Tuple[Type[BaseFederationServlet], ...]
DEFAULT_SERVLET_GROUPS = ( DEFAULT_SERVLET_GROUPS = (
"federation", "federation",

Some files were not shown because too many files have changed in this diff Show More