Merge remote-tracking branch 'origin/release-v1.90' into matrix-org-hotfixes
Erik Johnston
commit
94d4507d35
|
|
@ -29,6 +29,16 @@ jobs:
|
|||
- name: Inspect builder
|
||||
run: docker buildx inspect
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v3
|
||||
|
||||
- name: Extract version from pyproject.toml
|
||||
# Note: explicitly requesting bash will mean bash is invoked with `-eo pipefail`, see
|
||||
# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell
|
||||
shell: bash
|
||||
run: |
|
||||
echo "SYNAPSE_VERSION=$(grep "^version" pyproject.toml | sed -E 's/version\s*=\s*["]([^"]*)["]/\1/')" >> $GITHUB_ENV
|
||||
|
||||
- name: Log in to DockerHub
|
||||
uses: docker/login-action@v2
|
||||
with:
|
||||
|
|
@ -61,7 +71,9 @@ jobs:
|
|||
uses: docker/build-push-action@v4
|
||||
with:
|
||||
push: true
|
||||
labels: "gitsha1=${{ github.sha }}"
|
||||
labels: |
|
||||
gitsha1=${{ github.sha }}
|
||||
org.opencontainers.image.version=${{ env.SYNAPSE_VERSION }}
|
||||
tags: "${{ steps.set-tag.outputs.tags }}"
|
||||
file: "docker/Dockerfile"
|
||||
platforms: linux/amd64,linux/arm64
|
||||
|
|
|
|||
84
CHANGES.md
84
CHANGES.md
|
|
@ -1,25 +1,97 @@
|
|||
# Synapse 1.90.0rc1 (2023-08-08)
|
||||
|
||||
### Features
|
||||
|
||||
- Scope transaction IDs to devices (implement [MSC3970](https://github.com/matrix-org/matrix-spec-proposals/pull/3970)). ([\#15629](https://github.com/matrix-org/synapse/issues/15629))
|
||||
- Remove old rows from the `cache_invalidation_stream_by_instance` table automatically (this table is unused in SQLite). ([\#15868](https://github.com/matrix-org/synapse/issues/15868))
|
||||
|
||||
### Bugfixes
|
||||
|
||||
- Fix a long-standing bug where purging history and paginating simultaneously could lead to database corruption when using workers. ([\#15791](https://github.com/matrix-org/synapse/issues/15791))
|
||||
- Fix a long-standing bug where profile endpoint returned a 404 when the user's display name was empty. ([\#16012](https://github.com/matrix-org/synapse/issues/16012))
|
||||
- Fix a long-standing bug where the `synapse_port_db` failed to configure sequences for application services and partial stated rooms. ([\#16043](https://github.com/matrix-org/synapse/issues/16043))
|
||||
- Fix long-standing bug with deletion in dehydrated devices v2. ([\#16046](https://github.com/matrix-org/synapse/issues/16046))
|
||||
|
||||
### Updates to the Docker image
|
||||
|
||||
- Add `org.opencontainers.image.version` labels to Docker containers [published by Matrix.org](https://hub.docker.com/r/matrixdotorg/synapse). Contributed by Mo Balaa. ([\#15972](https://github.com/matrix-org/synapse/issues/15972), [\#16009](https://github.com/matrix-org/synapse/issues/16009))
|
||||
|
||||
### Improved Documentation
|
||||
|
||||
- Add a internal documentation page describing the ["streams" used within Synapse](https://matrix-org.github.io/synapse/v1.90/development/synapse_architecture/streams.html). ([\#16015](https://github.com/matrix-org/synapse/issues/16015))
|
||||
- Clarify comment on the keys/upload over replication enpoint. ([\#16016](https://github.com/matrix-org/synapse/issues/16016))
|
||||
- Do not expose Admin API in caddy reverse proxy example. Contributed by @NilsIrl. ([\#16027](https://github.com/matrix-org/synapse/issues/16027))
|
||||
|
||||
### Deprecations and Removals
|
||||
|
||||
- Remove support for legacy application service paths. ([\#15964](https://github.com/matrix-org/synapse/issues/15964))
|
||||
- Move support for application service query parameter authorization behind a configuration option. ([\#16017](https://github.com/matrix-org/synapse/issues/16017))
|
||||
|
||||
### Internal Changes
|
||||
|
||||
- Update SQL queries to inline boolean parameters as supported in SQLite 3.27. ([\#15525](https://github.com/matrix-org/synapse/issues/15525))
|
||||
- Allow for the configuration of the backoff algorithm for federation destinations. ([\#15754](https://github.com/matrix-org/synapse/issues/15754))
|
||||
- Allow modules to check whether the current worker is configured to run background tasks. ([\#15991](https://github.com/matrix-org/synapse/issues/15991))
|
||||
- Update support for [MSC3958](https://github.com/matrix-org/matrix-spec-proposals/pull/3958) to match the latest revision of the MSC. ([\#15992](https://github.com/matrix-org/synapse/issues/15992))
|
||||
- Allow modules to schedule delayed background calls. ([\#15993](https://github.com/matrix-org/synapse/issues/15993))
|
||||
- Properly overwrite the `redacts` content-property for forwards-compatibility with room versions 1 through 10. ([\#16013](https://github.com/matrix-org/synapse/issues/16013))
|
||||
- Fix building the nix development environment on MacOS systems. ([\#16019](https://github.com/matrix-org/synapse/issues/16019))
|
||||
- Remove leading and trailing spaces when setting a display name. ([\#16031](https://github.com/matrix-org/synapse/issues/16031))
|
||||
- Combine duplicated code. ([\#16023](https://github.com/matrix-org/synapse/issues/16023))
|
||||
- Collect additional metrics from `ResponseCache` for eviction. ([\#16028](https://github.com/matrix-org/synapse/issues/16028))
|
||||
- Fix endpoint improperly declaring support for MSC3814. ([\#16068](https://github.com/matrix-org/synapse/issues/16068))
|
||||
- Drop backwards compat hack for event serialization. ([\#16069](https://github.com/matrix-org/synapse/issues/16069))
|
||||
|
||||
### Updates to locked dependencies
|
||||
|
||||
* Update PyYAML to 6.0.1. ([\#16011](https://github.com/matrix-org/synapse/issues/16011))
|
||||
* Bump cryptography from 41.0.2 to 41.0.3. ([\#16048](https://github.com/matrix-org/synapse/issues/16048))
|
||||
* Bump furo from 2023.5.20 to 2023.7.26. ([\#16077](https://github.com/matrix-org/synapse/issues/16077))
|
||||
* Bump immutabledict from 2.2.4 to 3.0.0. ([\#16034](https://github.com/matrix-org/synapse/issues/16034))
|
||||
* Update certifi to 2023.7.22 and pygments to 2.15.1. ([\#16044](https://github.com/matrix-org/synapse/issues/16044))
|
||||
* Bump jsonschema from 4.18.3 to 4.19.0. ([\#16081](https://github.com/matrix-org/synapse/issues/16081))
|
||||
* Bump phonenumbers from 8.13.14 to 8.13.18. ([\#16076](https://github.com/matrix-org/synapse/issues/16076))
|
||||
* Bump regex from 1.9.1 to 1.9.3. ([\#16073](https://github.com/matrix-org/synapse/issues/16073))
|
||||
* Bump serde from 1.0.171 to 1.0.175. ([\#15982](https://github.com/matrix-org/synapse/issues/15982))
|
||||
* Bump serde from 1.0.175 to 1.0.179. ([\#16033](https://github.com/matrix-org/synapse/issues/16033))
|
||||
* Bump serde from 1.0.179 to 1.0.183. ([\#16074](https://github.com/matrix-org/synapse/issues/16074))
|
||||
* Bump serde_json from 1.0.103 to 1.0.104. ([\#16032](https://github.com/matrix-org/synapse/issues/16032))
|
||||
* Bump service-identity from 21.1.0 to 23.1.0. ([\#16038](https://github.com/matrix-org/synapse/issues/16038))
|
||||
* Bump types-commonmark from 0.9.2.3 to 0.9.2.4. ([\#16037](https://github.com/matrix-org/synapse/issues/16037))
|
||||
* Bump types-jsonschema from 4.17.0.8 to 4.17.0.10. ([\#16036](https://github.com/matrix-org/synapse/issues/16036))
|
||||
* Bump types-netaddr from 0.8.0.8 to 0.8.0.9. ([\#16035](https://github.com/matrix-org/synapse/issues/16035))
|
||||
* Bump types-opentracing from 2.4.10.5 to 2.4.10.6. ([\#16078](https://github.com/matrix-org/synapse/issues/16078))
|
||||
* Bump types-setuptools from 68.0.0.0 to 68.0.0.3. ([\#16079](https://github.com/matrix-org/synapse/issues/16079))
|
||||
|
||||
# Synapse 1.89.0 (2023-08-01)
|
||||
|
||||
No significant changes since 1.89.0rc1.
|
||||
|
||||
|
||||
# Synapse 1.89.0rc1 (2023-07-25)
|
||||
|
||||
### Features
|
||||
|
||||
- Add Unix Socket support for HTTP Replication Listeners. Document and provide usage instructions for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\#15708](https://github.com/matrix-org/synapse/issues/15708), [\#15924](https://github.com/matrix-org/synapse/issues/15924))
|
||||
- Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\#15708](https://github.com/matrix-org/synapse/issues/15708), [\#15924](https://github.com/matrix-org/synapse/issues/15924))
|
||||
- Allow `+` in Matrix IDs, per [MSC4009](https://github.com/matrix-org/matrix-spec-proposals/pull/4009). ([\#15911](https://github.com/matrix-org/synapse/issues/15911))
|
||||
- Support room version 11 from [MSC3820](https://github.com/matrix-org/matrix-spec-proposals/pull/3820). ([\#15912](https://github.com/matrix-org/synapse/issues/15912))
|
||||
- Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\#15913](https://github.com/matrix-org/synapse/issues/15913), [\#15969](https://github.com/matrix-org/synapse/issues/15969))
|
||||
- Implement [MSC3814](https://github.com/matrix-org/matrix-spec-proposals/pull/3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](https://github.com/matrix-org/matrix-spec-proposals/pull/2697) behind a config flag. Contributed by Nico from Famedly and H-Shay. ([\#15929](https://github.com/matrix-org/synapse/issues/15929))
|
||||
- Implement [MSC3814](https://github.com/matrix-org/matrix-spec-proposals/pull/3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](https://github.com/matrix-org/matrix-spec-proposals/pull/2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\#15929](https://github.com/matrix-org/synapse/issues/15929))
|
||||
|
||||
### Bugfixes
|
||||
|
||||
- Fix long-standing bug where remote invites weren't correctly pushed. ([\#15820](https://github.com/matrix-org/synapse/issues/15820))
|
||||
- Fix a long-standing bug where remote invites weren't correctly pushed. ([\#15820](https://github.com/matrix-org/synapse/issues/15820))
|
||||
- Fix background schema updates failing over a large upgrade gap. ([\#15887](https://github.com/matrix-org/synapse/issues/15887))
|
||||
- Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\#15925](https://github.com/matrix-org/synapse/issues/15925))
|
||||
- Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\#15957](https://github.com/matrix-org/synapse/issues/15957))
|
||||
- Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\#15960](https://github.com/matrix-org/synapse/issues/15960))
|
||||
- Properly handle redactions of creation events. ([\#15973](https://github.com/matrix-org/synapse/issues/15973))
|
||||
- Fix bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\#15975](https://github.com/matrix-org/synapse/issues/15975))
|
||||
- Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\#15975](https://github.com/matrix-org/synapse/issues/15975))
|
||||
|
||||
### Improved Documentation
|
||||
|
||||
- Better clarify how to run a worker instance (pass both configs). ([\#15921](https://github.com/matrix-org/synapse/issues/15921))
|
||||
- Improve the documentation for the login as a user admin API. ([\#15938](https://github.com/matrix-org/synapse/issues/15938))
|
||||
- Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\#15938](https://github.com/matrix-org/synapse/issues/15938))
|
||||
- Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\#15981](https://github.com/matrix-org/synapse/issues/15981))
|
||||
|
||||
### Deprecations and Removals
|
||||
|
|
@ -29,7 +101,6 @@
|
|||
### Internal Changes
|
||||
|
||||
- Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\#15884](https://github.com/matrix-org/synapse/issues/15884))
|
||||
- Fix background schema updates failing over a large upgrade gap. ([\#15887](https://github.com/matrix-org/synapse/issues/15887))
|
||||
- Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\#15909](https://github.com/matrix-org/synapse/issues/15909))
|
||||
- Add details to warning in log when we fail to fetch an alias. ([\#15922](https://github.com/matrix-org/synapse/issues/15922))
|
||||
- Remove unneeded `__init__`. ([\#15926](https://github.com/matrix-org/synapse/issues/15926))
|
||||
|
|
@ -37,7 +108,6 @@
|
|||
- Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\#15940](https://github.com/matrix-org/synapse/issues/15940))
|
||||
- Update presence metrics to differentiate remote vs local users. ([\#15952](https://github.com/matrix-org/synapse/issues/15952))
|
||||
- Stop reading from column `user_id` of table `profiles`. ([\#15955](https://github.com/matrix-org/synapse/issues/15955))
|
||||
- Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\#15960](https://github.com/matrix-org/synapse/issues/15960))
|
||||
- Build packages for Debian Trixie. ([\#15961](https://github.com/matrix-org/synapse/issues/15961))
|
||||
- Reduce the amount of state we pull out. ([\#15968](https://github.com/matrix-org/synapse/issues/15968))
|
||||
- Speed up updating state in large rooms. ([\#15971](https://github.com/matrix-org/synapse/issues/15971))
|
||||
|
|
|
|||
|
|
@ -291,9 +291,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "regex"
|
||||
version = "1.9.1"
|
||||
version = "1.9.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b2eae68fc220f7cf2532e4494aded17545fce192d59cd996e0fe7887f4ceb575"
|
||||
checksum = "81bc1d4caf89fac26a70747fe603c130093b53c773888797a6329091246d651a"
|
||||
dependencies = [
|
||||
"aho-corasick",
|
||||
"memchr",
|
||||
|
|
@ -303,9 +303,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "regex-automata"
|
||||
version = "0.3.2"
|
||||
version = "0.3.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "83d3daa6976cffb758ec878f108ba0e062a45b2d6ca3a2cca965338855476caf"
|
||||
checksum = "fed1ceff11a1dddaee50c9dc8e4938bd106e9d89ae372f192311e7da498e3b69"
|
||||
dependencies = [
|
||||
"aho-corasick",
|
||||
"memchr",
|
||||
|
|
@ -314,9 +314,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "regex-syntax"
|
||||
version = "0.7.3"
|
||||
version = "0.7.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2ab07dc67230e4a4718e70fd5c20055a4334b121f1f9db8fe63ef39ce9b8c846"
|
||||
checksum = "e5ea92a5b6195c6ef2a0295ea818b312502c6fc94dde986c5553242e18fd4ce2"
|
||||
|
||||
[[package]]
|
||||
name = "ryu"
|
||||
|
|
@ -332,29 +332,29 @@ checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
|
|||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.171"
|
||||
version = "1.0.183"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "30e27d1e4fd7659406c492fd6cfaf2066ba8773de45ca75e855590f856dc34a9"
|
||||
checksum = "32ac8da02677876d532745a130fc9d8e6edfa81a269b107c5b00829b91d8eb3c"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.171"
|
||||
version = "1.0.183"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "389894603bd18c46fa56231694f8d827779c0951a667087194cf9de94ed24682"
|
||||
checksum = "aafe972d60b0b9bee71a91b92fee2d4fb3c9d7e8f6b179aa99f27203d99a4816"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.25",
|
||||
"syn 2.0.28",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.103"
|
||||
version = "1.0.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d03b412469450d4404fe8499a268edd7f8b79fecb074b0d812ad64ca21f4031b"
|
||||
checksum = "076066c5f1078eac5b722a31827a8832fe108bed65dfa75e233c89f8206e976c"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"ryu",
|
||||
|
|
@ -386,9 +386,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.25"
|
||||
version = "2.0.28"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "15e3fc8c0c74267e2df136e5e5fb656a464158aa57624053375eb9c8c6e25ae2"
|
||||
checksum = "04361975b3f5e348b2189d8dc55bc942f278b2d482a6a0365de5bdd62d351567"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
|
|
|
|||
|
|
@ -1,3 +1,15 @@
|
|||
matrix-synapse-py3 (1.90.0~rc1) stable; urgency=medium
|
||||
|
||||
* New Synapse release 1.90.0rc1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 08 Aug 2023 15:29:34 +0100
|
||||
|
||||
matrix-synapse-py3 (1.89.0) stable; urgency=medium
|
||||
|
||||
* New Synapse release 1.89.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 01 Aug 2023 11:07:15 +0100
|
||||
|
||||
matrix-synapse-py3 (1.89.0~rc1) stable; urgency=medium
|
||||
|
||||
* New Synapse release 1.89.0rc1.
|
||||
|
|
|
|||
|
|
@ -97,6 +97,7 @@
|
|||
- [Cancellation](development/synapse_architecture/cancellation.md)
|
||||
- [Log Contexts](log_contexts.md)
|
||||
- [Replication](replication.md)
|
||||
- [Streams](development/synapse_architecture/streams.md)
|
||||
- [TCP Replication](tcp_replication.md)
|
||||
- [Faster remote joins](development/synapse_architecture/faster_joins.md)
|
||||
- [Internal Documentation](development/internal_documentation/README.md)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,157 @@
|
|||
## Streams
|
||||
|
||||
Synapse has a concept of "streams", which are roughly described in [`id_generators.py`](
|
||||
https://github.com/matrix-org/synapse/blob/develop/synapse/storage/util/id_generators.py
|
||||
).
|
||||
Generally speaking, streams are a series of notifications that something in Synapse's database has changed that the application might need to respond to.
|
||||
For example:
|
||||
|
||||
- The events stream reports new events (PDUs) that Synapse creates, or that Synapse accepts from another homeserver.
|
||||
- The account data stream reports changes to users' [account data](https://spec.matrix.org/v1.7/client-server-api/#client-config).
|
||||
- The to-device stream reports when a device has a new [to-device message](https://spec.matrix.org/v1.7/client-server-api/#send-to-device-messaging).
|
||||
|
||||
See [`synapse.replication.tcp.streams`](
|
||||
https://github.com/matrix-org/synapse/blob/develop/synapse/replication/tcp/streams/__init__.py
|
||||
) for the full list of streams.
|
||||
|
||||
It is very helpful to understand the streams mechanism when working on any part of Synapse that needs to respond to changes—especially if those changes are made by different workers.
|
||||
To that end, let's describe streams formally, paraphrasing from the docstring of [`AbstractStreamIdGenerator`](
|
||||
https://github.com/matrix-org/synapse/blob/a719b703d9bd0dade2565ddcad0e2f3a7a9d4c37/synapse/storage/util/id_generators.py#L96
|
||||
).
|
||||
|
||||
### Definition
|
||||
|
||||
A stream is an append-only log `T1, T2, ..., Tn, ...` of facts[^1] which grows over time.
|
||||
Only "writers" can add facts to a stream, and there may be multiple writers.
|
||||
|
||||
Each fact has an ID, called its "stream ID".
|
||||
Readers should only process facts in ascending stream ID order.
|
||||
|
||||
Roughly speaking, each stream is backed by a database table.
|
||||
It should have a `stream_id` (or similar) bigint column holding stream IDs, plus additional columns as necessary to describe the fact.
|
||||
Typically, a fact is expressed with a single row in its backing table.[^2]
|
||||
Within a stream, no two facts may have the same stream_id.
|
||||
|
||||
> _Aside_. Some additional notes on streams' backing tables.
|
||||
>
|
||||
> 1. Rich would like to [ditch the backing tables](https://github.com/matrix-org/synapse/issues/13456).
|
||||
> 2. The backing tables may have other uses.
|
||||
> For example, the events table serves backs the events stream, and is read when processing new events.
|
||||
> But old rows are read from the table all the time, whenever Synapse needs to lookup some facts about an event.
|
||||
> 3. Rich suspects that sometimes the stream is backed by multiple tables, so the stream proper is the union of those tables.
|
||||
|
||||
Stream writers can "reserve" a stream ID, and then later mark it as having being completed.
|
||||
Stream writers need to track the completion of each stream fact.
|
||||
In the happy case, completion means a fact has been written to the stream table.
|
||||
But unhappy cases (e.g. transaction rollback due to an error) also count as completion.
|
||||
Once completed, the rows written with that stream ID are fixed, and no new rows
|
||||
will be inserted with that ID.
|
||||
|
||||
### Current stream ID
|
||||
|
||||
For any given stream reader (including writers themselves), we may define a per-writer current stream ID:
|
||||
|
||||
> The current stream ID _for a writer W_ is the largest stream ID such that
|
||||
> all transactions added by W with equal or smaller ID have completed.
|
||||
|
||||
Similarly, there is a "linear" notion of current stream ID:
|
||||
|
||||
> The "linear" current stream ID is the largest stream ID such that
|
||||
> all facts (added by any writer) with equal or smaller ID have completed.
|
||||
|
||||
Because different stream readers A and B learn about new facts at different times, A and B may disagree about current stream IDs.
|
||||
Put differently: we should think of stream readers as being independent of each other, proceeding through a stream of facts at different rates.
|
||||
|
||||
**NB.** For both senses of "current", that if a writer opens a transaction that never completes, the current stream ID will never advance beyond that writer's last written stream ID.
|
||||
|
||||
For single-writer streams, the per-writer current ID and the linear current ID are the same.
|
||||
Both senses of current ID are monotonic, but they may "skip" or jump over IDs because facts complete out of order.
|
||||
|
||||
|
||||
_Example_.
|
||||
Consider a single-writer stream which is initially at ID 1.
|
||||
|
||||
| Action | Current stream ID | Notes |
|
||||
|------------|-------------------|-------------------------------------------------|
|
||||
| | 1 | |
|
||||
| Reserve 2 | 1 | |
|
||||
| Reserve 3 | 1 | |
|
||||
| Complete 3 | 1 | current ID unchanged, waiting for 2 to complete |
|
||||
| Complete 2 | 3 | current ID jumps from 1 -> 3 |
|
||||
| Reserve 4 | 3 | |
|
||||
| Reserve 5 | 3 | |
|
||||
| Reserve 6 | 3 | |
|
||||
| Complete 5 | 3 | |
|
||||
| Complete 4 | 5 | current ID jumps 3->5, even though 6 is pending |
|
||||
| Complete 6 | 6 | |
|
||||
|
||||
|
||||
### Multi-writer streams
|
||||
|
||||
There are two ways to view a multi-writer stream.
|
||||
|
||||
1. Treat it as a collection of distinct single-writer streams, one
|
||||
for each writer.
|
||||
2. Treat it as a single stream.
|
||||
|
||||
The single stream (option 2) is conceptually simpler, and easier to represent (a single stream id).
|
||||
However, it requires each reader to know about the entire set of writers, to ensures that readers don't erroneously advance their current stream position too early and miss a fact from an unknown writer.
|
||||
In contrast, multiple parallel streams (option 1) are more complex, requiring more state to represent (map from writer to stream id).
|
||||
The payoff for doing so is that readers can "peek" ahead to facts that completed on one writer no matter the state of the others, reducing latency.
|
||||
|
||||
Note that a multi-writer stream can be viewed in both ways.
|
||||
For example, the events stream is treated as multiple single-writer streams (option 1) by the sync handler, so that events are sent to clients as soon as possible.
|
||||
But the background process that works through events treats them as a single linear stream.
|
||||
|
||||
Another useful example is the cache invalidation stream.
|
||||
The facts this stream holds are instructions to "you should now invalidate these cache entries".
|
||||
We only ever treat this as a multiple single-writer streams as there is no important ordering between cache invalidations.
|
||||
(Invalidations are self-contained facts; and the invalidations commute/are idempotent).
|
||||
|
||||
### Writing to streams
|
||||
|
||||
Writers need to track:
|
||||
- track their current position (i.e. its own per-writer stream ID).
|
||||
- their facts currently awaiting completion.
|
||||
|
||||
At startup,
|
||||
- the current position of that writer can be found by querying the database (which suggests that facts need to be written to the database atomically, in a transaction); and
|
||||
- there are no facts awaiting completion.
|
||||
|
||||
To reserve a stream ID, call [`nextval`](https://www.postgresql.org/docs/current/functions-sequence.html) on the appropriate postgres sequence.
|
||||
|
||||
To write a fact to the stream: insert the appropriate rows to the appropriate backing table.
|
||||
|
||||
To complete a fact, first remove it from your map of facts currently awaiting completion.
|
||||
Then, if no earlier fact is awaiting completion, the writer can advance its current position in that stream.
|
||||
Upon doing so it should emit an `RDATA` message[^3], once for every fact between the old and the new stream ID.
|
||||
|
||||
### Subscribing to streams
|
||||
|
||||
Readers need to track the current position of every writer.
|
||||
|
||||
At startup, they can find this by contacting each writer with a `REPLICATE` message,
|
||||
requesting that all writers reply describing their current position in their streams.
|
||||
Writers reply with a `POSITION` message.
|
||||
|
||||
To learn about new facts, readers should listen for `RDATA` messages and process them to respond to the new fact.
|
||||
The `RDATA` itself is not a self-contained representation of the fact;
|
||||
readers will have to query the stream tables for the full details.
|
||||
Readers must also advance their record of the writer's current position for that stream.
|
||||
|
||||
# Summary
|
||||
|
||||
In a nutshell: we have an append-only log with a "buffer/scratchpad" at the end where we have to wait for the sequence to be linear and contiguous.
|
||||
|
||||
|
||||
---
|
||||
|
||||
[^1]: we use the word _fact_ here for two reasons.
|
||||
Firstly, the word "event" is already heavily overloaded (PDUs, EDUs, account data, ...) and we don't need to make that worse.
|
||||
Secondly, "fact" emphasises that the things we append to a stream cannot change after the fact.
|
||||
|
||||
[^2]: A fact might be expressed with 0 rows, e.g. if we opened a transaction to persist an event, but failed and rolled the transaction back before marking the fact as completed.
|
||||
In principle a fact might be expressed with 2 or more rows; if so, each of those rows should share the fact's stream ID.
|
||||
|
||||
[^3]: This communication used to happen directly with the writers [over TCP](../../tcp_replication.md);
|
||||
nowadays it's done via Redis's Pubsub.
|
||||
|
|
@ -95,7 +95,7 @@ matrix.example.com {
|
|||
}
|
||||
|
||||
example.com:8448 {
|
||||
reverse_proxy localhost:8008
|
||||
reverse_proxy /_matrix/* localhost:8008
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -88,6 +88,21 @@ process, for example:
|
|||
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
```
|
||||
|
||||
# Upgrading to v1.90.0
|
||||
|
||||
## App service query parameter authorization is now a configuration option
|
||||
|
||||
Synapse v1.81.0 deprecated application service authorization via query parameters as this is
|
||||
considered insecure - and from Synapse v1.71.0 forwards the application service token has also been sent via
|
||||
[the `Authorization` header](https://spec.matrix.org/v1.6/application-service-api/#authorization)], making the insecure
|
||||
query parameter authorization redundant. Since removing the ability to continue to use query parameters could break
|
||||
backwards compatibility it has now been put behind a configuration option, `use_appservice_legacy_authorization`.
|
||||
This option defaults to false, but can be activated by adding
|
||||
```yaml
|
||||
use_appservice_legacy_authorization: true
|
||||
```
|
||||
to your configuration.
|
||||
|
||||
# Upgrading to v1.89.0
|
||||
|
||||
## Removal of unspecced `user` property for `/register`
|
||||
|
|
@ -97,7 +112,6 @@ The standard `username` property should be used instead. See the
|
|||
[Application Service specification](https://spec.matrix.org/v1.7/application-service-api/#server-admin-style-permissions)
|
||||
for more information.
|
||||
|
||||
|
||||
# Upgrading to v1.88.0
|
||||
|
||||
## Minimum supported Python version
|
||||
|
|
|
|||
|
|
@ -1242,6 +1242,14 @@ like sending a federation transaction.
|
|||
* `max_short_retries`: maximum number of retries for the short retry algo. Default to 3 attempts.
|
||||
* `max_long_retries`: maximum number of retries for the long retry algo. Default to 10 attempts.
|
||||
|
||||
The following options control the retry logic when communicating with a specific homeserver destination.
|
||||
Unlike the previous configuration options, these values apply across all requests
|
||||
for a given destination and the state of the backoff is stored in the database.
|
||||
|
||||
* `destination_min_retry_interval`: the initial backoff, after the first request fails. Defaults to 10m.
|
||||
* `destination_retry_multiplier`: how much we multiply the backoff by after each subsequent fail. Defaults to 2.
|
||||
* `destination_max_retry_interval`: a cap on the backoff. Defaults to a week.
|
||||
|
||||
Example configuration:
|
||||
```yaml
|
||||
federation:
|
||||
|
|
@ -1250,6 +1258,9 @@ federation:
|
|||
max_long_retry_delay: 100s
|
||||
max_short_retries: 5
|
||||
max_long_retries: 20
|
||||
destination_min_retry_interval: 30s
|
||||
destination_retry_multiplier: 5
|
||||
destination_max_retry_interval: 12h
|
||||
```
|
||||
---
|
||||
## Caching
|
||||
|
|
@ -2837,6 +2848,20 @@ Example configuration:
|
|||
```yaml
|
||||
track_appservice_user_ips: true
|
||||
```
|
||||
---
|
||||
### `use_appservice_legacy_authorization`
|
||||
|
||||
Whether to send the application service access tokens via the `access_token` query parameter
|
||||
per older versions of the Matrix specification. Defaults to false. Set to true to enable sending
|
||||
access tokens via a query parameter.
|
||||
|
||||
**Enabling this option is considered insecure and is not recommended. **
|
||||
|
||||
Example configuration:
|
||||
```yaml
|
||||
use_appservice_legacy_authorization: true
|
||||
```
|
||||
|
||||
---
|
||||
### `macaroon_secret_key`
|
||||
|
||||
|
|
|
|||
60
flake.lock
60
flake.lock
|
|
@ -8,11 +8,11 @@
|
|||
"pre-commit-hooks": "pre-commit-hooks"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1683102061,
|
||||
"narHash": "sha256-kOphT6V0uQUlFNBP3GBjs7DAU7fyZGGqCs9ue1gNY6E=",
|
||||
"lastModified": 1690534632,
|
||||
"narHash": "sha256-kOXS9x5y17VKliC7wZxyszAYrWdRl1JzggbQl0gyo94=",
|
||||
"owner": "cachix",
|
||||
"repo": "devenv",
|
||||
"rev": "ff1f29e41756553174d596cafe3a9fa77595100b",
|
||||
"rev": "6568e7e485a46bbf32051e4d6347fa1fed8b2f25",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
@ -39,12 +39,15 @@
|
|||
}
|
||||
},
|
||||
"flake-utils": {
|
||||
"inputs": {
|
||||
"systems": "systems"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1667395993,
|
||||
"narHash": "sha256-nuEHfE/LcWyuSWnS8t12N1wc105Qtau+/OdUAjtQ0rA=",
|
||||
"lastModified": 1685518550,
|
||||
"narHash": "sha256-o2d0KcvaXzTrPRIo0kOLV0/QXHhDQ5DTi+OxcjO8xqY=",
|
||||
"owner": "numtide",
|
||||
"repo": "flake-utils",
|
||||
"rev": "5aed5285a952e0b949eb3ba02c12fa4fcfef535f",
|
||||
"rev": "a1720a10a6cfe8234c0e93907ffe81be440f4cef",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
@ -55,7 +58,7 @@
|
|||
},
|
||||
"flake-utils_2": {
|
||||
"inputs": {
|
||||
"systems": "systems"
|
||||
"systems": "systems_2"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1681202837,
|
||||
|
|
@ -167,27 +170,27 @@
|
|||
},
|
||||
"nixpkgs-stable": {
|
||||
"locked": {
|
||||
"lastModified": 1673800717,
|
||||
"narHash": "sha256-SFHraUqLSu5cC6IxTprex/nTsI81ZQAtDvlBvGDWfnA=",
|
||||
"lastModified": 1685801374,
|
||||
"narHash": "sha256-otaSUoFEMM+LjBI1XL/xGB5ao6IwnZOXc47qhIgJe8U=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "2f9fd351ec37f5d479556cd48be4ca340da59b8f",
|
||||
"rev": "c37ca420157f4abc31e26f436c1145f8951ff373",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixos-22.11",
|
||||
"ref": "nixos-23.05",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nixpkgs_2": {
|
||||
"locked": {
|
||||
"lastModified": 1682519441,
|
||||
"narHash": "sha256-Vsq/8NOtvW1AoC6shCBxRxZyMQ+LhvPuJT6ltbzuv+Y=",
|
||||
"lastModified": 1690535733,
|
||||
"narHash": "sha256-WgjUPscQOw3cB8yySDGlyzo6cZNihnRzUwE9kadv/5I=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "7a32a141db568abde9bc389845949dc2a454dfd3",
|
||||
"rev": "8cacc05fbfffeaab910e8c2c9e2a7c6b32ce881a",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
@ -228,11 +231,11 @@
|
|||
"nixpkgs-stable": "nixpkgs-stable"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1678376203,
|
||||
"narHash": "sha256-3tyYGyC8h7fBwncLZy5nCUjTJPrHbmNwp47LlNLOHSM=",
|
||||
"lastModified": 1688056373,
|
||||
"narHash": "sha256-2+SDlNRTKsgo3LBRiMUcoEUb6sDViRNQhzJquZ4koOI=",
|
||||
"owner": "cachix",
|
||||
"repo": "pre-commit-hooks.nix",
|
||||
"rev": "1a20b9708962096ec2481eeb2ddca29ed747770a",
|
||||
"rev": "5843cf069272d92b60c3ed9e55b7a8989c01d4c7",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
@ -246,7 +249,7 @@
|
|||
"devenv": "devenv",
|
||||
"nixpkgs": "nixpkgs_2",
|
||||
"rust-overlay": "rust-overlay",
|
||||
"systems": "systems_2"
|
||||
"systems": "systems_3"
|
||||
}
|
||||
},
|
||||
"rust-overlay": {
|
||||
|
|
@ -255,11 +258,11 @@
|
|||
"nixpkgs": "nixpkgs_3"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1689302058,
|
||||
"narHash": "sha256-yD74lcHTrw4niXcE9goJLbzsgyce48rQQoy5jK5ZK40=",
|
||||
"lastModified": 1690510705,
|
||||
"narHash": "sha256-6mjs3Gl9/xrseFh9iNcNq1u5yJ/MIoAmjoaG7SXZDIE=",
|
||||
"owner": "oxalica",
|
||||
"repo": "rust-overlay",
|
||||
"rev": "7b8dbbf4c67ed05a9bf3d9e658c12d4108bc24c8",
|
||||
"rev": "851ae4c128905a62834d53ce7704ebc1ba481bea",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
@ -297,6 +300,21 @@
|
|||
"repo": "default",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"systems_3": {
|
||||
"locked": {
|
||||
"lastModified": 1681028828,
|
||||
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"type": "github"
|
||||
}
|
||||
}
|
||||
},
|
||||
"root": "root",
|
||||
|
|
|
|||
|
|
@ -39,8 +39,8 @@
|
|||
|
||||
{
|
||||
inputs = {
|
||||
# Use the master/unstable branch of nixpkgs. The latest stable, 22.11,
|
||||
# does not contain 'perl536Packages.NetAsyncHTTP', needed by Sytest.
|
||||
# Use the master/unstable branch of nixpkgs. Used to fetch the latest
|
||||
# available versions of packages.
|
||||
nixpkgs.url = "github:NixOS/nixpkgs/master";
|
||||
# Output a development shell for x86_64/aarch64 Linux/Darwin (MacOS).
|
||||
systems.url = "github:nix-systems/default";
|
||||
|
|
|
|||
|
|
@ -226,13 +226,13 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "certifi"
|
||||
version = "2022.12.7"
|
||||
version = "2023.7.22"
|
||||
description = "Python package for providing Mozilla's CA Bundle."
|
||||
optional = false
|
||||
python-versions = ">=3.6"
|
||||
files = [
|
||||
{file = "certifi-2022.12.7-py3-none-any.whl", hash = "sha256:4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18"},
|
||||
{file = "certifi-2022.12.7.tar.gz", hash = "sha256:35824b4c3a97115964b408844d64aa14db1cc518f6562e8d7261699d1350a9e3"},
|
||||
{file = "certifi-2023.7.22-py3-none-any.whl", hash = "sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9"},
|
||||
{file = "certifi-2023.7.22.tar.gz", hash = "sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -460,34 +460,34 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "cryptography"
|
||||
version = "41.0.2"
|
||||
version = "41.0.3"
|
||||
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
|
||||
optional = false
|
||||
python-versions = ">=3.7"
|
||||
files = [
|
||||
{file = "cryptography-41.0.2-cp37-abi3-macosx_10_12_universal2.whl", hash = "sha256:01f1d9e537f9a15b037d5d9ee442b8c22e3ae11ce65ea1f3316a41c78756b711"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:079347de771f9282fbfe0e0236c716686950c19dee1b76240ab09ce1624d76d7"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:439c3cc4c0d42fa999b83ded80a9a1fb54d53c58d6e59234cfe97f241e6c781d"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f14ad275364c8b4e525d018f6716537ae7b6d369c094805cae45300847e0894f"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:84609ade00a6ec59a89729e87a503c6e36af98ddcd566d5f3be52e29ba993182"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:49c3222bb8f8e800aead2e376cbef687bc9e3cb9b58b29a261210456a7783d83"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:d73f419a56d74fef257955f51b18d046f3506270a5fd2ac5febbfa259d6c0fa5"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:2a034bf7d9ca894720f2ec1d8b7b5832d7e363571828037f9e0c4f18c1b58a58"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-win32.whl", hash = "sha256:d124682c7a23c9764e54ca9ab5b308b14b18eba02722b8659fb238546de83a76"},
|
||||
{file = "cryptography-41.0.2-cp37-abi3-win_amd64.whl", hash = "sha256:9c3fe6534d59d071ee82081ca3d71eed3210f76ebd0361798c74abc2bcf347d4"},
|
||||
{file = "cryptography-41.0.2-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:a719399b99377b218dac6cf547b6ec54e6ef20207b6165126a280b0ce97e0d2a"},
|
||||
{file = "cryptography-41.0.2-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:182be4171f9332b6741ee818ec27daff9fb00349f706629f5cbf417bd50e66fd"},
|
||||
{file = "cryptography-41.0.2-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:7a9a3bced53b7f09da251685224d6a260c3cb291768f54954e28f03ef14e3766"},
|
||||
{file = "cryptography-41.0.2-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:f0dc40e6f7aa37af01aba07277d3d64d5a03dc66d682097541ec4da03cc140ee"},
|
||||
{file = "cryptography-41.0.2-pp38-pypy38_pp73-macosx_10_12_x86_64.whl", hash = "sha256:674b669d5daa64206c38e507808aae49904c988fa0a71c935e7006a3e1e83831"},
|
||||
{file = "cryptography-41.0.2-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:7af244b012711a26196450d34f483357e42aeddb04128885d95a69bd8b14b69b"},
|
||||
{file = "cryptography-41.0.2-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:9b6d717393dbae53d4e52684ef4f022444fc1cce3c48c38cb74fca29e1f08eaa"},
|
||||
{file = "cryptography-41.0.2-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:192255f539d7a89f2102d07d7375b1e0a81f7478925b3bc2e0549ebf739dae0e"},
|
||||
{file = "cryptography-41.0.2-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:f772610fe364372de33d76edcd313636a25684edb94cee53fd790195f5989d14"},
|
||||
{file = "cryptography-41.0.2-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:b332cba64d99a70c1e0836902720887fb4529ea49ea7f5462cf6640e095e11d2"},
|
||||
{file = "cryptography-41.0.2-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:9a6673c1828db6270b76b22cc696f40cde9043eb90373da5c2f8f2158957f42f"},
|
||||
{file = "cryptography-41.0.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:342f3767e25876751e14f8459ad85e77e660537ca0a066e10e75df9c9e9099f0"},
|
||||
{file = "cryptography-41.0.2.tar.gz", hash = "sha256:7d230bf856164de164ecb615ccc14c7fc6de6906ddd5b491f3af90d3514c925c"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-macosx_10_12_universal2.whl", hash = "sha256:652627a055cb52a84f8c448185922241dd5217443ca194d5739b44612c5e6507"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:8f09daa483aedea50d249ef98ed500569841d6498aa9c9f4b0531b9964658922"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4fd871184321100fb400d759ad0cddddf284c4b696568204d281c902fc7b0d81"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:84537453d57f55a50a5b6835622ee405816999a7113267739a1b4581f83535bd"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:3fb248989b6363906827284cd20cca63bb1a757e0a2864d4c1682a985e3dca47"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:42cb413e01a5d36da9929baa9d70ca90d90b969269e5a12d39c1e0d475010116"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:aeb57c421b34af8f9fe830e1955bf493a86a7996cc1338fe41b30047d16e962c"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:6af1c6387c531cd364b72c28daa29232162010d952ceb7e5ca8e2827526aceae"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-win32.whl", hash = "sha256:0d09fb5356f975974dbcb595ad2d178305e5050656affb7890a1583f5e02a306"},
|
||||
{file = "cryptography-41.0.3-cp37-abi3-win_amd64.whl", hash = "sha256:a983e441a00a9d57a4d7c91b3116a37ae602907a7618b882c8013b5762e80574"},
|
||||
{file = "cryptography-41.0.3-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:5259cb659aa43005eb55a0e4ff2c825ca111a0da1814202c64d28a985d33b087"},
|
||||
{file = "cryptography-41.0.3-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:67e120e9a577c64fe1f611e53b30b3e69744e5910ff3b6e97e935aeb96005858"},
|
||||
{file = "cryptography-41.0.3-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:7efe8041897fe7a50863e51b77789b657a133c75c3b094e51b5e4b5cec7bf906"},
|
||||
{file = "cryptography-41.0.3-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:ce785cf81a7bdade534297ef9e490ddff800d956625020ab2ec2780a556c313e"},
|
||||
{file = "cryptography-41.0.3-pp38-pypy38_pp73-macosx_10_12_x86_64.whl", hash = "sha256:57a51b89f954f216a81c9d057bf1a24e2f36e764a1ca9a501a6964eb4a6800dd"},
|
||||
{file = "cryptography-41.0.3-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:4c2f0d35703d61002a2bbdcf15548ebb701cfdd83cdc12471d2bae80878a4207"},
|
||||
{file = "cryptography-41.0.3-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:23c2d778cf829f7d0ae180600b17e9fceea3c2ef8b31a99e3c694cbbf3a24b84"},
|
||||
{file = "cryptography-41.0.3-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:95dd7f261bb76948b52a5330ba5202b91a26fbac13ad0e9fc8a3ac04752058c7"},
|
||||
{file = "cryptography-41.0.3-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:41d7aa7cdfded09b3d73a47f429c298e80796c8e825ddfadc84c8a7f12df212d"},
|
||||
{file = "cryptography-41.0.3-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:d0d651aa754ef58d75cec6edfbd21259d93810b73f6ec246436a21b7841908de"},
|
||||
{file = "cryptography-41.0.3-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:ab8de0d091acbf778f74286f4989cf3d1528336af1b59f3e5d2ebca8b5fe49e1"},
|
||||
{file = "cryptography-41.0.3-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:a74fbcdb2a0d46fe00504f571a2a540532f4c188e6ccf26f1f178480117b33c4"},
|
||||
{file = "cryptography-41.0.3.tar.gz", hash = "sha256:6d192741113ef5e30d89dcb5b956ef4e1578f304708701b8b73d38e3e1461f34"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
|
|
@ -558,13 +558,13 @@ dev = ["Sphinx", "coverage", "flake8", "lxml", "lxml-stubs", "memory-profiler",
|
|||
|
||||
[[package]]
|
||||
name = "furo"
|
||||
version = "2023.5.20"
|
||||
version = "2023.7.26"
|
||||
description = "A clean customisable Sphinx documentation theme."
|
||||
optional = false
|
||||
python-versions = ">=3.7"
|
||||
files = [
|
||||
{file = "furo-2023.5.20-py3-none-any.whl", hash = "sha256:594a8436ddfe0c071f3a9e9a209c314a219d8341f3f1af33fdf7c69544fab9e6"},
|
||||
{file = "furo-2023.5.20.tar.gz", hash = "sha256:40e09fa17c6f4b22419d122e933089226dcdb59747b5b6c79363089827dea16f"},
|
||||
{file = "furo-2023.7.26-py3-none-any.whl", hash = "sha256:1c7936929ec57c5ddecc7c85f07fa8b2ce536b5c89137764cca508be90e11efd"},
|
||||
{file = "furo-2023.7.26.tar.gz", hash = "sha256:257f63bab97aa85213a1fa24303837a3c3f30be92901ec732fea74290800f59e"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
|
|
@ -824,13 +824,13 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "immutabledict"
|
||||
version = "2.2.4"
|
||||
version = "3.0.0"
|
||||
description = "Immutable wrapper around dictionaries (a fork of frozendict)"
|
||||
optional = false
|
||||
python-versions = ">=3.7,<4.0"
|
||||
python-versions = ">=3.8,<4.0"
|
||||
files = [
|
||||
{file = "immutabledict-2.2.4-py3-none-any.whl", hash = "sha256:c827715c147d2364522f9a7709cc424c7001015274a3c705250e673605bde64b"},
|
||||
{file = "immutabledict-2.2.4.tar.gz", hash = "sha256:3bedc0741faaa2846f6edf5c29183f993da3abaff6a5961bb70a5659bb9e68ab"},
|
||||
{file = "immutabledict-3.0.0-py3-none-any.whl", hash = "sha256:034bacc6c6872707c4ec0ea9515de6bbe0dcf0fcabd97ae19fd4e4c338f05798"},
|
||||
{file = "immutabledict-3.0.0.tar.gz", hash = "sha256:5a23cd369a6187f76a8c29d7d687980b092538eb9800e58964603f1b973c56fe"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -973,13 +973,13 @@ i18n = ["Babel (>=2.7)"]
|
|||
|
||||
[[package]]
|
||||
name = "jsonschema"
|
||||
version = "4.18.3"
|
||||
version = "4.19.0"
|
||||
description = "An implementation of JSON Schema validation for Python"
|
||||
optional = false
|
||||
python-versions = ">=3.8"
|
||||
files = [
|
||||
{file = "jsonschema-4.18.3-py3-none-any.whl", hash = "sha256:aab78b34c2de001c6b692232f08c21a97b436fe18e0b817bf0511046924fceef"},
|
||||
{file = "jsonschema-4.18.3.tar.gz", hash = "sha256:64b7104d72efe856bea49ca4af37a14a9eba31b40bb7238179f3803130fd34d9"},
|
||||
{file = "jsonschema-4.19.0-py3-none-any.whl", hash = "sha256:043dc26a3845ff09d20e4420d6012a9c91c9aa8999fa184e7efcfeccb41e32cb"},
|
||||
{file = "jsonschema-4.19.0.tar.gz", hash = "sha256:6e1e7569ac13be8139b2dd2c21a55d350066ee3f80df06c608b398cdc6f30e8f"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
|
|
@ -1610,13 +1610,13 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "phonenumbers"
|
||||
version = "8.13.14"
|
||||
version = "8.13.18"
|
||||
description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
files = [
|
||||
{file = "phonenumbers-8.13.14-py2.py3-none-any.whl", hash = "sha256:a4b20b6ba7dd402728f5cc8e86e1f29b1a873af45f5381dbee7e3083af497ff6"},
|
||||
{file = "phonenumbers-8.13.14.tar.gz", hash = "sha256:5fa952b4abf9fccdaf1f130d96114a520c48890d4091b50a064e22c0fdc12dec"},
|
||||
{file = "phonenumbers-8.13.18-py2.py3-none-any.whl", hash = "sha256:3d802739a22592e4127139349937753dee9b6a20bdd5d56847cd885bdc766b1f"},
|
||||
{file = "phonenumbers-8.13.18.tar.gz", hash = "sha256:b360c756252805d44b447b5bca6d250cf6bd6c69b6f0f4258f3bfe5ab81bef69"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -1898,13 +1898,13 @@ requests = ">=2.14.0"
|
|||
|
||||
[[package]]
|
||||
name = "pygments"
|
||||
version = "2.14.0"
|
||||
version = "2.15.1"
|
||||
description = "Pygments is a syntax highlighting package written in Python."
|
||||
optional = false
|
||||
python-versions = ">=3.6"
|
||||
python-versions = ">=3.7"
|
||||
files = [
|
||||
{file = "Pygments-2.14.0-py3-none-any.whl", hash = "sha256:fa7bd7bd2771287c0de303af8bfdfc731f51bd2c6a47ab69d117138893b82717"},
|
||||
{file = "Pygments-2.14.0.tar.gz", hash = "sha256:b3ed06a9e8ac9a9aae5a6f5dbe78a8a58655d17b43b93c078f094ddc476ae297"},
|
||||
{file = "Pygments-2.15.1-py3-none-any.whl", hash = "sha256:db2db3deb4b4179f399a09054b023b6a586b76499d36965813c71aa8ed7b5fd1"},
|
||||
{file = "Pygments-2.15.1.tar.gz", hash = "sha256:8ace4d3c1dd481894b2005f560ead0f9f19ee64fe983366be1a21e171d12775c"},
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
|
|
@ -2072,51 +2072,51 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "pyyaml"
|
||||
version = "6.0"
|
||||
version = "6.0.1"
|
||||
description = "YAML parser and emitter for Python"
|
||||
optional = false
|
||||
python-versions = ">=3.6"
|
||||
files = [
|
||||
{file = "PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53"},
|
||||
{file = "PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c"},
|
||||
{file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc"},
|
||||
{file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b"},
|
||||
{file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"},
|
||||
{file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"},
|
||||
{file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-win32.whl", hash = "sha256:bfaef573a63ba8923503d27530362590ff4f576c626d86a9fed95822a8255fd7"},
|
||||
{file = "PyYAML-6.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf"},
|
||||
{file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"},
|
||||
{file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"},
|
||||
{file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"},
|
||||
{file = "PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4"},
|
||||
{file = "PyYAML-6.0-cp36-cp36m-win32.whl", hash = "sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293"},
|
||||
{file = "PyYAML-6.0-cp36-cp36m-win_amd64.whl", hash = "sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57"},
|
||||
{file = "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c"},
|
||||
{file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0"},
|
||||
{file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4"},
|
||||
{file = "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9"},
|
||||
{file = "PyYAML-6.0-cp37-cp37m-win32.whl", hash = "sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737"},
|
||||
{file = "PyYAML-6.0-cp37-cp37m-win_amd64.whl", hash = "sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d"},
|
||||
{file = "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b"},
|
||||
{file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba"},
|
||||
{file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34"},
|
||||
{file = "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287"},
|
||||
{file = "PyYAML-6.0-cp38-cp38-win32.whl", hash = "sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78"},
|
||||
{file = "PyYAML-6.0-cp38-cp38-win_amd64.whl", hash = "sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-win32.whl", hash = "sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb"},
|
||||
{file = "PyYAML-6.0-cp39-cp39-win_amd64.whl", hash = "sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c"},
|
||||
{file = "PyYAML-6.0.tar.gz", hash = "sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-win32.whl", hash = "sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924"},
|
||||
{file = "PyYAML-6.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f003ed9ad21d6a4713f0a9b5a7a0a79e08dd0f221aff4525a2be4c346ee60aab"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-win32.whl", hash = "sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741"},
|
||||
{file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
|
||||
{file = "PyYAML-6.0.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47"},
|
||||
{file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98"},
|
||||
{file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c"},
|
||||
{file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:afd7e57eddb1a54f0f1a974bc4391af8bcce0b444685d936840f125cf046d5bd"},
|
||||
{file = "PyYAML-6.0.1-cp36-cp36m-win32.whl", hash = "sha256:fca0e3a251908a499833aa292323f32437106001d436eca0e6e7833256674585"},
|
||||
{file = "PyYAML-6.0.1-cp36-cp36m-win_amd64.whl", hash = "sha256:f22ac1c3cac4dbc50079e965eba2c1058622631e526bd9afd45fedd49ba781fa"},
|
||||
{file = "PyYAML-6.0.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:b1275ad35a5d18c62a7220633c913e1b42d44b46ee12554e5fd39c70a243d6a3"},
|
||||
{file = "PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:18aeb1bf9a78867dc38b259769503436b7c72f7a1f1f4c93ff9a17de54319b27"},
|
||||
{file = "PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:596106435fa6ad000c2991a98fa58eeb8656ef2325d7e158344fb33864ed87e3"},
|
||||
{file = "PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:baa90d3f661d43131ca170712d903e6295d1f7a0f595074f151c0aed377c9b9c"},
|
||||
{file = "PyYAML-6.0.1-cp37-cp37m-win32.whl", hash = "sha256:9046c58c4395dff28dd494285c82ba00b546adfc7ef001486fbf0324bc174fba"},
|
||||
{file = "PyYAML-6.0.1-cp37-cp37m-win_amd64.whl", hash = "sha256:4fb147e7a67ef577a588a0e2c17b6db51dda102c71de36f8549b6816a96e1867"},
|
||||
{file = "PyYAML-6.0.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:1d4c7e777c441b20e32f52bd377e0c409713e8bb1386e1099c2415f26e479595"},
|
||||
{file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5"},
|
||||
{file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696"},
|
||||
{file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735"},
|
||||
{file = "PyYAML-6.0.1-cp38-cp38-win32.whl", hash = "sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206"},
|
||||
{file = "PyYAML-6.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:c8098ddcc2a85b61647b2590f825f3db38891662cfc2fc776415143f599bb859"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-win32.whl", hash = "sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c"},
|
||||
{file = "PyYAML-6.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486"},
|
||||
{file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -2427,13 +2427,13 @@ tornado = ["tornado (>=5)"]
|
|||
|
||||
[[package]]
|
||||
name = "service-identity"
|
||||
version = "21.1.0"
|
||||
version = "23.1.0"
|
||||
description = "Service identity verification for pyOpenSSL & cryptography."
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
python-versions = ">=3.8"
|
||||
files = [
|
||||
{file = "service-identity-21.1.0.tar.gz", hash = "sha256:6e6c6086ca271dc11b033d17c3a8bea9f24ebff920c587da090afc9519419d34"},
|
||||
{file = "service_identity-21.1.0-py2.py3-none-any.whl", hash = "sha256:f0b0caac3d40627c3c04d7a51b6e06721857a0e10a8775f2d1d7e72901b3a7db"},
|
||||
{file = "service_identity-23.1.0-py3-none-any.whl", hash = "sha256:87415a691d52fcad954a500cb81f424d0273f8e7e3ee7d766128f4575080f383"},
|
||||
{file = "service_identity-23.1.0.tar.gz", hash = "sha256:ecb33cd96307755041e978ab14f8b14e13b40f1fbd525a4dc78f46d2b986431d"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
|
|
@ -2441,12 +2441,12 @@ attrs = ">=19.1.0"
|
|||
cryptography = "*"
|
||||
pyasn1 = "*"
|
||||
pyasn1-modules = "*"
|
||||
six = "*"
|
||||
|
||||
[package.extras]
|
||||
dev = ["coverage[toml] (>=5.0.2)", "furo", "idna", "pyOpenSSL", "pytest", "sphinx"]
|
||||
docs = ["furo", "sphinx"]
|
||||
dev = ["pyopenssl", "service-identity[docs,idna,mypy,tests]"]
|
||||
docs = ["furo", "myst-parser", "pyopenssl", "sphinx", "sphinx-notfound-page"]
|
||||
idna = ["idna"]
|
||||
mypy = ["idna", "mypy", "types-pyopenssl"]
|
||||
tests = ["coverage[toml] (>=5.0.2)", "pytest"]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -2947,46 +2947,46 @@ files = [
|
|||
|
||||
[[package]]
|
||||
name = "types-commonmark"
|
||||
version = "0.9.2.3"
|
||||
version = "0.9.2.4"
|
||||
description = "Typing stubs for commonmark"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
files = [
|
||||
{file = "types-commonmark-0.9.2.3.tar.gz", hash = "sha256:42769a2c194fd5b49fd9eedfd4a83cd1d2514c6d0a36f00f5c5ffe0b6a2d2fcf"},
|
||||
{file = "types_commonmark-0.9.2.3-py3-none-any.whl", hash = "sha256:b575156e1b8a292d43acb36f861110b85c4bc7aa53bbfb5ac64addec15d18cfa"},
|
||||
{file = "types-commonmark-0.9.2.4.tar.gz", hash = "sha256:2c6486f65735cf18215cca3e962b17787fa545be279306f79b801f64a5319959"},
|
||||
{file = "types_commonmark-0.9.2.4-py3-none-any.whl", hash = "sha256:d5090fa685c3e3c0ec3a5973ff842000baef6d86f762d52209b3c5e9fbd0b555"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "types-jsonschema"
|
||||
version = "4.17.0.8"
|
||||
version = "4.17.0.10"
|
||||
description = "Typing stubs for jsonschema"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
files = [
|
||||
{file = "types-jsonschema-4.17.0.8.tar.gz", hash = "sha256:96a56990910f405e62de58862c0bbb3ac29ee6dba6d3d99aa0ba7f874cc547de"},
|
||||
{file = "types_jsonschema-4.17.0.8-py3-none-any.whl", hash = "sha256:f5958eb7b53217dfb5125f0412aeaef226a8a9013eac95816c95b5b523f6796b"},
|
||||
{file = "types-jsonschema-4.17.0.10.tar.gz", hash = "sha256:8e979db34d69bc9f9b3d6e8b89bdbc60b3a41cfce4e1fb87bf191d205c7f5098"},
|
||||
{file = "types_jsonschema-4.17.0.10-py3-none-any.whl", hash = "sha256:3aa2a89afbd9eaa6ce0c15618b36f02692a621433889ce73014656f7d8caf971"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "types-netaddr"
|
||||
version = "0.8.0.8"
|
||||
version = "0.8.0.9"
|
||||
description = "Typing stubs for netaddr"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
files = [
|
||||
{file = "types-netaddr-0.8.0.8.tar.gz", hash = "sha256:db7e8cd16b1244e7c4541edd0df99d1039fc05fd5387c21840f0b958fc52aabc"},
|
||||
{file = "types_netaddr-0.8.0.8-py3-none-any.whl", hash = "sha256:6741b3824e2ec3f7a74842b394439b71107c7675f8ae42bb2b5e7a8ebfe8cf18"},
|
||||
{file = "types-netaddr-0.8.0.9.tar.gz", hash = "sha256:68900c267fd31627c1721c5c52b32a257657ac2777457dca49b6b096ba2faf74"},
|
||||
{file = "types_netaddr-0.8.0.9-py3-none-any.whl", hash = "sha256:63e871f064cd59473cec1177f372526f0fa3d565050247d5305bdc325be5c3f6"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "types-opentracing"
|
||||
version = "2.4.10.5"
|
||||
version = "2.4.10.6"
|
||||
description = "Typing stubs for opentracing"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
files = [
|
||||
{file = "types-opentracing-2.4.10.5.tar.gz", hash = "sha256:852d13ab1324832835d50c00cfd58b9267f0e79ec3189e5664c2a90c26880fd4"},
|
||||
{file = "types_opentracing-2.4.10.5-py3-none-any.whl", hash = "sha256:8f12ab4dce3e298a8e6655da9a6d52171e7a275357eae4cec22a1663d94023a7"},
|
||||
{file = "types-opentracing-2.4.10.6.tar.gz", hash = "sha256:87a1bdfce9de5e555e30497663583b9b9c3bb494d029ef9806aa1f137c19e744"},
|
||||
{file = "types_opentracing-2.4.10.6-py3-none-any.whl", hash = "sha256:25914c834db033a4a38fc322df0b5e5e14503b0ac97f78304ae180d721555e97"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -3052,13 +3052,13 @@ types-urllib3 = "*"
|
|||
|
||||
[[package]]
|
||||
name = "types-setuptools"
|
||||
version = "68.0.0.0"
|
||||
version = "68.0.0.3"
|
||||
description = "Typing stubs for setuptools"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
files = [
|
||||
{file = "types-setuptools-68.0.0.0.tar.gz", hash = "sha256:fc958b4123b155ffc069a66d3af5fe6c1f9d0600c35c0c8444b2ab4147112641"},
|
||||
{file = "types_setuptools-68.0.0.0-py3-none-any.whl", hash = "sha256:cc00e09ba8f535362cbe1ea8b8407d15d14b59c57f4190cceaf61a9e57616446"},
|
||||
{file = "types-setuptools-68.0.0.3.tar.gz", hash = "sha256:d57ae6076100b5704b3cc869fdefc671e1baf4c2cd6643f84265dfc0b955bf05"},
|
||||
{file = "types_setuptools-68.0.0.3-py3-none-any.whl", hash = "sha256:fec09e5c18264c5c09351c00be01a34456fb7a88e457abe97401325f84ad9d36"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
|
|||
|
|
@ -89,7 +89,7 @@ manifest-path = "rust/Cargo.toml"
|
|||
|
||||
[tool.poetry]
|
||||
name = "matrix-synapse"
|
||||
version = "1.89.0rc1"
|
||||
version = "1.90.0rc1"
|
||||
description = "Homeserver for the Matrix decentralised comms protocol"
|
||||
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -13,6 +13,9 @@
|
|||
// limitations under the License.
|
||||
|
||||
#![feature(test)]
|
||||
|
||||
use std::borrow::Cow;
|
||||
|
||||
use synapse::push::{
|
||||
evaluator::PushRuleEvaluator, Condition, EventMatchCondition, FilteredPushRules, JsonValue,
|
||||
PushRules, SimpleJsonValue,
|
||||
|
|
@ -26,15 +29,15 @@ fn bench_match_exact(b: &mut Bencher) {
|
|||
let flattened_keys = [
|
||||
(
|
||||
"type".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("m.text".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("m.text"))),
|
||||
),
|
||||
(
|
||||
"room_id".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("!room:server".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("!room:server"))),
|
||||
),
|
||||
(
|
||||
"content.body".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("test message".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("test message"))),
|
||||
),
|
||||
]
|
||||
.into_iter()
|
||||
|
|
@ -71,15 +74,15 @@ fn bench_match_word(b: &mut Bencher) {
|
|||
let flattened_keys = [
|
||||
(
|
||||
"type".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("m.text".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("m.text"))),
|
||||
),
|
||||
(
|
||||
"room_id".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("!room:server".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("!room:server"))),
|
||||
),
|
||||
(
|
||||
"content.body".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("test message".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("test message"))),
|
||||
),
|
||||
]
|
||||
.into_iter()
|
||||
|
|
@ -116,15 +119,15 @@ fn bench_match_word_miss(b: &mut Bencher) {
|
|||
let flattened_keys = [
|
||||
(
|
||||
"type".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("m.text".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("m.text"))),
|
||||
),
|
||||
(
|
||||
"room_id".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("!room:server".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("!room:server"))),
|
||||
),
|
||||
(
|
||||
"content.body".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("test message".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("test message"))),
|
||||
),
|
||||
]
|
||||
.into_iter()
|
||||
|
|
@ -161,15 +164,15 @@ fn bench_eval_message(b: &mut Bencher) {
|
|||
let flattened_keys = [
|
||||
(
|
||||
"type".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("m.text".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("m.text"))),
|
||||
),
|
||||
(
|
||||
"room_id".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("!room:server".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("!room:server"))),
|
||||
),
|
||||
(
|
||||
"content.body".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("test message".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("test message"))),
|
||||
),
|
||||
]
|
||||
.into_iter()
|
||||
|
|
|
|||
|
|
@ -63,22 +63,6 @@ pub const BASE_PREPEND_OVERRIDE_RULES: &[PushRule] = &[PushRule {
|
|||
}];
|
||||
|
||||
pub const BASE_APPEND_OVERRIDE_RULES: &[PushRule] = &[
|
||||
// We don't want to notify on edits. Not only can this be confusing in real
|
||||
// time (2 notifications, one message) but it's especially confusing
|
||||
// if a bridge needs to edit a previously backfilled message.
|
||||
PushRule {
|
||||
rule_id: Cow::Borrowed("global/override/.com.beeper.suppress_edits"),
|
||||
priority_class: 5,
|
||||
conditions: Cow::Borrowed(&[Condition::Known(KnownCondition::EventMatch(
|
||||
EventMatchCondition {
|
||||
key: Cow::Borrowed("content.m\\.relates_to.rel_type"),
|
||||
pattern: Cow::Borrowed("m.replace"),
|
||||
},
|
||||
))]),
|
||||
actions: Cow::Borrowed(&[]),
|
||||
default: true,
|
||||
default_enabled: true,
|
||||
},
|
||||
PushRule {
|
||||
rule_id: Cow::Borrowed("global/override/.m.rule.suppress_notices"),
|
||||
priority_class: 5,
|
||||
|
|
@ -146,7 +130,7 @@ pub const BASE_APPEND_OVERRIDE_RULES: &[PushRule] = &[
|
|||
priority_class: 5,
|
||||
conditions: Cow::Borrowed(&[Condition::Known(
|
||||
KnownCondition::ExactEventPropertyContainsType(EventPropertyIsTypeCondition {
|
||||
key: Cow::Borrowed("content.m\\.mentions.user_ids"),
|
||||
key: Cow::Borrowed(r"content.m\.mentions.user_ids"),
|
||||
value_type: Cow::Borrowed(&EventMatchPatternType::UserId),
|
||||
}),
|
||||
)]),
|
||||
|
|
@ -167,8 +151,8 @@ pub const BASE_APPEND_OVERRIDE_RULES: &[PushRule] = &[
|
|||
priority_class: 5,
|
||||
conditions: Cow::Borrowed(&[
|
||||
Condition::Known(KnownCondition::EventPropertyIs(EventPropertyIsCondition {
|
||||
key: Cow::Borrowed("content.m\\.mentions.room"),
|
||||
value: Cow::Borrowed(&SimpleJsonValue::Bool(true)),
|
||||
key: Cow::Borrowed(r"content.m\.mentions.room"),
|
||||
value: Cow::Owned(SimpleJsonValue::Bool(true)),
|
||||
})),
|
||||
Condition::Known(KnownCondition::SenderNotificationPermission {
|
||||
key: Cow::Borrowed("room"),
|
||||
|
|
@ -241,6 +225,21 @@ pub const BASE_APPEND_OVERRIDE_RULES: &[PushRule] = &[
|
|||
default: true,
|
||||
default_enabled: true,
|
||||
},
|
||||
// We don't want to notify on edits *unless* the edit directly mentions a
|
||||
// user, which is handled above.
|
||||
PushRule {
|
||||
rule_id: Cow::Borrowed("global/override/.org.matrix.msc3958.suppress_edits"),
|
||||
priority_class: 5,
|
||||
conditions: Cow::Borrowed(&[Condition::Known(KnownCondition::EventPropertyIs(
|
||||
EventPropertyIsCondition {
|
||||
key: Cow::Borrowed(r"content.m\.relates_to.rel_type"),
|
||||
value: Cow::Owned(SimpleJsonValue::Str(Cow::Borrowed("m.replace"))),
|
||||
},
|
||||
))]),
|
||||
actions: Cow::Borrowed(&[]),
|
||||
default: true,
|
||||
default_enabled: true,
|
||||
},
|
||||
PushRule {
|
||||
rule_id: Cow::Borrowed("global/override/.org.matrix.msc3930.rule.poll_response"),
|
||||
priority_class: 5,
|
||||
|
|
|
|||
|
|
@ -117,7 +117,7 @@ impl PushRuleEvaluator {
|
|||
msc3931_enabled: bool,
|
||||
) -> Result<Self, Error> {
|
||||
let body = match flattened_keys.get("content.body") {
|
||||
Some(JsonValue::Value(SimpleJsonValue::Str(s))) => s.clone(),
|
||||
Some(JsonValue::Value(SimpleJsonValue::Str(s))) => s.clone().into_owned(),
|
||||
_ => String::new(),
|
||||
};
|
||||
|
||||
|
|
@ -313,13 +313,15 @@ impl PushRuleEvaluator {
|
|||
};
|
||||
|
||||
let pattern = match &*exact_event_match.value_type {
|
||||
EventMatchPatternType::UserId => user_id,
|
||||
EventMatchPatternType::UserLocalpart => get_localpart_from_id(user_id)?,
|
||||
EventMatchPatternType::UserId => user_id.to_owned(),
|
||||
EventMatchPatternType::UserLocalpart => {
|
||||
get_localpart_from_id(user_id)?.to_owned()
|
||||
}
|
||||
};
|
||||
|
||||
self.match_event_property_contains(
|
||||
exact_event_match.key.clone(),
|
||||
Cow::Borrowed(&SimpleJsonValue::Str(pattern.to_string())),
|
||||
Cow::Borrowed(&SimpleJsonValue::Str(Cow::Owned(pattern))),
|
||||
)?
|
||||
}
|
||||
KnownCondition::ContainsDisplayName => {
|
||||
|
|
@ -494,7 +496,7 @@ fn push_rule_evaluator() {
|
|||
let mut flattened_keys = BTreeMap::new();
|
||||
flattened_keys.insert(
|
||||
"content.body".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("foo bar bob hello".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("foo bar bob hello"))),
|
||||
);
|
||||
let evaluator = PushRuleEvaluator::py_new(
|
||||
flattened_keys,
|
||||
|
|
@ -522,7 +524,7 @@ fn test_requires_room_version_supports_condition() {
|
|||
let mut flattened_keys = BTreeMap::new();
|
||||
flattened_keys.insert(
|
||||
"content.body".to_string(),
|
||||
JsonValue::Value(SimpleJsonValue::Str("foo bar bob hello".to_string())),
|
||||
JsonValue::Value(SimpleJsonValue::Str(Cow::Borrowed("foo bar bob hello"))),
|
||||
);
|
||||
let flags = vec![RoomVersionFeatures::ExtensibleEvents.as_str().to_string()];
|
||||
let evaluator = PushRuleEvaluator::py_new(
|
||||
|
|
|
|||
|
|
@ -256,7 +256,7 @@ impl<'de> Deserialize<'de> for Action {
|
|||
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
|
||||
#[serde(untagged)]
|
||||
pub enum SimpleJsonValue {
|
||||
Str(String),
|
||||
Str(Cow<'static, str>),
|
||||
Int(i64),
|
||||
Bool(bool),
|
||||
Null,
|
||||
|
|
@ -265,7 +265,7 @@ pub enum SimpleJsonValue {
|
|||
impl<'source> FromPyObject<'source> for SimpleJsonValue {
|
||||
fn extract(ob: &'source PyAny) -> PyResult<Self> {
|
||||
if let Ok(s) = <PyString as pyo3::PyTryFrom>::try_from(ob) {
|
||||
Ok(SimpleJsonValue::Str(s.to_string()))
|
||||
Ok(SimpleJsonValue::Str(Cow::Owned(s.to_string())))
|
||||
// A bool *is* an int, ensure we try bool first.
|
||||
} else if let Ok(b) = <PyBool as pyo3::PyTryFrom>::try_from(ob) {
|
||||
Ok(SimpleJsonValue::Bool(b.extract()?))
|
||||
|
|
@ -585,7 +585,7 @@ impl FilteredPushRules {
|
|||
}
|
||||
|
||||
if !self.msc3958_suppress_edits_enabled
|
||||
&& rule.rule_id == "global/override/.com.beeper.suppress_edits"
|
||||
&& rule.rule_id == "global/override/.org.matrix.msc3958.suppress_edits"
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -761,7 +761,7 @@ class Porter:
|
|||
|
||||
# Step 2. Set up sequences
|
||||
#
|
||||
# We do this before porting the tables so that event if we fail half
|
||||
# We do this before porting the tables so that even if we fail half
|
||||
# way through the postgres DB always have sequences that are greater
|
||||
# than their respective tables. If we don't then creating the
|
||||
# `DataStore` object will fail due to the inconsistency.
|
||||
|
|
@ -769,6 +769,10 @@ class Porter:
|
|||
await self._setup_state_group_id_seq()
|
||||
await self._setup_user_id_seq()
|
||||
await self._setup_events_stream_seqs()
|
||||
await self._setup_sequence(
|
||||
"un_partial_stated_event_stream_sequence",
|
||||
("un_partial_stated_event_stream",),
|
||||
)
|
||||
await self._setup_sequence(
|
||||
"device_inbox_sequence", ("device_inbox", "device_federation_outbox")
|
||||
)
|
||||
|
|
@ -779,6 +783,11 @@ class Porter:
|
|||
await self._setup_sequence("receipts_sequence", ("receipts_linearized",))
|
||||
await self._setup_sequence("presence_stream_sequence", ("presence_stream",))
|
||||
await self._setup_auth_chain_sequence()
|
||||
await self._setup_sequence(
|
||||
"application_services_txn_id_seq",
|
||||
("application_services_txns",),
|
||||
"txn_id",
|
||||
)
|
||||
|
||||
# Step 3. Get tables.
|
||||
self.progress.set_state("Fetching tables")
|
||||
|
|
@ -1083,7 +1092,10 @@ class Porter:
|
|||
)
|
||||
|
||||
async def _setup_sequence(
|
||||
self, sequence_name: str, stream_id_tables: Iterable[str]
|
||||
self,
|
||||
sequence_name: str,
|
||||
stream_id_tables: Iterable[str],
|
||||
column_name: str = "stream_id",
|
||||
) -> None:
|
||||
"""Set a sequence to the correct value."""
|
||||
current_stream_ids = []
|
||||
|
|
@ -1093,7 +1105,7 @@ class Porter:
|
|||
await self.sqlite_store.db_pool.simple_select_one_onecol(
|
||||
table=stream_id_table,
|
||||
keyvalues={},
|
||||
retcol="COALESCE(MAX(stream_id), 1)",
|
||||
retcol=f"COALESCE(MAX({column_name}), 1)",
|
||||
allow_none=True,
|
||||
),
|
||||
)
|
||||
|
|
|
|||
|
|
@ -16,9 +16,6 @@ import logging
|
|||
import urllib.parse
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
Any,
|
||||
Awaitable,
|
||||
Callable,
|
||||
Dict,
|
||||
Iterable,
|
||||
List,
|
||||
|
|
@ -27,10 +24,11 @@ from typing import (
|
|||
Sequence,
|
||||
Tuple,
|
||||
TypeVar,
|
||||
Union,
|
||||
)
|
||||
|
||||
from prometheus_client import Counter
|
||||
from typing_extensions import Concatenate, ParamSpec, TypeGuard
|
||||
from typing_extensions import ParamSpec, TypeGuard
|
||||
|
||||
from synapse.api.constants import EventTypes, Membership, ThirdPartyEntityKind
|
||||
from synapse.api.errors import CodeMessageException, HttpResponseException
|
||||
|
|
@ -80,9 +78,7 @@ sent_todevice_counter = Counter(
|
|||
|
||||
HOUR_IN_MS = 60 * 60 * 1000
|
||||
|
||||
|
||||
APP_SERVICE_PREFIX = "/_matrix/app/v1"
|
||||
APP_SERVICE_UNSTABLE_PREFIX = "/_matrix/app/unstable"
|
||||
|
||||
P = ParamSpec("P")
|
||||
R = TypeVar("R")
|
||||
|
|
@ -123,52 +119,12 @@ class ApplicationServiceApi(SimpleHttpClient):
|
|||
def __init__(self, hs: "HomeServer"):
|
||||
super().__init__(hs)
|
||||
self.clock = hs.get_clock()
|
||||
self.config = hs.config.appservice
|
||||
|
||||
self.protocol_meta_cache: ResponseCache[Tuple[str, str]] = ResponseCache(
|
||||
hs.get_clock(), "as_protocol_meta", timeout_ms=HOUR_IN_MS
|
||||
)
|
||||
|
||||
async def _send_with_fallbacks(
|
||||
self,
|
||||
service: "ApplicationService",
|
||||
prefixes: List[str],
|
||||
path: str,
|
||||
func: Callable[Concatenate[str, P], Awaitable[R]],
|
||||
*args: P.args,
|
||||
**kwargs: P.kwargs,
|
||||
) -> R:
|
||||
"""
|
||||
Attempt to call an application service with multiple paths, falling back
|
||||
until one succeeds.
|
||||
|
||||
Args:
|
||||
service: The appliacation service, this provides the base URL.
|
||||
prefixes: A last of paths to try in order for the requests.
|
||||
path: A suffix to append to each prefix.
|
||||
func: The function to call, the first argument will be the full
|
||||
endpoint to fetch. Other arguments are provided by args/kwargs.
|
||||
|
||||
Returns:
|
||||
The return value of func.
|
||||
"""
|
||||
for i, prefix in enumerate(prefixes, start=1):
|
||||
uri = f"{service.url}{prefix}{path}"
|
||||
try:
|
||||
return await func(uri, *args, **kwargs)
|
||||
except HttpResponseException as e:
|
||||
# If an error is received that is due to an unrecognised path,
|
||||
# fallback to next path (if one exists). Otherwise, consider it
|
||||
# a legitimate error and raise.
|
||||
if i < len(prefixes) and is_unknown_endpoint(e):
|
||||
continue
|
||||
raise
|
||||
except Exception:
|
||||
# Unexpected exceptions get sent to the caller.
|
||||
raise
|
||||
|
||||
# The function should always exit via the return or raise above this.
|
||||
raise RuntimeError("Unexpected fallback behaviour. This should never be seen.")
|
||||
|
||||
async def query_user(self, service: "ApplicationService", user_id: str) -> bool:
|
||||
if service.url is None:
|
||||
return False
|
||||
|
|
@ -177,12 +133,12 @@ class ApplicationServiceApi(SimpleHttpClient):
|
|||
assert service.hs_token is not None
|
||||
|
||||
try:
|
||||
response = await self._send_with_fallbacks(
|
||||
service,
|
||||
[APP_SERVICE_PREFIX, ""],
|
||||
f"/users/{urllib.parse.quote(user_id)}",
|
||||
self.get_json,
|
||||
{"access_token": service.hs_token},
|
||||
args = None
|
||||
if self.config.use_appservice_legacy_authorization:
|
||||
args = {"access_token": service.hs_token}
|
||||
response = await self.get_json(
|
||||
f"{service.url}{APP_SERVICE_PREFIX}/users/{urllib.parse.quote(user_id)}",
|
||||
args,
|
||||
headers={"Authorization": [f"Bearer {service.hs_token}"]},
|
||||
)
|
||||
if response is not None: # just an empty json object
|
||||
|
|
@ -203,12 +159,12 @@ class ApplicationServiceApi(SimpleHttpClient):
|
|||
assert service.hs_token is not None
|
||||
|
||||
try:
|
||||
response = await self._send_with_fallbacks(
|
||||
service,
|
||||
[APP_SERVICE_PREFIX, ""],
|
||||
f"/rooms/{urllib.parse.quote(alias)}",
|
||||
self.get_json,
|
||||
{"access_token": service.hs_token},
|
||||
args = None
|
||||
if self.config.use_appservice_legacy_authorization:
|
||||
args = {"access_token": service.hs_token}
|
||||
response = await self.get_json(
|
||||
f"{service.url}{APP_SERVICE_PREFIX}/rooms/{urllib.parse.quote(alias)}",
|
||||
args,
|
||||
headers={"Authorization": [f"Bearer {service.hs_token}"]},
|
||||
)
|
||||
if response is not None: # just an empty json object
|
||||
|
|
@ -241,15 +197,14 @@ class ApplicationServiceApi(SimpleHttpClient):
|
|||
assert service.hs_token is not None
|
||||
|
||||
try:
|
||||
args: Mapping[Any, Any] = {
|
||||
**fields,
|
||||
b"access_token": service.hs_token,
|
||||
}
|
||||
response = await self._send_with_fallbacks(
|
||||
service,
|
||||
[APP_SERVICE_PREFIX, APP_SERVICE_UNSTABLE_PREFIX],
|
||||
f"/thirdparty/{kind}/{urllib.parse.quote(protocol)}",
|
||||
self.get_json,
|
||||
args: Mapping[bytes, Union[List[bytes], str]] = fields
|
||||
if self.config.use_appservice_legacy_authorization:
|
||||
args = {
|
||||
**fields,
|
||||
b"access_token": service.hs_token,
|
||||
}
|
||||
response = await self.get_json(
|
||||
f"{service.url}{APP_SERVICE_PREFIX}/thirdparty/{kind}/{urllib.parse.quote(protocol)}",
|
||||
args=args,
|
||||
headers={"Authorization": [f"Bearer {service.hs_token}"]},
|
||||
)
|
||||
|
|
@ -285,12 +240,12 @@ class ApplicationServiceApi(SimpleHttpClient):
|
|||
# This is required by the configuration.
|
||||
assert service.hs_token is not None
|
||||
try:
|
||||
info = await self._send_with_fallbacks(
|
||||
service,
|
||||
[APP_SERVICE_PREFIX, APP_SERVICE_UNSTABLE_PREFIX],
|
||||
f"/thirdparty/protocol/{urllib.parse.quote(protocol)}",
|
||||
self.get_json,
|
||||
{"access_token": service.hs_token},
|
||||
args = None
|
||||
if self.config.use_appservice_legacy_authorization:
|
||||
args = {"access_token": service.hs_token}
|
||||
info = await self.get_json(
|
||||
f"{service.url}{APP_SERVICE_PREFIX}/thirdparty/protocol/{urllib.parse.quote(protocol)}",
|
||||
args,
|
||||
headers={"Authorization": [f"Bearer {service.hs_token}"]},
|
||||
)
|
||||
|
||||
|
|
@ -401,13 +356,14 @@ class ApplicationServiceApi(SimpleHttpClient):
|
|||
}
|
||||
|
||||
try:
|
||||
await self._send_with_fallbacks(
|
||||
service,
|
||||
[APP_SERVICE_PREFIX, ""],
|
||||
f"/transactions/{urllib.parse.quote(str(txn_id))}",
|
||||
self.put_json,
|
||||
args = None
|
||||
if self.config.use_appservice_legacy_authorization:
|
||||
args = {"access_token": service.hs_token}
|
||||
|
||||
await self.put_json(
|
||||
f"{service.url}{APP_SERVICE_PREFIX}/transactions/{urllib.parse.quote(str(txn_id))}",
|
||||
json_body=body,
|
||||
args={"access_token": service.hs_token},
|
||||
args=args,
|
||||
headers={"Authorization": [f"Bearer {service.hs_token}"]},
|
||||
)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
|
|
|
|||
|
|
@ -43,6 +43,14 @@ class AppServiceConfig(Config):
|
|||
)
|
||||
|
||||
self.track_appservice_user_ips = config.get("track_appservice_user_ips", False)
|
||||
self.use_appservice_legacy_authorization = config.get(
|
||||
"use_appservice_legacy_authorization", False
|
||||
)
|
||||
if self.use_appservice_legacy_authorization:
|
||||
logger.warning(
|
||||
"The use of appservice legacy authorization via query params is deprecated"
|
||||
" and should be considered insecure."
|
||||
)
|
||||
|
||||
|
||||
def load_appservices(
|
||||
|
|
|
|||
|
|
@ -216,12 +216,6 @@ class MSC3861:
|
|||
("session_lifetime",),
|
||||
)
|
||||
|
||||
if not root.experimental.msc3970_enabled:
|
||||
raise ConfigError(
|
||||
"experimental_features.msc3970_enabled must be 'true' when OAuth delegation is enabled",
|
||||
("experimental_features", "msc3970_enabled"),
|
||||
)
|
||||
|
||||
|
||||
@attr.s(auto_attribs=True, frozen=True, slots=True)
|
||||
class MSC3866Config:
|
||||
|
|
@ -397,9 +391,6 @@ class ExperimentalConfig(Config):
|
|||
"Invalid MSC3861 configuration", ("experimental", "msc3861")
|
||||
) from exc
|
||||
|
||||
# MSC3970: Scope transaction IDs to devices
|
||||
self.msc3970_enabled = experimental.get("msc3970_enabled", self.msc3861.enabled)
|
||||
|
||||
# Check that none of the other config options conflict with MSC3861 when enabled
|
||||
self.msc3861.check_config_conflicts(self.root)
|
||||
|
||||
|
|
|
|||
|
|
@ -65,5 +65,23 @@ class FederationConfig(Config):
|
|||
self.max_long_retries = federation_config.get("max_long_retries", 10)
|
||||
self.max_short_retries = federation_config.get("max_short_retries", 3)
|
||||
|
||||
# Allow for the configuration of the backoff algorithm used
|
||||
# when trying to reach an unavailable destination.
|
||||
# Unlike previous configuration those values applies across
|
||||
# multiple requests and the state of the backoff is stored on DB.
|
||||
self.destination_min_retry_interval_ms = Config.parse_duration(
|
||||
federation_config.get("destination_min_retry_interval", "10m")
|
||||
)
|
||||
self.destination_retry_multiplier = federation_config.get(
|
||||
"destination_retry_multiplier", 2
|
||||
)
|
||||
self.destination_max_retry_interval_ms = min(
|
||||
Config.parse_duration(
|
||||
federation_config.get("destination_max_retry_interval", "7d")
|
||||
),
|
||||
# Set a hard-limit to not overflow the database column.
|
||||
2**62,
|
||||
)
|
||||
|
||||
|
||||
_METRICS_FOR_DOMAINS_SCHEMA = {"type": "array", "items": {"type": "string"}}
|
||||
|
|
|
|||
|
|
@ -186,9 +186,6 @@ class EventContext(UnpersistedEventContextBase):
|
|||
),
|
||||
"app_service_id": self.app_service.id if self.app_service else None,
|
||||
"partial_state": self.partial_state,
|
||||
# add dummy delta_ids and prev_group for backwards compatibility
|
||||
"delta_ids": None,
|
||||
"prev_group": None,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -203,13 +200,6 @@ class EventContext(UnpersistedEventContextBase):
|
|||
Returns:
|
||||
The event context.
|
||||
"""
|
||||
# workaround for backwards/forwards compatibility: if the input doesn't have a value
|
||||
# for "state_group_deltas" just assign an empty dict
|
||||
state_group_deltas = input.get("state_group_deltas", None)
|
||||
if state_group_deltas:
|
||||
state_group_deltas = _decode_state_group_delta(state_group_deltas)
|
||||
else:
|
||||
state_group_deltas = {}
|
||||
|
||||
context = EventContext(
|
||||
# We use the state_group and prev_state_id stuff to pull the
|
||||
|
|
@ -217,7 +207,7 @@ class EventContext(UnpersistedEventContextBase):
|
|||
storage=storage,
|
||||
state_group=input["state_group"],
|
||||
state_group_before_event=input["state_group_before_event"],
|
||||
state_group_deltas=state_group_deltas,
|
||||
state_group_deltas=_decode_state_group_delta(input["state_group_deltas"]),
|
||||
state_delta_due_to_event=_decode_state_dict(
|
||||
input["state_delta_due_to_event"]
|
||||
),
|
||||
|
|
|
|||
|
|
@ -394,7 +394,6 @@ def serialize_event(
|
|||
time_now_ms: int,
|
||||
*,
|
||||
config: SerializeEventConfig = _DEFAULT_SERIALIZE_EVENT_CONFIG,
|
||||
msc3970_enabled: bool = False,
|
||||
) -> JsonDict:
|
||||
"""Serialize event for clients
|
||||
|
||||
|
|
@ -402,8 +401,6 @@ def serialize_event(
|
|||
e
|
||||
time_now_ms
|
||||
config: Event serialization config
|
||||
msc3970_enabled: Whether MSC3970 is enabled. It changes whether we should
|
||||
include the `transaction_id` in the event's `unsigned` section.
|
||||
|
||||
Returns:
|
||||
The serialized event dictionary.
|
||||
|
|
@ -429,38 +426,46 @@ def serialize_event(
|
|||
e.unsigned["redacted_because"],
|
||||
time_now_ms,
|
||||
config=config,
|
||||
msc3970_enabled=msc3970_enabled,
|
||||
)
|
||||
|
||||
# If we have a txn_id saved in the internal_metadata, we should include it in the
|
||||
# unsigned section of the event if it was sent by the same session as the one
|
||||
# requesting the event.
|
||||
txn_id: Optional[str] = getattr(e.internal_metadata, "txn_id", None)
|
||||
if txn_id is not None and config.requester is not None:
|
||||
# For the MSC3970 rules to be applied, we *need* to have the device ID in the
|
||||
# event internal metadata. Since we were not recording them before, if it hasn't
|
||||
# been recorded, we fallback to the old behaviour.
|
||||
if (
|
||||
txn_id is not None
|
||||
and config.requester is not None
|
||||
and config.requester.user.to_string() == e.sender
|
||||
):
|
||||
# Some events do not have the device ID stored in the internal metadata,
|
||||
# this includes old events as well as those created by appservice, guests,
|
||||
# or with tokens minted with the admin API. For those events, fallback
|
||||
# to using the access token instead.
|
||||
event_device_id: Optional[str] = getattr(e.internal_metadata, "device_id", None)
|
||||
if msc3970_enabled and event_device_id is not None:
|
||||
if event_device_id is not None:
|
||||
if event_device_id == config.requester.device_id:
|
||||
d["unsigned"]["transaction_id"] = txn_id
|
||||
|
||||
else:
|
||||
# The pre-MSC3970 behaviour is to only include the transaction ID if the
|
||||
# event was sent from the same access token. For regular users, we can use
|
||||
# the access token ID to determine this. For guests, we can't, but since
|
||||
# each guest only has one access token, we can just check that the event was
|
||||
# sent by the same user as the one requesting the event.
|
||||
# Fallback behaviour: only include the transaction ID if the event
|
||||
# was sent from the same access token.
|
||||
#
|
||||
# For regular users, the access token ID can be used to determine this.
|
||||
# This includes access tokens minted with the admin API.
|
||||
#
|
||||
# For guests and appservice users, we can't check the access token ID
|
||||
# so assume it is the same session.
|
||||
event_token_id: Optional[int] = getattr(
|
||||
e.internal_metadata, "token_id", None
|
||||
)
|
||||
if config.requester.user.to_string() == e.sender and (
|
||||
if (
|
||||
(
|
||||
event_token_id is not None
|
||||
and config.requester.access_token_id is not None
|
||||
and event_token_id == config.requester.access_token_id
|
||||
)
|
||||
or config.requester.is_guest
|
||||
or config.requester.app_service
|
||||
):
|
||||
d["unsigned"]["transaction_id"] = txn_id
|
||||
|
||||
|
|
@ -475,14 +480,16 @@ def serialize_event(
|
|||
if config.as_client_event:
|
||||
d = config.event_format(d)
|
||||
|
||||
# If the event is a redaction, copy the redacts field from the content to
|
||||
# top-level for backwards compatibility.
|
||||
if (
|
||||
e.type == EventTypes.Redaction
|
||||
and e.room_version.updated_redaction_rules
|
||||
and e.redacts is not None
|
||||
):
|
||||
d["redacts"] = e.redacts
|
||||
# If the event is a redaction, the field with the redacted event ID appears
|
||||
# in a different location depending on the room version. e.redacts handles
|
||||
# fetching from the proper location; copy it to the other location for forwards-
|
||||
# and backwards-compatibility with clients.
|
||||
if e.type == EventTypes.Redaction and e.redacts is not None:
|
||||
if e.room_version.updated_redaction_rules:
|
||||
d["redacts"] = e.redacts
|
||||
else:
|
||||
d["content"] = dict(d["content"])
|
||||
d["content"]["redacts"] = e.redacts
|
||||
|
||||
only_event_fields = config.only_event_fields
|
||||
if only_event_fields:
|
||||
|
|
@ -502,9 +509,6 @@ class EventClientSerializer:
|
|||
clients.
|
||||
"""
|
||||
|
||||
def __init__(self, *, msc3970_enabled: bool = False):
|
||||
self._msc3970_enabled = msc3970_enabled
|
||||
|
||||
def serialize_event(
|
||||
self,
|
||||
event: Union[JsonDict, EventBase],
|
||||
|
|
@ -529,9 +533,7 @@ class EventClientSerializer:
|
|||
if not isinstance(event, EventBase):
|
||||
return event
|
||||
|
||||
serialized_event = serialize_event(
|
||||
event, time_now, config=config, msc3970_enabled=self._msc3970_enabled
|
||||
)
|
||||
serialized_event = serialize_event(event, time_now, config=config)
|
||||
|
||||
# Check if there are any bundled aggregations to include with the event.
|
||||
if bundle_aggregations:
|
||||
|
|
|
|||
|
|
@ -63,6 +63,7 @@ from synapse.federation.federation_base import (
|
|||
)
|
||||
from synapse.federation.persistence import TransactionActions
|
||||
from synapse.federation.units import Edu, Transaction
|
||||
from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
|
||||
from synapse.http.servlet import assert_params_in_dict
|
||||
from synapse.logging.context import (
|
||||
make_deferred_yieldable,
|
||||
|
|
@ -137,6 +138,7 @@ class FederationServer(FederationBase):
|
|||
self._event_auth_handler = hs.get_event_auth_handler()
|
||||
self._room_member_handler = hs.get_room_member_handler()
|
||||
self._e2e_keys_handler = hs.get_e2e_keys_handler()
|
||||
self._worker_lock_handler = hs.get_worker_locks_handler()
|
||||
|
||||
self._state_storage_controller = hs.get_storage_controllers().state
|
||||
|
||||
|
|
@ -1236,9 +1238,18 @@ class FederationServer(FederationBase):
|
|||
logger.info("handling received PDU in room %s: %s", room_id, event)
|
||||
try:
|
||||
with nested_logging_context(event.event_id):
|
||||
await self._federation_event_handler.on_receive_pdu(
|
||||
origin, event
|
||||
)
|
||||
# We're taking out a lock within a lock, which could
|
||||
# lead to deadlocks if we're not careful. However, it is
|
||||
# safe on this occasion as we only ever take a write
|
||||
# lock when deleting a room, which we would never do
|
||||
# while holding the `_INBOUND_EVENT_HANDLING_LOCK_NAME`
|
||||
# lock.
|
||||
async with self._worker_lock_handler.acquire_read_write_lock(
|
||||
DELETE_ROOM_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
await self._federation_event_handler.on_receive_pdu(
|
||||
origin, event
|
||||
)
|
||||
except FederationError as e:
|
||||
# XXX: Ideally we'd inform the remote we failed to process
|
||||
# the event, but we can't return an error in the transaction
|
||||
|
|
|
|||
|
|
@ -722,6 +722,22 @@ class DeviceHandler(DeviceWorkerHandler):
|
|||
|
||||
return {"success": True}
|
||||
|
||||
async def delete_dehydrated_device(self, user_id: str, device_id: str) -> None:
|
||||
"""
|
||||
Delete a stored dehydrated device.
|
||||
|
||||
Args:
|
||||
user_id: the user_id to delete the device from
|
||||
device_id: id of the dehydrated device to delete
|
||||
"""
|
||||
success = await self.store.remove_dehydrated_device(user_id, device_id)
|
||||
|
||||
if not success:
|
||||
raise errors.NotFoundError()
|
||||
|
||||
await self.delete_devices(user_id, [device_id])
|
||||
await self.store.delete_e2e_keys_by_device(user_id=user_id, device_id=device_id)
|
||||
|
||||
@wrap_as_background_process("_handle_new_device_update_async")
|
||||
async def _handle_new_device_update_async(self) -> None:
|
||||
"""Called when we have a new local device list update that we need to
|
||||
|
|
|
|||
|
|
@ -53,6 +53,7 @@ from synapse.events.snapshot import EventContext, UnpersistedEventContextBase
|
|||
from synapse.events.utils import SerializeEventConfig, maybe_upsert_event_field
|
||||
from synapse.events.validator import EventValidator
|
||||
from synapse.handlers.directory import DirectoryHandler
|
||||
from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
|
||||
from synapse.logging import opentracing
|
||||
from synapse.logging.context import make_deferred_yieldable, run_in_background
|
||||
from synapse.metrics.background_process_metrics import run_as_background_process
|
||||
|
|
@ -485,6 +486,7 @@ class EventCreationHandler:
|
|||
self._events_shard_config = self.config.worker.events_shard_config
|
||||
self._instance_name = hs.get_instance_name()
|
||||
self._notifier = hs.get_notifier()
|
||||
self._worker_lock_handler = hs.get_worker_locks_handler()
|
||||
|
||||
self.room_prejoin_state_types = self.hs.config.api.room_prejoin_state
|
||||
|
||||
|
|
@ -559,8 +561,6 @@ class EventCreationHandler:
|
|||
expiry_ms=30 * 60 * 1000,
|
||||
)
|
||||
|
||||
self._msc3970_enabled = hs.config.experimental.msc3970_enabled
|
||||
|
||||
async def create_event(
|
||||
self,
|
||||
requester: Requester,
|
||||
|
|
@ -876,6 +876,53 @@ class EventCreationHandler:
|
|||
return prev_event
|
||||
return None
|
||||
|
||||
async def get_event_id_from_transaction(
|
||||
self,
|
||||
requester: Requester,
|
||||
txn_id: str,
|
||||
room_id: str,
|
||||
) -> Optional[str]:
|
||||
"""For the given transaction ID and room ID, check if there is a matching event ID.
|
||||
|
||||
Args:
|
||||
requester: The requester making the request in the context of which we want
|
||||
to fetch the event.
|
||||
txn_id: The transaction ID.
|
||||
room_id: The room ID.
|
||||
|
||||
Returns:
|
||||
An event ID if one could be found, None otherwise.
|
||||
"""
|
||||
existing_event_id = None
|
||||
|
||||
# According to the spec, transactions are scoped to a user's device ID.
|
||||
if requester.device_id:
|
||||
existing_event_id = (
|
||||
await self.store.get_event_id_from_transaction_id_and_device_id(
|
||||
room_id,
|
||||
requester.user.to_string(),
|
||||
requester.device_id,
|
||||
txn_id,
|
||||
)
|
||||
)
|
||||
if existing_event_id:
|
||||
return existing_event_id
|
||||
|
||||
# Some requsters don't have device IDs (appservice, guests, and access
|
||||
# tokens minted with the admin API), fallback to checking the access token
|
||||
# ID, which should be close enough.
|
||||
if requester.access_token_id:
|
||||
existing_event_id = (
|
||||
await self.store.get_event_id_from_transaction_id_and_token_id(
|
||||
room_id,
|
||||
requester.user.to_string(),
|
||||
requester.access_token_id,
|
||||
txn_id,
|
||||
)
|
||||
)
|
||||
|
||||
return existing_event_id
|
||||
|
||||
async def get_event_from_transaction(
|
||||
self,
|
||||
requester: Requester,
|
||||
|
|
@ -894,35 +941,11 @@ class EventCreationHandler:
|
|||
Returns:
|
||||
An event if one could be found, None otherwise.
|
||||
"""
|
||||
|
||||
if self._msc3970_enabled and requester.device_id:
|
||||
# When MSC3970 is enabled, we lookup for events sent by the same device first,
|
||||
# and fallback to the old behaviour if none were found.
|
||||
existing_event_id = (
|
||||
await self.store.get_event_id_from_transaction_id_and_device_id(
|
||||
room_id,
|
||||
requester.user.to_string(),
|
||||
requester.device_id,
|
||||
txn_id,
|
||||
)
|
||||
)
|
||||
if existing_event_id:
|
||||
return await self.store.get_event(existing_event_id)
|
||||
|
||||
# Pre-MSC3970, we looked up for events that were sent by the same session by
|
||||
# using the access token ID.
|
||||
if requester.access_token_id:
|
||||
existing_event_id = (
|
||||
await self.store.get_event_id_from_transaction_id_and_token_id(
|
||||
room_id,
|
||||
requester.user.to_string(),
|
||||
requester.access_token_id,
|
||||
txn_id,
|
||||
)
|
||||
)
|
||||
if existing_event_id:
|
||||
return await self.store.get_event(existing_event_id)
|
||||
|
||||
existing_event_id = await self.get_event_id_from_transaction(
|
||||
requester, txn_id, room_id
|
||||
)
|
||||
if existing_event_id:
|
||||
return await self.store.get_event(existing_event_id)
|
||||
return None
|
||||
|
||||
async def create_and_send_nonmember_event(
|
||||
|
|
@ -1010,6 +1033,37 @@ class EventCreationHandler:
|
|||
event.internal_metadata.stream_ordering,
|
||||
)
|
||||
|
||||
async with self._worker_lock_handler.acquire_read_write_lock(
|
||||
DELETE_ROOM_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
return await self._create_and_send_nonmember_event_locked(
|
||||
requester=requester,
|
||||
event_dict=event_dict,
|
||||
allow_no_prev_events=allow_no_prev_events,
|
||||
prev_event_ids=prev_event_ids,
|
||||
state_event_ids=state_event_ids,
|
||||
ratelimit=ratelimit,
|
||||
txn_id=txn_id,
|
||||
ignore_shadow_ban=ignore_shadow_ban,
|
||||
outlier=outlier,
|
||||
depth=depth,
|
||||
)
|
||||
|
||||
async def _create_and_send_nonmember_event_locked(
|
||||
self,
|
||||
requester: Requester,
|
||||
event_dict: dict,
|
||||
allow_no_prev_events: bool = False,
|
||||
prev_event_ids: Optional[List[str]] = None,
|
||||
state_event_ids: Optional[List[str]] = None,
|
||||
ratelimit: bool = True,
|
||||
txn_id: Optional[str] = None,
|
||||
ignore_shadow_ban: bool = False,
|
||||
outlier: bool = False,
|
||||
depth: Optional[int] = None,
|
||||
) -> Tuple[EventBase, int]:
|
||||
room_id = event_dict["room_id"]
|
||||
|
||||
# If we don't have any prev event IDs specified then we need to
|
||||
# check that the host is in the room (as otherwise populating the
|
||||
# prev events will fail), at which point we may as well check the
|
||||
|
|
@ -1923,7 +1977,10 @@ class EventCreationHandler:
|
|||
)
|
||||
|
||||
for room_id in room_ids:
|
||||
dummy_event_sent = await self._send_dummy_event_for_room(room_id)
|
||||
async with self._worker_lock_handler.acquire_read_write_lock(
|
||||
DELETE_ROOM_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
dummy_event_sent = await self._send_dummy_event_for_room(room_id)
|
||||
|
||||
if not dummy_event_sent:
|
||||
# Did not find a valid user in the room, so remove from future attempts
|
||||
|
|
|
|||
|
|
@ -46,6 +46,11 @@ logger = logging.getLogger(__name__)
|
|||
BACKFILL_BECAUSE_TOO_MANY_GAPS_THRESHOLD = 3
|
||||
|
||||
|
||||
PURGE_HISTORY_LOCK_NAME = "purge_history_lock"
|
||||
|
||||
DELETE_ROOM_LOCK_NAME = "delete_room_lock"
|
||||
|
||||
|
||||
@attr.s(slots=True, auto_attribs=True)
|
||||
class PurgeStatus:
|
||||
"""Object tracking the status of a purge request
|
||||
|
|
@ -142,6 +147,7 @@ class PaginationHandler:
|
|||
self._server_name = hs.hostname
|
||||
self._room_shutdown_handler = hs.get_room_shutdown_handler()
|
||||
self._relations_handler = hs.get_relations_handler()
|
||||
self._worker_locks = hs.get_worker_locks_handler()
|
||||
|
||||
self.pagination_lock = ReadWriteLock()
|
||||
# IDs of rooms in which there currently an active purge *or delete* operation.
|
||||
|
|
@ -356,7 +362,9 @@ class PaginationHandler:
|
|||
"""
|
||||
self._purges_in_progress_by_room.add(room_id)
|
||||
try:
|
||||
async with self.pagination_lock.write(room_id):
|
||||
async with self._worker_locks.acquire_read_write_lock(
|
||||
PURGE_HISTORY_LOCK_NAME, room_id, write=True
|
||||
):
|
||||
await self._storage_controllers.purge_events.purge_history(
|
||||
room_id, token, delete_local_events
|
||||
)
|
||||
|
|
@ -412,7 +420,10 @@ class PaginationHandler:
|
|||
room_id: room to be purged
|
||||
force: set true to skip checking for joined users.
|
||||
"""
|
||||
async with self.pagination_lock.write(room_id):
|
||||
async with self._worker_locks.acquire_multi_read_write_lock(
|
||||
[(PURGE_HISTORY_LOCK_NAME, room_id), (DELETE_ROOM_LOCK_NAME, room_id)],
|
||||
write=True,
|
||||
):
|
||||
# first check that we have no users in this room
|
||||
if not force:
|
||||
joined = await self.store.is_host_joined(room_id, self._server_name)
|
||||
|
|
@ -471,7 +482,9 @@ class PaginationHandler:
|
|||
|
||||
room_token = from_token.room_key
|
||||
|
||||
async with self.pagination_lock.read(room_id):
|
||||
async with self._worker_locks.acquire_read_write_lock(
|
||||
PURGE_HISTORY_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
(membership, member_event_id) = (None, None)
|
||||
if not use_admin_priviledge:
|
||||
(
|
||||
|
|
@ -747,7 +760,9 @@ class PaginationHandler:
|
|||
|
||||
self._purges_in_progress_by_room.add(room_id)
|
||||
try:
|
||||
async with self.pagination_lock.write(room_id):
|
||||
async with self._worker_locks.acquire_read_write_lock(
|
||||
PURGE_HISTORY_LOCK_NAME, room_id, write=True
|
||||
):
|
||||
self._delete_by_id[delete_id].status = DeleteStatus.STATUS_SHUTTING_DOWN
|
||||
self._delete_by_id[
|
||||
delete_id
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ class ProfileHandler:
|
|||
|
||||
if self.hs.is_mine(target_user):
|
||||
profileinfo = await self.store.get_profileinfo(target_user)
|
||||
if profileinfo.display_name is None:
|
||||
if profileinfo.display_name is None and profileinfo.avatar_url is None:
|
||||
raise SynapseError(404, "Profile was not found", Codes.NOT_FOUND)
|
||||
|
||||
return {
|
||||
|
|
@ -163,7 +163,7 @@ class ProfileHandler:
|
|||
400, "Displayname is too long (max %i)" % (MAX_DISPLAYNAME_LEN,)
|
||||
)
|
||||
|
||||
displayname_to_set: Optional[str] = new_displayname
|
||||
displayname_to_set: Optional[str] = new_displayname.strip()
|
||||
if new_displayname == "":
|
||||
displayname_to_set = None
|
||||
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ from synapse.events import EventBase
|
|||
from synapse.events.snapshot import EventContext
|
||||
from synapse.handlers.profile import MAX_AVATAR_URL_LEN, MAX_DISPLAYNAME_LEN
|
||||
from synapse.handlers.state_deltas import MatchChange, StateDeltasHandler
|
||||
from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
|
||||
from synapse.logging import opentracing
|
||||
from synapse.metrics import event_processing_positions
|
||||
from synapse.metrics.background_process_metrics import run_as_background_process
|
||||
|
|
@ -94,6 +95,7 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
|
|||
self.event_creation_handler = hs.get_event_creation_handler()
|
||||
self.account_data_handler = hs.get_account_data_handler()
|
||||
self.event_auth_handler = hs.get_event_auth_handler()
|
||||
self._worker_lock_handler = hs.get_worker_locks_handler()
|
||||
|
||||
self.member_linearizer: Linearizer = Linearizer(name="member")
|
||||
self.member_as_limiter = Linearizer(max_count=10, name="member_as_limiter")
|
||||
|
|
@ -174,8 +176,6 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
|
|||
self.request_ratelimiter = hs.get_request_ratelimiter()
|
||||
hs.get_notifier().add_new_join_in_room_callback(self._on_user_joined_room)
|
||||
|
||||
self._msc3970_enabled = hs.config.experimental.msc3970_enabled
|
||||
|
||||
def _on_user_joined_room(self, event_id: str, room_id: str) -> None:
|
||||
"""Notify the rate limiter that a room join has occurred.
|
||||
|
||||
|
|
@ -416,29 +416,11 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
|
|||
# do this check just before we persist an event as well, but may as well
|
||||
# do it up front for efficiency.)
|
||||
if txn_id:
|
||||
existing_event_id = None
|
||||
if self._msc3970_enabled and requester.device_id:
|
||||
# When MSC3970 is enabled, we lookup for events sent by the same device
|
||||
# first, and fallback to the old behaviour if none were found.
|
||||
existing_event_id = (
|
||||
await self.store.get_event_id_from_transaction_id_and_device_id(
|
||||
room_id,
|
||||
requester.user.to_string(),
|
||||
requester.device_id,
|
||||
txn_id,
|
||||
)
|
||||
existing_event_id = (
|
||||
await self.event_creation_handler.get_event_id_from_transaction(
|
||||
requester, txn_id, room_id
|
||||
)
|
||||
|
||||
if requester.access_token_id and not existing_event_id:
|
||||
existing_event_id = (
|
||||
await self.store.get_event_id_from_transaction_id_and_token_id(
|
||||
room_id,
|
||||
requester.user.to_string(),
|
||||
requester.access_token_id,
|
||||
txn_id,
|
||||
)
|
||||
)
|
||||
|
||||
)
|
||||
if existing_event_id:
|
||||
event_pos = await self.store.get_position_for_event(existing_event_id)
|
||||
return existing_event_id, event_pos.stream
|
||||
|
|
@ -646,32 +628,35 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
|
|||
raise SynapseError(504, "took to long to process")
|
||||
|
||||
async with self.member_linearizer.queue(key):
|
||||
diff = self.clock.time_msec() - then
|
||||
async with self._worker_lock_handler.acquire_read_write_lock(
|
||||
DELETE_ROOM_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
diff = self.clock.time_msec() - then
|
||||
|
||||
if diff > 80 * 1000:
|
||||
# haproxy would have timed the request out anyway...
|
||||
raise SynapseError(504, "took to long to process")
|
||||
if diff > 80 * 1000:
|
||||
# haproxy would have timed the request out anyway...
|
||||
raise SynapseError(504, "took to long to process")
|
||||
|
||||
with opentracing.start_active_span("update_membership_locked"):
|
||||
result = await self.update_membership_locked(
|
||||
requester,
|
||||
target,
|
||||
room_id,
|
||||
action,
|
||||
txn_id=txn_id,
|
||||
remote_room_hosts=remote_room_hosts,
|
||||
third_party_signed=third_party_signed,
|
||||
ratelimit=ratelimit,
|
||||
content=content,
|
||||
new_room=new_room,
|
||||
require_consent=require_consent,
|
||||
outlier=outlier,
|
||||
allow_no_prev_events=allow_no_prev_events,
|
||||
prev_event_ids=prev_event_ids,
|
||||
state_event_ids=state_event_ids,
|
||||
depth=depth,
|
||||
origin_server_ts=origin_server_ts,
|
||||
)
|
||||
with opentracing.start_active_span("update_membership_locked"):
|
||||
result = await self.update_membership_locked(
|
||||
requester,
|
||||
target,
|
||||
room_id,
|
||||
action,
|
||||
txn_id=txn_id,
|
||||
remote_room_hosts=remote_room_hosts,
|
||||
third_party_signed=third_party_signed,
|
||||
ratelimit=ratelimit,
|
||||
content=content,
|
||||
new_room=new_room,
|
||||
require_consent=require_consent,
|
||||
outlier=outlier,
|
||||
allow_no_prev_events=allow_no_prev_events,
|
||||
prev_event_ids=prev_event_ids,
|
||||
state_event_ids=state_event_ids,
|
||||
depth=depth,
|
||||
origin_server_ts=origin_server_ts,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,333 @@
|
|||
# Copyright 2023 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import random
|
||||
from types import TracebackType
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
AsyncContextManager,
|
||||
Collection,
|
||||
Dict,
|
||||
Optional,
|
||||
Tuple,
|
||||
Type,
|
||||
Union,
|
||||
)
|
||||
from weakref import WeakSet
|
||||
|
||||
import attr
|
||||
|
||||
from twisted.internet import defer
|
||||
from twisted.internet.interfaces import IReactorTime
|
||||
|
||||
from synapse.logging.context import PreserveLoggingContext
|
||||
from synapse.logging.opentracing import start_active_span
|
||||
from synapse.metrics.background_process_metrics import wrap_as_background_process
|
||||
from synapse.storage.databases.main.lock import Lock, LockStore
|
||||
from synapse.util.async_helpers import timeout_deferred
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from synapse.logging.opentracing import opentracing
|
||||
from synapse.server import HomeServer
|
||||
|
||||
|
||||
DELETE_ROOM_LOCK_NAME = "delete_room_lock"
|
||||
|
||||
|
||||
class WorkerLocksHandler:
|
||||
"""A class for waiting on taking out locks, rather than using the storage
|
||||
functions directly (which don't support awaiting).
|
||||
"""
|
||||
|
||||
def __init__(self, hs: "HomeServer") -> None:
|
||||
self._reactor = hs.get_reactor()
|
||||
self._store = hs.get_datastores().main
|
||||
self._clock = hs.get_clock()
|
||||
self._notifier = hs.get_notifier()
|
||||
self._instance_name = hs.get_instance_name()
|
||||
|
||||
# Map from lock name/key to set of `WaitingLock` that are active for
|
||||
# that lock.
|
||||
self._locks: Dict[
|
||||
Tuple[str, str], WeakSet[Union[WaitingLock, WaitingMultiLock]]
|
||||
] = {}
|
||||
|
||||
self._clock.looping_call(self._cleanup_locks, 30_000)
|
||||
|
||||
self._notifier.add_lock_released_callback(self._on_lock_released)
|
||||
|
||||
def acquire_lock(self, lock_name: str, lock_key: str) -> "WaitingLock":
|
||||
"""Acquire a standard lock, returns a context manager that will block
|
||||
until the lock is acquired.
|
||||
|
||||
Note: Care must be taken to avoid deadlocks. In particular, this
|
||||
function does *not* timeout.
|
||||
|
||||
Usage:
|
||||
async with handler.acquire_lock(name, key):
|
||||
# Do work while holding the lock...
|
||||
"""
|
||||
|
||||
lock = WaitingLock(
|
||||
reactor=self._reactor,
|
||||
store=self._store,
|
||||
handler=self,
|
||||
lock_name=lock_name,
|
||||
lock_key=lock_key,
|
||||
write=None,
|
||||
)
|
||||
|
||||
self._locks.setdefault((lock_name, lock_key), WeakSet()).add(lock)
|
||||
|
||||
return lock
|
||||
|
||||
def acquire_read_write_lock(
|
||||
self,
|
||||
lock_name: str,
|
||||
lock_key: str,
|
||||
*,
|
||||
write: bool,
|
||||
) -> "WaitingLock":
|
||||
"""Acquire a read/write lock, returns a context manager that will block
|
||||
until the lock is acquired.
|
||||
|
||||
Note: Care must be taken to avoid deadlocks. In particular, this
|
||||
function does *not* timeout.
|
||||
|
||||
Usage:
|
||||
async with handler.acquire_read_write_lock(name, key, write=True):
|
||||
# Do work while holding the lock...
|
||||
"""
|
||||
|
||||
lock = WaitingLock(
|
||||
reactor=self._reactor,
|
||||
store=self._store,
|
||||
handler=self,
|
||||
lock_name=lock_name,
|
||||
lock_key=lock_key,
|
||||
write=write,
|
||||
)
|
||||
|
||||
self._locks.setdefault((lock_name, lock_key), WeakSet()).add(lock)
|
||||
|
||||
return lock
|
||||
|
||||
def acquire_multi_read_write_lock(
|
||||
self,
|
||||
lock_names: Collection[Tuple[str, str]],
|
||||
*,
|
||||
write: bool,
|
||||
) -> "WaitingMultiLock":
|
||||
"""Acquires multi read/write locks at once, returns a context manager
|
||||
that will block until all the locks are acquired.
|
||||
|
||||
This will try and acquire all locks at once, and will never hold on to a
|
||||
subset of the locks. (This avoids accidentally creating deadlocks).
|
||||
|
||||
Note: Care must be taken to avoid deadlocks. In particular, this
|
||||
function does *not* timeout.
|
||||
"""
|
||||
|
||||
lock = WaitingMultiLock(
|
||||
lock_names=lock_names,
|
||||
write=write,
|
||||
reactor=self._reactor,
|
||||
store=self._store,
|
||||
handler=self,
|
||||
)
|
||||
|
||||
for lock_name, lock_key in lock_names:
|
||||
self._locks.setdefault((lock_name, lock_key), WeakSet()).add(lock)
|
||||
|
||||
return lock
|
||||
|
||||
def notify_lock_released(self, lock_name: str, lock_key: str) -> None:
|
||||
"""Notify that a lock has been released.
|
||||
|
||||
Pokes both the notifier and replication.
|
||||
"""
|
||||
|
||||
self._notifier.notify_lock_released(self._instance_name, lock_name, lock_key)
|
||||
|
||||
def _on_lock_released(
|
||||
self, instance_name: str, lock_name: str, lock_key: str
|
||||
) -> None:
|
||||
"""Called when a lock has been released.
|
||||
|
||||
Wakes up any locks that might be waiting on this.
|
||||
"""
|
||||
locks = self._locks.get((lock_name, lock_key))
|
||||
if not locks:
|
||||
return
|
||||
|
||||
def _wake_deferred(deferred: defer.Deferred) -> None:
|
||||
if not deferred.called:
|
||||
deferred.callback(None)
|
||||
|
||||
for lock in locks:
|
||||
self._clock.call_later(0, _wake_deferred, lock.deferred)
|
||||
|
||||
@wrap_as_background_process("_cleanup_locks")
|
||||
async def _cleanup_locks(self) -> None:
|
||||
"""Periodically cleans out stale entries in the locks map"""
|
||||
self._locks = {key: value for key, value in self._locks.items() if value}
|
||||
|
||||
|
||||
@attr.s(auto_attribs=True, eq=False)
|
||||
class WaitingLock:
|
||||
reactor: IReactorTime
|
||||
store: LockStore
|
||||
handler: WorkerLocksHandler
|
||||
lock_name: str
|
||||
lock_key: str
|
||||
write: Optional[bool]
|
||||
deferred: "defer.Deferred[None]" = attr.Factory(defer.Deferred)
|
||||
_inner_lock: Optional[Lock] = None
|
||||
_retry_interval: float = 0.1
|
||||
_lock_span: "opentracing.Scope" = attr.Factory(
|
||||
lambda: start_active_span("WaitingLock.lock")
|
||||
)
|
||||
|
||||
async def __aenter__(self) -> None:
|
||||
self._lock_span.__enter__()
|
||||
|
||||
with start_active_span("WaitingLock.waiting_for_lock"):
|
||||
while self._inner_lock is None:
|
||||
self.deferred = defer.Deferred()
|
||||
|
||||
if self.write is not None:
|
||||
lock = await self.store.try_acquire_read_write_lock(
|
||||
self.lock_name, self.lock_key, write=self.write
|
||||
)
|
||||
else:
|
||||
lock = await self.store.try_acquire_lock(
|
||||
self.lock_name, self.lock_key
|
||||
)
|
||||
|
||||
if lock:
|
||||
self._inner_lock = lock
|
||||
break
|
||||
|
||||
try:
|
||||
# Wait until the we get notified the lock might have been
|
||||
# released (by the deferred being resolved). We also
|
||||
# periodically wake up in case the lock was released but we
|
||||
# weren't notified.
|
||||
with PreserveLoggingContext():
|
||||
await timeout_deferred(
|
||||
deferred=self.deferred,
|
||||
timeout=self._get_next_retry_interval(),
|
||||
reactor=self.reactor,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return await self._inner_lock.__aenter__()
|
||||
|
||||
async def __aexit__(
|
||||
self,
|
||||
exc_type: Optional[Type[BaseException]],
|
||||
exc: Optional[BaseException],
|
||||
tb: Optional[TracebackType],
|
||||
) -> Optional[bool]:
|
||||
assert self._inner_lock
|
||||
|
||||
self.handler.notify_lock_released(self.lock_name, self.lock_key)
|
||||
|
||||
try:
|
||||
r = await self._inner_lock.__aexit__(exc_type, exc, tb)
|
||||
finally:
|
||||
self._lock_span.__exit__(exc_type, exc, tb)
|
||||
|
||||
return r
|
||||
|
||||
def _get_next_retry_interval(self) -> float:
|
||||
next = self._retry_interval
|
||||
self._retry_interval = max(5, next * 2)
|
||||
return next * random.uniform(0.9, 1.1)
|
||||
|
||||
|
||||
@attr.s(auto_attribs=True, eq=False)
|
||||
class WaitingMultiLock:
|
||||
lock_names: Collection[Tuple[str, str]]
|
||||
|
||||
write: bool
|
||||
|
||||
reactor: IReactorTime
|
||||
store: LockStore
|
||||
handler: WorkerLocksHandler
|
||||
|
||||
deferred: "defer.Deferred[None]" = attr.Factory(defer.Deferred)
|
||||
|
||||
_inner_lock_cm: Optional[AsyncContextManager] = None
|
||||
_retry_interval: float = 0.1
|
||||
_lock_span: "opentracing.Scope" = attr.Factory(
|
||||
lambda: start_active_span("WaitingLock.lock")
|
||||
)
|
||||
|
||||
async def __aenter__(self) -> None:
|
||||
self._lock_span.__enter__()
|
||||
|
||||
with start_active_span("WaitingLock.waiting_for_lock"):
|
||||
while self._inner_lock_cm is None:
|
||||
self.deferred = defer.Deferred()
|
||||
|
||||
lock_cm = await self.store.try_acquire_multi_read_write_lock(
|
||||
self.lock_names, write=self.write
|
||||
)
|
||||
|
||||
if lock_cm:
|
||||
self._inner_lock_cm = lock_cm
|
||||
break
|
||||
|
||||
try:
|
||||
# Wait until the we get notified the lock might have been
|
||||
# released (by the deferred being resolved). We also
|
||||
# periodically wake up in case the lock was released but we
|
||||
# weren't notified.
|
||||
with PreserveLoggingContext():
|
||||
await timeout_deferred(
|
||||
deferred=self.deferred,
|
||||
timeout=self._get_next_retry_interval(),
|
||||
reactor=self.reactor,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
assert self._inner_lock_cm
|
||||
await self._inner_lock_cm.__aenter__()
|
||||
return
|
||||
|
||||
async def __aexit__(
|
||||
self,
|
||||
exc_type: Optional[Type[BaseException]],
|
||||
exc: Optional[BaseException],
|
||||
tb: Optional[TracebackType],
|
||||
) -> Optional[bool]:
|
||||
assert self._inner_lock_cm
|
||||
|
||||
for lock_name, lock_key in self.lock_names:
|
||||
self.handler.notify_lock_released(lock_name, lock_key)
|
||||
|
||||
try:
|
||||
r = await self._inner_lock_cm.__aexit__(exc_type, exc, tb)
|
||||
finally:
|
||||
self._lock_span.__exit__(exc_type, exc, tb)
|
||||
|
||||
return r
|
||||
|
||||
def _get_next_retry_interval(self) -> float:
|
||||
next = self._retry_interval
|
||||
self._retry_interval = max(5, next * 2)
|
||||
return next * random.uniform(0.9, 1.1)
|
||||
|
|
@ -34,6 +34,7 @@ import jinja2
|
|||
from typing_extensions import ParamSpec
|
||||
|
||||
from twisted.internet import defer
|
||||
from twisted.internet.interfaces import IDelayedCall
|
||||
from twisted.web.resource import Resource
|
||||
|
||||
from synapse.api import errors
|
||||
|
|
@ -1230,6 +1231,58 @@ class ModuleApi:
|
|||
f,
|
||||
)
|
||||
|
||||
def should_run_background_tasks(self) -> bool:
|
||||
"""
|
||||
Return true if and only if the current worker is configured to run
|
||||
background tasks.
|
||||
There should only be one worker configured to run background tasks, so
|
||||
this is helpful when you need to only run a task on one worker but don't
|
||||
have any other good way to choose which one.
|
||||
|
||||
Added in Synapse v1.89.0.
|
||||
"""
|
||||
return self._hs.config.worker.run_background_tasks
|
||||
|
||||
def delayed_background_call(
|
||||
self,
|
||||
msec: float,
|
||||
f: Callable,
|
||||
*args: object,
|
||||
desc: Optional[str] = None,
|
||||
**kwargs: object,
|
||||
) -> IDelayedCall:
|
||||
"""Wraps a function as a background process and calls it in a given number of milliseconds.
|
||||
|
||||
The scheduled call is not persistent: if the current Synapse instance is
|
||||
restarted before the call is made, the call will not be made.
|
||||
|
||||
Added in Synapse v1.90.0.
|
||||
|
||||
Args:
|
||||
msec: How long to wait before calling, in milliseconds.
|
||||
f: The function to call once. f can be either synchronous or
|
||||
asynchronous, and must follow Synapse's logcontext rules.
|
||||
More info about logcontexts is available at
|
||||
https://matrix-org.github.io/synapse/latest/log_contexts.html
|
||||
*args: Positional arguments to pass to function.
|
||||
desc: The background task's description. Default to the function's name.
|
||||
**kwargs: Keyword arguments to pass to function.
|
||||
|
||||
Returns:
|
||||
IDelayedCall handle from twisted, which allows to cancel the delayed call if desired.
|
||||
"""
|
||||
|
||||
if desc is None:
|
||||
desc = f.__name__
|
||||
|
||||
return self._clock.call_later(
|
||||
# convert ms to seconds as needed by call_later.
|
||||
msec * 0.001,
|
||||
run_as_background_process,
|
||||
desc,
|
||||
lambda: maybe_awaitable(f(*args, **kwargs)),
|
||||
)
|
||||
|
||||
async def sleep(self, seconds: float) -> None:
|
||||
"""Sleeps for the given number of seconds.
|
||||
|
||||
|
|
|
|||
|
|
@ -234,6 +234,9 @@ class Notifier:
|
|||
|
||||
self._third_party_rules = hs.get_module_api_callbacks().third_party_event_rules
|
||||
|
||||
# List of callbacks to be notified when a lock is released
|
||||
self._lock_released_callback: List[Callable[[str, str, str], None]] = []
|
||||
|
||||
self.clock = hs.get_clock()
|
||||
self.appservice_handler = hs.get_application_service_handler()
|
||||
self._pusher_pool = hs.get_pusherpool()
|
||||
|
|
@ -785,6 +788,19 @@ class Notifier:
|
|||
# that any in flight requests can be immediately retried.
|
||||
self._federation_client.wake_destination(server)
|
||||
|
||||
def add_lock_released_callback(
|
||||
self, callback: Callable[[str, str, str], None]
|
||||
) -> None:
|
||||
"""Add a function to be called whenever we are notified about a released lock."""
|
||||
self._lock_released_callback.append(callback)
|
||||
|
||||
def notify_lock_released(
|
||||
self, instance_name: str, lock_name: str, lock_key: str
|
||||
) -> None:
|
||||
"""Notify the callbacks that a lock has been released."""
|
||||
for cb in self._lock_released_callback:
|
||||
cb(instance_name, lock_name, lock_key)
|
||||
|
||||
|
||||
@attr.s(auto_attribs=True)
|
||||
class ReplicationNotifier:
|
||||
|
|
|
|||
|
|
@ -107,8 +107,7 @@ class ReplicationUploadKeysForUserRestServlet(ReplicationEndpoint):
|
|||
Calls to e2e_keys_handler.upload_keys_for_user(user_id, device_id, keys) on
|
||||
the main process to accomplish this.
|
||||
|
||||
Defined in https://spec.matrix.org/v1.4/client-server-api/#post_matrixclientv3keysupload
|
||||
Request format(borrowed and expanded from KeyUploadServlet):
|
||||
Request format for this endpoint (borrowed and expanded from KeyUploadServlet):
|
||||
|
||||
POST /_synapse/replication/upload_keys_for_user
|
||||
|
||||
|
|
@ -117,6 +116,7 @@ class ReplicationUploadKeysForUserRestServlet(ReplicationEndpoint):
|
|||
"device_id": "<device_id>",
|
||||
"keys": {
|
||||
....this part can be found in KeyUploadServlet in rest/client/keys.py....
|
||||
or as defined in https://spec.matrix.org/v1.4/client-server-api/#post_matrixclientv3keysupload
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -422,6 +422,36 @@ class RemoteServerUpCommand(_SimpleCommand):
|
|||
NAME = "REMOTE_SERVER_UP"
|
||||
|
||||
|
||||
class LockReleasedCommand(Command):
|
||||
"""Sent to inform other instances that a given lock has been dropped.
|
||||
|
||||
Format::
|
||||
|
||||
LOCK_RELEASED ["<instance_name>", "<lock_name>", "<lock_key>"]
|
||||
"""
|
||||
|
||||
NAME = "LOCK_RELEASED"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
instance_name: str,
|
||||
lock_name: str,
|
||||
lock_key: str,
|
||||
):
|
||||
self.instance_name = instance_name
|
||||
self.lock_name = lock_name
|
||||
self.lock_key = lock_key
|
||||
|
||||
@classmethod
|
||||
def from_line(cls: Type["LockReleasedCommand"], line: str) -> "LockReleasedCommand":
|
||||
instance_name, lock_name, lock_key = json_decoder.decode(line)
|
||||
|
||||
return cls(instance_name, lock_name, lock_key)
|
||||
|
||||
def to_line(self) -> str:
|
||||
return json_encoder.encode([self.instance_name, self.lock_name, self.lock_key])
|
||||
|
||||
|
||||
_COMMANDS: Tuple[Type[Command], ...] = (
|
||||
ServerCommand,
|
||||
RdataCommand,
|
||||
|
|
@ -435,6 +465,7 @@ _COMMANDS: Tuple[Type[Command], ...] = (
|
|||
UserIpCommand,
|
||||
RemoteServerUpCommand,
|
||||
ClearUserSyncsCommand,
|
||||
LockReleasedCommand,
|
||||
)
|
||||
|
||||
# Map of command name to command type.
|
||||
|
|
@ -448,6 +479,7 @@ VALID_SERVER_COMMANDS = (
|
|||
ErrorCommand.NAME,
|
||||
PingCommand.NAME,
|
||||
RemoteServerUpCommand.NAME,
|
||||
LockReleasedCommand.NAME,
|
||||
)
|
||||
|
||||
# The commands the client is allowed to send
|
||||
|
|
@ -461,6 +493,7 @@ VALID_CLIENT_COMMANDS = (
|
|||
UserIpCommand.NAME,
|
||||
ErrorCommand.NAME,
|
||||
RemoteServerUpCommand.NAME,
|
||||
LockReleasedCommand.NAME,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ from synapse.replication.tcp.commands import (
|
|||
ClearUserSyncsCommand,
|
||||
Command,
|
||||
FederationAckCommand,
|
||||
LockReleasedCommand,
|
||||
PositionCommand,
|
||||
RdataCommand,
|
||||
RemoteServerUpCommand,
|
||||
|
|
@ -248,6 +249,9 @@ class ReplicationCommandHandler:
|
|||
if self._is_master or self._should_insert_client_ips:
|
||||
self.subscribe_to_channel("USER_IP")
|
||||
|
||||
if hs.config.redis.redis_enabled:
|
||||
self._notifier.add_lock_released_callback(self.on_lock_released)
|
||||
|
||||
def subscribe_to_channel(self, channel_name: str) -> None:
|
||||
"""
|
||||
Indicates that we wish to subscribe to a Redis channel by name.
|
||||
|
|
@ -648,6 +652,17 @@ class ReplicationCommandHandler:
|
|||
|
||||
self._notifier.notify_remote_server_up(cmd.data)
|
||||
|
||||
def on_LOCK_RELEASED(
|
||||
self, conn: IReplicationConnection, cmd: LockReleasedCommand
|
||||
) -> None:
|
||||
"""Called when we get a new LOCK_RELEASED command."""
|
||||
if cmd.instance_name == self._instance_name:
|
||||
return
|
||||
|
||||
self._notifier.notify_lock_released(
|
||||
cmd.instance_name, cmd.lock_name, cmd.lock_key
|
||||
)
|
||||
|
||||
def new_connection(self, connection: IReplicationConnection) -> None:
|
||||
"""Called when we have a new connection."""
|
||||
self._connections.append(connection)
|
||||
|
|
@ -754,6 +769,13 @@ class ReplicationCommandHandler:
|
|||
"""
|
||||
self.send_command(RdataCommand(stream_name, self._instance_name, token, data))
|
||||
|
||||
def on_lock_released(
|
||||
self, instance_name: str, lock_name: str, lock_key: str
|
||||
) -> None:
|
||||
"""Called when we released a lock and should notify other instances."""
|
||||
if instance_name == self._instance_name:
|
||||
self.send_command(LockReleasedCommand(instance_name, lock_name, lock_key))
|
||||
|
||||
|
||||
UpdateToken = TypeVar("UpdateToken")
|
||||
UpdateRow = TypeVar("UpdateRow")
|
||||
|
|
|
|||
|
|
@ -232,7 +232,7 @@ class DehydratedDeviceDataModel(RequestBodyModel):
|
|||
class DehydratedDeviceServlet(RestServlet):
|
||||
"""Retrieve or store a dehydrated device.
|
||||
|
||||
Implements either MSC2697 or MSC3814.
|
||||
Implements MSC2697.
|
||||
|
||||
GET /org.matrix.msc2697.v2/dehydrated_device
|
||||
|
||||
|
|
@ -266,7 +266,12 @@ class DehydratedDeviceServlet(RestServlet):
|
|||
|
||||
"""
|
||||
|
||||
def __init__(self, hs: "HomeServer", msc2697: bool = True):
|
||||
PATTERNS = client_patterns(
|
||||
"/org.matrix.msc2697.v2/dehydrated_device$",
|
||||
releases=(),
|
||||
)
|
||||
|
||||
def __init__(self, hs: "HomeServer"):
|
||||
super().__init__()
|
||||
self.hs = hs
|
||||
self.auth = hs.get_auth()
|
||||
|
|
@ -274,13 +279,6 @@ class DehydratedDeviceServlet(RestServlet):
|
|||
assert isinstance(handler, DeviceHandler)
|
||||
self.device_handler = handler
|
||||
|
||||
self.PATTERNS = client_patterns(
|
||||
"/org.matrix.msc2697.v2/dehydrated_device$"
|
||||
if msc2697
|
||||
else "/org.matrix.msc3814.v1/dehydrated_device$",
|
||||
releases=(),
|
||||
)
|
||||
|
||||
async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
|
||||
requester = await self.auth.get_user_by_req(request)
|
||||
dehydrated_device = await self.device_handler.get_dehydrated_device(
|
||||
|
|
@ -513,10 +511,8 @@ class DehydratedDeviceV2Servlet(RestServlet):
|
|||
if dehydrated_device is not None:
|
||||
(device_id, device_data) = dehydrated_device
|
||||
|
||||
result = await self.device_handler.rehydrate_device(
|
||||
requester.user.to_string(),
|
||||
self.auth.get_access_token_from_request(request),
|
||||
device_id,
|
||||
await self.device_handler.delete_dehydrated_device(
|
||||
requester.user.to_string(), device_id
|
||||
)
|
||||
|
||||
result = {"device_id": device_id}
|
||||
|
|
@ -538,6 +534,14 @@ class DehydratedDeviceV2Servlet(RestServlet):
|
|||
requester = await self.auth.get_user_by_req(request)
|
||||
user_id = requester.user.to_string()
|
||||
|
||||
old_dehydrated_device = await self.device_handler.get_dehydrated_device(user_id)
|
||||
|
||||
# if an old device exists, delete it before creating a new one
|
||||
if old_dehydrated_device:
|
||||
await self.device_handler.delete_dehydrated_device(
|
||||
user_id, old_dehydrated_device[0]
|
||||
)
|
||||
|
||||
device_info = submission.dict()
|
||||
if "device_keys" not in device_info.keys():
|
||||
raise SynapseError(
|
||||
|
|
@ -573,7 +577,7 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
|
|||
if hs.config.worker.worker_app is None:
|
||||
DeviceRestServlet(hs).register(http_server)
|
||||
if hs.config.experimental.msc2697_enabled:
|
||||
DehydratedDeviceServlet(hs, msc2697=True).register(http_server)
|
||||
DehydratedDeviceServlet(hs).register(http_server)
|
||||
ClaimDehydratedDeviceServlet(hs).register(http_server)
|
||||
if hs.config.experimental.msc3814_enabled:
|
||||
DehydratedDeviceV2Servlet(hs).register(http_server)
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from typing import TYPE_CHECKING, Tuple
|
|||
|
||||
from synapse.api.errors import Codes, ShadowBanError, SynapseError
|
||||
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
|
||||
from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
|
||||
from synapse.http.server import HttpServer
|
||||
from synapse.http.servlet import (
|
||||
RestServlet,
|
||||
|
|
@ -60,6 +61,7 @@ class RoomUpgradeRestServlet(RestServlet):
|
|||
self._hs = hs
|
||||
self._room_creation_handler = hs.get_room_creation_handler()
|
||||
self._auth = hs.get_auth()
|
||||
self._worker_lock_handler = hs.get_worker_locks_handler()
|
||||
|
||||
async def on_POST(
|
||||
self, request: SynapseRequest, room_id: str
|
||||
|
|
@ -78,9 +80,12 @@ class RoomUpgradeRestServlet(RestServlet):
|
|||
)
|
||||
|
||||
try:
|
||||
new_room_id = await self._room_creation_handler.upgrade_room(
|
||||
requester, room_id, new_version
|
||||
)
|
||||
async with self._worker_lock_handler.acquire_read_write_lock(
|
||||
DELETE_ROOM_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
new_room_id = await self._room_creation_handler.upgrade_room(
|
||||
requester, room_id, new_version
|
||||
)
|
||||
except ShadowBanError:
|
||||
# Generate a random room ID.
|
||||
new_room_id = stringutils.random_string(18)
|
||||
|
|
|
|||
|
|
@ -50,8 +50,6 @@ class HttpTransactionCache:
|
|||
# for at *LEAST* 30 mins, and at *MOST* 60 mins.
|
||||
self.cleaner = self.clock.looping_call(self._cleanup, CLEANUP_PERIOD_MS)
|
||||
|
||||
self._msc3970_enabled = hs.config.experimental.msc3970_enabled
|
||||
|
||||
def _get_transaction_key(self, request: IRequest, requester: Requester) -> Hashable:
|
||||
"""A helper function which returns a transaction key that can be used
|
||||
with TransactionCache for idempotent requests.
|
||||
|
|
@ -78,18 +76,20 @@ class HttpTransactionCache:
|
|||
elif requester.app_service is not None:
|
||||
return (path, "appservice", requester.app_service.id)
|
||||
|
||||
# With MSC3970, we use the user ID and device ID as the transaction key
|
||||
elif self._msc3970_enabled:
|
||||
# Use the user ID and device ID as the transaction key.
|
||||
elif requester.device_id:
|
||||
assert requester.user, "Requester must have a user"
|
||||
assert requester.device_id, "Requester must have a device_id"
|
||||
return (path, "user", requester.user, requester.device_id)
|
||||
|
||||
# Otherwise, the pre-MSC3970 behaviour is to use the access token ID
|
||||
# Some requsters don't have device IDs, these are mostly handled above
|
||||
# (appservice and guest users), but does not cover access tokens minted
|
||||
# by the admin API. Use the access token ID instead.
|
||||
else:
|
||||
assert (
|
||||
requester.access_token_id is not None
|
||||
), "Requester must have an access_token_id"
|
||||
return (path, "user", requester.access_token_id)
|
||||
return (path, "user_admin", requester.access_token_id)
|
||||
|
||||
def fetch_or_execute_request(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -107,6 +107,7 @@ from synapse.handlers.stats import StatsHandler
|
|||
from synapse.handlers.sync import SyncHandler
|
||||
from synapse.handlers.typing import FollowerTypingHandler, TypingWriterHandler
|
||||
from synapse.handlers.user_directory import UserDirectoryHandler
|
||||
from synapse.handlers.worker_lock import WorkerLocksHandler
|
||||
from synapse.http.client import (
|
||||
InsecureInterceptableContextFactory,
|
||||
ReplicationClient,
|
||||
|
|
@ -784,9 +785,7 @@ class HomeServer(metaclass=abc.ABCMeta):
|
|||
|
||||
@cache_in_self
|
||||
def get_event_client_serializer(self) -> EventClientSerializer:
|
||||
return EventClientSerializer(
|
||||
msc3970_enabled=self.config.experimental.msc3970_enabled
|
||||
)
|
||||
return EventClientSerializer()
|
||||
|
||||
@cache_in_self
|
||||
def get_password_policy_handler(self) -> PasswordPolicyHandler:
|
||||
|
|
@ -912,3 +911,7 @@ class HomeServer(metaclass=abc.ABCMeta):
|
|||
def get_common_usage_metrics_manager(self) -> CommonUsageMetricsManager:
|
||||
"""Usage metrics shared between phone home stats and the prometheus exporter."""
|
||||
return CommonUsageMetricsManager(self)
|
||||
|
||||
@cache_in_self
|
||||
def get_worker_locks_handler(self) -> WorkerLocksHandler:
|
||||
return WorkerLocksHandler(self)
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ from twisted.internet import defer
|
|||
from synapse.api.constants import EventTypes, Membership
|
||||
from synapse.events import EventBase
|
||||
from synapse.events.snapshot import EventContext
|
||||
from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
|
||||
from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
|
||||
from synapse.logging.opentracing import (
|
||||
SynapseTags,
|
||||
|
|
@ -338,6 +339,7 @@ class EventsPersistenceStorageController:
|
|||
)
|
||||
self._state_resolution_handler = hs.get_state_resolution_handler()
|
||||
self._state_controller = state_controller
|
||||
self.hs = hs
|
||||
|
||||
async def _process_event_persist_queue_task(
|
||||
self,
|
||||
|
|
@ -350,15 +352,22 @@ class EventsPersistenceStorageController:
|
|||
A dictionary of event ID to event ID we didn't persist as we already
|
||||
had another event persisted with the same TXN ID.
|
||||
"""
|
||||
if isinstance(task, _PersistEventsTask):
|
||||
return await self._persist_event_batch(room_id, task)
|
||||
elif isinstance(task, _UpdateCurrentStateTask):
|
||||
await self._update_current_state(room_id, task)
|
||||
return {}
|
||||
else:
|
||||
raise AssertionError(
|
||||
f"Found an unexpected task type in event persistence queue: {task}"
|
||||
)
|
||||
|
||||
# Ensure that the room can't be deleted while we're persisting events to
|
||||
# it. We might already have taken out the lock, but since this is just a
|
||||
# "read" lock its inherently reentrant.
|
||||
async with self.hs.get_worker_locks_handler().acquire_read_write_lock(
|
||||
DELETE_ROOM_LOCK_NAME, room_id, write=False
|
||||
):
|
||||
if isinstance(task, _PersistEventsTask):
|
||||
return await self._persist_event_batch(room_id, task)
|
||||
elif isinstance(task, _UpdateCurrentStateTask):
|
||||
await self._update_current_state(room_id, task)
|
||||
return {}
|
||||
else:
|
||||
raise AssertionError(
|
||||
f"Found an unexpected task type in event persistence queue: {task}"
|
||||
)
|
||||
|
||||
@trace
|
||||
async def persist_events(
|
||||
|
|
|
|||
|
|
@ -18,6 +18,8 @@ import logging
|
|||
from typing import TYPE_CHECKING, Any, Collection, Iterable, List, Optional, Tuple
|
||||
|
||||
from synapse.api.constants import EventTypes
|
||||
from synapse.config._base import Config
|
||||
from synapse.metrics.background_process_metrics import wrap_as_background_process
|
||||
from synapse.replication.tcp.streams import BackfillStream, CachesStream
|
||||
from synapse.replication.tcp.streams.events import (
|
||||
EventsStream,
|
||||
|
|
@ -52,6 +54,21 @@ PURGE_HISTORY_CACHE_NAME = "ph_cache_fake"
|
|||
# As above, but for invalidating room caches on room deletion
|
||||
DELETE_ROOM_CACHE_NAME = "dr_cache_fake"
|
||||
|
||||
# How long between cache invalidation table cleanups, once we have caught up
|
||||
# with the backlog.
|
||||
REGULAR_CLEANUP_INTERVAL_MS = Config.parse_duration("1h")
|
||||
|
||||
# How long between cache invalidation table cleanups, before we have caught
|
||||
# up with the backlog.
|
||||
CATCH_UP_CLEANUP_INTERVAL_MS = Config.parse_duration("1m")
|
||||
|
||||
# Maximum number of cache invalidation rows to delete at once.
|
||||
CLEAN_UP_MAX_BATCH_SIZE = 20_000
|
||||
|
||||
# Keep cache invalidations for 7 days
|
||||
# (This is likely to be quite excessive.)
|
||||
RETENTION_PERIOD_OF_CACHE_INVALIDATIONS_MS = Config.parse_duration("7d")
|
||||
|
||||
|
||||
class CacheInvalidationWorkerStore(SQLBaseStore):
|
||||
def __init__(
|
||||
|
|
@ -98,6 +115,18 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
|
|||
else:
|
||||
self._cache_id_gen = None
|
||||
|
||||
# Occasionally clean up the cache invalidations stream table by deleting
|
||||
# old rows.
|
||||
# This is only applicable when Postgres is in use; this table is unused
|
||||
# and not populated at all when SQLite is the active database engine.
|
||||
if hs.config.worker.run_background_tasks and isinstance(
|
||||
self.database_engine, PostgresEngine
|
||||
):
|
||||
self.hs.get_clock().call_later(
|
||||
CATCH_UP_CLEANUP_INTERVAL_MS / 1000,
|
||||
self._clean_up_cache_invalidation_wrapper,
|
||||
)
|
||||
|
||||
async def get_all_updated_caches(
|
||||
self, instance_name: str, last_id: int, current_id: int, limit: int
|
||||
) -> Tuple[List[Tuple[int, tuple]], int, bool]:
|
||||
|
|
@ -554,3 +583,104 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
|
|||
return self._cache_id_gen.get_current_token_for_writer(instance_name)
|
||||
else:
|
||||
return 0
|
||||
|
||||
@wrap_as_background_process("clean_up_old_cache_invalidations")
|
||||
async def _clean_up_cache_invalidation_wrapper(self) -> None:
|
||||
"""
|
||||
Clean up cache invalidation stream table entries occasionally.
|
||||
If we are behind (i.e. there are entries old enough to
|
||||
be deleted but too many of them to be deleted in one go),
|
||||
then we run slightly more frequently.
|
||||
"""
|
||||
delete_up_to: int = (
|
||||
self.hs.get_clock().time_msec() - RETENTION_PERIOD_OF_CACHE_INVALIDATIONS_MS
|
||||
)
|
||||
|
||||
in_backlog = await self._clean_up_batch_of_old_cache_invalidations(delete_up_to)
|
||||
|
||||
# Vary how long we wait before calling again depending on whether we
|
||||
# are still sifting through backlog or we have caught up.
|
||||
if in_backlog:
|
||||
next_interval = CATCH_UP_CLEANUP_INTERVAL_MS
|
||||
else:
|
||||
next_interval = REGULAR_CLEANUP_INTERVAL_MS
|
||||
|
||||
self.hs.get_clock().call_later(
|
||||
next_interval / 1000, self._clean_up_cache_invalidation_wrapper
|
||||
)
|
||||
|
||||
async def _clean_up_batch_of_old_cache_invalidations(
|
||||
self, delete_up_to_millisec: int
|
||||
) -> bool:
|
||||
"""
|
||||
Remove old rows from the `cache_invalidation_stream_by_instance` table automatically (this table is unused in SQLite).
|
||||
|
||||
Up to `CLEAN_UP_BATCH_SIZE` rows will be deleted at once.
|
||||
|
||||
Returns true if and only if we were limited by batch size (i.e. we are in backlog:
|
||||
there are more things to clean up).
|
||||
"""
|
||||
|
||||
def _clean_up_batch_of_old_cache_invalidations_txn(
|
||||
txn: LoggingTransaction,
|
||||
) -> bool:
|
||||
# First get the earliest stream ID
|
||||
txn.execute(
|
||||
"""
|
||||
SELECT stream_id FROM cache_invalidation_stream_by_instance
|
||||
ORDER BY stream_id ASC
|
||||
LIMIT 1
|
||||
"""
|
||||
)
|
||||
row = txn.fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
earliest_stream_id: int = row[0]
|
||||
|
||||
# Then find the last stream ID of the range we will delete
|
||||
txn.execute(
|
||||
"""
|
||||
SELECT stream_id FROM cache_invalidation_stream_by_instance
|
||||
WHERE stream_id <= ? AND invalidation_ts <= ?
|
||||
ORDER BY stream_id DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
(earliest_stream_id + CLEAN_UP_MAX_BATCH_SIZE, delete_up_to_millisec),
|
||||
)
|
||||
row = txn.fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
cutoff_stream_id: int = row[0]
|
||||
|
||||
# Determine whether we are caught up or still catching up
|
||||
txn.execute(
|
||||
"""
|
||||
SELECT invalidation_ts FROM cache_invalidation_stream_by_instance
|
||||
WHERE stream_id > ?
|
||||
ORDER BY stream_id ASC
|
||||
LIMIT 1
|
||||
""",
|
||||
(cutoff_stream_id,),
|
||||
)
|
||||
row = txn.fetchone()
|
||||
if row is None:
|
||||
in_backlog = False
|
||||
else:
|
||||
# We are in backlog if the next row could have been deleted
|
||||
# if we didn't have such a small batch size
|
||||
in_backlog = row[0] <= delete_up_to_millisec
|
||||
|
||||
txn.execute(
|
||||
"""
|
||||
DELETE FROM cache_invalidation_stream_by_instance
|
||||
WHERE ? <= stream_id AND stream_id <= ?
|
||||
""",
|
||||
(earliest_stream_id, cutoff_stream_id),
|
||||
)
|
||||
|
||||
return in_backlog
|
||||
|
||||
return await self.db_pool.runInteraction(
|
||||
"clean_up_old_cache_invalidations",
|
||||
_clean_up_batch_of_old_cache_invalidations_txn,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -843,7 +843,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
* because the schema change is in a background update, it's not
|
||||
* necessarily safe to assume that it will have been completed.
|
||||
*/
|
||||
AND edge.is_state is ? /* False */
|
||||
AND edge.is_state is FALSE
|
||||
/**
|
||||
* We only want backwards extremities that are older than or at
|
||||
* the same position of the given `current_depth` (where older
|
||||
|
|
@ -886,7 +886,6 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
|
|||
sql,
|
||||
(
|
||||
room_id,
|
||||
False,
|
||||
current_depth,
|
||||
self._clock.time_msec(),
|
||||
BACKFILL_EVENT_EXPONENTIAL_BACKOFF_MAXIMUM_DOUBLING_STEPS,
|
||||
|
|
|
|||
|
|
@ -127,8 +127,6 @@ class PersistEventsStore:
|
|||
self._backfill_id_gen: AbstractStreamIdGenerator = self.store._backfill_id_gen
|
||||
self._stream_id_gen: AbstractStreamIdGenerator = self.store._stream_id_gen
|
||||
|
||||
self._msc3970_enabled = hs.config.experimental.msc3970_enabled
|
||||
|
||||
@trace
|
||||
async def _persist_events_and_state_updates(
|
||||
self,
|
||||
|
|
@ -1012,9 +1010,11 @@ class PersistEventsStore:
|
|||
)
|
||||
)
|
||||
|
||||
# Pre-MSC3970, we rely on the access_token_id to scope the txn_id for events.
|
||||
# Since this is an experimental flag, we still store the mapping even if the
|
||||
# flag is disabled.
|
||||
# Synapse usually relies on the device_id to scope transactions for events,
|
||||
# except for users without device IDs (appservice, guests, and access
|
||||
# tokens minted with the admin API) which use the access token ID instead.
|
||||
#
|
||||
# TODO https://github.com/matrix-org/synapse/issues/16042
|
||||
if to_insert_token_id:
|
||||
self.db_pool.simple_insert_many_txn(
|
||||
txn,
|
||||
|
|
@ -1030,10 +1030,7 @@ class PersistEventsStore:
|
|||
values=to_insert_token_id,
|
||||
)
|
||||
|
||||
# With MSC3970, we rely on the device_id instead to scope the txn_id for events.
|
||||
# We're only inserting if MSC3970 is *enabled*, because else the pre-MSC3970
|
||||
# behaviour would allow for a UNIQUE constraint violation on this table
|
||||
if to_insert_device_id and self._msc3970_enabled:
|
||||
if to_insert_device_id:
|
||||
self.db_pool.simple_insert_many_txn(
|
||||
txn,
|
||||
table="event_txn_id_device_id",
|
||||
|
|
@ -1455,8 +1452,8 @@ class PersistEventsStore:
|
|||
},
|
||||
)
|
||||
|
||||
sql = "UPDATE events SET outlier = ? WHERE event_id = ?"
|
||||
txn.execute(sql, (False, event.event_id))
|
||||
sql = "UPDATE events SET outlier = FALSE WHERE event_id = ?"
|
||||
txn.execute(sql, (event.event_id,))
|
||||
|
||||
# Update the event_backward_extremities table now that this
|
||||
# event isn't an outlier any more.
|
||||
|
|
@ -1549,13 +1546,13 @@ class PersistEventsStore:
|
|||
for event, _ in events_and_contexts
|
||||
if not event.internal_metadata.is_redacted()
|
||||
]
|
||||
sql = "UPDATE redactions SET have_censored = ? WHERE "
|
||||
sql = "UPDATE redactions SET have_censored = FALSE WHERE "
|
||||
clause, args = make_in_list_sql_clause(
|
||||
self.database_engine,
|
||||
"redacts",
|
||||
unredacted_events,
|
||||
)
|
||||
txn.execute(sql + clause, [False] + args)
|
||||
txn.execute(sql + clause, args)
|
||||
|
||||
self.db_pool.simple_insert_many_txn(
|
||||
txn,
|
||||
|
|
@ -2318,14 +2315,14 @@ class PersistEventsStore:
|
|||
" SELECT 1 FROM events"
|
||||
" LEFT JOIN event_edges edge"
|
||||
" ON edge.event_id = events.event_id"
|
||||
" WHERE events.event_id = ? AND events.room_id = ? AND (events.outlier = ? OR edge.event_id IS NULL)"
|
||||
" WHERE events.event_id = ? AND events.room_id = ? AND (events.outlier = FALSE OR edge.event_id IS NULL)"
|
||||
" )"
|
||||
)
|
||||
|
||||
txn.execute_batch(
|
||||
query,
|
||||
[
|
||||
(e_id, ev.room_id, e_id, ev.room_id, e_id, ev.room_id, False)
|
||||
(e_id, ev.room_id, e_id, ev.room_id, e_id, ev.room_id)
|
||||
for ev in events
|
||||
for e_id in ev.prev_event_ids()
|
||||
if not ev.internal_metadata.is_outlier()
|
||||
|
|
|
|||
|
|
@ -12,8 +12,9 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import logging
|
||||
from contextlib import AsyncExitStack
|
||||
from types import TracebackType
|
||||
from typing import TYPE_CHECKING, Optional, Set, Tuple, Type
|
||||
from typing import TYPE_CHECKING, Collection, Optional, Set, Tuple, Type
|
||||
from weakref import WeakValueDictionary
|
||||
|
||||
from twisted.internet.interfaces import IReactorCore
|
||||
|
|
@ -208,77 +209,86 @@ class LockStore(SQLBaseStore):
|
|||
used (otherwise the lock will leak).
|
||||
"""
|
||||
|
||||
now = self._clock.time_msec()
|
||||
token = random_string(6)
|
||||
|
||||
def _try_acquire_read_write_lock_txn(txn: LoggingTransaction) -> None:
|
||||
# We attempt to acquire the lock by inserting into
|
||||
# `worker_read_write_locks` and seeing if that fails any
|
||||
# constraints. If it doesn't then we have acquired the lock,
|
||||
# otherwise we haven't.
|
||||
#
|
||||
# Before that though we clear the table of any stale locks.
|
||||
|
||||
delete_sql = """
|
||||
DELETE FROM worker_read_write_locks
|
||||
WHERE last_renewed_ts < ? AND lock_name = ? AND lock_key = ?;
|
||||
"""
|
||||
|
||||
insert_sql = """
|
||||
INSERT INTO worker_read_write_locks (lock_name, lock_key, write_lock, instance_name, token, last_renewed_ts)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
"""
|
||||
|
||||
if isinstance(self.database_engine, PostgresEngine):
|
||||
# For Postgres we can send these queries at the same time.
|
||||
txn.execute(
|
||||
delete_sql + ";" + insert_sql,
|
||||
(
|
||||
# DELETE args
|
||||
now - _LOCK_TIMEOUT_MS,
|
||||
lock_name,
|
||||
lock_key,
|
||||
# UPSERT args
|
||||
lock_name,
|
||||
lock_key,
|
||||
write,
|
||||
self._instance_name,
|
||||
token,
|
||||
now,
|
||||
),
|
||||
)
|
||||
else:
|
||||
# For SQLite these need to be two queries.
|
||||
txn.execute(
|
||||
delete_sql,
|
||||
(
|
||||
now - _LOCK_TIMEOUT_MS,
|
||||
lock_name,
|
||||
lock_key,
|
||||
),
|
||||
)
|
||||
txn.execute(
|
||||
insert_sql,
|
||||
(
|
||||
lock_name,
|
||||
lock_key,
|
||||
write,
|
||||
self._instance_name,
|
||||
token,
|
||||
now,
|
||||
),
|
||||
)
|
||||
|
||||
return
|
||||
|
||||
try:
|
||||
await self.db_pool.runInteraction(
|
||||
lock = await self.db_pool.runInteraction(
|
||||
"try_acquire_read_write_lock",
|
||||
_try_acquire_read_write_lock_txn,
|
||||
self._try_acquire_read_write_lock_txn,
|
||||
lock_name,
|
||||
lock_key,
|
||||
write,
|
||||
)
|
||||
except self.database_engine.module.IntegrityError:
|
||||
return None
|
||||
|
||||
return lock
|
||||
|
||||
def _try_acquire_read_write_lock_txn(
|
||||
self,
|
||||
txn: LoggingTransaction,
|
||||
lock_name: str,
|
||||
lock_key: str,
|
||||
write: bool,
|
||||
) -> "Lock":
|
||||
# We attempt to acquire the lock by inserting into
|
||||
# `worker_read_write_locks` and seeing if that fails any
|
||||
# constraints. If it doesn't then we have acquired the lock,
|
||||
# otherwise we haven't.
|
||||
#
|
||||
# Before that though we clear the table of any stale locks.
|
||||
|
||||
now = self._clock.time_msec()
|
||||
token = random_string(6)
|
||||
|
||||
delete_sql = """
|
||||
DELETE FROM worker_read_write_locks
|
||||
WHERE last_renewed_ts < ? AND lock_name = ? AND lock_key = ?;
|
||||
"""
|
||||
|
||||
insert_sql = """
|
||||
INSERT INTO worker_read_write_locks (lock_name, lock_key, write_lock, instance_name, token, last_renewed_ts)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
"""
|
||||
|
||||
if isinstance(self.database_engine, PostgresEngine):
|
||||
# For Postgres we can send these queries at the same time.
|
||||
txn.execute(
|
||||
delete_sql + ";" + insert_sql,
|
||||
(
|
||||
# DELETE args
|
||||
now - _LOCK_TIMEOUT_MS,
|
||||
lock_name,
|
||||
lock_key,
|
||||
# UPSERT args
|
||||
lock_name,
|
||||
lock_key,
|
||||
write,
|
||||
self._instance_name,
|
||||
token,
|
||||
now,
|
||||
),
|
||||
)
|
||||
else:
|
||||
# For SQLite these need to be two queries.
|
||||
txn.execute(
|
||||
delete_sql,
|
||||
(
|
||||
now - _LOCK_TIMEOUT_MS,
|
||||
lock_name,
|
||||
lock_key,
|
||||
),
|
||||
)
|
||||
txn.execute(
|
||||
insert_sql,
|
||||
(
|
||||
lock_name,
|
||||
lock_key,
|
||||
write,
|
||||
self._instance_name,
|
||||
token,
|
||||
now,
|
||||
),
|
||||
)
|
||||
|
||||
lock = Lock(
|
||||
self._reactor,
|
||||
self._clock,
|
||||
|
|
@ -289,10 +299,58 @@ class LockStore(SQLBaseStore):
|
|||
token=token,
|
||||
)
|
||||
|
||||
self._live_read_write_lock_tokens[(lock_name, lock_key, token)] = lock
|
||||
def set_lock() -> None:
|
||||
self._live_read_write_lock_tokens[(lock_name, lock_key, token)] = lock
|
||||
|
||||
txn.call_after(set_lock)
|
||||
|
||||
return lock
|
||||
|
||||
async def try_acquire_multi_read_write_lock(
|
||||
self,
|
||||
lock_names: Collection[Tuple[str, str]],
|
||||
write: bool,
|
||||
) -> Optional[AsyncExitStack]:
|
||||
"""Try to acquire multiple locks for the given names/keys. Will return
|
||||
an async context manager if the locks are successfully acquired, which
|
||||
*must* be used (otherwise the lock will leak).
|
||||
|
||||
If only a subset of the locks can be acquired then it will immediately
|
||||
drop them and return `None`.
|
||||
"""
|
||||
try:
|
||||
locks = await self.db_pool.runInteraction(
|
||||
"try_acquire_multi_read_write_lock",
|
||||
self._try_acquire_multi_read_write_lock_txn,
|
||||
lock_names,
|
||||
write,
|
||||
)
|
||||
except self.database_engine.module.IntegrityError:
|
||||
return None
|
||||
|
||||
stack = AsyncExitStack()
|
||||
|
||||
for lock in locks:
|
||||
await stack.enter_async_context(lock)
|
||||
|
||||
return stack
|
||||
|
||||
def _try_acquire_multi_read_write_lock_txn(
|
||||
self,
|
||||
txn: LoggingTransaction,
|
||||
lock_names: Collection[Tuple[str, str]],
|
||||
write: bool,
|
||||
) -> Collection["Lock"]:
|
||||
locks = []
|
||||
|
||||
for lock_name, lock_key in lock_names:
|
||||
lock = self._try_acquire_read_write_lock_txn(
|
||||
txn, lock_name, lock_key, write
|
||||
)
|
||||
locks.append(lock)
|
||||
|
||||
return locks
|
||||
|
||||
|
||||
class Lock:
|
||||
"""An async context manager that manages an acquired lock, ensuring it is
|
||||
|
|
|
|||
|
|
@ -249,12 +249,11 @@ class PurgeEventsStore(StateGroupWorkerStore, CacheInvalidationWorkerStore):
|
|||
# Mark all state and own events as outliers
|
||||
logger.info("[purge] marking remaining events as outliers")
|
||||
txn.execute(
|
||||
"UPDATE events SET outlier = ?"
|
||||
"UPDATE events SET outlier = TRUE"
|
||||
" WHERE event_id IN ("
|
||||
" SELECT event_id FROM events_to_purge "
|
||||
" WHERE NOT should_delete"
|
||||
")",
|
||||
(True,),
|
||||
" SELECT event_id FROM events_to_purge "
|
||||
" WHERE NOT should_delete"
|
||||
")"
|
||||
)
|
||||
|
||||
# synapse tries to take out an exclusive lock on room_depth whenever it
|
||||
|
|
|
|||
|
|
@ -560,19 +560,19 @@ class PushRuleStore(PushRulesWorkerStore):
|
|||
if isinstance(self.database_engine, PostgresEngine):
|
||||
sql = """
|
||||
INSERT INTO push_rules_enable (id, user_name, rule_id, enabled)
|
||||
VALUES (?, ?, ?, ?)
|
||||
VALUES (?, ?, ?, 1)
|
||||
ON CONFLICT DO NOTHING
|
||||
"""
|
||||
elif isinstance(self.database_engine, Sqlite3Engine):
|
||||
sql = """
|
||||
INSERT OR IGNORE INTO push_rules_enable (id, user_name, rule_id, enabled)
|
||||
VALUES (?, ?, ?, ?)
|
||||
VALUES (?, ?, ?, 1)
|
||||
"""
|
||||
else:
|
||||
raise RuntimeError("Unknown database engine")
|
||||
|
||||
new_enable_id = self._push_rules_enable_id_gen.get_next()
|
||||
txn.execute(sql, (new_enable_id, user_id, rule_id, 1))
|
||||
txn.execute(sql, (new_enable_id, user_id, rule_id))
|
||||
|
||||
async def delete_push_rule(self, user_id: str, rule_id: str) -> None:
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -454,9 +454,9 @@ class RegistrationWorkerStore(CacheInvalidationWorkerStore):
|
|||
) -> List[Tuple[str, int]]:
|
||||
sql = (
|
||||
"SELECT user_id, expiration_ts_ms FROM account_validity"
|
||||
" WHERE email_sent = ? AND (expiration_ts_ms - ?) <= ?"
|
||||
" WHERE email_sent = FALSE AND (expiration_ts_ms - ?) <= ?"
|
||||
)
|
||||
values = [False, now_ms, renew_at]
|
||||
values = [now_ms, renew_at]
|
||||
txn.execute(sql, values)
|
||||
return cast(List[Tuple[str, int]], txn.fetchall())
|
||||
|
||||
|
|
|
|||
|
|
@ -936,11 +936,11 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):
|
|||
JOIN event_json USING (room_id, event_id)
|
||||
WHERE room_id = ?
|
||||
%(where_clause)s
|
||||
AND contains_url = ? AND outlier = ?
|
||||
AND contains_url = TRUE AND outlier = FALSE
|
||||
ORDER BY stream_ordering DESC
|
||||
LIMIT ?
|
||||
"""
|
||||
txn.execute(sql % {"where_clause": ""}, (room_id, True, False, 100))
|
||||
txn.execute(sql % {"where_clause": ""}, (room_id, 100))
|
||||
|
||||
local_media_mxcs = []
|
||||
remote_media_mxcs = []
|
||||
|
|
@ -976,7 +976,7 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):
|
|||
|
||||
txn.execute(
|
||||
sql % {"where_clause": "AND stream_ordering < ?"},
|
||||
(room_id, next_token, True, False, 100),
|
||||
(room_id, next_token, 100),
|
||||
)
|
||||
|
||||
return local_media_mxcs, remote_media_mxcs
|
||||
|
|
@ -1086,9 +1086,9 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):
|
|||
|
||||
# set quarantine
|
||||
if quarantined_by is not None:
|
||||
sql += "AND safe_from_quarantine = ?"
|
||||
sql += "AND safe_from_quarantine = FALSE"
|
||||
txn.executemany(
|
||||
sql, [(quarantined_by, media_id, False) for media_id in local_mxcs]
|
||||
sql, [(quarantined_by, media_id) for media_id in local_mxcs]
|
||||
)
|
||||
# remove from quarantine
|
||||
else:
|
||||
|
|
|
|||
|
|
@ -1401,7 +1401,7 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
|
|||
`to_token`), or `limit` is zero.
|
||||
"""
|
||||
|
||||
args = [False, room_id]
|
||||
args: List[Any] = [room_id]
|
||||
|
||||
order, from_bound, to_bound = generate_pagination_bounds(
|
||||
direction, from_token, to_token
|
||||
|
|
@ -1475,7 +1475,7 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
|
|||
event.topological_ordering, event.stream_ordering
|
||||
FROM events AS event
|
||||
%(join_clause)s
|
||||
WHERE event.outlier = ? AND event.room_id = ? AND %(bounds)s
|
||||
WHERE event.outlier = FALSE AND event.room_id = ? AND %(bounds)s
|
||||
ORDER BY event.topological_ordering %(order)s,
|
||||
event.stream_ordering %(order)s LIMIT ?
|
||||
""" % {
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
SCHEMA_VERSION = 79 # remember to update the list below when updating
|
||||
SCHEMA_VERSION = 80 # remember to update the list below when updating
|
||||
"""Represents the expectations made by the codebase about the database schema
|
||||
|
||||
This should be incremented whenever the codebase changes its requirements on the
|
||||
|
|
@ -110,6 +110,9 @@ Changes in SCHEMA_VERSION = 78
|
|||
Changes in SCHEMA_VERSION = 79
|
||||
- Add tables to handle in DB read-write locks.
|
||||
- Add some mitigations for a painful race between foreground and background updates, cf #15677.
|
||||
|
||||
Changes in SCHEMA_VERSION = 80
|
||||
- The event_txn_id_device_id is always written to for new events.
|
||||
"""
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -117,11 +117,12 @@ class Requester:
|
|||
|
||||
Attributes:
|
||||
user: id of the user making the request
|
||||
access_token_id: *ID* of the access token used for this
|
||||
request, or None if it came via the appservice API or similar
|
||||
access_token_id: *ID* of the access token used for this request, or
|
||||
None for appservices, guests, and tokens generated by the admin API
|
||||
is_guest: True if the user making this request is a guest user
|
||||
shadow_banned: True if the user making this request has been shadow-banned.
|
||||
device_id: device_id which was set at authentication time
|
||||
device_id: device_id which was set at authentication time, or
|
||||
None for appservices, guests, and tokens generated by the admin API
|
||||
app_service: the AS requesting on behalf of the user
|
||||
authenticated_entity: The entity that authenticated when making the request.
|
||||
This is different to the user_id when an admin user or the server is
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ from synapse.logging.opentracing import (
|
|||
)
|
||||
from synapse.util import Clock
|
||||
from synapse.util.async_helpers import AbstractObservableDeferred, ObservableDeferred
|
||||
from synapse.util.caches import register_cache
|
||||
from synapse.util.caches import EvictionReason, register_cache
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -167,7 +167,7 @@ class ResponseCache(Generic[KV]):
|
|||
# the should_cache bit, we leave it in the cache for now and schedule
|
||||
# its removal later.
|
||||
if self.timeout_sec and context.should_cache:
|
||||
self.clock.call_later(self.timeout_sec, self.unset, key)
|
||||
self.clock.call_later(self.timeout_sec, self._entry_timeout, key)
|
||||
else:
|
||||
# otherwise, remove the result immediately.
|
||||
self.unset(key)
|
||||
|
|
@ -185,6 +185,12 @@ class ResponseCache(Generic[KV]):
|
|||
Args:
|
||||
key: key used to remove the cached value
|
||||
"""
|
||||
self._metrics.inc_evictions(EvictionReason.invalidation)
|
||||
self._result_cache.pop(key, None)
|
||||
|
||||
def _entry_timeout(self, key: KV) -> None:
|
||||
"""For the call_later to remove from the cache"""
|
||||
self._metrics.inc_evictions(EvictionReason.time)
|
||||
self._result_cache.pop(key, None)
|
||||
|
||||
async def wrap(
|
||||
|
|
|
|||
|
|
@ -27,15 +27,6 @@ if TYPE_CHECKING:
|
|||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# the initial backoff, after the first transaction fails
|
||||
MIN_RETRY_INTERVAL = 10 * 60 * 1000
|
||||
|
||||
# how much we multiply the backoff by after each subsequent fail
|
||||
RETRY_MULTIPLIER = 5
|
||||
|
||||
# a cap on the backoff. (Essentially none)
|
||||
MAX_RETRY_INTERVAL = 2**62
|
||||
|
||||
|
||||
class NotRetryingDestination(Exception):
|
||||
def __init__(self, retry_last_ts: int, retry_interval: int, destination: str):
|
||||
|
|
@ -169,6 +160,16 @@ class RetryDestinationLimiter:
|
|||
self.notifier = notifier
|
||||
self.replication_client = replication_client
|
||||
|
||||
self.destination_min_retry_interval_ms = (
|
||||
self.store.hs.config.federation.destination_min_retry_interval_ms
|
||||
)
|
||||
self.destination_retry_multiplier = (
|
||||
self.store.hs.config.federation.destination_retry_multiplier
|
||||
)
|
||||
self.destination_max_retry_interval_ms = (
|
||||
self.store.hs.config.federation.destination_max_retry_interval_ms
|
||||
)
|
||||
|
||||
def __enter__(self) -> None:
|
||||
pass
|
||||
|
||||
|
|
@ -220,13 +221,15 @@ class RetryDestinationLimiter:
|
|||
# We couldn't connect.
|
||||
if self.retry_interval:
|
||||
self.retry_interval = int(
|
||||
self.retry_interval * RETRY_MULTIPLIER * random.uniform(0.8, 1.4)
|
||||
self.retry_interval
|
||||
* self.destination_retry_multiplier
|
||||
* random.uniform(0.8, 1.4)
|
||||
)
|
||||
|
||||
if self.retry_interval >= MAX_RETRY_INTERVAL:
|
||||
self.retry_interval = MAX_RETRY_INTERVAL
|
||||
if self.retry_interval >= self.destination_max_retry_interval_ms:
|
||||
self.retry_interval = self.destination_max_retry_interval_ms
|
||||
else:
|
||||
self.retry_interval = MIN_RETRY_INTERVAL
|
||||
self.retry_interval = self.destination_min_retry_interval_ms
|
||||
|
||||
logger.info(
|
||||
"Connection to %s was unsuccessful (%s(%s)); backoff now %i",
|
||||
|
|
|
|||
|
|
@ -11,18 +11,18 @@
|
|||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from typing import Any, List, Mapping, Sequence, Union
|
||||
from typing import Any, List, Mapping, Optional, Sequence, Union
|
||||
from unittest.mock import Mock
|
||||
|
||||
from twisted.test.proto_helpers import MemoryReactor
|
||||
|
||||
from synapse.api.errors import HttpResponseException
|
||||
from synapse.appservice import ApplicationService
|
||||
from synapse.server import HomeServer
|
||||
from synapse.types import JsonDict
|
||||
from synapse.util import Clock
|
||||
|
||||
from tests import unittest
|
||||
from tests.unittest import override_config
|
||||
|
||||
PROTOCOL = "myproto"
|
||||
TOKEN = "myastoken"
|
||||
|
|
@ -40,7 +40,7 @@ class ApplicationServiceApiTestCase(unittest.HomeserverTestCase):
|
|||
hs_token=TOKEN,
|
||||
)
|
||||
|
||||
def test_query_3pe_authenticates_token(self) -> None:
|
||||
def test_query_3pe_authenticates_token_via_header(self) -> None:
|
||||
"""
|
||||
Tests that 3pe queries to the appservice are authenticated
|
||||
with the appservice's token.
|
||||
|
|
@ -75,12 +75,16 @@ class ApplicationServiceApiTestCase(unittest.HomeserverTestCase):
|
|||
args: Mapping[Any, Any],
|
||||
headers: Mapping[Union[str, bytes], Sequence[Union[str, bytes]]],
|
||||
) -> List[JsonDict]:
|
||||
# Ensure the access token is passed as both a header and query arg.
|
||||
if not headers.get("Authorization") or not args.get(b"access_token"):
|
||||
# Ensure the access token is passed as a header.
|
||||
if not headers or not headers.get("Authorization"):
|
||||
raise RuntimeError("Access token not provided")
|
||||
# ... and not as a query param
|
||||
if b"access_token" in args:
|
||||
raise RuntimeError(
|
||||
"Access token should not be passed as a query param."
|
||||
)
|
||||
|
||||
self.assertEqual(headers.get("Authorization"), [f"Bearer {TOKEN}"])
|
||||
self.assertEqual(args.get(b"access_token"), TOKEN)
|
||||
self.request_url = url
|
||||
if url == URL_USER:
|
||||
return SUCCESS_RESULT_USER
|
||||
|
|
@ -107,10 +111,13 @@ class ApplicationServiceApiTestCase(unittest.HomeserverTestCase):
|
|||
self.assertEqual(self.request_url, URL_LOCATION)
|
||||
self.assertEqual(result, SUCCESS_RESULT_LOCATION)
|
||||
|
||||
def test_fallback(self) -> None:
|
||||
@override_config({"use_appservice_legacy_authorization": True})
|
||||
def test_query_3pe_authenticates_token_via_param(self) -> None:
|
||||
"""
|
||||
Tests that the fallback to legacy URLs works.
|
||||
Tests that 3pe queries to the appservice are authenticated
|
||||
with the appservice's token.
|
||||
"""
|
||||
|
||||
SUCCESS_RESULT_USER = [
|
||||
{
|
||||
"protocol": PROTOCOL,
|
||||
|
|
@ -120,30 +127,41 @@ class ApplicationServiceApiTestCase(unittest.HomeserverTestCase):
|
|||
},
|
||||
}
|
||||
]
|
||||
SUCCESS_RESULT_LOCATION = [
|
||||
{
|
||||
"protocol": PROTOCOL,
|
||||
"alias": "#a:room",
|
||||
"fields": {
|
||||
"more": "fields",
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
URL_USER = f"{URL}/_matrix/app/v1/thirdparty/user/{PROTOCOL}"
|
||||
FALLBACK_URL_USER = f"{URL}/_matrix/app/unstable/thirdparty/user/{PROTOCOL}"
|
||||
URL_LOCATION = f"{URL}/_matrix/app/v1/thirdparty/location/{PROTOCOL}"
|
||||
|
||||
self.request_url = None
|
||||
self.v1_seen = False
|
||||
|
||||
async def get_json(
|
||||
url: str,
|
||||
args: Mapping[Any, Any],
|
||||
headers: Mapping[Union[str, bytes], Sequence[Union[str, bytes]]],
|
||||
headers: Optional[
|
||||
Mapping[Union[str, bytes], Sequence[Union[str, bytes]]]
|
||||
] = None,
|
||||
) -> List[JsonDict]:
|
||||
# Ensure the access token is passed as both a header and query arg.
|
||||
if not headers.get("Authorization") or not args.get(b"access_token"):
|
||||
raise RuntimeError("Access token not provided")
|
||||
# Ensure the access token is passed as a both a query param and in the headers.
|
||||
if not args.get(b"access_token"):
|
||||
raise RuntimeError("Access token should be provided in query params.")
|
||||
if not headers or not headers.get("Authorization"):
|
||||
raise RuntimeError("Access token should be provided in auth headers.")
|
||||
|
||||
self.assertEqual(headers.get("Authorization"), [f"Bearer {TOKEN}"])
|
||||
self.assertEqual(args.get(b"access_token"), TOKEN)
|
||||
self.assertEqual(headers.get("Authorization"), [f"Bearer {TOKEN}"])
|
||||
self.request_url = url
|
||||
if url == URL_USER:
|
||||
self.v1_seen = True
|
||||
raise HttpResponseException(404, "NOT_FOUND", b"NOT_FOUND")
|
||||
elif url == FALLBACK_URL_USER:
|
||||
return SUCCESS_RESULT_USER
|
||||
elif url == URL_LOCATION:
|
||||
return SUCCESS_RESULT_LOCATION
|
||||
else:
|
||||
raise RuntimeError(
|
||||
"URL provided was invalid. This should never be seen."
|
||||
|
|
@ -155,9 +173,15 @@ class ApplicationServiceApiTestCase(unittest.HomeserverTestCase):
|
|||
result = self.get_success(
|
||||
self.api.query_3pe(self.service, "user", PROTOCOL, {b"some": [b"field"]})
|
||||
)
|
||||
self.assertTrue(self.v1_seen)
|
||||
self.assertEqual(self.request_url, FALLBACK_URL_USER)
|
||||
self.assertEqual(self.request_url, URL_USER)
|
||||
self.assertEqual(result, SUCCESS_RESULT_USER)
|
||||
result = self.get_success(
|
||||
self.api.query_3pe(
|
||||
self.service, "location", PROTOCOL, {b"some": [b"field"]}
|
||||
)
|
||||
)
|
||||
self.assertEqual(self.request_url, URL_LOCATION)
|
||||
self.assertEqual(result, SUCCESS_RESULT_LOCATION)
|
||||
|
||||
def test_claim_keys(self) -> None:
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -179,6 +179,16 @@ class ProfileTestCase(unittest.HomeserverTestCase):
|
|||
|
||||
self.assertEqual("http://my.server/me.png", avatar_url)
|
||||
|
||||
def test_get_profile_empty_displayname(self) -> None:
|
||||
self.get_success(self.store.set_profile_displayname(self.frank, None))
|
||||
self.get_success(
|
||||
self.store.set_profile_avatar_url(self.frank, "http://my.server/me.png")
|
||||
)
|
||||
|
||||
profile = self.get_success(self.handler.get_profile(self.frank.to_string()))
|
||||
|
||||
self.assertEqual("http://my.server/me.png", profile["avatar_url"])
|
||||
|
||||
def test_set_my_avatar(self) -> None:
|
||||
self.get_success(
|
||||
self.handler.set_avatar_url(
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
# Copyright 2023 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from twisted.internet import defer
|
||||
from twisted.test.proto_helpers import MemoryReactor
|
||||
|
||||
from synapse.server import HomeServer
|
||||
from synapse.util import Clock
|
||||
|
||||
from tests import unittest
|
||||
from tests.replication._base import BaseMultiWorkerStreamTestCase
|
||||
|
||||
|
||||
class WorkerLockTestCase(unittest.HomeserverTestCase):
|
||||
def prepare(
|
||||
self, reactor: MemoryReactor, clock: Clock, homeserver: HomeServer
|
||||
) -> None:
|
||||
self.worker_lock_handler = self.hs.get_worker_locks_handler()
|
||||
|
||||
def test_wait_for_lock_locally(self) -> None:
|
||||
"""Test waiting for a lock on a single worker"""
|
||||
|
||||
lock1 = self.worker_lock_handler.acquire_lock("name", "key")
|
||||
self.get_success(lock1.__aenter__())
|
||||
|
||||
lock2 = self.worker_lock_handler.acquire_lock("name", "key")
|
||||
d2 = defer.ensureDeferred(lock2.__aenter__())
|
||||
self.assertNoResult(d2)
|
||||
|
||||
self.get_success(lock1.__aexit__(None, None, None))
|
||||
|
||||
self.get_success(d2)
|
||||
self.get_success(lock2.__aexit__(None, None, None))
|
||||
|
||||
|
||||
class WorkerLockWorkersTestCase(BaseMultiWorkerStreamTestCase):
|
||||
def prepare(
|
||||
self, reactor: MemoryReactor, clock: Clock, homeserver: HomeServer
|
||||
) -> None:
|
||||
self.main_worker_lock_handler = self.hs.get_worker_locks_handler()
|
||||
|
||||
def test_wait_for_lock_worker(self) -> None:
|
||||
"""Test waiting for a lock on another worker"""
|
||||
|
||||
worker = self.make_worker_hs(
|
||||
"synapse.app.generic_worker",
|
||||
extra_config={
|
||||
"redis": {"enabled": True},
|
||||
},
|
||||
)
|
||||
worker_lock_handler = worker.get_worker_locks_handler()
|
||||
|
||||
lock1 = self.main_worker_lock_handler.acquire_lock("name", "key")
|
||||
self.get_success(lock1.__aenter__())
|
||||
|
||||
lock2 = worker_lock_handler.acquire_lock("name", "key")
|
||||
d2 = defer.ensureDeferred(lock2.__aenter__())
|
||||
self.assertNoResult(d2)
|
||||
|
||||
self.get_success(lock1.__aexit__(None, None, None))
|
||||
|
||||
self.get_success(d2)
|
||||
self.get_success(lock2.__aexit__(None, None, None))
|
||||
|
|
@ -409,12 +409,12 @@ class TestBulkPushRuleEvaluator(HomeserverTestCase):
|
|||
)
|
||||
)
|
||||
|
||||
# Room mentions from those without power should not notify.
|
||||
# The edit should not cause a notification.
|
||||
self.assertFalse(
|
||||
self._create_and_process(
|
||||
bulk_evaluator,
|
||||
{
|
||||
"body": self.alice,
|
||||
"body": "Test message",
|
||||
"m.relates_to": {
|
||||
"rel_type": RelationTypes.REPLACE,
|
||||
"event_id": event.event_id,
|
||||
|
|
@ -422,3 +422,20 @@ class TestBulkPushRuleEvaluator(HomeserverTestCase):
|
|||
},
|
||||
)
|
||||
)
|
||||
|
||||
# An edit which is a mention will cause a notification.
|
||||
self.assertTrue(
|
||||
self._create_and_process(
|
||||
bulk_evaluator,
|
||||
{
|
||||
"body": "Test message",
|
||||
"m.relates_to": {
|
||||
"rel_type": RelationTypes.REPLACE,
|
||||
"event_id": event.event_id,
|
||||
},
|
||||
"m.mentions": {
|
||||
"user_ids": [self.alice],
|
||||
},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -379,4 +379,141 @@ class DehydratedDeviceTestCase(unittest.HomeserverTestCase):
|
|||
access_token=token,
|
||||
shorthand=False,
|
||||
)
|
||||
self.assertEqual(channel.code, 404)
|
||||
self.assertEqual(channel.code, 401)
|
||||
|
||||
@unittest.override_config(
|
||||
{"experimental_features": {"msc2697_enabled": False, "msc3814_enabled": True}}
|
||||
)
|
||||
def test_msc3814_dehydrated_device_delete_works(self) -> None:
|
||||
user = self.register_user("mikey", "pass")
|
||||
token = self.login(user, "pass", device_id="device1")
|
||||
content: JsonDict = {
|
||||
"device_data": {
|
||||
"algorithm": "m.dehydration.v1.olm",
|
||||
},
|
||||
"device_id": "device2",
|
||||
"initial_device_display_name": "foo bar",
|
||||
"device_keys": {
|
||||
"user_id": "@mikey:test",
|
||||
"device_id": "device2",
|
||||
"valid_until_ts": "80",
|
||||
"algorithms": [
|
||||
"m.olm.curve25519-aes-sha2",
|
||||
],
|
||||
"keys": {
|
||||
"<algorithm>:<device_id>": "<key_base64>",
|
||||
},
|
||||
"signatures": {
|
||||
"<user_id>": {"<algorithm>:<device_id>": "<signature_base64>"}
|
||||
},
|
||||
},
|
||||
}
|
||||
channel = self.make_request(
|
||||
"PUT",
|
||||
"_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
|
||||
content=content,
|
||||
access_token=token,
|
||||
shorthand=False,
|
||||
)
|
||||
self.assertEqual(channel.code, 200)
|
||||
device_id = channel.json_body.get("device_id")
|
||||
assert device_id is not None
|
||||
self.assertIsInstance(device_id, str)
|
||||
self.assertEqual("device2", device_id)
|
||||
|
||||
# ensure that keys were uploaded and available
|
||||
channel = self.make_request(
|
||||
"POST",
|
||||
"/_matrix/client/r0/keys/query",
|
||||
{
|
||||
"device_keys": {
|
||||
user: ["device2"],
|
||||
},
|
||||
},
|
||||
token,
|
||||
)
|
||||
self.assertEqual(
|
||||
channel.json_body["device_keys"][user]["device2"]["keys"],
|
||||
{
|
||||
"<algorithm>:<device_id>": "<key_base64>",
|
||||
},
|
||||
)
|
||||
|
||||
# delete the dehydrated device
|
||||
channel = self.make_request(
|
||||
"DELETE",
|
||||
"_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
|
||||
access_token=token,
|
||||
shorthand=False,
|
||||
)
|
||||
self.assertEqual(channel.code, 200)
|
||||
|
||||
# ensure that keys are no longer available for deleted device
|
||||
channel = self.make_request(
|
||||
"POST",
|
||||
"/_matrix/client/r0/keys/query",
|
||||
{
|
||||
"device_keys": {
|
||||
user: ["device2"],
|
||||
},
|
||||
},
|
||||
token,
|
||||
)
|
||||
self.assertEqual(channel.json_body["device_keys"], {"@mikey:test": {}})
|
||||
|
||||
# check that an old device is deleted when user PUTs a new device
|
||||
# First, create a device
|
||||
content["device_id"] = "device3"
|
||||
content["device_keys"]["device_id"] = "device3"
|
||||
channel = self.make_request(
|
||||
"PUT",
|
||||
"_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
|
||||
content=content,
|
||||
access_token=token,
|
||||
shorthand=False,
|
||||
)
|
||||
self.assertEqual(channel.code, 200)
|
||||
device_id = channel.json_body.get("device_id")
|
||||
assert device_id is not None
|
||||
self.assertIsInstance(device_id, str)
|
||||
self.assertEqual("device3", device_id)
|
||||
|
||||
# create a second device without deleting first device
|
||||
content["device_id"] = "device4"
|
||||
content["device_keys"]["device_id"] = "device4"
|
||||
channel = self.make_request(
|
||||
"PUT",
|
||||
"_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
|
||||
content=content,
|
||||
access_token=token,
|
||||
shorthand=False,
|
||||
)
|
||||
self.assertEqual(channel.code, 200)
|
||||
device_id = channel.json_body.get("device_id")
|
||||
assert device_id is not None
|
||||
self.assertIsInstance(device_id, str)
|
||||
self.assertEqual("device4", device_id)
|
||||
|
||||
# check that the second device that was created is what is returned when we GET
|
||||
channel = self.make_request(
|
||||
"GET",
|
||||
"_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device",
|
||||
access_token=token,
|
||||
shorthand=False,
|
||||
)
|
||||
self.assertEqual(channel.code, 200)
|
||||
returned_device_id = channel.json_body["device_id"]
|
||||
self.assertEqual(returned_device_id, "device4")
|
||||
|
||||
# and that if we query the keys for the first device they are not there
|
||||
channel = self.make_request(
|
||||
"POST",
|
||||
"/_matrix/client/r0/keys/query",
|
||||
{
|
||||
"device_keys": {
|
||||
user: ["device3"],
|
||||
},
|
||||
},
|
||||
token,
|
||||
)
|
||||
self.assertEqual(channel.json_body["device_keys"], {"@mikey:test": {}})
|
||||
|
|
|
|||
|
|
@ -68,6 +68,18 @@ class ProfileTestCase(unittest.HomeserverTestCase):
|
|||
res = self._get_displayname()
|
||||
self.assertEqual(res, "test")
|
||||
|
||||
def test_set_displayname_with_extra_spaces(self) -> None:
|
||||
channel = self.make_request(
|
||||
"PUT",
|
||||
"/profile/%s/displayname" % (self.owner,),
|
||||
content={"displayname": " test "},
|
||||
access_token=self.owner_tok,
|
||||
)
|
||||
self.assertEqual(channel.code, 200, channel.result)
|
||||
|
||||
res = self._get_displayname()
|
||||
self.assertEqual(res, "test")
|
||||
|
||||
def test_set_displayname_noauth(self) -> None:
|
||||
channel = self.make_request(
|
||||
"PUT",
|
||||
|
|
|
|||
|
|
@ -13,10 +13,12 @@
|
|||
# limitations under the License.
|
||||
from typing import List, Optional
|
||||
|
||||
from parameterized import parameterized
|
||||
|
||||
from twisted.test.proto_helpers import MemoryReactor
|
||||
|
||||
from synapse.api.constants import EventTypes, RelationTypes
|
||||
from synapse.api.room_versions import RoomVersions
|
||||
from synapse.api.room_versions import RoomVersion, RoomVersions
|
||||
from synapse.rest import admin
|
||||
from synapse.rest.client import login, room, sync
|
||||
from synapse.server import HomeServer
|
||||
|
|
@ -569,50 +571,81 @@ class RedactionsTestCase(HomeserverTestCase):
|
|||
self.assertIn("body", event_dict["content"], event_dict)
|
||||
self.assertEqual("I'm in a thread!", event_dict["content"]["body"])
|
||||
|
||||
def test_content_redaction(self) -> None:
|
||||
"""MSC2174 moved the redacts property to the content."""
|
||||
@parameterized.expand(
|
||||
[
|
||||
# Tuples of:
|
||||
# Room version
|
||||
# Boolean: True if the redaction event content should include the event ID.
|
||||
# Boolean: true if the resulting redaction event is expected to include the
|
||||
# event ID in the content.
|
||||
(RoomVersions.V10, False, False),
|
||||
(RoomVersions.V11, True, True),
|
||||
(RoomVersions.V11, False, True),
|
||||
]
|
||||
)
|
||||
def test_redaction_content(
|
||||
self, room_version: RoomVersion, include_content: bool, expect_content: bool
|
||||
) -> None:
|
||||
"""
|
||||
Room version 11 moved the redacts property to the content.
|
||||
|
||||
Ensure that the event gets created properly and that the Client-Server
|
||||
API servers the proper backwards-compatible version.
|
||||
"""
|
||||
# Create a room with the newer room version.
|
||||
room_id = self.helper.create_room_as(
|
||||
self.mod_user_id,
|
||||
tok=self.mod_access_token,
|
||||
room_version=RoomVersions.V11.identifier,
|
||||
room_version=room_version.identifier,
|
||||
)
|
||||
|
||||
# Create an event.
|
||||
b = self.helper.send(room_id=room_id, tok=self.mod_access_token)
|
||||
event_id = b["event_id"]
|
||||
|
||||
# Attempt to redact it with a bogus event ID.
|
||||
self._redact_event(
|
||||
# Ensure the event ID in the URL and the content must match.
|
||||
if include_content:
|
||||
self._redact_event(
|
||||
self.mod_access_token,
|
||||
room_id,
|
||||
event_id,
|
||||
expect_code=400,
|
||||
content={"redacts": "foo"},
|
||||
)
|
||||
|
||||
# Redact it for real.
|
||||
result = self._redact_event(
|
||||
self.mod_access_token,
|
||||
room_id,
|
||||
event_id,
|
||||
expect_code=400,
|
||||
content={"redacts": "foo"},
|
||||
content={"redacts": event_id} if include_content else {},
|
||||
)
|
||||
|
||||
# Redact it for real.
|
||||
self._redact_event(self.mod_access_token, room_id, event_id)
|
||||
redaction_event_id = result["event_id"]
|
||||
|
||||
# Sync the room, to get the id of the create event
|
||||
timeline = self._sync_room_timeline(self.mod_access_token, room_id)
|
||||
redact_event = timeline[-1]
|
||||
self.assertEqual(redact_event["type"], EventTypes.Redaction)
|
||||
# The redacts key should be in the content.
|
||||
# The redacts key should be in the content and the redacts keys.
|
||||
self.assertEquals(redact_event["content"]["redacts"], event_id)
|
||||
|
||||
# It should also be copied as the top-level redacts field for backwards
|
||||
# compatibility.
|
||||
self.assertEquals(redact_event["redacts"], event_id)
|
||||
|
||||
# But it isn't actually part of the event.
|
||||
def get_event(txn: LoggingTransaction) -> JsonDict:
|
||||
return db_to_json(
|
||||
main_datastore._fetch_event_rows(txn, [event_id])[event_id].json
|
||||
main_datastore._fetch_event_rows(txn, [redaction_event_id])[
|
||||
redaction_event_id
|
||||
].json
|
||||
)
|
||||
|
||||
main_datastore = self.hs.get_datastores().main
|
||||
event_json = self.get_success(
|
||||
main_datastore.db_pool.runInteraction("get_event", get_event)
|
||||
)
|
||||
self.assertNotIn("redacts", event_json)
|
||||
self.assertEquals(event_json["type"], EventTypes.Redaction)
|
||||
if expect_content:
|
||||
self.assertNotIn("redacts", event_json)
|
||||
self.assertEquals(event_json["content"]["redacts"], event_id)
|
||||
else:
|
||||
self.assertEquals(event_json["redacts"], event_id)
|
||||
self.assertNotIn("redacts", event_json["content"])
|
||||
|
|
|
|||
|
|
@ -711,7 +711,7 @@ class RoomsCreateTestCase(RoomBase):
|
|||
self.assertEqual(HTTPStatus.OK, channel.code, channel.result)
|
||||
self.assertTrue("room_id" in channel.json_body)
|
||||
assert channel.resource_usage is not None
|
||||
self.assertEqual(30, channel.resource_usage.db_txn_count)
|
||||
self.assertEqual(32, channel.resource_usage.db_txn_count)
|
||||
|
||||
def test_post_room_initial_state(self) -> None:
|
||||
# POST with initial_state config key, expect new room id
|
||||
|
|
@ -724,7 +724,7 @@ class RoomsCreateTestCase(RoomBase):
|
|||
self.assertEqual(HTTPStatus.OK, channel.code, channel.result)
|
||||
self.assertTrue("room_id" in channel.json_body)
|
||||
assert channel.resource_usage is not None
|
||||
self.assertEqual(32, channel.resource_usage.db_txn_count)
|
||||
self.assertEqual(34, channel.resource_usage.db_txn_count)
|
||||
|
||||
def test_post_room_visibility_key(self) -> None:
|
||||
# POST with visibility config key, expect new room id
|
||||
|
|
|
|||
|
|
@ -448,3 +448,55 @@ class ReadWriteLockTestCase(unittest.HomeserverTestCase):
|
|||
self.get_success(self.store._on_shutdown())
|
||||
|
||||
self.assertEqual(self.store._live_read_write_lock_tokens, {})
|
||||
|
||||
def test_acquire_multiple_locks(self) -> None:
|
||||
"""Tests that acquiring multiple locks at once works."""
|
||||
|
||||
# Take out multiple locks and ensure that we can't get those locks out
|
||||
# again.
|
||||
lock = self.get_success(
|
||||
self.store.try_acquire_multi_read_write_lock(
|
||||
[("name1", "key1"), ("name2", "key2")], write=True
|
||||
)
|
||||
)
|
||||
self.assertIsNotNone(lock)
|
||||
|
||||
assert lock is not None
|
||||
self.get_success(lock.__aenter__())
|
||||
|
||||
lock2 = self.get_success(
|
||||
self.store.try_acquire_read_write_lock("name1", "key1", write=True)
|
||||
)
|
||||
self.assertIsNone(lock2)
|
||||
|
||||
lock3 = self.get_success(
|
||||
self.store.try_acquire_read_write_lock("name2", "key2", write=False)
|
||||
)
|
||||
self.assertIsNone(lock3)
|
||||
|
||||
# Overlapping locks attempts will fail, and won't lock any locks.
|
||||
lock4 = self.get_success(
|
||||
self.store.try_acquire_multi_read_write_lock(
|
||||
[("name1", "key1"), ("name3", "key3")], write=True
|
||||
)
|
||||
)
|
||||
self.assertIsNone(lock4)
|
||||
|
||||
lock5 = self.get_success(
|
||||
self.store.try_acquire_read_write_lock("name3", "key3", write=True)
|
||||
)
|
||||
self.assertIsNotNone(lock5)
|
||||
assert lock5 is not None
|
||||
self.get_success(lock5.__aenter__())
|
||||
self.get_success(lock5.__aexit__(None, None, None))
|
||||
|
||||
# Once we release the lock we can take out the locks again.
|
||||
self.get_success(lock.__aexit__(None, None, None))
|
||||
|
||||
lock6 = self.get_success(
|
||||
self.store.try_acquire_read_write_lock("name1", "key1", write=True)
|
||||
)
|
||||
self.assertIsNotNone(lock6)
|
||||
assert lock6 is not None
|
||||
self.get_success(lock6.__aenter__())
|
||||
self.get_success(lock6.__aexit__(None, None, None))
|
||||
|
|
|
|||
|
|
@ -17,7 +17,6 @@ from twisted.test.proto_helpers import MemoryReactor
|
|||
from synapse.server import HomeServer
|
||||
from synapse.storage.databases.main.transactions import DestinationRetryTimings
|
||||
from synapse.util import Clock
|
||||
from synapse.util.retryutils import MAX_RETRY_INTERVAL
|
||||
|
||||
from tests.unittest import HomeserverTestCase
|
||||
|
||||
|
|
@ -57,8 +56,14 @@ class TransactionStoreTestCase(HomeserverTestCase):
|
|||
self.get_success(d)
|
||||
|
||||
def test_large_destination_retry(self) -> None:
|
||||
max_retry_interval_ms = (
|
||||
self.hs.config.federation.destination_max_retry_interval_ms
|
||||
)
|
||||
d = self.store.set_destination_retry_timings(
|
||||
"example.com", MAX_RETRY_INTERVAL, MAX_RETRY_INTERVAL, MAX_RETRY_INTERVAL
|
||||
"example.com",
|
||||
max_retry_interval_ms,
|
||||
max_retry_interval_ms,
|
||||
max_retry_interval_ms,
|
||||
)
|
||||
self.get_success(d)
|
||||
|
||||
|
|
|
|||
|
|
@ -11,12 +11,7 @@
|
|||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from synapse.util.retryutils import (
|
||||
MIN_RETRY_INTERVAL,
|
||||
RETRY_MULTIPLIER,
|
||||
NotRetryingDestination,
|
||||
get_retry_limiter,
|
||||
)
|
||||
from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
|
||||
|
||||
from tests.unittest import HomeserverTestCase
|
||||
|
||||
|
|
@ -42,6 +37,11 @@ class RetryLimiterTestCase(HomeserverTestCase):
|
|||
|
||||
limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
|
||||
|
||||
min_retry_interval_ms = (
|
||||
self.hs.config.federation.destination_min_retry_interval_ms
|
||||
)
|
||||
retry_multiplier = self.hs.config.federation.destination_retry_multiplier
|
||||
|
||||
self.pump(1)
|
||||
try:
|
||||
with limiter:
|
||||
|
|
@ -57,7 +57,7 @@ class RetryLimiterTestCase(HomeserverTestCase):
|
|||
assert new_timings is not None
|
||||
self.assertEqual(new_timings.failure_ts, failure_ts)
|
||||
self.assertEqual(new_timings.retry_last_ts, failure_ts)
|
||||
self.assertEqual(new_timings.retry_interval, MIN_RETRY_INTERVAL)
|
||||
self.assertEqual(new_timings.retry_interval, min_retry_interval_ms)
|
||||
|
||||
# now if we try again we should get a failure
|
||||
self.get_failure(
|
||||
|
|
@ -68,7 +68,7 @@ class RetryLimiterTestCase(HomeserverTestCase):
|
|||
# advance the clock and try again
|
||||
#
|
||||
|
||||
self.pump(MIN_RETRY_INTERVAL)
|
||||
self.pump(min_retry_interval_ms)
|
||||
limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
|
||||
|
||||
self.pump(1)
|
||||
|
|
@ -87,16 +87,16 @@ class RetryLimiterTestCase(HomeserverTestCase):
|
|||
self.assertEqual(new_timings.failure_ts, failure_ts)
|
||||
self.assertEqual(new_timings.retry_last_ts, retry_ts)
|
||||
self.assertGreaterEqual(
|
||||
new_timings.retry_interval, MIN_RETRY_INTERVAL * RETRY_MULTIPLIER * 0.5
|
||||
new_timings.retry_interval, min_retry_interval_ms * retry_multiplier * 0.5
|
||||
)
|
||||
self.assertLessEqual(
|
||||
new_timings.retry_interval, MIN_RETRY_INTERVAL * RETRY_MULTIPLIER * 2.0
|
||||
new_timings.retry_interval, min_retry_interval_ms * retry_multiplier * 2.0
|
||||
)
|
||||
|
||||
#
|
||||
# one more go, with success
|
||||
#
|
||||
self.reactor.advance(MIN_RETRY_INTERVAL * RETRY_MULTIPLIER * 2.0)
|
||||
self.reactor.advance(min_retry_interval_ms * retry_multiplier * 2.0)
|
||||
limiter = self.get_success(get_retry_limiter("test_dest", self.clock, store))
|
||||
|
||||
self.pump(1)
|
||||
|
|
|
|||
Loading…
Reference in New Issue