Commit Graph

21605 Commits (1eb8dcf4c9f375a5314e0197d1236d52c1fc36a2)

Author SHA1 Message Date
Dirk Klimpel 1eb8dcf4c9
Remove not needed `replication` listener in docker compose example (#14107) 2022-10-17 12:00:09 +01:00
Eric Eastwood 40bb37eb27
Stop getting missing `prev_events` after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
dependabot[bot] 616dcc1d18
Bump types-bleach from 4.1.4 to 5.0.3 (#14188)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-14 23:57:55 +00:00
dependabot[bot] 8cd22674a1
Bump types-pillow from 9.2.2 to 9.2.2.1 (#14187)
* Bump types-pillow from 9.2.2 to 9.2.2.1

Bumps [types-pillow](https://github.com/python/typeshed) from 9.2.2 to 9.2.2.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-15 00:16:25 +01:00
realtyem c75836fe77
Strip whitespace from worker types in Dockerfile-workers (#14165) 2022-10-14 19:38:04 +00:00
Patrick Cloke bc2bd92b93 Merge remote-tracking branch 'origin/release-v1.69' into develop 2022-10-14 14:11:27 -04:00
Patrick Cloke d1bdeccb50
Accept threaded receipts for events related to the root event. (#14174)
The root node of a thread (and events related to it) are considered
"part of a thread" when validating receipts. This allows clients which
show the root node in both the main timeline and the threaded timeline
to easily send receipts in either.

Note that threaded notifications are not created for these events, these
events created notifications on the main timeline.
2022-10-14 18:05:25 +00:00
Erik Johnston b43be004b4 1.69.0rc4 2022-10-14 15:05:04 +01:00
Erik Johnston d241a1350d
Fix background update to use an index (#14181) 2022-10-14 13:46:23 +00:00
Patrick Cloke 022f25b309
Advertise support for Matrix 1.4. (#14184)
All features / changes in Matrix 1.4 are now supported in
Synapse.
2022-10-14 13:21:55 +00:00
Patrick Cloke 5a983cccfb Fix changelog for #14171 to match #13394/#14175. 2022-10-14 08:56:37 -04:00
Andrew Morgan 97b3d037c0
Don't require optional `invite_room_state` field on fed v2 invite (#14083) 2022-10-14 13:48:33 +01:00
Patrick Cloke 126a15794c
Do not allow a None-limit on PaginationConfig. (#14146)
The callers either set a default limit or manually handle a None-limit
later on (by setting a default value).

Update the callers to always instantiate PaginationConfig with a default
limit and then assume the limit is non-None.
2022-10-14 12:30:05 +00:00
realtyem c7446906bd
Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182) 2022-10-14 13:29:49 +01:00
Patrick Cloke c3e4edb4d6
Stabilize the threads API. (#14175)
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.

This also marks the API as safe for workers since it is a read-only
API.
2022-10-14 07:16:50 -04:00
Patrick Cloke 9ff4155f6c
Properly invalidate get_thread_id cache. (#14163)
This was missed in 2b6d41ebd6 (#13824).
2022-10-14 07:10:44 -04:00
Erik Johnston 6e0dde3215 Revert "Disable push of docker images"
This reverts commit f3f303aa22.
2022-10-14 10:50:24 +01:00
Erik Johnston 424d1d28cc Fix docker workflow 2022-10-14 09:57:31 +01:00
Erik Johnston f3f303aa22 Disable push of docker images 2022-10-14 09:54:56 +01:00
Erik Johnston 29ee4b6698
Fix docker build OOMing in CI for arm64 builds (#14173)
Co-authored-by: David Robertson <davidr@element.io>
2022-10-13 18:16:21 +00:00
David Robertson 16c5d95b59
Optimise the event_push_backfill_thread_id bg job (#14172)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-10-13 17:32:16 +00:00
Patrick Cloke 2019b60f3b
Fix sqlite syntax for upserts. (#14171) 2022-10-13 12:53:24 -04:00
Patrick Cloke 7d59a515bb Properly return the thread ID down sync. (#14159)
Fix a broken conflict in e6e876b9b1,
by not stomping over a field right after creating it.
2022-10-13 12:15:41 -04:00
Patrick Cloke 3bbe532abb
Add an API for listing threads in a room. (#13394)
Implement the /threads endpoint from MSC3856.

This is currently unstable and behind an experimental configuration
flag.

It includes a background update to backfill data, results from
the /threads endpoint will be partial until that finishes.
2022-10-13 08:02:11 -04:00
Shay b6baa46db0
Fix a bug where the joined hosts for a given event were not being properly cached (#14125) 2022-10-12 11:01:00 -07:00
Patrick Cloke e6e876b9b1 Return the thread ID properly down sync. (#14159)
A receipt's thread ID, if one exists, should be added to the
body of a receipt.
2022-10-12 12:18:34 -04:00
Patrick Cloke 87099b6ea5
Return the main timeline for events which are not part of a thread. (#14140)
Fixes a bug where threaded receipts could not be sent for the
main timeline.
2022-10-12 12:15:52 -04:00
dependabot[bot] e4e55f8eef
Bump typing-extensions from 4.1.1 to 4.4.0 (#14134) 2022-10-12 14:37:36 +01:00
Andrew Morgan 9c23442ac9
Correct field name for stripped state events when knocking. `knock_state_events` -> `knock_room_state` (#14102) 2022-10-12 14:37:20 +01:00
dependabot[bot] c152e58116
Bump bleach from 4.1.0 to 5.0.1 (#14116) 2022-10-12 14:37:04 +01:00
dependabot[bot] 10a432a5f0
Bump idna from 3.3 to 3.4 (#14115) 2022-10-12 14:34:50 +01:00
Erik Johnston 821f74a8c0 Fixup changelog 2022-10-12 13:29:21 +01:00
Erik Johnston 19eb23bf32 1.69.0rc3 2022-10-12 13:24:22 +01:00
Patrick Cloke c604d2c218
Mark /relations endpoint as usable on workers. (#14028)
Co-authored-by: Eric Eastwood <erice@element.io>
2022-10-12 06:46:13 -04:00
Nick Mills-Barrett f9bc5428c4
Batch up calls to `get_rooms_for_users` (#14109) 2022-10-12 11:36:22 +01:00
Patrick Cloke 09be8ab5f9
Remove the experimental implementation of MSC3772. (#14094)
MSC3772 has been abandoned.
2022-10-12 06:26:39 -04:00
David Robertson 3f057e4c54
Use minimal Rust installation in docker images and CI (#14141) 2022-10-12 10:47:02 +01:00
Shay a86b2f6837
Fix a bug where redactions were not being sent over federation if we did not have the original event. (#13813) 2022-10-11 11:18:45 -07:00
David Robertson 6a92944854
Handle `gottestfmt` repository move (#14144) 2022-10-11 17:54:06 +00:00
Patrick Cloke 6136768e76
Remove the groups config code. (#14142)
This has been unused for a long time, but missed removal in #11584.
2022-10-11 13:14:57 -04:00
Erik Johnston 02086e1da0
Fix rotating existing notifications in push summary (#14138)
Broke by #14045. Fixes #14120.

Introduced in v1.69.0rc2.
2022-10-11 15:13:32 +00:00
Abdullah Osama a9934d48c1
Making parse_server_name more consistent (#14007)
Fixes #12122
2022-10-11 12:42:11 +00:00
Brendan Abolivier 422cff7df6
Fallback if 'approved' isn't included in a registration replication request (#14135) 2022-10-11 14:41:06 +02:00
Erik Johnston 17c031b251
Enable dependabot for Rust dependencies (#14132) 2022-10-11 12:26:40 +01:00
Erik Johnston d94bcbced3
Fix pinning Rust deps in docker images (#14129) 2022-10-11 10:53:34 +00:00
Andrew Morgan e580f03e02
Fix name of "alias_creation_rules" option in config manual (#14124) 2022-10-10 20:01:05 +00:00
dependabot[bot] 544cc400e3
Bump types-psycopg2 from 2.9.9 to 2.9.21.1 (#14114)
* Bump types-psycopg2 from 2.9.9 to 2.9.21.1

Bumps [types-psycopg2](https://github.com/python/typeshed) from 2.9.9 to 2.9.21.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-psycopg2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-10 13:41:21 +01:00
dependabot[bot] 475bc8acb9
Bump types-pillow from 9.0.15 to 9.2.2 (#14113)
* Bump types-pillow from 9.0.15 to 9.2.2

Bumps [types-pillow](https://github.com/python/typeshed) from 9.0.15 to 9.2.2.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-10 13:36:26 +01:00
Eric Eastwood 1bf2832714
Indicate what endpoint came back with a JSON response we were unable to parse (#14097)
**Before:**
```
WARNING - POST-11 - Unable to parse JSON: Expecting value: line 1 column 1 (char 0) (b'')
```

**After:**
```
WARNING - POST-11 - Unable to parse JSON from POST /_matrix/client/v3/join/%21ZlmJtelqFroDRJYZaq:hs1?server_name=hs1 response: Expecting value: line 1 column 1 (char 0) (b'')
```

---

It's possible to figure out which endpoint these warnings were coming from before but you had to follow the request ID `POST-11` to the log line that says `Completed request [...]`. Including this key information next to the JSON parsing error makes it much easier to reason whether it matters or not.

```
2022-09-29T08:23:25.7875506Z synapse_main | 2022-09-29 08:21:10,336 - synapse.http.matrixfederationclient - 299 - INFO - POST-11 - {GET-O-13} [hs1] Completed request: 200 OK in 0.53 secs, got 450 bytes - GET matrix://hs1/_matrix/federation/v1/make_join/%21ohtKoQiXlPePSycXwp%3Ahs1/%40charlie%3Ahs2?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=org.matrix.msc2176&ver=7&ver=8&ver=9&ver=org.matrix.msc3787&ver=10&ver=org.matrix.msc2716v4
```


---

As a note, having no `body` is normal for the `/join` endpoint and it can handle it.

0c853e0970/synapse/rest/client/room.py (L398-L403)

Alternatively we could remove these extra logs but they are probably more usually helpful to figure out what went wrong.
2022-10-07 11:39:45 -05:00
Patrick Cloke e03d7c5fd0
Remove support for the unstable dir flag on relations. (#14106)
From MSC3715, this was unused by clients (and there was no
way for clients to know it was supported).

Matrix 1.4 defines the stable field.
2022-10-07 12:38:46 -04:00