Commit Graph

12119 Commits (e739b205885a7132f3dde67545d56655d83096d7)

Author SHA1 Message Date
Erik Johnston 3a3a618460 Use get_users_in_room rather than state handler in typing for speed 2020-07-15 15:42:07 +01:00
Erik Johnston f13061d515
Fix client reader sharding tests (#7853)
* Fix client reader sharding tests

* Newsfile

* Fix typing

* Update changelog.d/7853.misc

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Move mocking of http_client to tests

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-07-15 15:27:35 +01:00
Patrick Cloke b11450dedc
Convert E2E key and room key handlers to async/await. (#7851) 2020-07-15 08:48:58 -04:00
Patrick Cloke 111e70d75c
Return the proper 403 Forbidden error during errors with JWT logins. (#7844) 2020-07-15 07:10:21 -04:00
Richard van der Hoff 1d9dca02f9
remove `retry_on_integrity_error` wrapper for persist_events (#7848)
As far as I can tell from the sentry logs, the only time this has actually done
anything in the last two years is when we had two master workers running at
once, and even then, it made a bit of a mess of it (see
https://github.com/matrix-org/synapse/issues/7845#issuecomment-658238739).

Generally I feel like this code is doing more harm than good.
2020-07-15 10:34:53 +01:00
Patrick Cloke 8d0097bef1
Fix bug in per-room message retention policies. (#7850) 2020-07-14 15:51:13 -04:00
Brendan Abolivier 85223106f3
Allow email subjects to be customised through Synapse's configuration (#7846) 2020-07-14 19:10:42 +01:00
Dirk Klimpel 491f0dab1b
Add delete room admin endpoint (#7613)
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-14 12:36:23 +01:00
Patrick Cloke 77d2c05410
Add the option to validate the `iss` and `aud` claims for JWT logins. (#7827) 2020-07-14 07:16:43 -04:00
Patrick Cloke 4db1509516
Improve the type hints of synapse.api.errors. (#7820) 2020-07-14 07:03:58 -04:00
Erik Johnston f886a69916
Correctly pass app_name to all email templates. (#7829)
We didn't do this for e.g. registration emails.
2020-07-14 10:00:53 +01:00
Patrick Cloke 457096e6df
Support handling registration requests across multiple client readers. (#7830) 2020-07-13 13:31:46 -04:00
Brendan Abolivier 504c8f3483
Fix handling of "off" in encryption_enabled_by_default_for_room_type (#7822)
Fixes https://github.com/matrix-org/synapse/issues/7821, introduced in https://github.com/matrix-org/synapse/pull/7639

Turns out PyYAML translates `off` into a `False` boolean if it's
unquoted (see https://stackoverflow.com/questions/36463531/pyyaml-automatically-converting-certain-keys-to-boolean-values),
which seems to be a liberal interpretation of this bit of the YAML spec: https://yaml.org/spec/1.1/current.html#id864510

An alternative fix would be to implement the solution mentioned in the
SO post linked above, but I'm aware it might break existing setups
(which might use these values in the configuration file) so it's
probably better just to add an extra check for this one. We should be
aware that this is a thing for the next times we do that though.

I didn't find any other occurrence of this bug elsewhere in the
codebase.
2020-07-13 17:14:42 +01:00
Richard van der Hoff 59e64b6d5b Merge branch 'master' into develop 2020-07-13 11:42:52 +01:00
Richard van der Hoff 29df3d0e9f 1.17.0 2020-07-13 10:20:36 +01:00
Patrick Cloke 66a4af8d96
Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802) 2020-07-10 14:30:08 -04:00
Patrick Cloke d9e47af617
Add types to the server code and remove unused parameter (#7813) 2020-07-10 14:28:42 -04:00
Sorunome 1bca21e1da
Include room states on invite events sent to ASes (#6455) 2020-07-10 18:44:56 +01:00
Richard van der Hoff 6cef918a4b Merge branch 'release-v1.17.0' into develop 2020-07-10 18:38:50 +01:00
Richard van der Hoff 8ccb7f08d9 Merge branch 'master' into release-v1.17.0 2020-07-10 18:38:18 +01:00
Erik Johnston f299441cc6
Add ability to shard the federation sender (#7798) 2020-07-10 18:26:36 +01:00
Erik Johnston f1245dc3c0
Fix resync remote devices on receive PDU in worker mode. (#7815)
The replication client requires that arguments are given as keyword
arguments, which was not done in this case. We also pull out the logic
so that we can catch and handle any exceptions raised, rather than
leaving them unhandled.
2020-07-10 18:23:17 +01:00
Erik Johnston e29c44340b
Fix recursion error when fetching auth chain over federation (#7817)
When fetching the state of a room over federation we receive the event
IDs of the state and auth chain. We then fetch those events that we
don't already have.

However, we used a function that recursively fetched any missing auth
events for the fetched events, which can lead to a lot of recursion if
the server is missing most of the auth chain. This work is entirely
pointless because would have queued up the missing events in the auth
chain to be fetched already.

Let's just diable the recursion, since it only gets called from one
place anyway.
2020-07-10 18:15:35 +01:00
Richard van der Hoff b1beb3ff59 fix migration, again 2020-07-10 12:18:35 +01:00
Richard van der Hoff c9f7c683ae 1.16.1 2020-07-10 12:11:12 +01:00
Richard van der Hoff cbabcec05c
Drop incorrectly-added table `local_rejections_stream`. (#7816) 2020-07-10 12:07:59 +01:00
Richard van der Hoff 43726783e4 1.17.0rc1 2020-07-09 16:53:19 +01:00
Patrick Cloke 38e1fac886
Fix some spelling mistakes / typos. (#7811) 2020-07-09 09:52:58 -04:00
Richard van der Hoff 53ee214f2f
`update_membership` declaration: now always returns an event id. (#7809) 2020-07-09 13:01:42 +01:00
Richard van der Hoff 8ca39bd2c3
Improve stacktraces from exceptions in background processes (#7808)
use `Failure()` to fish out the real exception.
2020-07-09 13:01:33 +01:00
Richard van der Hoff 08c5181a8d
Fix `can only concatenate list (not "tuple") to list` exception (#7810)
It seems auth_events can be either a list or a tuple, depending on Things.
2020-07-09 12:48:15 +01:00
Patrick Cloke 8fa7fdd4cb
Pass original request headers from workers to the main process. (#7797) 2020-07-09 07:34:46 -04:00
Richard van der Hoff 2ab0b021f1
Generate real events when we reject invites (#7804)
Fixes #2181. 

The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
2020-07-09 10:40:19 +01:00
Richard van der Hoff 67593b1728
Add `HomeServer.signing_key` property (#7805)
... instead of duplicating `config.signing_key[0]` everywhere
2020-07-08 17:51:56 +01:00
Patrick Cloke e7efd8f827
Do not use simplejson in Synapse. (#7800) 2020-07-08 07:15:08 -04:00
Patrick Cloke ff0680f69d
Stop passing bytes when dumping JSON (#7799) 2020-07-08 07:14:56 -04:00
Richard van der Hoff 59ddcd790b Merge branch 'master' into develop 2020-07-08 11:25:34 +01:00
Richard van der Hoff 98894341e7 1.16.0 2020-07-08 11:03:55 +01:00
Richard van der Hoff 76dbd7b8d6
Stop populating unused table `local_invites`. (#7793)
This table is no longer used, so we may as well stop populating it. Removing it
would prevent people rolling back to older releases of Synapse, so that can
happen in a future release.
2020-07-07 14:20:40 +01:00
Erik Johnston 67d7756fcf
Refactor getting replication updates from database v2. (#7740) 2020-07-07 12:11:35 +01:00
Patrick Cloke 2a266f4511
Add documentation for JWT login type and improve sample config. (#7776) 2020-07-06 08:31:51 -04:00
Patrick Cloke 6d687ebba1
Convert the appservice handler to async/await. (#7775) 2020-07-06 07:40:35 -04:00
reivilibre 57feeab364
Don't ignore `set_tweak` actions with no explicit `value`. (#7766)
* Fix spec compliance; tweaks without values are valid

(default to True, which is only concretely specified for
`highlight`, but it seems only reasonable to generalise)

* Changelog for 7766.

* Add documentation to `tweaks_for_actions`

May as well tidy up when I'm here.

* Add a test for `tweaks_for_actions`
2020-07-06 11:43:41 +01:00
Oliver Kurz 4e118742ca
Allow to use higher versions of prometheus_client (#7780)
Fixes https://github.com/matrix-org/synapse/issues/7641

The package was pinned to <0.8.0 without an obvious reasoning with
7ad1d7635
in https://github.com/matrix-org/synapse/pull/5636
while the version selection looks to just try to exclude an arbitrary
next minor version number that might introduce API breaking changes.
Selecting the next minor number might be a good conservative selection.

Downstream distributions already reported success patching out the version
requirements.

This also fixes the integration of upgraded packages into openSUSE packages,
e.g. for openSUSE Tumbleweed which already ships prometheus_client >= 0.8 .

Signed-off-by: Oliver Kurz <okurz@suse.de>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-07-06 10:21:41 +01:00
Will Hunt 62b1ce8539
isort 5 compatibility (#7786)
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
2020-07-05 16:32:02 +01:00
Erik Johnston 5cdca53aa0
Merge different Resource implementation classes (#7732) 2020-07-03 19:02:19 +01:00
Dirk Klimpel 21a212f8e5
Fix inconsistent handling of upper and lower cases of email addresses. (#7021)
fixes #7016
2020-07-03 14:03:13 +01:00
Patrick Cloke f3e0f16240 Synapse 1.16.0rc2 (2020-07-02)
==============================
 
 Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
 Please see [below](https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1152-2020-07-02) for more details.
 
 Improved Documentation
 ----------------------
 
 - Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))
 
 Internal Changes
 ----------------
 
 - Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl79+qgACgkQM/xY9qcR
 MEhcaRAAjWLW3ojN1F0DUfE85jziZK2VdnMQC3g+uEOLX6QRbfqFNaNNMjLdK+vl
 K/+2ZoHkRsg6g8noSPhPmI1z1+hb5xDJaxjltzHxonIipW8XSU8o2PQMkf8O/BAy
 VS58y3GyLkhEgzWC+/hcII+LBgcqXpLuNM0xrKTHmxclIjdewlwe1v+hxkP+6wsX
 9Whhn1f4sNHrCtyFVK9uzMFcVyzcQaiWZRjEDMj2uR7rWT6UbCUifN/G4fWmtGbY
 xWoNoC4Qv8xiqXOG4U7juPp9T3bRyWMKyjBFM5PWO6Ec2zfafDyFzhBxJhlQhODG
 g21tS4PowX/dM/pBpJFEOPh1BVrPZzzTD+YMmTcd3NO79HeaQGqEX/+tzFCFUyPp
 0daJK3Y85+l5w/M09WU8DDN8CiR3PFJyGDIZp+nweMsiJZkbEbLOkh1tx6TL+5/6
 zwewU6cq8nTVGrn53Tn58l8C7Sj4w+Qk+1XDzymAoidyoWqAKW9Y/fw53PaViUSx
 voDu0rpsEUXR1OzCBG8SAPQCFy9gdEWV04OvIpzHuq2uojkz66f7NAXy+Wz+Occ9
 AYb/s6Ei80bGCLgRd5jg+myqavwRbzCyv+LIC6dxpopxZJ3AzrFuD11eXKtrIxOC
 FZYf3U4KeBk4Q9TV5IFV1xcGFrq5aK36LdmP6WOsEl3PXVT9p/Q=
 =YaJn
 -----END PGP SIGNATURE-----

Merge tag 'v1.16.0rc2' into develop

Synapse 1.16.0rc2 (2020-07-02)
==============================

Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
Please see [below](https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1152-2020-07-02) for more details.

Improved Documentation
----------------------

- Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))

Internal Changes
----------------

- Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
2020-07-02 11:25:56 -04:00
Patrick Cloke 1319e53251 1.16.0rc2 2020-07-02 11:06:35 -04:00
Patrick Cloke f2bcc6ecbf Merge branch 'master' into release-v1.16.0 2020-07-02 11:02:42 -04:00
Patrick Cloke 4d978d7db4 Merge branch 'master' into develop 2020-07-02 10:55:41 -04:00
Patrick Cloke e8c36e527d 1.15.2 2020-07-02 10:35:59 -04:00
Erik Johnston 96e9afe625 Correctly handle outliers as prev events over federation 2020-07-02 10:00:33 -04:00
Patrick Cloke ea26e9a98b Ensure that HTML pages served from Synapse include headers to avoid embedding. 2020-07-02 09:58:31 -04:00
reivilibre e5808c4cfb
Hack to add push priority to push notifications (#7765)
* Remove obsolete comment about ancient temporary code

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Implement hack to set push priority

based on whether the tweaks indicate the event might cause
effects.

* Changelog for 7765

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Antilint

* Add tests for push priority

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Update synapse/push/httppusher.py

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Antilint

* Remove needless invites from tests.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2020-07-01 17:02:31 +01:00
Richard van der Hoff e866512367
Add early returns to `_check_for_soft_fail` (#7769)
my editor was complaining about unset variables, so let's add some early
returns to fix that and reduce indentation/cognitive load.
2020-07-01 16:41:19 +01:00
Richard van der Hoff f01e2ca039
Use symbolic names for replication stream names (#7768)
This makes it much easier to find where streams are referenced.
2020-07-01 16:35:40 +01:00
Richard van der Hoff a6eae69ffe
Type checking for `FederationHandler` (#7770)
fix a few things to make this pass mypy.
2020-07-01 16:21:02 +01:00
Erik Johnston 1e03513f9a
Fix new metric where we used ms instead of seconds (#7771)
Introduced in #7755, not yet released.
2020-07-01 15:23:58 +01:00
Richard van der Hoff 244dbb04f7
Fix incorrect error message when database CTYPE was set incorrectly. (#7760) 2020-07-01 13:56:16 +01:00
Andrew Morgan dc80a0762d 1.16.0rc1 2020-07-01 11:26:58 +01:00
Brendan Abolivier 74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Patrick Cloke 71cccf1593
Additional configuration options for auto-join rooms (#7763) 2020-06-30 15:41:36 -04:00
Erik Johnston a99658074d
Add some metrics for inbound and outbound federation processing times (#7755) 2020-06-30 16:58:06 +01:00
Andrew Morgan 2f6afdd8b4
Explain the purpose of the "tests" conditional dependency requirement (#7751) 2020-06-30 10:11:36 +01:00
Erik Johnston 831b31e563
Add another yield point to state res v2 (#7746) 2020-06-26 10:44:52 +01:00
Erik Johnston 0e0a2817a2
Yield during large v2 state res. (#7735)
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.

To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
2020-06-24 18:48:18 +01:00
Sorunome 6920e58136
add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675) 2020-06-24 10:23:55 +01:00
Christian Svensson 8bbe87f42d
Set Content-Length for Metrics requests (#7730)
HTTP requires the response to contain a Content-Length header unless chunked encoding is being used.
Prometheus metrics endpoint did not set this, causing software such as prometheus-proxy to not be able to scrape synapse for metrics.

Signed-off-by: Christian Svensson <blue@cmd.nu>
2020-06-23 18:06:01 +01:00
Patrick Cloke 95e41f368b
Allow local media to be marked as safe from being quarantined. (#7718) 2020-06-22 08:04:14 -04:00
Patrick Cloke e060bf4462
Convert directory handler to async/await (#7727) 2020-06-22 07:18:00 -04:00
Erik Johnston 91e886d615
Speed up state res v2 across large state differences. (#7725) 2020-06-19 13:56:35 +01:00
Richard van der Hoff 7d2824395f add a comment 2020-06-18 10:47:06 +01:00
Brendan Abolivier e35d44c01d
Merge pull request #7716 from matrix-org/babolivier/unread_fix
Fix unread counts in sync
2020-06-17 15:44:15 +01:00
Patrick Cloke 3630825612
Convert the typing handler to async/await. (#7679) 2020-06-17 10:37:59 -04:00
Oleg Girko 96bc110a68
Require parameterized package version to be at least 0.7.0. (#7680)
Older versions of `parameterized` package have no `parameterized_class` decorator. This decorator is used in tests.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2020-06-17 15:31:40 +01:00
Brendan Abolivier 5a5cf6460e
Fix unread counts in sync
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
2020-06-17 15:10:44 +01:00
Erik Johnston e07a8caf58
Add support for using rust-python-jaeger-reporter (#7697) 2020-06-17 14:13:41 +01:00
Erik Johnston b44bdd7f7b
Support running multiple media repos. (#7706)
This requires a new config option to specify which media repo should be
responsible for running background jobs to e.g. clear out expired URL
preview caches.
2020-06-17 14:13:30 +01:00
Patrick Cloke 434716e1d3
Fetch from the r0 media path instead of the unspecced v1. (#7714) 2020-06-17 08:36:46 -04:00
Brendan Abolivier 46613aaf79
Implement unread counter (MSC2625) (#7673)
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
Richard van der Hoff e452973fd2
fix broken link in sample config (#7712) 2020-06-16 19:50:16 +01:00
Erik Johnston f6f7511a4c
Refactor getting replication updates from database. (#7636)
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
2020-06-16 17:10:28 +01:00
Patrick Cloke 231252516c
Fix "argument of type 'ObservableDeferred' is not iterable" error (#7708) 2020-06-16 12:01:18 -04:00
Patrick Cloke ac51bd581a
Include a user agent in federation requests. (#7677) 2020-06-16 10:43:29 -04:00
Dagfinn Ilmari Mannsåker a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke 98c4e35e3c
Convert the device message and pagination handlers to async/await. (#7678) 2020-06-16 08:06:17 -04:00
Richard van der Hoff 03619324fc
Create a ListenerConfig object (#7681)
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Brendan Abolivier 789606577a Merge branch 'master' into develop 2020-06-16 10:36:29 +01:00
Brendan Abolivier 65eb078498
1.15.1 2020-06-16 10:28:58 +01:00
Andrew Morgan 3e6b5bba71
Wrap register_device coroutine in an ensureDeferred (#7684)
Fixes https://github.com/matrix-org/synapse/issues/7683

Broke in: #7649

We had a `yield` acting on a coroutine. To be fair this one is a bit difficult to notice as there's a function in the middle that just passes the coroutine along.
2020-06-16 10:13:59 +01:00
Patrick Cloke cc32fa7358
Ensure the body is a string before comparing push rules. (#7701) 2020-06-15 16:20:34 -04:00
Hubert Chathi 2b2344652b
Ensure etag is a string for GET room_keys/version response (#7691) 2020-06-15 13:42:44 -04:00
Andrew Morgan b8ee03caff
Update m.id.phone to use 'phone' instead of 'number' (#7687)
The spec [states](https://matrix.org/docs/spec/client_server/r0.6.1#phone-number) that `m.id.phone` requires the field `country` and `phone`.

In Synapse, we've been enforcing `country` and `number`.

I am not currently sure whether this affects any client implementations.

This issue was introduced in #1994.
2020-06-15 17:46:23 +01:00
Erik Johnston 4241a10673
Fix "There was no active span when trying to log." error (#7698) 2020-06-15 16:42:21 +01:00
Brendan Abolivier 6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Brendan Abolivier c2b4621630 Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:08 +01:00
Brendan Abolivier 6d5985e1f2
Enable 3PID add/bind/unbind endpoints on r0 routes 2020-06-15 16:14:13 +01:00
Patrick Cloke 7d2532be36
Discard RDATA from already seen positions. (#7648) 2020-06-15 08:44:54 -04:00
Patrick Cloke bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00