Commit Graph

921 Commits (c369d82df0eac691ccb549051dd61dd77b83d1e9)

Author SHA1 Message Date
Dirk Klimpel a068ad7dd4
Add information on uploaded media to user export command. () 2023-02-23 13:14:17 -05:00
dependabot[bot] 9bb2eac719
Bump black from 22.12.0 to 23.1.0 () 2023-02-22 15:29:09 -05:00
Dirk Klimpel 6def779a1a
Use `json.dump` in `FileExfiltrationWriter` ()
To directly write to the open file, instead of writing to an
in-memory string first.
2023-02-22 14:29:39 -05:00
Dirk Klimpel 61bfcd669a
Add account data to export command ()
* Add account data to to export command

* newsfile

* remove not needed function

* update newsfile

* adopt 
2023-02-17 13:54:55 +00:00
Sean Quah d0c713cc85
Return read-only collections from `@cached` methods ()
It's important that collections returned from `@cached` methods are not
modified, otherwise future retrievals from the cache will return the
modified collection.

This applies to the return values from `@cached` methods and the values
inside the dictionaries returned by `@cachedList` methods. It's not
necessary for the dictionaries returned by `@cachedList` methods
themselves to be read-only.

Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:29:00 +00:00
realtyem 58214dbb9b
Allow enabling the asyncio reactor in complement ()
Signed-off-by: Jason Little realtyem@gmail.com
2023-02-01 23:42:45 +00:00
Dirk Klimpel bf82b56bab
Add more user information to export-data command. ()
* The user's profile information.
* The user's devices.
* The user's connections / IP address information.
2023-02-01 15:45:19 +00:00
reivilibre 22cc93afe3
Enable Faster Remote Room Joins against worker-mode Synapse. ()
* Enable Complement tests for Faster Remote Room Joins on worker-mode

* (dangerous) Add an override to allow Complement to use FRRJ under workers

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Fix race where we didn't send out replication notification

* MORE HACKS

* Fix get_un_partial_stated_rooms_token to take instance_name

* Fix bad merge

* Remove warning

* Correctly advance un_partial_stated_room_stream

* Fix merge

* Add another notify_replication

* Fixups

* Create a separate ReplicationNotifier

* Fix test

* Fix portdb

* Create a separate ReplicationNotifier

* Fix test

* Fix portdb

* Fix presence test

* Newsfile

* Apply suggestions from code review

* Update changelog.d/14752.misc

Co-authored-by: Erik Johnston <erik@matrix.org>

* lint

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
2023-01-22 21:10:11 +00:00
Dirk Klimpel 73f097888e
Add listener `health` ()
Fixes: 
2023-01-11 12:00:38 +00:00
Patrick Cloke f685318c2a
Use ClientRestResource on both the main process and workers. ()
Add logic to ClientRestResource to decide whether to mount servlets
or not based on whether the current process is a worker.

This is clearer to see what a worker runs than the completely separate /
copy & pasted list of servlets being mounted for workers.
2022-12-02 13:10:05 -05:00
reivilibre 9af2be192a
Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. () 2022-11-24 09:09:17 +00:00
realtyem c15e9a0edb
Remove need for `worker_main_http_uri` setting to use /keys/upload. () 2022-11-16 22:16:25 +00:00
David Robertson 258b5285b6
Fix typechecking errors introduced in ()
* Fix typechecking errors introduced in 

* Changelog

* Correct annotations

so that context_factory works if you don't use TLS
2022-11-15 16:36:43 +00:00
Tuomas Ojamies b5ab2c428a
Support using SSL on worker endpoints. ()
* Fix missing SSL support in worker endpoints.

* Add changelog

* SSL for Replication endpoint

* Remove unit test change

* Refactor listener creation to reduce duplicated code

* Fix the logger message

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add config documentation for new TLS option

Co-authored-by: Tuomas Ojamies <tojamies@palantir.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-11-15 12:55:00 +00:00
Nick Mills-Barrett 3a4f80f8c6
Merge/remove `Slaved*` stores into `WorkerStores` () 2022-11-11 10:51:49 +00:00
reivilibre b922b54b61
Fix type annotation causing import time error in the Complement forking launcher. ()
Co-authored-by: David Robertson <davidr@element.io>
2022-11-01 10:30:43 +00:00
David Robertson 730b13dbc9
Improve `RawHeaders` type hints () 2022-10-28 16:04:02 +00:00
Richard van der Hoff d24346f530
Fix logging error on SIGHUP () 2022-10-21 16:03:44 +01:00
Patrick Cloke 755bfeee3a
Use servlets for /key/ endpoints. ()
To fix the response for unknown endpoints under that prefix.

See MSC3743.
2022-10-20 11:32:47 -04:00
Patrick Cloke c604d2c218
Mark /relations endpoint as usable on workers. ()
Co-authored-by: Eric Eastwood <erice@element.io>
2022-10-12 06:46:13 -04:00
David Robertson 285d72556b
Update mypy and mypy-zope, attempt 3 ()
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30 17:36:28 +01:00
David Robertson 8e52cb0bce
Revert "Update mypy and mypy-zope ()"
This reverts commit 6d543d6d9f.
2022-09-30 16:37:48 +01:00
David Robertson 6d543d6d9f
Update mypy and mypy-zope ()
* Update mypy and mypy-zope

* Unignore assigning to LogRecord attributes

Presumably https://github.com/python/typeshed/pull/8064 makes this ok

Cherry-picked from 

* Remove unused ignores due to mypy ParamSpec fixes

https://github.com/python/mypy/pull/12668

Cherry-picked from 

* Remove additional unused ignores

* Fix new mypy complaints related to `assertGreater`

Presumably due to https://github.com/python/typeshed/pull/8077

* Changelog

* Reword changelog

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30 16:34:47 +01:00
Erik Johnston e8318a4333
Handle the case of remote users leaving a partial join room for device lists () 2022-09-27 13:01:08 +01:00
Richard van der Hoff d6b85a2a7d
Complement image: propagate SIGTERM to all workers ()
This should mean that logs from worker processes are flushed before shutdown.

When a test completes, Complement stops the docker container, which means that
synapse will receive a SIGTERM. Currently, the `complement_fork_starter` exits
immediately (without notifying the worker processes), which means that the
workers never get a chance to flush their logs before the whole container is
vaped. We can fix this by propagating the SIGTERM to the children.
2022-09-26 23:07:02 +01:00
reivilibre 526f84bc2e
Fix Prometheus recording rules to not use legacy metric names. () 2022-09-08 15:01:42 +01:00
Patrick Cloke 32fc3b7ba4
Remove configuration options for direct TCP replication. ()
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-06 07:50:02 +00:00
Brendan Abolivier 898fef2789
Share some metrics between the Prometheus exporter and the phone home stats () 2022-09-05 10:26:43 +00:00
reivilibre be4250c7a8
Add experimental configuration option to allow disabling legacy Prometheus metric names. ()
Co-authored-by: David Robertson <davidr@element.io>
2022-08-24 11:35:54 +00:00
David Robertson 956e015413
Drop support for delegating email validation, round 2 () 2022-08-23 11:40:00 +00:00
Ayush Anand 22ea51faf9
Add support for compression to federation responses ()
Closes .

Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
2022-08-18 15:14:47 +01:00
Sean Quah f4ab6a4a96
Refuse to start when `faster_joins` is enabled on a worker deployment ()
Synapse does not currently support faster room joins on deployments with
workers.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-16 16:21:55 +01:00
David Robertson 19e5d44886
Revert "Update locked versions of mypy and mypy-zope ()"
This reverts commit f383b9b3ec. Other PRs
were seeing mypy failures that looked to be related to mypy-zope.
Confusingly, we didn't see this on .

Revert this for now and investigate later.
2022-08-15 14:51:05 +01:00
David Robertson f383b9b3ec
Update locked versions of mypy and mypy-zope () 2022-08-15 11:32:30 +01:00
3nprob 98fb610cc0
Revert "Drop support for delegating email validation ()" ()
Reverts commit fa71bb18b5, and tweaks documentation.

Signed-off-by: 3nprob <git@3n.anonaddy.com>
2022-07-29 10:29:23 +00:00
Nick Mills-Barrett 86e366a46e
Remove old empty/redundant slaved stores. () 2022-07-21 17:56:45 +00:00
Nick Mills-Barrett 190f49d8ab
Use cache store remove base slaved ()
This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
2022-07-21 11:51:30 +01:00
Richard van der Hoff fa71bb18b5
Drop support for delegating email validation ()
* Drop support for delegating email validation

Delegating email validation to an IS is insecure (since it allows the owner of
the IS to do a password reset on your HS), and has long been deprecated. It
will now cause a config error at startup.

* Update unit test which checks for email verification

Give it an `email` config instead of a threepid delegate

* Remove unused method `requestEmailToken`

* Simplify config handling for email verification

Rather than an enum and a boolean, all we need here is a single bool, which
says whether we are or are not doing email verification.

* update docs

* changelog

* upgrade.md: fix typo

* update version number

this will be in 1.64, not 1.63

* update version number

this one too
2022-07-12 19:18:53 +01:00
Erik Johnston e610128c50
Add a `filter_event_for_clients_with_state` function () 2022-07-11 14:14:09 +01:00
reivilibre 9667bad55d
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30 11:58:12 +00:00
Brendan Abolivier 417f4cf40b
Don't use keyword arguments when initialising modules () 2022-06-15 15:36:16 +02:00
reivilibre 5b645ae2ad
Refactor entry points so that they all have a `main` function. () 2022-06-14 17:41:06 +00:00
David Robertson f30bcbd84a
Fix Synapse git info missing in version strings () 2022-06-07 15:24:11 +01:00
Patrick Cloke cf05258f76
Remove groups replication code. ()
The replication logic for groups is no longer used, so the message
passing infrastructure can be removed.
2022-05-31 13:04:08 -04:00
Patrick Cloke a8db8c6eba
Remove user-visible groups/communities code ()
Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove:

* All API endpoints (including Client-Server, Server-Server, and admin).
* Documented configuration options (and the experimental flag, which is now unused).
* Special handling during room upgrades.
* The `groups` section of the `/sync` response.
2022-05-25 07:53:40 -04:00
Nick Mills-Barrett 1f9013ce60
Add the `batch_send` endpoint to generic workers () 2022-05-25 09:51:07 +00:00
Nick Mills-Barrett 2e5f88b5e6
Add the `/account/whoami` endpoint to generic workers () 2022-05-25 10:41:41 +01:00
David Robertson d38d242411
Reload cache factors from disk on SIGHUP () 2022-05-11 13:43:22 +00:00
reivilibre 699192fc1a
Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ()
Co-authored-by: Shay <hillerys@element.io>
2022-05-10 11:08:45 +01:00
David Robertson fa0eab9c8e
Use `ParamSpec` in a few places () 2022-05-09 10:27:39 +00:00