89700dfb8c
Check support room has only two users before sending a notice ( #8728 )
...
* Check support room has only two users
* Create 8728.bugfix
* Update synapse/server_notices/server_notices_manager.py
Co-authored-by: Erik Johnston <erik@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-11-11 14:23:16 +00:00
eedaf90c84
Better error message when a remote resource uses invalid Content-Type ( #8719 )
2020-11-11 14:22:40 +00:00
4c7587ef99
Catch exceptions in password_providers ( #8636 )
...
Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
2020-11-11 13:24:53 +00:00
fb56dfdccd
Fix SIGHUP handler ( #8697 )
...
Fixes:
```
builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given
```
2020-11-06 11:42:07 +00:00
c3119d1536
Add an admin API for users' media statistics ( #8700 )
...
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users.
Related to #6094
It is the first API for statistics.
Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats )
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-11-05 18:59:12 +00:00
e4676bd877
Add `displayname` to Shared-Secret Registration for admins ( #8722 )
...
Add `displayname` to Shared-Secret Registration for admins to `POST /_synapse/admin/v1/register`
2020-11-05 13:55:45 +00:00
243d427fbc
Block clients from sending server ACLs that lock the local server out. ( #8708 )
...
Fixes #4042
2020-11-03 12:13:48 +00:00
e89bd3ea92
Improve error messages of non-str displayname/avatar_url ( #8705 )
...
This PR fixes two things:
* Corrects the copy/paste error of telling the client their displayname is wrong when they are submitting an `avatar_url`.
* Returns a `M_INVALID_PARAM` instead of `M_UNKNOWN` for non-str type parameters.
Reported by @t3chguy.
2020-11-02 18:01:09 +00:00
59cc2472b3
Add base pushrule to notify for jitsi conferences ( #8286 )
...
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
2020-11-02 16:36:14 +00:00
1eb9de90c0
Improve start time by adding index to e2e_cross_signing_keys ( #8694 )
...
We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
2020-11-02 13:55:56 +00:00
7a0fd6f98d
Fix error handling around when completing an AS transaction ( #8693 )
2020-10-30 16:50:48 +00:00
f27a789697
Merge branch 'master' into develop
2020-10-30 16:27:02 +00:00
aef8514193
use correct version
2020-10-30 15:27:58 +00:00
90c900a8ff
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
...
Fix serialisation errors when using third-party event rules.
2020-10-30 15:24:10 +00:00
b37aa1643b
Tie together matches_user_in_member_list and get_users_in_room caches ( #8676 )
...
* Tie together matches_user_in_member_list and get_users_in_room
* changelog
* Remove type to fix mypy
* Add `on_invalidate` to the function signature in the hopes that may make things work well
* Remove **kwargs
* Update 8676.bugfix
2020-10-30 15:23:16 +00:00
8f1aefa694
Improve the sample config for SSO (OIDC, SAML, and CAS). ( #8635 )
2020-10-30 10:01:59 -04:00
cbc82aa09f
Implement and use an @lru_cache decorator ( #8595 )
...
We don't always need the full power of a DeferredCache.
2020-10-30 11:43:17 +00:00
46f4be94b4
Fix race for concurrent downloads of remote media. ( #8682 )
...
Fixes #6755
2020-10-30 10:55:24 +00:00
4504151546
Fix optional parameter in stripped state storage method ( #8688 )
...
Missed in #8671 .
2020-10-30 00:22:31 +00:00
70269fbd18
Tie together matches_user_in_member_list and get_users_in_room caches ( #8676 )
...
* Tie together matches_user_in_member_list and get_users_in_room
* changelog
* Remove type to fix mypy
* Add `on_invalidate` to the function signature in the hopes that may make things work well
* Remove **kwargs
* Update 8676.bugfix
2020-10-29 16:58:16 +00:00
8b42a4eefd
Gracefully handle a pending logging connection during shutdown. ( #8685 )
2020-10-29 12:53:57 -04:00
f21e24ffc2
Add ability for access tokens to belong to one user but grant access to another user. ( #8616 )
...
We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).
A future PR will add an API for creating such a token.
When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29 15:58:44 +00:00
22eeb6bc54
Fix cache call signature to accept `on_invalidate`. ( #8684 )
...
Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.
2020-10-29 15:18:17 +00:00
0073fe914a
Use `%r` rather than `%s` for stringifying events ( #8679 )
...
otherwise non-state events get written as `<FrozenEvent ... state_key='None'>`
which is indistinguishable from state events with the actual state_key `None`.
2020-10-29 12:16:49 +00:00
56f0ee78a9
Optimise createRoom with multiple invites ( #8559 )
...
By not dropping the membership lock between invites, we can stop joins from
grabbing the lock when we're half-done and slowing the whole thing down.
2020-10-29 11:48:39 +00:00
00b24aa545
Support generating structured logs in addition to standard logs. ( #8607 )
...
This modifies the configuration of structured logging to be usable from
the standard Python logging configuration.
This also separates the formatting of logs from the transport allowing
JSON logs to files or standard logs to sockets.
2020-10-29 07:27:37 -04:00
c97da1e45d
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
...
Fix serialisation errors when using third-party event rules.
2020-10-28 20:41:42 +00:00
e80eb69887
remove unused imports
2020-10-28 16:18:05 +00:00
b6ca69e4f1
Remove frozendict_json_encoder and support frozendicts everywhere
...
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
2020-10-28 15:56:57 +00:00
31d721fbf6
Add type hints to application services. ( #8655 )
2020-10-28 11:12:21 -04:00
2239813278
Add an admin APIs to allow server admins to list users' pushers ( #8610 )
...
Add an admin API `GET /_synapse/admin/v1/users/<user_id>/pushers` like https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers
2020-10-28 15:02:42 +00:00
a6ea1a957e
Don't pull event from DB when handling replication traffic. ( #8669 )
...
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-28 12:11:45 +00:00
aff1eb7c67
Tell Black to format code for Python 3.5 ( #8664 )
...
This allows trailing commas in multi-line arg lists.
Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.
Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27 23:26:36 +00:00
a699c044b6
Abstract code for stripping room state into a separate method ( #8671 )
...
This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739 ), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change.
For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid ) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included.
The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event.
This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem.
In essence this is just moving some inline code to a reusable store method.
2020-10-27 18:42:46 +00:00
4215a3acd4
Don't unnecessarily start bg process in replication sending loop. ( #8670 )
2020-10-27 17:37:08 +00:00
0c7f9cb81f
Don't unnecessarily start bg process while handling typing. ( #8668 )
...
There's no point starting a background process when all its going to do is bail if federation isn't enabled.
2020-10-27 15:32:19 +00:00
9b7c28283a
Add admin API to list users' local media ( #8647 )
...
Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
2020-10-27 14:12:31 +00:00
24229fac05
Merge branch 'master' into develop
2020-10-27 12:12:54 +00:00
fedfdfd750
1.22.0
2020-10-27 12:07:19 +00:00
2e380f0f18
e2e: ensure we have both master and self-signing key ( #8455 )
...
it seems to be possible that only one of them ends up to be cached.
when this was the case, the missing one was not fetched via federation,
and clients then failed to validate cross-signed devices.
Signed-off-by: Jonas Jelten <jj@sft.lol>
2020-10-26 18:37:47 +00:00
10f45d85bb
Add type hints for account validity handler ( #8620 )
...
This also fixes a bug by fixing handling of an account which doesn't expire.
2020-10-26 14:17:31 -04:00
66e6801c3e
Split admin API for reported events into a detail and a list view ( #8539 )
...
Split admin API for reported events in detail und list view.
API was introduced with #8217 in synapse v.1.21.0.
It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview.
The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`.
It is similar to room and users API.
It is a kind of regression in `GET /_synapse/admin/v1/event_reports`. `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec).
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-10-26 18:16:37 +00:00
49d72dea2a
Add an admin api to delete local media. ( #8519 )
...
Related to: #6459 , #3479
Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete
a single file from server.
2020-10-26 17:02:28 +00:00
4ac3a8c5dc
Fix a bug in the joined_rooms admin API ( #8643 )
...
If the user was not in any rooms then the API returned the same error
as if the user did not exist.
2020-10-26 12:25:48 -04:00
cf9a17a2b3
Synapse 1.22.0rc2 (2020-10-26)
...
==============================
Bugfixes
--------
- Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. ([\#8648](https://github.com/matrix-org/synapse/issues/8648 ), [\#8656](https://github.com/matrix-org/synapse/issues/8656 ))
- Fix `user_daily_visits` table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. ([\#8654](https://github.com/matrix-org/synapse/issues/8654 ))
-----BEGIN PGP SIGNATURE-----
iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAl+W6QIQHGVyaWtAbWF0
cml4Lm9yZwAKCRClQuTtGw+sCR/EB/9Afs/QiqD7uJ7XtAASwlOxcpSrvB6xSMzU
VIiYLrUSCvHe8SwcPG6nToy69hHmwLBPu9e8gvlQkOY4nJpW9VTGvhFcP5eCgeKd
vDp+elVD6BPg+6S3RaI/F5hyQ7gn9qZ/WexkVixDkVs5eAL3gNlZcjw9RwaT9ei9
fEOYlMi1CRSRzKVBPMEFCAvOpJJ8rA/iq4JK7+P0TplkhD+HydG0cYr6OoL7CmaD
L/au565Ynp6o0hSpNpTFqW6C5W5JRvz/TyG9/N95h9+WqubZzsPPDUzh98B3eBoX
43Wu9Qoc3mNamf0++HbHGUPI8dRV1gssxL01hs/iLUZz7CxXY+Ir
=Z3sQ
-----END PGP SIGNATURE-----
Merge tag 'v1.22.0rc2' into develop
Synapse 1.22.0rc2 (2020-10-26)
==============================
Bugfixes
--------
- Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. ([\#8648](https://github.com/matrix-org/synapse/issues/8648 ), [\#8656](https://github.com/matrix-org/synapse/issues/8656 ))
- Fix `user_daily_visits` table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. ([\#8654](https://github.com/matrix-org/synapse/issues/8654 ))
2020-10-26 15:23:13 +00:00
7a3adbd7af
1.22.0rc2
2020-10-26 15:11:03 +00:00
ff7f0e8a14
Merge branch 'release-v1.22.0' into develop
2020-10-26 15:02:55 +00:00
9e0f5a0ac4
Fix get|set_type_stream_id_for_appservice store functions ( #8648 )
2020-10-26 15:00:56 +00:00
e8dbbcb64c
Fix get|set_type_stream_id_for_appservice store functions ( #8648 )
2020-10-26 10:51:33 -04:00
913f8a06e4
Add field `total` to device list in admin API ( #8644 )
2020-10-26 14:07:51 +00:00
7b13780c54
Check status codes that profile handler returns ( #8580 )
...
Fixes #8520
Signed-off-by: Pavel Turinsky <pavel.turinsky@matfyz.cz>
Co-authored-by: Erik Johnston <erikj@jki.re>
2020-10-26 13:55:21 +00:00
5eda018561
Properly handle presence events for application services. ( #8656 )
2020-10-26 09:19:07 -04:00
437a99fb99
Fix user_daily_visits to not have duplicate rows for UA. ( #8654 )
...
* Fix user_daily_visits to not have duplicate rows for UA.
Fixes #8641 .
* Newsfile
* Fix typo.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-26 13:16:32 +00:00
2b7c180879
Start fewer opentracing spans ( #8640 )
...
#8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.
This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.
I don't really know how much this will help.
2020-10-26 09:30:19 +00:00
34a5696f93
Fix typos and spelling errors. ( #8639 )
2020-10-23 12:38:40 -04:00
c850dd9a8e
Fix handling of User-Agent headers with bad utf-8. ( #8632 )
2020-10-23 17:12:59 +01:00
db9ef792f0
Fix email notifications for invites without local state. ( #8627 )
...
This can happen if e.g. the room invited into is no longer on the
server (or if all users left the room).
2020-10-23 10:41:32 +01:00
054a6b9538
Synapse 1.22.0rc1 (2020-10-22)
...
==============================
Features
--------
- Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](https://github.com/matrix-org/synapse/issues/7658 ))
- Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](https://github.com/matrix-org/synapse/issues/8292 ), [\#8467](https://github.com/matrix-org/synapse/issues/8467 ))
- Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732 )). ([\#8312](https://github.com/matrix-org/synapse/issues/8312 ), [\#8501](https://github.com/matrix-org/synapse/issues/8501 ))
- Add support for running background tasks in a separate worker process. ([\#8369](https://github.com/matrix-org/synapse/issues/8369 ), [\#8458](https://github.com/matrix-org/synapse/issues/8458 ), [\#8489](https://github.com/matrix-org/synapse/issues/8489 ), [\#8513](https://github.com/matrix-org/synapse/issues/8513 ), [\#8544](https://github.com/matrix-org/synapse/issues/8544 ), [\#8599](https://github.com/matrix-org/synapse/issues/8599 ))
- Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697 )). ([\#8380](https://github.com/matrix-org/synapse/issues/8380 ))
- Add support for [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409 ), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](https://github.com/matrix-org/synapse/issues/8437 ), [\#8590](https://github.com/matrix-org/synapse/issues/8590 ))
- Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788 ). ([\#8461](https://github.com/matrix-org/synapse/issues/8461 ))
- Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](https://github.com/matrix-org/synapse/issues/8479 ))
- Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](https://github.com/matrix-org/synapse/issues/8502 ))
- Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](https://github.com/matrix-org/synapse/issues/8535 ), [\#8564](https://github.com/matrix-org/synapse/issues/8564 ))
Bugfixes
--------
- Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](https://github.com/matrix-org/synapse/issues/8454 ))
- Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](https://github.com/matrix-org/synapse/issues/8457 ))
- Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](https://github.com/matrix-org/synapse/issues/8465 ))
- Fix message duplication if something goes wrong after persisting the event. ([\#8476](https://github.com/matrix-org/synapse/issues/8476 ))
- Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](https://github.com/matrix-org/synapse/issues/8486 ))
- Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](https://github.com/matrix-org/synapse/issues/8504 ))
- Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](https://github.com/matrix-org/synapse/issues/8517 ))
- Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](https://github.com/matrix-org/synapse/issues/8527 ))
- Fix not sending events over federation when using sharded event writers. ([\#8536](https://github.com/matrix-org/synapse/issues/8536 ))
- Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](https://github.com/matrix-org/synapse/issues/8545 ))
- Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](https://github.com/matrix-org/synapse/issues/8567 ))
- Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](https://github.com/matrix-org/synapse/issues/8585 ))
- Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](https://github.com/matrix-org/synapse/issues/8606 ))
Updates to the Docker image
---------------------------
- Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](https://github.com/matrix-org/synapse/issues/7921 ))
- Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](https://github.com/matrix-org/synapse/issues/8390 ))
Improved Documentation
----------------------
- Update the directions for using the manhole with coroutines. ([\#8462](https://github.com/matrix-org/synapse/issues/8462 ))
- Improve readme by adding new shield.io badges. ([\#8493](https://github.com/matrix-org/synapse/issues/8493 ))
- Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. ([\#8526](https://github.com/matrix-org/synapse/issues/8526 ))
- Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](https://github.com/matrix-org/synapse/issues/8529 ))
Deprecations and Removals
-------------------------
- Drop unused `device_max_stream_id` table. ([\#8589](https://github.com/matrix-org/synapse/issues/8589 ))
Internal Changes
----------------
- Check for unreachable code with mypy. ([\#8432](https://github.com/matrix-org/synapse/issues/8432 ))
- Add unit test for event persister sharding. ([\#8433](https://github.com/matrix-org/synapse/issues/8433 ))
- Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](https://github.com/matrix-org/synapse/issues/8439 ), [\#8488](https://github.com/matrix-org/synapse/issues/8488 ), [\#8496](https://github.com/matrix-org/synapse/issues/8496 ), [\#8499](https://github.com/matrix-org/synapse/issues/8499 ))
- Configure `public_baseurl` when using demo scripts. ([\#8443](https://github.com/matrix-org/synapse/issues/8443 ))
- Add SQL logging on queries that happen during startup. ([\#8448](https://github.com/matrix-org/synapse/issues/8448 ))
- Speed up unit tests when using PostgreSQL. ([\#8450](https://github.com/matrix-org/synapse/issues/8450 ))
- Remove redundant database loads of stream_ordering for events we already have. ([\#8452](https://github.com/matrix-org/synapse/issues/8452 ))
- Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](https://github.com/matrix-org/synapse/issues/8463 ))
- Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](https://github.com/matrix-org/synapse/issues/8464 ))
- Additional testing for `ThirdPartyEventRules`. ([\#8468](https://github.com/matrix-org/synapse/issues/8468 ))
- Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](https://github.com/matrix-org/synapse/issues/8472 ))
- Unblacklist some sytests. ([\#8474](https://github.com/matrix-org/synapse/issues/8474 ))
- Include the log level in the phone home stats. ([\#8477](https://github.com/matrix-org/synapse/issues/8477 ))
- Remove outdated sphinx documentation, scripts and configuration. ([\#8480](https://github.com/matrix-org/synapse/issues/8480 ))
- Clarify error message when plugin config parsers raise an error. ([\#8492](https://github.com/matrix-org/synapse/issues/8492 ))
- Remove the deprecated `Handlers` object. ([\#8494](https://github.com/matrix-org/synapse/issues/8494 ))
- Fix a threadsafety bug in unit tests. ([\#8497](https://github.com/matrix-org/synapse/issues/8497 ))
- Add user agent to user_daily_visits table. ([\#8503](https://github.com/matrix-org/synapse/issues/8503 ))
- Add type hints to various parts of the code base. ([\#8407](https://github.com/matrix-org/synapse/issues/8407 ), [\#8505](https://github.com/matrix-org/synapse/issues/8505 ), [\#8507](https://github.com/matrix-org/synapse/issues/8507 ), [\#8547](https://github.com/matrix-org/synapse/issues/8547 ), [\#8562](https://github.com/matrix-org/synapse/issues/8562 ), [\#8609](https://github.com/matrix-org/synapse/issues/8609 ))
- Remove unused code from the test framework. ([\#8514](https://github.com/matrix-org/synapse/issues/8514 ))
- Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](https://github.com/matrix-org/synapse/issues/8515 ))
- Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](https://github.com/matrix-org/synapse/issues/8537 ))
- Improve database performance by executing more queries without starting transactions. ([\#8542](https://github.com/matrix-org/synapse/issues/8542 ))
- Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](https://github.com/matrix-org/synapse/issues/8548 ))
- Move metric registration code down into `LruCache`. ([\#8561](https://github.com/matrix-org/synapse/issues/8561 ), [\#8591](https://github.com/matrix-org/synapse/issues/8591 ))
- Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](https://github.com/matrix-org/synapse/issues/8563 ))
- Add virtualenv-generated folders to `.gitignore`. ([\#8566](https://github.com/matrix-org/synapse/issues/8566 ))
- Add `get_immediate` method to `DeferredCache`. ([\#8568](https://github.com/matrix-org/synapse/issues/8568 ))
- Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](https://github.com/matrix-org/synapse/issues/8569 ))
- Fix `synmark` benchmark runner. ([\#8571](https://github.com/matrix-org/synapse/issues/8571 ))
- Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](https://github.com/matrix-org/synapse/issues/8572 ))
- Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](https://github.com/matrix-org/synapse/issues/8577 ))
- Support macOS on the `synmark` benchmark runner. ([\#8578](https://github.com/matrix-org/synapse/issues/8578 ))
- Update `mypy` static type checker to 0.790. ([\#8583](https://github.com/matrix-org/synapse/issues/8583 ), [\#8600](https://github.com/matrix-org/synapse/issues/8600 ))
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](https://github.com/matrix-org/synapse/issues/8587 ))
- Remove extraneous unittest logging decorators from unit tests. ([\#8592](https://github.com/matrix-org/synapse/issues/8592 ))
- Minor optimisations in caching code. ([\#8593](https://github.com/matrix-org/synapse/issues/8593 ), [\#8594](https://github.com/matrix-org/synapse/issues/8594 ))
-----BEGIN PGP SIGNATURE-----
iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAl+RfFAQHGVyaWtAbWF0
cml4Lm9yZwAKCRClQuTtGw+sCZI4CACibRDTo+DFIp787ZMErW0xFc3S8DOA6K4u
G8J2GCrBT0pfWAi7jqfQO9GySlvS5ixqbWYxud9jG0elXtnVgjIcbToSCYvNFxzH
TDaQAaOwFVe/7fhZG6TvO2EcAXYXNfoqQKwc8xvTOcBOhObfLAzchV6t7mRTRYE4
+C3kM8BUJUnSp+OAY0zxRrUIVjX4W280BQ/+UxvPBLIi0IUQz/BufvTtgEhRBb6B
eK2qjDMTud3/WAzM/TYDoA/LmdAcFCogfkRRHYqDPw0gUIXm1tdHyLWdgHoV+WhX
6p/kacnKxqonKW5/UrqheYT8J0SMA1AlkSm2niy/K+Tde3Nga+Sw
=TLb5
-----END PGP SIGNATURE-----
Merge tag 'v1.22.0rc1' into develop
Synapse 1.22.0rc1 (2020-10-22)
==============================
Features
--------
- Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](https://github.com/matrix-org/synapse/issues/7658 ))
- Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](https://github.com/matrix-org/synapse/issues/8292 ), [\#8467](https://github.com/matrix-org/synapse/issues/8467 ))
- Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732 )). ([\#8312](https://github.com/matrix-org/synapse/issues/8312 ), [\#8501](https://github.com/matrix-org/synapse/issues/8501 ))
- Add support for running background tasks in a separate worker process. ([\#8369](https://github.com/matrix-org/synapse/issues/8369 ), [\#8458](https://github.com/matrix-org/synapse/issues/8458 ), [\#8489](https://github.com/matrix-org/synapse/issues/8489 ), [\#8513](https://github.com/matrix-org/synapse/issues/8513 ), [\#8544](https://github.com/matrix-org/synapse/issues/8544 ), [\#8599](https://github.com/matrix-org/synapse/issues/8599 ))
- Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697 )). ([\#8380](https://github.com/matrix-org/synapse/issues/8380 ))
- Add support for [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409 ), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](https://github.com/matrix-org/synapse/issues/8437 ), [\#8590](https://github.com/matrix-org/synapse/issues/8590 ))
- Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788 ). ([\#8461](https://github.com/matrix-org/synapse/issues/8461 ))
- Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](https://github.com/matrix-org/synapse/issues/8479 ))
- Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](https://github.com/matrix-org/synapse/issues/8502 ))
- Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](https://github.com/matrix-org/synapse/issues/8535 ), [\#8564](https://github.com/matrix-org/synapse/issues/8564 ))
Bugfixes
--------
- Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](https://github.com/matrix-org/synapse/issues/8454 ))
- Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](https://github.com/matrix-org/synapse/issues/8457 ))
- Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](https://github.com/matrix-org/synapse/issues/8465 ))
- Fix message duplication if something goes wrong after persisting the event. ([\#8476](https://github.com/matrix-org/synapse/issues/8476 ))
- Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](https://github.com/matrix-org/synapse/issues/8486 ))
- Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](https://github.com/matrix-org/synapse/issues/8504 ))
- Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](https://github.com/matrix-org/synapse/issues/8517 ))
- Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](https://github.com/matrix-org/synapse/issues/8527 ))
- Fix not sending events over federation when using sharded event writers. ([\#8536](https://github.com/matrix-org/synapse/issues/8536 ))
- Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](https://github.com/matrix-org/synapse/issues/8545 ))
- Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](https://github.com/matrix-org/synapse/issues/8567 ))
- Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](https://github.com/matrix-org/synapse/issues/8585 ))
- Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](https://github.com/matrix-org/synapse/issues/8606 ))
Updates to the Docker image
---------------------------
- Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](https://github.com/matrix-org/synapse/issues/7921 ))
- Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](https://github.com/matrix-org/synapse/issues/8390 ))
Improved Documentation
----------------------
- Update the directions for using the manhole with coroutines. ([\#8462](https://github.com/matrix-org/synapse/issues/8462 ))
- Improve readme by adding new shield.io badges. ([\#8493](https://github.com/matrix-org/synapse/issues/8493 ))
- Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. ([\#8526](https://github.com/matrix-org/synapse/issues/8526 ))
- Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](https://github.com/matrix-org/synapse/issues/8529 ))
Deprecations and Removals
-------------------------
- Drop unused `device_max_stream_id` table. ([\#8589](https://github.com/matrix-org/synapse/issues/8589 ))
Internal Changes
----------------
- Check for unreachable code with mypy. ([\#8432](https://github.com/matrix-org/synapse/issues/8432 ))
- Add unit test for event persister sharding. ([\#8433](https://github.com/matrix-org/synapse/issues/8433 ))
- Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](https://github.com/matrix-org/synapse/issues/8439 ), [\#8488](https://github.com/matrix-org/synapse/issues/8488 ), [\#8496](https://github.com/matrix-org/synapse/issues/8496 ), [\#8499](https://github.com/matrix-org/synapse/issues/8499 ))
- Configure `public_baseurl` when using demo scripts. ([\#8443](https://github.com/matrix-org/synapse/issues/8443 ))
- Add SQL logging on queries that happen during startup. ([\#8448](https://github.com/matrix-org/synapse/issues/8448 ))
- Speed up unit tests when using PostgreSQL. ([\#8450](https://github.com/matrix-org/synapse/issues/8450 ))
- Remove redundant database loads of stream_ordering for events we already have. ([\#8452](https://github.com/matrix-org/synapse/issues/8452 ))
- Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](https://github.com/matrix-org/synapse/issues/8463 ))
- Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](https://github.com/matrix-org/synapse/issues/8464 ))
- Additional testing for `ThirdPartyEventRules`. ([\#8468](https://github.com/matrix-org/synapse/issues/8468 ))
- Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](https://github.com/matrix-org/synapse/issues/8472 ))
- Unblacklist some sytests. ([\#8474](https://github.com/matrix-org/synapse/issues/8474 ))
- Include the log level in the phone home stats. ([\#8477](https://github.com/matrix-org/synapse/issues/8477 ))
- Remove outdated sphinx documentation, scripts and configuration. ([\#8480](https://github.com/matrix-org/synapse/issues/8480 ))
- Clarify error message when plugin config parsers raise an error. ([\#8492](https://github.com/matrix-org/synapse/issues/8492 ))
- Remove the deprecated `Handlers` object. ([\#8494](https://github.com/matrix-org/synapse/issues/8494 ))
- Fix a threadsafety bug in unit tests. ([\#8497](https://github.com/matrix-org/synapse/issues/8497 ))
- Add user agent to user_daily_visits table. ([\#8503](https://github.com/matrix-org/synapse/issues/8503 ))
- Add type hints to various parts of the code base. ([\#8407](https://github.com/matrix-org/synapse/issues/8407 ), [\#8505](https://github.com/matrix-org/synapse/issues/8505 ), [\#8507](https://github.com/matrix-org/synapse/issues/8507 ), [\#8547](https://github.com/matrix-org/synapse/issues/8547 ), [\#8562](https://github.com/matrix-org/synapse/issues/8562 ), [\#8609](https://github.com/matrix-org/synapse/issues/8609 ))
- Remove unused code from the test framework. ([\#8514](https://github.com/matrix-org/synapse/issues/8514 ))
- Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](https://github.com/matrix-org/synapse/issues/8515 ))
- Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](https://github.com/matrix-org/synapse/issues/8537 ))
- Improve database performance by executing more queries without starting transactions. ([\#8542](https://github.com/matrix-org/synapse/issues/8542 ))
- Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](https://github.com/matrix-org/synapse/issues/8548 ))
- Move metric registration code down into `LruCache`. ([\#8561](https://github.com/matrix-org/synapse/issues/8561 ), [\#8591](https://github.com/matrix-org/synapse/issues/8591 ))
- Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](https://github.com/matrix-org/synapse/issues/8563 ))
- Add virtualenv-generated folders to `.gitignore`. ([\#8566](https://github.com/matrix-org/synapse/issues/8566 ))
- Add `get_immediate` method to `DeferredCache`. ([\#8568](https://github.com/matrix-org/synapse/issues/8568 ))
- Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](https://github.com/matrix-org/synapse/issues/8569 ))
- Fix `synmark` benchmark runner. ([\#8571](https://github.com/matrix-org/synapse/issues/8571 ))
- Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](https://github.com/matrix-org/synapse/issues/8572 ))
- Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](https://github.com/matrix-org/synapse/issues/8577 ))
- Support macOS on the `synmark` benchmark runner. ([\#8578](https://github.com/matrix-org/synapse/issues/8578 ))
- Update `mypy` static type checker to 0.790. ([\#8583](https://github.com/matrix-org/synapse/issues/8583 ), [\#8600](https://github.com/matrix-org/synapse/issues/8600 ))
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](https://github.com/matrix-org/synapse/issues/8587 ))
- Remove extraneous unittest logging decorators from unit tests. ([\#8592](https://github.com/matrix-org/synapse/issues/8592 ))
- Minor optimisations in caching code. ([\#8593](https://github.com/matrix-org/synapse/issues/8593 ), [\#8594](https://github.com/matrix-org/synapse/issues/8594 ))
2020-10-22 13:37:08 +01:00
514a240aed
Remove unused OPTIONS handlers. ( #8621 )
...
The handling of OPTIONS requests was consolidated in #7534 , but the endpoint
specific handlers were not removed.
2020-10-22 08:35:55 -04:00
b19b63e6b4
Don't 500 for invalid group IDs ( #8628 )
2020-10-22 13:19:06 +01:00
ec0e9c4695
1.22.0rc1
2020-10-22 13:08:42 +01:00
a9f90fa73a
Type hints for RegistrationStore ( #8615 )
2020-10-22 11:56:58 +01:00
b28aaeb3a5
Optimise CacheDescriptor ( #8594 )
...
don't bother constricting a CacheContext unless we need one.
2020-10-21 22:57:45 +01:00
c13820bcee
fix failure case
2020-10-21 18:54:53 +01:00
2b3af01791
optimise DeferredCache.set
2020-10-21 17:55:53 +01:00
9146a8a691
Merge pull request #8572 from matrix-org/rav/cache_hacking/2
...
Push some deferred wrangling down into DeferredCache
2020-10-21 17:55:04 +01:00
1f4269700c
Push some deferred wrangling down into DeferredCache
2020-10-21 15:39:25 +01:00
70259d8c8c
Limit AS transactions to 100 events ( #8606 )
...
* Limit AS transactions to 100 events
* Update changelog.d/8606.feature
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add tests
* Update synapse/appservice/scheduler.py
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-10-21 15:36:53 +01:00
20a67aa70d
Separate the TCP and terse JSON formatting code. ( #8587 )
...
This should (theoretically) allow for using the TCP code with a different output type
and make it easier to use the JSON code with files / console.
2020-10-21 06:59:54 -04:00
de5cafe980
Add type hints to profile and base handlers. ( #8609 )
2020-10-21 06:44:31 -04:00
9e0f22874f
Consistently use wrap_as_background_task in more places ( #8599 )
2020-10-20 11:29:38 -04:00
626b8f0846
Move schema file for as_device_stream ( #8590 )
...
* Move schema file
* Add a .
* Add matching changelog entry
* Fix sqlite
2020-10-20 10:18:55 +01:00
96e7d3c4a0
Fix 'LruCache' object has no attribute '_on_resize' ( #8591 )
...
We need to make sure we are readu for the `set_cache_factor` callback.
2020-10-19 21:13:50 +01:00
34c20493b9
Drop unused `device_max_stream_id` table ( #8589 )
...
Signed-off-by: Vasilis Gerakaris <vasilis.gerakaris@navarino.gr>
2020-10-19 19:06:54 +01:00
21bb50ca3f
Fix mypy error: auth handler "checkpw" internal function type mismatch ( #8569 )
2020-10-19 18:32:24 +01:00
8f27b7fde1
Expose the experimental appservice login flow to clients. ( #8504 )
2020-10-19 13:03:55 -04:00
903d11c43a
Add `DeferredCache.get_immediate` method ( #8568 )
...
* Add `DeferredCache.get_immediate` method
A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.
* Remove unused 'default' parameter to DeferredCache.get()
* another get_immediate instance
2020-10-19 15:00:12 +01:00
c356b4bf42
Include a simple message in email notifications that include encrypted content ( #8545 )
2020-10-19 09:12:39 -04:00
1fcdbeb3ab
Start an opentracing span for background processes. ( #8567 )
...
This should reduce the number of `There was no active span` errors we
see.
Fixes #8510 .
2020-10-19 12:26:26 +01:00
97647b33c2
Replace DeferredCache with LruCache where possible ( #8563 )
...
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
2020-10-19 12:20:29 +01:00
79c1f973ce
Pre-emptively fix synapse.storage.types.Connection for future mypy release ( #8577 )
...
Fix the Connection protocol according to typeshed's assertions about sqlite3.Connection
2020-10-17 09:51:38 +01:00
d6094176d1
Type annotations for LruCache ( #8562 )
...
* type annotations for LruCache
* changelog
* Apply suggestions from code review
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* review comments
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-16 17:06:50 +01:00
1b70662be9
Clean-up old transaction IDs on the background worker. ( #8544 )
2020-10-16 12:06:17 -04:00
6d7b22041d
review comments
2020-10-16 16:25:15 +01:00
995cc615a0
Apply suggestions from code review
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-16 16:14:42 +01:00
0ec0bc3886
type annotations for LruCache
2020-10-16 15:56:39 +01:00
3ee17585cd
Make LruCache register its own metrics ( #8561 )
...
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
2020-10-16 15:51:57 +01:00
da0090fdff
Fix modifying events in `ThirdPartyRules` modules ( #8564 )
...
EventBuilder.build wants auth events these days
2020-10-16 13:39:46 +01:00
5649669c3c
Merge pull request #8535 from matrix-org/rav/third_party_events_updates
...
Support modifying event content from ThirdPartyRules modules
2020-10-15 20:55:41 +01:00
6b5a115c0a
Solidify the HomeServer constructor. ( #8515 )
...
This implements a more standard API for instantiating a homeserver and
moves some of the dependency injection into the test suite.
More concretely this stops using `setattr` on all `kwargs` passed to `HomeServer`.
2020-10-15 15:29:13 -04:00
c276bd9969
Send some ephemeral events to appservices ( #8437 )
...
Optionally sends typing, presence, and read receipt information to appservices.
2020-10-15 12:33:28 -04:00
74976a8e43
Merge branch 'master' into develop
2020-10-15 10:43:54 -04:00
9991aaa49c
1.21.2
2020-10-15 09:24:10 -04:00
20fa83f374
Remove racey assertion in MultiWriterIDGenerator ( #8530 )
...
We asserted that the IDs returned by postgres sequence was greater than
any we had seen, however this is technically racey as we may update the
current positions out of order.
We now assert that the sequences are correct on startup, so the
assertion is no longer really required, so we remove them.
2020-10-15 09:14:29 -04:00
8075504a60
Enable mypy for synapse.util.caches ( #8547 )
...
This seemed to entail dragging in a type stub for SortedList.
2020-10-15 11:44:39 +01:00
0a08cd1065
Merge pull request #8548 from matrix-org/rav/deferred_cache
...
Rename Cache to DeferredCache, and related changes
2020-10-15 11:42:07 +01:00
1f39155071
Include user agent in user daily visits table ( #8503 )
...
Include user agent in user daily visits table.
2020-10-15 10:36:40 +01:00
4433d01519
Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_invite
...
Simplify `_locally_reject_invite`
2020-10-15 10:20:19 +01:00
4182bb812f
move DeferredCache into its own module
2020-10-14 23:38:14 +01:00
9f87da0a84
Rename Cache->DeferredCache
2020-10-14 23:38:14 +01:00
Will Hunt