29364145b2
Pass str to twisted's IReactorTCP ( #10895 )
...
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.
Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
2021-09-30 12:51:47 +01:00
3412f5c8d8
1.44.0rc2
2021-09-30 12:40:24 +01:00
c4bf48ee6f
Fix event context for outliers in important MSC2716 spot ( #10938 )
...
Fix event context for outlier causing failures in all of the MSC2716
Complement tests.
The `EventContext.for_outlier` refactor happened in
https://github.com/matrix-org/synapse/pull/10883
and this spot was left out.
2021-09-30 11:34:44 +01:00
a03ed5e6ae
Fix issue causing sending presence to ASes to fail (due to incomplete type annotations) ( #10944 )
2021-09-30 11:06:47 +01:00
3aefc7b66d
Refactor user directory tests ( #10935 )
...
* Pull out GetUserDirectoryTables helper
* Don't rebuild the dir in tests that don't need it
In #10796 I changed registering a user to add directory entries under.
This means we don't have to force a directory regbuild in to tests of
the user directory search.
* Move test_initial to tests/storage
* Add type hints to both test_user_directory files
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-30 11:04:40 +01:00
428174f902
Split `event_auth.check` into two parts ( #10940 )
...
Broadly, the existing `event_auth.check` function has two parts:
* a validation section: checks that the event isn't too big, that it has the rught signatures, etc.
This bit is independent of the rest of the state in the room, and so need only be done once
for each event.
* an auth section: ensures that the event is allowed, given the rest of the state in the room.
This gets done multiple times, against various sets of room state, because it forms part of
the state res algorithm.
Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think
that makes everything hard to follow. Instead, we split the function in two and call each part
separately where it is needed.
2021-09-29 18:59:15 +01:00
a19aa8b162
Synapse 1.44.0rc1 (2021-09-29)
...
==============================
Features
--------
- Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776 ))
- Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814 ), [\#10819](https://github.com/matrix-org/synapse/issues/10819 ))
- Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868 ), [\#10905](https://github.com/matrix-org/synapse/issues/10905 ))
- Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898 ))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690 ))
- Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782 ))
- Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807 ))
- Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810 ))
- Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827 ))
- Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843 ))
- Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859 ))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873 ))
- Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875 ))
- Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881 ))
- Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907 ))
- Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911 ))
- Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913 ))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845 ))
- Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865 ))
- Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869 ))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659 ))
- Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777 ))
- Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785 ), [\#10817](https://github.com/matrix-org/synapse/issues/10817 ))
- Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796 ))
- Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812 ), [\#10885](https://github.com/matrix-org/synapse/issues/10885 ), [\#10893](https://github.com/matrix-org/synapse/issues/10893 ), [\#10897](https://github.com/matrix-org/synapse/issues/10897 ))
- Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815 ))
- Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816 ))
- Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820 ))
- Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823 ))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826 ))
- Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829 ))
- Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831 ), [\#10856](https://github.com/matrix-org/synapse/issues/10856 ))
- Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833 ))
- Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834 ))
- Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839 ))
- Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867 ))
- Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879 ))
- Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829 ). ([\#10880](https://github.com/matrix-org/synapse/issues/10880 ))
- Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883 ), [\#10884](https://github.com/matrix-org/synapse/issues/10884 ), [\#10896](https://github.com/matrix-org/synapse/issues/10896 ), [\#10901](https://github.com/matrix-org/synapse/issues/10901 ))
- Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ). ([\#10887](https://github.com/matrix-org/synapse/issues/10887 ))
- Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889 ))
- Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891 ))
- Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906 ))
- Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917 ))
- Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925 ))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmFUY3UACgkQLS76LzL7
4EdDdg//egZvwUizHJAEWi5Vz029S2D3LN8Hl57TG+/KR+GgIJvOfgX9sgWDNsLA
4wTlsw6xhH9VcDzFhiAPrUTwhi4Gi80BZmnGXuR97yAE1z3oyS4IQg/g5LUXq9el
a/2JsK1Sau354TBMvLQgZsplqC+RPxE0k0MSTvEC+d3ft929jLXUftnI/nhGcDpX
7pP9RscVThkLLNC5Xcuu2ev4yPSrNNWhBeqz5J8CfbEW6Roej7a1d3WdehhoWJ5V
hwPOxOhL89lWwJ8uL/H5sDFMkxk+ua+IVQx8gJkQ1eRa3RIdFFSh1+exFtYwjIhI
Pd4wwUH7hGe1VIvmlVOoDZ1hka0RTjRLeJg39vq+yvWCBQW7syVE6sRUxTjeUvRe
7Cyz4sVG0g8uvWsF3YYva7S4SlYPPSj2Z+5/yL4RHlMZsPk/apmCRZBzaWlVEGur
XqTExi2rKzKsi2HZctHDWOeYF+G3c13FDNeXG+N2g/DfGKuKCTuhbdMm7LeIJ2L2
B8ki/nq3wc1FCJ0mkVlM+/UX3i3wXG+2UbOx29hU9yuEUIl4p4jyE1OjuH5l45TX
OWfW25BOGh5gYKvX8p9Kj4A9MIJ2YFRDFfcfF7g+S0ZA8KdUAjZkYBzfJtEzn2iK
uxVsSVdYOVqbbSwE5wto8WuOWu1op6RP3l1Wp92SoM5vqWjRmyA=
=SzDp
-----END PGP SIGNATURE-----
Merge tag 'v1.44.0rc1' into develop
Synapse 1.44.0rc1 (2021-09-29)
==============================
Features
--------
- Only allow the
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
`/batch_send?chunk_id=xxx` endpoint to connect to an already existing
insertion event.
([\#10776](https://github.com/matrix-org/synapse/issues/10776 ))
- Improve oEmbed URL previews by processing the author name, photo, and
video information.
([\#10814](https://github.com/matrix-org/synapse/issues/10814 ),
[\#10819](https://github.com/matrix-org/synapse/issues/10819 ))
- Speed up responding with large JSON objects to requests.
([\#10868](https://github.com/matrix-org/synapse/issues/10868 ),
[\#10905](https://github.com/matrix-org/synapse/issues/10905 ))
- Add a `user_may_create_room_with_invites` spam checker callback to
allow modules to allow or deny a room creation request based on the
invites and/or 3PID invites it includes.
([\#10898](https://github.com/matrix-org/synapse/issues/10898 ))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging
history in certain rooms. Contributed by @Kokokokoka.
([\#10690](https://github.com/matrix-org/synapse/issues/10690 ))
- Fix a long-standing bug which caused deactivated users that were later
reactivated to be missing from the user directory.
([\#10782](https://github.com/matrix-org/synapse/issues/10782 ))
- Fix a long-standing bug that caused unbanning a user by sending a
membership event to fail. Contributed by @aaronraimist.
([\#10807](https://github.com/matrix-org/synapse/issues/10807 ))
- Fix a long-standing bug where logging contexts would go missing when
federation requests time out.
([\#10810](https://github.com/matrix-org/synapse/issues/10810 ))
- Fix a long-standing bug causing an error in the deprecated
`/initialSync` endpoint when using the undocumented `from` and `to`
parameters.
([\#10827](https://github.com/matrix-org/synapse/issues/10827 ))
- Fix a bug causing the `remove_stale_pushers` background job to
repeatedly fail and log errors. This bug affected Synapse servers that
had been upgraded from version 1.28 or older and are using SQLite.
([\#10843](https://github.com/matrix-org/synapse/issues/10843 ))
- Fix a long-standing bug in Unicode support of the room search admin
API breaking search for rooms with non-ASCII characters.
([\#10859](https://github.com/matrix-org/synapse/issues/10859 ))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership
events which we sent to remote servers to be incorrectly stored in the
local database.
([\#10873](https://github.com/matrix-org/synapse/issues/10873 ))
- Fix invalidating one-time key count cache after claiming keys. The bug
was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper.
([\#10875](https://github.com/matrix-org/synapse/issues/10875 ))
- Fix a long-standing bug causing application service users to be
subject to MAU blocking if the MAU limit had been reached, even if
configured not to be blocked.
([\#10881](https://github.com/matrix-org/synapse/issues/10881 ))
- Fix a long-standing bug which could cause events pulled over
federation to be incorrectly rejected.
([\#10907](https://github.com/matrix-org/synapse/issues/10907 ))
- Fix a long-standing bug causing URL cache files to be stored in
storage providers. Server admins may safely delete the `url_cache/` and
`url_cache_thumbnails/` directories from any configured storage
providers to reclaim space.
([\#10911](https://github.com/matrix-org/synapse/issues/10911 ))
- Fix a long-standing bug leading to race conditions when creating media
store and config directories.
([\#10913](https://github.com/matrix-org/synapse/issues/10913 ))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON
encoding/decoding.
([\#10845](https://github.com/matrix-org/synapse/issues/10845 ))
- Add developer documentation about experimental configuration flags.
([\#10865](https://github.com/matrix-org/synapse/issues/10865 ))
- Properly remove deleted files from GitHub pages when generating the
documentation.
([\#10869](https://github.com/matrix-org/synapse/issues/10869 ))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from
parallel branches.
([\#10659](https://github.com/matrix-org/synapse/issues/10659 ))
- Split out
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) meta
events to their own fields in the `/batch_send` response.
([\#10777](https://github.com/matrix-org/synapse/issues/10777 ))
- Add missing type hints to REST servlets.
([\#10785](https://github.com/matrix-org/synapse/issues/10785 ),
[\#10817](https://github.com/matrix-org/synapse/issues/10817 ))
- Simplify the internal logic which maintains the user directory
database tables.
([\#10796](https://github.com/matrix-org/synapse/issues/10796 ))
- Use direct references to config flags.
([\#10812](https://github.com/matrix-org/synapse/issues/10812 ),
[\#10885](https://github.com/matrix-org/synapse/issues/10885 ),
[\#10893](https://github.com/matrix-org/synapse/issues/10893 ),
[\#10897](https://github.com/matrix-org/synapse/issues/10897 ))
- Specify the type of token in generic "Invalid token" error messages.
([\#10815](https://github.com/matrix-org/synapse/issues/10815 ))
- Make `StateFilter` frozen so it is hashable.
([\#10816](https://github.com/matrix-org/synapse/issues/10816 ))
- Fix a long-standing bug where an `m.room.message` event containing a
null byte would cause an internal server error.
([\#10820](https://github.com/matrix-org/synapse/issues/10820 ))
- Add type hints to the state database.
([\#10823](https://github.com/matrix-org/synapse/issues/10823 ))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to
hopefully improve `/sync` performance when you
haven't synced recently.
([\#10826](https://github.com/matrix-org/synapse/issues/10826 ))
- Track cache eviction rates more finely in Prometheus's monitoring.
([\#10829](https://github.com/matrix-org/synapse/issues/10829 ))
- Add missing type hints to `synapse.handlers`.
([\#10831](https://github.com/matrix-org/synapse/issues/10831 ),
[\#10856](https://github.com/matrix-org/synapse/issues/10856 ))
- Extend the Module API to let plug-ins check whether an ID is local and
to access IP + User Agent data.
([\#10833](https://github.com/matrix-org/synapse/issues/10833 ))
- Factor out PNG image data to a constant to be used in several tests.
([\#10834](https://github.com/matrix-org/synapse/issues/10834 ))
- Add a test to ensure state events sent by modules get persisted
correctly.
([\#10835](https://github.com/matrix-org/synapse/issues/10835 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
fields and event types from `chunk` to `batch` to match the
`/batch_send` endpoint.
([\#10838](https://github.com/matrix-org/synapse/issues/10838 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
`/batch_send` query parameter from `?prev_event` to more obvious usage
with `?prev_event_id`.
([\#10839](https://github.com/matrix-org/synapse/issues/10839 ))
- Add type hints to `synapse.http.site`.
([\#10867](https://github.com/matrix-org/synapse/issues/10867 ))
- Include outlier status when we log V2 or V3 events.
([\#10879](https://github.com/matrix-org/synapse/issues/10879 ))
- Break down Grafana's cache expiry time series based on reason for
eviction, c.f.
[\#10829](https://github.com/matrix-org/synapse/issues/10829 ).
([\#10880](https://github.com/matrix-org/synapse/issues/10880 ))
- Clean up some of the federation event authentication code for clarity.
([\#10883](https://github.com/matrix-org/synapse/issues/10883 ),
[\#10884](https://github.com/matrix-org/synapse/issues/10884 ),
[\#10896](https://github.com/matrix-org/synapse/issues/10896 ),
[\#10901](https://github.com/matrix-org/synapse/issues/10901 ))
- Allow the `.` and `~` characters when creating registration tokens as
per the change to
[MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ).
([\#10887](https://github.com/matrix-org/synapse/issues/10887 ))
- Clean up some unnecessary parentheses in places around the codebase.
([\#10889](https://github.com/matrix-org/synapse/issues/10889 ))
- Improve type hinting in the user directory code.
([\#10891](https://github.com/matrix-org/synapse/issues/10891 ))
- Update development testing script `test_postgresql.sh` to use a
supported Python version and make re-runs quicker.
([\#10906](https://github.com/matrix-org/synapse/issues/10906 ))
- Document and summarize changes in schema version `61` – `64`.
([\#10917](https://github.com/matrix-org/synapse/issues/10917 ))
- Update release script to sign the newly created git tags.
([\#10925](https://github.com/matrix-org/synapse/issues/10925 ))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build
their docs.
([\#10931](https://github.com/matrix-org/synapse/issues/10931 ))
F124520CEEE062448FE1C8442D2EFA2F32FBE047
<olivier@librepush.net>" [ultimate]
2021-09-29 14:23:39 +01:00
176aa55fd5
add event id to logcontext when handling incoming PDUs ( #10936 )
2021-09-29 11:59:43 +01:00
e32b9f44ee
Update installation instructions ( #10919 )
...
Various updates to the install docs.
2021-09-29 10:57:53 +00:00
94b620a5ed
Use direct references for configuration variables (part 6). ( #10916 )
2021-09-29 06:44:15 -04:00
8cef1ab2ac
Implement MSC3069: Guest support on whoami ( #9655 )
2021-09-29 11:32:45 +01:00
13032b6603
Bump the date because the release ran over
2021-09-29 11:13:03 +01:00
1b9ce5e8a6
Indicate when bugs were introduced and tidy up
2021-09-29 11:09:00 +01:00
67815cc3db
Tweak changelog
2021-09-29 11:00:56 +01:00
5279b9161b
Use `RoomVersion` objects ( #10934 )
...
Various refactors to use `RoomVersion` objects instead of room version identifiers.
2021-09-29 10:57:10 +01:00
2be0fde3d6
Fix empty `url_cache_thumbnails/yyyy-mm-dd/` directories being left behind ( #10924 )
2021-09-29 10:24:37 +01:00
9fd057b8c5
Ensure `(room_id, next_batch_id)` is unique to avoid cross-talk/conflicts between batches (MSC2716) ( #10877 )
...
Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
Part of https://github.com/matrix-org/synapse/issues/10737
2021-09-28 21:23:16 -05:00
62800a8fe3
Add #10932 to release
2021-09-28 17:32:31 +01:00
0f007fe009
Update utility code to handle C implementations of frozendict ( #10902 )
...
* update _handle_frozendict to work with c implementations of frozen dict
* add changelog
* add clarifying comment to _handle_frozendict
2021-09-28 09:13:23 -07:00
8aaa4b7b5d
Drop backwards-compatibility support for "outlier" ( #10903 )
...
Before Synapse 1.31 (#9411 ), we relied on `outlier` being stored in the
`internal_metadata` column. We can now assume nobody will roll back their
deployment that far and drop the legacy support.
2021-09-28 15:25:36 +01:00
2622b28c5c
Inline `_check_event_auth` for outliers ( #10926 )
...
* Inline `_check_event_auth` for outliers
When we are persisting an outlier, most of `_check_event_auth` is redundant:
* `_update_auth_events_and_context_for_auth` does nothing, because the
`input_auth_events` are (now) exactly the event's auth_events,
which means that `missing_auth` is empty.
* we don't care about soft-fail, kicking guest users or `send_on_behalf_of`
for outliers
... so the only thing that matters is the auth itself, so let's just do that.
* `_auth_and_persist_fetched_events_inner`: de-async `prep`
`prep` no longer calls any `async` methods, so let's make it synchronous.
* Simplify `_check_event_auth`
We no longer need to support outliers here, which makes things rather simpler.
* changelog
* lint
2021-09-28 15:25:07 +01:00
37bb93d181
Fix exception responding to request that has been closed ( #10932 )
...
Introduced in #10905
2021-09-28 14:36:19 +01:00
eb2c7e51c4
Clean-up type hints in server config ( #10915 )
...
By using attrs instead of dicts to store configuration.
Also updates some of the attrs classes to use proper type
hints and auto_attribs.
2021-09-28 09:24:40 -04:00
2b9d174791
Fixup changelog
2021-09-28 13:50:05 +01:00
bc69d49362
Fixup changelog
2021-09-28 13:48:42 +01:00
c3ccad7785
Only do restricted join rules signature checks for room versions 8/9. ( #10927 )
...
Otherwise the presence of a (bogus, unused) field could cause
auth checks to fail.
2021-09-28 08:44:19 -04:00
3c50192d3f
1.44.0rc1
2021-09-28 13:42:21 +01:00
a8bbf08576
Fix debian package builds. ( #10931 )
...
This was due to dh-virtualenv builds being broken due to Shpinx removing
deprecated APIs.
2021-09-28 12:13:51 +01:00
707d5e4e48
Encode JSON responses on a thread in C, mk2 ( #10905 )
...
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.
Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:
1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.
I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
2021-09-28 09:37:58 +00:00
d37841787a
Sign the git tag in release script ( #10925 )
2021-09-27 15:39:49 +01:00
f7768f62cb
Avoid storing URL cache files in storage providers ( #10911 )
...
URL cache files are short-lived and it does not make sense to offload
them (eg. to the cloud) or back them up.
2021-09-27 12:55:27 +01:00
6c83c27107
Fix race conditions when creating media store and config directories ( #10913 )
2021-09-27 11:29:23 +01:00
d138187045
Document changes to schema version 61 - 64 ( #10917 )
...
As pointed out by @richvdh, https://github.com/matrix-org/synapse/pull/10838#discussion_r715424244
Retroactively summarize `61` - `64`
2021-09-24 17:09:12 -05:00
b10257e879
Add a spamchecker callback to allow or deny room creation based on invites ( #10898 )
...
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.
This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
2021-09-24 16:38:23 +02:00
ea01d4c2de
Update postgresql testing script ( #10906 )
...
- Use sytest:bionic. Sytest:latest is two years old (do we want
CI to push out latest at all?) and comes with Python 3.5, which we
explictly no longer support. The script now runs under PostgreSQL 10
as a result.
- Advertise script in the docs
- Move pg testing script to scripts-dev directory
- Write to host as the script's exector, not root
A few changes to make it speedier to re-run the tests:
- Create blank DB in the container, not the script, so we don't have to
`initdb` each time
- Use a named volume to persist the tox environment, so we don't have to
fetch and install a bunch of packages from PyPI each time
Co-authored-by: reivilibre <olivier@librepush.net>
2021-09-24 14:27:09 +00:00
0420d4e6a5
Stop trying to auth/persist events whose auth events we do not have. ( #10907 )
2021-09-24 14:01:45 +01:00
bb7fdd821b
Use direct references for configuration variables (part 5). ( #10897 )
2021-09-24 07:25:21 -04:00
85551b7a85
Factor out common code for persisting fetched auth events ( #10896 )
...
* Factor more stuff out of `_get_events_and_persist`
It turns out that the event-sorting algorithm in `_get_events_and_persist` is
also useful in other circumstances. Here we move the current
`_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`,
and then factor the sorting part out to `_auth_and_persist_fetched_events`.
* `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment
`get_event_auth` returns events with the outlier flag already set, so this is
redundant (though we need to update a test where `get_event_auth` is mocked).
* `_get_remote_auth_chain_for_event`: move existing-event tests earlier
Move a couple of tests outside the loop. This is a bit inefficient for now, but
a future commit will make it better. It should be functionally identical.
* `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events`
We can use the same codepath for persisting the events fetched as part of an
auth chain as for those fetched individually by `_get_events_and_persist` for
building the state at a backwards extremity.
* `_get_remote_auth_chain_for_event`: use a dict for efficiency
`_auth_and_persist_fetched_events` sorts the events itself, so we no longer
need to care about maintaining the ordering from `get_event_auth` (and no
longer need to sort by depth in `get_event_auth`).
That means that we can use a map, making it easier to filter out events we
already have, etc.
* changelog
* `_auth_and_persist_fetched_events`: improve docstring
2021-09-24 11:56:33 +01:00
261c9763c4
Simplify `_auth_and_persist_fetched_events` ( #10901 )
...
Combine the two loops over the list of events, and hence get rid of
`_NewEventInfo`. Also pass the event back alongside the context, so that it's
easier to process the result.
2021-09-24 11:56:13 +01:00
50022cff96
Add reactor to `SynapseRequest` and fix up types. ( #10868 )
2021-09-24 11:01:25 +01:00
fa74536384
Fix AuthBlocking check when requester is appservice ( #10881 )
...
If the MAU count had been reached, Synapse incorrectly blocked appservice users even though they've been explicitly configured not to be tracked (the default). This was due to bypassing the relevant if as it was chained behind another earlier hit if as an elif.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-09-24 10:41:18 +01:00
7f3352743e
Improve typing in user_directory files ( #10891 )
...
* Improve typing in user_directory files
This makes the user_directory.py in storage pass most of mypy's
checks (including `no-untyped-defs`). Unfortunately that file is in the
tangled web of Store class inheritance so doesn't pass mypy at the moment.
The handlers directory has already been mypyed.
Co-authored-by: reivilibre <olivier@librepush.net>
2021-09-24 10:38:22 +01:00
e704cc2a48
In `_purge_history_txn`, ensure that txn.fetchall has elements before accessing rows ( #10690 )
...
This change adds a check for row existence before accessing row element, this should fix issue #10669
Signed-off-by: Vasya Boytsov vasiliy.boytsov@phystech.edu
2021-09-24 09:19:51 +00:00
90d9fc7505
Allow `.` and `~` chars in registration tokens ( #10887 )
...
Per updates to MSC3231 in order to use the same grammar
as other identifiers.
2021-09-23 17:58:12 +00:00
a7304adc7d
Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth` ( #10884 )
...
* Reload auth events from db after fetching and persisting
In `_update_auth_events_and_context_for_auth`, when we fetch the remote auth
tree and persist the returned events: load the missing events from the database
rather than using the copies we got from the remote server.
This is mostly in preparation for additional refactors, but does have an
advantage in that if we later get around to checking the rejected status, we'll
be able to make use of it.
* Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth`
* changelog
2021-09-23 17:34:33 +01:00
47854c71e9
Use direct references for configuration variables (part 4). ( #10893 )
2021-09-23 12:03:01 -04:00
a10988983a
Break down cache expiry reasons in grafana ( #10880 )
...
A follow-up to #10829
2021-09-23 14:45:32 +01:00
dcfd864970
Fix reactivated users not being added to the user directory ( #10782 )
...
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-23 12:02:13 +00:00
e584534403
Use direct references for some configuration variables (part 3) ( #10885 )
...
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.
It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
aa2c027792
Remove unnecessary parentheses around tuples returned from methods ( #10889 )
2021-09-23 11:59:07 +01:00
David Robertson