b3bcacf3c1
Add missing `errcode` to `parse_string` and `parse_boolean` ( #11542 )
2021-12-09 11:23:34 +00:00
a6f1a3abec
Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp ( #9445 )
...
MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030
Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about.
```
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
"event_id": ...
"origin_server_ts": ...
}
```
Federation API endpoint:
```
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
"event_id": ...
"origin_server_ts": ...
}
```
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-12-02 01:02:20 -06:00
b64b6d12d4
Add more type hints to synapse.util. ( #11321 )
2021-11-12 13:43:06 +00:00
1bfd141205
Type hints for the remaining two files in `synapse.http`. ( #11164 )
...
* Teach MyPy that the sentinel context is False
This means that if `ctx: LoggingContextOrSentinel`
then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really
neat find!
* Annotate RequestMetrics
- Raise errors for sentry if we use the sentinel context
- Ensure we don't raise an error and carry on, but not recording stats
- Include stack trace in the error case to lower Sean's blood pressure
* Make mypy pass for synapse.http.request_metrics
* Make synapse.http.connectproxyclient pass mypy
Co-authored-by: reivilibre <oliverw@matrix.org>
2021-10-28 14:14:42 +01:00
2b82ec425f
Add type hints for most `HomeServer` parameters ( #11095 )
2021-10-22 18:15:41 +01:00
797ee7812d
Relax `ignore-missing-imports` for modules that have stubs now and update mypy ( #11006 )
...
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py
Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.
The rest of this change consists of fixups to make the newer mypy + stubs pass CI.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
52aefd5086
Catch AttributeErrors when calling registerProducer ( #10995 )
...
Looks like the wrong exception type was caught in #10932 .
2021-10-07 10:37:10 +00:00
a19aa8b162
Synapse 1.44.0rc1 (2021-09-29)
...
==============================
Features
--------
- Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776 ))
- Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814 ), [\#10819](https://github.com/matrix-org/synapse/issues/10819 ))
- Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868 ), [\#10905](https://github.com/matrix-org/synapse/issues/10905 ))
- Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898 ))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690 ))
- Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782 ))
- Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807 ))
- Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810 ))
- Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827 ))
- Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843 ))
- Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859 ))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873 ))
- Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875 ))
- Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881 ))
- Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907 ))
- Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911 ))
- Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913 ))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845 ))
- Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865 ))
- Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869 ))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659 ))
- Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777 ))
- Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785 ), [\#10817](https://github.com/matrix-org/synapse/issues/10817 ))
- Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796 ))
- Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812 ), [\#10885](https://github.com/matrix-org/synapse/issues/10885 ), [\#10893](https://github.com/matrix-org/synapse/issues/10893 ), [\#10897](https://github.com/matrix-org/synapse/issues/10897 ))
- Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815 ))
- Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816 ))
- Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820 ))
- Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823 ))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826 ))
- Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829 ))
- Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831 ), [\#10856](https://github.com/matrix-org/synapse/issues/10856 ))
- Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833 ))
- Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834 ))
- Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839 ))
- Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867 ))
- Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879 ))
- Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829 ). ([\#10880](https://github.com/matrix-org/synapse/issues/10880 ))
- Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883 ), [\#10884](https://github.com/matrix-org/synapse/issues/10884 ), [\#10896](https://github.com/matrix-org/synapse/issues/10896 ), [\#10901](https://github.com/matrix-org/synapse/issues/10901 ))
- Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ). ([\#10887](https://github.com/matrix-org/synapse/issues/10887 ))
- Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889 ))
- Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891 ))
- Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906 ))
- Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917 ))
- Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925 ))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmFUY3UACgkQLS76LzL7
4EdDdg//egZvwUizHJAEWi5Vz029S2D3LN8Hl57TG+/KR+GgIJvOfgX9sgWDNsLA
4wTlsw6xhH9VcDzFhiAPrUTwhi4Gi80BZmnGXuR97yAE1z3oyS4IQg/g5LUXq9el
a/2JsK1Sau354TBMvLQgZsplqC+RPxE0k0MSTvEC+d3ft929jLXUftnI/nhGcDpX
7pP9RscVThkLLNC5Xcuu2ev4yPSrNNWhBeqz5J8CfbEW6Roej7a1d3WdehhoWJ5V
hwPOxOhL89lWwJ8uL/H5sDFMkxk+ua+IVQx8gJkQ1eRa3RIdFFSh1+exFtYwjIhI
Pd4wwUH7hGe1VIvmlVOoDZ1hka0RTjRLeJg39vq+yvWCBQW7syVE6sRUxTjeUvRe
7Cyz4sVG0g8uvWsF3YYva7S4SlYPPSj2Z+5/yL4RHlMZsPk/apmCRZBzaWlVEGur
XqTExi2rKzKsi2HZctHDWOeYF+G3c13FDNeXG+N2g/DfGKuKCTuhbdMm7LeIJ2L2
B8ki/nq3wc1FCJ0mkVlM+/UX3i3wXG+2UbOx29hU9yuEUIl4p4jyE1OjuH5l45TX
OWfW25BOGh5gYKvX8p9Kj4A9MIJ2YFRDFfcfF7g+S0ZA8KdUAjZkYBzfJtEzn2iK
uxVsSVdYOVqbbSwE5wto8WuOWu1op6RP3l1Wp92SoM5vqWjRmyA=
=SzDp
-----END PGP SIGNATURE-----
Merge tag 'v1.44.0rc1' into develop
Synapse 1.44.0rc1 (2021-09-29)
==============================
Features
--------
- Only allow the
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
`/batch_send?chunk_id=xxx` endpoint to connect to an already existing
insertion event.
([\#10776](https://github.com/matrix-org/synapse/issues/10776 ))
- Improve oEmbed URL previews by processing the author name, photo, and
video information.
([\#10814](https://github.com/matrix-org/synapse/issues/10814 ),
[\#10819](https://github.com/matrix-org/synapse/issues/10819 ))
- Speed up responding with large JSON objects to requests.
([\#10868](https://github.com/matrix-org/synapse/issues/10868 ),
[\#10905](https://github.com/matrix-org/synapse/issues/10905 ))
- Add a `user_may_create_room_with_invites` spam checker callback to
allow modules to allow or deny a room creation request based on the
invites and/or 3PID invites it includes.
([\#10898](https://github.com/matrix-org/synapse/issues/10898 ))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging
history in certain rooms. Contributed by @Kokokokoka.
([\#10690](https://github.com/matrix-org/synapse/issues/10690 ))
- Fix a long-standing bug which caused deactivated users that were later
reactivated to be missing from the user directory.
([\#10782](https://github.com/matrix-org/synapse/issues/10782 ))
- Fix a long-standing bug that caused unbanning a user by sending a
membership event to fail. Contributed by @aaronraimist.
([\#10807](https://github.com/matrix-org/synapse/issues/10807 ))
- Fix a long-standing bug where logging contexts would go missing when
federation requests time out.
([\#10810](https://github.com/matrix-org/synapse/issues/10810 ))
- Fix a long-standing bug causing an error in the deprecated
`/initialSync` endpoint when using the undocumented `from` and `to`
parameters.
([\#10827](https://github.com/matrix-org/synapse/issues/10827 ))
- Fix a bug causing the `remove_stale_pushers` background job to
repeatedly fail and log errors. This bug affected Synapse servers that
had been upgraded from version 1.28 or older and are using SQLite.
([\#10843](https://github.com/matrix-org/synapse/issues/10843 ))
- Fix a long-standing bug in Unicode support of the room search admin
API breaking search for rooms with non-ASCII characters.
([\#10859](https://github.com/matrix-org/synapse/issues/10859 ))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership
events which we sent to remote servers to be incorrectly stored in the
local database.
([\#10873](https://github.com/matrix-org/synapse/issues/10873 ))
- Fix invalidating one-time key count cache after claiming keys. The bug
was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper.
([\#10875](https://github.com/matrix-org/synapse/issues/10875 ))
- Fix a long-standing bug causing application service users to be
subject to MAU blocking if the MAU limit had been reached, even if
configured not to be blocked.
([\#10881](https://github.com/matrix-org/synapse/issues/10881 ))
- Fix a long-standing bug which could cause events pulled over
federation to be incorrectly rejected.
([\#10907](https://github.com/matrix-org/synapse/issues/10907 ))
- Fix a long-standing bug causing URL cache files to be stored in
storage providers. Server admins may safely delete the `url_cache/` and
`url_cache_thumbnails/` directories from any configured storage
providers to reclaim space.
([\#10911](https://github.com/matrix-org/synapse/issues/10911 ))
- Fix a long-standing bug leading to race conditions when creating media
store and config directories.
([\#10913](https://github.com/matrix-org/synapse/issues/10913 ))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON
encoding/decoding.
([\#10845](https://github.com/matrix-org/synapse/issues/10845 ))
- Add developer documentation about experimental configuration flags.
([\#10865](https://github.com/matrix-org/synapse/issues/10865 ))
- Properly remove deleted files from GitHub pages when generating the
documentation.
([\#10869](https://github.com/matrix-org/synapse/issues/10869 ))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from
parallel branches.
([\#10659](https://github.com/matrix-org/synapse/issues/10659 ))
- Split out
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) meta
events to their own fields in the `/batch_send` response.
([\#10777](https://github.com/matrix-org/synapse/issues/10777 ))
- Add missing type hints to REST servlets.
([\#10785](https://github.com/matrix-org/synapse/issues/10785 ),
[\#10817](https://github.com/matrix-org/synapse/issues/10817 ))
- Simplify the internal logic which maintains the user directory
database tables.
([\#10796](https://github.com/matrix-org/synapse/issues/10796 ))
- Use direct references to config flags.
([\#10812](https://github.com/matrix-org/synapse/issues/10812 ),
[\#10885](https://github.com/matrix-org/synapse/issues/10885 ),
[\#10893](https://github.com/matrix-org/synapse/issues/10893 ),
[\#10897](https://github.com/matrix-org/synapse/issues/10897 ))
- Specify the type of token in generic "Invalid token" error messages.
([\#10815](https://github.com/matrix-org/synapse/issues/10815 ))
- Make `StateFilter` frozen so it is hashable.
([\#10816](https://github.com/matrix-org/synapse/issues/10816 ))
- Fix a long-standing bug where an `m.room.message` event containing a
null byte would cause an internal server error.
([\#10820](https://github.com/matrix-org/synapse/issues/10820 ))
- Add type hints to the state database.
([\#10823](https://github.com/matrix-org/synapse/issues/10823 ))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to
hopefully improve `/sync` performance when you
haven't synced recently.
([\#10826](https://github.com/matrix-org/synapse/issues/10826 ))
- Track cache eviction rates more finely in Prometheus's monitoring.
([\#10829](https://github.com/matrix-org/synapse/issues/10829 ))
- Add missing type hints to `synapse.handlers`.
([\#10831](https://github.com/matrix-org/synapse/issues/10831 ),
[\#10856](https://github.com/matrix-org/synapse/issues/10856 ))
- Extend the Module API to let plug-ins check whether an ID is local and
to access IP + User Agent data.
([\#10833](https://github.com/matrix-org/synapse/issues/10833 ))
- Factor out PNG image data to a constant to be used in several tests.
([\#10834](https://github.com/matrix-org/synapse/issues/10834 ))
- Add a test to ensure state events sent by modules get persisted
correctly.
([\#10835](https://github.com/matrix-org/synapse/issues/10835 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
fields and event types from `chunk` to `batch` to match the
`/batch_send` endpoint.
([\#10838](https://github.com/matrix-org/synapse/issues/10838 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
`/batch_send` query parameter from `?prev_event` to more obvious usage
with `?prev_event_id`.
([\#10839](https://github.com/matrix-org/synapse/issues/10839 ))
- Add type hints to `synapse.http.site`.
([\#10867](https://github.com/matrix-org/synapse/issues/10867 ))
- Include outlier status when we log V2 or V3 events.
([\#10879](https://github.com/matrix-org/synapse/issues/10879 ))
- Break down Grafana's cache expiry time series based on reason for
eviction, c.f.
[\#10829](https://github.com/matrix-org/synapse/issues/10829 ).
([\#10880](https://github.com/matrix-org/synapse/issues/10880 ))
- Clean up some of the federation event authentication code for clarity.
([\#10883](https://github.com/matrix-org/synapse/issues/10883 ),
[\#10884](https://github.com/matrix-org/synapse/issues/10884 ),
[\#10896](https://github.com/matrix-org/synapse/issues/10896 ),
[\#10901](https://github.com/matrix-org/synapse/issues/10901 ))
- Allow the `.` and `~` characters when creating registration tokens as
per the change to
[MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ).
([\#10887](https://github.com/matrix-org/synapse/issues/10887 ))
- Clean up some unnecessary parentheses in places around the codebase.
([\#10889](https://github.com/matrix-org/synapse/issues/10889 ))
- Improve type hinting in the user directory code.
([\#10891](https://github.com/matrix-org/synapse/issues/10891 ))
- Update development testing script `test_postgresql.sh` to use a
supported Python version and make re-runs quicker.
([\#10906](https://github.com/matrix-org/synapse/issues/10906 ))
- Document and summarize changes in schema version `61` – `64`.
([\#10917](https://github.com/matrix-org/synapse/issues/10917 ))
- Update release script to sign the newly created git tags.
([\#10925](https://github.com/matrix-org/synapse/issues/10925 ))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build
their docs.
([\#10931](https://github.com/matrix-org/synapse/issues/10931 ))
F124520CEEE062448FE1C8442D2EFA2F32FBE047
<olivier@librepush.net>" [ultimate]
2021-09-29 14:23:39 +01:00
94b620a5ed
Use direct references for configuration variables (part 6). ( #10916 )
2021-09-29 06:44:15 -04:00
37bb93d181
Fix exception responding to request that has been closed ( #10932 )
...
Introduced in #10905
2021-09-28 14:36:19 +01:00
707d5e4e48
Encode JSON responses on a thread in C, mk2 ( #10905 )
...
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.
Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:
1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.
I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
2021-09-28 09:37:58 +00:00
50022cff96
Add reactor to `SynapseRequest` and fix up types. ( #10868 )
2021-09-24 11:01:25 +01:00
47854c71e9
Use direct references for configuration variables (part 4). ( #10893 )
2021-09-23 12:03:01 -04:00
aa2c027792
Remove unnecessary parentheses around tuples returned from methods ( #10889 )
2021-09-23 11:59:07 +01:00
b25a494779
Add types to http.site ( #10867 )
2021-09-21 16:41:27 +00:00
8c7a531e27
Use direct references for some configuration variables (part 2) ( #10812 )
2021-09-15 08:34:52 -04:00
14b8c0476f
Prevent logging context going missing on federation request timeout ( #10810 )
...
In `MatrixFederationHttpClient._send_request()`, we make a HTTP request
using an `Agent`, wrap that request in a timeout and await the resulting
`Deferred`. On its own, the `Agent` performing the HTTP request
correctly stashes and restores the logging context while waiting.
The addition of the timeout introduces a path where the logging context
is not restored when execution resumes.
To address this, we wrap the timeout `Deferred` in a
`make_deferred_yieldable()` to stash the logging context and restore it
on completion of the `await`. However this is not sufficient, since by
the time we construct the timeout `Deferred`, the `Agent` has already
stashed and cleared the logging context when using
`make_deferred_yieldable()` to produce its `Deferred` for the request.
Hence, we wrap the `Agent` request in a `run_in_background()` to "fork"
and preserve the logging context so that we can stash and restore it
when `await`ing the timeout `Deferred`.
This approach is similar to the one used with `defer.gatherResults`.
Note that the code is still not fully correct. When a timeout occurs,
the request remains running in the background (existing behavior which
is nothing to do with the new call to `run_in_background`) and may
re-start the logging context after it has finished.
2021-09-14 13:01:30 +01:00
66d72b7e17
Change logging of puppeted requests to better differentiate users ( #10779 )
...
This used to be a comma and got accidentally changed to a period in #9654 , but a pipe character is more easier to parse visually.
2021-09-08 12:59:15 +00:00
ecbfa4fe4f
Additional type hints for client REST servlets (part 5) ( #10736 )
...
Additionally this enforce type hints on all function signatures inside
of the synapse.rest.client package.
2021-09-03 09:22:22 -04:00
0c3565da4c
Additional type hints for the proxy agent and SRV resolver modules. ( #10608 )
2021-08-18 13:53:20 -04:00
0ace38b7b3
Experimental support for MSC3266 Room Summary API. ( #10394 )
2021-08-16 14:49:12 +00:00
339c3918e1
support federation queries through http connect proxy ( #10475 )
...
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-11 15:34:59 +01:00
d9cb658c78
Fix up type hints for Twisted 21.7 ( #10490 )
...
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
2021-07-28 12:04:11 +00:00
076deade02
allow specifying https:// proxy ( #10411 )
2021-07-27 17:31:06 +01:00
d8324b8238
Fix a handful of type annotations. ( #10446 )
...
* switch from `types.CoroutineType` to `typing.Coroutine`
these should be identical semantically, and since `defer.ensureDeferred` is
defined to take a `typing.Coroutine`, will keep mypy happy
* Fix some annotations on inlineCallbacks functions
* changelog
2021-07-22 12:00:16 +01:00
590cc4e888
Add type hints to additional servlet functions ( #10437 )
...
Improves type hints for:
* parse_{boolean,integer}
* parse_{boolean,integer}_from_args
* parse_json_{value,object}_from_request
And fixes any incorrect calls that resulted from unknown types.
2021-07-21 18:12:22 +00:00
5db118626b
Add a return type to parse_string. ( #10438 )
...
And set the required attribute in a few places which will error if
a parameter is not provided.
2021-07-21 09:47:56 -04:00
95e47b2e78
[pyupgrade] `synapse/` ( #10348 )
...
This PR is tantamount to running
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```
Part of #9744
2021-07-19 15:28:05 +01:00
bdfde6dca1
Use inline type hints in `http/federation/`, `storage/` and `util/` ( #10381 )
2021-07-15 12:46:54 -04:00
bf72d10dbf
Use inline type hints in various other places (in `synapse/`) ( #10380 )
2021-07-15 11:02:43 +01:00
c7603af1d0
Allow providing credentials to `http_proxy` ( #10360 )
2021-07-15 10:37:08 +01:00
7695ca0618
Fix a number of logged errors caused by remote servers being down. ( #10400 )
2021-07-15 10:35:46 +01:00
0555d7b0dc
Add additional types to the federation transport server. ( #10213 )
2021-06-28 07:36:41 -04:00
27c06a6e06
Drop Origin & Accept from Access-Control-Allow-Headers value ( #10114 )
...
* Drop Origin & Accept from Access-Control-Allow-Headers value
This change drops the Origin and Accept header names from the value of the
Access-Control-Allow-Headers response header sent by Synapse. Per the CORS
protocol, it’s not necessary or useful to include those header names.
Details:
Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name , Origin
is a “forbidden header name” set by the browser and that frontend
JavaScript code is never allowed to set.
So the value of Access-Control-Allow-Headers isn’t relevant to Origin or
in general to other headers set by the browser itself — the browser
never ever consults the Access-Control-Allow-Headers value to confirm
that it’s OK for the request to include an Origin header.
And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header ,
Accept is a “CORS-safelisted request-header”, which means that browsers
allow requests to contain the Accept header regardless of whether the
Access-Control-Allow-Headers value contains "Accept".
So it’s unnecessary for the Access-Control-Allow-Headers to explicitly
include Accept. Browsers will not perform a CORS preflight for requests
containing an Accept request header.
Related: https://github.com/matrix-org/matrix-doc/pull/3225
Signed-off-by: Michael[tm] Smith <mike@w3.org>
2021-06-23 11:25:03 +01:00
8070b893db
update black to 21.6b0 ( #10197 )
...
Reformat all files with the new version.
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2021-06-17 15:20:06 +01:00
aac2c49b9b
Fix 'ip_range_whitelist' not working for federation servers ( #10115 )
...
Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers.
Signed-off-by: Michael Kutzner 1mikure@gmail.com
2021-06-15 08:53:55 +01:00
d936371b69
Implement knock feature ( #6739 )
...
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403
Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
1bf83a191b
Clean up the interface for injecting opentracing over HTTP ( #10143 )
...
* Remove unused helper functions
* Clean up the interface for injecting opentracing over HTTP
* changelog
2021-06-09 11:33:00 +01:00
c7f3fb2745
Add type hints to the federation server transport. ( #10080 )
2021-06-08 11:19:25 -04:00
9e4610cc27
Correct type hints for parse_string(s)_from_args. ( #10137 )
2021-06-08 08:30:48 -04:00
3ff6fe2851
Merge branch 'master' into develop
2021-06-01 13:47:27 +01:00
1641c5c707
Log method and path when dropping request due to size limit ( #10091 )
2021-05-28 16:29:09 +01:00
84cf3e47a0
Allow response of `/send_join` to be larger. ( #10093 )
...
Fixes #10087 .
2021-05-28 16:28:01 +01:00
3f96dbbda7
Log method and path when dropping request due to size limit ( #10091 )
2021-05-28 15:57:53 +01:00
ac3e02d089
Add `parse_strings_from_args` to get `prev_events` array ( #10048 )
...
Split out from https://github.com/matrix-org/synapse/pull/9247
Strings:
- `parse_string`
- `parse_string_from_args`
- `parse_strings_from_args`
For comparison with ints:
- `parse_integer`
- `parse_integer_from_args`
Previous discussions:
- https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687
- https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156
- https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791
Signed-off-by: Eric Eastwood <erice@element.io>
2021-05-28 14:19:06 +01:00
64887f06fc
Use ijson to parse the response to `/send_join`, reducing memory usage. ( #9958 )
...
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
2021-05-20 16:11:48 +01:00
3ff2251754
Improved validation for received requests ( #9817 )
...
* Simplify `start_listening` callpath
* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
59d24c5bef
pass a reactor into SynapseSite ( #9874 )
2021-04-23 17:06:47 +01:00
51a20914a8
Limit the size of HTTP responses read over federation. ( #9833 )
2021-04-23 11:08:41 +01:00
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
1d5f0e3529
Bump black configuration to target py36 ( #9781 )
...
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
2ca4e349e9
Bugbear: Add Mutable Parameter fixes ( #9682 )
...
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
48d44ab142
Record more information into structured logs. ( #9654 )
...
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
e2b8a90897
Update mypy configuration: `no_implicit_optional = True` ( #9742 )
2021-04-05 09:10:18 -04:00
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. ( #9442 )
2021-03-29 12:15:33 -04:00
b7748d3c00
Import HomeServer from the proper module. ( #9665 )
2021-03-23 07:12:48 -04:00
5b268997bd
Allow providing credentials to HTTPS_PROXY ( #9657 )
...
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70
This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization )) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.
Also added some type hints.
2021-03-22 17:20:47 +00:00
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. ( #9608 )
2021-03-15 11:14:39 -04:00
55da8df078
Fix additional type hints from Twisted 21.2.0. ( #9591 )
2021-03-12 11:37:57 -05:00
e55bd0e110
Add tests for blacklisting reactor/agent. ( #9563 )
2021-03-11 09:15:22 -05:00
9cd18cc588
Retry 5xx errors in federation client ( #9567 )
...
Fixes #8915
2021-03-09 13:15:12 +00:00
7fdc6cefb3
Fix additional type hints. ( #9543 )
...
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09 07:41:32 -05:00
58114f8a17
Create a SynapseReactor type which incorporates the necessary reactor interfaces. ( #9528 )
...
This helps fix some type hints when running with Twisted 21.2.0.
2021-03-08 08:25:43 -05:00
33a02f0f52
Fix additional type hints from Twisted upgrade. ( #9518 )
2021-03-03 15:47:38 -05:00
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. ( #9497 )
...
By consuming the response if the headers imply that the
content is too large.
2021-03-01 12:45:00 -05:00
ddb240293a
Add support for no_proxy and case insensitive env variables ( #9372 )
...
### Changes proposed in this PR
- Add support for the `no_proxy` and `NO_PROXY` environment variables
- Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519)bdb941be42/Lib/urllib/request.py (L2488)#9306 which also uses `ProxyAgent`
Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
15090de850
SSO: redirect to public URL before setting cookies ( #9436 )
...
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
d8e95e5452
Add support for X-Forwarded-Proto ( #9472 )
...
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
2021-02-24 18:11:33 +00:00
8ec2217103
Reduce the memory usage of previewing media files. ( #9421 )
...
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.
It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
2021-02-18 09:01:29 -05:00
0a00b7ff14
Update black, and run auto formatting over the codebase ( #9381 )
...
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md
2021-02-16 22:32:34 +00:00
b0f4119b8b
Add debug logging to DNS SRV requests. ( #9305 )
2021-02-03 16:47:30 -05:00
f78d07bf00
Split out a separate endpoint to complete SSO registration ( #9262 )
...
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
a737cc2713
Implement MSC2858 support ( #9183 )
...
Fixes #8928 .
2021-01-27 12:41:24 +00:00
0cd2938bc8
Support icons for Identity Providers ( #9154 )
2021-01-20 08:15:14 -05:00
2b467d0b61
Properly raise an exception when the body exceeds the max size. ( #9145 )
...
...instead of just creating the exception object and doing nothing with it.
2021-01-18 10:21:42 -05:00
74dd906041
Avoid raising the body exceeded error multiple times. ( #9108 )
...
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
2021-01-15 11:00:13 -05:00
12702be951
Fix wrong arguments being passed to BlacklistingAgentWrapper ( #9098 )
...
A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper
used by the WellyKnownResolver. This coulld cause exceptions when attempting to
connect to IP addresses that are blacklisted, but in reality this did not have any
observable affect since this code is not used for IP literals.
2021-01-14 06:59:26 -05:00
aee8e6a95d
Reduce scope of exception handler. ( #9106 )
...
Removes a bare `except Exception` clause and replaces it with
catching a specific exception around the portion that might throw.
2021-01-13 13:27:49 -05:00
aa4d8c1f9a
Merge branch 'master' into develop
2021-01-13 10:36:55 +00:00
e385c8b473
Don't apply the IP range blacklist to proxy connections ( #9084 )
...
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
2021-01-12 12:20:30 -05:00
723b19748a
Handle bad JSON data being returned from the federation API. ( #9070 )
2021-01-12 11:07:01 -05:00
2ec8ca5e60
Remove SynapseRequest.get_user_agent ( #9069 )
...
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the
LoggingContext and write the right entries to the request log).
Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.
In short: move get_user_agent out to a utility function.
2021-01-12 12:34:16 +00:00
ff5c4da128
Add a maximum size for well-known lookups. ( #8950 )
2020-12-16 17:25:24 -05:00
1619802228
Various clean-ups to the logging context code ( #8935 )
2020-12-14 14:19:47 -05:00
1821f7cc26
Fix buglet in DirectRenderJsonResource ( #8897 )
...
this was using `canonical_json` without setting it, so when you used it as a
standalone class, you would get exceptions.
2020-12-10 12:42:55 +00:00
30fba62108
Apply an IP range blacklist to push and key revocation requests. ( #8821 )
...
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:
* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events
The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
3f0cba657c
Allow Date header through CORS ( #8804 )
2020-12-01 13:24:56 +00:00
856eab606b
Remove special case of pretty printing JSON responses for curl ( #8833 )
...
* Remove special case of pretty printing JSON responses for curl
Signed-off-by: Tulir Asokan <tulir@maunium.net>
2020-11-27 12:37:55 +00:00
968939bdac
Add additional type hints to HTTP client. ( #8812 )
...
This also removes some duplicated code between the simple
HTTP client and matrix federation client.
2020-11-25 13:30:47 -05:00
f38676d161
Add type hints to matrix federation client / agent. ( #8806 )
2020-11-25 07:07:21 -05:00
eedaf90c84
Better error message when a remote resource uses invalid Content-Type ( #8719 )
2020-11-11 14:22:40 +00:00
f21e24ffc2
Add ability for access tokens to belong to one user but grant access to another user. ( #8616 )
...
We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).
A future PR will add an API for creating such a token.
When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29 15:58:44 +00:00
c97da1e45d
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
...
Fix serialisation errors when using third-party event rules.
2020-10-28 20:41:42 +00:00
e80eb69887
remove unused imports
2020-10-28 16:18:05 +00:00
b6ca69e4f1
Remove frozendict_json_encoder and support frozendicts everywhere
...
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
2020-10-28 15:56:57 +00:00
aff1eb7c67
Tell Black to format code for Python 3.5 ( #8664 )
...
This allows trailing commas in multi-line arg lists.
Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.
Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27 23:26:36 +00:00
34a5696f93
Fix typos and spelling errors. ( #8639 )
2020-10-23 12:38:40 -04:00
c850dd9a8e
Fix handling of User-Agent headers with bad utf-8. ( #8632 )
2020-10-23 17:12:59 +01:00
514a240aed
Remove unused OPTIONS handlers. ( #8621 )
...
The handling of OPTIONS requests was consolidated in #7534 , but the endpoint
specific handlers were not removed.
2020-10-22 08:35:55 -04:00
3e58ce72b4
Don't bother responding to client requests that have already disconnected ( #8465 )
...
This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so.
This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003 .
Fixes https://github.com/matrix-org/synapse/issues/6700
Fixes https://github.com/matrix-org/synapse/issues/5304
2020-10-06 10:03:39 +01:00
Dirk Klimpel