Commit Graph

705 Commits (c7603af1d06d65932c420ae76002b6ed94dbf23c)

Author SHA1 Message Date
Dirk Klimpel c7603af1d0
Allow providing credentials to `http_proxy` (#10360) 2021-07-15 10:37:08 +01:00
Erik Johnston 7695ca0618
Fix a number of logged errors caused by remote servers being down. (#10400) 2021-07-15 10:35:46 +01:00
Patrick Cloke 0555d7b0dc
Add additional types to the federation transport server. (#10213) 2021-06-28 07:36:41 -04:00
Michael[tm] Smith 27c06a6e06
Drop Origin & Accept from Access-Control-Allow-Headers value (#10114)
* Drop Origin & Accept from Access-Control-Allow-Headers value

This change drops the Origin and Accept header names from the value of the
Access-Control-Allow-Headers response header sent by Synapse. Per the CORS
protocol, it’s not necessary or useful to include those header names.

Details:

Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin
is a “forbidden header name” set by the browser and that frontend
JavaScript code is never allowed to set.

So the value of Access-Control-Allow-Headers isn’t relevant to Origin or
in general to other headers set by the browser itself — the browser
never ever consults the Access-Control-Allow-Headers value to confirm
that it’s OK for the request to include an Origin header.

And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header,
Accept is a “CORS-safelisted request-header”, which means that browsers
allow requests to contain the Accept header regardless of whether the
Access-Control-Allow-Headers value contains "Accept".

So it’s unnecessary for the Access-Control-Allow-Headers to explicitly
include Accept. Browsers will not perform a CORS preflight for requests
containing an Accept request header.

Related: https://github.com/matrix-org/matrix-doc/pull/3225

Signed-off-by: Michael[tm] Smith <mike@w3.org>
2021-06-23 11:25:03 +01:00
Marcus 8070b893db
update black to 21.6b0 (#10197)
Reformat all files with the new version.

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2021-06-17 15:20:06 +01:00
Michael Kutzner aac2c49b9b
Fix 'ip_range_whitelist' not working for federation servers (#10115)
Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers.

Signed-off-by: Michael Kutzner 1mikure@gmail.com
2021-06-15 08:53:55 +01:00
Sorunome d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Richard van der Hoff 1bf83a191b
Clean up the interface for injecting opentracing over HTTP (#10143)
* Remove unused helper functions

* Clean up the interface for injecting opentracing over HTTP

* changelog
2021-06-09 11:33:00 +01:00
Patrick Cloke c7f3fb2745
Add type hints to the federation server transport. (#10080) 2021-06-08 11:19:25 -04:00
Patrick Cloke 9e4610cc27
Correct type hints for parse_string(s)_from_args. (#10137) 2021-06-08 08:30:48 -04:00
Andrew Morgan 3ff6fe2851 Merge branch 'master' into develop 2021-06-01 13:47:27 +01:00
Erik Johnston 1641c5c707 Log method and path when dropping request due to size limit (#10091) 2021-05-28 16:29:09 +01:00
Erik Johnston 84cf3e47a0
Allow response of `/send_join` to be larger. (#10093)
Fixes #10087.
2021-05-28 16:28:01 +01:00
Erik Johnston 3f96dbbda7
Log method and path when dropping request due to size limit (#10091) 2021-05-28 15:57:53 +01:00
Eric Eastwood ac3e02d089
Add `parse_strings_from_args` to get `prev_events` array (#10048)
Split out from https://github.com/matrix-org/synapse/pull/9247

Strings:

 - `parse_string`
 - `parse_string_from_args`
 - `parse_strings_from_args`

For comparison with ints:

 - `parse_integer`
 - `parse_integer_from_args`

Previous discussions:

 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687
 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156
 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791

Signed-off-by: Eric Eastwood <erice@element.io>
2021-05-28 14:19:06 +01:00
Erik Johnston 64887f06fc
Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958)
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
2021-05-20 16:11:48 +01:00
Richard van der Hoff 3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Richard van der Hoff 59d24c5bef
pass a reactor into SynapseSite (#9874) 2021-04-23 17:06:47 +01:00
Richard van der Hoff 51a20914a8
Limit the size of HTTP responses read over federation. (#9833) 2021-04-23 11:08:41 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Dan Callahan 1d5f0e3529
Bump black configuration to target py36 (#9781)
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
Jonathan de Jong 2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Patrick Cloke 48d44ab142
Record more information into structured logs. (#9654)
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
Jonathan de Jong e2b8a90897
Update mypy configuration: `no_implicit_optional = True` (#9742) 2021-04-05 09:10:18 -04:00
Patrick Cloke 01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Patrick Cloke b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Andrew Morgan 5b268997bd
Allow providing credentials to HTTPS_PROXY (#9657)
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.
2021-03-22 17:20:47 +00:00
Patrick Cloke d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. (#9608) 2021-03-15 11:14:39 -04:00
Patrick Cloke 55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Patrick Cloke e55bd0e110
Add tests for blacklisting reactor/agent. (#9563) 2021-03-11 09:15:22 -05:00
Erik Johnston 9cd18cc588
Retry 5xx errors in federation client (#9567)
Fixes #8915
2021-03-09 13:15:12 +00:00
Patrick Cloke 7fdc6cefb3
Fix additional type hints. (#9543)
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09 07:41:32 -05:00
Patrick Cloke 58114f8a17
Create a SynapseReactor type which incorporates the necessary reactor interfaces. (#9528)
This helps fix some type hints when running with Twisted 21.2.0.
2021-03-08 08:25:43 -05:00
Patrick Cloke 33a02f0f52
Fix additional type hints from Twisted upgrade. (#9518) 2021-03-03 15:47:38 -05:00
Patrick Cloke 16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. (#9497)
By consuming the response if the headers imply that the
content is too large.
2021-03-01 12:45:00 -05:00
Tim Leung ddb240293a
Add support for no_proxy and case insensitive env variables (#9372)
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
Richard van der Hoff 15090de850
SSO: redirect to public URL before setting cookies (#9436)
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
Richard van der Hoff d8e95e5452
Add support for X-Forwarded-Proto (#9472)
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
2021-02-24 18:11:33 +00:00
Patrick Cloke 8ec2217103
Reduce the memory usage of previewing media files. (#9421)
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.

It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
2021-02-18 09:01:29 -05:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Marcus b0f4119b8b
Add debug logging to DNS SRV requests. (#9305) 2021-02-03 16:47:30 -05:00
Richard van der Hoff f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262)
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
Richard van der Hoff a737cc2713
Implement MSC2858 support (#9183)
Fixes #8928.
2021-01-27 12:41:24 +00:00
Richard van der Hoff 0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Patrick Cloke 2b467d0b61
Properly raise an exception when the body exceeds the max size. (#9145)
...instead of just creating the exception object and doing nothing with it.
2021-01-18 10:21:42 -05:00
Patrick Cloke 74dd906041
Avoid raising the body exceeded error multiple times. (#9108)
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
2021-01-15 11:00:13 -05:00
Tim Leung 12702be951
Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098)
A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper
used by the WellyKnownResolver. This coulld cause exceptions when attempting to
connect to IP addresses that are blacklisted, but in reality this did not have any
observable affect since this code is not used for IP literals.
2021-01-14 06:59:26 -05:00
Patrick Cloke aee8e6a95d
Reduce scope of exception handler. (#9106)
Removes a bare `except Exception` clause and replaces it with
catching a specific exception around the portion that might throw.
2021-01-13 13:27:49 -05:00
Erik Johnston aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Marcus e385c8b473
Don't apply the IP range blacklist to proxy connections (#9084)
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
2021-01-12 12:20:30 -05:00