Commit Graph

1736 Commits (a2070a2c4e008ceff6decce3d569f984d5e0f902)

Author SHA1 Message Date
Richard van der Hoff d6752ce5da
Clean up startup for the pusher (#6558)
* Remove redundant python2 support code

`str.decode()` doesn't exist on python3, so presumably this code was doing
nothing

* Filter out pushers with corrupt data

When we get a row with unparsable json, drop the row, rather than returning a
row with null `data`, which will then cause an explosion later on.

* Improve logging when we can't start a pusher

Log the ID to help us understand the problem

* Make email pusher setup more robust

We know we'll have a `data` member, since that comes from the database. What we
*don't* know is if that is a dict, and if that has a `brand` member, and if
that member is a string.
2019-12-18 14:26:58 +00:00
Erik Johnston 72acca6a32 Back out change preventing setting null avatar URLs 2019-12-11 11:46:55 +00:00
Andrew Morgan 5e8abe9013
Better errors regarding changing avatar_url (#6497) 2019-12-09 14:54:33 +00:00
Erik Johnston 2ace775d88 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_database_class 2019-12-06 11:33:34 +00:00
Manuel Stahl 649b6bc088 Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925) 2019-12-05 18:12:23 +00:00
Erik Johnston 9c41ba4c5f Port rest.client.v2 2019-12-05 16:56:23 +00:00
Erik Johnston 1a0997bbd5 Port rest/v1 to async/await 2019-12-05 15:57:28 +00:00
Erik Johnston 4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Brendan Abolivier 9dc84b7989
Merge branch 'develop' into babolivier/context_filters 2019-12-04 14:23:44 +00:00
Filip Štědronský 81731c6e75 Fix: Pillow error when uploading RGBA image (#3325) (#6241)
Signed-Off-By: Filip Štědronský <g@regnarg.cz>
2019-12-02 12:12:55 +00:00
Andrew Morgan 23ea572125
Add User-Interactive Auth to /account/3pid/add (#6119) 2019-11-29 13:51:14 +00:00
Erik Johnston 69d8fb83c6 MSC2367 Allow reason field on all member events 2019-11-28 11:02:04 +00:00
Hubert Chathi 0d27aba900
add etag and count to key backup endpoints (#5858) 2019-11-27 16:14:44 -05:00
Richard van der Hoff ef1a85e773
Fix startup error when http proxy is defined. (#6421)
Guess I only tested this on python 2 :/

Fixes #6419.
2019-11-26 18:10:50 +00:00
Andrew Morgan 3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Manuel Stahl 4f5ca455bf Move admin endpoints into separate files (#6308) 2019-11-20 11:49:11 +00:00
Brendan Abolivier 83446a18fb
Merge pull request #6335 from matrix-org/erikj/rc_login_cleanups
Only do `rc_login` ratelimiting on succesful login.
2019-11-20 09:52:38 +00:00
Brendan Abolivier 271c322d08
Lint 2019-11-20 09:29:48 +00:00
Erik Johnston c7376cdfe3
Apply suggestions from code review
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
2019-11-18 17:10:16 +00:00
Erik Johnston 541f1b92d9 Only do `rc_login` ratelimiting on succesful login.
We were doing this in a number of places which meant that some login
code paths incremented the counter multiple times.

It was also applying ratelimiting to UIA endpoints, which was probably
not intentional.

In particular, some custom auth modules were calling
`check_user_exists`, which incremented the counters, meaning that people
would fail to login sometimes.
2019-11-06 11:08:58 +00:00
Richard van der Hoff 5570d1c93f
Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2
Fix exception when OpenGraph tag values are ints
2019-11-05 17:28:11 +00:00
Richard van der Hoff 81d49cbb07 Fix exception when OpenGraph tag values are ints 2019-11-05 17:22:58 +00:00
Richard van der Hoff 55a7da247a
Merge branch 'develop' into rav/url_preview_limit_title 2019-11-05 17:08:07 +00:00
Richard van der Hoff e78167c94b
Apply suggestions from code review
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-11-05 16:46:39 +00:00
Richard van der Hoff e9bfe719ba Strip overlong OpenGraph data from url preview
... to stop people causing DoSes with malicious web pages
2019-11-05 15:51:18 +00:00
Brendan Abolivier 8822b33111
Update copyrights 2019-11-05 14:46:35 +00:00
Brendan Abolivier f496d25877
Merge pull request #6301 from matrix-org/babolivier/msc2326
Implement MSC2326 (label based filtering)
2019-11-01 17:04:45 +00:00
Richard van der Hoff 1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Brendan Abolivier e3689ac6f7
Add unstable feature flag 2019-11-01 10:41:23 +00:00
Amber Brown 020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Andrew Morgan 54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston 770d1ef673
Merge pull request #6280 from matrix-org/erikj/receipts_async_await
Port receipt and read markers to async/wait
2019-10-30 11:44:18 +01:00
Erik Johnston 2c35ffead2 Port receipt and read markers to async/wait 2019-10-29 15:08:22 +00:00
Erik Johnston 9be41bc121 Port room rest handlers to async/await 2019-10-29 13:09:29 +00:00
Michael Kaye e4d98188da Address codestyle concerns 2019-10-24 18:43:13 +01:00
Michael Kaye 8f4a808d9d Delay printf until logging is required.
Using % will cause the string to be generated even if debugging
is off.
2019-10-24 18:31:53 +01:00
Hubert Chathi 36adfaedab Merge branch 'develop' into cross-signing_sig_upload 2019-10-18 18:34:42 +01:00
Hubert Chathi cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Erik Johnston d98029ea89
Merge pull request #6196 from matrix-org/erikj/await
Move rest/admin to use async/await.
2019-10-18 11:53:02 +02:00
Erik Johnston 2e97a4c197 Port synapse/rest/client/_base.py to async/await 2019-10-11 15:29:26 +01:00
Erik Johnston 3c2d6c708c Add maybe_awaitable and fix __init__ bugs 2019-10-11 15:26:09 +01:00
Hubert Chathi 691dd67fcd
Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version
make version optional in body of e2e backup version update
2019-10-11 10:11:59 -04:00
Valérian Rousset be9b55e0d2 cas: support setting display name (#6114)
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Erik Johnston dfbb62c28d Port synaps/rest/client/users.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston f95325e22a Port synaps/rest/client/server_notice_servlet.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston 281f887090 Port synaps/rest/client/media.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston fca3a541e7 Port rest/admin/__init__.py to async/await 2019-10-11 12:17:52 +01:00
Erik Johnston ca3e01e50d Fix store_url_cache using bytes 2019-10-10 14:52:29 +01:00
krombel 2efd050c9d send 404 as http-status when filter-id is unknown to the server (#2380)
This fixed the weirdness of 400 vs 404 as http status code in the case
the filter id is not known by the server.
As e.g. matrix-js-sdk expects 404 to catch this situation this leads
to unwanted behaviour.
2019-10-10 12:59:55 +01:00
Hubert Chathi 4535a07f4a make version optional in body of e2e backup version update
to agree with latest version of the MSC
2019-10-09 17:54:03 -04:00
Anshul Angaria 474abf1eb6 add M_TOO_LARGE error code for uploading a too large file (#6151)
Fixes #6109
2019-10-08 13:55:16 +01:00
Michael Kaye dc795ba709 Log responder we are using. (#6139)
This prevents us logging "Responding to media request with responder %s".
2019-10-07 15:41:25 +01:00
Robert Swain 39b40d6d99 media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142)
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
Erik Johnston a5166e4d5f
Land improved room list based on room stats (#6019)
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Erik Johnston 67ee18daea
Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span
Add some helpful opentracing tags and remove get_user_by_req span
2019-09-27 10:02:57 +01:00
Richard van der Hoff 54569c787b
Kill off half-implemented password-reset via sms (#6101)
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.

See also matrix-org/matrix-doc#2303.
2019-09-26 15:38:25 +01:00
Richard van der Hoff 990928abde
Stop advertising unsupported flows for registration (#6107)
If email or msisdn verification aren't supported, let's stop advertising them
for registration.

Fixes #6100.
2019-09-25 12:10:26 +01:00
Erik Johnston dc2c97e1a3 isort 2019-09-25 11:59:05 +01:00
Erik Johnston 5d99713854 Add tags for event_id and txn_id in event sending
This will make it easier to search for sending event requests.
2019-09-25 11:39:15 +01:00
Richard van der Hoff 8004d6ca2f
Refactor code for calculating registration flows (#6106)
because, frankly, it looked like it was written by an axe-murderer.

This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
2019-09-25 11:32:05 +01:00
Richard van der Hoff ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Andrew Morgan 2b071a2ff1 Add an unstable feature flag for separate add/bind 3pid APIs (#6044)
Add a m.separate_add_and_bind flag set to True. See MSC2290's Backward Compatibility section for details.
2019-09-23 20:46:34 +01:00
Andrew Morgan 1ea3ed7620 Add m.id_access_token to /versions unstable_features (MSC2264) (#5974)
Adds a flag to /versions' unstable_features section indicating that this Synapse understands what an id_access_token is, as per MSC2264.

Fixes #5927
2019-09-23 20:19:25 +01:00
Andrew Morgan b38aa82b83 Add m.require_identity_server to /versions unstable_flags (#5972)
As MSC2263 states, m.require_identity_server must be set to false when it does not require an identity server to be provided by the client for the purposes of email registration or password reset.

Adds an m.require_identity_server flag to /versionss unstable_flags section. This will advertise that Synapse no longer needs id_server as a parameter.
2019-09-23 18:52:43 +01:00
Andrew Morgan 2c99c63453 Add POST submit_token endpoint for MSISDN (#6078)
First part of solving #6076
2019-09-23 17:49:00 +01:00
Andrew Morgan 30af161af2 Implement MSC2290 (#6043)
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.

This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints.

Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
Andrew Morgan 1b519e0272 Disable /register/available if registration is disabled (#6082)
Fixes #6066

This register endpoint should be disabled if registration is disabled, otherwise we're giving anyone the ability to check if a username exists on a server when we don't need to be.

Error code is 403 (Forbidden) as that's the same returned by /register when registration is disabled.
2019-09-23 15:38:38 +01:00
Andrew Morgan df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Andrew Morgan 7763dd3e95
Remove trailing slash ability from password reset's submit_token endpoint (#6074)
Remove trailing slash ability from the password reset submit_token endpoint. Since we provide the link in an email, and have never sent it with a trailing slash, there's no point for us to accept them on the endpoint.
2019-09-20 14:58:37 +01:00
Andrew Morgan 3ac614eb6c
Drop support for bind param on POST /account/3pid (MSC2290) (#6067)
As per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/files#diff-05cde9463e9209b701312b3baf2fb2ebR151), we're dropping the bind parameter from `/account/3pid`. This endpoint can now only be used for adding threepid's to the user's account on the homeserver.
2019-09-20 10:46:34 +01:00
J. Ryan Stinnett 36015d68ef Use unstable prefix for 3PID unbind API (#6062) 2019-09-19 22:28:29 +01:00
Richard van der Hoff b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Andrew Morgan 62e3ff92fd
Remove POST method from password reset submit_token endpoint (#6056)
Removes the POST method from `/password_reset/<medium>/submit_token/` as it's only used by phone number verification which Synapse does not support yet.
2019-09-19 10:53:14 +01:00
Richard van der Hoff a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Andrew Morgan 9fc71dc5ee
Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)
This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted.

Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140).

Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
2019-09-11 16:02:42 +01:00
Andrew Morgan b5833a2abf Add changelog 2019-09-10 17:56:10 +01:00
Andrew Morgan 60d3c57bd0 Use account_threepid_delegate for 3pid validation 2019-09-10 17:56:10 +01:00
Hubert Chathi 8e86f5b65c Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-09-07 13:20:34 -04:00
Hubert Chathi 0d61d1d735 Merge branch 'develop' into cross-signing_sig_upload 2019-09-07 13:14:45 -04:00
Andrew Morgan cf5a420c8a
Apply suggestions from code review
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-09-06 13:34:42 +01:00
Andrew Morgan ca74b140f2 Fix destructuring assumption bug 2019-09-06 13:25:06 +01:00
Andrew Morgan 0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Andrew Morgan a0d294c306
Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (#5892) 2019-09-05 14:31:22 +01:00
Andrew Morgan 90d17a3d28
Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (#5980)
Implements `POST /_matrix/client/r0/account/3pid/unbind` from [MSC2140](https://github.com/matrix-org/matrix-doc/blob/dbkr/tos_2/proposals/2140-terms-of-service-2.md#post-_matrixclientr0account3pidunbind).
2019-09-05 14:00:30 +01:00
Hubert Chathi c8dc740a94 update with newer coding style 2019-09-04 22:30:45 -04:00
Hubert Chathi 7d6c70fc7a make black happy 2019-09-04 22:04:12 -04:00
Hubert Chathi ac4746ac4b allow uploading signatures of master key signed by devices 2019-09-04 22:03:41 -04:00
Hubert Chathi 4bb4544784 implement device signature uploading/fetching 2019-09-04 20:02:56 -04:00
Hubert Chathi faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Andrew Morgan b736c6cd3a
Remove bind_email and bind_msisdn (#5964)
Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107).
2019-09-04 18:24:23 +01:00
Travis Ralston 2f416fc997
Ensure the list media admin API is always available (#5966)
* Ensure the list media admin API is always available

This API is required for some external media repo implementations to operate (mostly for doing quarantine operations on a room).

* changelog
2019-09-03 13:35:20 -06:00
Travis Ralston 0eac7077c9
Ensure an auth instance is available to ListMediaInRoom (#5967)
* Ensure an auth instance is available to ListMediaInRoom

Fixes https://github.com/matrix-org/synapse/issues/5737

* Changelog
2019-09-03 09:01:30 -06:00
Andrew Morgan 2a44782666
Remove double return statements (#5962)
Remove all the "double return" statements which were a result of us removing all the instances of

```
defer.returnValue(...)
return
```

statements when we switched to python3 fully.
2019-09-03 11:42:45 +01:00
Jorik Schellekens a90d16dabc
Opentrace device lists (#5853)
Trace device list changes.
2019-09-03 10:21:30 +01:00
L0ric0 ce7803b8b0 fix thumbnail storage location (#5915)
* fix thumbnail storage location

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>

* Add changelog file.

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>

* Update Changelog

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
2019-09-02 12:18:41 +01:00
Andrew Morgan 2a012e8a04
Revert "Add m.id_access_token flag (#5930)" (#5945)
This reverts commit 4765f0cfd9.
2019-08-30 17:13:37 +01:00
Andrew Morgan 4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Andrew Morgan 4765f0cfd9
Add m.id_access_token flag (#5930)
Adds a flag to `/versions`' `unstable_features` section indicating that this Synapse understands what an `id_access_token` is, as per https://github.com/matrix-org/synapse/issues/5927#issuecomment-523566043

Fixes #5927
2019-08-30 15:22:51 +01:00
Hubert Chathi e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
reivilibre a3f0635686
Merge pull request #5914 from matrix-org/rei/admin_getadmin
Add GET method to admin API /users/@user:dom/admin
2019-08-28 09:44:22 +01:00