Commit Graph

10756 Commits (9d06fb9cb105b33762a55b1d385da81c84507f71)

Author SHA1 Message Date
Brendan Abolivier ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
Brendan Abolivier 21d51ab598
Typo 2019-10-04 11:21:24 +01:00
Brendan Abolivier 4676732ca0
Lint 2019-10-04 11:18:28 +01:00
Brendan Abolivier 81d51ce48b
Incorporate review 2019-10-04 11:16:19 +01:00
Alexander Maznev 13c4345c84 Update `user_filters` table to have a unique index, and non-null columns (#1172) 2019-10-04 10:34:16 +01:00
Robert Swain 39b40d6d99 media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142)
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
Erik Johnston 6511071837 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships 2019-10-03 17:51:06 +01:00
Richard van der Hoff 66537e10ce
add some metrics on the federation sender (#6160) 2019-10-03 17:47:20 +01:00
Erik Johnston ab8a64772b Remove unused variable 2019-10-03 17:42:32 +01:00
Erik Johnston 693156aaf4 Don't regenerate numeric user ID if registration fails.
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
Erik Johnston 91f61fc6d7 Use the right error.... 2019-10-03 17:28:31 +01:00
Erik Johnston 84691da6c3 pep8 2019-10-03 17:27:18 +01:00
Erik Johnston a9610cdf02 Fixup names and comments 2019-10-03 17:26:56 +01:00
Erik Johnston d89ebf7c25 cachedList descriptor doesn't like typing 2019-10-03 17:23:11 +01:00
Erik Johnston c8145af8a9 Cache room membership lookups in _get_joined_users_from_context 2019-10-03 17:11:04 +01:00
Andrew Morgan 6018bec919 Merge branch 'master' into develop 2019-10-03 13:48:45 +01:00
Andrew Morgan ecb69d824a 1.4.0 2019-10-03 13:22:44 +01:00
Andrew Morgan 0f46bf5737
Replace client_secret with <redacted> in server logs (#6158)
Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
2019-10-03 12:57:26 +01:00
Brendan Abolivier 6527fa18c1 Add test case 2019-10-03 11:24:36 +01:00
Erik Johnston dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
Erik Johnston 22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
Erik Johnston 5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
Erik Johnston 7a5f080f91 Fix appservice room list pagination 2019-10-02 15:47:22 +01:00
Erik Johnston 4c4f44930d Fix not showing non-federatable rooms to remote room list queries 2019-10-02 15:20:36 +01:00
Erik Johnston 03cf4385e0 Fix public room list pagination.
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
Erik Johnston a5166e4d5f
Land improved room list based on room stats (#6019)
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Andrew Morgan aec1377d0b 1.4.0rc2 2019-10-02 13:55:00 +01:00
Amber Brown 864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Brendan Abolivier 972c9f65d7
Lint 2019-10-02 12:17:46 +01:00
Andrew Morgan 2a1470cd05
Fix yields and copy instead of move push rules on room upgrade (#6144)
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.

For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.

Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
Erik Johnston d69fd53f74 Bound find_next_generated_user_id DB query.
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
Brendan Abolivier 5705ecaec6
Don't 500 code when trying to exchange a revoked 3PID invite
While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state.
When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events.
When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it).

This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
2019-10-02 11:19:43 +01:00
Erik Johnston ecd254bc49 Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop 2019-10-02 11:08:07 +01:00
Erik Johnston b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
Erik Johnston 35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions
Fix fetching censored redactions from DB
2019-10-02 10:57:30 +01:00
Erik Johnston f44f1d2e83 Fix errors storing large retry intervals.
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.

To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
Erik Johnston ce7a3e7e27 Fix fetching censored redactions from DB
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
Erik Johnston 5e8387af9e Use `received_ts` to find uncensored redacted events
Joining against `events` and ordering by `stream_ordering` is
inefficient as it forced scanning the entirety of the redactions table.

This isn't the case if we use `redactions.received_ts` column as we can
then use an index.
2019-10-01 13:43:48 +01:00
Erik Johnston 898dde981b Add received_ts column to redactions.
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
Erik Johnston a27fb7d5ca Don't repeatedly attempt to censor events we don't have.
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.

When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
Erik Johnston 1d349fb159 Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop 2019-10-01 10:17:24 +01:00
Erik Johnston 9267741a5f Fix `devices_last_seen` background update.
Fixes #6134.
2019-09-30 11:58:36 +01:00
Richard van der Hoff 16cb9a71b8
Drop unused tables (#6115)
These tables are unused since #5893 (as amended by #6047), so we can now drop
them.

Fixes #6048.
2019-09-30 09:38:41 +01:00
Andrew Morgan f3451118a6
Edit SimpleHttpClient to reference that header keys can be passed as str or bytes (#6077) 2019-09-27 17:59:18 +01:00
Brendan Abolivier 3e42d47a5a
Incorporate review 2019-09-27 16:15:01 +01:00
Brendan Abolivier af92110c46
Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:12:15 +01:00
Brendan Abolivier bbe2a0f339
Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:36 +01:00
Brendan Abolivier 873fe7883c
Lint 2019-09-27 15:21:03 +01:00
Brendan Abolivier 72a2708ac6
Fixup and add some logging 2019-09-27 15:13:39 +01:00
Erik Johnston 132279a46f Patch inlinecallbacks for log contexts 2019-09-27 15:11:14 +01:00
Brendan Abolivier 5257a2fb1c
Reject pending invites on deactivation 2019-09-27 14:49:53 +01:00
Andrew Morgan 8c27bc8b60
Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978)
Just to have all the methods that make calls to identity services in one place.
2019-09-27 10:36:20 +01:00
Erik Johnston 67ee18daea
Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span
Add some helpful opentracing tags and remove get_user_by_req span
2019-09-27 10:02:57 +01:00
Erik Johnston 3423633d50 Fix 'redaction_retention_period' sampel config to match guidelines 2019-09-26 16:43:52 +01:00
Richard van der Hoff 54569c787b
Kill off half-implemented password-reset via sms (#6101)
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.

See also matrix-org/matrix-doc#2303.
2019-09-26 15:38:25 +01:00
Neil Johnson 8b8f8c7b3c Explicitly log when a homeserver does not have a trusted key server configured (#6090) 2019-09-26 12:57:01 +01:00
Richard van der Hoff 2927c6bc4c bump version 2019-09-26 12:29:59 +01:00
Neil Johnson 034db2ba21 Fix dummy event insertion consent bug (#6053)
Fixes #5905
2019-09-26 11:47:53 +01:00
Richard van der Hoff a96318127d Update comments and docstring 2019-09-25 18:17:39 +01:00
Erik Johnston 4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston 9614d3c9d1
Merge pull request #6089 from matrix-org/erikj/cleanup_user_ips
Move last seen info into devices table
2019-09-25 17:42:39 +01:00
Neil Johnson a4f3ca48b5 Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. (#5884) 2019-09-25 17:27:35 +01:00
Erik Johnston 39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Erik Johnston d2bd0bc6b1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips 2019-09-25 17:16:28 +01:00
Erik Johnston 50572db837
Use if `is not None`
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
Neil Johnson 77dc7093a7 Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. (#6104)
Fixes #6103
2019-09-25 12:29:35 +01:00
Richard van der Hoff 990928abde
Stop advertising unsupported flows for registration (#6107)
If email or msisdn verification aren't supported, let's stop advertising them
for registration.

Fixes #6100.
2019-09-25 12:10:26 +01:00
Erik Johnston dc2c97e1a3 isort 2019-09-25 11:59:05 +01:00
Erik Johnston dc01cad690 Add device and appservice tags 2019-09-25 11:59:00 +01:00
Erik Johnston 5d99713854 Add tags for event_id and txn_id in event sending
This will make it easier to search for sending event requests.
2019-09-25 11:39:15 +01:00
Richard van der Hoff 2cd98812ba
Refactor the user-interactive auth handling (#6105)
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.

This is also preparation for some more intelligent advertising of flows, as per #6100
2019-09-25 11:33:03 +01:00
Erik Johnston fde4ce2213 Don't create new span for get_user_by_req
We don't actually care about what happens in `get_user_by_req` and
having it as a separate span means that the entity tag isn't added to
the servlet spans, making it harder to search.
2019-09-25 11:32:41 +01:00
Richard van der Hoff 8004d6ca2f
Refactor code for calculating registration flows (#6106)
because, frankly, it looked like it was written by an axe-murderer.

This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
2019-09-25 11:32:05 +01:00
Brendan Abolivier f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
Richard van der Hoff 9b7c4f4078 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 17:03:50 +01:00
Richard van der Hoff 566ac40939
remove unused parameter to get_user_id_by_threepid (#6099)
Added in #5377, apparently in error
2019-09-24 17:01:09 +01:00
Erik Johnston 242017db8b Prune rows in user_ips older than configured period
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
Erik Johnston 2135c198d1 Add has_completed_background_update
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Erik Johnston 367158a609 Add wrap_as_background_process decorator.
This does the same thing as `run_as_background_process` but means we
don't need to create superfluous functions.
2019-09-24 15:53:17 +01:00
Richard van der Hoff bb82be9851
Merge pull request #6069 from matrix-org/rav/fix_attribute_mapping
Fix a bug with saml attribute maps.
2019-09-24 15:07:26 +01:00
Brendan Abolivier 12fe2a29bc
Incorporate review 2019-09-24 14:43:38 +01:00
J. Ryan Stinnett 40fb00f5b7
Add sid to next_link for email validation (#6097) 2019-09-24 14:39:50 +01:00
Brendan Abolivier f02f14e09a
Fix logging 2019-09-24 14:39:07 +01:00
Richard van der Hoff ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Richard van der Hoff a25b66d3f9 docstrings and comments 2019-09-24 11:15:08 +01:00
Andrew Morgan 50776261e1 Add submit_url response parameter to msisdn /requestToken (#6079)
Second part of solving #6076
Fixes #6076

We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
2019-09-23 21:21:03 +01:00
Andrew Morgan 2b071a2ff1 Add an unstable feature flag for separate add/bind 3pid APIs (#6044)
Add a m.separate_add_and_bind flag set to True. See MSC2290's Backward Compatibility section for details.
2019-09-23 20:46:34 +01:00
Richard van der Hoff 78e8ec368e
Merge pull request #6064 from matrix-org/rav/saml_config_cleanup
Make the sample saml config closer to our standards
2019-09-23 20:36:51 +01:00
Andrew Morgan e08ea43463 Use the federation blacklist for requests to untrusted Identity Servers (#6000)
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.

Fixes #5935
2019-09-23 20:23:20 +01:00
Andrew Morgan 1ea3ed7620 Add m.id_access_token to /versions unstable_features (MSC2264) (#5974)
Adds a flag to /versions' unstable_features section indicating that this Synapse understands what an id_access_token is, as per MSC2264.

Fixes #5927
2019-09-23 20:19:25 +01:00
Andrew Morgan b38aa82b83 Add m.require_identity_server to /versions unstable_flags (#5972)
As MSC2263 states, m.require_identity_server must be set to false when it does not require an identity server to be provided by the client for the purposes of email registration or password reset.

Adds an m.require_identity_server flag to /versionss unstable_flags section. This will advertise that Synapse no longer needs id_server as a parameter.
2019-09-23 18:52:43 +01:00
Andrew Morgan 2c99c63453 Add POST submit_token endpoint for MSISDN (#6078)
First part of solving #6076
2019-09-23 17:49:00 +01:00
Brendan Abolivier 323d685bf7
Typo 2019-09-23 17:23:49 +01:00
Brendan Abolivier 2858d10671
Fix the return value in the users_set_deactivated_flag background job 2019-09-23 17:22:01 +01:00
Erik Johnston 51d28272e2 Query devices table for last seen info.
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.

The storage functions now no longer return the access_token, since it
was unused.
2019-09-23 16:59:45 +01:00
Erik Johnston ed80231ade Add BG update to populate devices last seen info 2019-09-23 16:59:45 +01:00
Erik Johnston 2ade05dca3 Add last seen info to devices table.
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
Andrew Morgan 30af161af2 Implement MSC2290 (#6043)
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.

This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints.

Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
Andrew Morgan 1b519e0272 Disable /register/available if registration is disabled (#6082)
Fixes #6066

This register endpoint should be disabled if registration is disabled, otherwise we're giving anyone the ability to check if a username exists on a server when we don't need to be.

Error code is 403 (Forbidden) as that's the same returned by /register when registration is disabled.
2019-09-23 15:38:38 +01:00
Andrew Morgan 1c9feadf4b Generalize email sending logging (#6075)
In ancient times Synapse would only send emails when it was notifying a user about a message they received...

Now it can do all sorts of neat things!

Change the logging so it's not just about notifications.
2019-09-23 14:38:19 +01:00