Commit Graph

3120 Commits (f03c877b32c33275f47ead9e1e01d700a8d049c0)

Author SHA1 Message Date
Richard van der Hoff 366dc7d2d4 Merge commit '33757bad1' into rav/saml_mapping_work 2019-09-20 11:21:40 +01:00
Richard van der Hoff 33757bad19 More better logging 2019-09-20 11:20:02 +01:00
Richard van der Hoff b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff 7423fade92 better logging 2019-09-19 17:16:50 +01:00
Andrew Morgan 6670bd4072
v2 3PID Invites (part of MSC2140) (#5979)
3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is.

These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens).

Requires: ~~#5976~~
2019-09-17 18:05:13 +01:00
Erik Johnston 70c52821ce Fix race condition in room stats. (#6029)
Broke in #5971

Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)
2019-09-17 12:41:23 +01:00
Richard van der Hoff a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Andrew Morgan 9fc71dc5ee
Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)
This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted.

Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140).

Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
2019-09-11 16:02:42 +01:00
Erik Johnston cbcbfe64a2
Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction
Allow use of different ratelimits for admin redactions.
2019-09-11 15:39:38 +01:00
Andrew Morgan 9c555f37e3
Add note about extra arg to send_membership_event, remove arg in remote_reject_invite (#6009)
Some small fixes to `room_member.py` found while doing other PRs.

1. Add requester to the base `_remote_reject_invite` method.
2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.
2019-09-11 14:23:24 +01:00
Erik Johnston 57dd41a45b
Fix comments
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-11 13:54:50 +01:00
Andrew Morgan 3505ffcda7
Fix existing v2 identity server calls (MSC2140) (#6013)
Two things I missed while implementing [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R80).

1. Access tokens should be provided to the identity server as `access_token`, not `id_access_token`, even though the homeserver may accept the tokens as `id_access_token`.
2. Access tokens must be sent to the identity server in a query parameter, the JSON body is not allowed.

We now send the access token as part of an `Authorization: ...` header, which fixes both things.

The breaking code was added in https://github.com/matrix-org/synapse/pull/5892

Sytest PR: https://github.com/matrix-org/sytest/pull/697
2019-09-11 11:59:45 +01:00
Erik Johnston c64c3bb4c5 Fix how we check for self redaction 2019-09-11 11:18:04 +01:00
Erik Johnston 54ce81c86d Allow use of different ratelimits for admin redactions.
This is useful to allow room admins to quickly deal with a large number
of abusive messages.
2019-09-11 10:46:38 +01:00
Andrew Morgan cd17a2085e
Remove origin parameter from add_display_name_to_third_party_invite and add params to docstring (#6010)
Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument.

Since it's not used anywhere in the handler function though, we should remove it from the function arguments.
2019-09-11 10:37:17 +01:00
Erik Johnston 5e9b05d7da
Merge pull request #6011 from matrix-org/anoa/fix_3pid_validation
Use account_threepid_delegate for 3pid validation
2019-09-10 18:15:07 +01:00
Andrew Morgan b5833a2abf Add changelog 2019-09-10 17:56:10 +01:00
Andrew Morgan 60d3c57bd0 Use account_threepid_delegate for 3pid validation 2019-09-10 17:56:10 +01:00
Jason Robinson be618e0551 Only count real users when checking for auto-creation of auto-join room
Previously if the first registered user was a "support" or "bot" user,
when the first real user registers, the auto-join rooms were not
created.

Fix to exclude non-real (ie users with a special user type) users
when counting how many users there are to determine whether we should
auto-create a room.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:48:08 +03:00
Hubert Chathi d3f2fbcfe5 add function docs 2019-09-07 14:13:18 -04:00
Hubert Chathi 8e86f5b65c Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-09-07 13:20:34 -04:00
Hubert Chathi 0d61d1d735 Merge branch 'develop' into cross-signing_sig_upload 2019-09-07 13:14:45 -04:00
Hubert Chathi 415d0a00e0 run black 2019-09-06 16:46:45 -04:00
Hubert Chathi 561cbba057 split out signature processing into separate functions 2019-09-06 16:44:24 -04:00
Andrew Morgan 78801e7f9e
Ensure a sid parameter is passed to bind_threepid (#5995)
`sid` is required to be part of `three_pid_creds`. We were 500'ing if it wasn't provided instead of returning `M_MISSING_PARAM`.
2019-09-06 15:36:50 +01:00
Erik Johnston 142c9325c2 Correctly handle non-bool m.federate flag 2019-09-06 14:21:06 +01:00
Andrew Morgan 0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Hubert Chathi 369462da74 avoid modifying input parameter 2019-09-05 17:03:31 -04:00
Erik Johnston 1a6ae33309
Merge pull request #5984 from matrix-org/joriks/opentracing_link_send_to_edu_contexts
Link the send loop with the edus contexts
2019-09-05 15:22:24 +01:00
Jorik Schellekens 1d65292e94 Link the send loop with the edus contexts
The contexts were being filtered too early so  the send loop wasn't
being linked to them unless the destination
was whitelisted.
2019-09-05 14:42:37 +01:00
Andrew Morgan a0d294c306
Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (#5892) 2019-09-05 14:31:22 +01:00
Andrew Morgan 90d17a3d28
Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (#5980)
Implements `POST /_matrix/client/r0/account/3pid/unbind` from [MSC2140](https://github.com/matrix-org/matrix-doc/blob/dbkr/tos_2/proposals/2140-terms-of-service-2.md#post-_matrixclientr0account3pidunbind).
2019-09-05 14:00:30 +01:00
Hubert Chathi c8dc740a94 update with newer coding style 2019-09-04 22:30:45 -04:00
Hubert Chathi 7d6c70fc7a make black happy 2019-09-04 22:04:12 -04:00
Hubert Chathi ac4746ac4b allow uploading signatures of master key signed by devices 2019-09-04 22:03:41 -04:00
Hubert Chathi 4bb4544784 implement device signature uploading/fetching 2019-09-04 20:02:56 -04:00
Hubert Chathi faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Andrew Morgan b736c6cd3a
Remove bind_email and bind_msisdn (#5964)
Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107).
2019-09-04 18:24:23 +01:00
Erik Johnston 6e834e94fc
Fix and refactor room and user stats (#5971)
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
Andrew Morgan ea128a3e8e code cleanups 2019-09-03 21:05:06 +01:00
Andrew Morgan 6b6086b8bf Fix docstring 2019-09-03 20:00:09 +01:00
Andrew Morgan a98b8583c6 Remove unnecessary variable declaration 2019-09-03 19:58:51 +01:00
Matthew Hodgson 8401bcd206 fix typo 2019-09-03 12:44:14 +01:00
Andrew Morgan 2a44782666
Remove double return statements (#5962)
Remove all the "double return" statements which were a result of us removing all the instances of

```
defer.returnValue(...)
return
```

statements when we switched to python3 fully.
2019-09-03 11:42:45 +01:00
Jorik Schellekens a90d16dabc
Opentrace device lists (#5853)
Trace device list changes.
2019-09-03 10:21:30 +01:00
Andrew Morgan 36f34e6f3d
Remove unused methods from c/s api v1 in register.py (#5963)
These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
2019-09-02 18:29:21 +01:00
Andrew Morgan 4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Andrew Morgan 3057095a5d Revert "Use the v2 lookup API for 3PID invites (#5897)" (#5937)
This reverts commit 71fc04069a.

This broke 3PID invites as #5892 was required for it to work correctly.
2019-08-30 12:00:20 +01:00
Hubert Chathi e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
Hubert Chathi 96bda56370 black 2019-08-28 17:18:40 -07:00
Hubert Chathi 3b0b22cb05 use stream ID generator instead of timestamp 2019-08-28 17:17:21 -07:00
Will Hunt c8fa620d7a
Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent
Exempt support users from consent
2019-08-28 16:31:40 +01:00
Andrew Morgan 71fc04069a
Use the v2 lookup API for 3PID invites (#5897)
Fixes https://github.com/matrix-org/synapse/issues/5861

Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134). Currently this is only used for 3PID invites.

Sytest PR: https://github.com/matrix-org/sytest/pull/679
2019-08-28 14:59:26 +02:00
Amber Brown 7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
reivilibre a3f0635686
Merge pull request #5914 from matrix-org/rei/admin_getadmin
Add GET method to admin API /users/@user:dom/admin
2019-08-28 09:44:22 +01:00
reivilibre 7ccc251415
Merge pull request #5859 from matrix-org/rei/msc2197
MSC2197 Search Filters over Federation
2019-08-28 09:00:21 +01:00
Olivier Wilkinson (reivilibre) c88a119259 Add GET method to admin API /users/@user:dom/admin
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 13:12:27 +01:00
Richard van der Hoff ccb15a5bbe
Merge pull request #5906 from matrix-org/neilj/increase_display_name_limit
Increase profile display name limit
2019-08-27 11:52:59 +01:00
reivilibre 1a7e6eb633
Add Admin API capability to set adminship of a user (#5878)
Admin API: Set adminship of a user
2019-08-27 10:14:00 +01:00
Neil Johnson 27d3fc421a Increase max display name limit 2019-08-24 22:33:43 +01:00
Will Hunt c998f25006
Apply suggestions from code review
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-08-23 10:28:54 +01:00
Half-Shot 9ba32f6573 Exempt bot users 2019-08-23 09:56:31 +01:00
Half-Shot 886eceba3e Return user_type in get_user_by_id 2019-08-23 09:14:52 +01:00
Jorik Schellekens 8767b63a82
Propagate opentracing contexts through EDUs (#5852)
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Brendan Abolivier 1c5b8c6222 Revert "Add "require_consent" parameter for registration"
This reverts commit 3320aaab3a.
2019-08-22 14:47:34 +01:00
Half-Shot 3320aaab3a Add "require_consent" parameter for registration 2019-08-22 14:21:54 +01:00
Jorik Schellekens 9a6f2be572
Opentrace e2e keys (#5855)
Add opentracing tags and logs for e2e keys
2019-08-22 11:28:12 +01:00
Richard van der Hoff 119aa31b10
Servlet to purge old rooms (#5845) 2019-08-22 10:42:59 +01:00
Hubert Chathi 814f253f1b make isort happy 2019-08-21 13:22:15 -07:00
Hubert Chathi 7c3abc6572 apply PR review suggestions 2019-08-21 13:19:35 -07:00
Erik Johnston d514dac0b2
Merge pull request #5860 from matrix-org/erikj/update_5704_comments
Remove logging for #5407 and update comments
2019-08-19 10:20:59 +01:00
Erik Johnston 748aa38378 Remove logging for #5407 and update comments 2019-08-15 12:02:18 +01:00
Olivier Wilkinson (reivilibre) 6fadb560fc Support MSC2197 outbound with unstable prefix
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-15 10:59:37 +01:00
Michael Telatynski baee288fb4 Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633) 2019-08-15 09:45:57 +01:00
Erik Johnston 6881f21f3e Handle TimelineBatch being limited and empty.
This hopefully addresses #5407 by gracefully handling an empty but
limited TimelineBatch. We also add some logging to figure out how this
is happening.
2019-08-06 12:59:00 +01:00
Hubert Chathi fac1cdc562 make changes from PR review 2019-08-01 21:51:19 -04:00
Hubert Chathi 336c546d6a Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-08-01 16:31:40 -04:00
Brendan Abolivier 8ed9e63432
Account validity: allow defining HTML templates to serve the us… (#5807)
Account validity: allow defining HTML templates to serve the user on account renewal attempt
2019-08-01 16:09:25 +02:00
Erik Johnston 58af30a6c7
Merge pull request #5802 from matrix-org/erikj/deny_redacting_different_room
Deny redaction of events in a different room.
2019-08-01 13:14:46 +01:00
Erik Johnston 0f632f3a57
Merge pull request #5790 from matrix-org/erikj/groups_request_errors
Handle RequestSendFailed exception correctly in more places.
2019-08-01 13:14:08 +01:00
Brendan Abolivier a4a9ded4d0
Allow defining HTML templates to serve the user on account renewal 2019-08-01 11:59:27 +02:00
Erik Johnston cf89266b98 Deny redaction of events in a different room.
We already correctly filter out such redactions, but we should also deny
them over the CS API.
2019-07-31 16:12:27 +01:00
Andrew Morgan 58a755cdc3 Remove duplicate return statement 2019-07-31 13:24:51 +01:00
Erik Johnston b4d5ff0af7 Don't log as exception when failing durig backfill 2019-07-30 13:19:22 +01:00
Erik Johnston 1ec7d656dd Unwrap error 2019-07-30 13:09:02 +01:00
Erik Johnston 458e51df7a Fix error handling when fetching remote device keys 2019-07-30 13:07:02 +01:00
Richard van der Hoff 8c97f6414c
Remove non-functional 'expire_access_token' setting (#5782)
The `expire_access_token` didn't do what it sounded like it should do. What it
actually did was make Synapse enforce the 'time' caveat on macaroons used as
access tokens, but since our access token macaroons never contained such a
caveat, it was always a no-op.

(The code to add 'time' caveats was removed back in v0.18.5, in #1656)
2019-07-30 08:25:02 +01:00
Amber Brown 865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Erik Johnston aecae8f397 Correctly handle errors doing requests to group servers 2019-07-29 17:21:57 +01:00
Amber Brown 97a8b4caf7
Move some timeout checking logs to DEBUG #5785 2019-07-30 02:02:18 +10:00
Jorik Schellekens 85b0bd8fe0 Update the device list cache when keys/query is called (#5693) 2019-07-29 16:34:44 +01:00
Richard van der Hoff 1a93daf353
Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch
Log when we receive a /make_* request from a different origin
2019-07-26 12:38:37 +01:00
Richard van der Hoff d1020653fc Log when we receive a /make_* request from a different origin 2019-07-26 10:08:22 +01:00
Hubert Chathi c659b9f94f allow uploading keys for cross-signing 2019-07-25 11:08:24 -04:00
Erik Johnston 2276936bac
Merge pull request #5743 from matrix-org/erikj/log_origin_receipts_mismatch
Log when we receive receipt from a different origin
2019-07-24 13:27:57 +01:00
Richard van der Hoff f30a71a67b
Stop trying to fetch events with event_id=None. (#5753)
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.

I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
2019-07-24 13:16:18 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Erik Johnston 3db1377b26 Log when we receive receipt from a different origin 2019-07-23 13:31:03 +01:00
Richard van der Hoff 9c70a02a9c
Ignore redactions of m.room.create events (#5701) 2019-07-17 19:08:02 +01:00
Andrew Morgan 18c516698e
Return a different error from Invalid Password when a user is deactivated (#5674)
Return `This account has been deactivated` instead of `Invalid password` when a user is deactivated.
2019-07-15 11:45:29 +01:00
Erik Johnston d86321300a
Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data
Add basic function to get all data for a user out of synapse
2019-07-15 10:04:02 +01:00
Richard van der Hoff 5f158ec039
Implement access token expiry (#5660)
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Richard van der Hoff 1890cfcf82 Inline issue_access_token (#5659)
this is only used in one place, so it's clearer if we inline it and reduce the
API surface.

Also, fixes a buglet where we would create an access token even if we were
about to block the user (we would never return the AT, so the user could never
use it, but it was still created and added to the db.)
2019-07-11 04:10:07 +10:00
Brendan Abolivier 8ab3444fdf
Merge pull request #5658 from matrix-org/babolivier/is-json
Send 3PID bind requests as JSON data
2019-07-10 17:01:26 +01:00
Richard van der Hoff 953dbb7980
Remove access-token support from RegistrationStore.register (#5642)
The 'token' param is no longer used anywhere except the tests, so let's kill
that off too.
2019-07-10 16:26:49 +01:00
Brendan Abolivier b2a2e96ea6
Typo 2019-07-10 15:56:21 +01:00
Brendan Abolivier f77e997619
Send 3PID bind requests as JSON data 2019-07-10 15:46:42 +01:00
Brendan Abolivier 65434da75d
Merge pull request #5638 from matrix-org/babolivier/invite-json
Use JSON when querying the IS's /store-invite endpoint
2019-07-09 18:48:38 +01:00
Hubert Chathi 7b3bc755a3 remove unused and unnecessary check for FederationDeniedError (#5645)
FederationDeniedError is a subclass of SynapseError, which is a subclass of
CodeMessageException, so if e is a FederationDeniedError, then this check for
FederationDeniedError will never be reached since it will be caught by the
check for CodeMessageException above.  The check for CodeMessageException does
almost the same thing as this check (since FederationDeniedError initialises
with code=403 and msg="Federation denied with %s."), so may as well just keep
allowing it to handle this case.
2019-07-09 18:37:39 +01:00
Brendan Abolivier af67c7c1de
Merge pull request #5644 from matrix-org/babolivier/profile-allow-self
Allow newly-registered users to lookup their own profiles
2019-07-09 10:25:40 +01:00
Richard van der Hoff 824707383b
Remove access-token support from RegistrationHandler.register (#5641)
Nothing uses this now, so we can remove the dead code, and clean up the
API.

Since we're changing the shape of the return value anyway, we take the
opportunity to give the method a better name.
2019-07-08 19:01:08 +01:00
Brendan Abolivier 7556851665
Allow newly-registered users to lookup their own profiles
When a user creates an account and the 'require_auth_for_profile_requests' config flag is set, and a client that performed the registration wants to lookup the newly-created profile, the request will be denied because the user doesn't share a room with themselves yet.
2019-07-08 17:31:00 +01:00
Richard van der Hoff b70e080b59
Better logging for auto-join. (#5643)
It was pretty unclear what was going on, so I've added a couple of log lines.
2019-07-08 17:14:51 +01:00
Brendan Abolivier 57eacee4f4 Merge branch 'develop' into babolivier/invite-json 2019-07-08 15:49:23 +01:00
Richard van der Hoff 1af2fcd492 Move get_or_create_user to test code (#5628)
This is only used in tests, so...
2019-07-08 23:52:26 +10:00
Brendan Abolivier f05c7d62bc
Lint 2019-07-08 14:29:27 +01:00
Brendan Abolivier 1a807dfe68
Use application/json when querying the IS's /store-invite endpoint 2019-07-08 14:19:39 +01:00
Richard van der Hoff 80cc82a445
Remove support for invite_3pid_guest. (#5625)
This has never been documented, and I'm not sure it's ever been used outside
sytest.

It's quite a lot of poorly-maintained code, so I'd like to get rid of it.

For now I haven't removed the database table; I suggest we leave that for a
future clearout.
2019-07-05 16:47:58 +01:00
Erik Johnston b4f5416dd9 pep8 2019-07-05 14:41:29 +01:00
Erik Johnston eadb13d2e9 Remove FileExfiltrationWriter 2019-07-05 14:15:00 +01:00
Erik Johnston 7f0d8e4288 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data 2019-07-05 14:08:21 +01:00
Erik Johnston 9ccea16d45 Assume key existence. Update docstrings 2019-07-05 14:07:56 +01:00
Erik Johnston c061d4f237 Fixup from review comments. 2019-07-04 11:41:06 +01:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Erik Johnston d0b849c86d
Apply comment fixups from code review
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-07-03 15:03:38 +01:00
Richard van der Hoff 91753cae59
Fix a number of "Starting txn from sentinel context" warnings (#5605)
Fixes #5602, #5603
2019-07-03 09:31:27 +01:00
Erik Johnston 8ee69f299c Add basic function to get all data for a user out of synapse 2019-07-02 12:09:04 +01:00
Richard van der Hoff 6eecb6e500
Complete the SAML2 implementation (#5422)
* SAML2 Improvements and redirect stuff

Signed-off-by: Alexander Trost <galexrt@googlemail.com>

* Code cleanups and simplifications.

Also: share the saml client between redirect and response handlers.

* changelog

* Revert redundant changes to static js

* Move all the saml stuff out to a centralised handler

* Add support for tracking SAML2 sessions.

This allows us to correctly handle `allow_unsolicited: False`.

* update sample config

* cleanups

* update sample config

* rename BaseSSORedirectServlet for consistency

* Address review comments
2019-07-02 11:18:11 +01:00
Richard van der Hoff b4fd86a9b4 Merge branch 'develop' into rav/saml2_client 2019-07-01 14:21:03 +01:00
Richard van der Hoff 3bcb13edd0 Address review comments 2019-07-01 12:13:22 +01:00
Erik Johnston 04196a4dae
Merge pull request #5507 from matrix-org/erikj/presence_sync_tighloop
Fix sync tightloop bug.
2019-07-01 11:43:10 +01:00
Erik Johnston 915280f1ed Fixup comment 2019-07-01 10:22:42 +01:00
Brendan Abolivier c7ff297dde
Merge pull request #5576 from matrix-org/babolivier/3pid-invite-ratelimit
Don't update the ratelimiter before sending a 3PID invite
2019-06-28 17:43:48 +01:00
Brendan Abolivier 15d9fc31bd
Only ratelimit when sending the email
If we do the opposite, an event can arrive after or while sending the email and the 3PID invite event will get ratelimited.
2019-06-28 16:04:05 +01:00
Brendan Abolivier 01d0f8e701
Don't update the ratelimiter before sending a 3PID invite
This would cause emails being sent, but Synapse responding with a 429 when creating the event. The client would then retry, and with bad timing the same scenario would happen again. Some testing I did ended up sending me 10 emails for one single invite because of this.
2019-06-28 15:22:16 +01:00
Daniel Hoffend 9646a593ac Added possibilty to disable local password authentication (#5092)
Signed-off-by: Daniel Hoffend <dh@dotlan.net>
2019-06-27 18:37:29 +01:00
Erik Johnston 729f5a4fb6 Review comments 2019-06-27 16:06:23 +01:00
Richard van der Hoff 28db0ae537 cleanups 2019-06-27 00:37:41 +01:00
Richard van der Hoff 36f4953dec Add support for tracking SAML2 sessions.
This allows us to correctly handle `allow_unsolicited: False`.
2019-06-26 23:50:55 +01:00
Richard van der Hoff 3705322103 Move all the saml stuff out to a centralised handler 2019-06-26 22:52:02 +01:00
Erik Johnston 8624db3194 Refactor and comment sync device list code 2019-06-26 19:39:49 +01:00
Erik Johnston 806a06daf2 Rename get_users_whose_devices_changed 2019-06-26 19:39:19 +01:00
Erik Johnston a2f6d31a63 Refactor get_user_ids_changed to pull less from DB
When a client asks for users whose devices have changed since a token we
used to pull *all* users from the database since the token, which could
easily be thousands of rows for old tokens.

This PR changes this to only check for changes for users the client is
actually interested in.

Fixes #5553
2019-06-26 12:03:44 +01:00
Andrew Morgan ef8c62758c
Prevent multiple upgrades on the same room at once (#5051)
Closes #4583

Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time.

Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.
2019-06-25 14:19:21 +01:00
Erik Johnston 25433f212d
Merge pull request #5531 from matrix-org/erikj/workers_pagination_token
Fix /messages on workers when no from param specified.
2019-06-24 15:30:10 +01:00
Richard van der Hoff edea4bb5be Allow configuration of the path used for ACME account keys.
Because sticking it in the same place as the config isn't necessarily the right
thing to do.
2019-06-24 13:51:22 +01:00
Richard van der Hoff 21bf4318b5
Factor acme bits out to a separate file (#5521)
This makes some of the conditional-import hoop-jumping easier.
2019-06-24 11:33:56 +01:00
Erik Johnston dddf20e8e1 Fix /messages on workers when no from param specified.
If no `from` param is specified we calculate and use the "current
token" that inlcuded typing, presence, etc. These are unused during
pagination and are not available on workers, so we simply don't
calculate them.
2019-06-24 10:06:51 +01:00
Erik Johnston 8181e290a9 Fix sync tightloop bug.
If, for some reason, presence updates take a while to persist then it
can trigger clients to tightloop calling `/sync` due to the presence
handler returning updates but not advancing the stream token.

Fixes #5503.
2019-06-21 11:10:27 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston 7dcf984075
Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_error
Handle the case of `get_missing_events` failing
2019-06-19 13:20:09 +01:00
Erik Johnston e0be8d7016
Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events
Add experimental option to reduce extremities.
2019-06-19 13:19:18 +01:00
Erik Johnston 554609288b Run as background process and fix comments 2019-06-19 11:33:03 +01:00
David Baker f2d2ae03da Add some logging to 3pid invite sig verification (#5015)
I had to add quite a lot of logging to diagnose a problem with 3pid
invites - we only logged the one failure which isn't all that
informative.

NB. I'm not convinced the logic of this loop is right: I think it
should just accept a single valid signature from a trusted source
rather than fail if *any* signature is invalid. Also it should
probably not skip the rest of middle loop if a check fails? However,
I'm deliberately not changing the logic here.
2019-06-18 22:51:24 +01:00
Erik Johnston 2b20d0fb59 Fix logline 2019-06-18 16:12:53 +01:00
Erik Johnston 19b80fe68a Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_get_missing_events_error 2019-06-18 16:11:43 +01:00
Erik Johnston b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Brendan Abolivier 112cf5a73a Add third party rules hook for 3PID invites 2019-06-17 17:39:09 +01:00
Erik Johnston 187d2837a9 Add third party rules hook into create room 2019-06-17 16:41:19 +01:00
Erik Johnston 6840ebeef8
Merge pull request #5385 from matrix-org/erikj/reduce_http_exceptions
Handle HttpResponseException when using federation client.
2019-06-17 13:54:47 +01:00
Erik Johnston a9dab970b8
Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_state
Fix 3PID invite room state over federation.
2019-06-17 10:18:28 +01:00
Brendan Abolivier f12e1f029c
Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules
Allow server admins to define implementations of extra rules for allowing or denying incoming events
2019-06-14 19:37:59 +01:00
Brendan Abolivier f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Erik Johnston 304a1376c2 Fix 3PID invite room state over federation.
Fixes that when a user exchanges a 3PID invite for a proper invite over
federation it does not include the `invite_room_state` key.

This was due to synapse incorrectly sending out two invite requests.
2019-06-14 16:19:11 +01:00
Brendan Abolivier 6d56a694f4 Don't send renewal emails to deactivated users 2019-06-14 15:05:56 +01:00
Brendan Abolivier d0530382ee
Track deactivated accounts in the database (#5378) 2019-06-14 13:18:24 +01:00
Erik Johnston 49e01e5710 Fix defaults on checking threepids 2019-06-10 23:09:31 +01:00
Erik Johnston a46ef1e3a4 Handle HttpResponseException when using federation client.
Otherwise we just log exceptions everywhere.
2019-06-07 10:29:35 +01:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Richard van der Hoff 71063a69b8
Fix missing logcontext for PresenceHandler.on_shutdown. (#5369)
Fixes some warnings, and a scary-looking stacktrace when sytest kills the
process.
2019-06-06 14:45:17 +01:00
Brendan Abolivier 8f06344e11
Merge pull request #5089 from dnaf/m-heroes-empty-room-name
Make /sync return heroes if room name or canonical alias are empty
2019-06-06 11:18:13 +01:00
Brendan Abolivier 64fa928792
Simplify condition 2019-06-06 10:34:12 +01:00
Brendan Abolivier b9c43c8463
Merge pull request #5355 from matrix-org/babolivier/heroes_left_members
Include left members in room summaries' heroes
2019-06-06 10:05:27 +01:00
Erik Johnston 75538813fc Fix background updates to handle redactions/rejections (#5352)
* Fix background updates to handle redactions/rejections

In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
Brendan Abolivier 0a2f522644
Simplify condition 2019-06-05 14:02:29 +01:00
Brendan Abolivier 40596aec0e Merge branch 'develop' into m-heroes-empty-room-name 2019-06-05 13:38:01 +01:00
Brendan Abolivier dbbaf25dd3
Do user_id != me checks before deciding whether we should pick heroes from the joined members or the parted ones 2019-06-05 11:50:27 +01:00
Erik Johnston bc3d6b918b Add logging when request fails and clarify we ignore errors. 2019-06-05 11:37:51 +01:00
Erik Johnston 220a733d73 Fix handling of failures when calling /event_auth.
When processing an incoming event over federation, we may try and
resolve any unexpected differences in auth events. This is a
non-essential process and so should not stop the processing of the event
if it fails (e.g. due to the remote disappearing or not implementing the
necessary endpoints).

Fixes #3330
2019-06-03 09:56:45 +01:00
Richard van der Hoff d828d1dc57
Merge pull request #5309 from matrix-org/rav/limit_displayname_length
Limit displaynames and avatar URLs
2019-06-01 11:34:50 +01:00
Richard van der Hoff d16c6375fe Limit displaynames and avatar URLs
These end up in join events everywhere, so let's limit them.

Fixes #5079
2019-06-01 10:44:36 +01:00
Richard van der Hoff 5726378ece
Fix "db txn 'update_presence' from sentinel context" log messages (#5275)
Fixes #4414.
2019-05-28 21:20:11 +01:00
Erik Johnston d16f5574b6
Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events
Don't bundle aggregations with events in /sync or /events or state queries
2019-05-24 10:36:31 +01:00
Erik Johnston 4cb577c23f Don't bundle aggs for /state and /members etc APIs 2019-05-24 09:52:33 +01:00
Andrew Morgan 6368150a74
Add config option for setting homeserver's default room version (#5223)
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
2019-05-23 15:00:20 +01:00
Richard van der Hoff 85d1e03b9d
Simplifications and comments in do_auth (#5227)
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
Amber Brown 4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Erik Johnston 9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
Erik Johnston 44b8ba484e Fix words 2019-05-21 16:51:45 +01:00
Richard van der Hoff c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146)
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
2019-05-21 16:10:54 +01:00
Erik Johnston de7672b78f Don't bundle events in /sync or /events
As we'll send down the annotations too anyway, so this just ends up
confusing clients.
2019-05-21 13:54:09 +01:00
Erik Johnston c7ec06e8a6 Block attempts to annotate the same event twice 2019-05-20 17:39:05 +01:00
ReidAnderson 3787133c9e Limit UserIds to a length that fits in a state key (#5198) 2019-05-20 11:20:08 +01:00
bytepoets-blo 291e1eea5e fix mapping of return values for get_or_register_3pid_guest (#5177)
* fix mapping of return values for get_or_register_3pid_guest
2019-05-17 17:27:14 +01:00
Brendan Abolivier cd32375846 Add option to disable per-room profiles 2019-05-16 14:34:28 +01:00
Matthew Hodgson 4a6d5de98c Make /sync attempt to return device updates for both joined and invited users (#3484) 2019-05-16 13:23:43 +01:00
Amber Brown f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Erik Johnston b54b03f9e1 Allow client event serialization to be async 2019-05-14 11:58:01 +01:00
Matthew Hodgson c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083)
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
2019-05-08 18:26:56 +01:00
Brendan Abolivier 1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128)
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
2019-05-08 17:01:30 +01:00