Commit Graph

3590 Commits (22eeb6bc54d480e170277a340a219fa8c62ba508)

Author SHA1 Message Date
Patrick Cloke 4a739c73b4
Convert simple_update* and simple_select* to async (#8173) 2020-08-27 07:08:38 -04:00
Patrick Cloke 6fe12c9512
Do not propagate typing notifications from shadow-banned users. (#8176) 2020-08-26 12:05:36 -04:00
Andrew Morgan e0d6244beb
Remove unused parameter from, and add safeguard in, get_room_data (#8174)
Small cleanup PR.

* Removed the unused `is_guest` argument
* Added a safeguard to a (currently) impossible code path, fixing static checking at the same time.
2020-08-26 15:07:35 +01:00
Patrick Cloke 2e6c90ff84
Do not propagate profile changes of shadow-banned users into rooms. (#8157) 2020-08-26 08:49:01 -04:00
Patrick Cloke 5099bd68da
Do not allow send_nonmember_event to be called with shadow-banned users. (#8158) 2020-08-25 10:52:15 -04:00
Brendan Abolivier 6e1c64a668 Synapse 1.19.1rc1 (2020-08-25)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in v1.19.0 where appservices with ratelimiting disabled would still be ratelimited when joining rooms. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
 - Fix a bug introduced in v1.19.0 that would cause e.g. profile updates to fail due to incorrect application of rate limits on join requests. ([\#8153](https://github.com/matrix-org/synapse/issues/8153))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAl9FIKkACgkQHgFcFF8Z
 Fs3zIw/9H40ieb73Iay6ecQeOSfHiMVMzvJqYbKgho/a6h5JDHir+NGpwuLFdeGM
 eHR07QkQgUU9+dLNTMOQpCKTIsU70wvzH1vTDINS3ChjnRBdrHKqhAG6ZyEt1dJx
 kxYX54zsQUwiwshMKbJ+DPclHaBFnL+SY5OFfqCNjvaNob59DbHAL3tlSktPc2go
 tGmj81q0dWY6maMCGI3IIYcrW7oLi+4TwosZual5Hz/xgRBiGaKHXRIJnInvkXpl
 R+rSOmpYraapfDPHzPyQgLN4Dt7aAccGho843tt7dAVfd2GRSaUkLGXVXCdoruQG
 CRjY1P3BRBzRBx6o80Uw7Ah0hsoVgpJTSzY008KigJce+IiRWG5sgPjoubhfK0MA
 BqzmCa3/lrR+/WUOf4+w6HSfRncKawgAp7Y7wVj4nQF5fc8mwpFLz4pA/C2YOyjp
 nYXCHf60/KSBDhnr0ZRAhAby4MJoYSf03djFG1oef5SVzOzHD7zho9oBnEz15Tab
 XXkg1iJ7AhNFiQjsY4H1sl2onoF4T7B53NOnUEwD0oll+nXIYGe6hlNuq6x4j6l9
 39ZlMoe9zK28LoKKWa1RDug8z+PmarKRJ2zATlHIb2RGeVX+oFfaKVsIbJtupJVC
 8HSFt7gcgLCdUazk6taKpOHeVyGxK6WUkLnCMHzD2rzPhzpSyws=
 =0rHM
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAl9FJK4ACgkQHgFcFF8Z
 Fs0O2xAAgxqCfAHylVuwunRppwV49Bvq6H0mMM29hYBogGB5cmh1vRnUm5GsPQt0
 9vTKlwMz6XKjLs3TLqYsnZfXqK+lyaN0xFXd7xWCNzFtEXoIvDnq+u3h5WGaKOC9
 PcXb2LSZoHC5yECBpoh58ZQEPKtYILEjo+OSDboIqHz4N5HSjMSPGPvMkUn6xMNG
 roWTAWKPd9juyE2dPzZxIoBWwJGn3D8EMkeTQDlExTTvmnDPyPvJ5MVUY/xaHLgy
 XV8lapFu/SzWAKotc5+9qkVN64obaxwovYTU9JnlqEc5+WlD+Jl+g0258Q1bV1H9
 341aQQJX08iYw3xw13xVgT0zLPRbp82O3/SHC3S1nz27HUWKXqUtsm6woDbgHIz5
 UPvKFQsp2dEN4tFXxkEHiossIVNGuXdRYwEjFQrxOwayCuS4cQwDADhqnzDU4hio
 LSVhtxs9rgLps4iKpcaRAqK8kifTrsomlQfh/7axPJQ43pmBR2PiItetlBW/9Le6
 KTH90ghLQzJwKFkIcFcvPhFMVqSyXI32+g5++YAPmNVy9M/7LdJxuEc9ifTWgwds
 LtV3/F8xlqd0qwl5IbwC6Wf19N06jdlRv/q1zL/Hb6qu3FLQeGd+/1aiC0rsbq15
 grdHVZkZi1iVF/zrOx24ctxQvgLyGHA+M7n/oIaIgxlT1S6+FUI=
 =49ZC
 -----END PGP SIGNATURE-----

Merge tag 'v1.19.1rc1' into develop

Synapse 1.19.1rc1 (2020-08-25)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.19.0 where appservices with ratelimiting disabled would still be ratelimited when joining rooms. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
- Fix a bug introduced in v1.19.0 that would cause e.g. profile updates to fail due to incorrect application of rate limits on join requests. ([\#8153](https://github.com/matrix-org/synapse/issues/8153))
2020-08-25 15:48:11 +01:00
Patrick Cloke 5758dcf30c
Add type hints for state. (#8140) 2020-08-24 14:25:27 -04:00
Patrick Cloke cbd8d83da7
Stop shadow-banned users from sending non-member events. (#8142) 2020-08-24 13:58:56 -04:00
Brendan Abolivier 420484a334
Allow capping a room's retention policy (#8104) 2020-08-24 18:21:04 +01:00
Brendan Abolivier 393a811a41
Fix join ratelimiter breaking profile updates and idempotency (#8153) 2020-08-24 18:06:04 +01:00
Will Hunt 2df82ae451
Do not apply ratelimiting on joins to appservices (#8139)
Add new method ratelimiter.can_requester_do_action and ensure that appservices are exempt from being ratelimited.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-08-24 14:53:53 +01:00
Will Hunt cbbf9126cb
Do not apply ratelimiting on joins to appservices (#8139)
Add new method ratelimiter.can_requester_do_action and ensure that appservices are exempt from being ratelimited.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-08-21 15:07:56 +01:00
Patrick Cloke 3f91638da6
Allow denying or shadow banning registrations via the spam checker (#8034) 2020-08-20 15:42:58 -04:00
Patrick Cloke e259d63f73
Stop shadow-banned users from sending invites. (#8095) 2020-08-20 15:07:42 -04:00
Patrick Cloke 592cdf73be
Improve the error code when trying to register using a name reserved for guests. (#8135) 2020-08-20 10:39:41 -04:00
Patrick Cloke dbc630a628
Use the JSON encoder without whitespace in more places. (#8124) 2020-08-20 10:32:33 -04:00
Patrick Cloke eebf52be06
Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
Patrick Cloke f40645e60b
Convert events worker database to async/await. (#8071) 2020-08-18 16:20:49 -04:00
Patrick Cloke 3c01724b33
Fix the return type of send_nonmember_events. (#8112) 2020-08-18 09:53:13 -04:00
Patrick Cloke 25e55d2598
Return the previous stream token if a non-member event is a duplicate. (#8093) 2020-08-18 07:53:23 -04:00
Andrew Morgan e04e465b4d
Use the default templates when a custom template file cannot be found (#8037)
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
Patrick Cloke ad6190c925
Convert stream database to async/await. (#8074) 2020-08-17 07:24:46 -04:00
Patrick Cloke ac77cdb64e
Add a shadow-banned flag to users. (#8092) 2020-08-14 12:37:59 -04:00
Erik Johnston dc22090a67
Add type hints to synapse.handlers.room (#8090) 2020-08-14 14:47:53 +01:00
Richard van der Hoff 53834bb9c4
Run `remove_push_actions_from_staging` in foreground (#8081)
If we got an error persisting an event, we would try to remove the push actions
asynchronously, which would lead to a 'Re-starting finished log context'
warning.

I don't think there's any need for this to be asynchronous.
2020-08-13 17:05:31 +01:00
Erik Johnston 5dd73d029e
Add type hints to handlers.message and events.builder (#8067) 2020-08-12 15:05:50 +01:00
Erik Johnston fdb46b5442 Merge remote-tracking branch 'origin/develop' into erikj/type_server 2020-08-11 22:03:14 +01:00
Erik Johnston a1e9bb9eae
Add typing info to Notifier (#8058) 2020-08-11 19:40:02 +01:00
Erik Johnston 0f1afbe8dc Change HomeServer definition to work with typing.
Duplicating function signatures between server.py and server.pyi is
silly. This commit changes that by changing all `build_*` methods to
`get_*` methods and changing the `_make_dependency_method` to work work
as a descriptor that caches the produced value.

There are some changes in other files that were made to fix the typing
in server.py.
2020-08-11 18:00:17 +01:00
Richard van der Hoff 0cb169900e
Implement login blocking based on SAML attributes (#8052)
Hopefully this mostly speaks for itself. I also did a bit of cleaning up of the
error handling.

Fixes #8047
2020-08-11 16:08:10 +01:00
Richard van der Hoff fcbab08cbd
Add an assertion on prev_events in create_new_client_event (#8041)
I think this would have caught all the cases in
https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes
more sense here than a 403
2020-08-10 12:29:47 +01:00
Brendan Abolivier 2ffd6783c7
Revert #7736 (#8039) 2020-08-06 17:15:35 +01:00
Patrick Cloke d4a7829b12
Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Patrick Cloke c36228c403
Convert run_as_background_process inner function to async. (#8032) 2020-08-06 08:20:42 -04:00
Patrick Cloke 66f24449dd
Improve performance of the register endpoint (#8009) 2020-08-06 08:09:55 -04:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke e19de43eb5
Convert streams to async. (#8014) 2020-08-04 07:21:47 -04:00
Andrew Morgan 481f76c7aa
Remove signature check on v1 identity server lookups (#8001)
We've [decided](https://github.com/matrix-org/synapse/issues/5253#issuecomment-665976308) to remove the signature check for v1 lookups.

The signature check has been removed in v2 lookups. v1 lookups are currently deprecated. As mentioned in the above linked issue, this verification was causing deployments for the vector.im and matrix.org IS deployments, and this change is the simplest solution, without being unjustified.

Implementations are encouraged to use the v2 lookup API as it has [increased privacy benefits](https://github.com/matrix-org/matrix-doc/pull/2134).
2020-08-03 21:56:43 +01:00
Andrew Morgan 5d92a1428c
Prevent join->join membership transitions changing member count (#7977)
`StatsHandler` handles updates to the `current_state_delta_stream`, and updates room stats such as the amount of state events, joined users, etc.

However, it counts every new join membership as a new user entering a room (and that user being in another room), whereas it's possible for a user's membership status to go from join -> join, for instance when they change their per-room profile information.

This PR adds a check for join->join membership transitions, and bails out early, as none of the further checks are necessary at that point.

Due to this bug, membership stats in many rooms have ended up being wildly larger than their true values. I am not sure if we also want to include a migration step which recalculates these statistics (possibly using the `_populate_stats_process_rooms` bg update).

Bug introduced in the initial implementation https://github.com/matrix-org/synapse/pull/4338.
2020-08-03 21:54:24 +01:00
Patrick Cloke 3b415e23a5
Convert replication code to async/await. (#7987) 2020-08-03 07:12:55 -04:00
Patrick Cloke db5970ac6d
Convert ACME code to async/await. (#7989) 2020-08-03 07:09:33 -04:00
Erik Johnston faba873d4b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/add_rate_limiting_to_joins 2020-07-31 15:07:01 +01:00
Erik Johnston 18de00adb4 Add ratelimiting on joins 2020-07-31 15:06:56 +01:00
Richard van der Hoff 0a7fb24716
Fix invite rejection when we have no forward-extremeties (#7980)
Thanks to some slightly overzealous cleanup in the
`delete_old_current_state_events`, it's possible to end up with no
`event_forward_extremities` in a room where we have outstanding local
invites. The user would then get a "no create event in auth events" when trying
to reject the invite.

We can hack around it by using the dangling invite as the prev event.
2020-07-30 16:58:57 +01:00
Patrick Cloke c978f6c451
Convert federation client to async/await. (#7975) 2020-07-30 08:01:33 -04:00
Patrick Cloke 4cce8ef74e
Convert appservice to async. (#7973) 2020-07-30 07:27:39 -04:00
Patrick Cloke 3a00bd1378
Add additional logging for SAML sessions. (#7971) 2020-07-29 13:54:44 -04:00
Brendan Abolivier 8dff4a1242
Re-implement unread counts (#7736) 2020-07-29 18:26:55 +01:00
lugino-emeritus 3857de2194
Option to allow server admins to join complex rooms (#7902)
Fixes #7901.

Signed-off-by: Niklas Tittjung <nik_t.01@web.de>
2020-07-28 13:41:44 +01:00
Patrick Cloke 8553f46498
Convert a synapse.events to async/await. (#7949) 2020-07-27 13:40:22 -04:00
Patrick Cloke d8a9cd8d3e
Remove hacky error handling for inlineDeferreds. (#7950) 2020-07-27 08:35:56 -04:00
Patrick Cloke b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Patrick Cloke e739b20588
Fix up types and comments that refer to Deferreds. (#7945) 2020-07-24 10:53:25 -04:00
Patrick Cloke fefe9943ef
Convert presence handler helpers to async/await. (#7939) 2020-07-23 16:47:36 -04:00
Patrick Cloke 83434df381
Update the auth providers to be async. (#7935) 2020-07-23 15:45:39 -04:00
Patrick Cloke 13d77464c9
Follow-up to admin API to re-activate accounts (#7908) 2020-07-22 12:33:19 -04:00
Patrick Cloke cc9bb3dc3f
Convert the message handler to async/await. (#7884) 2020-07-22 12:29:15 -04:00
Richard van der Hoff b74919c72e
Add debugging to sync response generation (#7929) 2020-07-22 13:43:10 +01:00
Patrick Cloke de119063f2
Convert room list handler to async/await. (#7912) 2020-07-21 07:51:48 -04:00
Karthikeyan Singaravelan a7b06a81f0
Fix deprecation warning: import ABC from collections.abc (#7892) 2020-07-20 13:33:04 -04:00
Erik Johnston a3ad045286
Fix TypeError in synapse.notifier (#7880)
Fixes #7774
2020-07-17 14:11:05 +01:00
Patrick Cloke 6b3ac3b8cd
Convert device handler to async/await (#7871) 2020-07-17 07:09:25 -04:00
Patrick Cloke 6fca1b3506
Convert _base, profile, and _receipts handlers to async/await (#7860) 2020-07-17 07:08:30 -04:00
Richard van der Hoff 346476df21
Reject attempts to join empty rooms over federation (#7859)
We shouldn't allow others to make_join through us if we've left the room;
reject such attempts with a 404.

Fixes #7835. Fixes #6958.
2020-07-16 15:17:31 +01:00
Erik Johnston f2e38ca867
Allow moving typing off master (#7869) 2020-07-16 15:12:54 +01:00
Richard van der Hoff a827838706
Merge pull request #7866 from matrix-org/rav/fix_guest_user_id
Fix guest user registration with lots of client readers
2020-07-16 13:54:45 +01:00
Richard van der Hoff a973bcb8a4
Add some tiny type annotations (#7870)
I found these made pycharm have more of a clue as to what was going on in other places.
2020-07-16 13:52:29 +01:00
Richard van der Hoff c445bc0cad Use a postgres sequence to generate guest user IDs 2020-07-16 13:00:25 +01:00
Patrick Cloke 35450519de
Ensure that calls to `json.dumps` are compatible with the standard library json. (#7836) 2020-07-15 13:40:54 -04:00
Erik Johnston 97e1159ac1 Merge branch 'erikj/faster_typing' of github.com:matrix-org/synapse into develop 2020-07-15 16:54:30 +01:00
Patrick Cloke 8c7d0f163d
Allow accounts to be re-activated from the admin APIs. (#7847) 2020-07-15 11:00:21 -04:00
Erik Johnston 62352c3a1b Fix typo 2020-07-15 15:46:16 +01:00
Erik Johnston 3a3a618460 Use get_users_in_room rather than state handler in typing for speed 2020-07-15 15:42:07 +01:00
Patrick Cloke b11450dedc
Convert E2E key and room key handlers to async/await. (#7851) 2020-07-15 08:48:58 -04:00
Dirk Klimpel 491f0dab1b
Add delete room admin endpoint (#7613)
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-14 12:36:23 +01:00
Patrick Cloke 457096e6df
Support handling registration requests across multiple client readers. (#7830) 2020-07-13 13:31:46 -04:00
Patrick Cloke 66a4af8d96
Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802) 2020-07-10 14:30:08 -04:00
Erik Johnston f1245dc3c0
Fix resync remote devices on receive PDU in worker mode. (#7815)
The replication client requires that arguments are given as keyword
arguments, which was not done in this case. We also pull out the logic
so that we can catch and handle any exceptions raised, rather than
leaving them unhandled.
2020-07-10 18:23:17 +01:00
Erik Johnston e29c44340b
Fix recursion error when fetching auth chain over federation (#7817)
When fetching the state of a room over federation we receive the event
IDs of the state and auth chain. We then fetch those events that we
don't already have.

However, we used a function that recursively fetched any missing auth
events for the fetched events, which can lead to a lot of recursion if
the server is missing most of the auth chain. This work is entirely
pointless because would have queued up the missing events in the auth
chain to be fetched already.

Let's just diable the recursion, since it only gets called from one
place anyway.
2020-07-10 18:15:35 +01:00
Richard van der Hoff 53ee214f2f
`update_membership` declaration: now always returns an event id. (#7809) 2020-07-09 13:01:42 +01:00
Richard van der Hoff 08c5181a8d
Fix `can only concatenate list (not "tuple") to list` exception (#7810)
It seems auth_events can be either a list or a tuple, depending on Things.
2020-07-09 12:48:15 +01:00
Richard van der Hoff 2ab0b021f1
Generate real events when we reject invites (#7804)
Fixes #2181. 

The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
2020-07-09 10:40:19 +01:00
Richard van der Hoff 67593b1728
Add `HomeServer.signing_key` property (#7805)
... instead of duplicating `config.signing_key[0]` everywhere
2020-07-08 17:51:56 +01:00
Patrick Cloke ff0680f69d
Stop passing bytes when dumping JSON (#7799) 2020-07-08 07:14:56 -04:00
Erik Johnston 67d7756fcf
Refactor getting replication updates from database v2. (#7740) 2020-07-07 12:11:35 +01:00
Patrick Cloke 6d687ebba1
Convert the appservice handler to async/await. (#7775) 2020-07-06 07:40:35 -04:00
Will Hunt 62b1ce8539
isort 5 compatibility (#7786)
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
2020-07-05 16:32:02 +01:00
Dirk Klimpel 21a212f8e5
Fix inconsistent handling of upper and lower cases of email addresses. (#7021)
fixes #7016
2020-07-03 14:03:13 +01:00
Patrick Cloke f3e0f16240 Synapse 1.16.0rc2 (2020-07-02)
==============================
 
 Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
 Please see [below](https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1152-2020-07-02) for more details.
 
 Improved Documentation
 ----------------------
 
 - Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))
 
 Internal Changes
 ----------------
 
 - Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl79+qgACgkQM/xY9qcR
 MEhcaRAAjWLW3ojN1F0DUfE85jziZK2VdnMQC3g+uEOLX6QRbfqFNaNNMjLdK+vl
 K/+2ZoHkRsg6g8noSPhPmI1z1+hb5xDJaxjltzHxonIipW8XSU8o2PQMkf8O/BAy
 VS58y3GyLkhEgzWC+/hcII+LBgcqXpLuNM0xrKTHmxclIjdewlwe1v+hxkP+6wsX
 9Whhn1f4sNHrCtyFVK9uzMFcVyzcQaiWZRjEDMj2uR7rWT6UbCUifN/G4fWmtGbY
 xWoNoC4Qv8xiqXOG4U7juPp9T3bRyWMKyjBFM5PWO6Ec2zfafDyFzhBxJhlQhODG
 g21tS4PowX/dM/pBpJFEOPh1BVrPZzzTD+YMmTcd3NO79HeaQGqEX/+tzFCFUyPp
 0daJK3Y85+l5w/M09WU8DDN8CiR3PFJyGDIZp+nweMsiJZkbEbLOkh1tx6TL+5/6
 zwewU6cq8nTVGrn53Tn58l8C7Sj4w+Qk+1XDzymAoidyoWqAKW9Y/fw53PaViUSx
 voDu0rpsEUXR1OzCBG8SAPQCFy9gdEWV04OvIpzHuq2uojkz66f7NAXy+Wz+Occ9
 AYb/s6Ei80bGCLgRd5jg+myqavwRbzCyv+LIC6dxpopxZJ3AzrFuD11eXKtrIxOC
 FZYf3U4KeBk4Q9TV5IFV1xcGFrq5aK36LdmP6WOsEl3PXVT9p/Q=
 =YaJn
 -----END PGP SIGNATURE-----

Merge tag 'v1.16.0rc2' into develop

Synapse 1.16.0rc2 (2020-07-02)
==============================

Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
Please see [below](https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1152-2020-07-02) for more details.

Improved Documentation
----------------------

- Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))

Internal Changes
----------------

- Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
2020-07-02 11:25:56 -04:00
Patrick Cloke f2bcc6ecbf Merge branch 'master' into release-v1.16.0 2020-07-02 11:02:42 -04:00
Patrick Cloke 4d978d7db4 Merge branch 'master' into develop 2020-07-02 10:55:41 -04:00
Erik Johnston 96e9afe625 Correctly handle outliers as prev events over federation 2020-07-02 10:00:33 -04:00
Patrick Cloke ea26e9a98b Ensure that HTML pages served from Synapse include headers to avoid embedding. 2020-07-02 09:58:31 -04:00
Richard van der Hoff e866512367
Add early returns to `_check_for_soft_fail` (#7769)
my editor was complaining about unset variables, so let's add some early
returns to fix that and reduce indentation/cognitive load.
2020-07-01 16:41:19 +01:00
Richard van der Hoff a6eae69ffe
Type checking for `FederationHandler` (#7770)
fix a few things to make this pass mypy.
2020-07-01 16:21:02 +01:00
Erik Johnston 1e03513f9a
Fix new metric where we used ms instead of seconds (#7771)
Introduced in #7755, not yet released.
2020-07-01 15:23:58 +01:00
Brendan Abolivier 74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Patrick Cloke 71cccf1593
Additional configuration options for auto-join rooms (#7763) 2020-06-30 15:41:36 -04:00
Erik Johnston a99658074d
Add some metrics for inbound and outbound federation processing times (#7755) 2020-06-30 16:58:06 +01:00
Erik Johnston 0e0a2817a2
Yield during large v2 state res. (#7735)
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.

To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
2020-06-24 18:48:18 +01:00
Patrick Cloke e060bf4462
Convert directory handler to async/await (#7727) 2020-06-22 07:18:00 -04:00
Patrick Cloke 3630825612
Convert the typing handler to async/await. (#7679) 2020-06-17 10:37:59 -04:00
Brendan Abolivier 46613aaf79
Implement unread counter (MSC2625) (#7673)
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
Erik Johnston f6f7511a4c
Refactor getting replication updates from database. (#7636)
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
2020-06-16 17:10:28 +01:00
Dagfinn Ilmari Mannsåker a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke 98c4e35e3c
Convert the device message and pagination handlers to async/await. (#7678) 2020-06-16 08:06:17 -04:00
Hubert Chathi 2b2344652b
Ensure etag is a string for GET room_keys/version response (#7691) 2020-06-15 13:42:44 -04:00
Erik Johnston 4241a10673
Fix "There was no active span when trying to log." error (#7698) 2020-06-15 16:42:21 +01:00
Brendan Abolivier 6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Brendan Abolivier c2b4621630 Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:08 +01:00
Patrick Cloke bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Patrick Cloke d0a43d431e
Fix a typo when comparing the URI & method during UI Auth. (#7689) 2020-06-12 14:12:04 -04:00
Brendan Abolivier 9dbd006607
Appease mypy 2020-06-10 20:44:24 +01:00
Brendan Abolivier 243f0ba6ce
Lint 2020-06-10 20:35:35 +01:00
Brendan Abolivier df3323a7cf
Use temporary prefixes as per the MSC 2020-06-10 20:32:01 +01:00
Andrew Morgan fcd6961441
Add option to enable encryption by default for new rooms (#7639)
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on #7637
2020-06-10 17:44:34 +01:00
Brendan Abolivier ef345c5a7b
Add a new unread_counter to sync responses 2020-06-10 16:21:16 +01:00
Patrick Cloke 3c45a78090
Convert the registration handler to async/await. (#7649) 2020-06-08 11:15:02 -04:00
Patrick Cloke 737b4a936e
Convert user directory handler and related classes to async/await. (#7640) 2020-06-05 14:42:55 -04:00
Travis Ralston 09099313e6
Add an option to disable autojoin for guest accounts (#6637)
Fixes https://github.com/matrix-org/synapse/issues/3177
2020-06-05 18:18:15 +01:00
Patrick Cloke f1e61ef85c Typo fixes. 2020-06-05 08:43:21 -04:00
Dirk Klimpel 908f9e2d24
Allow new users to be registered via the admin API even if the monthly active user limit has been reached (#7263) 2020-06-05 13:08:49 +01:00
Andrew Morgan f4e6495b5d
Performance improvements and refactor of Ratelimiter (#7595)
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:

* Rather undocumented, and
* causing a *lot* of config checks

This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. 

Best to be reviewed commit-by-commit.
2020-06-05 10:47:20 +01:00
Richard van der Hoff 11de843626
Cleanups to the OpenID Connect integration (#7628)
docs, default configs, comments. Nothing very significant.
2020-06-03 21:13:17 +01:00
Brendan Abolivier c9507be989
Check if the localpart is reserved for guests earlier in the registration flow (#7625)
This is so the user is warned about the username not being valid as soon as possible, rather than only once they've finished UIA.
2020-06-03 16:55:02 +02:00
Erik Johnston 11dc2b4698
Fix exceptions when fetching events from a down host. (#7622)
We already caught some exceptions, but not all.
2020-06-03 14:12:13 +01:00
Richard van der Hoff 1bbc9e2df6
Clean up exception handling in SAML2ResponseResource (#7614)
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string

* Clean up exception handling in SAML2ResponseResource

  * use the existing code in `return_html_error` instead of re-implementing it
    (giving it a jinja2 template rather than inventing a new form of template)

  * do the exception-catching in the REST layer rather than in the handler
    layer, to make sure we catch all exceptions.
2020-06-03 10:41:12 +01:00
Brendan Abolivier 33c39ab93c
Process cross-signing keys when resyncing device lists (#7594)
It looks like `user_device_resync` was ignoring cross-signing keys from the results received from the remote server. This patch fixes this, by processing these keys using the same process `_handle_signing_key_updates` does (and effectively factor that part out of that function).
2020-06-01 17:47:30 +02:00
Patrick Cloke 6af9cdca24
Convert groups local and server to async/await. (#7600) 2020-06-01 07:28:43 -04:00
Brendan Abolivier c1bdd4fac7
Don't fail all of an iteration of the device list retry loop on error (#7609)
Without this patch, if an error happens which isn't caught by `user_device_resync`, then `_maybe_retry_device_resync` would fail, without retrying the next users in the iteration. This patch fixes this so that it now only logs an error in this case.
2020-06-01 12:55:14 +02:00
Erik Johnston cb495f526d
Fix 'FederationGroupsRoomsServlet' API when group has room server is not in. (#7599) 2020-05-29 17:49:47 +01:00
Christopher Cooper c4a820b32a
allow emails to be passed through SAML (#7385)
Signed-off-by: Christopher Cooper <cooperc@ocf.berkeley.edu>
2020-05-27 17:40:08 +01:00
Patrick Cloke ef884f6d04
Convert identity handler to async/await. (#7561) 2020-05-26 13:46:22 -04:00
Richard van der Hoff f4269694ce
Optimise some references to hs.config (#7546)
These are surprisingly expensive, and we only really need to do them at startup.
2020-05-22 21:47:07 +01:00
Erik Johnston e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Erik Johnston 1531b214fc
Add ability to wait for replication streams (#7542)
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
Erik Johnston 06a02bc1ce
Convert sending mail to async/await. (#7557)
Mainly because sometimes the email push code raises exceptions where the
stack traces have gotten lost, which is hopefully fixed by this.
2020-05-22 13:41:11 +01:00
Erik Johnston 710d958c64
On upgrade room only send canonical alias once. (#7547)
Instead of doing a complicated dance of deleting and moving aliases one
by one, which sends a canonical alias update into the old room for each
one, lets do it all in one go.

This also changes the function to move *all* local alias events to the new
room, however that happens later on anyway.
2020-05-22 11:41:41 +01:00
Erik Johnston 547e4dd83e
Fix exception reporting due to HTTP request errors. (#7556)
These are business as usual errors, rather than stuff we want to log at
error.
2020-05-22 11:39:20 +01:00
Brendan Abolivier d1ae1015ec
Retry to sync out of sync device lists (#7453)
When a call to `user_device_resync` fails, we don't currently mark the remote user's device list as out of sync, nor do we retry to sync it.

https://github.com/matrix-org/synapse/pull/6776 introduced some code infrastructure to mark device lists as stale/out of sync.

This commit uses that code infrastructure to mark device lists as out of sync if processing an incoming device list update makes the device handler realise that the device list is out of sync, but we can't resync right now.

It also adds a looping call to retry all failed resync every 30s. This shouldn't cause too much spam in the logs as this commit also removes the "Failed to handle device list update for..." warning logs when catching `NotRetryingDestination`.

Fixes #7418
2020-05-21 17:41:12 +02:00
Patrick Cloke 9dc6f3075a
Hash passwords earlier in the password reset process (#7538)
This now matches the logic of the registration process as modified in
56db0b1365 / #7523.
2020-05-20 09:48:03 -04:00
Aaron Raimist 250f3eb991
Omit displayname or avatar_url if they aren't set instead of returning null (#7497)
Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec.

Fixes https://github.com/matrix-org/synapse/issues/7333

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-05-19 10:31:25 +01:00
Patrick Cloke ee421e5244 Synapse 1.13.0rc3 (2020-05-18)
Bugfixes:
 
 - Hash passwords as early as possible during registration. #7523
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl7CpGYACgkQM/xY9qcR
 MEhVixAAk2hDWVXxbGzUk2LmfiIsFA2eV55sw+VqEw0eRfe1d/mP6aH75VmTt3pw
 IymZUVxDXdbTnPNPw+ldyGhzu9C6JJjXnNRBZnIkR5vcSbWsV0mPl/qHFu/4FnZI
 m4Nj1Sx3sG0CyNDpWjVrzTW6SbDX9J68DXbLwnNTSX3KPa7gNn6TUmFfKzlrNI23
 pPmD+EITYMn/H9HOhxhTzq//Ja7UOViAKQ0q4N2I4GxmLP6ufx9P3s5FG/oJqA+H
 Pka2+9JnfHq2Ze22CoDcg8q5f5MgVkQzGeir0ZsGJwJqOYjeTmbCvD3T/RYWO5g+
 ZghON3tsMQmdzUQqGRxcn/YLOZY9ZqrX2kBf5E6Wapwj9MfKg2ToLZM4yrWN0+RX
 KDuWaKXYtkSQCo1nDS2KooVMWjGNZautWWnHzZ0KNQCIkxVpGC234JYI685grKXb
 dg7R41kdXI7NJzqS4iM1fxXoLx64fpoREa/pbLF6VeLaYXBlzMjfhiIx2pQBN3L/
 q/y3ftev9VCp+2wPxiKUkiC4Sh7dgWUzNuyHU+4lsPUbI1H/MN5dN2ryObdEGWc/
 5YU3tv2MTQJ7jECHR+/fastnG+5d2kVm/FK+zVhG17JvA2VmDaLnSde+mzGbsO1N
 gIUx5VrTEP7y0tC8C/VgbS3c2KqCSOopqd3j2slLLrtQlXM71VE=
 =lpDI
 -----END PGP SIGNATURE-----

Merge tag 'v1.13.0rc3' into develop

Synapse 1.13.0rc3 (2020-05-18)

Bugfixes:

- Hash passwords as early as possible during registration. #7523
2020-05-18 11:10:04 -04:00
Patrick Cloke 56db0b1365
Hash passwords earlier in the registration process (#7523) 2020-05-18 09:46:18 -04:00
Patrick Cloke c29915bd05
Add type hints to room member handlers (#7513) 2020-05-15 15:05:25 -04:00
Patrick Cloke a3cf36f76e
Support UI Authentication for OpenID Connect accounts (#7457) 2020-05-15 12:26:02 -04:00
Patrick Cloke e9f3de0bab
Update the room member handler to use async/await. (#7507) 2020-05-15 09:32:13 -04:00
Andrew Morgan 02d97fc3ba
Ignore incoming presence updates when presence is disabled (#7508) 2020-05-15 11:44:00 +01:00
Richard van der Hoff dede23ff1e Synapse 1.13.0rc2 (2020-05-14)
==============================
 
 Bugfixes
 --------
 
 - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
 - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
 
 Internal Changes
 ----------------
 
 - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl69IQ8ACgkQOSor00I9
 eP87lAf8DK+v6cs2U0BoD5opzQ7ZazJT6JYTmnMBaTzHU6Wx20V2ttkF7Vpwm3WU
 Zsz0048tdYtHFyYBQ1kF5RNIBBJwV8SA/QUcPkR7FVpwZMLR2q4aJn0EE7kC9OMf
 tYsmdbHeBdyfLXpXzazxWlgHquLyEIt52ykAcCphjx/Jl2fAExFEhtfsxpECoJ2f
 8Dqhjg3WFjd6QWU6AFkElbwHUYCdIWdJOcsC8N1p8OvBmDz5QXv/RlYipHE00Cpx
 QQQOgEjdRc6dlz2mbetMklnfII3p2kO9bzNdmEpOzT0Zt7nFaGdntW4I1QA0yJfa
 gows9bYMzhqYk7YSiyTYOZ4qyavVtw==
 =N/zZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.13.0rc2' into develop

Synapse 1.13.0rc2 (2020-05-14)
==============================

Bugfixes
--------

- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))

Internal Changes
----------------

- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
2020-05-14 11:46:38 +01:00
Patrick Cloke 5d64fefd6c
Do not validate that the client dict is stable during UI Auth. (#7483)
This backs out some of the validation for the client dictionary and logs if
this changes during a user interactive authentication session instead.
2020-05-13 14:26:44 -04:00