505ea932f5
Merge branch 'develop' into matrix-org-hotfixes
2020-09-03 15:30:00 +01:00
82c1ee1c22
Add experimental support for sharding event persister. ( #8170 )
...
This is *not* ready for production yet. Caveats:
1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-02 15:48:37 +01:00
da77520cd1
Convert additional databases to async/await part 2 ( #8200 )
2020-09-01 08:39:04 -04:00
3f488bfded
Merge branch 'develop' into matrix-org-hotfixes
2020-08-27 10:16:21 -04:00
e0d6244beb
Remove unused parameter from, and add safeguard in, get_room_data ( #8174 )
...
Small cleanup PR.
* Removed the unused `is_guest` argument
* Added a safeguard to a (currently) impossible code path, fixing static checking at the same time.
2020-08-26 15:07:35 +01:00
5099bd68da
Do not allow send_nonmember_event to be called with shadow-banned users. ( #8158 )
2020-08-25 10:52:15 -04:00
cbd8d83da7
Stop shadow-banned users from sending non-member events. ( #8142 )
2020-08-24 13:58:56 -04:00
eebf52be06
Be stricter about JSON that is accepted by Synapse ( #8106 )
2020-08-19 07:26:03 -04:00
f40645e60b
Convert events worker database to async/await. ( #8071 )
2020-08-18 16:20:49 -04:00
b4c1cfacc2
Merge branch 'develop' into matrix-org-hotfixes
2020-08-18 18:20:01 +01:00
afe4c4e02e
Merge branch 'develop' into matrix-org-hotfixes
2020-08-18 18:13:47 +01:00
3c01724b33
Fix the return type of send_nonmember_events. ( #8112 )
2020-08-18 09:53:13 -04:00
25e55d2598
Return the previous stream token if a non-member event is a duplicate. ( #8093 )
2020-08-18 07:53:23 -04:00
53834bb9c4
Run `remove_push_actions_from_staging` in foreground ( #8081 )
...
If we got an error persisting an event, we would try to remove the push actions
asynchronously, which would lead to a 'Re-starting finished log context'
warning.
I don't think there's any need for this to be asynchronous.
2020-08-13 17:05:31 +01:00
527f73d902
Merge branch 'develop' into matrix-org-hotfixes
2020-08-13 11:45:08 +01:00
5dd73d029e
Add type hints to handlers.message and events.builder ( #8067 )
2020-08-12 15:05:50 +01:00
fcbab08cbd
Add an assertion on prev_events in create_new_client_event ( #8041 )
...
I think this would have caught all the cases in
https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes
more sense here than a 403
2020-08-10 12:29:47 +01:00
d4a7829b12
Convert synapse.api to async/await ( #8031 )
2020-08-06 08:30:06 -04:00
a7bdf98d01
Rename database classes to make some sense ( #8033 )
2020-08-05 21:38:57 +01:00
b2ccc72a00
Merge branch 'release-v1.18.0' into matrix-org-hotfixes
2020-07-28 10:15:22 +01:00
d8a9cd8d3e
Remove hacky error handling for inlineDeferreds. ( #7950 )
2020-07-27 08:35:56 -04:00
be777e325d
Merge branch 'develop' into matrix-org-hotfixes
2020-07-24 09:57:49 +01:00
cc9bb3dc3f
Convert the message handler to async/await. ( #7884 )
2020-07-22 12:29:15 -04:00
6fca1b3506
Convert _base, profile, and _receipts handlers to async/await ( #7860 )
2020-07-17 07:08:30 -04:00
cc86fbc9ad
Merge branch 'develop' into matrix-org-hotfixes
2020-07-09 11:06:52 +01:00
2ab0b021f1
Generate real events when we reject invites ( #7804 )
...
Fixes #2181 .
The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
2020-07-09 10:40:19 +01:00
c16bb06d25
Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes
2020-06-25 09:39:01 +01:00
e060bf4462
Convert directory handler to async/await ( #7727 )
2020-06-22 07:18:00 -04:00
8ba1086801
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2020-06-17 14:38:15 +01:00
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents ( #7704 )
2020-06-16 08:51:47 -04:00
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. ( #7692 )
2020-06-15 07:03:36 -04:00
d6c7550cf5
Synapse 1.15.0rc1 (2020-06-09)
...
==============================
Features
--------
- Advertise support for Client-Server API r0.6.0 and remove related unstable feature flags. ([\#6585](https://github.com/matrix-org/synapse/issues/6585 ))
- Add an option to disable autojoining rooms for guest accounts. ([\#6637](https://github.com/matrix-org/synapse/issues/6637 ))
- For SAML authentication, add the ability to pass email addresses to be added to new users' accounts via SAML attributes. Contributed by Christopher Cooper. ([\#7385](https://github.com/matrix-org/synapse/issues/7385 ))
- Add admin APIs to allow server admins to manage users' devices. Contributed by @dklimpel. ([\#7481](https://github.com/matrix-org/synapse/issues/7481 ))
- Add support for generating thumbnails for WebP images. Previously, users would see an empty box instead of preview image. ([\#7586](https://github.com/matrix-org/synapse/issues/7586 ))
- Support the standardized `m.login.sso` user-interactive authentication flow. ([\#7630](https://github.com/matrix-org/synapse/issues/7630 ))
Bugfixes
--------
- Allow new users to be registered via the admin API even if the monthly active user limit has been reached. Contributed by @dkimpel. ([\#7263](https://github.com/matrix-org/synapse/issues/7263 ))
- Fix email notifications not being enabled for new users when created via the Admin API. ([\#7267](https://github.com/matrix-org/synapse/issues/7267 ))
- Fix str placeholders in an instance of `PrepareDatabaseException`. Introduced in Synapse v1.8.0. ([\#7575](https://github.com/matrix-org/synapse/issues/7575 ))
- Fix a bug in automatic user creation during first time login with `m.login.jwt`. Regression in v1.6.0. Contributed by @olof. ([\#7585](https://github.com/matrix-org/synapse/issues/7585 ))
- Fix a bug causing the cross-signing keys to be ignored when resyncing a device list. ([\#7594](https://github.com/matrix-org/synapse/issues/7594 ))
- Fix metrics failing when there is a large number of active background processes. ([\#7597](https://github.com/matrix-org/synapse/issues/7597 ))
- Fix bug where returning rooms for a group would fail if it included a room that the server was not in. ([\#7599](https://github.com/matrix-org/synapse/issues/7599 ))
- Fix duplicate key violation when persisting read markers. ([\#7607](https://github.com/matrix-org/synapse/issues/7607 ))
- Prevent an entire iteration of the device list resync loop from failing if one server responds with a malformed result. ([\#7609](https://github.com/matrix-org/synapse/issues/7609 ))
- Fix exceptions when fetching events from a remote host fails. ([\#7622](https://github.com/matrix-org/synapse/issues/7622 ))
- Make `synctl restart` start synapse if it wasn't running. ([\#7624](https://github.com/matrix-org/synapse/issues/7624 ))
- Pass device information through to the login endpoint when using the login fallback. ([\#7629](https://github.com/matrix-org/synapse/issues/7629 ))
- Advertise the `m.login.token` login flow when OpenID Connect is enabled. ([\#7631](https://github.com/matrix-org/synapse/issues/7631 ))
- Fix bug in account data replication stream. ([\#7656](https://github.com/matrix-org/synapse/issues/7656 ))
Improved Documentation
----------------------
- Update the OpenBSD installation instructions. ([\#7587](https://github.com/matrix-org/synapse/issues/7587 ))
- Advertise Python 3.8 support in `setup.py`. ([\#7602](https://github.com/matrix-org/synapse/issues/7602 ))
- Add a link to `#synapse:matrix.org` in the troubleshooting section of the README. ([\#7603](https://github.com/matrix-org/synapse/issues/7603 ))
- Clarifications to the admin api documentation. ([\#7647](https://github.com/matrix-org/synapse/issues/7647 ))
Internal Changes
----------------
- Convert the identity handler to async/await. ([\#7561](https://github.com/matrix-org/synapse/issues/7561 ))
- Improve query performance for fetching state from a PostgreSQL database. ([\#7567](https://github.com/matrix-org/synapse/issues/7567 ))
- Speed up processing of federation stream RDATA rows. ([\#7584](https://github.com/matrix-org/synapse/issues/7584 ))
- Add comment to systemd example to show postgresql dependency. ([\#7591](https://github.com/matrix-org/synapse/issues/7591 ))
- Refactor `Ratelimiter` to limit the amount of expensive config value accesses. ([\#7595](https://github.com/matrix-org/synapse/issues/7595 ))
- Convert groups handlers to async/await. ([\#7600](https://github.com/matrix-org/synapse/issues/7600 ))
- Clean up exception handling in `SAML2ResponseResource`. ([\#7614](https://github.com/matrix-org/synapse/issues/7614 ))
- Check that all asynchronous tasks succeed and general cleanup of `MonthlyActiveUsersTestCase` and `TestMauLimit`. ([\#7619](https://github.com/matrix-org/synapse/issues/7619 ))
- Convert `get_user_id_by_threepid` to async/await. ([\#7620](https://github.com/matrix-org/synapse/issues/7620 ))
- Switch to upstream `dh-virtualenv` rather than our fork for Debian package builds. ([\#7621](https://github.com/matrix-org/synapse/issues/7621 ))
- Update CI scripts to check the number in the newsfile fragment. ([\#7623](https://github.com/matrix-org/synapse/issues/7623 ))
- Check if the localpart of a Matrix ID is reserved for guest users earlier in the registration flow, as well as when responding to requests to `/register/available`. ([\#7625](https://github.com/matrix-org/synapse/issues/7625 ))
- Minor cleanups to OpenID Connect integration. ([\#7628](https://github.com/matrix-org/synapse/issues/7628 ))
- Attempt to fix flaky test: `PhoneHomeStatsTestCase.test_performance_100`. ([\#7634](https://github.com/matrix-org/synapse/issues/7634 ))
- Fix typos of `m.olm.curve25519-aes-sha2` and `m.megolm.v1.aes-sha2` in comments, test files. ([\#7637](https://github.com/matrix-org/synapse/issues/7637 ))
- Convert user directory, state deltas, and stats handlers to async/await. ([\#7640](https://github.com/matrix-org/synapse/issues/7640 ))
- Remove some unused constants. ([\#7644](https://github.com/matrix-org/synapse/issues/7644 ))
- Fix type information on `assert_*_is_admin` methods. ([\#7645](https://github.com/matrix-org/synapse/issues/7645 ))
- Convert registration handler to async/await. ([\#7649](https://github.com/matrix-org/synapse/issues/7649 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAl7fsrsACgkQHgFcFF8Z
Fs0+4g/+L9rn5pnCdga0D5S2mBsmDzaU6kemNfZqfJCHhtK6e3BTvCbJ2cVwAJ5C
E/KctWfrrAWqN3DHPd2PetEjUNvDrJSwnVKmkmZCBLB2kybbGd6fBys75N0cj0Tk
jPedske3J/J9XZhnphJb1Zy+EjFlqQ+9Nm4xAo95G+pSoFhUrcAmAxLXcGU+bci5
ObFWNSd7eZ7vtHWQXomNIs0d/45iQGmd1ElRcZ/HxW0dSxwQhLUgM3xbSIquAcsK
q+P9zDvEkyrhSpnVYNdqBuFHzW44PktvpJMSfox0BS7WVIPaIm3MQE2yym44PqU9
UIEh2I56J/yWywKHE3ABX9Eyh5ROL+k2i6zzMPZYnttMWj4/O9Km8RbJ2pjKm695
JLaU2p1X67S1nEE8G8h/7GeWVFZh6ZnfyPqRt1F5iwYIg+vIqKH2oEtiRh7lLbW4
JCsOQHKdN85w+88iehPu5SWSxKETcm+BpeLiQasXXkWVb6hr4ZJ5h9afrwQyXJYY
i3Q/bBF4DMKMfwp3pMsUYlSdZZHXoX7BdOpkSqKeFK/QDgJBEDORmMU+krIXFObp
6NGWj2xIIzHKXJx/4HCl/m6J285q/VXGOLVDHEGmOarBfgFGgq3tXn5xSFl1f/Fm
/Gt11uIrtIwwSgDbGL92KUkySljZ4YbFywEqUR8pPlgTqQZJ2XI=
=KXx8
-----END PGP SIGNATURE-----
Merge tag 'v1.15.0rc1' of github.com:matrix-org/synapse into matrix-org-hotfixes
Synapse 1.15.0rc1 (2020-06-09)
==============================
Features
--------
- Advertise support for Client-Server API r0.6.0 and remove related unstable feature flags. ([\#6585](https://github.com/matrix-org/synapse/issues/6585 ))
- Add an option to disable autojoining rooms for guest accounts. ([\#6637](https://github.com/matrix-org/synapse/issues/6637 ))
- For SAML authentication, add the ability to pass email addresses to be added to new users' accounts via SAML attributes. Contributed by Christopher Cooper. ([\#7385](https://github.com/matrix-org/synapse/issues/7385 ))
- Add admin APIs to allow server admins to manage users' devices. Contributed by @dklimpel. ([\#7481](https://github.com/matrix-org/synapse/issues/7481 ))
- Add support for generating thumbnails for WebP images. Previously, users would see an empty box instead of preview image. ([\#7586](https://github.com/matrix-org/synapse/issues/7586 ))
- Support the standardized `m.login.sso` user-interactive authentication flow. ([\#7630](https://github.com/matrix-org/synapse/issues/7630 ))
Bugfixes
--------
- Allow new users to be registered via the admin API even if the monthly active user limit has been reached. Contributed by @dkimpel. ([\#7263](https://github.com/matrix-org/synapse/issues/7263 ))
- Fix email notifications not being enabled for new users when created via the Admin API. ([\#7267](https://github.com/matrix-org/synapse/issues/7267 ))
- Fix str placeholders in an instance of `PrepareDatabaseException`. Introduced in Synapse v1.8.0. ([\#7575](https://github.com/matrix-org/synapse/issues/7575 ))
- Fix a bug in automatic user creation during first time login with `m.login.jwt`. Regression in v1.6.0. Contributed by @olof. ([\#7585](https://github.com/matrix-org/synapse/issues/7585 ))
- Fix a bug causing the cross-signing keys to be ignored when resyncing a device list. ([\#7594](https://github.com/matrix-org/synapse/issues/7594 ))
- Fix metrics failing when there is a large number of active background processes. ([\#7597](https://github.com/matrix-org/synapse/issues/7597 ))
- Fix bug where returning rooms for a group would fail if it included a room that the server was not in. ([\#7599](https://github.com/matrix-org/synapse/issues/7599 ))
- Fix duplicate key violation when persisting read markers. ([\#7607](https://github.com/matrix-org/synapse/issues/7607 ))
- Prevent an entire iteration of the device list resync loop from failing if one server responds with a malformed result. ([\#7609](https://github.com/matrix-org/synapse/issues/7609 ))
- Fix exceptions when fetching events from a remote host fails. ([\#7622](https://github.com/matrix-org/synapse/issues/7622 ))
- Make `synctl restart` start synapse if it wasn't running. ([\#7624](https://github.com/matrix-org/synapse/issues/7624 ))
- Pass device information through to the login endpoint when using the login fallback. ([\#7629](https://github.com/matrix-org/synapse/issues/7629 ))
- Advertise the `m.login.token` login flow when OpenID Connect is enabled. ([\#7631](https://github.com/matrix-org/synapse/issues/7631 ))
- Fix bug in account data replication stream. ([\#7656](https://github.com/matrix-org/synapse/issues/7656 ))
Improved Documentation
----------------------
- Update the OpenBSD installation instructions. ([\#7587](https://github.com/matrix-org/synapse/issues/7587 ))
- Advertise Python 3.8 support in `setup.py`. ([\#7602](https://github.com/matrix-org/synapse/issues/7602 ))
- Add a link to `#synapse:matrix.org` in the troubleshooting section of the README. ([\#7603](https://github.com/matrix-org/synapse/issues/7603 ))
- Clarifications to the admin api documentation. ([\#7647](https://github.com/matrix-org/synapse/issues/7647 ))
Internal Changes
----------------
- Convert the identity handler to async/await. ([\#7561](https://github.com/matrix-org/synapse/issues/7561 ))
- Improve query performance for fetching state from a PostgreSQL database. ([\#7567](https://github.com/matrix-org/synapse/issues/7567 ))
- Speed up processing of federation stream RDATA rows. ([\#7584](https://github.com/matrix-org/synapse/issues/7584 ))
- Add comment to systemd example to show postgresql dependency. ([\#7591](https://github.com/matrix-org/synapse/issues/7591 ))
- Refactor `Ratelimiter` to limit the amount of expensive config value accesses. ([\#7595](https://github.com/matrix-org/synapse/issues/7595 ))
- Convert groups handlers to async/await. ([\#7600](https://github.com/matrix-org/synapse/issues/7600 ))
- Clean up exception handling in `SAML2ResponseResource`. ([\#7614](https://github.com/matrix-org/synapse/issues/7614 ))
- Check that all asynchronous tasks succeed and general cleanup of `MonthlyActiveUsersTestCase` and `TestMauLimit`. ([\#7619](https://github.com/matrix-org/synapse/issues/7619 ))
- Convert `get_user_id_by_threepid` to async/await. ([\#7620](https://github.com/matrix-org/synapse/issues/7620 ))
- Switch to upstream `dh-virtualenv` rather than our fork for Debian package builds. ([\#7621](https://github.com/matrix-org/synapse/issues/7621 ))
- Update CI scripts to check the number in the newsfile fragment. ([\#7623](https://github.com/matrix-org/synapse/issues/7623 ))
- Check if the localpart of a Matrix ID is reserved for guest users earlier in the registration flow, as well as when responding to requests to `/register/available`. ([\#7625](https://github.com/matrix-org/synapse/issues/7625 ))
- Minor cleanups to OpenID Connect integration. ([\#7628](https://github.com/matrix-org/synapse/issues/7628 ))
- Attempt to fix flaky test: `PhoneHomeStatsTestCase.test_performance_100`. ([\#7634](https://github.com/matrix-org/synapse/issues/7634 ))
- Fix typos of `m.olm.curve25519-aes-sha2` and `m.megolm.v1.aes-sha2` in comments, test files. ([\#7637](https://github.com/matrix-org/synapse/issues/7637 ))
- Convert user directory, state deltas, and stats handlers to async/await. ([\#7640](https://github.com/matrix-org/synapse/issues/7640 ))
- Remove some unused constants. ([\#7644](https://github.com/matrix-org/synapse/issues/7644 ))
- Fix type information on `assert_*_is_admin` methods. ([\#7645](https://github.com/matrix-org/synapse/issues/7645 ))
- Convert registration handler to async/await. ([\#7649](https://github.com/matrix-org/synapse/issues/7649 ))
2020-06-10 10:57:26 +01:00
f4e6495b5d
Performance improvements and refactor of Ratelimiter ( #7595 )
...
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:
* Rather undocumented, and
* causing a *lot* of config checks
This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation.
Best to be reviewed commit-by-commit.
2020-06-05 10:47:20 +01:00
8beca8e21f
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2020-05-26 09:43:21 +01:00
f4269694ce
Optimise some references to hs.config ( #7546 )
...
These are surprisingly expensive, and we only really need to do them at startup.
2020-05-22 21:47:07 +01:00
e5c67d04db
Add option to move event persistence off master ( #7517 )
2020-05-22 16:11:35 +01:00
1531b214fc
Add ability to wait for replication streams ( #7542 )
...
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).
Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.
People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
cf92310da2
Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
2020-05-21 15:19:00 +01:00
89f795fe8a
Merge branch 'rav/matrix_hacks' into matrix-org-hotfixes
2020-05-20 23:40:22 +01:00
1c347c84bf
inline some config references
2020-05-20 23:33:13 +01:00
250f3eb991
Omit displayname or avatar_url if they aren't set instead of returning null ( #7497 )
...
Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec.
Fixes https://github.com/matrix-org/synapse/issues/7333
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-05-19 10:31:25 +01:00
1124111a12
Allow censoring of events to happen on workers. ( #7492 )
...
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
2020-05-13 17:15:40 +01:00
ae4f6140f1
Merge branch 'release-v1.13.0' into matrix-org-hotfixes
2020-05-07 10:42:56 +01:00
d9b8d27494
Add a configuration setting for the dummy event threshold ( #7422 )
...
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
ff20747703
Merge branch 'release-v1.13.0' into matrix-org-hotfixes
2020-05-06 11:57:36 +01:00
6b22921b19
async/await is_server_admin ( #7363 )
2020-05-01 15:15:36 +01:00
649e48a799
Merge branch 'develop' into matrix-org-hotfixes
2020-04-24 14:07:47 +01:00
190ab593b7
Use the proper error code when a canonical alias that does not exist is used. ( #7109 )
2020-03-23 15:21:54 -04:00
69ce55c510
Don't filter out dummy events when we're checking the visibility of state
2020-03-11 17:52:54 +00:00
936686ed2d
Don't filter out events when we're checking the visibility of state
2020-03-11 15:21:25 +00:00
Brendan Abolivier