Commit Graph

17882 Commits (1b15a3d92cbe1ee9475319ff81abe8760d6be19f)

Author SHA1 Message Date
Erik Johnston 1b15a3d92c
Fix port script so that it can be run again after failure. (#8755)
If the script fails (or is CTRL-C'ed) between porting some of the events table and copying of the sequences then the port script will immediately die if run again due to the postgres DB having inconsistencies between sequences and tables.

The fix is to move the porting of sequences to before porting the tables, so that there is never a period where the Postgres DB is inconsistent. To do that we need to change how we port the sequences so that it calculates the values from the SQLite DB rather than the Postgres DB.

Fixes #8619
2020-11-13 11:53:51 +00:00
Erik Johnston 4cb00d297f
Cache event ID to auth event IDs lookups (#8752)
This should hopefully speed up `get_auth_chain_difference` a bit in the case of repeated state res on the same rooms.

`get_auth_chain_difference` does a breadth first walk of the auth graphs by repeatedly looking up events' auth events. Different state resolutions on the same room will end up doing a lot of the same event to auth events lookups, so by caching them we should speed things up in cases of repeated state resolutions on the same room.
2020-11-13 11:29:18 +00:00
Erik Johnston c2d4467cd4
Enable reconnection in DB pool (#8726)
`adbapi.ConnectionPool` let's you turn on auto reconnect of DB connections. This is off by default.
As far as I can tell if its not enabled dead connections never get removed from the pool.

Maybe helps #8574
2020-11-12 14:26:24 +00:00
Erik Johnston 41a389934e
Fix port script fails when DB has no backfilled events. (#8729)
Fixes #8618
2020-11-11 15:08:03 +00:00
Erik Johnston 5829872bec
Fix port script to handle foreign key constraints (#8730) 2020-11-11 15:07:34 +00:00
Will Hunt 89700dfb8c
Check support room has only two users before sending a notice (#8728)
* Check support room has only two users

* Create 8728.bugfix

* Update synapse/server_notices/server_notices_manager.py

Co-authored-by: Erik Johnston <erik@matrix.org>

Co-authored-by: Erik Johnston <erik@matrix.org>
2020-11-11 14:23:16 +00:00
Andrew Morgan eedaf90c84
Better error message when a remote resource uses invalid Content-Type (#8719) 2020-11-11 14:22:40 +00:00
Nicolai Søborg 4c7587ef99
Catch exceptions in password_providers (#8636)
Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
2020-11-11 13:24:53 +00:00
Marcus Schopen c059413001
Notes on SSO logins and media_repository worker (#8701)
If SSO login is used (e.g. SAML) in a multi worker setup, it should be mentioned that currently all SAML logins must run on the same worker, see https://github.com/matrix-org/synapse/issues/7530

Also, if you are using different ports (for example 443 and 8448) in a reverse proxy for client and federation, the path `/_matrix/media` on the client and federation port must point to the listener of the `media_repository` worker, otherwise you'll get a 404 on the federation port for the path `/_matrix/media`, if a remote server is trying to get the media object on federation port, see https://github.com/matrix-org/synapse/issues/8695
2020-11-06 14:33:07 +00:00
Andrew Morgan 2a6b685294
Add documentation about documentation to CONTRIBUTING.md (#8714)
This PR adds some documentation that:

* Describes who the audience for the `docs/`, `docs/dev/` and `docs/admin/` directories are, as well as Synapse's wiki page.
* Stresses that we'd like all documentation to be down in markdown.
2020-11-06 11:59:22 +00:00
Richard van der Hoff fb56dfdccd
Fix SIGHUP handler (#8697)
Fixes:

```
builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given
```
2020-11-06 11:42:07 +00:00
Dirk Klimpel c3119d1536
Add an admin API for users' media statistics (#8700)
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users.
Related to #6094
It is the first API for statistics.
Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats)

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-11-05 18:59:12 +00:00
Dirk Klimpel e4676bd877
Add `displayname` to Shared-Secret Registration for admins (#8722)
Add `displayname` to Shared-Secret Registration for admins to `POST /_synapse/admin/v1/register`
2020-11-05 13:55:45 +00:00
Andrew Morgan 6abb1ad0be
Consolidate purge table lists to prevent desyncronisation (#8713)
I idly noticed that these lists were out of sync with each other, causing us to miss a table in a test case (`local_invites`). Let's consolidate this list instead to prevent this from happening in the future.
2020-11-04 11:26:05 +00:00
Dirk Klimpel 4fda58ddd2
Remove the "draft" status of the Room Details Admin API (#8702)
Fixes #8550
2020-11-03 12:48:25 +00:00
Erik Johnston 243d427fbc
Block clients from sending server ACLs that lock the local server out. (#8708)
Fixes #4042
2020-11-03 12:13:48 +00:00
Erik Johnston 4b09b7438e
Document how to set up multiple event persisters (#8706) 2020-11-03 10:27:11 +00:00
Matthew Hodgson d04c2d19b3 grammar 2020-11-02 21:22:36 +00:00
Andrew Morgan e89bd3ea92
Improve error messages of non-str displayname/avatar_url (#8705)
This PR fixes two things:

* Corrects the copy/paste error of telling the client their displayname is wrong when they are submitting an `avatar_url`.
* Returns a `M_INVALID_PARAM` instead of `M_UNKNOWN` for non-str type parameters.

Reported by @t3chguy.
2020-11-02 18:01:09 +00:00
David Baker 59cc2472b3
Add base pushrule to notify for jitsi conferences (#8286)
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
2020-11-02 16:36:14 +00:00
Dan Callahan ca39e67f3d
Use Python 3.8 in Docker images by default (#8698)
This bumps us closer to current Python without going all the way to 3.9.

Fixes #8674

Signed-off-by: Dan Callahan <danc@element.io>
2020-11-02 16:33:06 +00:00
Erik Johnston 1eb9de90c0
Improve start time by adding index to e2e_cross_signing_keys (#8694)
We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
2020-11-02 13:55:56 +00:00
Matthew Hodgson 11fd90a2b7 typo 2020-11-02 13:33:56 +00:00
Andrew Morgan 26b46796ea Fix typos in systemd-with-workers doc 2020-11-02 12:56:16 +00:00
Andrew Morgan 305545682d Fix typo in workers doc 2020-11-02 12:36:18 +00:00
Brendan Abolivier 7a0fd6f98d
Fix error handling around when completing an AS transaction (#8693) 2020-10-30 16:50:48 +00:00
Erik Johnston f27a789697 Merge branch 'master' into develop 2020-10-30 16:27:02 +00:00
Erik Johnston b176f1036a Fix changelog 2020-10-30 15:33:33 +00:00
Erik Johnston aef8514193 use correct version 2020-10-30 15:27:58 +00:00
Erik Johnston b4289795ea 1.22.1 2020-10-30 15:25:44 +00:00
Erik Johnston 1b831f2bec Merge branch 'release-v1.22.1' into develop 2020-10-30 15:24:48 +00:00
Richard van der Hoff 90c900a8ff Merge pull request #8678 from matrix-org/rav/fix_frozen_events
Fix serialisation errors when using third-party event rules.
2020-10-30 15:24:10 +00:00
Will Hunt b37aa1643b Tie together matches_user_in_member_list and get_users_in_room caches (#8676)
* Tie together matches_user_in_member_list and get_users_in_room

* changelog

* Remove type to fix mypy

* Add `on_invalidate` to the function signature in the hopes that may make things work well

* Remove **kwargs

* Update 8676.bugfix
2020-10-30 15:23:16 +00:00
Patrick Cloke 8f1aefa694
Improve the sample config for SSO (OIDC, SAML, and CAS). (#8635) 2020-10-30 10:01:59 -04:00
Richard van der Hoff cbc82aa09f
Implement and use an @lru_cache decorator (#8595)
We don't always need the full power of a DeferredCache.
2020-10-30 11:43:17 +00:00
Patrick Cloke fd7c743445
Fail test cases if they fail to await all awaitables (#8690) 2020-10-30 07:15:07 -04:00
Erik Johnston 46f4be94b4
Fix race for concurrent downloads of remote media. (#8682)
Fixes #6755
2020-10-30 10:55:24 +00:00
Andrew Morgan 4504151546
Fix optional parameter in stripped state storage method (#8688)
Missed in #8671.
2020-10-30 00:22:31 +00:00
Erik Johnston ef2d627015
Fix unit tests (#8689)
* Fix unit tests

* Newsfile
2020-10-29 18:21:49 +00:00
Will Hunt 70269fbd18
Tie together matches_user_in_member_list and get_users_in_room caches (#8676)
* Tie together matches_user_in_member_list and get_users_in_room

* changelog

* Remove type to fix mypy

* Add `on_invalidate` to the function signature in the hopes that may make things work well

* Remove **kwargs

* Update 8676.bugfix
2020-10-29 16:58:16 +00:00
Patrick Cloke 8b42a4eefd
Gracefully handle a pending logging connection during shutdown. (#8685) 2020-10-29 12:53:57 -04:00
Erik Johnston f21e24ffc2
Add ability for access tokens to belong to one user but grant access to another user. (#8616)
We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).

A future PR will add an API for creating such a token.

When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29 15:58:44 +00:00
Erik Johnston 22eeb6bc54
Fix cache call signature to accept `on_invalidate`. (#8684)
Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.
2020-10-29 15:18:17 +00:00
Richard van der Hoff 0073fe914a
Use `%r` rather than `%s` for stringifying events (#8679)
otherwise non-state events get written as `<FrozenEvent ... state_key='None'>`
which is indistinguishable from state events with the actual state_key `None`.
2020-10-29 12:16:49 +00:00
Richard van der Hoff 56f0ee78a9
Optimise createRoom with multiple invites (#8559)
By not dropping the membership lock between invites, we can stop joins from
grabbing the lock when we're half-done and slowing the whole thing down.
2020-10-29 11:48:39 +00:00
Patrick Cloke 00b24aa545
Support generating structured logs in addition to standard logs. (#8607)
This modifies the configuration of structured logging to be usable from
the standard Python logging configuration.

This also separates the formatting of logs from the transport allowing
JSON logs to files or standard logs to sockets.
2020-10-29 07:27:37 -04:00
Erik Johnston 9a7e0d2ea6
Don't require hiredis to run unit tests (#8680) 2020-10-29 11:17:35 +00:00
Richard van der Hoff c97da1e45d
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
Fix serialisation errors when using third-party event rules.
2020-10-28 20:41:42 +00:00
Richard van der Hoff e80eb69887 remove unused imports 2020-10-28 16:18:05 +00:00
Richard van der Hoff b6ca69e4f1 Remove frozendict_json_encoder and support frozendicts everywhere
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
2020-10-28 15:56:57 +00:00