Commit Graph

156 Commits (e90fad5cbacebcf00f90e9432089533008ccc48b)

Author SHA1 Message Date
Andrew Morgan a699c044b6
Abstract code for stripping room state into a separate method (#8671)
This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change.

For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included.

The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event.

This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem.

In essence this is just moving some inline code to a reusable store method.
2020-10-27 18:42:46 +00:00
Dirk Klimpel 9b7c28283a
Add admin API to list users' local media (#8647)
Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
2020-10-27 14:12:31 +00:00
Patrick Cloke 10f45d85bb
Add type hints for account validity handler (#8620)
This also fixes a bug by fixing handling of an account which doesn't expire.
2020-10-26 14:17:31 -04:00
Dirk Klimpel 66e6801c3e
Split admin API for reported events into a detail and a list view (#8539)
Split admin API for reported events in detail und list view.
API was introduced with #8217 in synapse v.1.21.0.

It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview.
The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`.
It is similar to room and users API.

It is a kind of regression in `GET /_synapse/admin/v1/event_reports`.  `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec).

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-10-26 18:16:37 +00:00
Dirk Klimpel 49d72dea2a
Add an admin api to delete local media. (#8519)
Related to: #6459, #3479

Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete
a single file from server.
2020-10-26 17:02:28 +00:00
Erik Johnston ff7f0e8a14 Merge branch 'release-v1.22.0' into develop 2020-10-26 15:02:55 +00:00
Will Hunt 9e0f5a0ac4 Fix get|set_type_stream_id_for_appservice store functions (#8648) 2020-10-26 15:00:56 +00:00
Will Hunt e8dbbcb64c
Fix get|set_type_stream_id_for_appservice store functions (#8648) 2020-10-26 10:51:33 -04:00
Erik Johnston 437a99fb99
Fix user_daily_visits to not have duplicate rows for UA. (#8654)
* Fix user_daily_visits to not have duplicate rows for UA.

Fixes #8641.

* Newsfile

* Fix typo.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-26 13:16:32 +00:00
Erik Johnston a9f90fa73a
Type hints for RegistrationStore (#8615) 2020-10-22 11:56:58 +01:00
Patrick Cloke de5cafe980
Add type hints to profile and base handlers. (#8609) 2020-10-21 06:44:31 -04:00
Patrick Cloke 9e0f22874f
Consistently use wrap_as_background_task in more places (#8599) 2020-10-20 11:29:38 -04:00
Will Hunt 626b8f0846
Move schema file for as_device_stream (#8590)
* Move schema file

* Add a .

* Add matching changelog entry

* Fix sqlite
2020-10-20 10:18:55 +01:00
Vasilis Gerakaris 34c20493b9
Drop unused `device_max_stream_id` table (#8589)
Signed-off-by: Vasilis Gerakaris <vasilis.gerakaris@navarino.gr>
2020-10-19 19:06:54 +01:00
Richard van der Hoff 903d11c43a
Add `DeferredCache.get_immediate` method (#8568)
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
2020-10-19 15:00:12 +01:00
Richard van der Hoff 97647b33c2
Replace DeferredCache with LruCache where possible (#8563)
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
2020-10-19 12:20:29 +01:00
Patrick Cloke 1b70662be9
Clean-up old transaction IDs on the background worker. (#8544) 2020-10-16 12:06:17 -04:00
Will Hunt c276bd9969
Send some ephemeral events to appservices (#8437)
Optionally sends typing, presence, and read receipt information to appservices.
2020-10-15 12:33:28 -04:00
Richard van der Hoff 0a08cd1065
Merge pull request #8548 from matrix-org/rav/deferred_cache
Rename Cache to DeferredCache, and related changes
2020-10-15 11:42:07 +01:00
Neil Johnson 1f39155071
Include user agent in user daily visits table (#8503)
Include user agent in user daily visits table.
2020-10-15 10:36:40 +01:00
Richard van der Hoff 4182bb812f move DeferredCache into its own module 2020-10-14 23:38:14 +01:00
Richard van der Hoff 9f87da0a84 Rename Cache->DeferredCache 2020-10-14 23:38:14 +01:00
Erik Johnston 19b15d63e8
Use autocommit mode for single statement DB functions. (#8542)
Autocommit means that we don't wrap the functions in transactions, and instead get executed directly. Introduced in #8456. This will help:

1. reduce the number of `could not serialize access due to concurrent delete` errors that we see (though there are a few functions that often cause serialization errors that we don't fix here);
2. improve the DB performance, as it no longer needs to deal with the overhead of `REPEATABLE READ` isolation levels; and
3. improve wall clock speed of these functions, as we no longer need to send `BEGIN` and `COMMIT` to the DB.

Some notes about the differences between autocommit mode and our default `REPEATABLE READ` transactions:

1. Currently `autocommit` only applies when using PostgreSQL, and is ignored when using SQLite (due to silliness with [Twisted DB classes](https://twistedmatrix.com/trac/ticket/9998)).
2. Autocommit functions may get retried on error, which means they can get applied *twice* (or more) to the DB (since they are not in a transaction the previous call would not get rolled back). This means that the functions need to be idempotent (or otherwise not care about being called multiple times). Read queries, simple deletes, and updates/upserts that replace rows (rather than generating new values from existing rows) are all idempotent.
3. Autocommit functions no longer get executed in [`REPEATABLE READ`](https://www.postgresql.org/docs/current/transaction-iso.html) isolation level, and so data can change queries, which is fine for single statement queries.
2020-10-14 15:50:59 +01:00
Brendan Abolivier 3ee97a2748
Make sure a retention policy is a state event (#8527)
* Make sure a retention policy is a state event

* Changelog
2020-10-14 12:00:52 +01:00
Patrick Cloke 629a951b49
Move additional tasks to the background worker, part 4 (#8513) 2020-10-13 08:20:32 -04:00
Erik Johnston b2486f6656
Fix message duplication if something goes wrong after persisting the event (#8476)
Should fix #3365.
2020-10-13 12:07:56 +01:00
Erik Johnston 5009ffcaa4
Only send RDATA for instance local events. (#8496)
When pulling events out of the DB to send over replication we were not
filtering by instance name, and so we were sending events for other
instances.
2020-10-09 13:10:33 +01:00
Patrick Cloke fe0f4a3591
Move additional tasks to the background worker, part 3 (#8489) 2020-10-09 07:37:51 -04:00
Patrick Cloke a93f3121f8
Add type hints to some handlers (#8505) 2020-10-09 07:20:51 -04:00
Hubert Chathi a97cec18bb
Invalidate the cache when an olm fallback key is uploaded (#8501) 2020-10-08 13:24:46 -04:00
Patrick Cloke e4f72ddc44
Move additional tasks to the background worker (#8458) 2020-10-07 11:27:56 -04:00
Erik Johnston 52a50e8686
Use vector clocks for room stream tokens. (#8439)
Currently when using multiple event persisters we (in the worst case) don't tell clients about events until all event persisters have persisted new events after the original event. This is a suboptimal, especially if one of the event persisters goes down.

To handle this, we encode the position of each event persister in the room tokens so that we can send events to clients immediately. To reduce the size of the token we do two things:

1. We create a unique immutable persistent mapping between instance names and a generated small integer ID, which we can encode in the tokens instead of the instance name; and
2. We encode the "persisted upto position" of the room token and then only explicitly include instances that have positions strictly greater than that.

The new tokens look something like: `m3478~1.3488~2.3489`, where the first number is the min position, and the subsequent `-` separated pairs are the instance ID to positions map. (We use `.` and `~` as separators as they're URL safe and not already used by `StreamToken`).
2020-10-07 15:15:33 +01:00
Patrick Cloke b460a088c6
Add typing information to the device handler. (#8407) 2020-10-07 08:58:21 -04:00
Hubert Chathi 4cb44a1585
Add support for MSC2697: Dehydrated devices (#8380)
This allows a user to store an offline device on the server and
then restore it at a subsequent login.
2020-10-07 08:00:17 -04:00
Hubert Chathi 3cd78bbe9e
Add support for MSC2732: olm fallback keys (#8312) 2020-10-06 13:26:29 -04:00
Richard van der Hoff f31f8e6319
Remove stream ordering from Metadata dict (#8452)
There's no need for it to be in the dict as well as the events table. Instead,
we store it in a separate attribute in the EventInternalMetadata object, and
populate that on load.

This means that we can rely on it being correctly populated for any event which
has been persited to the database.
2020-10-05 14:43:14 +01:00
Patrick Cloke c5251c6fbd
Do not assume that account data is of the correct form. (#8454)
This fixes a bug where `m.ignored_user_list` was assumed to be a dict,
leading to odd behavior for users who set it to something else.
2020-10-05 09:28:05 -04:00
Erik Johnston e3debf9682
Add logging on startup/shutdown (#8448)
This is so we can tell what is going on when things are taking a while to start up.

The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
2020-10-02 15:20:45 +01:00
Erik Johnston ec10bdd32b
Speed up unit tests when using PostgreSQL (#8450) 2020-10-02 15:09:31 +01:00
Patrick Cloke 62894673e6
Allow background tasks to be run on a separate worker. (#8369) 2020-10-02 08:23:15 -04:00
Patrick Cloke 4ff0201e62
Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
Erik Johnston 7941372ec8
Make token serializing/deserializing async (#8427)
The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
2020-09-30 20:29:19 +01:00
Richard van der Hoff 20e7c4de26 Add an improved "forward extremities" metric
Hopefully, N(extremities) * N(state_events) is a more realistic approximation
to "how big a problem is this room?".
2020-09-30 16:49:15 +01:00
Richard van der Hoff 6d2d42f8fb Rewrite BucketCollector
This was a bit unweildy for what I wanted: in particular, I wanted to assign
each measurement straight into a bucket, rather than storing an intermediate
Counter which didn't do any bucketing at all.

I've replaced it with something that is hopefully a bit easier to use.

(I'm not entirely sure what the difference between a HistogramMetricFamily and
a GaugeHistogramMetricFamily is, but given our counters can go down as well as
up the latter *sounds* more accurate?)
2020-09-30 16:49:15 +01:00
Erik Johnston ea70f1c362
Various clean ups to room stream tokens. (#8423) 2020-09-29 21:48:33 +01:00
Richard van der Hoff 2649d545a5
Mypy fixes for `synapse.handlers.federation` (#8422)
For some reason, an apparently unrelated PR upset mypy about this module. Here are a number of little fixes.
2020-09-29 15:57:36 +01:00
Will Hunt 8676d8ab2e
Filter out appservices from mau count (#8404)
This is an attempt to fix #8403.
2020-09-29 13:11:02 +01:00
Erik Johnston bd380d942f
Add checks for postgres sequence consistency (#8402) 2020-09-28 18:00:30 +01:00
Matthew Hodgson 4b3a1faa08 typo 2020-09-28 00:23:35 +01:00
Tdxdxoz abd04b6af0
Allow existing users to login via OpenID Connect. (#8345)
Co-authored-by: Benjamin Koch <bbbsnowball@gmail.com>

This adds configuration flags that will match a user to pre-existing users
when logging in via OpenID Connect. This is useful when switching to
an existing SSO system.
2020-09-25 07:01:45 -04:00