6f5c6c8f60
Merge pull request #6195 from matrix-org/erikj/opentracing_preview_url
...
Trace non-JSON APIs, /media, /key etc
2019-10-11 13:23:52 +01:00
be9b55e0d2
cas: support setting display name ( #6114 )
...
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
dfbb62c28d
Port synaps/rest/client/users.py to async/await
2019-10-11 12:21:40 +01:00
f95325e22a
Port synaps/rest/client/server_notice_servlet.py to async/await
2019-10-11 12:21:40 +01:00
281f887090
Port synaps/rest/client/media.py to async/await
2019-10-11 12:21:40 +01:00
fca3a541e7
Port rest/admin/__init__.py to async/await
2019-10-11 12:17:52 +01:00
f3ceaf4323
Trace non-JSON APIs, /media, /key etc
2019-10-11 11:58:52 +01:00
a0d0ba7862
Fix MAU reaping where reserved users are specified. ( #6168 )
2019-10-11 09:38:26 +01:00
7a0dce9259
make sure we actually return something
2019-10-10 20:31:30 -04:00
4908fb3b30
make storage layer in charge of interpreting the device key data
2019-10-10 19:15:30 -04:00
83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any
...
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
b54b1e759a
Fix SQLite take 2
2019-10-10 16:19:40 +01:00
afb6d9d53b
Fix SQLite
2019-10-10 15:55:41 +01:00
3bc687508f
Remove add_in_list_sql_clause
2019-10-10 15:35:46 +01:00
ca3e01e50d
Fix store_url_cache using bytes
2019-10-10 14:52:29 +01:00
9d06fb9cb1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any
2019-10-10 14:38:21 +01:00
8bc529c04d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 14:30:51 +01:00
933034e2fe
Merge pull request #6127 from matrix-org/erikj/patch_inner
...
Add more log context checks when patching inlineCallbacks
2019-10-10 13:47:50 +01:00
fe1c1e6c28
Fixup comments
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-10 13:17:19 +01:00
b161786c14
Replace IN usage with helper funcs
2019-10-10 13:15:49 +01:00
b4fbf71187
Add helper funcs to use postgres ANY
...
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
91f43dca39
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 13:10:57 +01:00
b5b03b7079
Add domain validation when creating room with list of invitees ( #6121 )
2019-10-10 13:05:48 +01:00
9a84d74417
before fulfilling a group invite,check if user is already joined/invited ( #3436 )
...
Fixes vector-im/riot-web#5645
2019-10-10 13:03:44 +01:00
2efd050c9d
send 404 as http-status when filter-id is unknown to the server ( #2380 )
...
This fixed the weirdness of 400 vs 404 as http status code in the case
the filter id is not known by the server.
As e.g. matrix-js-sdk expects 404 to catch this situation this leads
to unwanted behaviour.
2019-10-10 12:59:55 +01:00
59e0ed8306
Fix py3.5
2019-10-10 12:47:07 +01:00
c349e3ebaf
Fix py3.5
2019-10-10 12:29:38 +01:00
f735aeec65
sort
2019-10-10 12:20:29 +01:00
941edad583
Appease mypy
2019-10-10 12:15:17 +01:00
791a8c559b
Add coments
2019-10-10 11:53:57 +01:00
9970f955ce
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-10 11:51:50 +01:00
a139420a3c
Fix races in room stats (and other) updates. ( #6187 )
...
Hopefully this will fix the occasional failures we were seeing in the room directory.
The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.
We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
562b4e51dd
Rewrite the user_filter migration again ( #6184 )
...
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
128d5948c4
Fix packaging
2019-10-10 11:16:26 +01:00
ec0596f2ab
Log correct context
2019-10-10 11:11:38 +01:00
3e4272961a
Test for sentinel commit
2019-10-10 10:58:32 +01:00
1d6dd1c294
Move patch_inline_callbacks into synapse/
2019-10-10 10:53:06 +01:00
da815c1f69
Move tag/push rules room upgrade checking ealier ( #6155 )
...
It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers.
The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).
2019-10-10 10:06:45 +01:00
f743108a94
Refactor HomeserverConfig so it can be typechecked ( #6137 )
2019-10-10 09:39:35 +01:00
4535a07f4a
make version optional in body of e2e backup version update
...
to agree with latest version of the MSC
2019-10-09 17:54:03 -04:00
5c1f886c75
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-09 16:52:21 +01:00
7f18b3d526
Do the update as a background index
2019-10-09 16:03:24 +01:00
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
...
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
59d6290ed9
Merge pull request #6178 from matrix-org/babolivier/factor_out_bg_updates
...
Factor out backgroung updates
2019-10-09 12:29:01 +01:00
1d3858371e
Disable bytes usage with postgres
...
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
e7631d84e6
Fix existing hex encoded json values in DB
2019-10-08 16:18:43 +01:00
ced4784592
Fix inserting bytes as text
2019-10-08 16:18:43 +01:00
c69324ffb5
Fix RegistrationStore
2019-10-08 14:48:33 +01:00
b1c0a4ceb3
Cleanup client_ips
2019-10-08 14:38:14 +01:00
8f1b385acc
Don't end up with 4 classes in registration
2019-10-08 14:36:33 +01:00
474abf1eb6
add M_TOO_LARGE error code for uploading a too large file ( #6151 )
...
Fixes #6109
2019-10-08 13:55:16 +01:00
ea7d938bca
Remove unused public room list timeout param ( #6179 )
...
* Remove unused public room list timeout param
* Add changelog
2019-10-08 13:51:25 +01:00
b94a401852
Fix /federation/v1/state for recent room versions ( #6170 )
...
* Fix /federation/v1/state for recent room versions
Turns out this endpoint was completely broken for v3 rooms. Hopefully this
re-signing code is irrelevant nowadays anyway.
2019-10-08 09:35:37 +01:00
66ebea1723
Lint
2019-10-07 17:44:41 +01:00
0496eafbf4
Move roommember's bg updates to a dedicated store
2019-10-07 17:35:01 +01:00
e106a0e4db
Move user_directory's bg updates to a dedicated store
2019-10-07 17:34:45 +01:00
cfccd2d78a
Move state's bg updates to a dedicated store
2019-10-07 17:34:39 +01:00
841054ad96
Move search's bg updates to a dedicated store
2019-10-07 17:34:35 +01:00
81e6ffb536
Move registration's bg updates to a dedicated store
2019-10-07 17:34:29 +01:00
54f87e0734
Move media_repository's bg updates to a dedicated store
2019-10-07 17:34:26 +01:00
cef9f6753e
Move devices's bg updates to a dedicated store
2019-10-07 17:34:20 +01:00
2d3b4f42f0
Move deviceinbox's bg updates to a dedicated store
2019-10-07 17:34:16 +01:00
88957199e7
Move client_ips's bg updates to a dedicated store
2019-10-07 17:34:12 +01:00
2cb7466993
Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure
...
Don't regenerate numeric user ID if registration fails.
2019-10-07 16:56:23 +01:00
dc795ba709
Log responder we are using. ( #6139 )
...
This prevents us logging "Responding to media request with responder %s".
2019-10-07 15:41:25 +01:00
276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test ( #6167 )
2019-10-07 14:41:39 +01:00
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
...
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
c8e6c308c6
Fix unique_user_filter_index schema update
2019-10-07 13:15:35 +01:00
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships
...
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
5119a4cac7
Fix bug where we didn't pull out event ID
2019-10-07 12:21:17 +01:00
ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
...
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
21d51ab598
Typo
2019-10-04 11:21:24 +01:00
4676732ca0
Lint
2019-10-04 11:18:28 +01:00
81d51ce48b
Incorporate review
2019-10-04 11:16:19 +01:00
13c4345c84
Update `user_filters` table to have a unique index, and non-null columns ( #1172 )
2019-10-04 10:34:16 +01:00
39b40d6d99
media/thumbnailer: Better quality for 1-bit / 8-bit color palette images ( #2142 )
...
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
6511071837
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships
2019-10-03 17:51:06 +01:00
66537e10ce
add some metrics on the federation sender ( #6160 )
2019-10-03 17:47:20 +01:00
ab8a64772b
Remove unused variable
2019-10-03 17:42:32 +01:00
693156aaf4
Don't regenerate numeric user ID if registration fails.
...
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
91f61fc6d7
Use the right error....
2019-10-03 17:28:31 +01:00
84691da6c3
pep8
2019-10-03 17:27:18 +01:00
a9610cdf02
Fixup names and comments
2019-10-03 17:26:56 +01:00
d89ebf7c25
cachedList descriptor doesn't like typing
2019-10-03 17:23:11 +01:00
c8145af8a9
Cache room membership lookups in _get_joined_users_from_context
2019-10-03 17:11:04 +01:00
6018bec919
Merge branch 'master' into develop
2019-10-03 13:48:45 +01:00
ecb69d824a
1.4.0
2019-10-03 13:22:44 +01:00
0f46bf5737
Replace client_secret with <redacted> in server logs ( #6158 )
...
Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
2019-10-03 12:57:26 +01:00
6527fa18c1
Add test case
2019-10-03 11:24:36 +01:00
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
...
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
...
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
...
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
7a5f080f91
Fix appservice room list pagination
2019-10-02 15:47:22 +01:00
4c4f44930d
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 15:20:36 +01:00
03cf4385e0
Fix public room list pagination.
...
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
a5166e4d5f
Land improved room list based on room stats ( #6019 )
...
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
aec1377d0b
1.4.0rc2
2019-10-02 13:55:00 +01:00
864f144543
Fix up some typechecking ( #6150 )
...
* type checking fixes
* changelog
2019-10-02 05:29:01 -07:00
972c9f65d7
Lint
2019-10-02 12:17:46 +01:00
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade ( #6144 )
...
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.
For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.
Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
d69fd53f74
Bound find_next_generated_user_id DB query.
...
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
5705ecaec6
Don't 500 code when trying to exchange a revoked 3PID invite
...
While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state.
When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events.
When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it).
This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
2019-10-02 11:19:43 +01:00
ecd254bc49
Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop
2019-10-02 11:08:07 +01:00
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
...
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions
...
Fix fetching censored redactions from DB
2019-10-02 10:57:30 +01:00
f44f1d2e83
Fix errors storing large retry intervals.
...
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.
To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
ce7a3e7e27
Fix fetching censored redactions from DB
...
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
5e8387af9e
Use `received_ts` to find uncensored redacted events
...
Joining against `events` and ordering by `stream_ordering` is
inefficient as it forced scanning the entirety of the redactions table.
This isn't the case if we use `redactions.received_ts` column as we can
then use an index.
2019-10-01 13:43:48 +01:00
898dde981b
Add received_ts column to redactions.
...
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
a27fb7d5ca
Don't repeatedly attempt to censor events we don't have.
...
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.
When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
1d349fb159
Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop
2019-10-01 10:17:24 +01:00
9267741a5f
Fix `devices_last_seen` background update.
...
Fixes #6134 .
2019-09-30 11:58:36 +01:00
16cb9a71b8
Drop unused tables ( #6115 )
...
These tables are unused since #5893 (as amended by #6047 ), so we can now drop
them.
Fixes #6048 .
2019-09-30 09:38:41 +01:00
f3451118a6
Edit SimpleHttpClient to reference that header keys can be passed as str or bytes ( #6077 )
2019-09-27 17:59:18 +01:00
3e42d47a5a
Incorporate review
2019-09-27 16:15:01 +01:00
af92110c46
Update synapse/handlers/deactivate_account.py
...
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:12:15 +01:00
bbe2a0f339
Update synapse/handlers/deactivate_account.py
...
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:36 +01:00
873fe7883c
Lint
2019-09-27 15:21:03 +01:00
72a2708ac6
Fixup and add some logging
2019-09-27 15:13:39 +01:00
132279a46f
Patch inlinecallbacks for log contexts
2019-09-27 15:11:14 +01:00
5257a2fb1c
Reject pending invites on deactivation
2019-09-27 14:49:53 +01:00
8c27bc8b60
Move lookup-related functions from RoomMemberHandler to IdentityHandler ( #5978 )
...
Just to have all the methods that make calls to identity services in one place.
2019-09-27 10:36:20 +01:00
67ee18daea
Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span
...
Add some helpful opentracing tags and remove get_user_by_req span
2019-09-27 10:02:57 +01:00
3423633d50
Fix 'redaction_retention_period' sampel config to match guidelines
2019-09-26 16:43:52 +01:00
54569c787b
Kill off half-implemented password-reset via sms ( #6101 )
...
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.
See also matrix-org/matrix-doc#2303 .
2019-09-26 15:38:25 +01:00
8b8f8c7b3c
Explicitly log when a homeserver does not have a trusted key server configured ( #6090 )
2019-09-26 12:57:01 +01:00
2927c6bc4c
bump version
2019-09-26 12:29:59 +01:00
034db2ba21
Fix dummy event insertion consent bug ( #6053 )
...
Fixes #5905
2019-09-26 11:47:53 +01:00
a96318127d
Update comments and docstring
2019-09-25 18:17:39 +01:00
4fb3c129aa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2
2019-09-25 17:53:13 +01:00
9614d3c9d1
Merge pull request #6089 from matrix-org/erikj/cleanup_user_ips
...
Move last seen info into devices table
2019-09-25 17:42:39 +01:00
a4f3ca48b5
Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ( #5884 )
2019-09-25 17:27:35 +01:00
39b50ad42a
Review comments
2019-09-25 17:22:33 +01:00
d2bd0bc6b1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips
2019-09-25 17:16:28 +01:00
50572db837
Use if `is not None`
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
77dc7093a7
Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. ( #6104 )
...
Fixes #6103
2019-09-25 12:29:35 +01:00
990928abde
Stop advertising unsupported flows for registration ( #6107 )
...
If email or msisdn verification aren't supported, let's stop advertising them
for registration.
Fixes #6100 .
2019-09-25 12:10:26 +01:00
dc2c97e1a3
isort
2019-09-25 11:59:05 +01:00
dc01cad690
Add device and appservice tags
2019-09-25 11:59:00 +01:00
5d99713854
Add tags for event_id and txn_id in event sending
...
This will make it easier to search for sending event requests.
2019-09-25 11:39:15 +01:00
2cd98812ba
Refactor the user-interactive auth handling ( #6105 )
...
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.
This is also preparation for some more intelligent advertising of flows, as per #6100
2019-09-25 11:33:03 +01:00
fde4ce2213
Don't create new span for get_user_by_req
...
We don't actually care about what happens in `get_user_by_req` and
having it as a separate span means that the entity tag isn't added to
the servlet spans, making it harder to search.
2019-09-25 11:32:41 +01:00
8004d6ca2f
Refactor code for calculating registration flows ( #6106 )
...
because, frankly, it looked like it was written by an axe-murderer.
This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
2019-09-25 11:32:05 +01:00
c3635c9459
make isort happy
2019-09-24 16:21:03 -04:00
f4b6d43ec3
add some comments
2019-09-24 16:19:54 -04:00
39864f45ec
drop some logger lines to debug
2019-09-24 15:26:45 -04:00
26113fb7de
make changes based on PR feedback
2019-09-24 14:12:20 -04:00
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
...
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
9b7c4f4078
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 17:03:50 +01:00
566ac40939
remove unused parameter to get_user_id_by_threepid ( #6099 )
...
Added in #5377 , apparently in error
2019-09-24 17:01:09 +01:00
Erik Johnston