499d4a32cd
Add metrics for len of new extremities persisted.
...
Of new events being persisted add metrics for total size of forward
extremities and number of unchanged, "stale" extremities.
2019-06-17 17:02:48 +01:00
eba7caf09f
Remove Postgres 9.4 support ( #5448 )
2019-06-18 00:59:00 +10:00
9ca4ae7131
Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive
...
Prometheus histograms are cumalative
2019-06-14 18:21:42 +01:00
e0b77b004d
Fix background job for deactivated flag
2019-06-14 16:00:45 +01:00
6d56a694f4
Don't send renewal emails to deactivated users
2019-06-14 15:05:56 +01:00
3ed595e327
Prometheus histograms are cumalative
2019-06-14 14:07:32 +01:00
d0530382ee
Track deactivated accounts in the database ( #5378 )
2019-06-14 13:18:24 +01:00
6312d6cc7c
Expose statistics on extrems to prometheus ( #5384 )
2019-06-13 22:40:52 +10:00
5bc9484537
Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop
2019-06-11 10:37:43 +01:00
94dac0f3e5
add monthly active users to phonehome stats ( #5252 )
...
* add monthly active users to phonehome stats
2019-06-10 23:33:59 +01:00
49e01e5710
Fix defaults on checking threepids
2019-06-10 23:09:31 +01:00
c413540fb9
Fix bug sending federation transactions with lots of EDUs
...
If we try and send a transaction with lots of EDUs and we run out of
space, we call get_new_device_msgs_for_remote with a limit of 0, which
then failed.
2019-06-10 16:22:45 +01:00
43badd2cd4
Fix key verification when key stored with null valid_until_ms
...
Some keys are stored in the synapse database with a null valid_until_ms
which caused an exception to be thrown when using that key. We fix this
by treating nulls as zeroes, i.e. they keys will match verification
requests with a minimum_valid_until_ms of zero (i.e. don't validate ts)
but will not match requests with a non-zero minimum_valid_until_ms.
Fixes #5391 .
2019-06-10 15:37:10 +01:00
2d1d7b7e6f
Prevent multiple device list updates from breaking a batch send ( #5156 )
...
fixes #5153
2019-06-06 23:54:00 +01:00
1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
...
Make a full SQL schema
2019-06-06 18:35:38 +01:00
ed872db8df
fix maybe
2019-06-07 02:53:47 +10:00
3719680ee4
Add ability to perform password reset via email without trusting the identity server ( #5377 )
...
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.
This PR is a culmination of 3 smaller PRs which have each been separately reviewed:
* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
837e32ef55
just user dir?
2019-06-07 01:49:25 +10:00
3b6645d3bf
remove background updates that arent needed
2019-06-07 01:20:58 +10:00
89d3d7b2c0
Merge pull request #5221 from matrix-org/erikj/fix_worker_sytest
...
Fix get_max_topological_token to never return None
2019-06-06 13:54:46 +01:00
7f08a3523a
Better words
2019-06-06 11:09:38 +01:00
6362e3af14
add more comments
2019-06-06 04:20:35 +10:00
75538813fc
Fix background updates to handle redactions/rejections ( #5352 )
...
* Fix background updates to handle redactions/rejections
In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
0a56966f7d
Fix
2019-06-03 17:42:52 +01:00
fa4b54aca5
Ignore room state with null bytes in for room stats
2019-06-03 17:06:54 +01:00
2198b7ce2a
add stuff in bg updates
2019-06-04 01:06:00 +10:00
4e75c5e02a
WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING
2019-06-03 22:42:12 +10:00
be452fc9ac
more fix
2019-06-03 22:24:23 +10:00
7f81b967ca
fix schemas
2019-06-03 22:23:40 +10:00
dc72b90cd6
full schema
2019-06-03 22:03:28 +10:00
37057d5d60
prepare
2019-06-03 22:02:47 +10:00
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
...
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
4d794dae21
Move delta from +10% to -10%
2019-05-31 11:09:39 +01:00
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
...
Speed up room stats background update
2019-05-31 10:48:51 +01:00
5037326d66
Add indices. Remove room_ids accidentally added
...
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.
We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
847b9dcd1c
Make max_delta equal to period * 10%
2019-05-31 09:54:46 +01:00
5ac75fc9a2
Join against events to use its room_id index
2019-05-30 15:26:55 +01:00
e2c46ed851
Move deletion from table inside txn
2019-05-30 15:26:38 +01:00
04710cc2d7
Fetch membership counts all at once
2019-05-30 15:25:41 +01:00
54d50fbfdf
Get events all at once
2019-05-30 15:15:13 +01:00
6cdfb0207e
Add index to temp table
2019-05-30 15:02:31 +01:00
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities
...
Add DB bg update to cleanup extremities.
2019-05-30 14:28:26 +01:00
cb967e2346
Update synapse/storage/events_bg_updates.py
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-30 14:06:42 +01:00
468bd090ff
Rename constant
2019-05-30 11:24:42 +01:00
5c1ece0ffc
Move event background updates to a separate file
2019-05-30 11:22:59 +01:00
640fcbb07f
Fixup comments and logging
2019-05-30 10:55:55 +01:00
46c8f7a517
Implement the SHHS complexity API ( #5216 )
2019-05-30 01:47:16 +10:00
7e8e683754
Log actual number of entries deleted
2019-05-29 15:11:28 +01:00
d79c9994f4
Add DB bg update to cleanup extremities.
...
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
58c8ed5b0d
Correctly filter out extremities with soft failed prevs ( #5274 )
...
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.
Fixes #5269
2019-05-29 11:56:24 +01:00
52839886d6
Allow configuring a range for the account validity startup job
...
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
2019-05-28 16:52:45 +01:00
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
...
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
ba17de7fbc
Fix schema update for account validity
2019-05-28 10:11:38 +01:00
bc4b2ecf70
Fix logging for room stats background update
2019-05-25 12:02:48 -06:00
b75537beaf
Store key validity time in the storage layer
...
This is a first step to checking that the key is valid at the required moment.
The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
2e052110ee
Rewrite store_server_verify_key to store several keys at once ( #5234 )
...
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
85d1e03b9d
Simplifications and comments in do_auth ( #5227 )
...
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
4a30e4acb4
Room Statistics ( #4338 )
2019-05-21 11:36:50 -05:00
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender
...
Only count aggregations from distinct senders
2019-05-21 17:10:48 +01:00
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
...
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. ( #5146 )
...
When considering the candidates to be forward-extremities, we must exclude soft
failures.
Hopefully fixes #5090 .
2019-05-21 16:10:54 +01:00
7b0e804a4a
Fix get_max_topological_token to never return None
2019-05-21 15:21:43 +01:00
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
...
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
5ceee46c6b
Do the select and insert in a single transaction
2019-05-21 13:38:51 +01:00
c7ec06e8a6
Block attempts to annotate the same event twice
2019-05-20 17:39:05 +01:00
1dff859d6a
Rename relation types to match MSC
2019-05-20 14:31:19 +01:00
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
...
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
935af0da38
Correctly update aggregation counts after redaction
2019-05-20 12:09:27 +01:00
ad5b4074e1
Add startup background job for account validity
...
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
2019-05-17 19:37:31 +01:00
b63cc325a9
Only count aggregations from distinct senders
...
As a user isn't allowed to send a single emoji more than once.
2019-05-17 18:03:10 +01:00
8dd9cca8ea
Spelling and clarifications
2019-05-17 16:40:51 +01:00
5dbff34509
Fixup bsaed on review comments
2019-05-17 15:48:04 +01:00
d46aab3fa8
Add basic editing support
2019-05-16 16:54:45 +01:00
895179a4dc
Update docstring
2019-05-16 16:41:05 +01:00
7a7eba8302
Move parsing of tokens out of storage layer
2019-05-16 14:26:23 +01:00
2c662ddde4
Indirect tuple conversion
2019-05-16 14:21:39 +01:00
b5c62c6b26
Fix relations in worker mode
2019-05-16 10:38:13 +01:00
33453419b0
Add cache to relations
2019-05-16 10:02:14 +01:00
a0603523d2
Add aggregations API
2019-05-16 09:37:20 +01:00
5be34fc3e3
Actually check for None rather falsey
2019-05-15 17:30:23 +01:00
b50641e357
Add simple pagination API
2019-05-15 13:36:51 +01:00
efe3c7977a
Add simple send_relation API and track in DB
2019-05-15 13:36:51 +01:00
54d77107c1
Make generating SQL bounds for pagination generic
...
This will allow us to reuse the same structure when we paginate e.g.
relations
2019-05-15 11:30:05 +01:00
52ddc6c0ed
Update docstring with correct type
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-15 09:52:15 +01:00
4fb44fb5b9
Expose DataStore._get_events as get_events_as_list
...
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
11ea16777f
Limit the number of EDUs in transactions to 100 as expected by receiver ( #5138 )
...
Fixes #3951 .
2019-05-09 11:01:41 +01:00
bd0d45ca69
Fix infinite loop in presence handler
...
Fixes #5102
2019-04-26 11:14:49 +01:00
ca90336a69
Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration
2019-04-17 19:44:40 +01:00
eaf41a943b
Add management endpoints for account validity
2019-04-17 19:34:45 +01:00
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
...
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
20f0617e87
Send out emails with links to extend an account's validity period
2019-04-17 14:42:20 +01:00
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta
...
Fix schema upgrade when dropping tables
2019-04-10 10:22:35 +01:00
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
...
Add time-based account expiration
2019-04-09 17:02:41 +01:00
747aa9f8ca
Add account expiration feature
2019-04-09 16:46:04 +01:00
50d2a3059d
Fix schema upgrade when dropping tables
...
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.
Introduced in #4992 .
2019-04-09 14:39:18 +01:00
18b69be00f
Rewrite Datastore.get_server_verify_keys
...
Rewrite this so that it doesn't hammer the database.
2019-04-09 00:00:10 +01:00
3352baac4b
Remove unused server_tls_certificates functions ( #5028 )
...
These have been unused since #4120 , and with the demise of perspectives, it is
unlikely that they will ever be used again.
2019-04-08 21:50:18 +01:00
2d951686a7
drop tables listed in #1830 ( #4992 )
...
Tables dropped:
* application_services,
* application_services_regex,
* transaction_id_to_pdu,
* stats_reporting
* current_state_resets
* event_content_hashes
* event_destinations
* event_edge_hashes
* event_signatures
* feedback
* room_hosts
* state_forward_extremities
2019-04-08 15:37:26 +01:00
7fc1e17f4c
Merge pull request #5001 from matrix-org/rav/keyring_cleanups
...
Cleanups in the Keyring
2019-04-08 12:47:09 +01:00
b43d9a920b
Fix docstring on get_server_keys_json
2019-04-04 18:54:03 +01:00
a33a5abc4c
Clean up the database pagination code ( #5007 )
...
* rewrite & simplify
* changelog
* cleanup potential sql injection
2019-04-05 00:21:16 +11:00
4a2e13631d
Add functions to delete a group
2019-04-03 16:29:52 +01:00
4a4d5c4fd6
Fix grammar and document get_current_users_in_room ( #4998 )
2019-04-03 14:32:20 +01:00
e8419554ff
Remove presence lists ( #4989 )
...
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
8f549c1177
Merge pull request #4982 from matrix-org/erikj/msc1915
...
Implement MSC1915 - 3PID unbind APIs
2019-04-03 11:07:09 +01:00
7efd1d87c2
Run black on the rest of the storage module ( #4996 )
2019-04-03 10:07:29 +01:00
5bec8d660d
Make starting pushers faster during start up
...
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.
Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
2019-04-02 16:59:13 +01:00
297bf2547e
Fix sync bug when accepting invites ( #4956 )
...
Hopefully this time we really will fix #4422 .
We need to make sure that the cache on
`get_rooms_for_user_with_stream_ordering` is invalidated *before* the
SyncHandler is notified for the new events, and we can now do so reliably via
the `events` stream.
2019-04-02 12:42:39 +01:00
c75e2017f1
Fixup docstrings
2019-04-02 11:20:06 +01:00
3715c124b3
Grandfather in existing user threepids
...
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
2019-04-01 15:25:19 +01:00
1666c0696a
Track IS used to bind 3PIDs
...
This will then be used to know which IS to default to when unbinding the
threepid.
2019-04-01 15:23:01 +01:00
54a87a7b08
Collect room-version variations into one place ( #4969 )
...
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
902cdc63b6
Merge pull request #4955 from matrix-org/rav/merge_state_into_events
...
Combine the CurrentStateDeltaStream into the EventStream
2019-03-28 18:32:13 +00:00
2e060774ad
Run `black` on some storage modules that the stats branch touches ( #4959 )
2019-03-29 00:37:16 +11:00
4b91c313a9
Combine the CurrentStateDeltaStream into the EventStream
2019-03-27 22:07:05 +00:00
903f04c21f
Use the state event amount for userdir import batching, not room count ( #4944 )
2019-03-27 02:49:28 +11:00
9bde730ef8
Fix bug where read-receipts lost their timestamps ( #4927 )
...
Make sure that they are sent correctly over the replication stream.
Fixes : #4898
2019-03-25 16:38:05 +00:00
ac396a0d32
Refactor out state delta handling into its own class ( #4917 )
2019-03-25 20:37:08 +11:00
3ecec5ede2
Fix upsert
2019-03-21 10:21:15 +00:00
8d8834d3e7
comment block_room
2019-03-20 17:49:56 +00:00
67d618e111
Allow blocking a room multiple times
2019-03-20 16:50:05 +00:00
4d53017432
Batching in the user directory import ( #4900 )
2019-03-21 03:06:36 +11:00
e9eeca1314
Fix user directory background update ( #4887 )
2019-03-19 22:13:53 +11:00
282c97327f
Migrate the user directory initial population to a background task ( #4864 )
2019-03-19 04:50:24 +11:00
9073cfc8bd
Merge pull request #4846 from matrix-org/hawkowl/userdir-search
...
Improve searching in the userdir
2019-03-14 13:54:15 +00:00
d42c81d724
Transfer local user's push rules on room upgrade ( #4838 )
...
Transfer push rules (notifications) on room upgrade
2019-03-12 14:42:53 +00:00
c0332d095f
fixup
2019-03-13 01:30:54 +11:00
797b6a63fc
fixup
2019-03-13 01:17:51 +11:00
81d9d1bee6
fixup
2019-03-12 22:28:48 +11:00
6f5890b2fa
fixup
2019-03-12 22:27:56 +11:00
10480c4348
fixup
2019-03-12 21:47:14 +11:00
8b618041ef
fixup
2019-03-12 18:06:28 +11:00
c980c7e31f
use the old method
2019-03-12 17:51:14 +11:00
78a6b950b3
fix
2019-03-12 00:50:28 +11:00
1b77bd69fb
pep8
2019-03-12 00:39:12 +11:00
5ba8ceab4c
fixes
2019-03-12 00:35:31 +11:00
30a8deeb68
Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search
2019-03-11 21:12:30 +11:00
4abc988c6a
initial
2019-03-11 21:11:36 +11:00
fe6c12e6cd
Add comment to schema
2019-03-08 16:38:23 +00:00
5536ddba75
Make `prev_state` field optional
...
The `prev_state` field on events is not specced and so synapse shouldn't
explode if an event is missing the field.
Fixes #4787
2019-03-08 15:05:32 +00:00
8c4896668f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl
2019-03-08 11:44:20 +00:00
f6135d06cf
Rewrite userdir to be faster ( #4537 )
2019-03-07 01:22:53 -08:00
6d13bdec91
Add docstrings from matrix-org-hotfixes
2019-03-06 17:24:26 +00:00
a9de04be72
Implement soft fail
2019-03-06 16:22:16 +00:00
b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room
...
Stop backpaginating when events not visible
2019-03-05 09:32:33 +00:00
c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys
...
Allow /keys/{changes,query} API to run on worker
2019-03-05 09:06:25 +00:00
2db49ea476
Add some debug about processing read receipts.
...
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
2019-03-04 18:19:40 +00:00
a84b8d56c2
Fixup slave stores
2019-03-04 18:04:57 +00:00
8b63fe4c26
s/get_forward_events/get_successor_events/
2019-03-04 11:56:03 +00:00
fbc047f2a5
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room
2019-03-04 11:54:58 +00:00
54f9ce11a7
Move /account/3pid to client_reader
2019-02-27 14:26:08 +00:00
6bb1c028f1
Limit cache invalidation replication line length ( #4748 )
2019-02-27 10:28:37 +00:00
f191be822b
Add database version to phonehome stats. ( #4753 )
2019-02-27 10:21:49 +00:00
80467bbac3
Fix state cache invalidation on workers
2019-02-22 14:38:14 +00:00
71b625d808
Stop backpaginating when events not visible
2019-02-20 18:14:12 +00:00
a06614bd2a
UPSERT many functionality ( #4644 )
2019-02-20 23:03:30 +11:00
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation
...
Batch cache invalidation over replication
2019-02-19 13:14:30 +00:00
62175a20e5
Docs
2019-02-19 11:38:40 +00:00
1bb35e3a83
Use itertools
2019-02-19 11:34:40 +00:00
bc8fa1509d
Documentation
2019-02-19 11:24:59 +00:00
a9b5ea6fc1
Batch cache invalidation over replication
...
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.
Hopefully this will reduce load on the master process by substantially
reducing traffic.
2019-02-18 17:53:31 +00:00
4cc4400b4d
Split /login into client_reader
2019-02-18 17:19:01 +00:00
eb2b8523ae
Split out registration to worker
...
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.
Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
eaf4d11af9
Add configurable room list publishing rules
...
This allows specifying who and what is allowed to be published onto the
public room list
2019-02-14 16:02:23 +00:00
bb4fd8f927
Run `black` on user directory code ( #4635 )
2019-02-13 23:05:32 +11:00
cf82338930
Merge pull request #4627 from matrix-org/erikj/user_ips_analyze
...
Analyze user_ips before running deduplication
2019-02-12 13:05:09 +00:00
495ea92350
Fix pep8
2019-02-12 12:40:42 +00:00
483ba85c7a
Analyze user_ips before running deduplication
...
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
2019-02-12 11:55:27 +00:00
362d80b770
Reduce user_ips bloat during dedupe background update
...
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.
This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.
2019-02-12 11:39:34 +00:00
719e073f00
Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating
...
add updating of backup versions
2019-02-11 13:45:49 +00:00
56710c7df5
Fix 'no unique or exclusion constraint' error ( #4591 )
...
Add more tables to the list of tables which need a background update to
complete before we can upsert into them, which fixes a race against the
background updates.
2019-02-08 18:30:46 +00:00
8248637173
add new endpoint to update backup versions
2019-02-06 17:57:10 -05:00
627ecd358e
Filter user directory state query to a subset of state events ( #4462 )
...
* Filter user directory state query to a subset of state events
* Add changelog
2019-02-05 12:16:28 +00:00
c45fd0dda0
Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable
...
Copy over non-federatable trait on room upgrade
2019-01-31 20:28:37 +00:00
d239f67c25
Raise an exception instead of returning None
2019-01-31 18:34:15 +00:00
3ed3cb4339
New function for getting room's create event
2019-01-31 18:21:39 +00:00
e707e7b38d
Fix infinite loop when an event is redacted in a v3 room ( #4535 )
2019-01-31 15:34:17 +00:00
3f189c902e
Fix flake8 ( #4519 )
2019-01-30 10:53:17 +00:00
a696c48133
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah
2019-01-29 22:00:33 +00:00
c21b7cbc09
Update synapse/storage/events_worker.py
2019-01-29 21:53:48 +00:00
4db252c073
Check redaction state when event is pulled out of the database
2019-01-29 21:48:36 +00:00
6d23ec2111
Fix typo
2019-01-29 21:45:53 +00:00
b5d510ad64
Remove unused arg
2019-01-29 21:45:28 +00:00
82165eeb05
Update synapse/storage/events_worker.py
...
Co-Authored-By: erikjohnston <erikj@jki.re>
2019-01-29 21:14:39 +00:00
7709d2bd16
Implement rechecking of redactions
2019-01-29 11:56:20 +00:00
be47cfa9c9
Refactor event building into EventBuilder
...
This is so that everything is done in one place, making it easier to
change the event format based on room version
2019-01-29 11:13:00 +00:00
94fb63e44f
Fix typo in upserts code ( #4505 )
...
* fix obvious problem :|
* changelog
2019-01-29 10:04:23 +00:00
7072fe3084
Fix UPSERTs on SQLite 3.24+ ( #4477 )
2019-01-28 15:43:32 +00:00
4026d555fa
Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade
2019-01-28 14:08:24 +00:00
b6dce9b9fd
Merge pull request #4470 from matrix-org/erikj/require_format_version
...
Require event format version to parse or create events
2019-01-25 15:59:36 +00:00
431e485914
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert
2019-01-25 14:11:17 +00:00
0b3fd1401f
Don't require sqlite3 when using postgres ( #4466 )
2019-01-25 22:25:02 +11:00
821b65aeb5
Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade
2019-01-25 11:09:53 +00:00
b1b6dba2d2
Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms
...
Ability to search entire room history after upgrading room
2019-01-25 10:49:58 +00:00
1953067136
Disable native upserts for sqlite, as they don't work
2019-01-25 10:46:49 +00:00
678a92cb56
Replace missed usages of FrozenEvent
2019-01-25 10:32:30 +00:00
be6a7e47fa
Revert "Require event format version to parse or create events"
2019-01-25 10:23:51 +00:00
efb8ed1d45
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version
2019-01-24 18:52:34 +00:00
80bcca659e
Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites
...
Store rejected remote invite events as outliers
2019-01-24 18:33:27 +00:00
664b7a2920
Merge pull request #4437 from matrix-org/erikj/event_format_version_v2
...
Add support for persisting event format versions
2019-01-24 18:32:30 +00:00
b8082a5445
Use term 'out of band membership' instead
2019-01-24 17:33:19 +00:00
a383289b0d
Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms
2019-01-24 17:23:51 +00:00
e1781b043b
Remove redundant create event None check
2019-01-24 17:23:39 +00:00
03c85335d1
Apply suggestions from code review
...
Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>
2019-01-24 17:22:09 +00:00
0e27501ee5
Fix UPSERT check ( #4459 )
2019-01-24 22:57:41 +11:00
e8c9f15397
Replace missed usages of FrozenEvent
2019-01-24 11:14:07 +00:00
58f6c48183
Use native UPSERTs where possible ( #4306 )
2019-01-24 21:31:54 +11:00
17898a5ba6
Merge branch 'erikj/fixup_rejecting_invites' of github.com:matrix-org/synapse into erikj/require_format_version
2019-01-23 20:13:07 +00:00
7c288c2250
Clarify the invite flows
2019-01-23 20:07:47 +00:00
886e5acc76
Store rejected remote invite events as outliers
...
Currently they're stored as non-outliers even though the server isn't in
the room, which can be problematic in places where the code assumes it
has the state for all non outlier events.
In particular, there is an edge case where persisting the leave event
triggers a state resolution, which requires looking up the room version
from state. Since the server doesn't have the state, this causes an
exception to be thrown.
2019-01-23 20:04:57 +00:00
2a8edbaf74
Merge branch 'erikj/get_pdu_versions' into erikj/require_format_version
2019-01-23 17:27:49 +00:00
be1065af59
isort
2019-01-23 11:48:16 +00:00
c5a296b10c
Add support for persisting event format versions
...
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.
We specify the format version rather than the room version as:
1. We don't necessarily know the room version, existing events may be
either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
events in the same room reported to be of different versions, which
sounds annoying.
2019-01-23 11:30:01 +00:00
90743c9d89
Fixup removal of duplicate `user_ips` rows ( #4432 )
...
* Remove unnecessary ORDER BY clause
* Add logging
* Newsfile
2019-01-23 19:45:18 +11:00
7f503f83b9
Refactor to rewrite the SQL instead
2019-01-22 16:31:05 +00:00
1c9704f8ab
Don't shadow params
2019-01-22 16:20:33 +00:00
2557531f0f
Fix bug when removing duplicate rows from user_ips
...
This was caused by accidentally overwritting a `last_seen` variable
in a for loop, causing the wrong value to be written to the progress
table. The result of which was that we didn't scan sections of the table
when searching for duplicates, and so some duplicates did not get
deleted.
2019-01-22 13:33:46 +00:00
277e50462d
Do not return in a deferred function
2019-01-22 12:40:26 +00:00
c433f61091
Ensure new filter is actually created
2019-01-22 12:06:36 +00:00
c9bfb058d8
Fix a bug with single-room search searching all rooms
...
* Create a new method for getting predecessor rooms
* Remove formatting change
2019-01-22 12:00:41 +00:00
25d64a846a
Fix typos
2019-01-22 11:00:04 +00:00
25dd56ace3
Fix race when persisting create event ( #4404 )
...
* Fix race when persisting create event
When persisting a chunk of DAG it is sometimes requried to do a state
resolution, which requires knowledge of the room version. If this
happens while we're persisting the create event then we need to use that
event rather than attempting to look it up in the database.
2019-01-18 23:17:04 +11:00
df3a661e4a
Search for messages across predecessor rooms
...
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
2019-01-18 11:19:20 +00:00
a35c66a00b
Remove duplicates in the user_ips table and add an index ( #4370 )
2019-01-12 06:21:50 +11:00
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. ( #4369 )
2019-01-09 22:26:25 +11:00
84b6fae1f5
Ensure synchrotrons can access is_support_user in the storage layer
2019-01-02 10:19:59 +00:00
0708f437cc
Log roomid along with Unknown room ( #4297 )
2018-12-24 10:49:42 +01:00
d2f7c4e6b1
create support user ( #4141 )
...
Allow for the creation of a support user.
A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
e93a0ebf50
Settings Fix deleting e2e room keys on xenial ( #4295 )
2018-12-14 05:10:31 +11:00
30da50a5b8
Initialise user displayname from SAML2 data ( #4272 )
...
When we register a new user from SAML2 data, initialise their displayname
correctly.
2018-12-07 14:44:46 +01:00
9a3e24a13d
drop undocumented dependency on dateutil ( #4266 )
...
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
2018-12-06 04:52:42 +11:00
158ffb92f1
Add an option to disable search for homeservers which may not be interested in it ( #4230 )
...
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
2018-12-04 12:01:02 +01:00
b5ac0ffa0a
add more detail to logging regarding "More than one row matched" error ( #4234 )
2018-12-04 11:57:39 +01:00
ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english ( #4253 )
...
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252 . Fundamentally the right answer is to drop support for Python 2.
2018-12-04 11:55:52 +01:00
704c5298f0
Drop sent_transactions
...
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2018-12-01 23:07:35 -06:00
f9b136a886
Neilj/fix mau initial reserved users ( #4211 )
...
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail
* towncrier
* white space
2018-11-28 20:33:41 +11:00
6c18cc4b50
Ignore __pycache__ directories in schema delta dir
...
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
2018-11-20 22:52:34 +00:00
835779f7fb
Add option to track MAU stats (but not limit people) ( #3830 )
2018-11-15 18:08:27 +00:00
bca3b91c2d
Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers
2018-11-09 18:35:02 +00:00
d44dea0223
pep8
2018-11-09 14:38:31 +00:00
4f93abd62d
add docs
2018-11-09 13:25:38 +00:00
5ebed18692
Lets convert bytes to unicode instead
2018-11-08 12:33:13 +00:00
b1a22b24ab
Fix noop checks when updating device keys
...
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.
The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
2018-11-08 12:18:38 +00:00
bc80b3f454
Add helpers for getting prev and auth events ( #4139 )
...
* Add helpers for getting prev and auth events
This is in preparation for allowing the event format to change between
room versions.
2018-11-06 00:35:15 +11:00
350f654e7b
Add unique indexes to a couple of tables
...
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
2018-11-02 10:36:13 +00:00
50e328d1e7
Remove redundant database locks for device list updates
...
We can rely on the application-level per-user linearizer.
2018-11-02 10:36:13 +00:00
0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups
...
Delete unreferenced state groups during purge
2018-10-30 16:58:22 +00:00
e0934acdbb
Cast to int here too
2018-10-30 11:12:23 +00:00
12941f5f8b
Cast bacjup version to int when querying
2018-10-30 11:01:07 +00:00
2f0f911c52
Convert version back to a string
2018-10-30 10:35:18 +00:00
Erik Johnston