ba17de7fbc
Fix schema update for account validity
2019-05-28 10:11:38 +01:00
bc4b2ecf70
Fix logging for room stats background update
2019-05-25 12:02:48 -06:00
b75537beaf
Store key validity time in the storage layer
...
This is a first step to checking that the key is valid at the required moment.
The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
2e052110ee
Rewrite store_server_verify_key to store several keys at once ( #5234 )
...
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
85d1e03b9d
Simplifications and comments in do_auth ( #5227 )
...
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
4a30e4acb4
Room Statistics ( #4338 )
2019-05-21 11:36:50 -05:00
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender
...
Only count aggregations from distinct senders
2019-05-21 17:10:48 +01:00
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
...
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. ( #5146 )
...
When considering the candidates to be forward-extremities, we must exclude soft
failures.
Hopefully fixes #5090 .
2019-05-21 16:10:54 +01:00
7b0e804a4a
Fix get_max_topological_token to never return None
2019-05-21 15:21:43 +01:00
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
...
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
5ceee46c6b
Do the select and insert in a single transaction
2019-05-21 13:38:51 +01:00
c7ec06e8a6
Block attempts to annotate the same event twice
2019-05-20 17:39:05 +01:00
1dff859d6a
Rename relation types to match MSC
2019-05-20 14:31:19 +01:00
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
...
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
935af0da38
Correctly update aggregation counts after redaction
2019-05-20 12:09:27 +01:00
ad5b4074e1
Add startup background job for account validity
...
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
2019-05-17 19:37:31 +01:00
b63cc325a9
Only count aggregations from distinct senders
...
As a user isn't allowed to send a single emoji more than once.
2019-05-17 18:03:10 +01:00
8dd9cca8ea
Spelling and clarifications
2019-05-17 16:40:51 +01:00
5dbff34509
Fixup bsaed on review comments
2019-05-17 15:48:04 +01:00
d46aab3fa8
Add basic editing support
2019-05-16 16:54:45 +01:00
895179a4dc
Update docstring
2019-05-16 16:41:05 +01:00
7a7eba8302
Move parsing of tokens out of storage layer
2019-05-16 14:26:23 +01:00
2c662ddde4
Indirect tuple conversion
2019-05-16 14:21:39 +01:00
b5c62c6b26
Fix relations in worker mode
2019-05-16 10:38:13 +01:00
33453419b0
Add cache to relations
2019-05-16 10:02:14 +01:00
a0603523d2
Add aggregations API
2019-05-16 09:37:20 +01:00
5be34fc3e3
Actually check for None rather falsey
2019-05-15 17:30:23 +01:00
b50641e357
Add simple pagination API
2019-05-15 13:36:51 +01:00
efe3c7977a
Add simple send_relation API and track in DB
2019-05-15 13:36:51 +01:00
54d77107c1
Make generating SQL bounds for pagination generic
...
This will allow us to reuse the same structure when we paginate e.g.
relations
2019-05-15 11:30:05 +01:00
52ddc6c0ed
Update docstring with correct type
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-15 09:52:15 +01:00
4fb44fb5b9
Expose DataStore._get_events as get_events_as_list
...
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
11ea16777f
Limit the number of EDUs in transactions to 100 as expected by receiver ( #5138 )
...
Fixes #3951 .
2019-05-09 11:01:41 +01:00
bd0d45ca69
Fix infinite loop in presence handler
...
Fixes #5102
2019-04-26 11:14:49 +01:00
ca90336a69
Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration
2019-04-17 19:44:40 +01:00
eaf41a943b
Add management endpoints for account validity
2019-04-17 19:34:45 +01:00
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
...
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
20f0617e87
Send out emails with links to extend an account's validity period
2019-04-17 14:42:20 +01:00
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta
...
Fix schema upgrade when dropping tables
2019-04-10 10:22:35 +01:00
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
...
Add time-based account expiration
2019-04-09 17:02:41 +01:00
747aa9f8ca
Add account expiration feature
2019-04-09 16:46:04 +01:00
50d2a3059d
Fix schema upgrade when dropping tables
...
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.
Introduced in #4992 .
2019-04-09 14:39:18 +01:00
18b69be00f
Rewrite Datastore.get_server_verify_keys
...
Rewrite this so that it doesn't hammer the database.
2019-04-09 00:00:10 +01:00
3352baac4b
Remove unused server_tls_certificates functions ( #5028 )
...
These have been unused since #4120 , and with the demise of perspectives, it is
unlikely that they will ever be used again.
2019-04-08 21:50:18 +01:00
2d951686a7
drop tables listed in #1830 ( #4992 )
...
Tables dropped:
* application_services,
* application_services_regex,
* transaction_id_to_pdu,
* stats_reporting
* current_state_resets
* event_content_hashes
* event_destinations
* event_edge_hashes
* event_signatures
* feedback
* room_hosts
* state_forward_extremities
2019-04-08 15:37:26 +01:00
7fc1e17f4c
Merge pull request #5001 from matrix-org/rav/keyring_cleanups
...
Cleanups in the Keyring
2019-04-08 12:47:09 +01:00
b43d9a920b
Fix docstring on get_server_keys_json
2019-04-04 18:54:03 +01:00
a33a5abc4c
Clean up the database pagination code ( #5007 )
...
* rewrite & simplify
* changelog
* cleanup potential sql injection
2019-04-05 00:21:16 +11:00
4a2e13631d
Add functions to delete a group
2019-04-03 16:29:52 +01:00
4a4d5c4fd6
Fix grammar and document get_current_users_in_room ( #4998 )
2019-04-03 14:32:20 +01:00
e8419554ff
Remove presence lists ( #4989 )
...
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
8f549c1177
Merge pull request #4982 from matrix-org/erikj/msc1915
...
Implement MSC1915 - 3PID unbind APIs
2019-04-03 11:07:09 +01:00
7efd1d87c2
Run black on the rest of the storage module ( #4996 )
2019-04-03 10:07:29 +01:00
5bec8d660d
Make starting pushers faster during start up
...
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.
Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
2019-04-02 16:59:13 +01:00
297bf2547e
Fix sync bug when accepting invites ( #4956 )
...
Hopefully this time we really will fix #4422 .
We need to make sure that the cache on
`get_rooms_for_user_with_stream_ordering` is invalidated *before* the
SyncHandler is notified for the new events, and we can now do so reliably via
the `events` stream.
2019-04-02 12:42:39 +01:00
c75e2017f1
Fixup docstrings
2019-04-02 11:20:06 +01:00
3715c124b3
Grandfather in existing user threepids
...
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
2019-04-01 15:25:19 +01:00
1666c0696a
Track IS used to bind 3PIDs
...
This will then be used to know which IS to default to when unbinding the
threepid.
2019-04-01 15:23:01 +01:00
54a87a7b08
Collect room-version variations into one place ( #4969 )
...
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
902cdc63b6
Merge pull request #4955 from matrix-org/rav/merge_state_into_events
...
Combine the CurrentStateDeltaStream into the EventStream
2019-03-28 18:32:13 +00:00
2e060774ad
Run `black` on some storage modules that the stats branch touches ( #4959 )
2019-03-29 00:37:16 +11:00
4b91c313a9
Combine the CurrentStateDeltaStream into the EventStream
2019-03-27 22:07:05 +00:00
903f04c21f
Use the state event amount for userdir import batching, not room count ( #4944 )
2019-03-27 02:49:28 +11:00
9bde730ef8
Fix bug where read-receipts lost their timestamps ( #4927 )
...
Make sure that they are sent correctly over the replication stream.
Fixes : #4898
2019-03-25 16:38:05 +00:00
ac396a0d32
Refactor out state delta handling into its own class ( #4917 )
2019-03-25 20:37:08 +11:00
3ecec5ede2
Fix upsert
2019-03-21 10:21:15 +00:00
8d8834d3e7
comment block_room
2019-03-20 17:49:56 +00:00
67d618e111
Allow blocking a room multiple times
2019-03-20 16:50:05 +00:00
4d53017432
Batching in the user directory import ( #4900 )
2019-03-21 03:06:36 +11:00
e9eeca1314
Fix user directory background update ( #4887 )
2019-03-19 22:13:53 +11:00
282c97327f
Migrate the user directory initial population to a background task ( #4864 )
2019-03-19 04:50:24 +11:00
9073cfc8bd
Merge pull request #4846 from matrix-org/hawkowl/userdir-search
...
Improve searching in the userdir
2019-03-14 13:54:15 +00:00
d42c81d724
Transfer local user's push rules on room upgrade ( #4838 )
...
Transfer push rules (notifications) on room upgrade
2019-03-12 14:42:53 +00:00
c0332d095f
fixup
2019-03-13 01:30:54 +11:00
797b6a63fc
fixup
2019-03-13 01:17:51 +11:00
81d9d1bee6
fixup
2019-03-12 22:28:48 +11:00
6f5890b2fa
fixup
2019-03-12 22:27:56 +11:00
10480c4348
fixup
2019-03-12 21:47:14 +11:00
8b618041ef
fixup
2019-03-12 18:06:28 +11:00
c980c7e31f
use the old method
2019-03-12 17:51:14 +11:00
78a6b950b3
fix
2019-03-12 00:50:28 +11:00
1b77bd69fb
pep8
2019-03-12 00:39:12 +11:00
5ba8ceab4c
fixes
2019-03-12 00:35:31 +11:00
30a8deeb68
Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search
2019-03-11 21:12:30 +11:00
4abc988c6a
initial
2019-03-11 21:11:36 +11:00
fe6c12e6cd
Add comment to schema
2019-03-08 16:38:23 +00:00
5536ddba75
Make `prev_state` field optional
...
The `prev_state` field on events is not specced and so synapse shouldn't
explode if an event is missing the field.
Fixes #4787
2019-03-08 15:05:32 +00:00
8c4896668f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl
2019-03-08 11:44:20 +00:00
f6135d06cf
Rewrite userdir to be faster ( #4537 )
2019-03-07 01:22:53 -08:00
6d13bdec91
Add docstrings from matrix-org-hotfixes
2019-03-06 17:24:26 +00:00
a9de04be72
Implement soft fail
2019-03-06 16:22:16 +00:00
b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room
...
Stop backpaginating when events not visible
2019-03-05 09:32:33 +00:00
c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys
...
Allow /keys/{changes,query} API to run on worker
2019-03-05 09:06:25 +00:00
2db49ea476
Add some debug about processing read receipts.
...
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
2019-03-04 18:19:40 +00:00
a84b8d56c2
Fixup slave stores
2019-03-04 18:04:57 +00:00
8b63fe4c26
s/get_forward_events/get_successor_events/
2019-03-04 11:56:03 +00:00
fbc047f2a5
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room
2019-03-04 11:54:58 +00:00
54f9ce11a7
Move /account/3pid to client_reader
2019-02-27 14:26:08 +00:00
6bb1c028f1
Limit cache invalidation replication line length ( #4748 )
2019-02-27 10:28:37 +00:00
f191be822b
Add database version to phonehome stats. ( #4753 )
2019-02-27 10:21:49 +00:00
80467bbac3
Fix state cache invalidation on workers
2019-02-22 14:38:14 +00:00
71b625d808
Stop backpaginating when events not visible
2019-02-20 18:14:12 +00:00
a06614bd2a
UPSERT many functionality ( #4644 )
2019-02-20 23:03:30 +11:00
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation
...
Batch cache invalidation over replication
2019-02-19 13:14:30 +00:00
62175a20e5
Docs
2019-02-19 11:38:40 +00:00
1bb35e3a83
Use itertools
2019-02-19 11:34:40 +00:00
bc8fa1509d
Documentation
2019-02-19 11:24:59 +00:00
a9b5ea6fc1
Batch cache invalidation over replication
...
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.
Hopefully this will reduce load on the master process by substantially
reducing traffic.
2019-02-18 17:53:31 +00:00
4cc4400b4d
Split /login into client_reader
2019-02-18 17:19:01 +00:00
eb2b8523ae
Split out registration to worker
...
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.
Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
eaf4d11af9
Add configurable room list publishing rules
...
This allows specifying who and what is allowed to be published onto the
public room list
2019-02-14 16:02:23 +00:00
bb4fd8f927
Run `black` on user directory code ( #4635 )
2019-02-13 23:05:32 +11:00
cf82338930
Merge pull request #4627 from matrix-org/erikj/user_ips_analyze
...
Analyze user_ips before running deduplication
2019-02-12 13:05:09 +00:00
495ea92350
Fix pep8
2019-02-12 12:40:42 +00:00
483ba85c7a
Analyze user_ips before running deduplication
...
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
2019-02-12 11:55:27 +00:00
362d80b770
Reduce user_ips bloat during dedupe background update
...
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.
This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.
2019-02-12 11:39:34 +00:00
719e073f00
Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating
...
add updating of backup versions
2019-02-11 13:45:49 +00:00
56710c7df5
Fix 'no unique or exclusion constraint' error ( #4591 )
...
Add more tables to the list of tables which need a background update to
complete before we can upsert into them, which fixes a race against the
background updates.
2019-02-08 18:30:46 +00:00
8248637173
add new endpoint to update backup versions
2019-02-06 17:57:10 -05:00
627ecd358e
Filter user directory state query to a subset of state events ( #4462 )
...
* Filter user directory state query to a subset of state events
* Add changelog
2019-02-05 12:16:28 +00:00
c45fd0dda0
Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable
...
Copy over non-federatable trait on room upgrade
2019-01-31 20:28:37 +00:00
d239f67c25
Raise an exception instead of returning None
2019-01-31 18:34:15 +00:00
3ed3cb4339
New function for getting room's create event
2019-01-31 18:21:39 +00:00
e707e7b38d
Fix infinite loop when an event is redacted in a v3 room ( #4535 )
2019-01-31 15:34:17 +00:00
3f189c902e
Fix flake8 ( #4519 )
2019-01-30 10:53:17 +00:00
a696c48133
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah
2019-01-29 22:00:33 +00:00
c21b7cbc09
Update synapse/storage/events_worker.py
2019-01-29 21:53:48 +00:00
4db252c073
Check redaction state when event is pulled out of the database
2019-01-29 21:48:36 +00:00
6d23ec2111
Fix typo
2019-01-29 21:45:53 +00:00
b5d510ad64
Remove unused arg
2019-01-29 21:45:28 +00:00
82165eeb05
Update synapse/storage/events_worker.py
...
Co-Authored-By: erikjohnston <erikj@jki.re>
2019-01-29 21:14:39 +00:00
7709d2bd16
Implement rechecking of redactions
2019-01-29 11:56:20 +00:00
be47cfa9c9
Refactor event building into EventBuilder
...
This is so that everything is done in one place, making it easier to
change the event format based on room version
2019-01-29 11:13:00 +00:00
94fb63e44f
Fix typo in upserts code ( #4505 )
...
* fix obvious problem :|
* changelog
2019-01-29 10:04:23 +00:00
7072fe3084
Fix UPSERTs on SQLite 3.24+ ( #4477 )
2019-01-28 15:43:32 +00:00
4026d555fa
Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade
2019-01-28 14:08:24 +00:00
b6dce9b9fd
Merge pull request #4470 from matrix-org/erikj/require_format_version
...
Require event format version to parse or create events
2019-01-25 15:59:36 +00:00
431e485914
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert
2019-01-25 14:11:17 +00:00
0b3fd1401f
Don't require sqlite3 when using postgres ( #4466 )
2019-01-25 22:25:02 +11:00
821b65aeb5
Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade
2019-01-25 11:09:53 +00:00
b1b6dba2d2
Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms
...
Ability to search entire room history after upgrading room
2019-01-25 10:49:58 +00:00
1953067136
Disable native upserts for sqlite, as they don't work
2019-01-25 10:46:49 +00:00
678a92cb56
Replace missed usages of FrozenEvent
2019-01-25 10:32:30 +00:00
be6a7e47fa
Revert "Require event format version to parse or create events"
2019-01-25 10:23:51 +00:00
efb8ed1d45
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version
2019-01-24 18:52:34 +00:00
80bcca659e
Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites
...
Store rejected remote invite events as outliers
2019-01-24 18:33:27 +00:00
664b7a2920
Merge pull request #4437 from matrix-org/erikj/event_format_version_v2
...
Add support for persisting event format versions
2019-01-24 18:32:30 +00:00
b8082a5445
Use term 'out of band membership' instead
2019-01-24 17:33:19 +00:00
a383289b0d
Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms
2019-01-24 17:23:51 +00:00
Brendan Abolivier