07cb38e965
Use a sensible default value for labels
2019-11-04 09:56:10 +00:00
a46574281d
Use the right format for rows
2019-11-04 09:56:10 +00:00
c9a1b80a74
MSC2326: Add background update to take previous events into account
2019-11-04 09:56:04 +00:00
f496d25877
Merge pull request #6301 from matrix-org/babolivier/msc2326
...
Implement MSC2326 (label based filtering)
2019-11-01 17:04:45 +00:00
988d8d6507
Incorporate review
2019-11-01 16:22:44 +00:00
5598445655
Update synapse/storage/data_stores/main/schema/delta/56/event_labels.sql
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-11-01 16:18:34 +00:00
fa7e52caf1
Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schema
...
fix hidden field in devices table for older sqlite
2019-11-01 10:52:46 -04:00
a2c63c619a
Add more data to the event_labels table and fix the indexes
2019-11-01 11:47:28 +00:00
669b6cbda3
Fix up comment
2019-11-01 11:32:20 +00:00
57cdb046e4
Lint
2019-11-01 10:39:14 +00:00
c6dbca2422
Incorporate review
2019-11-01 10:30:51 +00:00
c61db13183
fix hidden field in devices table for older sqlite
2019-10-31 22:52:55 -04:00
c3fc176c60
Update synapse/storage/data_stores/main/devices.py
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 22:49:48 -04:00
6f4bc6d01d
Merge branch 'develop' into cross-signing_federation
2019-10-31 22:38:21 -04:00
9c94b48bf1
Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify
2019-10-31 12:32:07 -04:00
fb1a6914cf
Update log line to lie a little less
2019-10-31 15:45:48 +00:00
020add5099
Update black to 19.10b0 ( #6304 )
...
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
61be1a2926
Add state_groups.room_id index
2019-10-31 15:39:26 +00:00
f91f2a1f92
Docstrings
2019-10-31 15:26:00 +00:00
8f5bbdb987
Fix purge room API
2019-10-31 15:22:08 +00:00
cd581338cf
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history
2019-10-31 15:19:26 +00:00
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
...
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
64f2b8c3d8
Apply suggestions from code review
...
Fix docstring
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 15:44:31 +01:00
c6bcd38841
Fix /purge_room API.
...
It fails trying to clean the `topic` table which was recently removed.
2019-10-31 11:17:23 +00:00
54fef094b3
Remove usage of deprecated logger.warn method from codebase ( #6271 )
...
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
998f7fe7d4
make user signatures a separate stream
2019-10-30 17:22:52 -04:00
670972c0e1
Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify
2019-10-30 16:46:31 -04:00
bb6cec27a5
rename get_devices_by_remote to get_device_updates_by_remote
2019-10-30 14:57:34 -04:00
0467f33584
fix delete_existing for _persist_events ( #6300 )
...
this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
2019-10-30 18:05:00 +00:00
dcc069a2e2
Lint
2019-10-30 18:01:56 +00:00
fe51d6cacf
Add more integration testing
2019-10-30 17:28:41 +00:00
233b14ebe1
Add index on label
2019-10-30 15:58:05 +00:00
acd16ad86a
Implement filtering
2019-10-30 15:56:33 +00:00
7c8c97e635
Split purge API into events vs state
2019-10-30 15:23:37 +00:00
5db03535d5
Add StateGroupStorage interface
2019-10-30 14:46:49 +00:00
fa0dcbc8fa
Store labels for new events
2019-10-30 14:27:15 +00:00
7d7eac61be
Merge branch 'develop' into cross-signing_federation
2019-10-30 10:17:10 -04:00
d78b1e339d
apply changes as a result of PR review
2019-10-30 10:01:53 -04:00
ec6de1cc7d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store
2019-10-30 13:37:04 +00:00
a8d16f6c00
Review comments
2019-10-30 13:36:12 +00:00
47f767269c
Add database table for keeping track of labels on events
2019-10-29 16:56:22 +00:00
2c35ffead2
Port receipt and read markers to async/wait
2019-10-29 15:08:22 +00:00
561133c3c5
Merge pull request #6263 from matrix-org/erikj/caches_return_deferreds
...
Quick fix to ensure cache descriptors always return deferreds
2019-10-29 12:53:21 +01:00
e419c44ba4
Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop
2019-10-29 11:41:27 +00:00
342d871d76
Make room directory search case insensitive
2019-10-29 10:24:02 +00:00
14504ad573
Add CI for synapse_port_db ( #6140 )
...
This adds:
* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)
as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:
1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)
Step `2` is done via a new script located at `scripts-dev/update_database`.
The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
d0d8a22c13
Quick fix to ensure cache descriptors always return deferreds
2019-10-28 13:33:04 +00:00
9aee28927b
Convert EventContext to attrs ( #6218 )
...
* make EventContext use an attr
2019-10-28 14:29:55 +02:00
da78f61778
Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys
...
delete keys when deleting backup versions
2019-10-25 11:28:11 -04:00
a71b8c87ec
Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop
2019-10-25 11:32:24 +01:00
a411f2b177
Fix /keys/query API on workers.
...
The necessary getters were added only to the master store and not the
worker stores.
2019-10-25 11:08:03 +01:00
44ab048cfe
Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_logging
...
Reduce debug logging overhead
2019-10-25 10:05:44 +01:00
c40d7244f8
Merge branch 'develop' into cross-signing_federation
2019-10-24 22:31:25 -04:00
8ac766c44a
make notification of signatures work with workers
2019-10-24 22:14:58 -04:00
848cd388d9
delete keys when deleting backups
2019-10-24 21:21:51 -04:00
e4d98188da
Address codestyle concerns
2019-10-24 18:43:13 +01:00
9eebc1e73b
use %r to __repr__ objects
...
This avoids calculating __repr__ unless we are going to log.
2019-10-24 18:18:56 +01:00
39266a9c9f
Make user/room stats log line less verbose.
2019-10-24 17:55:53 +01:00
3aa2a90556
Move schema delta files to the correct data store.
...
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
73cf63784b
Add DataStores and Storage classes.
2019-10-23 16:15:03 +01:00
dc2cd6f79d
move get_e2e_cross_signing_key to EndToEndKeyWorkerStore so it works with workers
2019-10-23 09:13:47 -04:00
22a9847670
Move persist_events out from main data store.
...
This is in preparation for splitting out of state_groups_state from the
main store into it own one, as persisting events depends on calculating
state.
2019-10-23 13:29:44 +01:00
404e8c8532
vendor-prefix the EDU name until MSC1756 is merged into the spec
2019-10-22 22:33:23 -04:00
1fabf82d50
update to work with newer code, and fix formatting
2019-10-22 21:44:58 -04:00
cfdb84422d
make black happy
2019-10-22 19:06:06 -04:00
a1aaf3eea6
don't crash if the user doesn't have cross-signing keys
2019-10-22 19:04:37 -04:00
8d3542a64e
implement federation parts of cross-signing
2019-10-22 19:04:35 -04:00
c17efdc01c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores
2019-10-22 18:44:29 +01:00
23d62eded2
Clean up prepare_database.py a bit and add comments
2019-10-22 18:43:31 +01:00
6cc497f99b
Delete background_update table creation in main data_store
2019-10-22 18:02:50 +01:00
1bbc5444a8
Move README into synapse/storage/__init__.py
2019-10-22 17:59:31 +01:00
acf47c7698
Add a basic README to synapse.storage
2019-10-22 11:55:46 +01:00
bb6264be0b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores
2019-10-22 10:41:18 +01:00
ffd24545bb
Fix schema management to work with multiple data stores.
2019-10-21 16:08:40 +01:00
c66a06ac6b
Move storage classes into a main "data store".
...
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
36adfaedab
Merge branch 'develop' into cross-signing_sig_upload
2019-10-18 18:34:42 +01:00
cfc28325a6
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-10-18 11:39:40 +01:00
f0f6a2b360
use the right function for when we're already in runInteraction
2019-10-18 10:56:54 +01:00
06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
...
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
132b251e29
expand on comment
2019-10-11 14:24:52 -04:00
a0d0ba7862
Fix MAU reaping where reserved users are specified. ( #6168 )
2019-10-11 09:38:26 +01:00
7a0dce9259
make sure we actually return something
2019-10-10 20:31:30 -04:00
4908fb3b30
make storage layer in charge of interpreting the device key data
2019-10-10 19:15:30 -04:00
83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any
...
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
b54b1e759a
Fix SQLite take 2
2019-10-10 16:19:40 +01:00
afb6d9d53b
Fix SQLite
2019-10-10 15:55:41 +01:00
3bc687508f
Remove add_in_list_sql_clause
2019-10-10 15:35:46 +01:00
9d06fb9cb1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any
2019-10-10 14:38:21 +01:00
8bc529c04d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 14:30:51 +01:00
b161786c14
Replace IN usage with helper funcs
2019-10-10 13:15:49 +01:00
b4fbf71187
Add helper funcs to use postgres ANY
...
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
91f43dca39
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 13:10:57 +01:00
9970f955ce
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-10 11:51:50 +01:00
a139420a3c
Fix races in room stats (and other) updates. ( #6187 )
...
Hopefully this will fix the occasional failures we were seeing in the room directory.
The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.
We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
562b4e51dd
Rewrite the user_filter migration again ( #6184 )
...
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
5c1f886c75
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-09 16:52:21 +01:00
7f18b3d526
Do the update as a background index
2019-10-09 16:03:24 +01:00
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
...
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
1d3858371e
Disable bytes usage with postgres
...
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
e7631d84e6
Fix existing hex encoded json values in DB
2019-10-08 16:18:43 +01:00
ced4784592
Fix inserting bytes as text
2019-10-08 16:18:43 +01:00
c69324ffb5
Fix RegistrationStore
2019-10-08 14:48:33 +01:00
b1c0a4ceb3
Cleanup client_ips
2019-10-08 14:38:14 +01:00
8f1b385acc
Don't end up with 4 classes in registration
2019-10-08 14:36:33 +01:00
66ebea1723
Lint
2019-10-07 17:44:41 +01:00
0496eafbf4
Move roommember's bg updates to a dedicated store
2019-10-07 17:35:01 +01:00
e106a0e4db
Move user_directory's bg updates to a dedicated store
2019-10-07 17:34:45 +01:00
cfccd2d78a
Move state's bg updates to a dedicated store
2019-10-07 17:34:39 +01:00
841054ad96
Move search's bg updates to a dedicated store
2019-10-07 17:34:35 +01:00
81e6ffb536
Move registration's bg updates to a dedicated store
2019-10-07 17:34:29 +01:00
54f87e0734
Move media_repository's bg updates to a dedicated store
2019-10-07 17:34:26 +01:00
cef9f6753e
Move devices's bg updates to a dedicated store
2019-10-07 17:34:20 +01:00
2d3b4f42f0
Move deviceinbox's bg updates to a dedicated store
2019-10-07 17:34:16 +01:00
88957199e7
Move client_ips's bg updates to a dedicated store
2019-10-07 17:34:12 +01:00
276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test ( #6167 )
2019-10-07 14:41:39 +01:00
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
...
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
c8e6c308c6
Fix unique_user_filter_index schema update
2019-10-07 13:15:35 +01:00
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships
...
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
5119a4cac7
Fix bug where we didn't pull out event ID
2019-10-07 12:21:17 +01:00
13c4345c84
Update `user_filters` table to have a unique index, and non-null columns ( #1172 )
2019-10-04 10:34:16 +01:00
6511071837
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships
2019-10-03 17:51:06 +01:00
66537e10ce
add some metrics on the federation sender ( #6160 )
2019-10-03 17:47:20 +01:00
91f61fc6d7
Use the right error....
2019-10-03 17:28:31 +01:00
84691da6c3
pep8
2019-10-03 17:27:18 +01:00
a9610cdf02
Fixup names and comments
2019-10-03 17:26:56 +01:00
d89ebf7c25
cachedList descriptor doesn't like typing
2019-10-03 17:23:11 +01:00
c8145af8a9
Cache room membership lookups in _get_joined_users_from_context
2019-10-03 17:11:04 +01:00
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
...
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
...
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
...
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
7a5f080f91
Fix appservice room list pagination
2019-10-02 15:47:22 +01:00
4c4f44930d
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 15:20:36 +01:00
03cf4385e0
Fix public room list pagination.
...
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
a5166e4d5f
Land improved room list based on room stats ( #6019 )
...
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade ( #6144 )
...
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.
For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.
Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
d69fd53f74
Bound find_next_generated_user_id DB query.
...
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
ecd254bc49
Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop
2019-10-02 11:08:07 +01:00
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
...
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions
...
Fix fetching censored redactions from DB
2019-10-02 10:57:30 +01:00
f44f1d2e83
Fix errors storing large retry intervals.
...
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.
To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
ce7a3e7e27
Fix fetching censored redactions from DB
...
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
5e8387af9e
Use `received_ts` to find uncensored redacted events
...
Joining against `events` and ordering by `stream_ordering` is
inefficient as it forced scanning the entirety of the redactions table.
This isn't the case if we use `redactions.received_ts` column as we can
then use an index.
2019-10-01 13:43:48 +01:00
898dde981b
Add received_ts column to redactions.
...
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
a27fb7d5ca
Don't repeatedly attempt to censor events we don't have.
...
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.
When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
1d349fb159
Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop
2019-10-01 10:17:24 +01:00
9267741a5f
Fix `devices_last_seen` background update.
...
Fixes #6134 .
2019-09-30 11:58:36 +01:00
16cb9a71b8
Drop unused tables ( #6115 )
...
These tables are unused since #5893 (as amended by #6047 ), so we can now drop
them.
Fixes #6048 .
2019-09-30 09:38:41 +01:00
132279a46f
Patch inlinecallbacks for log contexts
2019-09-27 15:11:14 +01:00
034db2ba21
Fix dummy event insertion consent bug ( #6053 )
...
Fixes #5905
2019-09-26 11:47:53 +01:00
4fb3c129aa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2
2019-09-25 17:53:13 +01:00
39b50ad42a
Review comments
2019-09-25 17:22:33 +01:00
d2bd0bc6b1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips
2019-09-25 17:16:28 +01:00
50572db837
Use if `is not None`
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
26113fb7de
make changes based on PR feedback
2019-09-24 14:12:20 -04:00
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
...
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
9b7c4f4078
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 17:03:50 +01:00
566ac40939
remove unused parameter to get_user_id_by_threepid ( #6099 )
...
Added in #5377 , apparently in error
2019-09-24 17:01:09 +01:00
242017db8b
Prune rows in user_ips older than configured period
...
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
2135c198d1
Add has_completed_background_update
...
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
12fe2a29bc
Incorporate review
2019-09-24 14:43:38 +01:00
f02f14e09a
Fix logging
2019-09-24 14:39:07 +01:00
ed8b92f0d2
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 12:57:32 +01:00
323d685bf7
Typo
2019-09-23 17:23:49 +01:00
2858d10671
Fix the return value in the users_set_deactivated_flag background job
2019-09-23 17:22:01 +01:00
51d28272e2
Query devices table for last seen info.
...
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.
The storage functions now no longer return the access_token, since it
was unused.
2019-09-23 16:59:45 +01:00
ed80231ade
Add BG update to populate devices last seen info
2019-09-23 16:59:45 +01:00
2ade05dca3
Add last seen info to devices table.
...
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
30af161af2
Implement MSC2290 ( #6043 )
...
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.
This PR also removes the blacklist on some 3PID tests which occurs in #6042 , as the corresponding Sytest PR changes them to use the new endpoints.
Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
df3401a71d
Allow HS to send emails when adding an email to the HS ( #6042 )
2019-09-20 15:21:30 +01:00
b65327ff66
Merge branch 'develop' into rav/saml_mapping_work
2019-09-19 18:13:31 +01:00
bcd9132869
Undo the deletion of some tables ( #6047 )
...
This is a partial revert of #5893 . The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.
So let's leave the tables in place for now, and remember to drop them in a
subsequent release.
(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
2019-09-19 15:06:27 +01:00
1e19ce00bf
Add 'failure_ts' column to 'destinations' table ( #6016 )
...
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
a8ac40445c
Record mappings from saml users in an external table
...
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
7902bf1e1d
Clean up some code in the retry logic ( #6017 )
...
* remove some unused code
* make things which were constants into constants for efficiency and clarity
2019-09-11 15:14:56 +01:00
63f9317b8e
Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users
...
Only count real users when checking for auto-creation of auto-join room
2019-09-09 17:37:52 +03:00
470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions
...
Censor redactions in DB after a month
2019-09-09 15:29:39 +01:00
aaed6b39e1
Fix code style, again
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 17:10:02 +03:00
580f3df9b2
Fix comments
2019-09-09 15:08:24 +01:00
e89fea4f04
Simplify count_real_users SQL to only count user_type is null rows
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 16:43:32 +03:00
8c03cd0e5f
Simplify is_real_user_txn check to trust user_type is null if real user
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 16:40:40 +03:00
e7184a4370
Use better names in SQL
2019-09-09 13:33:38 +01:00
916c697228
Fixup comment
2019-09-09 13:31:00 +01:00
fffe17b77d
Don't start looping call unless enabled
2019-09-09 13:24:24 +01:00
80e14a8546
Handle setting retention period to 0
2019-09-09 13:23:41 +01:00
62fac9d969
Auto-fix a few code style issues
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:59:35 +03:00
be618e0551
Only count real users when checking for auto-creation of auto-join room
...
Previously if the first registered user was a "support" or "bot" user,
when the first real user registers, the auto-join rooms were not
created.
Fix to exclude non-real (ie users with a special user type) users
when counting how many users there are to determine whether we should
auto-create a room.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:48:08 +03:00
8e86f5b65c
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-09-07 13:20:34 -04:00
0d61d1d735
Merge branch 'develop' into cross-signing_sig_upload
2019-09-07 13:14:45 -04:00
55d5b3af88
Servers-known-about statistic ( #5981 )
2019-09-07 01:45:51 +10:00
a2a695b7ec
Merge pull request #5998 from matrix-org/erikj/fixup_federate_flag
...
Correctly handle non-bool m.federate flag
2019-09-06 15:32:43 +01:00
142c9325c2
Correctly handle non-bool m.federate flag
2019-09-06 14:21:06 +01:00
6ddda8152e
Move delete_threepid_session into RegistrationWorkerStore
2019-09-06 13:23:10 +01:00
e059c5e648
Move get_threepid_validation_session into RegistrationWorkerStore
2019-09-06 13:10:11 +01:00
591d82f06b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions
2019-09-05 17:27:46 +01:00
3ff0422d2d
Make redaction retention period configurable
2019-09-05 17:16:45 +01:00
5ade977d08
Opentracing context cannot be none
2019-09-05 15:06:13 +01:00
7d6c70fc7a
make black happy
2019-09-04 22:04:12 -04:00
ac4746ac4b
allow uploading signatures of master key signed by devices
2019-09-04 22:03:41 -04:00
4bb4544784
implement device signature uploading/fetching
2019-09-04 20:02:56 -04:00
faf72a4c40
Merge branch 'develop' into cross-signing_keys
2019-09-04 19:12:29 -04:00
6e834e94fc
Fix and refactor room and user stats ( #5971 )
...
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
2a44782666
Remove double return statements ( #5962 )
...
Remove all the "double return" statements which were a result of us removing all the instances of
```
defer.returnValue(...)
return
```
statements when we switched to python3 fully.
2019-09-03 11:42:45 +01:00
a90d16dabc
Opentrace device lists ( #5853 )
...
Trace device list changes.
2019-09-03 10:21:30 +01:00
4548d1f87e
Remove unnecessary parentheses around return statements ( #5931 )
...
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
a4bf72c30c
Censor redactions in DB after a month
2019-08-30 11:29:17 +01:00
e3d3fbf63f
Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys
2019-08-28 17:36:46 -07:00
72d296a7f3
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-08-28 17:33:44 -07:00
3b0b22cb05
use stream ID generator instead of timestamp
2019-08-28 17:17:21 -07:00
c8fa620d7a
Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent
...
Exempt support users from consent
2019-08-28 16:31:40 +01:00
49ef8ec399
Fix a cache-invalidation bug for worker-based deployments ( #5920 )
...
Some of the caches on worker processes were not being correctly invalidated
when a room's state was changed in a way that did not affect the membership
list of the room.
We need to make sure we send out cache invalidations even when no memberships
are changing.
2019-08-28 10:18:16 +01:00
322ccac33f
Allow schema deltas to be engine-specific ( #5911 )
...
* Allow schema deltas to be engine-specific
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Code style (Black)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 11:53:21 +01:00
1a7e6eb633
Add Admin API capability to set adminship of a user ( #5878 )
...
Admin API: Set adminship of a user
2019-08-27 10:14:00 +01:00
d1e0b91083
Code style (Black)
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:39:11 +01:00
aefa76f5cd
Allow schema deltas to be engine-specific
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:14:00 +01:00
ae38e0569f
Ignore consent for support users
2019-08-23 09:15:10 +01:00
8767b63a82
Propagate opentracing contexts through EDUs ( #5852 )
...
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
9a6f2be572
Opentrace e2e keys ( #5855 )
...
Add opentracing tags and logs for e2e keys
2019-08-22 11:28:12 +01:00
c9f11d09fc
Add missing index on users_in_public_rooms. ( #5894 )
2019-08-22 10:43:13 +01:00
119aa31b10
Servlet to purge old rooms ( #5845 )
2019-08-22 10:42:59 +01:00
7c3abc6572
apply PR review suggestions
2019-08-21 13:19:35 -07:00
4dab867288
Drop some unused tables. ( #5893 )
...
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
8a5f6ed130
Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index
...
add the version field to the index for e2e_room_keys
2019-08-16 17:45:50 -07:00
c058aeb88d
update set_e2e_room_key to agree with fixed index
2019-08-14 18:02:58 -07:00
b7f7cc7ace
add the version field to the index for e2e_room_keys
2019-08-14 17:14:40 -07:00
af187805b3
Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop
...
Handle pusher being deleted during processing.
2019-08-13 14:08:29 +01:00
fb3469f53a
Clarify docstring
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-13 10:17:23 +01:00
2546f32b90
Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses
...
Don't unnecessarily block notifying of new events.
2019-08-13 09:36:25 +01:00
f63ba7a795
Cross-signing [1/4] -- hidden devices ( #5759 )
...
* allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions
...
Fix handling of redactions of redactions
2019-08-12 12:25:19 +01:00
bf4db42920
Don't unnecessarily block notifying of new events.
...
When persisting events we calculate new stream orderings up front.
Before we notify about an event all events with lower stream orderings
must have finished being persisted.
This PR moves the assignment of stream ordering till *after* calculated
the new current state and split the batch of events into separate chunks
for persistence. This means that if it takes a long time to calculate
new current state then it will not block events in other rooms being
notified about.
This should help reduce some global pauses in the events stream which
can last for tens of seconds (if not longer), caused by some
particularly expensive state resolutions.
2019-08-06 13:32:02 +01:00
fac1cdc562
make changes from PR review
2019-08-01 21:51:19 -04:00
336c546d6a
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-08-01 16:31:40 -04:00
5d018d23f0
Have ClientReaderSlavedStore inherit RegistrationStore ( #5806 )
...
Fixes #5803
2019-08-01 13:54:56 +01:00
d02e41dcb2
Handle pusher being deleted during processing.
...
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
2019-08-01 13:44:12 +01:00
d78a4fe156
don't need to return the hidden column any more
2019-08-01 02:16:09 -04:00
73b26f827c
really fix queries to work with Postgres
...
(by going back to not using SQL directly)
2019-07-31 18:37:05 -04:00
430ea08186
PostgreSQL, Y U no like?
2019-07-31 15:38:11 -04:00
185188be03
remove extra SQL query param
2019-07-31 15:18:15 -04:00
c1f0a5636c
Merge branch 'develop' into cross-signing_hidden
2019-07-30 23:33:06 -04:00
bc95890822
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-07-30 23:21:26 -04:00
781ade836b
apply changes from PR review
2019-07-30 23:09:50 -04:00
123c04daa7
Don't recreate column
2019-07-30 16:42:48 +01:00
15056ca208
Fix current_state_events membership background update.
...
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
2019-07-30 14:51:41 +01:00
4e97eb89e5
Handle loops in redaction events
2019-07-30 00:00:34 +01:00
448bcfd0f9
recursively fetch redactions
2019-07-30 00:00:34 +01:00
e6a6c4fbab
split _get_events_from_db out of _enqueue_events
2019-07-29 23:15:15 +01:00
c9964ba600
Return dicts from _fetch_event_list
2019-07-29 23:15:15 +01:00
7c8c3b8437
Merge pull request #5774 from matrix-org/erikj/fix_rejected_membership
...
Fix room summary when rejected events are in state
2019-07-29 17:15:15 +01:00
3e013b7c8e
Merge pull request #5752 from matrix-org/erikj/forgotten_user
...
Remove some more joins on room_memberships
2019-07-29 17:15:01 +01:00
df3a5db629
Expand comment
2019-07-29 16:40:25 +01:00
Brendan Abolivier