Commit Graph

2993 Commits (199aa72d3519dcb472cf60952d56930bc3c4dc29)

Author SHA1 Message Date
Amber Brown 199aa72d35 Merge branch 'develop' of ssh://github.com/matrix-org/synapse into
matrix-org-hotfixes
2019-03-07 21:43:10 +11:00
Amber Brown f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Erik Johnston 8f7dbbc14a Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-03-06 19:30:30 +00:00
Erik Johnston 6d13bdec91 Add docstrings from matrix-org-hotfixes 2019-03-06 17:24:26 +00:00
Erik Johnston dc510e0e43 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-03-05 14:41:13 +00:00
Erik Johnston b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room
Stop backpaginating when events not visible
2019-03-05 09:32:33 +00:00
Erik Johnston c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys
Allow /keys/{changes,query} API to run on worker
2019-03-05 09:06:25 +00:00
Richard van der Hoff 2db49ea476 Add some debug about processing read receipts.
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
2019-03-04 18:19:40 +00:00
Erik Johnston a84b8d56c2 Fixup slave stores 2019-03-04 18:04:57 +00:00
Erik Johnston 8b63fe4c26 s/get_forward_events/get_successor_events/ 2019-03-04 11:56:03 +00:00
Erik Johnston fbc047f2a5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room 2019-03-04 11:54:58 +00:00
Erik Johnston 54f9ce11a7 Move /account/3pid to client_reader 2019-02-27 14:26:08 +00:00
Richard van der Hoff 67acd1aa1b Merge branch 'develop' into matrix-org-hotfixes 2019-02-27 10:29:24 +00:00
Erik Johnston 6bb1c028f1 Limit cache invalidation replication line length (#4748) 2019-02-27 10:28:37 +00:00
Richard van der Hoff f191be822b
Add database version to phonehome stats. (#4753) 2019-02-27 10:21:49 +00:00
Richard van der Hoff 6087c53830
Add more debug for membership syncing issues (#4719) 2019-02-25 17:00:18 +00:00
Erik Johnston 0ea52872ab Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-22 15:29:41 +00:00
Erik Johnston 80467bbac3 Fix state cache invalidation on workers 2019-02-22 14:38:14 +00:00
Erik Johnston 71b625d808 Stop backpaginating when events not visible 2019-02-20 18:14:12 +00:00
Richard van der Hoff 68af15637b Merge branch 'develop' into matrix-org-hotfixes 2019-02-20 14:24:17 +00:00
Amber Brown a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Erik Johnston 0e07d2c7d5 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-19 13:24:37 +00:00
Erik Johnston 90ec885805 Revert "Merge pull request #4654 from matrix-org/hawkowl/registration-worker"
This reverts commit 5bd2e2c31d, reversing
changes made to d97c3a6ce6.
2019-02-19 13:23:17 +00:00
Erik Johnston c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation
Batch cache invalidation over replication
2019-02-19 13:14:30 +00:00
Erik Johnston 62175a20e5 Docs 2019-02-19 11:38:40 +00:00
Erik Johnston 1bb35e3a83 Use itertools 2019-02-19 11:34:40 +00:00
Erik Johnston bc8fa1509d Documentation 2019-02-19 11:24:59 +00:00
Erik Johnston a9b5ea6fc1 Batch cache invalidation over replication
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.

Hopefully this will reduce load on the master process by substantially
reducing traffic.
2019-02-18 17:53:31 +00:00
Erik Johnston 4cc4400b4d Split /login into client_reader 2019-02-18 17:19:01 +00:00
Erik Johnston eb2b8523ae Split out registration to worker
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.

Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
Erik Johnston 88af0317a2 Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-15 22:39:13 +00:00
Amber Brown f5bafd70f4 add cache remover endpoint and wire it up 2019-02-16 04:34:23 +11:00
Erik Johnston eaf4d11af9 Add configurable room list publishing rules
This allows specifying who and what is allowed to be published onto the
public room list
2019-02-14 16:02:23 +00:00
Amber Brown bb4fd8f927
Run `black` on user directory code (#4635) 2019-02-13 23:05:32 +11:00
Erik Johnston 341c35614a Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes 2019-02-13 10:29:31 +00:00
Erik Johnston cf82338930
Merge pull request #4627 from matrix-org/erikj/user_ips_analyze
Analyze user_ips before running deduplication
2019-02-12 13:05:09 +00:00
Erik Johnston 495ea92350 Fix pep8 2019-02-12 12:40:42 +00:00
Erik Johnston 483ba85c7a Analyze user_ips before running deduplication
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
2019-02-12 11:55:27 +00:00
Erik Johnston 362d80b770 Reduce user_ips bloat during dedupe background update
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.

This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.
2019-02-12 11:39:34 +00:00
Erik Johnston 719e073f00
Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating
add updating of backup versions
2019-02-11 13:45:49 +00:00
Richard van der Hoff 56710c7df5
Fix 'no unique or exclusion constraint' error (#4591)
Add more tables to the list of tables which need a background update to
complete before we can upsert into them, which fixes a race against the
background updates.
2019-02-08 18:30:46 +00:00
Hubert Chathi 8248637173 add new endpoint to update backup versions 2019-02-06 17:57:10 -05:00
Andrew Morgan 627ecd358e Filter user directory state query to a subset of state events (#4462)
* Filter user directory state query to a subset of state events

* Add changelog
2019-02-05 12:16:28 +00:00
Andrew Morgan c45fd0dda0
Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable
Copy over non-federatable trait on room upgrade
2019-01-31 20:28:37 +00:00
Richard van der Hoff b60d005156 Merge branch 'develop' into matrix-org-hotfixes 2019-01-31 18:44:04 +00:00
Richard van der Hoff 6c232a69df Revert "Break infinite loop on redaction in v3 rooms"
We've got a better fix of this now.

This reverts commit decb5698b3.
2019-01-31 18:43:49 +00:00
Andrew Morgan d239f67c25 Raise an exception instead of returning None 2019-01-31 18:34:15 +00:00
Andrew Morgan 3ed3cb4339 New function for getting room's create event 2019-01-31 18:21:39 +00:00
Richard van der Hoff e707e7b38d
Fix infinite loop when an event is redacted in a v3 room (#4535) 2019-01-31 15:34:17 +00:00
Richard van der Hoff decb5698b3 Break infinite loop on redaction in v3 rooms 2019-01-31 00:23:58 +00:00