Commit Graph

2025 Commits (191f2e5d5d12cde1145f29f9d3a9aa0f993f4306)

Author SHA1 Message Date
Will Hunt 9e0f5a0ac4 Fix get|set_type_stream_id_for_appservice store functions (#8648) 2020-10-26 15:00:56 +00:00
Richard van der Hoff 9146a8a691
Merge pull request #8572 from matrix-org/rav/cache_hacking/2
Push some deferred wrangling down into DeferredCache
2020-10-21 17:55:04 +01:00
Richard van der Hoff 6d3905c7c7 Add some more tests 2020-10-21 15:39:25 +01:00
Richard van der Hoff 1f4269700c Push some deferred wrangling down into DeferredCache 2020-10-21 15:39:25 +01:00
Richard van der Hoff 7b71695388 Combine the two sets of tests for CacheDescriptor 2020-10-21 15:38:29 +01:00
Will Hunt 70259d8c8c
Limit AS transactions to 100 events (#8606)
* Limit AS transactions to 100 events

* Update changelog.d/8606.feature

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Add tests

* Update synapse/appservice/scheduler.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-10-21 15:36:53 +01:00
Patrick Cloke 20a67aa70d
Separate the TCP and terse JSON formatting code. (#8587)
This should (theoretically) allow for using the TCP code with a different output type
and make it easier to use the JSON code with files / console.
2020-10-21 06:59:54 -04:00
Andrew Morgan 74f29284aa
Remove some extraneous @unittest.INFOs on unit tests (#8592) 2020-10-20 11:49:15 +01:00
Richard van der Hoff 96e7d3c4a0
Fix 'LruCache' object has no attribute '_on_resize' (#8591)
We need to make sure we are readu for the `set_cache_factor` callback.
2020-10-19 21:13:50 +01:00
Richard van der Hoff 903d11c43a
Add `DeferredCache.get_immediate` method (#8568)
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
2020-10-19 15:00:12 +01:00
Patrick Cloke c356b4bf42
Include a simple message in email notifications that include encrypted content (#8545) 2020-10-19 09:12:39 -04:00
Richard van der Hoff 3ee17585cd
Make LruCache register its own metrics (#8561)
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
2020-10-16 15:51:57 +01:00
Richard van der Hoff 5649669c3c
Merge pull request #8535 from matrix-org/rav/third_party_events_updates
Support modifying event content from ThirdPartyRules modules
2020-10-15 20:55:41 +01:00
Jonathan de Jong 6b5a115c0a
Solidify the HomeServer constructor. (#8515)
This implements a more standard API for instantiating a homeserver and
moves some of the dependency injection into the test suite.

More concretely this stops using `setattr` on all `kwargs` passed to `HomeServer`.
2020-10-15 15:29:13 -04:00
Will Hunt c276bd9969
Send some ephemeral events to appservices (#8437)
Optionally sends typing, presence, and read receipt information to appservices.
2020-10-15 12:33:28 -04:00
Richard van der Hoff 0a08cd1065
Merge pull request #8548 from matrix-org/rav/deferred_cache
Rename Cache to DeferredCache, and related changes
2020-10-15 11:42:07 +01:00
Richard van der Hoff 4433d01519
Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_invite
Simplify `_locally_reject_invite`
2020-10-15 10:20:19 +01:00
Richard van der Hoff 470dedd266 Combine the two sets of DeferredCache tests 2020-10-14 23:49:27 +01:00
Richard van der Hoff 4182bb812f move DeferredCache into its own module 2020-10-14 23:38:14 +01:00
Richard van der Hoff 9f87da0a84 Rename Cache->DeferredCache 2020-10-14 23:38:14 +01:00
Erik Johnston 1264c8ac89
Add basic tests for sync/pagination with vector clock tokens. (#8488)
These are tests for #8439
2020-10-14 13:53:20 +01:00
Erik Johnston 921a3f8a59
Fix not sending events over federation when using sharded event persisters (#8536)
* Fix outbound federaion with multiple event persisters.

We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.

Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.

* Change some interfaces to use RoomStreamToken.

By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
2020-10-14 13:27:51 +01:00
Richard van der Hoff a34b17e492 Simplify `_locally_reject_invite`
Update `EventCreationHandler.create_event` to accept an auth_events param, and
use it in `_locally_reject_invite` instead of reinventing the wheel.
2020-10-13 23:58:48 +01:00
Richard van der Hoff 898196f1cc guard against accidental modification 2020-10-13 23:24:50 +01:00
Richard van der Hoff 617e8a4653 Allow ThirdPartyRules modules to replace event content
Support returning a new event dict from `check_event_allowed`.
2020-10-13 23:24:50 +01:00
Richard van der Hoff d9d86c2996 Remove redundant `token_id` parameter to create_event
this is always the same as requester.access_token_id.
2020-10-13 23:06:36 +01:00
Patrick Cloke 629a951b49
Move additional tasks to the background worker, part 4 (#8513) 2020-10-13 08:20:32 -04:00
Erik Johnston b2486f6656
Fix message duplication if something goes wrong after persisting the event (#8476)
Should fix #3365.
2020-10-13 12:07:56 +01:00
Erik Johnston 8de3703d21
Make event persisters periodically announce position over replication. (#8499)
Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress.

This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
2020-10-12 15:51:41 +01:00
Patrick Cloke d35a451399
Clean-up some broken/unused code in the test framework (#8514) 2020-10-09 14:19:29 -04:00
Richard van der Hoff 9789b1fba5
Fix threadsafety in ThreadedMemoryReactorClock (#8497)
This could, very occasionally, cause:

```
tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/src/tests/rest/media/v1/test_media_storage.py", line 86, in test_ensure_media_is_in_local_cache
    self.wait_on_thread(x)
  File "/src/tests/unittest.py", line 296, in wait_on_thread
    self.reactor.advance(0.01)
  File "/src/.tox/py35/lib/python3.5/site-packages/twisted/internet/task.py", line 826, in advance
    self._sortCalls()
  File "/src/.tox/py35/lib/python3.5/site-packages/twisted/internet/task.py", line 787, in _sortCalls
    self.calls.sort(key=lambda a: a.getTime())
builtins.ValueError: list modified during sort

tests.rest.media.v1.test_media_storage.MediaStorageTests.test_ensure_media_is_in_local_cache
```
2020-10-09 17:22:25 +01:00
Andrew Morgan 66ac4b1e34
Allow modules to create and send events into rooms (#8479)
This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room.

The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.
2020-10-09 13:46:36 +01:00
Patrick Cloke c9c0ad5e20
Remove the deprecated Handlers object (#8494)
All handlers now available via get_*_handler() methods on the HomeServer.
2020-10-09 07:24:34 -04:00
Hubert Chathi a97cec18bb
Invalidate the cache when an olm fallback key is uploaded (#8501) 2020-10-08 13:24:46 -04:00
Erik Johnston ae5b2a72c0
Reduce serialization errors in MultiWriterIdGen (#8456)
We call `_update_stream_positions_table_txn` a lot, which is an UPSERT
that can conflict in `REPEATABLE READ` isolation level. Instead of doing
a transaction consisting of a single query we may as well run it outside
of a transaction.
2020-10-07 15:15:57 +01:00
Hubert Chathi 4cb44a1585
Add support for MSC2697: Dehydrated devices (#8380)
This allows a user to store an offline device on the server and
then restore it at a subsequent login.
2020-10-07 08:00:17 -04:00
Richard van der Hoff 43c622885c
Merge pull request #8463 from matrix-org/rav/clean_up_event_handling
Reduce inconsistencies between codepaths for membership and non-membership events.
2020-10-07 12:20:44 +01:00
Richard van der Hoff 4f0637346a
Combine `SpamCheckerApi` with the more generic `ModuleApi`. (#8464)
Lots of different module apis is not easy to maintain.

Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
2020-10-07 12:03:26 +01:00
Hubert Chathi 3cd78bbe9e
Add support for MSC2732: olm fallback keys (#8312) 2020-10-06 13:26:29 -04:00
Richard van der Hoff a024461130
Additional tests for third-party event rules (#8468)
* Optimise and test state fetching for 3p event rules

Getting all the events at once is much more efficient than getting them
individually

* Test that 3p event rules can modify events
2020-10-06 16:31:31 +01:00
Richard van der Hoff 9c0b168cff
Merge pull request #8467 from matrix-org/rav/fix_3pevent_rules
Fix third-party event modules for `check_visibility_can_be_modified` check
2020-10-06 11:32:53 +01:00
Richard van der Hoff 785437dc0d
Update default room version to 6 (#8461)
Per https://github.com/matrix-org/matrix-doc/pull/2788
2020-10-05 21:40:51 +01:00
Richard van der Hoff 4cd1448d0e Fix third-party event modules for `check_visibility_can_be_modified` check
PR #8292 tried to maintain backwards compat with modules which don't provide a
`check_visibility_can_be_modified` method, but the tests weren't being run,
and the check didn't work.
2020-10-05 20:29:52 +01:00
Richard van der Hoff e775b5bb5b kill off `send_nonmember_event`
This is now redundant, and we can just call `handle_new_client_event` directly.
2020-10-05 19:04:10 +01:00
Andrew Morgan 0991a2da93
Allow ThirdPartyEventRules modules to manipulate public room state (#8292)
This PR allows `ThirdPartyEventRules` modules to view, manipulate and block changes to the state of whether a room is published in the public rooms directory.

While the idea of whether a room is in the public rooms list is not kept within an event in the room, `ThirdPartyEventRules` generally deal with controlling which modifications can happen to a room. Public rooms fits within that idea, even if its toggle state isn't controlled through a state event.
2020-10-05 14:57:46 +01:00
Erik Johnston e3debf9682
Add logging on startup/shutdown (#8448)
This is so we can tell what is going on when things are taking a while to start up.

The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
2020-10-02 15:20:45 +01:00
Erik Johnston ec10bdd32b
Speed up unit tests when using PostgreSQL (#8450) 2020-10-02 15:09:31 +01:00
Patrick Cloke 62894673e6
Allow background tasks to be run on a separate worker. (#8369) 2020-10-02 08:23:15 -04:00
Erik Johnston 6c5d5e507e
Add unit test for event persister sharding (#8433) 2020-10-02 09:57:12 +01:00
BBBSnowball 05ee048f2c
Add config option for always using "userinfo endpoint" for OIDC (#7658)
This allows for connecting to certain IdPs, e.g. GitLab.
2020-10-01 13:54:35 -04:00