f431ff3fb8
Require event format version to parse or create events
2019-01-23 20:21:33 +00:00
9b827c40ca
Log some bits about event creation ( #4121 )
...
I found these helpful in debugging my room upgrade tests.
2018-10-31 15:42:23 +00:00
cb53ce9d64
Refactor state group lookup to reduce DB hits ( #4011 )
...
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
e7a16c6210
Remove redundant run_as_background_process() from pusherpool
...
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
2018-10-22 16:12:11 +01:00
8174c6725b
Avoid reraise, to improve stacktraces
2018-10-01 18:50:34 +01:00
c144252a8c
Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates
...
Fix logcontexts for running pushers
2018-08-17 16:21:49 +01:00
66f7dc8c87
Fix logcontexts for running pushers
...
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.
Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
372bf073c1
block event creation and room creation on hitting resource limits
2018-08-16 21:25:16 +01:00
2f78f432c4
speed up /members and add at= and membership params ( #3568 )
2018-08-15 16:35:22 +01:00
b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
536bc63a4e
Merge branch 'develop' into erikj/client_apis_move
2018-07-24 09:57:05 +01:00
0faa3223cd
Fix missing attributes on workers.
...
This was missed during the transition from attribute to getter for
getting state from context.
2018-07-23 16:28:00 +01:00
0b0b24cb82
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move
2018-07-23 13:21:15 +01:00
e42510ba63
Use new getters
2018-07-23 13:17:22 +01:00
5c88bb722f
Move PaginationHandler to its own file
2018-07-20 15:32:23 +01:00
0ecf68aedc
Move check_in_room_or_world_readable to Auth
2018-07-20 15:30:59 +01:00
7c712f95bb
Combine Limiter and Linearizer
...
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
2018-07-20 13:11:43 +01:00
8462c26485
Improvements to the Limiter
...
* give them names, to improve logging
* use a deque rather than a list for efficiency
2018-07-20 12:50:27 +01:00
8cb8df55e9
Split MessageHandler into read only and writers
...
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
2018-07-18 15:33:03 +01:00
ea752bdd99
s/becuase/because/g
2018-07-10 17:58:18 +01:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
0269367f18
allow non-consented users to still part rooms (to let us autopart them)
2018-06-25 17:56:10 +01:00
07cad26d65
Remove all global reactor imports & pass it around explicitly ( #3424 )
2018-06-25 14:08:28 +01:00
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
a61738b316
Remove run_on_reactor ( #3395 )
2018-06-14 18:27:37 +10:00
4a9cbdbc15
Exempt AS-registered users from doing gdpr
2018-05-29 19:54:32 +01:00
17a70cf6e9
Misc. py3 fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
a0b3946fe2
Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri
2018-05-23 16:06:03 +01:00
3ff6f50eac
Use the localpart in the consent uri
...
... because it's shorter.
2018-05-23 15:28:23 +01:00
82191b08f6
Support for putting %(consent_uri)s in messages
...
Make it possible to put the URI in the error message and the server notice that
get sent by the server
2018-05-23 15:24:31 +01:00
a5e2941aad
Reject attempts to send event before privacy consent is given
...
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252 ) if consent is not yet
given.
2018-05-22 12:00:47 +01:00
5f27ed75ad
Make purge_history operate on tokens
...
As we're soon going to change how topological_ordering works
2018-05-15 16:23:50 +01:00
ca7211104e
Merge branch 'release-v0.28.1' into develop
2018-05-01 18:16:57 +01:00
33f469ba19
Apply some limits to depth to counter abuse
...
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth
As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
2018-05-01 17:54:19 +01:00
453adf00b6
pep8; remove spurious import
2018-04-27 14:32:08 +01:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
41d4b07a53
Merge pull request #3142 from matrix-org/rav/reraise
...
reraise exceptions more carefully
2018-04-27 12:16:19 +01:00
6493b22b42
reraise exceptions more carefully
...
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
2018-04-27 11:40:06 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
f8e8ec013b
Note why we're limiting concurrent event sends
2018-04-10 14:00:46 +01:00
e5082494eb
Limit concurrent event sends for a room
2018-04-09 12:07:39 +01:00
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
d0fcc48f9d
extra_users is actually a list of UserIDs
2018-03-13 11:20:06 +00:00
e48c7aac4d
Add transactional API to history purge
...
Make the purge request return quickly, and allow scripts to poll for updates.
2018-03-12 16:22:55 +00:00
1708412f56
Return an error when doing two purges on a room
...
Queuing up purges doesn't sound like a good thing.
2018-03-12 16:22:54 +00:00
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
f381d63813
Check event auth on the worker
2018-03-01 10:18:37 +00:00
6b8604239f
Correctly send ratelimit and extra_users params
2018-03-01 10:08:39 +00:00
f756f961ea
Fixup comments
2018-03-01 10:05:27 +00:00
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
50fe92cd26
Move presence handling into handle_new_client_event
...
As we want to have it run on the main synapse instance
2018-02-07 10:34:09 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
e3624fad5f
Remove pointless ratelimit check
...
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
2018-02-07 10:30:25 +00:00
3e1e69ccaf
Update copyright
2018-02-06 16:40:38 +00:00
770b2252ca
s/_create_new_client_event/create_new_client_event/
2018-02-06 16:40:30 +00:00
5ff3d23564
Split event creation into a separate handler
2018-02-05 16:01:48 +00:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
b78bae2d51
fix isinstance
2017-10-11 14:49:09 +01:00
c3b7a45e84
Allow error strings from spam checker
2017-10-11 14:39:22 +01:00
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
bc7f3eb32f
Merge pull request #2483 from jeremycline/unfreeze-ujson-dump
...
Unfreeze event before serializing with ujson
2017-10-11 13:18:52 +01:00
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
cafb8de132
Unfreeze event before serializing with ujson
...
In newer versions of https://github.com/esnme/ultrajson , ujson does not
serialize frozendicts (introduced in esnme/ultrajson@53f85b1 ). Although
the PyPI version is still 1.35, Fedora ships with a build from commit
esnme/ultrajson@2f1d487 . This causes the serialization to fail if the
distribution-provided package is used.
This runs the event through the unfreeze utility before serializing it.
Thanks to @ignatenkobrain for tracking down the root cause.
fixes #2351
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
2017-09-30 11:22:37 -04:00
8090fd4664
Fix /joined_members to work with AS users
2017-09-28 10:09:32 +01:00
adec03395d
Fix bug where /joined_members didn't check user was in room
2017-09-27 15:14:39 +01:00
6cd5fcd536
Make the spam checker a module
2017-09-26 19:20:23 +01:00
5ed109d59f
PoC for filtering spammy events ( #2456 )
...
Demonstration of how you might add some hooks to filter out spammy events.
2017-09-19 12:20:11 +01:00
bf81f3cf2c
Split out profile handler to fix tests
2017-08-25 14:34:56 +01:00
5a7f561a9b
Fix bug where pusherpool didn't start and broke some rooms
...
Since we didn't instansiate the PusherPool at start time it could fail
at run time, which it did for some users.
This may or may not fix things for those users, but it should happen at
start time and stop the server from starting.
2017-07-06 17:55:51 +01:00
1bce3e6b35
Remove unused variables
2017-06-26 14:03:27 +01:00
fcf01dd88e
Reject local events that don't round trip the DB
2017-06-19 11:33:40 +01:00
cafe659f72
Store ActionGenerator in HomeServer
2017-05-19 10:09:56 +01:00
66d8ffabbd
Faster push rule calculation via push specific cache
...
We add a push rule specific cache that ensures that we can reuse
calculated push rules appropriately when a user join/leaves.
2017-05-17 16:55:40 +01:00
b990b2fce5
Add per user ratelimiting overrides
2017-05-10 11:05:43 +01:00
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
34e682d385
Fix invite state to always include all events
2017-04-26 16:18:08 +01:00
7eb9f34cc3
Remove spurious yield
...
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
2017-04-03 15:44:19 +01:00
10e48d8310
Don't clobber a displayname or avatar_url if provided by an m.room.member event
2017-01-24 18:06:07 +00:00
3b4de17d2b
Comment
2017-01-10 14:05:53 +00:00
586f474a44
Don't block messages sending on bumping presence
2017-01-10 12:46:00 +00:00
d53a80af25
Merge pull request #1620 from matrix-org/erikj/concurrent_room_access
...
Limit the number of events that can be created on a given room concurrently
2016-12-12 10:30:23 +00:00
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
64038b806c
Comments
2016-11-11 10:42:08 +00:00
2bd4513a4d
Limit the number of events that can be created on a given room concurretnly
2016-11-10 16:44:35 +00:00
a4632783fb
Sample correctly
2016-11-08 11:20:26 +00:00
eeda4e618c
Limit the number of prev_events of new events
2016-11-08 11:02:29 +00:00
d04e2ff3a4
Fix incredubly slow back pagination query
...
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
2016-10-24 13:35:51 +01:00
f2f74ffce6
Comment
2016-10-19 14:21:28 +01:00
550308c7a1
Check whether to ratelimit sooner to avoid work
2016-10-19 10:45:24 +01:00
87528f0756
Support /initialSync in synchrotron worker
2016-09-21 11:46:28 +01:00
69a2d4e38c
Use get_joined_users_from_context instead of manually looking up hosts
2016-09-05 13:44:40 +01:00
c10cb581c6
Correctly handle the difference between prev and current state
2016-08-31 14:26:22 +01:00
a3dc1e9cbe
Replace context.current_state with context.current_state_ids
2016-08-25 17:32:22 +01:00
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
a98d215204
Add filter param to /messages API
2016-07-14 16:30:56 +01:00
8f8798bc0d
Add ReadWriteLock for pagination and history prune
2016-07-05 15:30:25 +01:00
2d21d43c34
Add purge_history API
2016-07-05 10:28:51 +01:00
4c04222fa5
Poke notifier on next reactor tick
2016-06-03 11:24:16 +01:00
b153f5b150
Merge pull request #787 from matrix-org/markjh/liberate_presence_handler
...
Move the presence handler out of the Handlers object
2016-05-17 10:09:43 +01:00
1a3a2002ff
Spell "domain" correctly
...
s/domian/domain/g
2016-05-16 19:17:23 +01:00
3b86ecfa79
Move the presence handler out of the Handlers object
2016-05-16 18:56:37 +01:00
7e23476814
move filter_events_for_client out of base handler
2016-05-11 13:42:37 +01:00
1620578b13
Shuffle when we get the signing_key attribute.
...
Wait until we sign a message to get the signing key from the homeserver
config. This means that the message handler can be created without
having a signing key in the config which means that separate processes
like the pusher that don't send messages and don't need to sign them can
still access the handlers.
2016-05-11 12:20:57 +01:00
30057b1e15
Move _create_new_client_event and handle_new_client_event out of base handler
2016-05-11 09:09:20 +01:00
424a7f48f8
Run filter_events_for_client
...
so we don't accidentally mail out events people shouldn't see
2016-04-27 17:50:49 +01:00
7e2f971c08
Remove some unused functions ( #711 )
...
* Remove some unused functions
* get_room_events_stream is only used in tests
* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
86be915cce
Call profile handler get_displayname directly rather than using collect_presencelike_data
2016-04-07 18:11:49 +01:00
d76d89323c
Use computed prev event ids
2016-04-01 17:39:32 +01:00
8d73cd502b
Add concurrently_execute function
2016-04-01 14:06:00 +01:00
b7a3be693b
Merge pull request #618 from matrix-org/markjh/pushrule_stream
...
Add a stream for push rule updates
2016-03-04 16:35:08 +00:00
13cbd31040
Spelling
2016-03-04 15:22:39 +00:00
ec7460b4f2
Merge branch 'develop' into markjh/pushrule_stream
2016-03-04 14:44:34 +00:00
ed61a49169
Add profile information to invites
2016-03-04 14:35:02 +00:00
b4022cc487
Pass whole requester to ratelimiting
...
This will enable more detailed decisions
2016-03-03 16:43:42 +00:00
ddf9e7b302
Hook up the push rules to the notifier
2016-03-03 14:57:45 +00:00
e5ad2e5267
Merge pull request #582 from matrix-org/erikj/presence
...
Rewrite presence for performance.
2016-02-19 09:37:50 +00:00
f8d21e1431
Review comments
2016-02-18 11:02:14 +00:00
e5999bfb1a
Initial cut
2016-02-17 15:40:50 +00:00
d1fb790818
Some cleanup
2016-02-16 14:25:23 +00:00
4bfb32f685
Branch off member and non member sends
...
Unclean, needs tidy-up, but works
2016-02-15 18:21:30 +00:00
e71095801f
Merge implementation of /join by alias or ID
...
This code is kind of rough (passing the remote servers down a long
chain), but is a step towards improvement.
2016-02-15 15:39:16 +00:00
4e7948b47a
Allow paginating backwards from stream token
2016-01-28 11:52:34 +00:00
5cba88ea7c
Make it possible to paginate forwards from stream tokens
...
In order that we can fill the gap after a /sync, make it possible to paginate
forwards from a stream token.
2016-01-27 17:42:45 +00:00
da417aa56d
Allow non-guests to peek on rooms using /events
2016-01-20 15:34:07 +00:00
ac5a4477ad
Require unbanning before other membership changes
2016-01-15 16:27:26 +00:00
2ef6de928d
Skip, rather than erroring, invalid guest requests
...
Erroring causes problems when people make illegal requests, because they
don't know what limit parameter they should pass.
This is definitely buggy. It leaks message counts for rooms people don't
have permission to see, via tokens. But apparently we already
consciously decided to allow that as a team, so this preserves that
behaviour.
2016-01-05 18:12:37 +00:00
517fb9a023
Move the doc string to the public facing method
2015-12-22 18:53:47 +00:00
9ac417fa88
Add a cache for initialSync responses that expires after 5 minutes
2015-12-22 18:27:56 +00:00
491f3d16dc
Make state updates in the C+S API idempotent
2015-12-02 15:50:50 +00:00
5eb4d13aaa
Fix typo in collect_presencelike_data
2015-12-02 10:50:58 +00:00
a9526831a4
Wrap calls to distributor.fire in appropriately named functions so that static analysis can work out want is calling what
2015-12-01 20:53:04 +00:00
95f30ecd1f
Add API for setting account_data globaly or on a per room basis
2015-12-01 18:41:32 +00:00
924d85a75e
Merge pull request #375 from matrix-org/daniel/guestroominitialsync
...
Clean up room initialSync for guest users
2015-11-23 16:10:49 +00:00
1c960fbb80
s/private_user_data/account_data/
2015-11-18 15:31:04 +00:00
641420c5e0
Clean up room initialSync for guest users
2015-11-13 15:44:57 +00:00
06bfd0a3c0
Merge pull request #367 from matrix-org/daniel/readafterleave
...
Merge pull request #367 from matrix-org/daniel/readafterleave
Tweak guest access permissions
2015-11-12 15:22:02 +00:00
0d08670f61
Merge pull request #360 from matrix-org/daniel/guestroominitialsync
...
Merge pull request #360 from matrix-org/daniel/guestroominitialsync
Allow guest access to room initialSync
2015-11-12 15:19:55 +00:00
fb7e260a20
Tweak guest access permissions
...
* Allow world_readable rooms to be read by guests who have joined and
left
* Allow regular users to access world_readable rooms
2015-11-12 15:02:00 +00:00
50f1afbd5b
Consider joined guest users as joined users
...
Otherwise they're inconveniently allowed to write events to the room
but not to read them from the room.
2015-11-12 13:37:07 +00:00
Erik Johnston