5019945828
Refactor the Appservice scheduler code
...
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).
Add some docstrings too.
2019-08-20 11:50:23 +01:00
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
52e87fbfbe
Run the AS senders as background processes ( #4189 )
...
This should fix some "Starting db connection from sentinel context" warnings,
and will mean we get metrics for these processes.
2018-12-04 10:53:49 +01:00
2d2828dcbc
Port http/ to Python 3 ( #3771 )
2018-09-06 00:10:47 +10:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
f4f1cda928
add ip_range_whitelist parameter to limit where ASes can connect from
2018-06-28 20:32:00 +01:00
13d211edc1
Merge pull request #3344 from Half-Shot/hs/as-metrics
...
Add metrics to track appservice transactions
2018-06-07 11:37:12 +01:00
592c162516
also redact __str__ of ApplicationService used for logging
2018-06-06 10:35:29 +01:00
d6e3c2c79b
Let's try labels instead of label, that might work
2018-06-05 17:30:45 +01:00
604cff1a06
Add metrics to track appservice transactions
2018-06-05 13:21:30 +01:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
e54c202b81
Replace some type checks with six type checks
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-07 01:02:32 +02:00
7fb0e98b03
Extract group_id from the dict for multiple use
2017-11-16 18:18:30 +00:00
d8391f0541
Remove unused GROUP_ID_REGEX
2017-11-16 18:05:57 +00:00
4e8374856d
Document get_groups_for_user
2017-11-16 18:03:46 +00:00
270f9cd23a
Flake8
2017-11-16 18:03:31 +00:00
9d83d52027
Use a generator instead of a list
2017-11-16 17:57:34 +00:00
b1edf26051
Check group_id belongs to this domain
2017-11-16 17:54:27 +00:00
97bd18af4e
Add automagical AS Publicised Group(s)
...
via registration file "users" namespace:
```YAML
...
namespaces:
users:
- exclusive: true
regex: '.*luke.*'
group_id: '+all_the_lukes:hsdomain'
...
```
This is part of giving App Services their own groups for matching users. With this, ghost users will be given the appeareance that they are in a group and that they have publicised the fact, but _only_ from the perspective of the `get_publicised_groups_for_user` API.
2017-11-16 16:44:55 +00:00
2a7e9faeec
Do logcontexts outside ResponseCache
2017-10-25 15:21:08 +01:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
760625acba
Make get_if_app_services_interested_in_user faster
2017-05-18 16:34:44 +01:00
69efd77749
Add comment
2017-03-29 09:50:05 +01:00
51b156d48a
Cache whether an AS is interested based on members
2017-03-28 13:27:21 +01:00
30f5ffdca2
Remove param and cast at call site
2017-03-28 13:27:21 +01:00
650f0e69f2
Compile the regex's used in ASes
2017-03-28 13:27:21 +01:00
2bf5a47b3e
Rename network_id to instance_id on client side
2016-12-12 16:05:45 +00:00
3727d66a0e
Don't include appservice id
2016-12-06 17:04:26 +00:00
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
5b54d51d1e
Allow Configurable Rate Limiting Per AS
...
This adds a flag loaded from the registration file of an AS that will determine whether or not its users are rate limited (by ratelimit in _base.py). Needed for IRC bridge reasons - see https://github.com/matrix-org/matrix-appservice-irc/issues/240 .
2016-10-18 17:04:09 +01:00
776594f99d
Log if rejecting 3PE query metadata result due to type check
2016-09-09 15:11:31 +01:00
ed44c475d8
Reject malformed 3PE query metadata results earlier in AS API handling code
2016-09-09 15:07:04 +01:00
572acde483
Use None instead of the empty string
...
Change how we validate the 'url' field as a result.
2016-08-30 17:16:00 +01:00
16b652f0a3
Flake8
2016-08-30 16:30:12 +01:00
e82247f990
Allow application services to have an optional 'url'
...
If 'url' is not specified, they will not be pushed for events or queries. This
is useful for bots who simply wish to reserve large chunks of user/alias
namespace, and don't care about being pushed for events.
2016-08-30 16:21:16 +01:00
9459137f1e
Just sprintf the 'kind' argument into uri directly
2016-08-25 18:35:38 +01:00
1294d4a329
Move ThirdPartyEntityKind into api.constants so the expectation becomes that the value is significant
2016-08-25 18:34:47 +01:00
e7af8be5ae
Root the 3PE lookup API within /_matrix/app/unstable instead of at toplevel
2016-08-25 18:06:29 +01:00
d9dcb2ba3a
Merge pull request #1041 from matrix-org/paul/third-party-lookup
...
Extend 3PE lookup APIs for metadata query
2016-08-25 17:06:53 +01:00
adf53f04ce
appease pep8
2016-08-25 16:00:31 +01:00
db7283cc6b
Implement a ResponseCache around 3PE lookup metadata lookups
2016-08-25 15:56:27 +01:00
d0b8d49f71
Kill PROTOCOL_META since I'm not using it any more
2016-08-25 15:45:28 +01:00
5474824975
Actually query over AS API for 3PE lookup metadata
2016-08-25 15:29:36 +01:00
9899824b85
Initial hack at the 3PN protocols metadata lookup API
2016-08-24 12:33:01 +01:00
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
63c19e1df9
Move 3PU/3PL lookup APIs into /thirdparty containing entity
2016-08-24 11:55:57 +01:00
65201631a4
Move validation logic for AS 3PE query response into ApplicationServiceApi class, to keep the handler logic neater
2016-08-18 17:33:56 +01:00
b515f844ee
Avoid so much copypasta between 3PU and 3PL query by unifying around a ThirdPartyEntityKind enumeration
2016-08-18 17:19:55 +01:00
2a91799fcc
Minor syntax neatenings
2016-08-18 16:58:25 +01:00
06964c4a0a
Copypasta the 3PU support code to also do 3PL
2016-08-18 16:09:50 +01:00
434bbf2cb5
Filter 3PU lookups by only ASes that declare knowledge of that protocol
2016-08-18 14:56:02 +01:00
d5bf7a4a99
Merge remote-tracking branch 'origin/develop' into paul/thirdpartylookup
2016-08-18 14:21:01 +01:00
718ffcf8bb
Since empty lookups now return 200/empty list not 404, we can safely log failures as exceptions
2016-08-18 14:18:37 +01:00
b3511adb92
Don't catch the return-value-as-exception that @defer.inlineCallbacks will use
2016-08-18 13:45:18 +01:00
3ec10dffd6
Actually make 3PU lookup calls out to ASes
2016-08-18 00:39:09 +01:00
320dfe523c
Make notify_interested_services faster
2016-08-17 17:20:50 +01:00
fa87c981e1
Thread 3PU lookup through as far as the AS API object; which currently noöps it
2016-08-17 16:17:28 +01:00
f743471380
Change name of metric
2016-08-17 15:05:50 +01:00
b9e888858c
Move Measure block inside loop
2016-08-17 14:52:26 +01:00
7321f45457
Clean up _ServiceQueuer
2016-08-17 12:03:04 +01:00
c626fc576a
Move the AS handler out of the Handlers object.
...
Access it directly from the homeserver itself. It already wasn't
inheriting from BaseHandler storing it on the Handlers object was
already somewhat dubious.
2016-05-31 13:53:48 +01:00
7e2f971c08
Remove some unused functions ( #711 )
...
* Remove some unused functions
* get_room_events_stream is only used in tests
* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
4dec901c76
Cap the time to retry txns to appservices to 8.5 minutes
...
There's been numerous issues with people playing around with their
application service and then not receiving events from their HS for
ages due to backoff timers reaching crazy heights (albeit capped at
< 1 day).
Reduce the max time between pokes to be 8.5 minutes (2^9 secs) which
is quick enough for people to wait it out (avg wait time being 4.25 min)
but long enough to actually give the AS breathing room if it needs it.
2015-10-21 10:10:55 +01:00
f43544eecc
Make the appservice use 'users_in_room' rather than get_room_members since it is cached
2015-05-22 11:01:28 +01:00
813e54bd5b
Fix more AS sender ID thinkos.
...
Specifically, the ASes own user ID wasn't being treated as 'exclusive' so
a human could nab it. Also, the HS would needlessly send user queries to the
AS for its own user ID.
2015-04-01 14:05:24 +01:00
9fa8bda099
Merge branch 'develop' into application-services-registration-script
2015-04-01 10:19:17 +01:00
09cbff174a
Fix thinko whereby events *for the AS specifically* were not passed on.
...
This was caused by not explicitly checking the service.sender field. This
has now been fixed and a regression test has been added.
2015-03-31 16:44:45 +01:00
b59aa74556
Fix tests and missing returns on deferreds.
2015-03-31 11:35:45 +01:00
32206dde3f
Fixes from PR comments
2015-03-26 10:11:52 +00:00
d04fa1f712
Implement ServiceQueuer with tests.
2015-03-16 14:03:16 +00:00
6279285b2a
Replace EventGrouper for ServiceQueuer to move to push-based txns. Fix tests and add stub tests for ServiceQueuer.
2015-03-16 13:15:40 +00:00
c9c444f562
Wrap polling/retry blocks in try/excepts to avoid sending to other ASes breaking permanently should an error occur.
2015-03-16 10:38:02 +00:00
835e01fc70
Minor PR comment tweaks.
2015-03-16 10:16:59 +00:00
db1fbc6c6f
Fix remaining scheduler bugs. Add more informative logging.
2015-03-10 10:04:20 +00:00
04c9751f24
Bug fixes whilst putting it all together
2015-03-09 17:45:41 +00:00
21fd84dcb8
Use seconds; start gluing in the AS scheduler into the AS handler.
2015-03-09 17:01:19 +00:00
4a6afa6abf
Assign the AS ID from the database; replace old placeholder txn id.
2015-03-06 17:27:55 +00:00
64345b7559
Upper bound the backoff.
2015-03-06 16:41:19 +00:00
2602ddc379
Apply clarity and docstrings
2015-03-06 16:16:14 +00:00
0354659f9d
Finish synapse.appservice.scheduler implementation.
...
With tests to assert behaviour. Not hooked up yet. Stub datastore methods
not implemented yet.
2015-03-06 16:09:05 +00:00
7d3491c741
Add some loggers
2015-03-06 15:17:50 +00:00
f260cb72cd
Flesh out more stub functions.
2015-03-06 15:12:24 +00:00
141ec04d19
Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions.
2015-03-06 14:53:35 +00:00
0fbfe1b08a
Add more tests; fix bugs.
2015-03-06 14:36:52 +00:00
192e228a98
Start adding some tests
2015-03-06 11:50:27 +00:00
d516d68b29
Rejig structure given the appservice_handler already filters the correct ASes to use.
2015-03-06 10:25:50 +00:00
0c838f9f5e
Minor tweaks
2015-03-05 17:45:52 +00:00
773cb3b688
Add stub architecture for txn reliability.
2015-03-05 17:35:07 +00:00
e319071191
Add stub scheduler module for txn reliability
2015-03-05 16:30:33 +00:00
be09c23ff0
Add txn_id kwarg to push methods
2015-03-05 15:40:07 +00:00
40c9896705
Add functions to return whether an AS has exclusively claimed a matching namespace.
2015-02-27 11:03:56 +00:00
16b90764ad
Convert expected format for AS regex to include exclusivity.
...
Previously you just specified the regex as a string, now it expects a JSON
object with a 'regex' key and an 'exclusive' boolean, as per spec.
2015-02-27 10:44:32 +00:00
Richard van der Hoff