2e4f0b2bd7
Replace the @metrics.counted annotations in federation with specifically-written counters and distributions
2015-03-12 16:24:51 +00:00
c1cdd7954d
Add an .inc_by() method to CounterMetric; implement DistributionMetric a neater way
2015-03-12 16:24:51 +00:00
63cb7ece62
Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer
2015-03-12 16:24:51 +00:00
493e3fa0ca
Don't forbid '_' in metric basenames any more, to allow things like foo_time
2015-03-12 16:24:51 +00:00
f1fbe3e09f
Rename TimerMetric to DistributionMetric; as it could count more than just time
2015-03-12 16:24:51 +00:00
642f725fd7
Pretend the 'getEvent' cache is just another cache in the set of all the others for metric
2015-03-12 16:24:51 +00:00
cbc0406be8
Export CacheMetric as hits+total, rather than hits+misses, as it's easier to derive hit ratio from that
2015-03-12 16:24:51 +00:00
1748605c5d
Count incoming HTTP requests per servlet that responds
2015-03-12 16:24:51 +00:00
4d661ec0f3
Remember to emit final linefeed from /metrics page, or Prometheus gets upset
2015-03-12 16:24:51 +00:00
0e847540c3
Prometheus needs "escaped" label values
2015-03-12 16:24:51 +00:00
22b37b75db
Kill unused CounterMetric.fetch() method
2015-03-12 16:24:51 +00:00
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
0b96bb793e
Have all @metrics.counted use a single metric name vectored on the method name, rather than a brand new scalar counter per counted method
2015-03-12 16:24:51 +00:00
b3a0179d64
Bugfix to rendering output of vectored TimerMetrics
2015-03-12 16:24:51 +00:00
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
399689dcc7
Provide some process resource usage metrics
2015-03-12 16:24:51 +00:00
fa319a5786
Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely
2015-03-12 16:24:51 +00:00
6d146e15df
Put some gauge metrics on the number of notifier listeners, and notified-on objects (users, rooms, appservices)
2015-03-12 16:24:51 +00:00
25187ab674
Collect per-SQL-verb timer stats on query execution time
2015-03-12 16:24:50 +00:00
f52acf3b12
Neater register_* methods on overall Metrics container
2015-03-12 16:24:50 +00:00
a99d6edc05
Neater implementation of metric render methods by pulling out 'render' as a base method that calls self.render_item
2015-03-12 16:24:50 +00:00
72625f2f4d
Initial hack at a TimerMetric; for storing counts + duration accumulators
2015-03-12 16:24:50 +00:00
e1a7e3564f
Delete a couple of TODO markers of monitoring stats now done
2015-03-12 16:24:50 +00:00
094803cf82
Put vector gauges on transaction queue pending PDU and EDU dicts
2015-03-12 16:24:50 +00:00
e9c4b0d178
Ensure that /_synapse/metrics response is UTF-8 encoded
2015-03-12 16:24:50 +00:00
23ab0c68c2
Implement vector CallbackMetrics
2015-03-12 16:24:50 +00:00
849300bc73
Neater introspection methods on BaseMetric so that subclasses don't need to touch self.keys directly
2015-03-12 16:24:50 +00:00
8664599af7
Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache
2015-03-12 16:24:50 +00:00
e02cc249da
Ensure that exceptions while rendering individual metrics don't stop others from being rendered anyway - especially useful for CallbackMetric
2015-03-12 16:24:50 +00:00
59c448f074
Add a scalar gauge metric on the size of the presence user cachemap
2015-03-12 16:24:50 +00:00
d8caa5454d
Initial attempt at a scalar callback-based metric to give instantaneous snapshot gauges
2015-03-12 16:24:50 +00:00
b0cdf097f4
Sprinkle some CacheCounterMetrics around the synapse.storage layer
2015-03-12 16:24:50 +00:00
ce8b5769f7
Create the concept of a cachecounter metric; generating two counters specific to caches
2015-03-12 16:24:50 +00:00
7d72e44eb9
Add vector counters to HTTP clients and servers; count the requests by method and responses by method and response code
2015-03-12 16:24:50 +00:00
c53ec53d80
Pull out all uses of the underlying HTTP user agent .request() method into a single wrapper function, to make adding metrics easier
2015-03-12 16:24:50 +00:00
9470412316
Initial attempt at sprinkling some @metrics.counted decorations around the federation code
2015-03-12 16:24:50 +00:00
a594087f06
Have the MetricsResource actually render metric counters
2015-03-12 16:24:50 +00:00
74bc42cfdd
An initial implementation of a 'metrics' instance, similar to a 'logger' for keeping counter stats on method calls
2015-03-12 16:24:50 +00:00
120b689284
Delete pointless (and unreachable) __init__ method from FederationClient
2015-03-12 16:24:50 +00:00
e7420a3bef
Initial tiny attempt at (vectorable) counter metrics
2015-03-12 16:24:50 +00:00
e07fc62833
A trivial 'hello world'-style resource on /_synapse/metrics, with optional commandline flag
2015-03-12 16:24:50 +00:00
5b6e11d560
Commandline option to enable metrics system
2015-03-12 16:24:50 +00:00
211c14c391
No need to explicitly pass 'web_client' in to create_resource_tree as it can be found via config
2015-03-12 16:24:50 +00:00
ad5701f50f
Expose 'config' as a real HomeServer dependency key
2015-03-12 16:24:50 +00:00
c92fdf88a3
Log the matching push rule.
2015-03-11 22:17:31 +00:00
a7a28f85ae
Appease pep8
2015-03-11 17:32:43 +00:00
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
cdb2e045ee
Again, underscore, not hyphen
2015-03-11 14:22:35 +00:00
465354ffde
'false' is not False
2015-03-11 11:24:50 +00:00
83b1e7fb3c
PEP8 blank lines
2015-03-11 10:01:17 +00:00
04f8478aaa
Add the master push rule for the break-my-push button. Allow server default rules to be disabled by default.
2015-03-10 17:26:25 +00:00
8916acbc13
These aren't defined for redacted events so don't crash
2015-03-10 11:21:37 +00:00
abaf47bbb6
Fix bug in logging.
2015-03-10 10:28:29 +00:00
045afd6b61
in_thread takes no arguments
2015-03-10 10:19:03 +00:00
98b867f7b7
Fix bug in logging.
2015-03-10 10:16:09 +00:00
db1fbc6c6f
Fix remaining scheduler bugs. Add more informative logging.
2015-03-10 10:04:20 +00:00
e84fe3599b
Merge pull request #105 from matrix-org/erikj-perf
...
Add a Twisted Service to synapse.app.homeserver
2015-03-10 10:02:26 +00:00
c37eceeb9e
Split out the 'run' from 'setup'
2015-03-10 09:58:33 +00:00
b8a6692657
Add documentation. When starting via twistd respect soft_file_limit config option.
2015-03-10 09:39:42 +00:00
7e0bba555c
Remove unused import
2015-03-09 17:48:37 +00:00
04c9751f24
Bug fixes whilst putting it all together
2015-03-09 17:45:41 +00:00
019422ebba
Merge pull request #101 from matrix-org/neaten-federation-servlets
...
Neaten federation servlets
2015-03-09 17:39:06 +00:00
b98cd03193
Use event IDs instead of dumping event content in the txns table.
2015-03-09 17:25:20 +00:00
21fd84dcb8
Use seconds; start gluing in the AS scheduler into the AS handler.
2015-03-09 17:01:19 +00:00
8e28db5cc9
Change room handlers get_rooms_for_user to get_joined_rooms_for_user. This uses the a storage api that is cached.
2015-03-09 16:43:09 +00:00
0a60bbf4fa
Finish appservice txn storage impl and tests.
2015-03-09 15:53:03 +00:00
1ead1caa18
Implement create_appservice_txn with tests.
2015-03-09 13:54:20 +00:00
f31e65ca8b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj-perf
2015-03-09 13:29:41 +00:00
1c2dcf762a
Partially implement txn store methods with tests.
2015-03-09 13:10:31 +00:00
1df3ccf7ee
D'oh: underscore, not hyphen
2015-03-09 12:39:56 +00:00
118c883429
Call notifications should be override else they'll get clobbered by sender/room rules.
2015-03-06 19:41:36 +00:00
406d32f8b5
Start implementing ApplicationServiceTransactionStore
2015-03-06 17:35:14 +00:00
34ce2ca62f
Merge branch 'develop' into application-services-txn-reliability
2015-03-06 17:28:49 +00:00
4a6afa6abf
Assign the AS ID from the database; replace old placeholder txn id.
2015-03-06 17:27:55 +00:00
01c099d9ef
Add appservice txns sql schema
2015-03-06 17:16:47 +00:00
64345b7559
Upper bound the backoff.
2015-03-06 16:41:19 +00:00
5d43eaed61
Merge branch 'develop' into release-v0.8.0
2015-03-06 16:25:19 +00:00
9ccccd4874
When setting display name more graciously handle failures to update room state.
2015-03-06 16:24:05 +00:00
2602ddc379
Apply clarity and docstrings
2015-03-06 16:16:14 +00:00
0354659f9d
Finish synapse.appservice.scheduler implementation.
...
With tests to assert behaviour. Not hooked up yet. Stub datastore methods
not implemented yet.
2015-03-06 16:09:05 +00:00
be9dafcd37
Dial down logging for failed pushers
2015-03-06 15:32:38 +00:00
7d3491c741
Add some loggers
2015-03-06 15:17:50 +00:00
96eda876a4
Specify when we don't want to highlight
2015-03-06 15:12:37 +00:00
f260cb72cd
Flesh out more stub functions.
2015-03-06 15:12:24 +00:00
e7d7152c3c
Remove the fallback rule - we probably don't want to be notifying for everything even if we don't know what it is.
2015-03-06 15:03:34 +00:00
141ec04d19
Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions.
2015-03-06 14:53:35 +00:00
5ecc768970
Add attribute so push gateways can tell if a member event is about the user in question
2015-03-06 14:41:50 +00:00
0fbfe1b08a
Add more tests; fix bugs.
2015-03-06 14:36:52 +00:00
369449827d
Bump version
2015-03-06 14:24:53 +00:00
c54773473f
Merge branch 'master' of github.com:matrix-org/synapse into develop
2015-03-06 14:23:41 +00:00
b102a87348
Merge pull request #96 from matrix-org/pushrules2
...
Evolution of push rules
2015-03-06 14:20:04 +00:00
cf66ddc1b4
Schema change as delta in v14
2015-03-06 14:11:49 +00:00
c06b45129c
Add more server default rules so we have default rules for whether you get notifs for invites / random member events
2015-03-06 11:50:51 +00:00
192e228a98
Start adding some tests
2015-03-06 11:50:27 +00:00
e49d6b1568
Unused import
2015-03-06 11:37:24 +00:00
657a0d2568
Comment typo
2015-03-06 11:34:30 +00:00
3ce8540484
Don't look for an TLS private key if we have set --no-tls
2015-03-06 11:34:06 +00:00
1487bba226
Suppress notices should trump content/room/sender rules.
2015-03-06 10:27:32 +00:00
83d31144eb
Add the highlight tweak where messages should be highlighted a different colour in appropriate clients.
2015-03-06 10:26:08 +00:00
d516d68b29
Rejig structure given the appservice_handler already filters the correct ASes to use.
2015-03-06 10:25:50 +00:00
130df8fb01
Add some randomness to the user specified timeout on event streams to mitigate against thundering herds problems
2015-03-06 10:25:36 +00:00
d79d91a4a7
Appease pep8
2015-03-05 20:55:40 +00:00
5eab2549ab
Append a $ on PATH at registration time, meaning each PATH attribute doesn't need it
2015-03-05 20:36:05 +00:00
7644cb79b2
Slightly neater(?) arrangement of authentication wrapper for HTTP servlet methods
2015-03-05 20:33:16 +00:00
ba8ac996f9
Remove the dead 'rate_limit_origin' method from TransportLayerServer
2015-03-05 19:43:17 +00:00
a901ed16b5
Move federation API responding code out of weird mix of lambdas into Servlet-style methods on instances
2015-03-05 19:10:57 +00:00
0c838f9f5e
Minor tweaks
2015-03-05 17:45:52 +00:00
773cb3b688
Add stub architecture for txn reliability.
2015-03-05 17:35:07 +00:00
5b5c7a28d6
Log error message when we fail to fetch remote server keys
2015-03-05 17:09:13 +00:00
12bcf3d179
Merge pull request #100 from matrix-org/missing_pdu_compat
...
Handle if get_missing_pdu returns 400 or not all events.
2015-03-05 16:42:15 +00:00
9708f49abf
Docs
2015-03-05 16:35:16 +00:00
96fee64421
Remove unecessary check
2015-03-05 16:31:47 +00:00
39aa968a76
Respect min_depth argument
2015-03-05 16:31:32 +00:00
6dfd8c73fc
Docs.
2015-03-05 16:31:13 +00:00
e319071191
Add stub scheduler module for txn reliability
2015-03-05 16:30:33 +00:00
9d9d39536b
Slightly reduce the insane amounts of indentation in main http server response path, by 'continue'ing around a non-match or falling through
2015-03-05 16:24:13 +00:00
ae702d161a
Handle if get_missing_pdu returns 400 or not all events.
2015-03-05 16:08:02 +00:00
be09c23ff0
Add txn_id kwarg to push methods
2015-03-05 15:40:07 +00:00
dc4b774f1e
Rename rooms_to_listeners to room_to_listeners, for consistency with user_ and appservice_*
2015-03-05 14:30:20 +00:00
027fd1242c
Give LruCache a __len__, so that len(cache) works
2015-03-04 17:32:28 +00:00
590b544f67
Add default rule to suppress notices.
2015-03-04 15:29:02 +00:00
ed72fc3a50
Merge branch 'develop' into pushrules2
...
Conflicts:
synapse/storage/schema/pusher.sql
2015-03-04 15:24:21 +00:00
d56c01fff4
Note that we don't specify execution order
2015-03-04 15:10:05 +00:00
17d319a20d
s/schema_deltas/applied_schema_deltas/
2015-03-04 15:06:22 +00:00
92b3dc3219
Merge branch 'develop' into pushrules2
2015-03-04 14:56:41 +00:00
5681264faa
s/%r/%s/
2015-03-04 14:21:53 +00:00
f701197227
Add example directory structures in doc
2015-03-04 14:20:14 +00:00
2a45f3d448
Use if not results rather than len, as per feedback.
2015-03-04 14:17:59 +00:00
16dd87d848
Don't assume db conn is a Context Manager.
...
Twisted adbapi wrapped connections aren't context managers.
2015-03-04 14:03:41 +00:00
5eefd1f618
Add unique constraint on schema_version.lock schema. Use conflict clause in sql.
2015-03-04 13:52:18 +00:00
b4c38738f4
Change to use logger in db upgrade script
2015-03-04 13:43:35 +00:00
640e53935d
Use context manager with db conn to correctly commit and rollback
2015-03-04 13:43:17 +00:00
8c8354e85a
Actually add full_schemas dir
2015-03-04 13:34:38 +00:00
c3530c3fb3
More docs. Rename 'schema/current' to 'schema/full_schemas'
2015-03-04 13:34:11 +00:00
811355ccd0
Add some docs and remove unused variables
2015-03-04 13:11:01 +00:00
82b34e813d
SYN-67: Finish up implementing new database schema management
2015-03-04 12:04:19 +00:00
527e0c43a5
Bump version
2015-03-03 14:49:34 +00:00
ede89ae3b4
Also bump version of downloaded syweb
2015-03-03 14:49:19 +00:00
da877aad15
Bump syweb dependency
2015-03-03 13:31:50 +00:00
8d33adfbbb
SYN-67: Begin changing the way we handle schema versioning
2015-03-02 18:23:55 +00:00
6fab7bd2c1
s/user_name/user/ as per mjark's comment
2015-03-02 18:17:19 +00:00
09f9e8493c
Oops, missed a replacement.
2015-03-02 17:37:22 +00:00
9f03553f48
Add missing comma
2015-03-02 16:38:40 +00:00
b41dc68773
We purposefully don't have a version 14 delta script.
2015-03-02 16:36:19 +00:00
20436cdf75
Blank lines
2015-03-02 15:58:12 +00:00
2de5b14fe0
Fix bug which prevented the HS pushing events to the AS due to FrozenEvents
2015-03-02 15:36:37 +00:00
8486910b64
Bump webclient version
2015-03-02 14:57:37 +00:00
8ad024ea80
Merge pull request #93 from matrix-org/application-services-exclusive
...
Application services exclusive flag support
2015-03-02 14:56:32 +00:00
0a036944bd
Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu
2015-03-02 13:53:30 +00:00
3fce185c77
Merge pull request #83 from matrix-org/nofile_limit_config
...
Add config option to set the soft fd limit on start
2015-03-02 13:52:16 +00:00
4195e55ccc
Merge branch 'develop' of github.com:matrix-org/synapse into federation_rate_limit
2015-03-02 13:39:22 +00:00
c3c01641d2
Run deltas and bump user_version in upgrade script
2015-03-02 13:38:57 +00:00
210d3c5d72
Merge pull request #95 from matrix-org/serialize_transaction_processing
...
Process transactions serially.
2015-03-02 13:33:05 +00:00
3077cb2915
Use contextlib.contextmanager instead of a custom class
2015-03-02 13:32:44 +00:00
769f8b58e8
Rename the room-with-two-people rule to be more compatible if we have actual one to one rooms.
2015-03-02 13:28:24 +00:00
33f93d389e
Merge pull request #92 from matrix-org/application-services-event-stream
...
Application services event stream support
2015-03-02 12:02:48 +00:00
29481690c5
If we're yielding don't add errback
2015-03-02 11:50:43 +00:00
23d9bd1d74
Process transactions serially.
...
Since the events received in a transaction are ordered, later events
might depend on earlier events and so we shouldn't blindly process them
in parellel.
2015-03-02 11:39:57 +00:00
9d9b230501
Make the federation server ratelimiting configurable.
2015-03-02 11:33:45 +00:00
cb97ea3ec2
PEP8
2015-03-02 11:23:46 +00:00
377ae369c1
Wrap all of get_app_service_rooms in a txn.
2015-03-02 11:20:51 +00:00
b216b36892
JOIN state_events rather than parsing unrecognized_keys to pull out member state_keys
2015-03-02 10:41:35 +00:00
3d73383d18
Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users.
2015-03-02 10:16:24 +00:00
ebc4830666
PR tweaks: set earlier on and use 'as json' for compat
2015-03-02 09:53:00 +00:00
2a6dedd7cc
It's set_tweak now, not set_sound
2015-02-27 18:38:56 +00:00
0554d07082
Move federation rate limiting out of transport layer
2015-02-27 15:41:52 +00:00
9dc9118e55
Document FederationRateLimiter
2015-02-27 15:16:47 +00:00
58ff066064
Implement exclusive namespace checks.
2015-02-27 13:51:41 +00:00
40c9896705
Add functions to return whether an AS has exclusively claimed a matching namespace.
2015-02-27 11:03:56 +00:00
16b90764ad
Convert expected format for AS regex to include exclusivity.
...
Previously you just specified the regex as a string, now it expects a JSON
object with a 'regex' key and an 'exclusive' boolean, as per spec.
2015-02-27 10:44:32 +00:00
806a6c886a
PEP8
2015-02-27 09:48:57 +00:00
1cc77145d4
Notify appservices of invites mid-poll.
...
This requires the notifier to have knowledge of appservice listeners so it can
do the regex checks on incoming invites to see if the state_key matches. It
isn't enough to just rely on the room listeners and store.get_app_service_rooms
as the room will initially not exist or won't be on the ASes radar due to
having none of its users in the room.
2015-02-27 09:39:12 +00:00
cfac3b7873
SYN-267 Add a fallback rule as an explicit server default rule and make the default dont-notify so you effectively have a "notify for everything else" switch you can turn on and off.
2015-02-26 18:58:14 +00:00
1959088156
Add API for getting/setting enabled-ness of push rules.
2015-02-26 18:07:44 +00:00
f0995436e7
Check for membership invite events correctly.
2015-02-26 17:21:17 +00:00
dcec7175dc
Finish impl to get new events for AS. ASes should now be able to poll /events
2015-02-26 16:23:01 +00:00
93d90765c4
Initial implementation of federation server rate limiting
2015-02-26 16:15:26 +00:00
59362454dd
Must update pending_transactions map before yield'ing
2015-02-26 15:47:35 +00:00
92478e96d6
Finish impl to extract all room IDs an AS may be interested in when polling the event stream.
2015-02-26 14:35:28 +00:00
944003021b
whitespace
2015-02-26 13:43:05 +00:00
94fa334b01
Add enable/disable overlay for push rules (REST API not yet hooked up)
2015-02-25 19:17:07 +00:00
29267cf9d7
PEP8 and pyflakes
2015-02-25 17:42:28 +00:00
978ce87c86
Comment unused variables.
2015-02-25 17:37:48 +00:00
2c79c4dc7f
Fix alias query.
2015-02-25 17:37:14 +00:00
2b8ca84296
Add support for extracting matching room_ids and room_aliases for a given AS.
2015-02-25 17:15:25 +00:00
2d20466f9a
Add stub functions and work out execution flow to implement AS event stream polling.
2015-02-25 15:00:59 +00:00
a025055643
SYWEB-278 Don't allow rules with no rule_id.
2015-02-25 14:02:38 +00:00
255f989c7b
turns uris config options should append since it's a list
2015-02-24 20:57:58 +00:00
443ba4eecc
%s for strings otherwise you end up sending 'u"foo"'
2015-02-24 15:00:12 +00:00
c0aaf9fe76
Merge pull request #89 from matrix-org/registration-fallback
...
Registration fallback
2015-02-24 10:00:33 +00:00
9640510de2
Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased
2015-02-23 18:41:58 +00:00
f53fcbce97
Use cache.pop() instead of a separate membership test + del []
2015-02-23 18:30:45 +00:00
27080698e7
Fix code style warning
2015-02-23 18:19:13 +00:00
74048bdd41
Remove unused import
2015-02-23 18:17:43 +00:00
28d8614f48
Trailing comma
2015-02-23 17:36:37 +00:00
bd84755e64
Merge remote-tracking branch 'origin/develop' into performance-cache-improvements
2015-02-23 17:16:03 +00:00
a09e59a698
Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing
2015-02-23 16:55:57 +00:00
044d813ef7
Use the @cached decorator to implement the destination_retry_timings cache
2015-02-23 16:04:40 +00:00
357fba2c24
RoomMemberStore no longer needs a _user_rooms_cache member
2015-02-23 15:57:41 +00:00
Paul "LeoNerd" Evans