617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
36c59ce669
Use six.itervalues in some places
...
There's more where that came from
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
87478c5a60
Merge pull request #3061 from NotAFile/add-some-byte-strings
...
Add b prefixes to some strings that are bytes in py3
2018-04-09 23:54:05 +01:00
15e8ed874f
more verbosity in synctl
2018-04-06 09:28:36 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
e4570c53dd
phone home cache size configurations
2018-04-04 16:46:58 +01:00
6168351877
Add b prefixes to some strings that are bytes in py3
...
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style
...
Replace old style error catching with 'as' keyword
2018-04-04 10:45:22 +01:00
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
33f6195d9a
Handle review comments
2018-03-28 14:25:25 +01:00
792d340572
rename stat to future proof
2018-03-28 12:25:02 +01:00
4ceaa7433a
As daemonizing will make a new process, defer call to init.
2018-03-28 12:19:01 +01:00
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
ef520d8d0e
Include coarse CPU and Memory use in stats callbacks.
...
This requires the psutil module, and is still opt-in based on the report_stats
config option.
2018-03-27 17:56:03 +01:00
8cbbfaefc1
404 correctly on missing paths via NoResource
...
fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
2018-03-23 10:32:50 +00:00
2cc9f76bc3
replace old style error catching with 'as' keyword
...
This is both easier to read and compatible with python3 (not that that
matters)
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-15 16:11:17 +01:00
4f28018c83
Register membership/state servlets in event_creator
2018-03-14 14:30:06 +00:00
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer
...
Remove ReplicationLayer and user Client/Server directly
2018-03-13 15:45:08 +00:00
cb9f8e527c
s/replication_client/federation_client/
2018-03-13 13:26:52 +00:00
d023ecb810
Don't build handlers on workers unnecessarily
2018-03-13 11:08:10 +00:00
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
25adde9a04
Factor out get_db_conn to HomeServer base class
...
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
2018-01-26 00:56:49 +00:00
f4d93ae424
Actually make it work
2018-01-12 10:39:27 +00:00
f68e4cf690
Refactor
2018-01-12 10:11:12 +00:00
7cd34512d8
When using synctl with workers, don't start the main synapse automatically
2018-01-11 11:37:39 +00:00
df0f602796
Implement listen_tcp method in remaining workers
...
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
26cd3f5690
Remove logger argument and do not catch replication listener
...
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
ed48ecc58c
Add methods for listening on multiple addresses
...
Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP
and reactor.listenSSL for multiple addresses.
Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:15:48 +01:00
37d1a90025
Allow binds to both :: and 0.0.0.0
...
Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed.
This causes a warning explaining the behaviour.
Configuration changed to match.
See #2232
Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:10:31 +01:00
795b0849f3
Add a comment which might save some confusion
2017-11-24 00:34:56 +00:00
7f14f0ae38
Remove dead sync_callback
...
This is never used; let's remove it to stop confusing things.
2017-11-24 00:32:04 +00:00
68ca864141
Add config option to disable media_repo on main synapse
...
... to stop us doing the cache cleanup jobs on the master.
2017-11-22 16:20:27 +00:00
e1fd4751de
Build MediaRepositoryResource as a homeserver dependency
...
This avoids the scenario where we have four different PreviewUrlResources
configured on a single app, each of which have their own caches and cache
clearing jobs.
2017-11-22 16:19:49 +00:00
d4553818a0
Merge pull request #2627 from matrix-org/rav/custom_rest_endpoints
...
Add a hook for custom rest endpoints
2017-11-02 15:18:37 +00:00
fcdfc911ee
Add a hook for custom rest endpoints
...
Let the user specify custom modules which can be used for implementing extra
endpoints.
2017-11-02 14:36:55 +00:00
6650a07ede
Factor out _configure_named_resource
...
This was a bit of a code vomit, so let's factor it out to preserve some sanity
2017-11-02 14:33:37 +00:00
8b56977b6f
Merge pull request #2586 from matrix-org/rav/frontend_proxy_auth_header
...
Front-end proxy: pass through auth header
2017-10-27 11:01:50 +01:00
54a2525133
Front-end proxy: pass through auth header
...
So that access-token-in-an-auth-header works.
2017-10-26 18:19:01 +01:00
0a5866bec9
Support /keys/upload on /r0 as well as /unstable
...
(So that we can stop riot relying on it in /unstable)
2017-10-26 18:18:23 +01:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
ea87cb1ba5
Make 'affinity' package optional
2017-10-02 18:03:59 +01:00
3fed5bb25f
Move quit_with_error
2017-10-02 17:59:34 +01:00
b5feaa5a49
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_merged
2017-09-19 11:07:45 +01:00
692250c6be
Fix user_dir startup
...
Add missing parameter to _base.start_worker_reactor
2017-08-16 15:11:29 +01:00
10d8b701a1
Allow configuration of CPU affinity
...
Make it possible to set the CPU affinity in the config file, so that we don't
need to remember to do it manually every time.
2017-08-15 17:08:28 +01:00
543c794a76
Factor out common application start
...
We have 10 copies of this code, and I don't really want to update each one
separately.
2017-08-15 17:04:40 +01:00
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
d4d12daed9
Include registration and as stores in frontend proxy
2017-07-07 18:36:45 +01:00
1fc4a962e4
Add a frontend proxy
2017-07-07 18:19:46 +01:00
78cefd78d6
Make workers report to master for user ip updates
2017-06-27 14:58:10 +01:00
8177563ebe
Fix for workers
2017-06-21 13:57:49 +01:00
dfeca6cf40
Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
...
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
5ddd199870
Typo
2017-06-15 10:49:10 +01:00
4b461a6931
Add some more stats
2017-06-15 09:39:39 +01:00
617304b2cf
Fix phone home stats
2017-06-14 19:47:15 +01:00
f87b287291
Merge pull request #2127 from APwhitehat/alreadystarted
...
print something legible if synapse already running
2017-04-27 15:46:53 +01:00
d4040e9e28
Queried CONDITIONAL_REQUIREMENTS
2017-04-18 16:19:48 -05:00
54f9a4cb59
Fixed travis build failure
...
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 01:38:27 -05:00
8e780b113d
web_server_root documentation fix
...
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 00:49:11 -05:00
c1f52a321d
synctl.py: Check if synapse is already running
2017-04-13 18:00:02 +05:30
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
9c712a366f
Move get_presence_list_* to SlaveStore
2017-04-11 16:07:33 +01:00
414522aed5
Move get_interested_parties
2017-04-11 15:33:26 +01:00
28a4649785
Remove HTTP replication APIs
2017-04-11 09:52:11 +01:00
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
449d1297ca
Fix up federation SendQueue and document types
2017-04-07 11:48:33 +01:00
54f59bd7d4
Merge pull request #2107 from HarHarLinks/patch-1
...
fix typo in synctl help
2017-04-07 09:54:37 +01:00
98ce212093
Merge pull request #2103 from matrix-org/erikj/no-double-encode
...
Don't double encode replication data
2017-04-07 09:39:52 +01:00
8a1137ceab
fix typo in synctl help
2017-04-06 17:10:20 +02:00
391712a4f9
Comment
2017-04-06 13:35:00 +01:00
dbf87282d3
Docs
2017-04-06 13:11:21 +01:00
b5cb6347a4
Don't immediately notify the master about users whose syncs have gone away
2017-04-05 13:25:40 +01:00
96b9b6c127
Don't double json encode typing replication data
2017-04-05 11:34:20 +01:00
f10ce8944b
Don't double json encode federation replication data
2017-04-05 11:10:28 +01:00
a5c401bd12
Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
...
Move to using TCP replication
2017-04-05 09:36:21 +01:00
d1d5362267
Add comment
2017-04-04 16:41:03 +01:00
ac66e11f2b
Add the appropriate amount of preserve_fn
2017-04-04 15:22:54 +01:00
3376f16012
Shuffle and comment synchrotron presence
2017-04-04 11:14:16 +01:00
6ce6bbedcb
Move where we ack federation
2017-04-04 11:02:44 +01:00
62b89daac6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server
2017-04-04 09:46:16 +01:00
36c28bc467
Update all the workers and master to use TCP replication
2017-04-03 15:35:52 +01:00
8a240e4f9c
Merge pull request #2078 from APwhitehat/assertuserfriendly
...
add user friendly report of assertion error in synctl.py
2017-03-31 14:41:49 +01:00
3ba2859e0c
Add tcp replication listener type and hook it up
2017-03-30 13:31:10 +01:00
6cdca71079
synctl.py: wait for synapse to stop before restarting ( #2020 )
2017-03-29 19:20:13 +01:00
305d16d612
add user friendly report of assertion error in synctl.py
...
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
2017-03-29 20:41:39 +05:30
48e7697911
Add slave transaction store
2017-03-24 13:05:30 +00:00
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
aac6d1fc9b
PEP8
2017-03-20 13:47:56 +00:00
067b00d49d
Run the reactor with the sentinel logcontext
...
This fixes a class of 'Unexpected logcontext' messages, which were happening
because the logcontext was somewhat arbitrarily swapping between the sentinel
and the `run` logcontext.
2017-03-18 00:07:43 +00:00
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
6957bfdca6
Don't recreate so many sets
2017-03-16 11:54:26 +00:00
d84bd51e95
Refactor logger config for workers
...
- to make it easier to add more config options.
2017-03-10 15:34:01 +00:00
3a46280ca3
Add db functions needed for room initial sync to slave
2017-02-13 11:16:53 +00:00
4ac363a168
Remove debug logging
2017-01-30 14:10:12 +00:00
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
edd6cdfc9a
Restore default bind address
2017-01-10 17:21:41 +00:00
8a0fddfd73
Remove spurious for..else..
2017-01-10 16:30:53 +00:00
702c020e58
Fix check for bind_address
...
The empty string is a valid setting for the bind_address option, so
explicitly check for None here instead.
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-20 01:37:50 +01:00
7dfd70fc83
Add support for specifying multiple bind addresses
...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 21:51:56 +01:00
4d9b5c60f9
Comment
2016-11-23 11:11:41 +00:00
4c79a63fd7
Explicit federation ack
2016-11-23 10:40:44 +00:00
50934ce460
Comments
2016-11-21 16:55:23 +00:00
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
24772ba56e
Respect use_frozen_dicts option in workers
2016-11-08 11:07:18 +00:00
1cc22da600
Set up the process collector during metrics __init__; that way all split-process workers have it
2016-10-27 18:09:34 +01:00
aac13b1f9a
Pass the Metrics group into the process collector instead of having it find its own one; this avoids it needing to import from synapse.metrics
2016-10-27 18:08:15 +01:00
3ae48a1f99
Move the process metrics collector code into its own file
2016-10-19 18:10:24 +01:00
a68ade6ed3
Merge pull request #1162 from larroy/master
...
Use sys.executable instead of hardcoded python. fixes #1161
2016-10-14 21:42:55 +01:00
c57f871184
Use sys.executable instead of hardcoded python. fixes #1161
2016-10-08 23:55:20 +02:00
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
87528f0756
Support /initialSync in synchrotron worker
2016-09-21 11:46:28 +01:00
883df2e983
fix logger for client_reader worker
2016-09-17 14:12:04 +01:00
4be85281f9
Enable state caches on workers
2016-09-15 14:31:22 +01:00
e0b6db29ed
Split out public room list into a worker process
2016-09-14 14:42:51 +01:00
44330a21e9
Comment
2016-09-09 17:22:07 +01:00
7ed5acacf4
Fix up the calls to the notifier for device messages
2016-09-01 18:08:40 +01:00
873f870e5a
Add new direct message storage functions to slave store
2016-08-30 09:40:32 +01:00
689f4cb914
Update comment
2016-08-22 16:17:31 +01:00
c8f9b45bc2
Add usage stats to prometheus monitoring
2016-08-22 15:34:38 +01:00
bcbd74dc5b
Remove log lines
2016-08-18 15:52:10 +01:00
07229bbdae
Add appservice worker
2016-08-18 14:59:55 +01:00
403ecd8a2c
Missed a s/federation reader/media repository/ in a log message
2016-08-18 10:26:15 +01:00
396624864a
Add a media repository worker
2016-08-18 09:38:42 +01:00
784a2d4f2c
Remove broken cache stuff
2016-08-15 11:25:48 +01:00
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
4e1cebd56f
Make synchrotron accept /events
2016-08-12 15:31:44 +01:00
5b5148b7ec
Synced up synchrotron set_state with PresenceHandler set_state
2016-08-11 11:48:30 +01:00
24f36469bc
Add federation /version API
2016-08-05 16:36:07 +01:00
5aa024e501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader
2016-07-29 11:24:56 +01:00
ec8b217722
Add destination retry to slave store
2016-07-28 17:35:53 +01:00
76b89d0edb
Add slace storage functions for public room list
2016-07-28 17:03:40 +01:00
05e7e5e972
Fix flake8 violation
...
Apparently flake8 v3 puts the error on a different line to v2. Easiest way to
make sure that happens is by putting the whole statement on one line :)
2016-07-26 11:59:08 +01:00
aede7248ab
Split out a FederationReader process
2016-07-21 17:37:44 +01:00
66868119dc
Add metrics for psutil derived memory usage
2016-07-20 16:00:21 +01:00
05f1a4596a
Merge branch 'master' into develop
2016-06-23 11:17:48 +01:00
5cc7564c5c
Optionally start or stop workers in synctl.
...
Optionally start or stop an individual worker by passing -w with
the path to the worker config.
Optionally start or stop every worker and the main synapse by
passing -a with a path to a directory containing worker configs.
The "-w" is intended to be used to bounce individual workers proceses.
THe "-a" is intended for when you want to restart all the workers
simultaneuously, for example when performing database upgrades.
2016-06-21 16:38:05 +01:00
13e334506c
Remove the legacy v0 content upload API.
...
The existing content can still be downloaded. The last upload to the
matrix.org server was in January 2015, so it is probably safe to remove
the upload API.
2016-06-21 11:47:39 +01:00
8c75040c25
Fix setting gc thresholds in the workers
2016-06-17 11:48:12 +01:00
a352b68acf
Use worker_ prefixes for worker config, use existing support for multiple config files
2016-06-16 17:29:50 +01:00
364d616792
Access the event_cache_size directly from the server object.
...
This means that the workers can override the event_cache_size
directly without clobbering the value in the main synapse config.
2016-06-16 12:53:15 +01:00
bde13833cb
Access replication_url from the worker config directly
2016-06-16 12:44:40 +01:00
885ee861f7
Inline the synchrotron and pusher configs into the main config
2016-06-16 11:06:12 +01:00
7dbb473339
Add function to load config without generating it
...
Renames ``load_config`` to ``load_or_generate_config``
Adds a method called ``load_config`` that just loads the
config.
The main synapse.app.homeserver will continue to use
``load_or_generate_config`` to retain backwards compat.
However new worker processes can use ``load_config`` to
load the config avoiding some of the cruft needed to generate
the config.
As the new ``load_config`` method is expected to be used by new
configs it removes support for the legacy commandline overrides
that ``load_or_generate_config`` supports
2016-06-09 18:50:38 +01:00
2d1d1025fa
Add gc_threshold to pusher and synchrotron
2016-06-07 16:26:25 +01:00
dded389ac1
Allow setting of gc.set_thresholds
2016-06-07 15:45:56 +01:00
b0932b34cb
Merge pull request #845 from matrix-org/markjh/synchrotron_presence
...
Fix a KeyError in the synchrotron presence
2016-06-06 16:52:27 +01:00
4a5bbb1941
Fix a KeyError in the synchrotron presence
2016-06-06 16:37:12 +01:00
5ef84da4f1
Yield on the sleeps intended to backoff replication
2016-06-06 16:05:28 +01:00
ac9716f154
Fix spelling
2016-06-03 18:10:00 +01:00
8f79084bd4
Add get_presence_list_accepted to the broken caches in synchrotron
2016-06-03 18:03:40 +01:00
48340e4f13
Clear the list of ongoing syncs on shutdown
2016-06-03 15:02:27 +01:00
da491e75b2
Appease flake8
2016-06-03 14:56:36 +01:00
0b3c80a234
Use ClientIpStore to record client ips
2016-06-03 14:55:01 +01:00
80aade3805
Send updates to the syncing users every ten seconds or immediately if they've just come online
2016-06-03 14:24:19 +01:00
abb151f3c9
Add a separate process that can handle /sync requests
2016-06-03 11:57:26 +01:00
1f31cc37f8
Working unsubscribe links going straight to the HS
...
and authed by macaroons that let you delete pushers and nothing else
2016-06-02 17:21:31 +01:00
c39f305067
os.environ requires a string
2016-05-16 17:21:30 +01:00
c5c5a7403b
Make synctl read a cache factor from config file
2016-05-16 17:01:57 +01:00
b3f29dc1e5
Manually expire broken caches like the who_forgot_in_room
2016-05-13 17:16:27 +01:00
f03ddc98ec
Use the SlavedAccountDataStore
2016-05-13 17:01:28 +01:00
206eb9fd94
Shift some of the state_group methods into the SlavedEventStore
2016-05-13 16:58:14 +01:00
b5e646a18c
Make email notifs work on the pusher synapse
...
Plus general bugfix to email notif code
2016-05-13 11:36:50 +01:00
ae1af262f6
Pass through _get_state_group_for_events
2016-05-10 19:18:03 +02:00
90afc07f39
StateStore, not EventsStore
2016-05-10 19:10:46 +02:00
89b5ef7c4b
Cached functions must be accessed through the dict
2016-05-10 19:05:22 +02:00
35b6e6d2a8
Pass though _get_state_group_for_events
2016-05-10 18:56:40 +02:00
3367e65476
Pass through get_state_groups
2016-05-10 18:53:15 +02:00
0c4ccdcb83
Also pass through get_profile_displayname
2016-05-10 18:51:14 +02:00
9ef05a12c3
Add date header & message id
2016-05-04 14:52:10 +01:00
8cc82aad87
Add db functions used for email to the pusher app
2016-05-04 11:47:59 +01:00
71df327190
Actually start the pusher daemon
2016-04-26 17:07:09 +01:00
c9eab73f2a
Fix typo in default pusher config
2016-04-26 17:06:18 +01:00
b80b93ea0f
Add a log context to the daemonized pusher
2016-04-26 15:57:28 +01:00
6df5a6a833
Optionally daemonize the pusher
2016-04-26 15:37:41 +01:00
9c417c54d4
Add a couple of update methods to the PusherSlaveStore
2016-04-26 10:45:02 +01:00
f15e9e8de4
Remove the uncomments from the comments
2016-04-25 17:56:24 +01:00
72e2fafa20
Add a metrics listener and a ssh listener to the pusher
2016-04-25 17:34:25 +01:00
f22f46f4f9
Move the listenTCP call outside the manhole function
2016-04-25 14:59:21 +01:00
5905f36f05
Split out setting up the manhole to a separate file
2016-04-22 17:09:15 +01:00
62607d5452
Merge branch 'develop' into markjh/split_out_site.py
...
Conflicts:
synapse/app/homeserver.py
2016-04-22 16:26:57 +01:00
e856036f4c
Move SynapseSite to its own file
2016-04-22 16:09:55 +01:00
9e7aa98c22
Split out create_resource_tree to a separate file
2016-04-22 15:40:51 +01:00
a3ac837599
Optionally split out the pushers into a separate process
2016-04-21 17:22:37 +01:00
82d7eea7e3
Move the versionstring code out of app.homeserver into util
2016-04-11 14:57:09 +01:00
8aab9d87fa
Don't require config to create database
2016-04-06 14:15:45 +01:00
239badea9b
Use syntax that works on both py2.7 and py3
2016-03-07 20:13:10 +00:00
95481e7ba7
Merge pull request #571 from matrix-org/daniel/asids
...
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
763360594d
Mark AS users with their AS's ID
2016-02-11 17:26:42 +00:00
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
737c4223ef
Host /media/r0 as well as /media/v1
2016-02-05 10:47:46 +00:00
6a9f1209df
Error if macaroon key is missing from config
...
Currently we store all access tokens in the DB, and fall back to that
check if we can't validate the macaroon, so our fallback works here, but
for guests, their macaroons don't get persisted, so we don't get to
find them in the database. Each restart, we generate a new ephemeral
key, so guests lose access after each server restart.
I tried to fix up the config stuff to be less insane, but gave up, so
instead I bolt on yet another piece of custom one-off insanity.
Also, add some basic tests for config generation and loading.
2016-02-05 01:58:23 +00:00
f8aae79a72
Simplify get_rooms
2016-02-03 13:24:35 +00:00
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
167d1df699
Merge pull request #534 from matrix-org/erikj/setup
...
Add a Homeserver.setup method
2016-01-28 14:45:36 +00:00
7ed2bbeb11
Clean up a bit. Add comment
2016-01-28 14:32:05 +00:00
a6477d5933
Remove chdir
2016-01-28 09:19:55 +00:00
fa90c180ee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/setup
2016-01-27 17:37:33 +00:00
9fda8b5193
Don't turn on profiling
2016-01-26 18:27:23 +00:00
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
9959d9ece8
Remove redundated BaseHomeServer
2016-01-26 13:52:29 +00:00
44b4fc5f50
Use compiled regex
2016-01-13 11:47:32 +00:00
8f8b884430
Don't log urlencoded access_tokens
2016-01-08 17:48:08 +00:00
33bef689c1
Turn on core dumps
2016-01-07 15:34:30 +00:00
5727922106
Merge pull request #473 from matrix-org/erikj/ssh_manhole
...
Change manhole to use ssh
2016-01-07 14:36:16 +00:00
c9ae1d1ee5
Change manhole to use ssh
2016-01-07 13:59:02 +00:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
11a974da21
Add /_matrix/versions to report supported versions
2016-01-06 18:08:52 +00:00
bf32922e5a
Log when starting stats reporting
2016-01-06 14:13:34 +00:00
797691f908
Log on stats scheduling
2016-01-06 14:04:27 +00:00
7a8ba4c9a0
Actually host r0 and unstable prefixes
2015-12-08 15:26:52 +00:00
6a5ff5f223
Track the time spent in the database per request.
...
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
d6059bdd2a
Fix warnings
2015-12-04 11:34:23 +00:00
a412b9a465
Run the background updates when starting synapse.
2015-11-10 15:50:58 +00:00
fa1cf5ef34
Move static folder into synapse
...
This is because otherwise it won't get picked up by python packaging.
This also fixes the problem where the "static" folder was found if
synapse wasn't started from that directory.
2015-10-26 15:37:44 +00:00
9020860479
Only turn on the twisted deferred debugging if full_twisted_stacktraces is set in the config
2015-10-13 17:50:44 +01:00
7639c3d9e5
Bounce all deferreds through the reactor to make debugging easier.
...
If all deferreds wait a reactor tick before resolving then there is
always a chance to add an errback to the deferred so that stacktraces
get reported, rather than being discarded.
2015-10-13 17:13:04 +01:00
17c80c8a3d
rename schema_prepare to prepare_database
2015-10-13 13:56:22 +01:00
ec398af41c
Expose error more nicely
2015-10-13 11:43:43 +01:00
0a4b7226fc
Don't change cwd in synctl
2015-10-01 09:21:36 +01:00
301141515a
Merge pull request #288 from matrix-org/markjh/unused_definitions
...
Remove some of the unused definitions from synapse
2015-09-28 14:22:44 +01:00
f87a11e0fd
Fix restart
2015-09-24 21:59:38 +00:00
76328b85f6
Allow config file path to be configurable in in synctl
...
Also, allow it to be run from directories other than the synapse directory
2015-09-24 21:50:20 +00:00
7a5818ed81
Note that GzipFile was removed in comment that referenced it
2015-09-22 18:27:22 +01:00
184ba0968a
synapse/app/homeserver.py:GzipFile was unused
2015-09-22 18:25:30 +01:00
b6e0303c83
Catch stats-reporting errors
2015-09-22 13:34:29 +01:00
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
0b5c9adeb5
Merge pull request #267 from matrix-org/markjh/missing_requirements
...
Print an example "pip install" line for a missing requirement
2015-09-18 18:52:08 +01:00
2c746382e0
Merge branch 'daniel/insecureclient' into develop
2015-09-09 14:27:30 +01:00
6485f03d91
Fix random formatting
2015-09-09 13:05:00 +01:00
81a93ddcc8
Allow configuration to ignore invalid SSL certs
...
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
73e4ad4b8b
Merge branch 'master' into develop
...
Conflicts:
setup.py
2015-09-07 15:06:46 +01:00
1002bbd732
Change log level to info
2015-09-03 09:51:01 +01:00
3cdfd37d95
Print an example "pip install" line for a missing requirement
2015-09-01 16:47:26 +01:00
d71af2ee12
don't log the whole DB config (including postgres password...)
2015-08-29 22:23:21 +01:00
d33f31d741
Print the correct pip install line when failing due to lack of matrix-angular-sdk
2015-08-25 15:33:23 +01:00
37403ab06c
Update the log message
2015-08-25 14:19:09 +01:00
d9088c923f
Remove dependency on matrix-angular-sdk
2015-08-25 13:34:50 +01:00
86cef6a91b
Allow specifying a directory to host a web client from
2015-08-25 12:01:23 +01:00
efe60d5e8c
Only print the pidfile path on startup if requested by a commandline flag
2015-08-07 16:36:42 +01:00
0e58d19163
Merge pull request #187 from matrix-org/erikj/sanitize_logging
...
Sanitize logging
2015-06-19 11:35:59 +01:00
eb928c9f52
Add site_tag to logger
2015-06-19 10:16:48 +01:00
9e5a353663
Make upload dir a configurable path.
...
Fixes SYN-425.
Signed-off-by: Eric Myhre <hash@exultant.us>
2015-06-18 23:38:20 -05:00
aaa749d366
Disable twisted access logging. Move access logging to SynapseRequest object
2015-06-15 18:18:05 +01:00
cee69441d3
Log more when we have processed the request
2015-06-15 17:11:44 +01:00
b5209c5744
Create SynapseRequest that overrides __repr__ to not print access_token
2015-06-15 16:37:04 +01:00
a005b7269a
Add backwards compat support for metrics, manhole and webclient config options
2015-06-12 17:44:23 +01:00
942e39e87c
PEP8
2015-06-12 17:13:54 +01:00
9c5fc81c2d
Correctly handle x_forwaded listener option
2015-06-12 17:13:23 +01:00
fd2c07bfed
Use config.listeners
2015-06-12 15:33:07 +01:00
c42ed47660
Fix up create_resource_tree
2015-06-12 11:52:52 +01:00
522f285f9b
Add config option to disable compression of http responses
2015-06-01 13:36:30 +01:00
a7b65bdedf
Add config option to turn off freezing events. Use new encode_json api and ujson.loads
2015-05-29 12:17:33 +01:00
1ce1509989
s/metric_interface/metric_bind_host/
2015-05-22 14:51:22 +01:00
8bb85c8c5a
Update log line
2015-05-22 14:48:06 +01:00
59a0682f3e
Enable changing the interface the metrics listener binds to
2015-05-22 13:13:07 +01:00
19505e0392
Disable GZip encoding on static file resources as per comment
2015-05-19 13:20:25 +01:00
677be13ffc
Revert accidental commit
2015-05-19 13:12:28 +01:00
2b7120e233
SYN-383: Handle the fact the server might not have signed things
2015-05-19 12:49:38 +01:00
6e1ad283cf
Support gzip encoding for client, client v2 and web client resources (SYN-176).
2015-05-14 16:39:19 +01:00
4a7a4a5b6c
Optional profiling using cProfile
2015-05-06 17:08:00 +01:00
b994fb2b96
Don't read from the config file before checking it exists
2015-05-06 12:56:47 +01:00
6ea9cf58be
missing import
2015-04-30 17:21:21 +01:00
c95480963e
read the pid_file from the config file in synctl
2015-04-30 17:12:15 +01:00
2d4d2bbae4
Merge branch 'develop' into markjh/config_cleanup
...
Conflicts:
synapse/config/captcha.py
2015-04-30 16:54:55 +01:00
054aa0d58c
Do access log using python's logging stuff, just under a separate logger name
2015-04-30 16:21:38 +01:00
d89a9f7283
Add an access_log
...
SYN-161 #resolve
2015-04-30 13:58:13 +01:00
6b69ddd17a
remove duplicate parse_size method
2015-04-30 04:26:29 +01:00
d624e2a638
Manually generate the default config yaml, remove most of the commandline arguments for synapse anticipating that people will use the yaml instead. Simpify implementing config options by not requiring the classes to hit the super class
2015-04-30 04:24:44 +01:00
e7a6edb0ee
Revert previous
2015-04-29 18:37:30 +01:00
c27d6ad6b5
Only start pushers when synapse has fully started
2015-04-29 18:25:24 +01:00
f2d90d5c02
Fix whitespace
2015-04-29 14:53:23 +01:00
845b0b2c97
Check requirements before doing anything else when running the homeserver
2015-04-29 14:52:42 +01:00
64991b0c8b
Merge pull request #129 from matrix-org/key_distribution
...
Key distribution v2
2015-04-29 13:34:38 +01:00
4ad8b45155
Merge branch 'develop' into key_distribution
...
Conflicts:
synapse/config/homeserver.py
2015-04-29 13:15:14 +01:00
cd0864121b
Make postgres database error slightly more helpful
2015-04-29 12:12:25 +01:00
204132a998
Check that postgres database has correct charset set
2015-04-29 11:42:28 +01:00
073b891ec1
Remove unused imports
2015-04-28 13:44:23 +01:00
327ca883ec
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-28 13:39:42 +01:00
42b7139dec
Remove unused import
2015-04-27 15:59:56 +01:00
1ef66cc3bd
Move database configuration into config module
2015-04-27 15:57:43 +01:00
416a3e6c4f
Ensure check_same_thread is enabled for sqlite3
2015-04-27 15:44:30 +01:00
df75914791
pep8
2015-04-27 11:48:33 +01:00
b02e1006b9
Run database check before daemonizing, at the cost of database hygiene.
2015-04-27 11:46:00 +01:00
f8152f2708
rename db method to be more informative
2015-04-27 10:16:26 +01:00
a7b51f4539
Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266
2015-04-24 18:11:21 +01:00
ffad75bd62
Remove mysql/maria support
2015-04-15 17:00:50 +01:00
6606f7c659
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-15 10:27:20 +01:00
d488463fa3
Add a version 2 of the key server api
2015-04-14 16:04:52 +01:00
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
3cbc286d06
Move server key api into rest/key/v1
2015-04-14 13:28:11 +01:00
c5365dee56
Use case sensitive collations
2015-04-10 18:46:33 +01:00
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
0bfa78b39b
PEP8
2015-04-07 12:16:05 +01:00
d0e444a648
Explicitly name the __main__ module logger
2015-04-07 12:04:02 +01:00
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
e24c32e6f3
Fix SQLite support
2015-04-01 15:09:51 +01:00
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
3470cb36a8
Pyflakes
2015-03-31 13:03:31 +01:00
e7887e37a8
Remove appservice REST servlets
2015-03-31 09:32:40 +01:00
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
455579ca90
Make database selection configurable
2015-03-20 10:55:55 +00:00
7564dac8cb
Wire up the webclient option
...
It existed but was hardcoded to True.
Give it an underscore for consistency.
Also don't pull in syweb unless we're actually using the web client.
2015-03-17 12:45:37 +01:00
c8032aec17
actually uphold the bind_host parameter. in theory should make ipv6 binds work like bind_host: 'fe80::1%lo0'
2015-03-14 00:12:20 +00:00
b98b4c135d
Option to serve metrics from their own localhost-only TCP port instead of muxed on the main listener
2015-03-12 16:24:51 +00:00
a2cdd11d4a
Fold the slightly-odd bind_port/secure_port/etc.. logic into SynapseHomeServer.start_listening()
2015-03-12 16:24:51 +00:00
e0214a263b
Build MetricsResource as a specific HomeServer dependency
2015-03-12 16:24:51 +00:00
e07fc62833
A trivial 'hello world'-style resource on /_synapse/metrics, with optional commandline flag
2015-03-12 16:24:50 +00:00
211c14c391
No need to explicitly pass 'web_client' in to create_resource_tree as it can be found via config
2015-03-12 16:24:50 +00:00
045afd6b61
in_thread takes no arguments
2015-03-10 10:19:03 +00:00
c37eceeb9e
Split out the 'run' from 'setup'
2015-03-10 09:58:33 +00:00
b8a6692657
Add documentation. When starting via twistd respect soft_file_limit config option.
2015-03-10 09:39:42 +00:00
f31e65ca8b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj-perf
2015-03-09 13:29:41 +00:00
c54773473f
Merge branch 'master' of github.com:matrix-org/synapse into develop
2015-03-06 14:23:41 +00:00
82b34e813d
SYN-67: Finish up implementing new database schema management
2015-03-04 12:04:19 +00:00
3fce185c77
Merge pull request #83 from matrix-org/nofile_limit_config
...
Add config option to set the soft fd limit on start
2015-03-02 13:52:16 +00:00
28d8614f48
Trailing comma
2015-02-23 17:36:37 +00:00
0696dfd94b
Actually treat this as static content, not random Resources.
2015-02-23 15:35:09 +00:00
22399d3d8f
Add RegisterFallbackResource to /_matrix/static/client/register
...
Try to keep both forms of registration logic (native/fallback) close
together for sanity.
2015-02-23 15:14:56 +00:00
1f29fafc95
Don't exit if we can't work out if we're running in a git repo
2015-02-21 13:44:46 +00:00
7c56210f20
By default set soft limit to hard limit
2015-02-20 16:09:44 +00:00
939273c4b0
Rename resource variable so as to not shadow module import
2015-02-19 11:53:13 +00:00
c3eb7dd9c5
Add config option to set the soft fd limit on start
2015-02-19 11:50:49 +00:00
6311ae8968
Conform to header spec take two
2015-02-18 17:34:26 +00:00
b6771037a6
Make version_string conform to User-Agent and Server spec
2015-02-18 17:31:50 +00:00
1df8bad63e
pyflakes
2015-02-18 16:54:25 +00:00
aa577df064
When computing git version run git commands in same dir as source files
2015-02-18 16:52:04 +00:00
d122e215ff
Generate a version string that includes git details if run from git checkout
2015-02-18 16:52:04 +00:00
2635d4e634
Merge branch 'develop' of github.com:matrix-org/synapse into develop
2015-02-18 10:29:54 +00:00
ec847059f3
Rename _fail_fetch_pdu_cache to _get_pdu_cache
2015-02-18 10:14:10 +00:00
1177245e86
Merge branch 'hotfixes-v0.7.0g' of github.com:matrix-org/synapse into develop
2015-02-17 17:30:11 +00:00
8b919c00f3
Start the get_pdu cache
2015-02-17 15:44:01 +00:00
ea1d6c16cd
Don't write bytecode
2015-02-17 10:54:06 +00:00
0d872f5aa6
Merge pull request #50 from matrix-org/application-services
...
Application Services
2015-02-13 15:06:14 +00:00
183b3d4e47
Prepare the database whenever a connection is opened from the db_pool so that in-memory databases will work
2015-02-13 14:38:24 +00:00
5758dafb4e
Merge branch 'develop' of github.com:matrix-org/synapse into state-chache
2015-02-11 10:35:13 +00:00
9978c5c103
Merge branch 'develop' into application-services
2015-02-11 10:03:24 +00:00
84a769cdb7
Fix code-style
2015-02-10 17:58:36 +00:00
c86ebe7673
Merge branch 'develop' of github.com:matrix-org/synapse into state-chache
...
Conflicts:
synapse/app/homeserver.py
synapse/state.py
2015-02-10 11:04:37 +00:00
53557fc532
Merge branch 'develop' into application-services
2015-02-09 15:20:56 +00:00
Neil Johnson