Commit Graph

620 Commits (ad8b909ce98cce5e955b10a3cbe04f3e868ecb70)

Author SHA1 Message Date
Erik Johnston 24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Richard van der Hoff 25adde9a04 Factor out get_db_conn to HomeServer base class
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
2018-01-26 00:56:49 +00:00
Erik Johnston f4d93ae424 Actually make it work 2018-01-12 10:39:27 +00:00
Erik Johnston f68e4cf690 Refactor 2018-01-12 10:11:12 +00:00
Erik Johnston 7cd34512d8 When using synctl with workers, don't start the main synapse automatically 2018-01-11 11:37:39 +00:00
Silke df0f602796 Implement listen_tcp method in remaining workers
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
Silke 26cd3f5690 Remove logger argument and do not catch replication listener
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
Silke Hofstra ed48ecc58c Add methods for listening on multiple addresses
Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP
and reactor.listenSSL for multiple addresses.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:15:48 +01:00
Silke Hofstra 37d1a90025 Allow binds to both :: and 0.0.0.0
Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed.
This causes a warning explaining the behaviour.
Configuration changed to match.

See #2232

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:10:31 +01:00
Richard van der Hoff 795b0849f3 Add a comment which might save some confusion 2017-11-24 00:34:56 +00:00
Richard van der Hoff 7f14f0ae38 Remove dead sync_callback
This is never used; let's remove it to stop confusing things.
2017-11-24 00:32:04 +00:00
Richard van der Hoff 68ca864141 Add config option to disable media_repo on main synapse
... to stop us doing the cache cleanup jobs on the master.
2017-11-22 16:20:27 +00:00
Richard van der Hoff e1fd4751de Build MediaRepositoryResource as a homeserver dependency
This avoids the scenario where we have four different PreviewUrlResources
configured on a single app, each of which have their own caches and cache
clearing jobs.
2017-11-22 16:19:49 +00:00
David Baker d4553818a0
Merge pull request #2627 from matrix-org/rav/custom_rest_endpoints
Add a hook for custom rest endpoints
2017-11-02 15:18:37 +00:00
Richard van der Hoff fcdfc911ee Add a hook for custom rest endpoints
Let the user specify custom modules which can be used for implementing extra
endpoints.
2017-11-02 14:36:55 +00:00
Richard van der Hoff 6650a07ede Factor out _configure_named_resource
This was a bit of a code vomit, so let's factor it out to preserve some sanity
2017-11-02 14:33:37 +00:00
Richard van der Hoff 8b56977b6f Merge pull request #2586 from matrix-org/rav/frontend_proxy_auth_header
Front-end proxy: pass through auth header
2017-10-27 11:01:50 +01:00
Richard van der Hoff 54a2525133 Front-end proxy: pass through auth header
So that access-token-in-an-auth-header works.
2017-10-26 18:19:01 +01:00
Richard van der Hoff 0a5866bec9 Support /keys/upload on /r0 as well as /unstable
(So that we can stop riot relying on it in /unstable)
2017-10-26 18:18:23 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston 535cc49f27 Merge pull request #2466 from matrix-org/erikj/groups_merged
Initial Group Implementation
2017-10-11 13:20:07 +01:00
Erik Johnston ea87cb1ba5 Make 'affinity' package optional 2017-10-02 18:03:59 +01:00
Erik Johnston 3fed5bb25f Move quit_with_error 2017-10-02 17:59:34 +01:00
Erik Johnston b5feaa5a49 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_merged 2017-09-19 11:07:45 +01:00
Richard van der Hoff 692250c6be Fix user_dir startup
Add missing parameter to _base.start_worker_reactor
2017-08-16 15:11:29 +01:00
Richard van der Hoff 10d8b701a1 Allow configuration of CPU affinity
Make it possible to set the CPU affinity in the config file, so that we don't
need to remember to do it manually every time.
2017-08-15 17:08:28 +01:00
Richard van der Hoff 543c794a76 Factor out common application start
We have 10 copies of this code, and I don't really want to update each one
separately.
2017-08-15 17:04:40 +01:00
Erik Johnston 2cc998fed8 Fix replication. And notify 2017-07-20 17:13:18 +01:00
Erik Johnston d4d12daed9 Include registration and as stores in frontend proxy 2017-07-07 18:36:45 +01:00
Erik Johnston 1fc4a962e4 Add a frontend proxy 2017-07-07 18:19:46 +01:00
Erik Johnston 78cefd78d6 Make workers report to master for user ip updates 2017-06-27 14:58:10 +01:00
Erik Johnston 8177563ebe Fix for workers 2017-06-21 13:57:49 +01:00
Erik Johnston dfeca6cf40 Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston 6aa5bc8635 Initial worker impl 2017-06-16 11:47:11 +01:00
Erik Johnston 5ddd199870 Typo 2017-06-15 10:49:10 +01:00
Erik Johnston 4b461a6931 Add some more stats 2017-06-15 09:39:39 +01:00
Erik Johnston 617304b2cf Fix phone home stats 2017-06-14 19:47:15 +01:00
Erik Johnston f87b287291 Merge pull request #2127 from APwhitehat/alreadystarted
print something legible if synapse already running
2017-04-27 15:46:53 +01:00
Matthew Wolff d4040e9e28
Queried CONDITIONAL_REQUIREMENTS 2017-04-18 16:19:48 -05:00
Matthew Wolff 54f9a4cb59
Fixed travis build failure
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 01:38:27 -05:00
Matthew Wolff 8e780b113d
web_server_root documentation fix
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 00:49:11 -05:00
Anant Prakash c1f52a321d synctl.py: Check if synapse is already running 2017-04-13 18:00:02 +05:30
Erik Johnston 247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
Erik Johnston 9c712a366f Move get_presence_list_* to SlaveStore 2017-04-11 16:07:33 +01:00
Erik Johnston 414522aed5 Move get_interested_parties 2017-04-11 15:33:26 +01:00
Erik Johnston 28a4649785 Remove HTTP replication APIs 2017-04-11 09:52:11 +01:00
Erik Johnston 29574fd5b3 Reduce federation presence replication traffic
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston 449d1297ca Fix up federation SendQueue and document types 2017-04-07 11:48:33 +01:00
Erik Johnston 54f59bd7d4 Merge pull request #2107 from HarHarLinks/patch-1
fix typo in synctl help
2017-04-07 09:54:37 +01:00
Erik Johnston 98ce212093 Merge pull request #2103 from matrix-org/erikj/no-double-encode
Don't double encode replication data
2017-04-07 09:39:52 +01:00
Kim Brose 8a1137ceab fix typo in synctl help 2017-04-06 17:10:20 +02:00
Erik Johnston 391712a4f9 Comment 2017-04-06 13:35:00 +01:00
Erik Johnston dbf87282d3 Docs 2017-04-06 13:11:21 +01:00
Erik Johnston b5cb6347a4 Don't immediately notify the master about users whose syncs have gone away 2017-04-05 13:25:40 +01:00
Erik Johnston 96b9b6c127 Don't double json encode typing replication data 2017-04-05 11:34:20 +01:00
Erik Johnston f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston a5c401bd12 Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
Move to using TCP replication
2017-04-05 09:36:21 +01:00
Erik Johnston d1d5362267 Add comment 2017-04-04 16:41:03 +01:00
Erik Johnston ac66e11f2b Add the appropriate amount of preserve_fn 2017-04-04 15:22:54 +01:00
Erik Johnston 3376f16012 Shuffle and comment synchrotron presence 2017-04-04 11:14:16 +01:00
Erik Johnston 6ce6bbedcb Move where we ack federation 2017-04-04 11:02:44 +01:00
Erik Johnston 62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Erik Johnston 36c28bc467 Update all the workers and master to use TCP replication 2017-04-03 15:35:52 +01:00
Erik Johnston 8a240e4f9c Merge pull request #2078 from APwhitehat/assertuserfriendly
add user friendly report of assertion error in synctl.py
2017-03-31 14:41:49 +01:00
Erik Johnston 3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Anant Prakash 6cdca71079 synctl.py: wait for synapse to stop before restarting (#2020) 2017-03-29 19:20:13 +01:00
Anant Prakash 305d16d612 add user friendly report of assertion error in synctl.py
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
2017-03-29 20:41:39 +05:30
Erik Johnston 48e7697911 Add slave transaction store 2017-03-24 13:05:30 +00:00
Erik Johnston 37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
Erik Johnston aac6d1fc9b PEP8 2017-03-20 13:47:56 +00:00
Richard van der Hoff 067b00d49d Run the reactor with the sentinel logcontext
This fixes a class of 'Unexpected logcontext' messages, which were happening
because the logcontext was somewhat arbitrarily swapping between the sentinel
and the `run` logcontext.
2017-03-18 00:07:43 +00:00
Erik Johnston 61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Erik Johnston 6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Richard van der Hoff d84bd51e95 Refactor logger config for workers
- to make it easier to add more config options.
2017-03-10 15:34:01 +00:00
Mark Haines 3a46280ca3 Add db functions needed for room initial sync to slave 2017-02-13 11:16:53 +00:00
Erik Johnston 4ac363a168 Remove debug logging 2017-01-30 14:10:12 +00:00
Erik Johnston 252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston edd6cdfc9a Restore default bind address 2017-01-10 17:21:41 +00:00
Erik Johnston 8a0fddfd73 Remove spurious for..else.. 2017-01-10 16:30:53 +00:00
Johannes Löthberg 702c020e58 Fix check for bind_address
The empty string is a valid setting for the bind_address option, so
explicitly check for None here instead.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-20 01:37:50 +01:00
Johannes Löthberg 7dfd70fc83 Add support for specifying multiple bind addresses
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 21:51:56 +01:00
Erik Johnston 4d9b5c60f9 Comment 2016-11-23 11:11:41 +00:00
Erik Johnston 4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston 50934ce460 Comments 2016-11-21 16:55:23 +00:00
Erik Johnston 7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Erik Johnston f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston 24772ba56e Respect use_frozen_dicts option in workers 2016-11-08 11:07:18 +00:00
Paul "LeoNerd" Evans 1cc22da600 Set up the process collector during metrics __init__; that way all split-process workers have it 2016-10-27 18:09:34 +01:00
Paul "LeoNerd" Evans aac13b1f9a Pass the Metrics group into the process collector instead of having it find its own one; this avoids it needing to import from synapse.metrics 2016-10-27 18:08:15 +01:00
Paul "LeoNerd" Evans 3ae48a1f99 Move the process metrics collector code into its own file 2016-10-19 18:10:24 +01:00
Erik Johnston a68ade6ed3 Merge pull request #1162 from larroy/master
Use sys.executable instead of hardcoded python. fixes #1161
2016-10-14 21:42:55 +01:00
Pedro Larroy c57f871184 Use sys.executable instead of hardcoded python. fixes #1161 2016-10-08 23:55:20 +02:00
Erik Johnston 748d8fdc7b Reduce DB hits for replication
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.

This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston 87528f0756 Support /initialSync in synchrotron worker 2016-09-21 11:46:28 +01:00
Matthew Hodgson 883df2e983 fix logger for client_reader worker 2016-09-17 14:12:04 +01:00
Erik Johnston 4be85281f9 Enable state caches on workers 2016-09-15 14:31:22 +01:00
Erik Johnston e0b6db29ed Split out public room list into a worker process 2016-09-14 14:42:51 +01:00
Erik Johnston 44330a21e9 Comment 2016-09-09 17:22:07 +01:00
Mark Haines 7ed5acacf4 Fix up the calls to the notifier for device messages 2016-09-01 18:08:40 +01:00
Erik Johnston 873f870e5a Add new direct message storage functions to slave store 2016-08-30 09:40:32 +01:00
Mark Haines 689f4cb914 Update comment 2016-08-22 16:17:31 +01:00
Mark Haines c8f9b45bc2 Add usage stats to prometheus monitoring 2016-08-22 15:34:38 +01:00
Erik Johnston bcbd74dc5b Remove log lines 2016-08-18 15:52:10 +01:00
Erik Johnston 07229bbdae Add appservice worker 2016-08-18 14:59:55 +01:00
Mark Haines 403ecd8a2c Missed a s/federation reader/media repository/ in a log message 2016-08-18 10:26:15 +01:00
Mark Haines 396624864a Add a media repository worker 2016-08-18 09:38:42 +01:00
Erik Johnston 784a2d4f2c Remove broken cache stuff 2016-08-15 11:25:48 +01:00
Erik Johnston 64e7e11853 Implement cache replication stream 2016-08-15 11:16:45 +01:00
Erik Johnston 4e1cebd56f Make synchrotron accept /events 2016-08-12 15:31:44 +01:00
Will Hunt 5b5148b7ec Synced up synchrotron set_state with PresenceHandler set_state 2016-08-11 11:48:30 +01:00
Erik Johnston 24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston 5aa024e501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:24:56 +01:00
Erik Johnston ec8b217722 Add destination retry to slave store 2016-07-28 17:35:53 +01:00
Erik Johnston 76b89d0edb Add slace storage functions for public room list 2016-07-28 17:03:40 +01:00
Richard van der Hoff 05e7e5e972 Fix flake8 violation
Apparently flake8 v3 puts the error on a different line to v2. Easiest way to
make sure that happens is by putting the whole statement on one line :)
2016-07-26 11:59:08 +01:00
Erik Johnston aede7248ab Split out a FederationReader process 2016-07-21 17:37:44 +01:00
Erik Johnston 66868119dc Add metrics for psutil derived memory usage 2016-07-20 16:00:21 +01:00
Mark Haines 05f1a4596a Merge branch 'master' into develop 2016-06-23 11:17:48 +01:00
Mark Haines 5cc7564c5c Optionally start or stop workers in synctl.
Optionally start or stop an individual worker by passing -w with
the path to the worker config.
Optionally start or stop every worker and the main synapse by
passing -a with a path to a directory containing worker configs.

The "-w" is intended to be used to bounce individual workers proceses.
THe "-a" is intended for when you want to restart all the workers
simultaneuously, for example when performing database upgrades.
2016-06-21 16:38:05 +01:00
Mark Haines 13e334506c Remove the legacy v0 content upload API.
The existing content can still be downloaded. The last upload to the
matrix.org server was in January 2015, so it is probably safe to remove
the upload API.
2016-06-21 11:47:39 +01:00
Mark Haines 8c75040c25 Fix setting gc thresholds in the workers 2016-06-17 11:48:12 +01:00
Mark Haines a352b68acf Use worker_ prefixes for worker config, use existing support for multiple config files 2016-06-16 17:29:50 +01:00
Mark Haines 364d616792 Access the event_cache_size directly from the server object.
This means that the workers can override the event_cache_size
directly without clobbering the value in the main synapse config.
2016-06-16 12:53:15 +01:00
Mark Haines bde13833cb Access replication_url from the worker config directly 2016-06-16 12:44:40 +01:00
Mark Haines 885ee861f7 Inline the synchrotron and pusher configs into the main config 2016-06-16 11:06:12 +01:00
Mark Haines 7dbb473339 Add function to load config without generating it
Renames ``load_config`` to ``load_or_generate_config``
Adds a method called ``load_config`` that just loads the
config.

The main synapse.app.homeserver will continue to use
``load_or_generate_config`` to retain backwards compat.
However new worker processes can use ``load_config`` to
load the config avoiding some of the cruft needed to generate
the config.

As the new ``load_config`` method is expected to be used by new
configs it removes support for the legacy commandline overrides
that ``load_or_generate_config`` supports
2016-06-09 18:50:38 +01:00
Erik Johnston 2d1d1025fa Add gc_threshold to pusher and synchrotron 2016-06-07 16:26:25 +01:00
Erik Johnston dded389ac1 Allow setting of gc.set_thresholds 2016-06-07 15:45:56 +01:00
Mark Haines b0932b34cb Merge pull request #845 from matrix-org/markjh/synchrotron_presence
Fix a KeyError in the synchrotron presence
2016-06-06 16:52:27 +01:00
Mark Haines 4a5bbb1941 Fix a KeyError in the synchrotron presence 2016-06-06 16:37:12 +01:00
Mark Haines 5ef84da4f1 Yield on the sleeps intended to backoff replication 2016-06-06 16:05:28 +01:00
Mark Haines ac9716f154 Fix spelling 2016-06-03 18:10:00 +01:00
Mark Haines 8f79084bd4 Add get_presence_list_accepted to the broken caches in synchrotron 2016-06-03 18:03:40 +01:00
Mark Haines 48340e4f13 Clear the list of ongoing syncs on shutdown 2016-06-03 15:02:27 +01:00
Mark Haines da491e75b2 Appease flake8 2016-06-03 14:56:36 +01:00
Mark Haines 0b3c80a234 Use ClientIpStore to record client ips 2016-06-03 14:55:01 +01:00
Mark Haines 80aade3805 Send updates to the syncing users every ten seconds or immediately if they've just come online 2016-06-03 14:24:19 +01:00
Mark Haines abb151f3c9 Add a separate process that can handle /sync requests 2016-06-03 11:57:26 +01:00
David Baker 1f31cc37f8 Working unsubscribe links going straight to the HS
and authed by macaroons that let you delete pushers and nothing else
2016-06-02 17:21:31 +01:00
Erik Johnston c39f305067 os.environ requires a string 2016-05-16 17:21:30 +01:00
Erik Johnston c5c5a7403b Make synctl read a cache factor from config file 2016-05-16 17:01:57 +01:00
Mark Haines b3f29dc1e5 Manually expire broken caches like the who_forgot_in_room 2016-05-13 17:16:27 +01:00
Mark Haines f03ddc98ec Use the SlavedAccountDataStore 2016-05-13 17:01:28 +01:00
Mark Haines 206eb9fd94 Shift some of the state_group methods into the SlavedEventStore 2016-05-13 16:58:14 +01:00
David Baker b5e646a18c Make email notifs work on the pusher synapse
Plus general bugfix to email notif code
2016-05-13 11:36:50 +01:00
David Baker ae1af262f6 Pass through _get_state_group_for_events 2016-05-10 19:18:03 +02:00
David Baker 90afc07f39 StateStore, not EventsStore 2016-05-10 19:10:46 +02:00
David Baker 89b5ef7c4b Cached functions must be accessed through the dict 2016-05-10 19:05:22 +02:00
David Baker 35b6e6d2a8 Pass though _get_state_group_for_events 2016-05-10 18:56:40 +02:00
David Baker 3367e65476 Pass through get_state_groups 2016-05-10 18:53:15 +02:00
David Baker 0c4ccdcb83 Also pass through get_profile_displayname 2016-05-10 18:51:14 +02:00
David Baker 9ef05a12c3 Add date header & message id 2016-05-04 14:52:10 +01:00
David Baker 8cc82aad87 Add db functions used for email to the pusher app 2016-05-04 11:47:59 +01:00
Mark Haines 71df327190 Actually start the pusher daemon 2016-04-26 17:07:09 +01:00
Mark Haines c9eab73f2a Fix typo in default pusher config 2016-04-26 17:06:18 +01:00
Mark Haines b80b93ea0f Add a log context to the daemonized pusher 2016-04-26 15:57:28 +01:00
Mark Haines 6df5a6a833 Optionally daemonize the pusher 2016-04-26 15:37:41 +01:00
Mark Haines 9c417c54d4 Add a couple of update methods to the PusherSlaveStore 2016-04-26 10:45:02 +01:00
Mark Haines f15e9e8de4 Remove the uncomments from the comments 2016-04-25 17:56:24 +01:00
Mark Haines 72e2fafa20 Add a metrics listener and a ssh listener to the pusher 2016-04-25 17:34:25 +01:00
Mark Haines f22f46f4f9 Move the listenTCP call outside the manhole function 2016-04-25 14:59:21 +01:00
Mark Haines 5905f36f05 Split out setting up the manhole to a separate file 2016-04-22 17:09:15 +01:00
Mark Haines 62607d5452 Merge branch 'develop' into markjh/split_out_site.py
Conflicts:
	synapse/app/homeserver.py
2016-04-22 16:26:57 +01:00
Mark Haines e856036f4c Move SynapseSite to its own file 2016-04-22 16:09:55 +01:00
Mark Haines 9e7aa98c22 Split out create_resource_tree to a separate file 2016-04-22 15:40:51 +01:00
Mark Haines a3ac837599 Optionally split out the pushers into a separate process 2016-04-21 17:22:37 +01:00
Mark Haines 82d7eea7e3 Move the versionstring code out of app.homeserver into util 2016-04-11 14:57:09 +01:00
Erik Johnston 8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Mark Haines 239badea9b Use syntax that works on both py2.7 and py3 2016-03-07 20:13:10 +00:00
Daniel Wagner-Hall 95481e7ba7 Merge pull request #571 from matrix-org/daniel/asids
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
Mark Haines 60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Daniel Wagner-Hall 763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Erik Johnston 2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall 737c4223ef Host /media/r0 as well as /media/v1 2016-02-05 10:47:46 +00:00
Daniel Wagner-Hall 6a9f1209df Error if macaroon key is missing from config
Currently we store all access tokens in the DB, and fall back to that
check if we can't validate the macaroon, so our fallback works here, but
for guests, their macaroons don't get persisted, so we don't get to
find them in the database. Each restart, we generate a new ephemeral
key, so guests lose access after each server restart.

I tried to fix up the config stuff to be less insane, but gave up, so
instead I bolt on yet another piece of custom one-off insanity.

Also, add some basic tests for config generation and loading.
2016-02-05 01:58:23 +00:00
Erik Johnston f8aae79a72 Simplify get_rooms 2016-02-03 13:24:35 +00:00
Daniel Wagner-Hall d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston 167d1df699 Merge pull request #534 from matrix-org/erikj/setup
Add a Homeserver.setup method
2016-01-28 14:45:36 +00:00
Erik Johnston 7ed2bbeb11 Clean up a bit. Add comment 2016-01-28 14:32:05 +00:00
Erik Johnston a6477d5933 Remove chdir 2016-01-28 09:19:55 +00:00
Erik Johnston fa90c180ee Merge branch 'develop' of github.com:matrix-org/synapse into erikj/setup 2016-01-27 17:37:33 +00:00
Erik Johnston 9fda8b5193 Don't turn on profiling 2016-01-26 18:27:23 +00:00
Erik Johnston 87f9477b10 Add a Homeserver.setup method.
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
Erik Johnston 9959d9ece8 Remove redundated BaseHomeServer 2016-01-26 13:52:29 +00:00
Erik Johnston 44b4fc5f50 Use compiled regex 2016-01-13 11:47:32 +00:00
Erik Johnston 8f8b884430 Don't log urlencoded access_tokens 2016-01-08 17:48:08 +00:00
Erik Johnston 33bef689c1 Turn on core dumps 2016-01-07 15:34:30 +00:00
Erik Johnston 5727922106 Merge pull request #473 from matrix-org/erikj/ssh_manhole
Change manhole to use ssh
2016-01-07 14:36:16 +00:00
Erik Johnston c9ae1d1ee5 Change manhole to use ssh 2016-01-07 13:59:02 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall 11a974da21 Add /_matrix/versions to report supported versions 2016-01-06 18:08:52 +00:00
Daniel Wagner-Hall bf32922e5a Log when starting stats reporting 2016-01-06 14:13:34 +00:00
Daniel Wagner-Hall 797691f908 Log on stats scheduling 2016-01-06 14:04:27 +00:00
Daniel Wagner-Hall 7a8ba4c9a0 Actually host r0 and unstable prefixes 2015-12-08 15:26:52 +00:00
Mark Haines 6a5ff5f223 Track the time spent in the database per request.
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
Mark Haines d6059bdd2a Fix warnings 2015-12-04 11:34:23 +00:00
Mark Haines a412b9a465 Run the background updates when starting synapse. 2015-11-10 15:50:58 +00:00
Erik Johnston fa1cf5ef34 Move static folder into synapse
This is because otherwise it won't get picked up by python packaging.

This also fixes the problem where the "static" folder was found if
synapse wasn't started from that directory.
2015-10-26 15:37:44 +00:00
Mark Haines 9020860479 Only turn on the twisted deferred debugging if full_twisted_stacktraces is set in the config 2015-10-13 17:50:44 +01:00
Mark Haines 7639c3d9e5 Bounce all deferreds through the reactor to make debugging easier.
If all deferreds wait a reactor tick before resolving then there is
always a chance to add an errback to the deferred so that stacktraces
get reported, rather than being discarded.
2015-10-13 17:13:04 +01:00
Erik Johnston 17c80c8a3d rename schema_prepare to prepare_database 2015-10-13 13:56:22 +01:00
Erik Johnston ec398af41c Expose error more nicely 2015-10-13 11:43:43 +01:00
Erik Johnston 0a4b7226fc Don't change cwd in synctl 2015-10-01 09:21:36 +01:00
Mark Haines 301141515a Merge pull request #288 from matrix-org/markjh/unused_definitions
Remove some of the unused definitions from synapse
2015-09-28 14:22:44 +01:00
Daniel Wagner-Hall f87a11e0fd Fix restart 2015-09-24 21:59:38 +00:00
Daniel Wagner-Hall 76328b85f6 Allow config file path to be configurable in in synctl
Also, allow it to be run from directories other than the synapse directory
2015-09-24 21:50:20 +00:00
Mark Haines 7a5818ed81 Note that GzipFile was removed in comment that referenced it 2015-09-22 18:27:22 +01:00
Mark Haines 184ba0968a synapse/app/homeserver.py:GzipFile was unused 2015-09-22 18:25:30 +01:00
Daniel Wagner-Hall b6e0303c83 Catch stats-reporting errors 2015-09-22 13:34:29 +01:00
Daniel Wagner-Hall 7213588083 Implement configurable stats reporting
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Mark Haines 0b5c9adeb5 Merge pull request #267 from matrix-org/markjh/missing_requirements
Print an example "pip install" line for a missing requirement
2015-09-18 18:52:08 +01:00
Daniel Wagner-Hall 2c746382e0 Merge branch 'daniel/insecureclient' into develop 2015-09-09 14:27:30 +01:00
Daniel Wagner-Hall 6485f03d91 Fix random formatting 2015-09-09 13:05:00 +01:00
Daniel Wagner-Hall 81a93ddcc8 Allow configuration to ignore invalid SSL certs
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
Mark Haines 73e4ad4b8b Merge branch 'master' into develop
Conflicts:
	setup.py
2015-09-07 15:06:46 +01:00
Erik Johnston 1002bbd732 Change log level to info 2015-09-03 09:51:01 +01:00
Mark Haines 3cdfd37d95 Print an example "pip install" line for a missing requirement 2015-09-01 16:47:26 +01:00
Matthew Hodgson d71af2ee12 don't log the whole DB config (including postgres password...) 2015-08-29 22:23:21 +01:00
Erik Johnston d33f31d741 Print the correct pip install line when failing due to lack of matrix-angular-sdk 2015-08-25 15:33:23 +01:00
Erik Johnston 37403ab06c Update the log message 2015-08-25 14:19:09 +01:00
Erik Johnston d9088c923f Remove dependency on matrix-angular-sdk 2015-08-25 13:34:50 +01:00
Erik Johnston 86cef6a91b Allow specifying a directory to host a web client from 2015-08-25 12:01:23 +01:00
Paul "LeoNerd" Evans efe60d5e8c Only print the pidfile path on startup if requested by a commandline flag 2015-08-07 16:36:42 +01:00
Erik Johnston 0e58d19163 Merge pull request #187 from matrix-org/erikj/sanitize_logging
Sanitize logging
2015-06-19 11:35:59 +01:00
Erik Johnston eb928c9f52 Add site_tag to logger 2015-06-19 10:16:48 +01:00
Eric Myhre 9e5a353663 Make upload dir a configurable path.
Fixes SYN-425.

Signed-off-by: Eric Myhre <hash@exultant.us>
2015-06-18 23:38:20 -05:00
Erik Johnston aaa749d366 Disable twisted access logging. Move access logging to SynapseRequest object 2015-06-15 18:18:05 +01:00
Erik Johnston cee69441d3 Log more when we have processed the request 2015-06-15 17:11:44 +01:00
Erik Johnston b5209c5744 Create SynapseRequest that overrides __repr__ to not print access_token 2015-06-15 16:37:04 +01:00
Erik Johnston a005b7269a Add backwards compat support for metrics, manhole and webclient config options 2015-06-12 17:44:23 +01:00
Erik Johnston 942e39e87c PEP8 2015-06-12 17:13:54 +01:00
Erik Johnston 9c5fc81c2d Correctly handle x_forwaded listener option 2015-06-12 17:13:23 +01:00
Erik Johnston fd2c07bfed Use config.listeners 2015-06-12 15:33:07 +01:00
Erik Johnston c42ed47660 Fix up create_resource_tree 2015-06-12 11:52:52 +01:00
Erik Johnston 522f285f9b Add config option to disable compression of http responses 2015-06-01 13:36:30 +01:00
Erik Johnston a7b65bdedf Add config option to turn off freezing events. Use new encode_json api and ujson.loads 2015-05-29 12:17:33 +01:00
Erik Johnston 1ce1509989 s/metric_interface/metric_bind_host/ 2015-05-22 14:51:22 +01:00
Erik Johnston 8bb85c8c5a Update log line 2015-05-22 14:48:06 +01:00
Erik Johnston 59a0682f3e Enable changing the interface the metrics listener binds to 2015-05-22 13:13:07 +01:00
David Baker 19505e0392 Disable GZip encoding on static file resources as per comment 2015-05-19 13:20:25 +01:00
Erik Johnston 677be13ffc Revert accidental commit 2015-05-19 13:12:28 +01:00
Erik Johnston 2b7120e233 SYN-383: Handle the fact the server might not have signed things 2015-05-19 12:49:38 +01:00
David Baker 6e1ad283cf Support gzip encoding for client, client v2 and web client resources (SYN-176). 2015-05-14 16:39:19 +01:00
Mark Haines 4a7a4a5b6c Optional profiling using cProfile 2015-05-06 17:08:00 +01:00
Mark Haines b994fb2b96 Don't read from the config file before checking it exists 2015-05-06 12:56:47 +01:00
Mark Haines 6ea9cf58be missing import 2015-04-30 17:21:21 +01:00
Mark Haines c95480963e read the pid_file from the config file in synctl 2015-04-30 17:12:15 +01:00
Mark Haines 2d4d2bbae4 Merge branch 'develop' into markjh/config_cleanup
Conflicts:
	synapse/config/captcha.py
2015-04-30 16:54:55 +01:00
David Baker 054aa0d58c Do access log using python's logging stuff, just under a separate logger name 2015-04-30 16:21:38 +01:00