a90a0f5c8a
Propagate errors sensibly from proxied IS requests
...
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
2017-04-21 11:32:48 +01:00
4903ccf159
Fix some lies, and other clarifications, in docstrings
...
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
a380f041c2
try not to drop context after federation requests
...
preserve_context_over_fn uses a ContextPreservingDeferred, which only restores
context for the duration of its callbacks, which isn't really correct, and
means that subsequent operations in the same request can end up without their
logcontexts.
2017-03-23 22:36:21 +00:00
5a16cb4bf0
Ignore backoff history for invites, aliases, and roomdirs
...
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
4bd597d9fc
push federation retry limiter down to matrixfederationclient
...
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
ad8a26e361
MatrixFederationHttpClient: clean up
...
rename _create_request to _request, and push ascii-encoding of `destination`
and `path` down into it
2017-03-23 00:27:04 +00:00
7827251daf
Merge pull request #1994 from matrix-org/dbkr/msisdn_signin_2
...
Phone number registration / login support v2
2017-03-15 09:59:54 +00:00
73a5f06652
Support registration / login with phone number
...
Changes from https://github.com/matrix-org/synapse/pull/1971
2017-03-13 17:27:51 +00:00
170ccc9de5
Fix routing loop when fetching remote media
...
When we proxy a media request to a remote server, add a query-param, which will
tell the remote server to 404 if it doesn't recognise the server_name.
This should fix a routing loop where the server keeps forwarding back to
itself.
Also improves the error handling on remote media fetches, so that we don't
always return a rather obscure 502.
2017-03-13 16:30:36 +00:00
7eae6eaa2f
Revert "Support registration & login with phone number"
2017-03-13 09:59:33 +00:00
2e27339add
Refector out assert_params_in_request
...
and replace requestEmailToken where we meant requestMsisdnToken
2017-03-08 11:37:34 +00:00
342fb8dae9
Merge branch 'release-v0.18.6' into develop
2016-12-29 17:33:46 +00:00
97ffc5690b
Manually abort the underlying TLS connection.
...
The abort() method calls loseConnection() which tries to shutdown the
TLS connection cleanly. We now call abortConnection() directly which
should promptly close both the TLS connection and the underlying TCP
connection.
I also added some TODO markers to consider cancelling the old previous
timeout rather than checking time.time(). But given how urgently we want
to get this code released I'd rather leave the existing code with the
duplicate timeouts and the time.time() check.
2016-12-29 15:51:04 +00:00
b4bc6fef5b
Respect long_retries param and default to off
2016-12-29 00:58:34 +00:00
68030fd37b
Spelling and comments
2016-12-29 00:10:49 +00:00
b7336ff32d
Clean up
2016-12-29 00:09:33 +00:00
5b6672c66d
Wrap connections in an N minute timeout to ensure they get reaped correctly
2016-12-29 00:06:53 +00:00
a58e4e0d48
Merge pull request #1696 from kyrias/ipv6
...
IPv6 support
2016-12-19 23:49:07 +00:00
0648e76979
Remove spurious newline
...
Apparently I just removed the spaces instead...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 18:41:30 +01:00
d3bd94805f
Fixup for #1689 and #1690
...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 16:32:47 +01:00
9f07f4c559
IPv6 support for endpoint.py
...
Similar to https://github.com/matrix-org/synapse/pull/1689 , but for endpoint.py
2016-12-11 11:10:32 +01:00
6e18805ac2
IPv6 support for client.py
...
This is an (untested) general sketch of how to use wrapClientTLS to implement TLS over IPv6, as well as faster connections over IPv4.
2016-12-11 11:10:32 +01:00
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
b5b3a7e867
More intelligent Content-Type parsing
...
Content-Type is allowed to contain options (`; charset=utf-8`, for
instance). We should allow that.
2016-11-30 15:07:32 +00:00
b1c27975d0
Set CORs headers on responses from the media repo
2016-11-02 11:29:25 +00:00
5810cffd33
Pass since/from parameters over federation
2016-09-15 10:36:19 +01:00
949c2c5435
Add a timeout parameter for end2end key queries.
...
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.
Fetch keys from different servers in parallel rather than in series.
Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
33f3624ff7
Add exception logging. Fix typo
2016-08-22 10:49:31 +01:00
8c52160b07
Allow request handlers to override metric name
2016-08-22 10:44:45 +01:00
a093fab253
Use top level measure
2016-08-22 10:18:12 +01:00
27e0178da9
Add a top level measure
2016-08-19 18:49:37 +01:00
6d1a94d218
Remove redundant measure
2016-08-19 18:40:31 +01:00
39b900b316
Measure http.server render
2016-08-19 18:23:44 +01:00
ca8abfbf30
Clean up TransactionQueue
2016-08-10 16:24:16 +01:00
33d08e8433
Log when adding listeners
2016-07-25 17:22:15 +01:00
255c229f23
Work around TLS bug in twisted
...
Wrap up twisted's FileBodyProducer to work around
https://twistedmatrix.com/trac/ticket/8473 . Hopefully this fixes
https://matrix.org/jira/browse/SYN-700 .
2016-06-15 10:39:08 +01:00
eb79110beb
Clean up the blacklist/whitelist handling.
...
Always set the config key with an empty list, even if a list isn't specified.
This means that the codepaths are the same for both the empty list and
for a missing key. Since the behaviour is the same for both cases this
makes the code somewhat easier to reason about.
2016-05-16 13:03:59 +01:00
792def4928
add a url_preview_ip_range_whitelist config param so we can whitelist the matrix.org IP space
2016-05-01 12:44:24 +01:00
351b50a887
Fix more typos in per-request metrics
2016-04-28 15:29:46 +01:00
dcfc10b129
Fix typo in request metrics
2016-04-28 15:11:06 +01:00
1a12766e3b
Add a comment explaining why automatic metric reporting is disabled for JsonResource
2016-04-28 12:31:26 +01:00
6037349512
Check if report_metrics is True
2016-04-28 12:26:07 +01:00
8d7ad44331
Report per request metrics for all of the things using request_handler
2016-04-28 10:57:49 +01:00
e856036f4c
Move SynapseSite to its own file
2016-04-22 16:09:55 +01:00
f505575f69
Make InsecureInterceptableContextFactory work with SpiderEndpoint
2016-04-19 16:08:14 +01:00
4bd3d25218
Merge pull request #688 from matrix-org/matthew/preview_urls
...
URL previewing support
2016-04-11 10:40:29 +01:00
83b2f83da0
actually throw meaningful errors
2016-04-08 21:36:59 +01:00
dafef5a688
Add url_preview_enabled config option to turn on/off preview_url endpoint. defaults to off.
...
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
2016-04-08 18:37:15 +01:00
a28d066732
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dns_cache
2016-04-07 11:11:17 +01:00
9f7dc2bef7
Merge branch 'develop' into matthew/preview_urls
2016-04-04 00:38:21 +01:00
8b98a7e8c3
pep8
2016-04-03 12:56:29 +01:00
7426c86eb8
add a persistent cache of URL lookups, and fix up the in-memory one to work
2016-04-03 00:31:57 +01:00
d1b154a10f
support gzip compression, and don't pass through error msgs
2016-04-02 03:06:39 +01:00
2a37467fa1
Use google style doc strings.
...
pycharm supports them so there is no need to use the other format.
Might as well convert the existing strings to reduce the risk of
people accidentally cargo culting the wrong doc string format.
2016-04-01 16:12:07 +01:00
683e564815
handle spidered relative images correctly
2016-03-31 23:52:58 +01:00
0d3d7de6fc
sync in changes from matrixfederationclient
2016-03-31 12:42:27 +01:00
f9d3665c88
Allow clock to be passed in to func
2016-03-31 10:23:48 +01:00
f699b8f997
Read from DNS cache if within TTL
2016-03-31 10:04:28 +01:00
a8a5dd3b44
handle requests with missing content-length headers (e.g. YouTube)
2016-03-31 01:55:21 +01:00
721b2bfa85
implement redirects
2016-03-29 03:32:52 +01:00
64b4aead15
make it work
2016-03-29 03:13:25 +01:00
dd4287ca5d
make it build
2016-03-29 02:07:57 +01:00
e0c2490a14
Merge branch 'develop' into matthew/preview_urls
2016-03-29 01:20:25 +01:00
d9d48aad2d
Merge branch 'develop' into matthew/preview_urls
2016-03-27 22:54:42 +01:00
acdfef7b14
Intern all the things
2016-03-23 16:25:54 +00:00
398cd1edfb
Fix regression where synapse checked whether push rules were valid JSON before the compatibility hack that handled clients sending invalid JSON
2016-03-14 14:16:41 +00:00
e9c1cabac2
Use parse_json_object_from_request to parse JSON out of request bodies
2016-03-11 16:41:03 +00:00
b7dbe5147a
Add a parse_json_object function
...
to deduplicate all the copy+pasted _parse_json functions. Also document
the parse_.* functions.
2016-03-09 11:26:26 +00:00
9a3c80a348
pep8
2016-03-08 10:09:07 +00:00
7bcee4733a
Encode unicode objects given to post_urlencode* otherwise urllib.urlencode chokes.
2016-03-08 10:04:38 +00:00
58c9f20692
Catch the exceptions thrown by twisted when you write to a closed connection
2016-02-12 13:46:59 +00:00
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
13e6262659
Add metrics to pushers
2016-02-08 14:26:45 +00:00
d4f72a5bfb
Allowing tagging log contexts
2016-02-03 13:52:27 +00:00
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
7dd0c1730a
initial WIP of a tentative preview_url endpoint - incomplete, untested, experimental, etc. just putting it here for safekeeping for now
2016-01-24 18:47:27 -05:00
c6549117a2
Fix AttributeError
2016-01-21 14:02:14 +00:00
191070123d
Cache dns lookups, and use the cache if we fail to lookup servers later
2016-01-20 11:34:09 +00:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
2b0f8a9482
Fix typo
2015-12-16 17:59:44 +01:00
338c0a8a69
Include errcode on Internal Server Error
2015-12-14 13:50:50 +00:00
6a5ff5f223
Track the time spent in the database per request.
...
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
99e1d6777f
Add metrics to track the cpu on the main thread consumed by each type of request
2015-12-04 14:42:24 +00:00
14d7acfad4
Host /unstable and /r0 versions of r0 APIs
2015-12-01 17:34:32 +00:00
0eabfa55f6
Fix typo
2015-11-20 17:17:58 +00:00
6408541075
Don't limit connections to perspective servers
2015-11-20 17:15:44 +00:00
2f2bbb4d06
Use min and not max to set an upper bound on retry interval
2015-11-20 09:34:58 +00:00
cbf3cd6151
Add comment
2015-11-17 18:29:29 +00:00
cf4ef5f3c7
Only retry federation requests for a long time for background requests
2015-11-17 18:26:50 +00:00
afdfd12bdf
Implement required method 'resumeProducing'
2015-11-17 16:57:06 +00:00
bceec65913
Slightly more aggressive retry timers at HTTP level
2015-11-17 15:10:05 +00:00
eacb068ac2
Retry dead servers a lot less often
2015-11-02 16:56:30 +00:00
b8dd5b1a2d
Verify third party ID server certificates
2015-10-16 14:54:54 +01:00
a80ef851f7
Fix previous merge to s/version_string/user_agent/
2015-10-10 12:35:39 +01:00
625e13bfde
Add get_raw method to SimpleHttpClient, use this in CAS auth rather than requests
2015-10-10 10:54:19 +01:00
492beb62a8
Use space not dash as delimiter
2015-10-06 09:53:33 -05:00
e0b466bcfd
Use space not dash as delimiter
2015-10-06 09:32:26 -05:00
b28c7da0a4
Preserve version string in user agent
2015-10-05 20:49:39 -05:00
8fc52bc56a
Allow synapse's useragent to be customized
...
This will allow me to write tests which verify which server made HTTP
requests in a federation context.
2015-10-02 17:13:51 -05:00
3bcbabc9fb
Rename context factory
...
Mjark is officially no fun.
2015-09-15 15:46:22 +01:00
6485f03d91
Fix random formatting
2015-09-09 13:05:00 +01:00
81a93ddcc8
Allow configuration to ignore invalid SSL certs
...
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
adbd720fab
PEP8
2015-08-13 11:47:38 +01:00
998a72d4d9
Merge branch 'develop' into markjh/twisted-15
...
Conflicts:
synapse/http/matrixfederationclient.py
2015-08-12 17:21:14 +01:00
a7dcbfe430
Don't 500 if a group is missing from the regex
2015-07-06 16:47:17 +01:00
653533a3da
Fix log context when sending requests
2015-06-19 11:46:49 +01:00
18968efa0a
Remove stale debug lines
2015-06-19 10:18:02 +01:00
9d112f4440
Add IDs to outbound transactions
2015-06-19 10:13:03 +01:00
aaa749d366
Disable twisted access logging. Move access logging to SynapseRequest object
2015-06-15 18:18:05 +01:00
cee69441d3
Log more when we have processed the request
2015-06-15 17:11:44 +01:00
b5209c5744
Create SynapseRequest that overrides __repr__ to not print access_token
2015-06-15 16:37:04 +01:00
f00f8346f1
Make http.server request logging more verbose, but redact access_tokens
2015-06-15 13:37:58 +01:00
83f119a84a
Log requests and responses sent via http.client
2015-06-15 13:14:12 +01:00
b8d49be5a1
Merge branch 'develop' into markjh/twisted-15
...
Conflicts:
synapse/python_dependencies.py
2015-06-01 10:56:05 +01:00
90abdaf3bc
Use Twisted-15.2.1, Use Agent.usingEndpointFactory rather than implement our own Agent
2015-06-01 10:51:50 +01:00
d15f166093
Remove log line
2015-05-29 15:03:24 +01:00
a7b65bdedf
Add config option to turn off freezing events. Use new encode_json api and ujson.loads
2015-05-29 12:17:33 +01:00
79e37a7ecb
Correctly pass connection pool parameter
2015-05-28 16:48:53 +01:00
11f51e6ded
Up maxPersistentPerHost count
2015-05-28 15:45:46 +01:00
086df80790
Add connection pooling to SimpleHttpClient
2015-05-28 15:43:21 +01:00
291e942332
Use connection pool for federation connections
2015-05-28 15:43:21 +01:00
284f55a7fb
Add doc strings
2015-05-22 15:18:04 +01:00
5b1631a4a9
Add a timeout param to get_event
2015-05-19 14:53:32 +01:00
2236ef6c92
Fix up leak. Add warnings.
2015-05-08 19:53:34 +01:00
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
0c1b7f843b
Unused import
2015-04-30 13:33:30 +01:00
4b46fbec5b
Doesn't look like this is used anymore
2015-04-30 12:04:08 +01:00
6532b6e607
Merge branch 'develop' into csauth
...
Conflicts:
synapse/http/server.py
2015-04-24 09:37:54 +01:00
1967650bc4
Combine the request wrappers in rest/media/v1 and http/server into a single wrapper decorator
2015-04-21 16:35:53 +01:00
1ebff9736b
Split out the JsonResource request logging and error handling into a separate wrapper function
2015-04-21 16:07:20 +01:00
cb03fafdf1
Merge branch 'develop' into csauth
2015-04-17 13:51:10 +01:00
3cbc286d06
Move server key api into rest/key/v1
2015-04-14 13:28:11 +01:00
c5bf0343e8
Explain how I justified to myself making JsonResource not always send JSON.
2015-04-01 15:13:14 +01:00
e9c908ebc0
Completely replace fallback auth for C/S V2:
...
* Now only the auth part goes to fallback, not the whole operation
* Auth fallback is a normal API endpoint, not a static page
* Params like the recaptcha pubkey can just live in the config
Involves a little engineering on JsonResource so its servlets aren't always forced to return JSON. I should document this more, in fact I'll do that now.
2015-04-01 15:05:30 +01:00
f129ee1e18
Make docs a bit more true
2015-03-31 18:25:10 +01:00
d18e7779ca
Grammar and deduplication
2015-03-31 14:40:02 +01:00
59bf16eddc
New registration for C/S API v2. Only ReCAPTCHA working currently.
2015-03-30 18:13:10 +01:00
ed4d44d833
Merge pull request #109 from matrix-org/default_registration
...
Disable registration by default. Add script to register new users.
2015-03-18 11:38:52 +00:00
3f7a31d366
Add a DistributionMetric to HTTP request/response processing time in the server
2015-03-16 18:31:29 +00:00
c8032aec17
actually uphold the bind_host parameter. in theory should make ipv6 binds work like bind_host: 'fe80::1%lo0'
2015-03-14 00:12:20 +00:00
dea236e4fa
Add missing commas
2015-03-13 15:24:03 +00:00
128cf2daf7
Appease pep8
2015-03-12 16:24:51 +00:00
1748605c5d
Count incoming HTTP requests per servlet that responds
2015-03-12 16:24:51 +00:00
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
7d72e44eb9
Add vector counters to HTTP clients and servers; count the requests by method and responses by method and response code
2015-03-12 16:24:50 +00:00
c53ec53d80
Pull out all uses of the underlying HTTP user agent .request() method into a single wrapper function, to make adding metrics easier
2015-03-12 16:24:50 +00:00
9d9d39536b
Slightly reduce the insane amounts of indentation in main http server response path, by 'continue'ing around a non-match or falling through
2015-03-05 16:24:13 +00:00
2de5b14fe0
Fix bug which prevented the HS pushing events to the AS due to FrozenEvents
2015-03-02 15:36:37 +00:00
David Baker