18b69be00f
Rewrite Datastore.get_server_verify_keys
...
Rewrite this so that it doesn't hammer the database.
2019-04-09 00:00:10 +01:00
f88a9e6323
Remove redundant merged_keys dict
...
There's no point in collecting a merged dict of keys: it is sufficient to
consider just the new keys which have been fetched by the most recent
key_fetch_fns.
2019-04-08 22:36:18 +01:00
7d2a0c848e
Fix from_server buglet in get_keys_from_perspectives
...
make sure we store the name of the server the keys came from, rather than the
origin server, after doing a fetch-from-perspectives.
2019-04-08 12:51:16 +01:00
6ae9361510
Hoist server_name check out of process_v2_response
...
It's easier to check it in the caller than to complicate the interface with an
extra param.
2019-04-04 19:12:54 +01:00
ef27d434d1
Clean up Keyring.process_v2_response
...
Make this just return the key dict, rather than a single-entry dict mapping the
server name to the key dict. It's easy for the caller to get the server name
from from the response object anyway.
2019-04-04 19:12:54 +01:00
78c563b77c
Correctly log expected errors when fetching server keys
2019-03-11 14:11:10 +00:00
65d1003d01
raise_from already raises
2019-02-25 14:34:03 +00:00
41285ffe5b
Handle errors when fetching remote server keys
2019-02-23 15:09:39 +00:00
561eebe170
fix to use makeContext so that we don't need to rebuild the certificateoptions each time
2019-02-19 16:18:05 +11:00
7fc1196a36
Correctly handle RequestSendFailed exceptions
...
This mainly reduces the number of exceptions we log.
2019-02-14 14:01:04 +00:00
9645728619
Don't create server contexts when TLS is disabled
...
we aren't going to use them anyway.
2019-02-11 21:32:01 +00:00
554ca58ea1
Make add_hashes_and_signatures operate on dicts
2019-01-29 11:12:38 +00:00
855a151015
Refactor event signing to work on dicts
...
This is in preparation for making EventBuilder format agnostic, which
means event signing should be done against the event dict rather than
the EventBuilder object.
2019-01-28 16:42:10 +00:00
97fd29c019
Don't send IP addresses as SNI ( #4452 )
...
The problem here is that we have cut-and-pasted an impl from Twisted, and then
failed to maintain it. It was fixed in Twisted in
https://github.com/twisted/twisted/pull/1047/files ; let's do the same here.
2019-01-24 09:34:44 +00:00
6bfa735a69
Make key fetches use regular federation client ( #4426 )
...
All this magic is redundant.
2019-01-22 11:04:20 +00:00
23b0813599
Require ECDH key exchange & remove dh_params ( #4429 )
...
* remove dh_params and set better cipher string
2019-01-22 21:58:50 +11:00
916efc8249
Remove fetching keys via the deprecated v1 kex method ( #4120 )
2018-10-31 23:14:39 +11:00
ef771cc4c2
Fix a number of flake8 errors
...
Broadly three things here:
* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
them
* use `r""` for strings which include backslashes
Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
2018-10-24 10:39:03 +01:00
33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring
...
add some logging for the keyring queue
2018-09-12 20:43:47 +10:00
8fd93b5eea
Port crypto/ to Python 3 ( #3822 )
2018-09-12 20:16:31 +10:00
806964b5de
add some logging for the keyring queue
...
why is it so damn slow?
2018-09-06 18:51:06 +01:00
808d8e06aa
Don't log exceptions when failing to fetch server keys
...
Not being able to resolve or connect to remote servers is an expected
error, so we shouldn't log at ERROR with stacktraces.
2018-08-21 11:19:26 +01:00
2e9c73e8ca
more generic conversion of str/bytes to unicode
2018-08-09 21:31:26 +02:00
64899341dc
include private functions from twisted
2018-08-09 21:04:22 +02:00
d5c0ce4cad
updated docstring for ServerContextFactory
2018-08-08 19:25:01 +02:00
2903e65aff
fix isort
2018-07-29 19:47:08 +02:00
8e3f75b39a
fix accidental removal of hs
2018-07-27 12:17:31 +02:00
505530f36a
Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
...
# Conflicts:
# synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
b5e157d895
Merge branch 'develop' into send_sni_for_federation_requests
...
# Conflicts:
# synapse/http/endpoint.py
2018-07-09 08:51:11 +02:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
95341a8f6f
take idna implementation from twisted
2018-06-26 21:15:14 +02:00
b7f34ee348
allow self-signed certificates
2018-06-26 20:41:05 +02:00
07b4f88de9
formatting changes for pep8
2018-06-25 12:31:16 +02:00
3d605853c8
send SNI for federation requests
2018-06-24 22:38:43 +02:00
e82db24a0e
Try to log more helpful info when a sig verification fails
...
Firstly, don't swallow the reason for the failure
Secondly, don't assume all exceptions are verification failures
Thirdly, log a bit of info about the key being used if debug is enabled
2018-06-08 12:13:08 +01:00
2ad3fc36e6
Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve ( #3157 )
...
fixes #3135
Signed-off-by: Will Hunt will@half-shot.uk
2018-04-30 16:21:11 +01:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
1515560f5c
Use str(e) instead of e.message
...
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:32:42 +02:00
7303ed65e1
Fix 500 when joining matrix-dev
...
matrix-dev has an event (`$/6ANj/9QWQyd71N6DpRQPf+SDUu11+HVMeKSpMzBCwM:zemos.net`)
which has no `hashes` member.
Check for missing `hashes` element in events.
2017-11-29 16:00:46 +00:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
94133d7ce8
Merge branch 'develop' into develop
2017-09-25 11:50:11 +01:00
79b3cf3e02
Fix logcontxt leak in keyclient ( #2465 )
...
preserve_context_over_function doesn't do what you want it to do.
2017-09-25 09:51:39 +01:00
c5c24c239b
Fix logcontext handling in verify_json_objects_for_server
...
preserve_context_over_fn is essentially broken, because (a) it pointlessly
drops the current logcontext before calling its wrapped function, which means
we don't get any useful logcontexts for _handle_key_deferred; (b) it wraps the
resulting deferred in a _PreservingContextDeferred, which is very dangerous
because you then can't yield on it without leaking context back into the
reactor.
Instead, let's specify that the resultant deferreds call their callbacks with
no logcontext.
2017-09-20 01:32:42 +01:00
c5b0e9f485
Turn _start_key_lookups into an inlineCallbacks function
...
... which means that logcontexts can be correctly preserved for the stuff it
does.
get_server_verify_keys is now called with the logcontext, so needs to
preserve_fn when it fires off its nested inlineCallbacks function.
Also renames get_server_verify_keys to reflect the fact it's meant to be
private.
2017-09-20 01:32:42 +01:00
abdefb8a01
Fix potential race in _start_key_lookups
...
If the verify_request.deferred has already completed, then `remove_deferreds`
will be called immediately. It therefore might resolve the server_to_deferred
deferred while there are still other requests for that server in flight.
To avoid that, we should build the complete list of requests, and *then* add the
callbacks.
2017-09-20 01:32:42 +01:00
afbd773dc6
Add some comments to _start_key_lookups
2017-09-20 01:32:42 +01:00
2a4b9ea233
Consistency for how verify_request.deferred is called
...
Define that it is run with no log context, and make sure that happens.
If we aren't careful to reset the logcontext, we can't bung the deferreds into
defer.gatherResults etc. We don't actually do that directly, but we *do*
resolve other deferreds from affected callbacks (notably the server_to_deferred
map in _start_key_lookups), and those *do* get passed into
defer.gatherResults. It turns out that this way ends up being least confusing.
2017-09-20 01:32:42 +01:00
3b98439eca
Factor out _start_key_lookups
...
... to make it easier to see what's going on.
2017-09-20 01:32:42 +01:00
fde63b880d
Replace `server_and_json` with `verify_requests`
...
This is a precursor to factoring some of this code out.
2017-09-20 01:32:42 +01:00
2d511defd9
pull out handle_key_deferred to top level
...
There's no need for this to be a nested definition; pulling it out not only
makes it more efficient, but makes it easier to check that it's not accessing
any local variables it shouldn't be.
2017-09-20 01:32:42 +01:00
dd1ea9763a
Fix incorrect key_ids in error message
2017-09-20 01:32:42 +01:00
9864efa532
Fix concurrent server_key requests ( #2458 )
...
Fix a bug where we could end up firing off multiple requests for server_keys
for the same server at the same time.
2017-09-19 23:25:44 +01:00
290777b3d9
Clean up and document handling of logcontexts in Keyring ( #2452 )
...
I'm still unclear on what the intended behaviour for
`verify_json_objects_for_server` is, but at least I now understand the
behaviour of most of the things it calls...
2017-09-18 18:31:01 +01:00
f18373dc5d
Fix iteration of requests_missing_keys; list doesn't have .values()
...
Signed-off-by: Kenny Keslar <r3dey3@r3dey3.com>
2017-07-26 22:44:19 -05:00
9397edb28b
Merge pull request #2050 from matrix-org/rav/federation_backoff
...
push federation retry limiter down to matrixfederationclient
2017-03-23 22:27:01 +00:00
4bd597d9fc
push federation retry limiter down to matrixfederationclient
...
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
64778693be
fix up some key verif docstrings
2017-03-21 13:27:50 +00:00
c36d15d2de
Add some debug to help diagnose weird federation issue
2017-03-20 15:36:14 +00:00
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
04fc8bbcb0
Update keyring Measure
2016-08-19 18:23:44 +01:00
2426c2f21a
Measure keyrings
2016-08-19 18:23:44 +01:00
fa1ce4d8ad
Don't print stack traces when failing to get remote keys
2016-08-10 10:44:37 +01:00
a285194021
Merge branch 'erikj/key_client_fix' of github.com:matrix-org/synapse into release-v0.17.0
2016-07-28 10:47:06 +01:00
29b25d59c6
Merge branch 'develop' into markjh/verify
...
Conflicts:
synapse/crypto/keyring.py
2016-07-27 15:11:02 +01:00
884b800899
Merge pull request #955 from matrix-org/markjh/only_from2
...
Add a couple more checks to the keyring
2016-07-27 15:08:22 +01:00
fe1b369946
Clean up verify_json_objects_for_server
2016-07-27 14:10:43 +01:00
a4b06b619c
Add a couple more checks to the keyring
2016-07-26 19:50:11 +01:00
87ffd21b29
Fix a couple of bugs in the transaction and keyring code
2016-07-26 19:19:08 +01:00
d26b660aa6
Cache getPeer
2016-07-21 17:38:51 +01:00
cf94a78872
Set host not path
2016-07-21 11:45:53 +01:00
081e5d55e6
Send the correct host header when fetching keys
2016-07-21 11:14:54 +01:00
f28643cea9
Uncommit accidentally commited edit to cipher list
2016-05-10 18:44:32 +02:00
d46b18a00f
Pass through _get_event_txn
2016-05-10 18:27:06 +02:00
2e2be463f8
Make key client send a Host header
2016-03-11 10:29:05 +00:00
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
a6ba41e078
Actually look up required remote server key IDs
...
set.union() is a side-effect-free function that returns the union of two
sets. This clearly wanted .update(), which is the side-effecting mutator
version.
2015-12-18 21:36:42 +00:00
0eabfa55f6
Fix typo
2015-11-20 17:17:58 +00:00
6408541075
Don't limit connections to perspective servers
2015-11-20 17:15:44 +00:00
ffe8cf7e59
Fix bug where we sometimes didn't fetch all the keys requested for a
...
server.
2015-09-17 10:21:32 +01:00
2c8f16257a
Merge pull request #272 from matrix-org/daniel/insecureclient
...
Allow configuration to ignore invalid SSL certs
2015-09-15 16:52:38 +01:00
dd0867f5ba
Various bug fixes to crypto.keyring
2015-09-09 17:02:39 +01:00
81a93ddcc8
Allow configuration to ignore invalid SSL certs
...
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
0b3389bcd2
Merge pull request #194 from matrix-org/erikj/bulk_verify_sigs
...
Implement bulk verify_signed_json API
2015-07-10 13:46:53 +01:00
fb8d2862c1
remove the tls_certificate_chain_path param and simply support tls_certificate_path pointing to a file containing a chain of certificates
2015-07-09 00:45:41 +01:00
f26a3df1bf
oops, context.tls_certificate_chain_file() expects a file, not a certificate.
2015-07-08 21:33:02 +01:00
19fa3731ae
typo
2015-07-08 18:53:41 +01:00
64afbe6ccd
add new optional config for tls_certificate_chain_path for folks with intermediary SSL certs
2015-07-08 18:20:02 +01:00
f0dd568e16
Wait for previous attempts at fetching keys for a given server before trying to fetch more
2015-06-26 11:25:00 +01:00
b5f55a1d85
Implement bulk verify_signed_json API
2015-06-26 10:39:34 +01:00
291cba284b
Handle the case when things return empty but non none things
2015-05-19 14:42:46 +01:00
253f76a0a5
Don't always hit get_server_verify_key_v1_direct
2015-05-19 14:42:38 +01:00
d3e09f12d0
SYN-383: Actually, we expect this value to be a dict
2015-05-19 13:12:41 +01:00
2b7120e233
SYN-383: Handle the fact the server might not have signed things
2015-05-19 12:49:38 +01:00
8b256a7296
Don't reuse var names
2015-05-19 11:58:22 +01:00
2aeee2a905
SYN-383: Fix parsing of verify_keys and catching of _DefGen_Return
2015-05-19 11:56:18 +01:00
c6a03c46e6
SYN-383: Extract the response list from 'server_keys' in the response JSON as it might work better than iterating over the top level dict
2015-05-19 10:23:02 +01:00
ec07dba29e
Merge pull request #143 from matrix-org/erikj/SYN-375
...
SYN-375 - Lots of unhandled deferred exceptions.
2015-05-12 15:25:54 +01:00
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
fca28d243e
Change the way we create observers to deferreds so that we don't get spammed by 'unhandled errors'
2015-05-08 16:28:08 +01:00
1319905d7a
Use a defer.gatherResults to collect results from the perspective servers
2015-04-29 13:31:14 +01:00
74874ffda7
Update the query format used by keyring to match current key v2 spec
2015-04-29 12:14:08 +01:00
46d200a3a1
Implement minimum_valid_until_ts in the remote key resource
2015-04-29 11:57:26 +01:00
f8b865264a
Merge branch 'develop' into key_distribution
...
Conflicts:
synapse/crypto/keyring.py
2015-04-27 18:29:32 +01:00
2c70849dc3
Fix newlines
2015-04-27 14:38:29 +01:00
0a016b0525
Pull inner function out.
2015-04-27 14:37:24 +01:00
e701aec2d1
Implement locks using create_observer for fetching media and server keys
2015-04-27 14:20:26 +01:00
288702170d
Add config for setting the perspective servers
2015-04-24 17:01:34 +01:00
4bbf7156ef
Update to match the specification for key/v2
2015-04-23 16:39:13 +01:00
f30d47c876
Implement remote key lookup api
2015-04-22 14:21:08 +01:00
2f9157b427
Implement v2 key lookup
2015-04-20 16:23:47 +01:00
8d761134c2
Fail quicker for 4xx responses in the key client, optional hit a different API path
2015-04-15 16:57:58 +01:00
3ce8540484
Don't look for an TLS private key if we have set --no-tls
2015-03-06 11:34:06 +00:00
5b5c7a28d6
Log error message when we fail to fetch remote server keys
2015-03-05 17:09:13 +00:00
9371019133
Try to only back off if we think we failed to connect to the remote
2015-02-17 18:13:34 +00:00
2b8f1a956c
Add per server retry limiting.
...
Factor out the pre destination retry logic from TransactionQueue so it
can be reused in both get_pdu and crypto.keyring
2015-02-17 17:20:56 +00:00
5025305fb2
Rate limit retries when fetching server keys.
2015-02-17 15:57:42 +00:00
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
84a769cdb7
Fix code-style
2015-02-10 17:58:36 +00:00
823999716e
Fix bug in timeout handling in keyclient
2015-01-30 11:08:01 +00:00
adb04b1e57
Update copyright notices
2015-01-06 13:21:39 +00:00
95aa903ffa
Try and figure out how and why signatures are being changed.
2014-12-10 11:37:47 +00:00
609c31e8df
More bug fixes
2014-12-08 17:50:56 +00:00
c31dba86ec
Convert rest and handlers to use new event structure
2014-12-04 15:50:01 +00:00
75b4329aaa
WIP for new way of managing events.
2014-12-03 16:07:21 +00:00
f1c7f8e813
Merge branch 'develop' into http_client_refactor
2014-11-20 17:49:48 +00:00
e377d33652
Separate out the matrix http client completely because just about all of its code it now separate from the simple case we need for standard HTTP(S)
2014-11-20 17:41:56 +00:00
dfdda2c871
Use module loggers rather than the root logger. Exceptions caused by bad clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str'
2014-11-20 17:10:37 +00:00
32090aee16
Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts
2014-11-20 16:24:00 +00:00
cb4b6c844a
Merge PDUs and Events into one object
2014-11-14 21:25:02 +00:00
8c2b5ea7c4
Fix PDU and event signatures
2014-11-14 19:11:04 +00:00
de1ec90133
Validate signatures on incoming events
2014-11-14 19:11:04 +00:00
e903c941cb
Merge branch 'develop' into request_logging
...
Conflicts:
setup.py
synapse/storage/_base.py
synapse/util/async.py
2014-11-14 11:16:50 +00:00
6cb6cb9e69
Tidy up some of the unused sql tables
2014-11-10 10:31:00 +00:00
1c06806f90
Finish redaction algorithm.
2014-11-10 10:21:32 +00:00
97a096b507
Add hash of current state to events
2014-11-07 11:37:06 +00:00
68698e0ac8
Fix bugs in generating event signatures and hashing
2014-11-03 17:51:42 +00:00
82a6b83524
Don't assume event has hashes key already
2014-11-03 11:32:12 +00:00
ecabff7eb4
Sign evnets
2014-10-31 17:08:52 +00:00
d30d79b5be
Make prev_event signing work again.
2014-10-31 15:35:39 +00:00
ef9c4476a0
Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization
2014-10-30 11:18:28 +00:00
7c06399512
Merge branch 'develop' into request_logging
...
Conflicts:
synapse/config/logger.py
2014-10-30 11:13:58 +00:00
7d709542ca
Fix pep8 warnings
2014-10-30 11:10:17 +00:00
b29517bd01
Add a request-id to each log line
2014-10-30 01:21:33 +00:00
5e2236f9ff
fix pyflakes warnings
2014-10-27 11:19:15 +00:00
acb2d171e8
Merge branch 'develop' into event_signing
2014-10-27 11:14:11 +00:00
Richard van der Hoff