bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib
...
move httplib import to six
2018-04-30 01:26:42 +01:00
683149c1f9
Merge pull request #3151 from NotAFile/py3-xrange-1
...
Move more xrange to six
2018-04-30 01:20:06 +01:00
3b0e431c82
Merge pull request #3150 from NotAFile/py3-listcomp-yield
...
Don't yield in list comprehensions
2018-04-30 01:11:41 +01:00
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
2fd96727b1
Merge pull request #3085 from NotAFile/py3-config-text-mode
...
Open config file in non-bytes mode
2018-04-30 01:00:23 +01:00
b8ee12b978
Merge pull request #3084 from NotAFile/py3-certs-byte-mode
...
Open certificate files as bytes
2018-04-30 01:00:05 +01:00
049b0b5af2
Merge pull request #3154 from NotAFile/py3-stringio
...
Replace stringIO imports with six
2018-04-30 00:59:04 +01:00
af3cc50511
Remove redundant call to preserve_fn
...
submit_event_for_as doesn't return a deferred anyway, so this is pointless.
2018-04-30 00:48:36 +01:00
dbf6f28d64
Merge pull request #3155 from NotAFile/py3-bytes-1
...
more bytes strings
2018-04-30 00:38:21 +01:00
aab2e4da60
Merge pull request #3140 from matrix-org/rav/use_run_in_background
...
Use run_in_background in preference to preserve_fn
2018-04-30 00:34:28 +01:00
1315d374cc
Merge pull request #3156 from NotAFile/py3-hmac-bytes
...
Construct HMAC as bytes on py3
2018-04-30 00:33:20 +01:00
9e2601f830
Merge pull request #3108 from NotAFile/py3-six-urlparse
...
Use six.moves.urlparse
2018-04-30 00:33:05 +01:00
122593265b
Construct HMAC as bytes on py3
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:19:41 +02:00
e9143b6593
more bytes strings
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:13:57 +02:00
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
4f2f5171b7
replace stringIO imports
2018-04-28 13:46:23 +02:00
94f4d7f49e
move httplib import to six
2018-04-28 13:43:34 +02:00
57b58e2174
make imports local
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:41:41 +02:00
cdb4647a80
Don't yield in list comprehensions
...
I've tried to grep for more of this with no success.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:36:30 +02:00
a376d8f761
open log_config in text mode too
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:34:13 +02:00
453adf00b6
pep8; remove spurious import
2018-04-27 14:32:08 +01:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
6146332387
Merge remote-tracking branch 'origin/develop' into rav/deferred_timeout
2018-04-27 14:18:00 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
3d1ae61399
Merge branch 'develop' into rav/deferred_timeout
2018-04-27 12:54:43 +01:00
9d2c1b8429
Backport deferred.addTimeout
...
Twisted 16.0 doesn't have addTimeout, so let's backport it.
2018-04-27 12:52:30 +01:00
13843f771e
Trap exceptions thrown within run_in_background
...
Turn any exceptions that get thrown synchronously within run_in_background into
Failures instead.
2018-04-27 12:17:13 +01:00
41d4b07a53
Merge pull request #3142 from matrix-org/rav/reraise
...
reraise exceptions more carefully
2018-04-27 12:16:19 +01:00
268e40341b
Merge pull request #3136 from matrix-org/rav/fix_dependencies
...
Miscellaneous fixes to python_dependencies
2018-04-27 11:48:28 +01:00
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
53494c34df
Merge pull request #3139 from matrix-org/rav/consume_errors
...
Add missing consumeErrors to improve exception handling
2018-04-27 11:47:21 +01:00
6493b22b42
reraise exceptions more carefully
...
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
2018-04-27 11:40:06 +01:00
6e10eed28e
Refactor event storage to not require state
...
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.
We also add a check to see if the state groups of the old extremities
are the same as the new ones.
2018-04-27 11:38:02 +01:00
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
28dd536e80
update changelog and bump version to 0.28.0
2018-04-26 15:51:39 +01:00
d78ada3166
Miscellaneous fixes to python_dependencies
...
* add some doc about wtf this thing does
* pin Twisted to < 18.4
* add explicit dep on six (fixes #3089 )
2018-04-26 13:11:03 +01:00
7ec8e798b4
Fix media admin APIs
2018-04-26 11:31:22 +01:00
fb6015d0a6
pep8
2018-04-25 17:56:11 +01:00
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
22881b3d69
Also fix reindexing of search
2018-04-25 15:32:04 +01:00
ba3166743c
Fix quarantine media admin API
2018-04-25 15:11:18 +01:00
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
2c3e995f38
Bump version and update changelog
2018-04-24 15:33:22 +01:00
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events
...
Reject events which have lots of prev_events
2018-04-23 17:51:38 +01:00
9b9c38373c
Remove spurious param
2018-04-23 12:00:06 +01:00
1ea904b9f0
Use deferred.addTimeout instead of time_bound_deferred
...
This doesn't feel like a wheel we need to reinvent.
2018-04-23 00:53:18 +01:00
dc875d2712
Merge pull request #3106 from NotAFile/py3-six-itervalues-1
...
Use six.itervalues in some places
2018-04-20 15:43:52 +01:00
8dc4a6144b
Merge pull request #3107 from NotAFile/py3-bool-nonzero
...
add __bool__ alias to __nonzero__ methods
2018-04-20 15:43:39 +01:00
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
...
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
692a3cc806
Merge pull request #3103 from NotAFile/py3-baseexcepton-message
...
Use str(e) instead of e.message
2018-04-20 15:34:49 +01:00
3de7d9fe99
accept stupid events over backfill
2018-04-20 11:41:03 +01:00
11a67b7c9d
Merge pull request #3093 from matrix-org/rav/response_cache_wrap
...
Refactor ResponseCache usage
2018-04-20 11:31:17 +01:00
0c280d4d99
Reinstate linearizer for federation_server.on_context_state_request
2018-04-20 11:10:04 +01:00
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
1f4b498b73
Add some comments
2018-04-18 00:15:36 +01:00
e585228860
Check events on backfill too
2018-04-18 00:06:42 +01:00
9b7794262f
Reject events which have too many auth_events or prev_events
...
... this should protect us from being dossed by people making silly events
(deliberately or otherwise)
2018-04-18 00:06:42 +01:00
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
878995e660
Replace Queue with six.moves.queue
...
and a six.range change which I missed the last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
2a3c33ff03
Use six.moves.urlparse
...
The imports were shuffled around a bunch in py3
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 21:22:43 +02:00
f63ff73c7f
add __bool__ alias to __nonzero__ methods
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:40:47 +02:00
36c59ce669
Use six.itervalues in some places
...
There's more where that came from
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
1515560f5c
Use str(e) instead of e.message
...
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:32:42 +02:00
bfc2ade9b3
Make event properties raise AttributeError instead
...
They raised KeyError before. I'm changing this because the code uses
hasattr() to check for the presence of a key. This worked accidentally
before, because hasattr() silences all exceptions in python 2. However,
in python3, this isn't the case anymore.
I had a look around to see if anything depended on this raising a
KeyError and I couldn't find anything. Of course, I could have simply
missed it.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:16:59 +02:00
154b44c249
Merge branch 'master' of https://github.com/matrix-org/synapse into develop
2018-04-13 17:07:54 +01:00
78a9698650
fix federation_domain_whitelist
...
we were checking the wrong server_name on inbound requests
2018-04-13 15:47:43 +01:00
25b0ba30b1
revert last to PR properly
2018-04-13 15:46:37 +01:00
f8d46cad3c
correctly auth inbound federation_domain_whitelist reqs
2018-04-13 15:41:52 +01:00
d4b2e05852
Merge branch 'master' of https://github.com/matrix-org/synapse into develop
2018-04-13 12:16:27 +01:00
51d628d28d
Bump version and Change log
2018-04-13 12:08:19 +01:00
d3347ad485
Revert "Use sortedcontainers instead of blist"
...
This reverts commit 9fbe70a7dc
2018-04-13 11:16:43 +01:00
fac3f9e678
Bump canonicaljson to 1.1.3
...
1.1.2 was a bit broken too :/
2018-04-13 10:21:38 +01:00
60f6014bb7
ResponseCache: fix handling of completed results
...
Turns out that ObservableDeferred.observe doesn't return a deferred if the
result is already completed. Fix handling and improve documentation.
2018-04-13 07:32:29 +01:00
119596ab8f
Update canonicaljson dependency
...
1.1.0 and 1.1.1 were broken, so we're updating this to help people make sure
they don't end up on a broken version.
Also, 1.1.0 is speedier...
2018-04-12 17:31:44 +01:00
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
d5c74b9f6c
Merge pull request #3092 from matrix-org/rav/response_cache_metrics
...
Add metrics for ResponseCache
2018-04-12 12:59:36 +01:00
415aeefd89
Format docstring
2018-04-12 12:07:09 +01:00
19ceb4851f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag
2018-04-12 11:36:07 +01:00
261124396e
Merge pull request #3059 from matrix-org/rav/doc_response_cache
...
Document the behaviour of ResponseCache
2018-04-12 11:22:30 +01:00
23a7f9d7f4
Doc we raise on unknown event
2018-04-12 11:20:51 +01:00
d7bf3a68f0
s/list/tuple
2018-04-12 11:19:04 +01:00
f67e906e18
Set all metrics at the same time
2018-04-12 11:18:19 +01:00
971059a733
Merge pull request #3088 from matrix-org/erikj/as_parallel
...
Send events to ASes concurrently
2018-04-12 10:42:36 +01:00
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
ab825aa328
Add GaugeMetric
2018-04-11 12:13:40 +01:00
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
781cd8c54f
bump version/changelog
2018-04-11 10:54:43 +01:00
9ef0b179e0
Merge commit '11d2609da70af797405241cdf7d9df19db5628f2' of https://github.com/matrix-org/synapse into release-v0.27.0
2018-04-11 10:51:59 +01:00
121591568b
Send events to ASes concurrently
2018-04-11 09:56:00 +01:00
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
eaa2ebf20b
Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends
...
Limit concurrent event sends for a room
2018-04-10 16:43:58 +01:00
9daf82278f
Merge pull request #3078 from matrix-org/erikj/federation_sender
...
Send federation events concurrently
2018-04-10 16:43:48 +01:00
a3f9ddbede
Open certificate files as bytes
...
That's what pyOpenSSL expects on python3
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-10 17:36:29 +02:00
7f8eebc8ee
Open config file in non-bytes mode
...
Nothing written into it is encoded, so it makes little sense, but it
does break in python3 the way it was before.
The variable names were adjusted to be less misleading.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-10 17:32:40 +02:00
dd723267b2
Merge branch 'release-v0.27.0' of https://github.com/matrix-org/synapse into develop
2018-04-10 15:03:29 +01:00
a060dfa132
Use run_in_background instead
2018-04-10 14:25:11 +01:00
f8e8ec013b
Note why we're limiting concurrent event sends
2018-04-10 14:00:46 +01:00
1246d23710
Preserve log contexts correctly
2018-04-10 12:04:32 +01:00
d49cbf712f
Log event ID on exception
2018-04-10 12:03:41 +01:00
ce72d590ed
Merge pull request #3082 from matrix-org/erikj/urlencode_paths
...
URL quote path segments over federation
2018-04-10 11:31:16 +01:00
11d2609da7
Ensure slashes are escaped
2018-04-10 11:24:40 +01:00
dab87b84a3
URL quote path segments over federation
2018-04-10 11:16:08 +01:00
6d7f0f8dd3
Don't disable GC when running on PyPy
...
PyPy's incminimark GC can't be triggered manually. From what I observed
there are no obvious issues with just letting it run normally. And
unlike CPython, it actually returns unused RAM to the system.
Signed-off-by: Vincent Breitmoser <look@my.amazin.horse>
2018-04-10 11:35:34 +02:00
f4284d943a
In DomainSpecificString, override __repr__ in addition to __str__
...
For some reason, string interpolation on a DomainSpecificString object
like "%r" % (domainSpecificStringObj) fails under PyPy, because the
default __repr__ implementation wants to iterate over the object. I'm
not sure why that happens, but overriding __repr__ instead of __str__
fixes this problem, and is arguably the more appropriate thing to do
anyways.
2018-04-10 11:35:29 +02:00
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
9fbe70a7dc
Use sortedcontainers instead of blist
...
This commit drop-in replaces blist with SortedContainers. They are
written in pure python so work with pypy, but perform as good as
native implementations, at least in a couple benchmarks:
http://www.grantjenks.com/docs/sortedcontainers/performance.html
2018-04-10 11:29:51 +02:00
a3599dda97
Merge pull request #2996 from krombel/allow_auto_join_rooms
...
move handling of auto_join_rooms to RegisterHandler
2018-04-10 01:11:00 +01:00
87478c5a60
Merge pull request #3061 from NotAFile/add-some-byte-strings
...
Add b prefixes to some strings that are bytes in py3
2018-04-09 23:54:05 +01:00
c508b2f2f0
Merge pull request #3073 from NotAFile/use-six-reraise
...
Replace old-style raise with six.reraise
2018-04-09 23:53:40 +01:00
37354b55c9
Merge pull request #2938 from dklug/develop
...
Return 401 for invalid access_token on logout
2018-04-09 23:52:56 +01:00
0e9aa1d091
Merge pull request #3074 from NotAFile/fix-py3-prints
...
use python3-compatible prints
2018-04-09 23:44:41 +01:00
8eaa141d8f
Merge pull request #3075 from NotAFile/six-type-checks
...
Replace some type checks with six type checks
2018-04-09 23:40:44 +01:00
664adb4236
Merge pull request #3016 from silkeh/improve-service-lookups
...
Improve handling of SRV records for federation connections
2018-04-09 23:40:06 +01:00
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
64bc2162ef
Fix psycopg2 interpolation
2018-04-09 17:50:36 +01:00
d2c6f4d626
Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin
...
Return a 404 rather than a 500 on rejoining empty rooms
2018-04-09 17:32:36 +01:00
5232d3bfb1
version bump v0.27.3-rc2
2018-04-09 17:25:57 +01:00
5e785d4d5b
Merge branch 'develop' of https://github.com/matrix-org/synapse into release-v0.27.0
2018-04-09 17:21:34 +01:00
6e025a97b4
Handle all events in a room correctly
2018-04-09 16:02:48 +01:00
d9ae2bc826
bump version to release candidate
2018-04-09 16:00:31 +01:00
072fb59446
bump version
2018-04-09 13:49:25 +01:00
13decdbf96
Revert "Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics"
...
We aren't ready to release this yet, so I'm reverting it for now.
This reverts commit d1679a4ed7e089100c62
2018-04-09 12:59:12 +01:00
f3ef60662f
Return a 404 rather than a 500 on rejoining empty rooms
...
Filter ourselves out of the server list before checking for an empty remote
host list, to fix 500 error
Fixes #2141
2018-04-09 12:56:22 +01:00
e5082494eb
Limit concurrent event sends for a room
2018-04-09 12:07:39 +01:00
56b0589865
Use create_and_send_nonmember_event everywhere
2018-04-09 12:04:18 +01:00
11974f3787
Send federation events concurrently
2018-04-09 11:47:10 +01:00
145d14656b
Handle exceptions in get_hosts_for_room when sending events over federation
2018-04-09 11:47:01 +01:00
e54c202b81
Replace some type checks with six type checks
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-07 01:02:32 +02:00
b0500d3774
use python3-compatible prints
2018-04-06 23:35:27 +02:00
4f40d058cc
Replace old-style raise with six.reraise
...
The old style raise is invalid syntax in python3. As noted in the docs,
this adds one more frame in the traceback, but I think this is
acceptable:
<ipython-input-7-bcc5cba3de3f> in <module>()
16 except:
17 pass
---> 18 six.reraise(*x)
/usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
691 if value.__traceback__ is not tb:
692 raise value.with_traceback(tb)
--> 693 raise value
694 finally:
695 value = None
<ipython-input-7-bcc5cba3de3f> in <module>()
9
10 try:
---> 11 x()
12 except:
13 x = sys.exc_info()
Also note that this uses six, which is not formally a dependency yet,
but is included indirectly since most packages depend on it.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-06 23:06:24 +02:00
135fc5b9cd
Merge pull request #3046 from matrix-org/dbkr/join_group
...
Implement group join API
2018-04-06 16:24:32 +01:00
020a501354
de-lint, quote consistency
2018-04-06 16:02:06 +01:00
db2fd801f7
Explicitly grab individual columns from group object
2018-04-06 15:57:25 +01:00
7945435587
When exposing group state, return is_openly_joinable
...
as opposed to join_policy, which is really only pertinent to the
synapse implementation of the group server.
By doing this we keep the group server concept extensible by
allowing arbitrarily complex rules for deciding whether a group
is openly joinable.
2018-04-06 15:43:27 +01:00
6bd1b7053e
By default, join policy is "invite"
2018-04-06 15:43:27 +01:00
b4478e586f
add_user -> _add_user
2018-04-06 15:43:27 +01:00
112c2253e2
pep8
2018-04-06 15:43:27 +01:00
6850f8aea3
Get group_info from existing call to check_group_is_ours
2018-04-06 15:43:27 +01:00
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
f8d1917fce
Fix federation client `set_group_joinable` typo
2018-04-06 15:43:27 +01:00
6eb3aa94b6
Factor out add_user from accept_invite and join_group
2018-04-06 15:43:27 +01:00
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
7b824f1475
Add response size metrics
2018-04-06 13:20:11 +01:00
1d71f484d4
use PUT instead of POST for federating groups/m.join_policy
2018-04-06 12:54:09 +02:00
15e8ed874f
more verbosity in synctl
2018-04-06 09:28:36 +01:00
c7f0969731
Merge pull request #2986 from jplatte/join_reponse_room_id
...
Add room_id to the response of `rooms/{roomId}/join`
2018-04-05 17:29:06 +01:00
3449da3bc7
Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation
...
Improve database cache performance
2018-04-05 17:21:44 +01:00
d1679a4ed7
Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics
...
Remove redundant metrics which were deprecated in 0.27.0.
2018-04-05 17:21:18 +01:00
01afc563c3
Fix overzealous cache invalidation
...
Fixes an issue where a cache invalidation would invalidate *all* pending
entries, rather than just the entry that we intended to invalidate.
2018-04-05 16:24:04 +01:00
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
104c0bc1d5
Use "/settings/" (plural)
2018-04-05 14:07:16 +01:00
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
518f6de088
Remove redundant metrics which were deprecated in 0.27.0.
2018-04-04 19:46:28 +01:00
e4570c53dd
phone home cache size configurations
2018-04-04 16:46:58 +01:00
6168351877
Add b prefixes to some strings that are bytes in py3
...
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
f92963f5db
Revert "improve mxid check performance"
2018-04-04 12:08:29 +01:00
72251d1b97
Remove address resolution of hosts in SRV records
...
Signed-off-by: Silke Hofstra <silke@slxh.eu>
2018-04-04 12:26:50 +02:00
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style
...
Replace old style error catching with 'as' keyword
2018-04-04 10:45:22 +01:00
a89f9f830c
Merge pull request #3044 from matrix-org/michaelk/performance_stats
...
Add basic performance statistics to phone home
2018-04-04 10:37:25 +01:00
39ce38b024
Merge pull request #3053 from NotAFile/speedup-mxid-check
...
improve mxid check performance
2018-04-04 10:19:52 +01:00
a9a74101a4
Document the behaviour of ResponseCache
...
it looks like everything that uses ResponseCache expects to have to
`make_deferred_yieldable` its results. It's debatable whether that is the best
approach, but let's document it for now to avoid further confusion.
2018-04-04 09:06:22 +01:00
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
3ee4ad09eb
Fix json encoding bug in replication
...
json encoders have an encode method, not a dumps method.
2018-04-03 15:09:48 +01:00
11597ddea5
improve mxid check performance ~4x
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-31 02:00:22 +02:00
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
88cc9cc69e
Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation
...
Measure time it takes to calculate state group ID
2018-03-28 17:40:14 +01:00
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
c5de6987c2
This should probably be a PUT
2018-03-28 16:44:11 +01:00
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
9a0db062af
Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors
...
Fix error when claiming e2e keys from offline servers
2018-03-28 14:50:38 +01:00
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
32260baa41
pep8
2018-03-28 14:29:42 +01:00
33f6195d9a
Handle review comments
2018-03-28 14:25:25 +01:00
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
6152e253d8
Merge branch 'develop' of into allow_auto_join_rooms
2018-03-28 14:45:28 +02:00
792d340572
rename stat to future proof
2018-03-28 12:25:02 +01:00
4ceaa7433a
As daemonizing will make a new process, defer call to init.
2018-03-28 12:19:01 +01:00
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
0f890f477e
No need to cast in count_daily_users
2018-03-28 11:49:57 +01:00
545001b9e4
Fix search_user_dir multiple sqlite versions do different things
2018-03-28 11:19:45 +01:00
01ccc9e6f2
Measure time it takes to calculate state group ID
2018-03-28 11:03:52 +01:00
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
9187e0762f
count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home
2018-03-28 10:02:32 +01:00
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
...
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
...
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
004cc8a328
Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics
...
Add counter metrics for calculating state delta
2018-03-28 09:59:42 +01:00
ef520d8d0e
Include coarse CPU and Memory use in stats callbacks.
...
This requires the psutil module, and is still opt-in based on the report_stats
config option.
2018-03-27 17:56:03 +01:00
a134c572a6
Stringify exceptions for keys/{query,claim}
...
Make sure we stringify any exceptions we return from keys/query and keys/claim,
to avoid a 'not JSON serializable' error later
Fixes #3010
2018-03-27 17:15:06 +01:00
c2a5cf2fe3
factor out exception handling for keys/claim and keys/query
...
this stuff is badly c&p'ed
2018-03-27 17:11:23 +01:00
800cfd5774
Comment
2018-03-27 13:30:39 +01:00
152c2ac19e
Fix indent
2018-03-27 13:13:46 +01:00
e70287cff3
Comment
2018-03-27 13:13:38 +01:00
03a26e28d9
Merge pull request #3017 from matrix-org/erikj/add_cache_control_headers
...
Add Cache-Control headers to all JSON APIs
2018-03-27 13:10:38 +01:00
3e0c0660b3
Also do check inside linearizer
2018-03-27 13:01:34 +01:00
3f49e131d9
Add counter metrics for calculating state delta
...
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
9b8c0fb162
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-26 21:41:55 +01:00
a9d7d98d3f
Bum version and changelog
2018-03-26 16:36:53 +01:00
bdbb1eec65
Merge branch 'erikj/simplejson_replication' of github.com:matrix-org/synapse into release-v0.27.0
2018-03-26 16:35:55 +01:00
9187862002
Bump version and changelog
2018-03-26 16:20:24 +01:00
51406dab96
version bump
2018-03-26 14:48:19 +01:00
fecb45e0c3
Remove last usage of ujson
2018-03-26 13:32:29 +01:00
44cd6e1358
PEP8
2018-03-26 12:06:48 +01:00
8d6dc106d1
Don't use _cursor_to_dict in find_next_generated_user_id_localpart
2018-03-26 12:02:44 +01:00
a052aa42e7
Linearize calls to _generate_user_id
2018-03-26 12:02:20 +01:00
8cbbfaefc1
404 correctly on missing paths via NoResource
...
fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
2018-03-23 10:32:50 +00:00
84b5cc69f5
Merge pull request #3006 from matrix-org/erikj/state_iter
...
Use .iter* to avoid copies in StateHandler
2018-03-22 11:51:13 +00:00
fde8e8f09f
Fix s/iteriterms/itervalues
2018-03-22 11:42:16 +00:00
eb9fc021e3
Merge branch 'release-v0.27.0' of github.com:matrix-org/synapse into develop
2018-03-22 10:19:53 +00:00
1c41b05c8c
Add Cache-Control headers to all JSON APIs
...
It is especially important that sync requests don't get cached, as if a
sync returns the same token given then the client will call sync with
the same parameters again. If the previous response was cached it will
get reused, resulting in the client tight looping making the same
request and never making any progress.
In general, clients will expect to get up to date data when requesting
APIs, and so its safer to do a blanket no cache policy than only
whitelisting APIs that we know will break things if they get cached.
2018-03-21 17:46:26 +00:00
e66fbcbb02
fix merge conflicts
2018-03-20 14:25:31 +00:00
9aa5a0af51
Explicitly use simplejson
2018-03-20 09:58:13 +00:00
610accbb7f
Fix replication after switch to simplejson
...
Turns out that simplejson serialises namedtuple's as dictionaries rather
than tuples by default.
2018-03-19 16:12:48 +00:00
c384705ee8
Update __init__.py
...
bump version
2018-03-19 15:11:58 +00:00
3f961e638a
Merge pull request #3005 from matrix-org/erikj/fix_cache_size
...
Fix bug where state cache used lots of memory
2018-03-19 11:44:26 +00:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
9a0d783c11
Add comments
2018-03-19 11:35:53 +00:00
38f952b9bc
spell out not to massively increase bcrypt rounds
2018-03-19 09:27:36 +00:00
a8ce159be4
Replace some ujson with simplejson to make it work
2018-03-16 00:27:09 +00:00
5b631ff41a
Remove wrong comment
2018-03-16 00:07:08 +00:00
ba48755d56
Bump version and changelog
2018-03-15 23:57:26 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
5a6e54264d
Make 'unexpected logging context' into warnings
...
I think we've now fixed enough of these that the rest can be logged at
warning.
2018-03-15 18:40:38 +00:00
9cf519769b
Use .iter* to avoid copies in StateHandler
2018-03-15 17:50:26 +00:00
7c7706f42b
Fix bug where state cache used lots of memory
...
The state cache bases its size on the sum of the size of entries. The
size of the entry is calculated once on insertion, so it is important
that the size of entries does not change.
The DictionaryCache modified the entries size, which caused the state
cache to incorrectly think it was smaller than it actually was.
2018-03-15 15:46:54 +00:00
2cc9f76bc3
replace old style error catching with 'as' keyword
...
This is both easier to read and compatible with python3 (not that that
matters)
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-15 16:11:17 +01:00
ddb00efc1d
Bump version number
2018-03-15 14:41:30 +00:00
91ea0202e6
move handling of auto_join_rooms to RegisterHandler
...
Currently the handling of auto_join_rooms only works when a user
registers itself via public register api. Registrations via
registration_shared_secret and ModuleApi do not work
This auto_joins the users in the registration handler which enables
the auto join feature for all 3 registration paths.
This is related to issue #2725
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-14 16:45:37 +01:00
4f28018c83
Register membership/state servlets in event_creator
2018-03-14 14:30:06 +00:00
57db62e554
Merge pull request #2992 from matrix-org/erikj/implement_member_workre
...
Implement RoomMemberWorkerHandler
2018-03-14 14:29:33 +00:00
0011ede3b0
Fix imports
2018-03-14 14:19:23 +00:00
62ad701326
s/join/joined/ in notify_user_membership_change
2018-03-14 14:17:43 +00:00
3f0f06cb31
Split RoomMemberWorkerHandler to separate file
2018-03-14 11:41:45 +00:00
3e839e0548
Merge pull request #2989 from matrix-org/erikj/profile_cache_master
...
Only update remote profile cache on master
2018-03-14 09:42:27 +00:00
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
...
Split up ProfileStore
2018-03-14 09:41:06 +00:00
cfe75a9fb6
Merge pull request #2991 from matrix-org/erikj/fixup_rm
...
_remote_join and co take a requester
2018-03-14 09:40:57 +00:00
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
...
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) ( #2990 )
...
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
a08726fc42
Add is_blocked to worker store
2018-03-13 18:28:44 +00:00
b27320b550
Implement RoomMemberWorkerHandler
2018-03-13 18:26:00 +00:00
350331d466
_remote_join and co take a requester
2018-03-13 17:50:39 +00:00
1a69c6d590
Merge pull request #2987 from matrix-org/erikj/split_room_member_handler
...
Split RoomMemberHandler into base and master class
2018-03-13 17:40:00 +00:00
df8ff682a7
Only update remote profile cache on master
2018-03-13 17:38:21 +00:00
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
d45a114824
Raise, don't return, exception
2018-03-13 17:24:34 +00:00
6dbebef141
Add missing param to docstrings
2018-03-13 17:15:32 +00:00
16adb11cc0
Correct import order
2018-03-13 16:57:07 +00:00
82f16faa78
Move user_*_room distributor stuff to master class
...
I added yields when calling user_left_room, but they shouldn't matter on
the master process as they always return None anyway.
2018-03-13 16:38:15 +00:00
b78717b87b
Split RoomMemberHandler into base and master class
...
The intention here is to split the class into the bits that can be done
on workers and the bits that have to be done on the master.
In future there will also be a class that can be run on the worker,
which will delegate work to the master when necessary.
2018-03-13 16:37:41 +00:00
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer
...
Remove ReplicationLayer and user Client/Server directly
2018-03-13 15:45:08 +00:00
5d8476d8ff
Merge pull request #2981 from matrix-org/erikj/factor_remote_leave
...
Factor out _remote_reject_invite in RoomMember
2018-03-13 15:44:56 +00:00
47ce527f45
Add room_id to the response of `rooms/{roomId}/join`
...
Fixes #2349
2018-03-13 14:48:12 +01:00
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers
...
Don't build handlers on workers unnecessarily
2018-03-13 13:46:38 +00:00
cb9f8e527c
s/replication_client/federation_client/
2018-03-13 13:26:52 +00:00
cea462e285
s/replication_server/federation_server
2018-03-13 13:22:21 +00:00
bf8e97bd3c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave
2018-03-13 13:17:08 +00:00
ea3442c15c
Add docstring
2018-03-13 13:16:21 +00:00
16469a4f15
Merge pull request #2980 from matrix-org/erikj/rm_priv
...
Make RoomMemberHandler functions private that can be
2018-03-13 13:11:11 +00:00
c82111a55f
Merge pull request #2982 from matrix-org/erikj/fix_extra_users
...
extra_users is actually a list of UserIDs
2018-03-13 13:11:04 +00:00
da87791975
Merge pull request #2983 from matrix-org/erikj/rename_register_3pid
...
Refactor get_or_register_3pid_guest
2018-03-13 13:10:52 +00:00
99e9b4f26c
Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix
...
RoomMembershipRestServlet doesn't handle /forget
2018-03-13 13:10:44 +00:00
f5160d4a3e
RoomMembershipRestServlet doesn't handle /forget
...
Due to the order we register the REST handlers `/forget` was handled by
the correct handler.
2018-03-13 12:12:55 +00:00
8b3573a8b2
Refactor get_or_register_3pid_guest
2018-03-13 12:08:58 +00:00
299fd740c7
Merge pull request #2975 from matrix-org/rav/measure_persist_events
...
Add Measure block for persist_events
2018-03-13 12:06:25 +00:00
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props
...
Move property setting from ReplicationLayer to base classes
2018-03-13 11:45:25 +00:00
f43b6d6d9b
Fix docstring types
2018-03-13 11:29:35 +00:00
0f942f68c1
Factor out _remote_reject_invite in RoomMember
2018-03-13 11:22:45 +00:00
d0fcc48f9d
extra_users is actually a list of UserIDs
2018-03-13 11:20:06 +00:00
31becf4ac3
Make functions private that can be
2018-03-13 11:15:16 +00:00
d023ecb810
Don't build handlers on workers unnecessarily
2018-03-13 11:08:10 +00:00
ea7b3c4b1b
Remove unused ReplicationLayer
2018-03-13 11:00:04 +00:00
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
e05bf34117
Move property setting from ReplicationLayer to FederationBase
2018-03-13 10:51:30 +00:00
Richard van der Hoff