eaaa2248ff
Refactor request logging code
...
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.
Advantages of this are:
* we get logs for *all* requests (including OPTIONS and HEADs), rather than
just those that end up hitting handlers we've remembered to decorate
correctly.
* when a request handler wires up a Producer (as the media stuff does
currently, and as other things will do soon), we log at the point that all
of the traffic has been sent to the client.
2018-08-15 13:47:52 +01:00
87a824bad1
clean up AuthError
2018-08-15 11:58:03 +01:00
c4eb97518f
Merge branch 'neilj/update_limits_error_codes' of github.com:matrix-org/synapse into neilj/admin_email
2018-08-15 11:53:01 +01:00
55afba0fc5
update admin email to uri
2018-08-15 11:41:18 +01:00
1c5e690a6b
Merge pull request #3690 from matrix-org/neilj/change_prometheus_mau_metric_name
...
combine mau metrics into one group
2018-08-15 09:53:59 +00:00
fef2e65d12
Merge pull request #3667 from matrix-org/erikj/fixup_unbind
...
Don't fail requests to unbind 3pids for non supporting ID servers
2018-08-15 10:32:12 +01:00
b8429c7c81
update error codes for resource limiting
2018-08-15 10:19:48 +01:00
ab035bdeac
replace admin_email with admin_uri for greater flexibility
2018-08-15 10:16:41 +01:00
19b433e3f4
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email
2018-08-14 17:44:46 +01:00
b586b8b986
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 17:43:22 +01:00
70e48cbbb1
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/change_prometheus_mau_metric_name
2018-08-14 17:40:59 +01:00
9ecbaf8ba8
adding missing yield
2018-08-14 16:55:28 +01:00
e5962f845c
pep8
2018-08-14 16:36:14 +01:00
e7d091fb86
combine mau metrics into one group
2018-08-14 16:26:55 +01:00
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
...
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
2545993ce4
make comments clearer
2018-08-14 15:48:12 +01:00
8f9a7eb58d
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-14 15:11:54 +01:00
c74c71128d
remove blank line
2018-08-14 15:06:24 +01:00
ed4bc3d2fc
fix off by 1s on mau
2018-08-14 15:04:48 +01:00
bd92c8eaa7
Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:56:45 +01:00
99ebaed8e6
Update register.py
...
remove comments
2018-08-14 14:55:55 +01:00
9b5bf3d858
Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:51:38 +01:00
e25d87d97b
Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:32:18 +01:00
e2c9fe0a6a
backout ability to pass in event type to server notices
2018-08-14 13:32:56 +01:00
591bf87c6a
Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres
2018-08-14 20:56:23 +10:00
9b75c78b4d
support server notice state events for resource limits
2018-08-14 11:20:41 +01:00
63417c31e9
fix typo
2018-08-13 22:36:52 +01:00
f4b49152e2
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-13 21:09:47 +01:00
50bcaf1a27
Merge pull request #3677 from andrewshadura/master
...
Use recaptcha_ajax.js directly from Google
2018-08-13 18:52:56 +01:00
34f51babc5
Revert "Use recaptcha_ajax.js directly from Google"
2018-08-13 18:48:03 +01:00
ce7de9ae6b
Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases"
...
This reverts commit 0d43f991a1
2018-08-13 18:06:18 +01:00
0d43f991a1
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-13 18:00:23 +01:00
99dd975dae
Run tests under PostgreSQL ( #3423 )
2018-08-13 16:47:46 +10:00
31fa743567
fix sqlite/postgres incompatibility in reap_monthly_active_users
2018-08-11 22:38:34 +01:00
6c6aba76e1
implementation of server notices to alert on hitting resource limits
2018-08-10 15:12:59 +01:00
c31793a784
Merge branch 'rav/fix_linearizer_cancellation' into develop
2018-08-10 14:57:27 +01:00
c08f9d95b2
log *after* reloading log config
...
... because logging *before* reloading means the log message gets lost in the old MemoryLogger
2018-08-10 14:56:48 +01:00
b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
c75b71a397
Use recaptcha_ajax.js directly from Google
...
The script recaptcha_ajax.js contains minified non-free code which
we probably cannot redistribute. Since it talks to Google servers
anyway, it is better to just download it from Google directly instead
of shipping it.
This fixes #1932 .
2018-08-10 13:36:14 +02:00
3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
...
send SNI for federation requests
2018-08-10 12:23:54 +01:00
638d35ef08
Fix linearizer cancellation on twisted < 18.7
...
Turns out that cancellation of inlineDeferreds didn't really work properly
until Twisted 18.7. This commit refactors Linearizer.queue to avoid
inlineCallbacks.
2018-08-10 10:59:09 +01:00
01021c812f
wip at implementing MSC 7075
2018-08-09 22:16:00 +01:00
2e9c73e8ca
more generic conversion of str/bytes to unicode
2018-08-09 21:31:26 +02:00
64899341dc
include private functions from twisted
2018-08-09 21:04:22 +02:00
885ea9c602
rename _user_last_seen_monthly_active
2018-08-09 18:02:12 +01:00
c1f9dec92a
fix errant parenthesis
2018-08-09 17:43:26 +01:00
09cf130898
only block on sync where user is not part of the mau cohort
2018-08-09 17:39:12 +01:00
bb99b1f550
Add fast path in state res for zero prev events
2018-08-09 14:58:49 +01:00
ce6db0e547
Choose state algorithm based on room version
2018-08-09 14:58:47 +01:00
152c0aa58e
Add constants for room versions
2018-08-09 14:55:47 +01:00
119451dcd1
Refactor state module
...
We split out the actual state resolution algorithm to prepare for having
multiple versions.
2018-08-09 14:55:47 +01:00
69ce057ea6
block sync if auth checks fail
2018-08-09 12:26:27 +01:00
3dce9050cf
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block
2018-08-09 11:42:02 +01:00
0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs
...
Flag to disable HS without disabling federation
2018-08-09 10:41:43 +00:00
a5ef110749
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block
2018-08-09 11:40:37 +01:00
a6c813761a
Docstrings
2018-08-09 10:41:08 +01:00
b179537f2a
Move clean_room_for_join to master
2018-08-09 10:37:38 +01:00
72d1902bbe
Fixup doc comments
2018-08-09 10:23:49 +01:00
984376745b
Merge branch 'master' into develop
2018-08-09 19:23:47 +10:00
dafe90a6be
0.33.2
2018-08-09 19:20:41 +10:00
484a0ebdfc
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-09 10:16:29 +01:00
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
bf7598f582
Log when we 3pid/unbind request fails
2018-08-09 10:09:56 +01:00
2bdafaf3c1
Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet
...
Add helper base class for generating new replication endpoints
2018-08-09 10:06:23 +01:00
62564797f5
Fixup wording and remove dead code
2018-08-09 09:56:10 +01:00
bb89c84614
Merge pull request #3664 from matrix-org/rav/federation_metrics
...
more metrics for the federation and appservice senders
2018-08-08 23:16:58 +01:00
d5c0ce4cad
updated docstring for ServerContextFactory
2018-08-08 19:25:01 +02:00
e92fb00f32
sync auth blocking
2018-08-08 17:54:49 +01:00
839a317c96
fix pep8 too many lines
2018-08-08 17:39:04 +01:00
5298d79fb5
Merge branch 'develop' into neilj/disable_hs
2018-08-08 16:13:03 +00:00
8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
...
Support for room versioning
2018-08-08 17:10:53 +01:00
d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs
2018-08-08 15:55:47 +00:00
2b5baebeba
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-08 13:47:07 +01:00
be59910b93
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users
2018-08-08 13:45:21 +01:00
990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
...
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
312ae74746
typos
2018-08-08 13:33:16 +01:00
360ba89c50
Don't fail requests to unbind 3pids for non supporting ID servers
...
Older identity servers may not support the unbind 3pid request, so we
shouldn't fail the requests if we received one of 400/404/501. The
request still fails if we receive e.g. 500 responses, allowing clients
to retry requests on transient identity server errors that otherwise do
support the API.
Fixes #3661
2018-08-08 12:06:18 +01:00
bebe325e6c
Rename POST param to METHOD
2018-08-08 10:36:18 +01:00
5011417632
Fixup logging and docstrings
2018-08-08 10:29:58 +01:00
bab94da79c
fix metric name
2018-08-07 22:11:45 +01:00
53bca4690b
more metrics for the federation and appservice senders
2018-08-07 19:09:48 +01:00
ef3589063a
prevent total number of reserved users being too large
2018-08-07 17:59:27 +01:00
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
3523f5432a
Don't expose default_room_version as config opt
2018-08-07 12:51:57 +01:00
848431be1d
fix for rc1
2018-08-07 21:13:22 +10:00
3b9662339b
version
2018-08-07 21:10:52 +10:00
cd9765805e
Allow ratelimiting on workers
2018-08-07 10:50:28 +01:00
495cb100d1
Allow profile changes to happen on workers
2018-08-07 10:50:26 +01:00
ca9bc1f4fe
Fix occasional glitches in the synapse_event_persisted_position metric
...
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
a74b25faaa
WIP building out mau reserved users
2018-08-06 23:25:25 +01:00
7daa8a78c5
load mau limit threepids
2018-08-06 22:55:05 +01:00
89834d9a29
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:56:56 +01:00
e54794f5b6
Fix postgres compatibility bug
2018-08-06 21:55:54 +01:00
7bcf126b18
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:39:44 +01:00
1911c037cb
update comments to reflect new sig
2018-08-06 18:01:46 +01:00
33bd07d062
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 17:52:28 +01:00
16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
19a17068f1
Check m.room.create for sane room_versions
2018-08-06 16:11:24 +01:00
96a9a29645
Pull in necessary stores in federation_reader
2018-08-06 15:23:57 +01:00
62ace05c45
Move necessary storage functions to worker classes
2018-08-06 15:23:38 +01:00
1e2bed9656
Import all functions from TransactionStore
2018-08-06 15:23:38 +01:00
a3f5bf79a0
Add EDU/query handling over replication
2018-08-06 15:23:31 +01:00
e26dbd82ef
Add replication APIs for persisting federation events
2018-08-06 15:02:28 +01:00
051a99c400
Fix isort
2018-08-06 14:29:31 +01:00
f900d50824
include known room versions in outgoing make_joins
2018-08-06 13:45:37 +01:00
42c6823827
disable HS from config
2018-08-04 22:07:04 +01:00
e40a510fbf
py3 fix
2018-08-03 23:19:13 +01:00
886be75ad1
bug fixes
2018-08-03 22:29:03 +01:00
16d9701892
Return M_NOT_FOUND when a profile could not be found. ( #3596 )
2018-08-03 19:08:05 +01:00
e10830e976
wip commit - tests failing
2018-08-03 17:55:50 +01:00
3777fa26aa
sanity check response from make_join
2018-08-03 16:08:32 +01:00
0d63d93ca8
Enforce compatibility when processing make_join requests
...
Reject make_join requests from servers which do not support the room version.
Also include the room version in the response.
2018-08-03 16:08:32 +01:00
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
15c1ae45e5
Docstrings for BaseFederationServlet
...
... to save me reverse-engineering this stuff again.
2018-08-03 16:08:32 +01:00
5593ff6773
fix (lots of) py3 test failures
2018-08-03 14:59:17 +01:00
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
897c51d274
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-03 13:40:47 +01:00
cb298ff623
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet
2018-08-03 09:25:15 +01:00
c0affa7b4f
update generate_monthly_active_users, and reap_monthly_active_users
2018-08-02 23:03:01 +01:00
5e2f7b8084
typo
2018-08-02 22:47:48 +01:00
4ecb4bdac9
wip attempt at caching
2018-08-02 22:41:05 +01:00
1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling
...
Clean up handling of errors from outbound requests
2018-08-02 17:38:24 +01:00
bdae8f2e68
Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling
...
Factor out exception handling in federation_client
2018-08-02 17:37:46 +01:00
74b1d46ad9
do mau checks based on monthly_active_users table
2018-08-02 16:57:35 +01:00
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
43ecfe0b10
Synapse 0.33.1 (2018-08-02)
...
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx
Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7
SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g
UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd
BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP
RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc=
=U/ir
-----END PGP SIGNATURE-----
Merge tag 'v0.33.1'
Synapse 0.33.1 (2018-08-02)
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
2018-08-02 15:40:44 +01:00
14a4e7d5a4
Prepare 0.33.1
2018-08-02 15:31:04 +01:00
0bf5ec0db7
Check room visibility for /event/ requests
...
Make sure that the user has permission to view the requeseted event for
/event/{eventId} and /room/{roomId}/event/{eventId} requests.
Also check that the event is in the given room for
/room/{roomId}/event/{eventId}, for sanity.
2018-08-02 15:03:27 +01:00
14fa9d4d92
Avoid extra db lookups
...
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
4a6725d9d1
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-02 11:04:18 +01:00
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
...
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
08281fe6b7
self.db_conn unused
2018-08-01 23:26:24 +01:00
c21d82bab3
normalise reaping query
2018-08-01 23:24:38 +01:00
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
...
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
b8d7d3996b
Merge pull request #3620 from fuzzmz/return-404-room-not-found
...
return 404 if room not found
2018-08-01 16:34:32 +01:00
b7f203a566
count_monthly_users is now async
2018-08-01 16:17:42 +01:00
7ff44d9215
improve clarity
2018-08-01 16:17:00 +01:00
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
018d75a148
Refactor code for turning HttpResponseException into SynapseError
...
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.
The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
2018-08-01 16:02:46 +01:00
fa7dc889f1
Be more careful which errors we send back over the C-S API
...
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.
That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.
The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
c82ccd3027
Factor out exception handling in federation_client
...
Factor out the error handling from make_membership_event, send_join, and
send_leave, so that it can be shared.
2018-08-01 16:01:04 +01:00
da7785147d
Python 3: Convert some unicode/bytes uses ( #3569 )
2018-08-02 00:54:06 +10:00
303f1c851f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits
2018-08-01 13:42:50 +01:00
a6d7b74915
update docs
2018-08-01 13:39:14 +01:00
Richard van der Hoff