Commit Graph

9011 Commits (e3758c8c929be11fe38cebb2b4f7d43185f80197)

Author SHA1 Message Date
Richard van der Hoff 286d6930b7
Merge pull request #3879 from matrix-org/matthew/fix-autojoin
don't ratelimit autojoins
2018-09-18 13:07:01 +01:00
Richard van der Hoff 1e09a1d48a
Merge pull request #3889 from matrix-org/rav/404_on_remove_unknown_alias
Return a 404 when deleting unknown room alias
2018-09-18 12:59:30 +01:00
Will Hunt 5baa087312
typo 2018-09-17 17:37:56 +01:00
Will Hunt b58714789f
make pip happy? 2018-09-17 17:35:54 +01:00
Richard van der Hoff ac80cb08fe Fix more b'abcd' noise in metrics 2018-09-17 17:16:50 +01:00
Will Hunt 9a1cceeca9
Use a string for versions 2018-09-17 17:09:06 +01:00
Richard van der Hoff f75b9961c6 Reinstate missing null check 2018-09-17 16:52:02 +01:00
Will Hunt 2b39494cd5
Add python_version phone home stat 2018-09-17 16:35:18 +01:00
Richard van der Hoff f00a9d2636 Fix some b'abcd' noise in logs and metrics
Python 3 compatibility: make sure that we decode some byte sequences before we
use them to create log lines and metrics labels.
2018-09-17 16:15:42 +01:00
Amber Brown fe88907d04 version 2018-09-17 22:33:22 +10:00
Richard van der Hoff 85a43f4167 Return a 404 when deleting unknown room alias
As per https://github.com/matrix-org/matrix-doc/issues/1675

Fixes https://github.com/matrix-org/synapse/issues/2782
2018-09-17 13:19:00 +01:00
Matthew Hodgson d42d79e3c3 don't ratelimit autojoins 2018-09-15 22:27:41 +01:00
Erik Johnston 24efb2a70d Fix timeout function
Turns out deferred.cancel sometimes throws, so we do that last to ensure
that we always do resolve the new deferred.
2018-09-15 11:38:39 +01:00
Erik Johnston fcfe7a850d Add an awful secondary timeout to fix wedged requests
This is an attempt to mitigate #3842 by adding yet-another-timeout
2018-09-14 19:23:07 +01:00
Matthew Hodgson 024be6cf18
don't filter membership events based on history visibility (#3874)
don't filter membership events based on history visibility
as we will already have filtered the messages in the timeline, and state events
are always visible.

and because @erikjohnston said so.
2018-09-14 18:12:52 +01:00
Erik Johnston 3e6e94fe9f
Merge pull request #3872 from matrix-org/hawkowl/timeouts-2
timeouts 2: electric boogaloo
2018-09-14 16:58:44 +01:00
Amber Brown bc9af88a2d fix 2018-09-15 00:26:00 +10:00
Erik Johnston d0f6c1ce21 Remove spurious comment 2018-09-14 15:12:36 +01:00
Erik Johnston 9e2f9a7b57 Measure outbound requests 2018-09-14 15:11:26 +01:00
Erik Johnston 0a81038ea0 Add in flight real time metrics for Measure blocks 2018-09-14 15:08:37 +01:00
Neil Johnson 7de1989ea2 fix link for case that config.email_riot_base_url is set 2018-09-13 22:43:50 +01:00
Amber Brown c971aa7b9d fix 2018-09-14 03:57:02 +10:00
Amber Brown 8f08d848f5 fix 2018-09-14 03:53:56 +10:00
Travis Ralston f1a7264663
Fix minor typo in exception 2018-09-13 11:51:12 -06:00
Amber Brown 7c33ab76da redact better 2018-09-14 03:45:34 +10:00
Amber Brown 63755fa4c2 we do that higher up 2018-09-14 03:21:47 +10:00
Amber Brown 73884ebac5 Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2 2018-09-14 03:11:25 +10:00
Amber Brown 7c27c4d51c
merge (#3576) 2018-09-14 03:11:11 +10:00
Amber Brown 1c3f4d9ca5 buffer? 2018-09-14 03:09:13 +10:00
David Baker bc74925c5b WIP e2e key backups
Continues from uhoreg's branch

This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
2018-09-13 17:02:59 +01:00
Erik Johnston 6c0f8d9d50
Merge pull request #3856 from matrix-org/erikj/speed_up_purge
Make purge history slightly faster
2018-09-13 16:14:46 +01:00
Erik Johnston ed5331a627 comment 2018-09-13 16:10:56 +01:00
Erik Johnston 89a76d1889 Fix handling of redacted events from federation
If we receive an event that doesn't pass their content hash check (e.g.
due to already being redacted) then we hit a bug which causes an
exception to be raised, which then promplty stops the event (and
request) from being processed.

This effects all sorts of federation APIs, including joining rooms with
a redacted state event.
2018-09-13 15:44:12 +01:00
Amber Brown bfa0b759e0
Attempt to figure out what's going on with timeouts (#3857) 2018-09-14 00:15:51 +10:00
Erik Johnston 9cbd0094f0 pep8 2018-09-13 15:15:35 +01:00
Erik Johnston 9dbe38ea7d Create indices after insertion 2018-09-13 15:05:52 +01:00
Erik Johnston 93139a1fb8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_purge 2018-09-13 12:57:09 +01:00
Erik Johnston c857f5ef9b Make purge history slightly faster
Don't pull out events that are outliers and won't be deleted, as nothing
should happen to them.
2018-09-13 12:48:10 +01:00
Amber Brown b7d2fb5eb9
Remove some superfluous logging (#3855) 2018-09-13 19:59:32 +10:00
Neil Johnson f30a303590
Merge pull request #3846 from matrix-org/neilj/expose-registered-users
expose number of real reserved users
2018-09-12 17:14:04 +01:00
Matthew Hodgson 2ac1abbc7e
show heroes if a room has a 'deleted' name/canonical_alias (#3851) 2018-09-12 17:11:05 +01:00
Erik Johnston fa0d464fa4
Merge pull request #3853 from matrix-org/erikj/log_outbound_each_time
Log outbound requests when we retry
2018-09-12 16:55:40 +01:00
Matthew Hodgson 0403cf0783 argh pep8 2018-09-12 16:54:28 +01:00
Matthew Hodgson 0e200e366d correctly log gappy sync metrics 2018-09-12 16:47:20 +01:00
Matthew Hodgson 11bfc2af1c fix logline 2018-09-12 16:45:42 +01:00
Neil Johnson 8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Erik Johnston 8c5b84441b Log outbound requests when we retry 2018-09-12 16:22:14 +01:00
Erik Johnston 54f8616d2c
Merge pull request #3841 from matrix-org/erikj/manhole_key_length
Change the manhole SSH key to have more bits
2018-09-12 14:33:39 +01:00
Amber Brown 7ca097f77e
Port federation/ to py3 (#3847) 2018-09-12 23:23:32 +10:00
Neil Johnson 0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown 546aee7e52
Merge pull request #3835 from krombel/fix_3821
fix VOIP crashes under Python 3
2018-09-12 20:44:18 +10:00
Amber Brown 33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring
add some logging for the keyring queue
2018-09-12 20:43:47 +10:00
Amber Brown bc635026c5
Merge pull request #3824 from matrix-org/rav/fix_jwt_import
Fix jwt import check
2018-09-12 20:41:57 +10:00
Amber Brown 02aa41809b
Port rest/ to Python 3 (#3823) 2018-09-12 20:41:31 +10:00
Amber Brown 8fd93b5eea
Port crypto/ to Python 3 (#3822) 2018-09-12 20:16:31 +10:00
Erik Johnston 4084a774a8 Timeout reading body for outbound HTTP requests 2018-09-12 10:10:20 +01:00
Matthew Hodgson b041115415
Speed up lazy loading (#3827)
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
2018-09-12 00:50:39 +01:00
Erik Johnston 9e05c8d309 Change the manhole SSH key to have more bits
Newer versions of openssh client refuse to connect to the old key due to
its length.
2018-09-11 10:42:10 +01:00
Krombel 7bc22539ff fix VOIP crashes under Python 3 (#3821) 2018-09-10 14:30:08 +02:00
Amber Brown 9a5ea511b5
Merge pull request #3810 from matrix-org/erikj/send_tags_down_sync_on_join
Send existing room tags down sync on join
2018-09-07 23:28:42 +10:00
Richard van der Hoff b8ad756bd0 Fix jwt import check
This handy code attempted to check that we could import jwt, but utterly failed
to check it was the right jwt.

Fixes https://github.com/matrix-org/synapse/issues/3793
2018-09-07 14:20:54 +01:00
Amber Brown 771d213ac5 Merge branch 'master' into develop 2018-09-07 21:45:38 +10:00
Amber Brown 6febd8e8f7 version 2018-09-07 21:40:57 +10:00
Richard van der Hoff cd7ef43872 clearer logging when things fail, too 2018-09-06 23:56:47 +01:00
Richard van der Hoff 806964b5de add some logging for the keyring queue
why is it so damn slow?
2018-09-06 18:51:06 +01:00
Neil Johnson c5440b2ca0
Merge pull request #3800 from matrix-org/neilj/remove-guests-from-mau-count
guest users should not be part of mau total
2018-09-06 17:45:55 +01:00
Neil Johnson 84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Erik Johnston f60c9e2a01 Don't send empty tags list down sync 2018-09-06 17:01:41 +01:00
Erik Johnston 7baf66ef5d Send existing room tags down sync on join
When a user joined a room any existing tags were not sent down the sync
stream. Ordinarily this isn't a problem because the user needs to be in
the room to have set tags in it, however synapse will sometimes add tags
for a user to a room, e.g. for server notices, which need to come down
sync.
2018-09-06 16:46:51 +01:00
Hubert Chathi 3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Amber Brown 654324eded
Merge pull request #3805 from matrix-org/erikj/limit_transaction_pdus_edus
Limit the number of PDUs/EDUs per fedreation transaction
2018-09-07 01:33:31 +10:00
Amber Brown 70fd75cd1d
Merge pull request #3788 from matrix-org/erikj/remove_conn_id
Remove conn_id from repl prometheus metrics
2018-09-07 00:48:19 +10:00
Amber Brown ee7c8bd2b5
Merge pull request #3795 from matrix-org/erikj/faster_sync_state
User iter* during sync state calculations
2018-09-07 00:24:43 +10:00
Erik Johnston 6707a3212c Limit the number of PDUs/EDUs per fedreation transaction 2018-09-06 15:23:55 +01:00
Amber Brown 135f3b4390
Merge pull request #3804 from matrix-org/rav/fix_openssl_dep
bump dep on pyopenssl to 16.x
2018-09-07 00:23:39 +10:00
Amber Brown 2608ebc04c
Port handlers/ to Python 3 (#3803) 2018-09-07 00:22:23 +10:00
Erik Johnston 417e7077aa Bump version and changelog 2018-09-06 14:08:55 +01:00
Erik Johnston d64b24dfe6 Synapse 0.33.3.1 (2018-09-06)
=============================
 
 SECURITY FIXES
 --------------
 
 - Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796))
 - Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796))
 
 Internal Changes
 ----------------
 
 - Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802))
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2
 
 iQEcBAABCAAGBQJbkPLrAAoJEIofk9V1tejV7K4IAItYIX98DKN9x3FNs9Hd69Pw
 mZtqmuJ12YAyiVZoR7IJ5GfyctHCYUUXcqmNR1O+2/IRvezFnU6ZTcPW3OfNfnuD
 vlnMiK53F21T96ul3Wu47Z2wyO+WkoKeXdvlqt3Wa8HpnPbU6y5CSac2vK57ppTU
 DcMZKXaDae6vA7bsjryYgTRRplH6eFWZEexjbuZudbDvqkySi2zGUTs6SUesMC5B
 FRPgI6p6sCeQgGMJY+d1i+ZdzWkZhH5OLH2icf2MI1hnhSnRSJ/scSGm5OYDCwYy
 03hOy0ZVMr8SB5/j0TjSaFXonKsmPKqxnU2g7crJmvFP4BfJrEbym0y9YhYXnVQ=
 =8PDU
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3.1' into release-v0.33.4

Synapse 0.33.3.1 (2018-09-06)
=============================

SECURITY FIXES
--------------

- Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796))
- Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796))

Internal Changes
----------------

- Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802))
2018-09-06 14:08:33 +01:00
Richard van der Hoff 4f8baab0c4 Merge branch 'master' into develop 2018-09-06 13:05:22 +01:00
Richard van der Hoff 625542878d bump dep on pyopenssl to 16.x 2018-09-06 12:53:15 +01:00
Richard van der Hoff 80189ed27c prepare v0.33.3.1 2018-09-06 10:26:23 +01:00
Neil Johnson 61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Richard van der Hoff dfba1d843d
Merge pull request #3790 from matrix-org/rav/respect_event_format_in_filter
Implement 'event_format' filter param in /sync
2018-09-05 16:24:14 +01:00
Erik Johnston 7419764351 User iter* during sync state calculations 2018-09-05 16:19:50 +01:00
Amber Brown 2d2828dcbc
Port http/ to Python 3 (#3771) 2018-09-06 00:10:47 +10:00
Richard van der Hoff c127c8d042 Fix origin handling for pushed transactions
Use the actual origin for push transactions, rather than whatever the remote
server claimed.
2018-09-05 13:08:07 +01:00
Richard van der Hoff 804dd41e18 Check that signatures on events are valid
We should check that both the sender's server, and the server which created the
event_id (which may be different from whatever the remote server has told us
the origin is), have signed the event.
2018-09-05 13:08:07 +01:00
Erik Johnston 5f02017aea Improve performance of getting typing updates for replication
Fetching the list of all new typing notifications involved iterating
over all rooms and comparing their serial. Lets move to using a stream
change cache, like we do for other streams.
2018-09-05 10:20:40 +01:00
Richard van der Hoff 87c18d12ee Implement 'event_format' filter param in /sync
This has been specced and part-implemented; let's implement it for /sync (but
no other endpoints yet :/).
2018-09-04 15:20:09 +01:00
Neil Johnson a6cf7d9d9a
Merge pull request #3789 from matrix-org/neilj/improve_threepid_error_strings
improve human readable error messages
2018-09-04 13:16:00 +00:00
Amber Brown 7e9ced4178 version and towncrier 2018-09-04 21:12:04 +10:00
Neil Johnson bae37cd811 improve human readable error message 2018-09-04 12:07:00 +01:00
Neil Johnson c42f7fd7b9 improve human readable error messages 2018-09-04 12:03:17 +01:00
Erik Johnston 3e242dc149 Remove conn_id 2018-09-04 11:45:52 +01:00
Erik Johnston b13836da7f Remove conn_id from repl prometheus metrics
`conn_id` gets set to a random string, and so we end up filling up
prometheus with tonnes of data series, which is bad.
2018-09-03 17:22:49 +01:00
Erik Johnston 567363e497
Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func
Remove unnecessary resolve_events_with_state_map
2018-09-03 16:19:41 +01:00
Neil Johnson 301cb60d0b assert rather than warn 2018-08-31 17:29:35 +01:00
Neil Johnson 0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson e8e540630e fix reference to is_threepid_reserved 2018-08-31 16:09:15 +01:00
Neil Johnson 09f3cf1a7e ensure post registration auth checks do not fail erroneously 2018-08-31 15:42:51 +01:00
Neil Johnson ea068d6f3c fix bug where preserved threepid user comes to sign up and server is mau blocked 2018-08-31 10:49:14 +01:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff 475253a88e
Merge pull request #3764 from matrix-org/rav/close_db_conn_after_init
Make sure that we close db connections opened during init
2018-08-30 10:47:27 +01:00
Krombel 79a8a347a6 fix #3445
itervalues(d) calls d.itervalues() [PY2] and d.values() [PY3]
but SortedDict only implements d.values()
2018-08-29 16:28:25 +02:00
Matthew Hodgson b1580f50fe
don't return non-LL-member state in incremental sync state blocks (#3760)
don't return non-LL-member state in incremental sync state blocks
2018-08-28 23:25:58 +01:00
Richard van der Hoff 32eb1dedd2 use abc.abstractproperty
This gives clearer messages when someone gets it wrong
2018-08-28 17:10:43 +01:00
Richard van der Hoff 0b07f02e19 Make sure that we close db connections opened during init
We should explicitly close any db connections we open, because failing to do so
can block other transactions as per
https://github.com/matrix-org/synapse/issues/3682.

Let's also try to factor out some of the boilerplate by having server classes
define their datastore class rather than duplicating the whole of `setup`.
2018-08-28 13:39:49 +01:00
Hubert Chathi 16a31c6fce update to newer Synapse APIs 2018-08-24 22:51:25 -04:00
Erik Johnston 9db2476991 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact 2018-08-24 17:00:37 +01:00
Erik Johnston 05077e06fa Change admin_uri to admin_contact in config and errors 2018-08-24 16:51:27 +01:00
Hubert Chathi 83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston 01a5a8b9e3 Fix checking if service notice room is already tagged
This manifested in synapse repeatedly setting the tag for the room
2018-08-24 16:22:37 +01:00
Erik Johnston cdd24449ee Ensure we wake up /sync when we add tag to notice room 2018-08-24 14:50:03 +01:00
Erik Johnston 14d49c51db Make content of tag an empty object rather than null 2018-08-24 14:44:16 +01:00
Erik Johnston 84b4e76fed
Merge pull request #3754 from matrix-org/erikj/fix_whitelist
Allow federation_domain_whitelist to be emtpy list
2018-08-24 12:23:39 +01:00
Erik Johnston 1d67b13674 Fix bug when federation_domain_whitelist is an emtpy list
Outbound federation were incorrectly allowed when the config option was
set to an empty list
2018-08-24 12:13:12 +01:00
Erik Johnston 92d50e3c2a
Merge pull request #3753 from matrix-org/erikj/fix_no_server_noticse
Fix bug where we broke sync when using limit_usage_by_mau
2018-08-24 11:56:08 +01:00
Richard van der Hoff e94cdbaecf
Merge pull request #3751 from matrix-org/rav/twisted_17
Pin to twisted 17.1 or later
2018-08-24 11:55:52 +01:00
Erik Johnston 7f147d623b Fix bug where we broke sync when using limit_usage_by_mau
We assumed that we always had service notices configured, but that is
not always true
2018-08-24 11:33:50 +01:00
Richard van der Hoff f584d6108f Pin to twisted 17.1 or later
Fixes https://github.com/matrix-org/synapse/issues/3741.
2018-08-24 10:02:31 +01:00
Erik Johnston cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston 7e6e588e60 Fix bug where we resent "limit exceeded" server notices
This was due to a bug where we mutated a cached event's contents
2018-08-23 16:21:20 +01:00
Erik Johnston 7a0da69eee Add missing yield 2018-08-23 10:28:12 +01:00
Erik Johnston fd2dbf1836 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-22 17:06:10 +01:00
Erik Johnston 9643a6f7f2 Update notice format 2018-08-22 17:00:29 +01:00
Erik Johnston 5c261107c9 Remove unnecessary resolve_events_with_state_map
We only ever used the synchronous resolve_events_with_state_map in one
place, which is trivial to replace with the async version.
2018-08-22 15:41:15 +01:00
Richard van der Hoff c7181dcc6c Merge branch 'master' into develop 2018-08-22 14:37:11 +01:00
Richard van der Hoff 74854a9719 Use recaptcha_ajax.js directly from Google
This was originally done in commit c75b71a397,
but got reverted on this branch due to the PR (#3677) being based on the wrong
branch.

We're ready to merge this to master now, so let's make it match
release-v0.33.3.
2018-08-22 14:30:49 +01:00
Richard van der Hoff 48fec67536 Features
--------
 
 - Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
 - Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
 - speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
 - implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
 - Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
 - Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
 - Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
 - Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
 - Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
 - Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
 - Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
 - Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
 - set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
 - Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
 - For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))
 
 Bugfixes
 --------
 
 - Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
 - Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
 - Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
 - Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
 - Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
 - Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
 - Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
 - Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
 - Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
 - Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
 - Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
 - Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
 - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 - Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))
 
 Deprecations and Removals
 -------------------------
 
 - The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))
 
 Internal Changes
 ----------------
 
 - The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
 - Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
 - Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
 - Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
 - Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
 - Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
 - Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
 - Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
 - Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
 - Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
 - add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
 - Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
 - Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQEcBAABAgAGBQJbfWKDAAoJEIofk9V1tejVHOMH/RmP+axF2K+Socnnb5Fa6RTo
 pDJXVwsvNQxcwzZIJXh2TRilVr88jx7A1PHUde0TmkHRmis/bDzZxa59QRA4rFEy
 fLYMx9Tyt0RzAJrEd0p7jbIkJxxPm5PbMWfCgVfzg44q0kepRW6ZVl1+kFHDrsCo
 ocE03WdLe2sRHSudaVtBLOVMA1lDE22v40/7xiu9IbUOnF+WZ3cjXYX+SNr5eb6g
 7YGbXOk4/E8XoUnpzH5oEnNtiDK5piRrr7o1282kecY20ZMNzKgseBHU6nYMv5ZO
 TUpCPwbukmzeGHITupY1RkzpmBbEIJTCnmLP8fiUw+RWznYyFC/aRFMiWTzle3A=
 =ngmn
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3'

Features
--------

- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))

Bugfixes
--------

- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))

Deprecations and Removals
-------------------------

- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))

Internal Changes
----------------

- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))
2018-08-22 14:28:55 +01:00
Richard van der Hoff 4e5a4549b6 bump version to 0.33.3 2018-08-22 14:07:10 +01:00
Erik Johnston db10f553ba
Merge pull request #3724 from Half-Shot/hs/guest-fetch-event
Allow guests to use /rooms/:roomId/event/:eventId
2018-08-22 13:41:08 +01:00
Erik Johnston 764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles
Allow profile updates to happen on workers
2018-08-22 11:35:55 +01:00
Erik Johnston 8432e2ebd7 Rename WorkerProfileHandler to BaseProfileHandler 2018-08-22 10:13:40 +01:00
Erik Johnston a81f140880 Add assert to ensure handler is only run on master 2018-08-22 10:11:21 +01:00
Erik Johnston 47b25ba5f3 Remove redundant vars 2018-08-22 10:09:05 +01:00
Erik Johnston 3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Richard van der Hoff a4cf660a32
Merge pull request #3735 from matrix-org/travis/federation-spelling
limt -> limit
2018-08-22 09:34:21 +01:00
Richard van der Hoff 0d568ff403 Merge remote-tracking branch 'origin/release-v0.33.3' into develop 2018-08-22 09:15:44 +01:00
Matthew Hodgson bb81e78ec6
Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Richard van der Hoff f7bf181a90 fix another consent encoding fail 2018-08-21 23:14:25 +01:00
Richard van der Hoff f7baff6f7b Fix 500 error from /consent form
Fixes #3731
2018-08-21 22:47:07 +01:00
Richard van der Hoff a52f276990 Bugfixes
--------
 
 - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
 ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS
 ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6
 Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9
 698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ
 YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV
 ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU
 Z2lL1C5ggBCkD43g3orDvehbNWCL5g==
 =0vEe
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3rc2' into develop

Bugfixes
--------

- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
([\#3723](https://github.com/matrix-org/synapse/issues/3723))
2018-08-21 20:30:09 +01:00
Hubert Chathi 42a394caa2 allow session_data to be any JSON instead of just a string 2018-08-21 14:51:34 -04:00
Erik Johnston 46c832eaac
Merge pull request #3727 from matrix-org/erikj/dont_error_on_missing_keys
Don't log exceptions when failing to fetch server keys
2018-08-21 17:07:20 +01:00
Erik Johnston cd6937fb26 Fix typo 2018-08-21 16:28:10 +01:00
Hubert Chathi 8550a7e9c2 allow auth_data to be any JSON instead of a string 2018-08-21 10:38:00 -04:00
Erik Johnston c2c153dd3b Log more detail when we fail to authenticate request 2018-08-21 11:42:49 +01:00
Erik Johnston 808d8e06aa Don't log exceptions when failing to fetch server keys
Not being able to resolve or connect to remote servers is an expected
error, so we shouldn't log at ERROR with stacktraces.
2018-08-21 11:19:26 +01:00
Erik Johnston 3f6762f0bb isort 2018-08-21 09:38:38 +01:00
Amber Brown 3b5b64ac99 changelog 2018-08-21 03:48:55 +10:00
Will Hunt f89f6b7c09 Allow guests to access /rooms/:roomId/event/:eventId 2018-08-20 18:25:54 +01:00
Richard van der Hoff be6527325a Fix exceptions when a connection is closed before we read the headers
This fixes bugs introduced in #3700, by making sure that we behave sanely
when an incoming connection is closed before the headers are read.
2018-08-20 18:21:10 +01:00
Richard van der Hoff 55e6bdf287 Robustness fix for logcontext filter
Make the logcontext filter not explode if it somehow ends up with a logcontext
of None, since that infinite-loops the whole logging system.
2018-08-20 18:20:07 +01:00
Erik Johnston b01a755498 Make the in flight requests metrics thread safe 2018-08-20 17:27:52 +01:00
Erik Johnston 1058d14127 Make the in flight background process metrics thread safe 2018-08-20 17:27:24 +01:00
Amber Brown 9a2f960736 version 2018-08-21 00:00:19 +10:00
Amber Brown 324525f40c
Port over enough to get some sytests running on Python 3 (#3668) 2018-08-20 23:54:49 +10:00
Erik Johnston 4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston 8dee601054 Remove redundant room_version checks 2018-08-20 14:48:53 +01:00
Erik Johnston e21c368b8b Revert spurious change 2018-08-20 13:54:51 +01:00
Erik Johnston f2a48d87df Use get_cache_factor_for function for `state_cache`
This allows the cache factor for `state_cache` to be individually
specified in the enviroment
2018-08-20 13:01:46 +01:00
Erik Johnston 2aa7cc6a46
Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging
Fix logging bug in EDU handling over replication
2018-08-20 10:51:45 +01:00
Neil Johnson e07970165f rename error code 2018-08-18 14:39:45 +01:00
Neil Johnson c5171bf171 special case server_notices_mxid 2018-08-18 12:33:07 +01:00
Neil Johnson ba1fbf7d5b special case server_notices_mxid 2018-08-18 12:31:08 +01:00
Richard van der Hoff 3cef867cc1
Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commands
Logcontexts for replication command handlers
2018-08-17 16:22:07 +01:00
Richard van der Hoff c144252a8c
Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates
Fix logcontexts for running pushers
2018-08-17 16:21:49 +01:00
Amber Brown c334ca67bb
Integrate presence from hotfixes (#3694) 2018-08-18 01:08:45 +10:00
Amber Brown 04f5d2db62
Remove v1/register's broken shared secret functionality (#3703) 2018-08-18 00:55:01 +10:00
Erik Johnston ab822a2d1f Add some fixmes 2018-08-17 15:31:50 +01:00
Erik Johnston 91cdb6de08 Call UserDirectoryHandler methods directly
Turns out that the user directory handling is fairly racey as a bunch
of stuff assumes that the processing happens on master, which it doesn't
when there is a synapse.app.user_dir worker. So lets just call the
function directly until we actually get round to fixing it, since it
doesn't make the situation any worse.
2018-08-17 15:26:13 +01:00
Neil Johnson d49b77404b clean up, no functional changes 2018-08-17 15:21:34 +01:00
Richard van der Hoff 63260397c6
Merge pull request #3701 from matrix-org/rav/use_producer_for_responses
Use a producer to stream back responses
2018-08-17 14:58:45 +01:00
Richard van der Hoff 3f8709ffe4
Merge pull request #3700 from matrix-org/rav/wait_for_producers
Refactor request logging code
2018-08-17 14:57:45 +01:00
Neil Johnson 3ee57bdcbb Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:34:10 +01:00
Erik Johnston 782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston ca87ad1def Split ProfileHandler into master and worker 2018-08-17 14:15:14 +01:00
Neil Johnson b5f638f1f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:04:15 +01:00
Neil Johnson 9fd161c6fb Merge branch 'neilj/limit_exceeded_error' of github.com:matrix-org/synapse into neilj/limit_exceeded_error 2018-08-17 13:58:40 +01:00
Neil Johnson 0195dfbf52 server limits config docs 2018-08-17 13:58:25 +01:00
Neil Johnson 69c49d3fa3
Merge branch 'develop' into neilj/limit_exceeded_error 2018-08-17 12:44:26 +00:00
Erik Johnston 38f708a2bb Remote profile cache should remain in master worker 2018-08-17 11:37:42 +01:00
Erik Johnston 3b2dcfff78 Fix logging bug in EDU handling over replication 2018-08-17 11:11:06 +01:00
Neil Johnson 521d369e7a remove errant yield 2018-08-17 10:12:11 +01:00
Richard van der Hoff 0e8d78f6aa Logcontexts for replication command handlers
Run the handlers for replication commands as background processes. This should
improve the visibility in our metrics, and reduce the number of "running db
transaction from sentinel context" warnings.

Ideally it means converting the things that fire off deferreds into the night
into things that actually return a Deferred when they are done. I've made a bit
of a stab at this, but it will probably be leaky.
2018-08-17 00:43:43 +01:00
Richard van der Hoff 66f7dc8c87 Fix logcontexts for running pushers
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.

Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
Neil Johnson bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Neil Johnson 372bf073c1 block event creation and room creation on hitting resource limits 2018-08-16 21:25:16 +01:00
Neil Johnson 13ad9930c8 add new error type ResourceLimit 2018-08-16 18:02:02 +01:00
Neil Johnson 51b17ec566 flake8 2018-08-16 17:32:22 +01:00
Neil Johnson 3c1080b6e4 refactor for readability, and reuse caching for setting tags 2018-08-16 17:02:04 +01:00
Neil Johnson eff3ae3b9a add room tagging 2018-08-16 15:48:34 +01:00
Erik Johnston 7d5b1a60a3 Fix inbound federation on reader worker
Inbound federation requires calculating push, which in turn relies on
having access to account data.
2018-08-16 15:32:20 +01:00
Neil Johnson bfb6c58624 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-16 15:10:16 +01:00
Neil Johnson a675f9c556 check for room state before deciding on action 2018-08-16 14:53:35 +01:00
Will Hunt c151b32b1d Add GET media/v1/config (#3184) 2018-08-16 14:23:38 +01:00
Matthew Hodgson 762a758fea lazyload aware /messages (#3589) 2018-08-16 14:22:47 +01:00
Neil Johnson 25d2b5d55f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-16 11:37:17 +01:00
Neil Johnson df1e4f259f WIP impl commiting to get feedback 2018-08-16 11:10:53 +01:00
Neil Johnson c055c91655 fix case where empty string state check is evaulated as False 2018-08-16 11:10:19 +01:00
Matthew Hodgson 3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Neil Johnson 8cfad2e686 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 17:20:38 +01:00
Neil Johnson 81d727efa9
Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus
Fix Mau off by one errors
2018-08-15 16:19:41 +00:00
Matthew Hodgson 2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Neil Johnson fc5d937550 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 16:31:40 +01:00
Neil Johnson 86a00e05e1 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-15 16:27:08 +01:00
Neil Johnson eabc5f8271 wip cut at sending resource server notices 2018-08-15 15:04:52 +01:00
Neil Johnson c24fc9797b add new event types 2018-08-15 15:04:30 +01:00
Richard van der Hoff afcd655ab6 Use a producer to stream back responses
The problem with dumping all of the json response into the Request object at
once is that doing so starts the timeout for the next request to be received:
so if it takes longer than 60s to stream back the response to the client, the
client never gets it.

The correct solution is to use a Producer; then the timeout is only started
once all of the content is sent over the TCP connection.
2018-08-15 15:04:16 +01:00
Erik Johnston dc56c47dc0
Merge pull request #3653 from matrix-org/erikj/split_federation
Move more federation APIs to workers
2018-08-15 14:59:02 +01:00
Neil Johnson 4601129c44
Merge pull request #3687 from matrix-org/neilj/admin_email
support admin_email config and pass through into blocking errors,
2018-08-15 13:52:25 +00:00
Erik Johnston ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Erik Johnston 488ffe6fdb Use federation handler function rather than duplicate
This involves renaming _persist_events to be a public function.
2018-08-15 14:17:18 +01:00
Erik Johnston 773db62a22 Rename slave TransactionStore to SlaveTransactionStore 2018-08-15 14:17:06 +01:00
Richard van der Hoff eaaa2248ff Refactor request logging code
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.

Advantages of this are:
 * we get logs for *all* requests (including OPTIONS and HEADs), rather than
   just those that end up hitting handlers we've remembered to decorate
   correctly.

 * when a request handler wires up a Producer (as the media stuff does
   currently, and as other things will do soon), we log at the point that all
   of the traffic has been sent to the client.
2018-08-15 13:47:52 +01:00
Neil Johnson 87a824bad1 clean up AuthError 2018-08-15 11:58:03 +01:00
Neil Johnson c4eb97518f Merge branch 'neilj/update_limits_error_codes' of github.com:matrix-org/synapse into neilj/admin_email 2018-08-15 11:53:01 +01:00
Neil Johnson 55afba0fc5 update admin email to uri 2018-08-15 11:41:18 +01:00
Neil Johnson 1c5e690a6b
Merge pull request #3690 from matrix-org/neilj/change_prometheus_mau_metric_name
combine mau metrics into one group
2018-08-15 09:53:59 +00:00
Erik Johnston fef2e65d12
Merge pull request #3667 from matrix-org/erikj/fixup_unbind
Don't fail requests to unbind 3pids for non supporting ID servers
2018-08-15 10:32:12 +01:00
Neil Johnson b8429c7c81 update error codes for resource limiting 2018-08-15 10:19:48 +01:00
Neil Johnson ab035bdeac replace admin_email with admin_uri for greater flexibility 2018-08-15 10:16:41 +01:00
Neil Johnson 19b433e3f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email 2018-08-14 17:44:46 +01:00
Neil Johnson b586b8b986 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 17:43:22 +01:00
Neil Johnson 70e48cbbb1 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/change_prometheus_mau_metric_name 2018-08-14 17:40:59 +01:00
Neil Johnson 9ecbaf8ba8 adding missing yield 2018-08-14 16:55:28 +01:00
Neil Johnson e5962f845c pep8 2018-08-14 16:36:14 +01:00
Neil Johnson e7d091fb86 combine mau metrics into one group 2018-08-14 16:26:55 +01:00
Neil Johnson 414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
Neil Johnson 2545993ce4 make comments clearer 2018-08-14 15:48:12 +01:00
Neil Johnson 8f9a7eb58d support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-14 15:11:54 +01:00
Neil Johnson c74c71128d remove blank line 2018-08-14 15:06:24 +01:00
Neil Johnson ed4bc3d2fc fix off by 1s on mau 2018-08-14 15:04:48 +01:00
Neil Johnson bd92c8eaa7 Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:56:45 +01:00
Neil Johnson 99ebaed8e6
Update register.py
remove comments
2018-08-14 14:55:55 +01:00
Neil Johnson 9b5bf3d858 Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:51:38 +01:00
Neil Johnson e25d87d97b Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:32:18 +01:00
Neil Johnson e2c9fe0a6a backout ability to pass in event type to server notices 2018-08-14 13:32:56 +01:00
Amber Brown 591bf87c6a Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres 2018-08-14 20:56:23 +10:00
Neil Johnson 9b75c78b4d support server notice state events for resource limits 2018-08-14 11:20:41 +01:00
Neil Johnson 63417c31e9 fix typo 2018-08-13 22:36:52 +01:00
Neil Johnson f4b49152e2 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 21:09:47 +01:00
Richard van der Hoff 50bcaf1a27 Merge pull request #3677 from andrewshadura/master
Use recaptcha_ajax.js directly from Google
2018-08-13 18:52:56 +01:00
Richard van der Hoff 34f51babc5
Revert "Use recaptcha_ajax.js directly from Google" 2018-08-13 18:48:03 +01:00
Neil Johnson ce7de9ae6b Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases"
This reverts commit 0d43f991a1.
2018-08-13 18:06:18 +01:00
Neil Johnson 0d43f991a1 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 18:00:23 +01:00
Amber Brown 99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
Matthew Hodgson 4f7064f6b5 missing import 2018-08-12 19:14:31 -04:00
Matthew Hodgson f0cede5556 missing import 2018-08-12 19:14:31 -04:00
Matthew Hodgson 54ac18e832 use parse_string 2018-08-12 19:14:31 -04:00
Matthew Hodgson 66a4ca1d28 404 nicely if you try to interact with a missing current version 2018-08-12 19:14:31 -04:00
Matthew Hodgson 72788cf9c1 support DELETE /version with no args 2018-08-12 19:14:31 -04:00
Matthew Hodgson edc427a351 flake8 2018-08-12 19:14:31 -04:00
Matthew Hodgson fe87890b18 implement remaining tests and make them work 2018-08-12 19:14:31 -04:00
Matthew Hodgson 15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson b5eee511c7 don't needlessly return user_id 2018-08-12 19:14:31 -04:00
Matthew Hodgson 93d174bcc4 improve docstring 2018-08-12 19:14:31 -04:00
Matthew Hodgson 5e42c45c96 switch get_current_version_info back to being get_version_info 2018-08-12 19:14:31 -04:00
Matthew Hodgson 982edca380 fix flakes 2018-08-12 19:14:31 -04:00
Matthew Hodgson 234611f347 fix typos 2018-08-12 19:14:31 -04:00
Matthew Hodgson 14b3da63a3 add a tonne of docstring; make upload_room_keys properly assert version 2018-08-12 19:14:31 -04:00
Matthew Hodgson 9f0791b7bd add a tonne of docstring; make upload_room_keys properly assert version 2018-08-12 19:14:31 -04:00
Matthew Hodgson 9f500cb39e more docstring for the e2e_room_keys rest 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8d14598e90 add storage docstring; remove unused set_e2e_room_keys 2018-08-12 19:14:31 -04:00
Matthew Hodgson ca0b052307 fix factoring out of _should_replace_room_key 2018-08-12 19:14:31 -04:00
Matthew Hodgson cac0253799 rename room_key_version table correctly, and fix opt args 2018-08-12 19:14:31 -04:00
Matthew Hodgson 0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson 69e51c7ba4 make /room_keys/version work 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Matthew Hodgson cf1e2000f6 document the API 2018-08-12 19:13:09 -04:00
Matthew Hodgson 6b8c07abc2 make it work and fix pep8 2018-08-12 19:13:09 -04:00
Matthew Hodgson 0bc4627a73 interim WIP checkin; doesn't build yet 2018-08-12 18:23:10 -04:00
Matthew Hodgson 53ace904b2 total WIP skeleton for /room_keys API 2018-08-12 18:23:10 -04:00
Neil Johnson 31fa743567 fix sqlite/postgres incompatibility in reap_monthly_active_users 2018-08-11 22:38:34 +01:00
Neil Johnson 6c6aba76e1 implementation of server notices to alert on hitting resource limits 2018-08-10 15:12:59 +01:00
Richard van der Hoff c31793a784 Merge branch 'rav/fix_linearizer_cancellation' into develop 2018-08-10 14:57:27 +01:00
Richard van der Hoff c08f9d95b2 log *after* reloading log config
... because logging *before* reloading means the log message gets lost in the old MemoryLogger
2018-08-10 14:56:48 +01:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Andrej Shadura c75b71a397
Use recaptcha_ajax.js directly from Google
The script recaptcha_ajax.js contains minified non-free code which
we probably cannot redistribute. Since it talks to Google servers
anyway, it is better to just download it from Google directly instead
of shipping it.

This fixes #1932.
2018-08-10 13:36:14 +02:00
Richard van der Hoff 3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Richard van der Hoff 638d35ef08 Fix linearizer cancellation on twisted < 18.7
Turns out that cancellation of inlineDeferreds didn't really work properly
until Twisted 18.7. This commit refactors Linearizer.queue to avoid
inlineCallbacks.
2018-08-10 10:59:09 +01:00
Neil Johnson 01021c812f wip at implementing MSC 7075 2018-08-09 22:16:00 +01:00
Jeroen 2e9c73e8ca more generic conversion of str/bytes to unicode 2018-08-09 21:31:26 +02:00
Jeroen 64899341dc include private functions from twisted 2018-08-09 21:04:22 +02:00
Neil Johnson 885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Neil Johnson c1f9dec92a fix errant parenthesis 2018-08-09 17:43:26 +01:00
Neil Johnson 09cf130898 only block on sync where user is not part of the mau cohort 2018-08-09 17:39:12 +01:00
Erik Johnston bb99b1f550 Add fast path in state res for zero prev events 2018-08-09 14:58:49 +01:00
Erik Johnston ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston 152c0aa58e Add constants for room versions 2018-08-09 14:55:47 +01:00
Erik Johnston 119451dcd1 Refactor state module
We split out the actual state resolution algorithm to prepare for having
multiple versions.
2018-08-09 14:55:47 +01:00
Neil Johnson 69ce057ea6 block sync if auth checks fail 2018-08-09 12:26:27 +01:00
Neil Johnson 3dce9050cf Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:42:02 +01:00
Neil Johnson 0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs
Flag to disable HS without disabling federation
2018-08-09 10:41:43 +00:00
Neil Johnson a5ef110749 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:40:37 +01:00
Erik Johnston a6c813761a Docstrings 2018-08-09 10:41:08 +01:00
Erik Johnston b179537f2a Move clean_room_for_join to master 2018-08-09 10:37:38 +01:00
Erik Johnston 72d1902bbe Fixup doc comments 2018-08-09 10:23:49 +01:00
Amber Brown 984376745b Merge branch 'master' into develop 2018-08-09 19:23:47 +10:00
Amber Brown dafe90a6be 0.33.2 2018-08-09 19:20:41 +10:00