Commit Graph

205 Commits (f608e6c8cf48d8d5327c5ac04746f4f7f3992e31)

Author SHA1 Message Date
Oleg Girko 96bc110a68
Require parameterized package version to be at least 0.7.0. (#7680)
Older versions of `parameterized` package have no `parameterized_class` decorator. This decorator is used in tests.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2020-06-17 15:31:40 +01:00
Dagfinn Ilmari Mannsåker a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Richard van der Hoff 03619324fc
Create a ListenerConfig object (#7681)
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Quentin Gliech 616af44137
Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Richard van der Hoff e053c86a96
Make redis go faster with hiredis (#7439)
For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.
2020-05-06 17:36:46 +01:00
Erik Johnston 51f7eaf908
Add ability to run replication protocol over redis. (#7040)
This is configured via the `redis` config options.
2020-04-22 13:07:41 +01:00
Richard van der Hoff 5a246611e3
Type defintions for use in refactoring for redaction changes (#6803)
* Bump signedjson to 1.1

... so that we can use the type definitions

* Fix breakage caused by upgrade to signedjson 1.1

Thanks, @illicitonion...
2020-01-30 11:25:59 +00:00
Amber Brown 4e1c7b79fa
Remove the psutil dependency (#6318)
* remove psutil and replace with resource
2019-11-05 05:05:48 +11:00
Amber Brown 864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Amber Brown b617864cd9
Fix for structured logging tests stomping on logs (#6023) 2019-09-13 02:29:55 +10:00
Jorik Schellekens 6604b64fae
Check dependencies on setup in the nicer way. (#5989) 2019-09-11 14:00:37 +01:00
Amber Brown 7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Richard van der Hoff 412c6e21a8
Drop dependency on sdnotify (#5871)
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
2019-08-17 09:09:52 +01:00
Erik Johnston c560b791e1 Add process hooks to tell systemd our state.
Fixes #5676.
2019-07-22 14:52:18 +01:00
Amber Brown 7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Jorik Schellekens 38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d906

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
PauRE 856ea04eb3 Fix JWT login (#5555)
* Fix JWT login with register

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Add pyjwt conditional dependency

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Added changelog file

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Improved changelog description

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>
2019-06-27 12:02:41 +01:00
Andrew Morgan 5d6644efe0
Only import jinja2 when needed (#5514)
Fixes https://github.com/matrix-org/synapse/issues/5431

`jinja2` was being imported even when it wasn't strictly necessary. This made it required to run Synapse, even if the functionality that required it wasn't enabled. This was causing new Synapse installations to crash on startup.

Email modules are now required.
2019-06-21 16:52:09 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown 09e9a26b71
Remove Python 2.7 support. (#5425)
* remove 2.7 from CI and publishing

* fill out classifiers and also make it not be installed on 3.5

* some minor bumps so that the old deps work on python 3.5
2019-06-12 21:31:59 +10:00
Erik Johnston a766c41d25 Bump bleach version so that tests can run on old deps. 2019-06-11 12:34:18 +01:00
Richard van der Hoff efe7b3176e Fix federation connections to literal IP addresses
turns out we need a shiny version of service_identity to enforce this
correctly.
2019-06-10 15:58:35 +01:00
Richard van der Hoff d11c634ced clean up impl, and import idna directly 2019-06-10 15:55:12 +01:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Marcus Hoffmann 62388a1e44 remove urllib3 pin (#5230)
requests 2.22.0 as been released supporting urllib3 1.25.2

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2019-05-22 16:48:12 +01:00
Richard van der Hoff bab3eddac4
Pin eliot to <1.8 on python 3.5.2 (#5218)
* Pin eliot to <1.8 on python 3.5.2

Fixes https://github.com/matrix-org/synapse/issues/5199

* Add support for 'markers' to python_dependencies

* tell xargs not to strip quotes
2019-05-21 15:58:01 +01:00
PauRE f89f688a55 Fix image orientation when generating thumbnail (#5039) 2019-05-16 19:04:26 +01:00
Richard van der Hoff 836d3adcce Merge branch 'master' into develop 2019-05-03 19:25:01 +01:00
Richard van der Hoff e3281d7d26 pin urllib3 to <1.25 2019-05-03 18:33:10 +01:00
Silke Hofstra a137f4eac0 Add systemd-python to optional dependencies (#4339)
Using systemd-python allows for logging to the systemd journal,
as is documented in: `synapse/contrib/systemd/log_config.yaml`.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2019-04-16 20:41:17 +10:00
Richard van der Hoff 4abf5aa81a
Bump psycopg requirement (#5032) 2019-04-09 11:29:50 +01:00
Erik Johnston d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry
Add basic optional sentry.io integration
2019-02-18 17:22:45 +00:00
Richard van der Hoff c6e75c9f2d
Merge pull request #4450 from 14mRh4X0r/fix-dependency-message
Fix error message for optional dependencies
2019-02-13 16:12:49 +00:00
Erik Johnston ef2228c890 Basic sentry integration 2019-02-12 13:55:58 +00:00
Erik Johnston b201149c7e
Merge pull request #4420 from matrix-org/jaywink/openid-listener
New listener resource for the federation API "openid/userinfo" endpoint
2019-02-11 09:44:00 +00:00
Willem Mulder d528406cb8 Fix error message for optional dependencies
Signed-off-by: Willem Mulder <willemmaster@hotmail.com>
2019-01-23 18:44:57 +01:00
Amber Brown 6129e52f43
Support ACME for certificate provisioning (#4384) 2019-01-23 19:39:06 +11:00
Jason Robinson 899e60be80 Add parameterized Python module to test dependencies
Allows running parameterized tests. BSD license.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:32:41 +02:00
Richard van der Hoff 676cf2ee26
Fix incorrect logcontexts after a Deferred was cancelled (#4407) 2019-01-17 14:00:23 +00:00
Andrej Shadura 64cf6788d9
Depend on pymacaroons >= 0.13.0 instead on pymacaroons-pynacl
Since 0.13.0, pymacaroons works correctly with pynacl, so there
isn’t any more reason to depend on an outdated pynacl fork.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
2019-01-16 10:54:41 +01:00
Andrej Shadura fab948120f
Use msgpack instead of msgpack-python
The package msgpack-python has been deprecated.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
2019-01-16 10:53:49 +01:00
Richard van der Hoff b7c0218812
Check jinja version for consent resource (#4327)
* Raise a ConfigError if an invalid resource is specified

* Require Jinja 2.9 for the consent resource

* changelog
2019-01-07 10:14:31 +00:00
Amber Brown c26f49a664
Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298) 2018-12-22 01:37:26 +11:00
Richard van der Hoff f208f608cb Merge branch 'release-v0.34.0' into develop 2018-12-11 15:43:20 +00:00
Richard van der Hoff df96177ca7 Stop installing Matrix Console by default
This is based on the work done by @krombel in #2601.
2018-12-11 13:20:33 +00:00
Richard van der Hoff c7401a697f
Implement SAML2 authentication (#4267)
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.

What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
2018-12-07 13:11:11 +01:00
Richard van der Hoff b0c24a66ec Rip out half-implemented m.login.saml2 support (#4265)
* Rip out half-implemented m.login.saml2 support

This was implemented in an odd way that left most of the work to the client, in
a way that I really didn't understand. It's going to be a pain to maintain, so
let's start by ripping it out.

* drop undocumented dependency on dateutil

It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
2018-12-06 19:44:38 +11:00
Amber Brown 4cd1c9f2ff
Delete the disused & unspecced identicon functionality (#4106) 2018-10-29 23:57:24 +11:00
Richard van der Hoff e5b52d0f94 Make psutil an explicit dependency
As of #4027, we require psutil to be installed, so it should be in our
dependency list. We can also remove some of the conditional import code
introduced by #992.

Fixes #4062.
2018-10-19 21:51:15 +01:00
Will Hunt d6a7797dd1 Fix roomlist since tokens on Python 3 (#4046)
Thanks @Half-Shot !!!
2018-10-17 23:04:55 +11:00
Amber Brown 381d2cfdf0
Make workers work on Py3 (#4027) 2018-10-13 00:14:08 +11:00
Richard van der Hoff a59d899668 Pin to prometheus_client<0.4 to avoid renaming all of our metrics 2018-10-03 17:20:15 +01:00
Amber Brown b3064532d0
Run our oldest supported configuration in CI (#3952) 2018-09-27 23:21:54 +10:00
Richard van der Hoff a9d84f4e44 We require attrs 16.0.0
Ref: https://github.com/matrix-org/synapse/issues/3945
2018-09-25 10:43:39 +01:00
Richard van der Hoff 625542878d bump dep on pyopenssl to 16.x 2018-09-06 12:53:15 +01:00
Amber Brown 2d2828dcbc
Port http/ to Python 3 (#3771) 2018-09-06 00:10:47 +10:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff f584d6108f Pin to twisted 17.1 or later
Fixes https://github.com/matrix-org/synapse/issues/3741.
2018-08-24 10:02:31 +01:00
Amber Brown e845fd41c2
Correct attrs package name in requirements (#3492) 2018-07-07 10:46:59 +10:00
Richard van der Hoff 1464a0578a Add explicit dependency on netaddr
the dependencies file, causing failures on upgrade (and presumably for new
installs).
2018-07-06 16:27:17 +01:00
Richard van der Hoff 57e3f923d2 Add missing dependency on attr
We've rcently added a dep on `attr`. I don't know why the CI didn't pick this
up, but we should make it explicit anyway.
2018-06-06 17:12:41 +01:00
Amber Brown f7869f8f8b
Port to sortedcontainers (with tests!) (#3332) 2018-06-06 00:13:57 +10:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Will Hunt 2ad3fc36e6 Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve (#3157)
fixes #3135

Signed-off-by: Will Hunt will@half-shot.uk
2018-04-30 16:21:11 +01:00
Richard van der Hoff d78ada3166 Miscellaneous fixes to python_dependencies
* add some doc about wtf this thing does
* pin Twisted to < 18.4
* add explicit dep on six (fixes #3089)
2018-04-26 13:11:03 +01:00
Richard van der Hoff fac3f9e678 Bump canonicaljson to 1.1.3
1.1.2 was a bit broken too :/
2018-04-13 10:21:38 +01:00
Richard van der Hoff 119596ab8f Update canonicaljson dependency
1.1.0 and 1.1.1 were broken, so we're updating this to help people make sure
they don't end up on a broken version.

Also, 1.1.0 is speedier...
2018-04-12 17:31:44 +01:00
Erik Johnston fecb45e0c3 Remove last usage of ujson 2018-03-26 13:32:29 +01:00
Krombel ed9b5eced4 use bcrypt.checkpw
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago)
This makes use of that with enhancements which might get introduced
by that

Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-05 18:02:59 +01:00
Pascal Bach 3acd616979 Update pynacl dependency to 1.2.1 or higher
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
2018-02-19 10:45:22 +01:00
Oliver Kurz 83d8d4d8cd Allow use of higher versions of saml2
The package was pinned to <4.0 with 07cf96eb because "from saml2 import
config" did not work. This seems to have been fixed in the mean time in the
saml2 package and therefore should not stop to use a more recent version.

Signed-off-by: Oliver Kurz <okurz@suse.de>
2017-11-20 11:14:39 +01:00
Erik Johnston ea87cb1ba5 Make 'affinity' package optional 2017-10-02 18:03:59 +01:00
Richard van der Hoff a7effa8400 Merge pull request #2288 from kyrias/bcrypt
python_dependencies: Use bcrypt module instead of py-bcrypt
2017-08-23 14:14:56 +01:00
Richard van der Hoff 10d8b701a1 Allow configuration of CPU affinity
Make it possible to set the CPU affinity in the config file, so that we don't
need to remember to do it manually every time.
2017-08-15 17:08:28 +01:00
Johannes Löthberg 4f66312df8 python_dependencies: Use bcrypt module instead of py-bcrypt
py-bcrypt has been unmaintained for a long while, while bcrypt is
actively maintained. And since ff8b87118d
we're compatible with the bcrypt anyway.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2017-06-17 17:39:35 +02:00
pik 250ce11ab9 Add jsonschema to python_dependencies.py
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:47 -03:00
David Baker 73a5f06652 Support registration / login with phone number
Changes from https://github.com/matrix-org/synapse/pull/1971
2017-03-13 17:27:51 +00:00
Erik Johnston 7eae6eaa2f Revert "Support registration & login with phone number" 2017-03-13 09:59:33 +00:00
David Baker ce3e583d94 WIP support for msisdn 3pid proxy methods 2017-02-14 15:05:55 +00:00
Johannes Löthberg a9c1b419a9 Bump twisted dependency
At least 16.0.0 is needed for wrapClientTLS support.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 23:16:43 +01:00
Erik Johnston d56c39cf24 Use external ldap auth pacakge 2016-11-15 13:03:19 +00:00
Erik Johnston f9d156d270 New Flake8 fixes 2016-11-15 11:22:29 +00:00
Erik Johnston 4f181f361d Accept optional token to public room list 2016-09-15 09:08:57 +01:00
Erik Johnston 7c1a92274c Make psutil optional 2016-08-08 11:12:21 +01:00
Erik Johnston 66868119dc Add metrics for psutil derived memory usage 2016-07-20 16:00:21 +01:00
Martin Weinelt 0a32208e5d Rework ldap integration with ldap3
Use the pure-python ldap3 library, which eliminates the need for a
system dependency.

Offer both a `search` and `simple_bind` mode, for more sophisticated
ldap scenarios.
- `search` tries to find a matching DN within the `user_base` while
  employing the `user_filter`, then tries the bind when a single
  matching DN was found.
- `simple_bind` tries the bind against a specific DN by combining the
  localpart and `user_base`

Offer support for STARTTLS on a plain connection.

The configuration was changed to reflect these new possibilities.

Signed-off-by: Martin Weinelt <hexa@darmstadt.ccc.de>
2016-06-22 17:51:59 +02:00
David Baker acded821c4 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-04-29 10:05:20 +01:00
David Baker fa12209c1b Hopefully all remaining bits for email notifs
Add public facing base url to the server so synapse knows what URL to use when converting mxc to http urls for use in emails
2016-04-27 15:09:55 +01:00
Erik Johnston 52ecbc2843 Make pyjwt dependency optional 2016-04-25 14:30:15 +01:00
David Baker c10ed26c30 Flesh out email templating
Mostly WIP porting the room name calculation logic from the web client so our room names in the email mirror the clients.
2016-04-21 19:19:07 +01:00
David Baker 2ed0adb075 Generate mails from a template 2016-04-20 18:35:29 +01:00
Erik Johnston d0633e6dbe Sanitize the optional dependencies for spider API 2016-04-13 13:38:09 +01:00
Matthew Hodgson dafef5a688 Add url_preview_enabled config option to turn on/off preview_url endpoint. defaults to off.
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
2016-04-08 18:37:15 +01:00
Matthew Hodgson 9f7dc2bef7 Merge branch 'develop' into matthew/preview_urls 2016-04-04 00:38:21 +01:00
Niklas Riekenbrauck 3f9948a069 Add JWT support 2016-03-29 14:36:36 +02:00
Matthew Hodgson dd4287ca5d make it build 2016-03-29 02:07:57 +01:00
Erik Johnston 07cf96ebf7 Pin pysaml2 version to 3.x
This is due to the fact that `from saml2 import config` fails in version
4.x
2016-03-09 11:54:56 +00:00
Daniel Wagner-Hall 577951b032 Allow third_party_signed to be specified on /join 2016-02-23 15:11:25 +00:00
Matthew Hodgson 7634687057 Merge branch 'master' of git+ssh://github.com/matrix-org/synapse 2016-02-10 16:27:15 +00:00
Matthew Hodgson b3ecb96e36 try to bump syweb to 0.6.8 2016-02-10 16:27:12 +00:00