Commit Graph

166 Commits (a4e63e5a47a855884ae3aea41dfbfa464bddb744)

Author SHA1 Message Date
Jonathan de Jong efb6b6629c
Move lint dependencies to extras_require (#8330)
Lint dependencies can now be installed with pip install -e ".[lint]"

This should help keep the version in sync between tox and documentation.
2020-09-17 16:45:22 -04:00
Alexandre Morignot 7586fdf1e8
Bump canonicaljson to version 1.4.0 (#8262)
The version 1.3.0 has a bug with unicode charecters:
```
>>> from canonicaljson import encode_pretty_printed_json
>>> encode_pretty_printed_json({'a': 'à'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/erdnaxeli/.pyenv/versions/3.6.7/lib/python3.6/site-packages/canonicaljson.py", line 96, in encode_pretty_printed_json
    return _pretty_encoder.encode(json_object).encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode character '\xe0' in position 12: ordinal not in range(128)
```

Signed-off-by: Alexandre Morignot <erdnaxeli@cervoi.se>

Co-authored-by: Alexandre Morignot <erdnaxeli@cervoi.se>
2020-09-07 12:21:38 +01:00
Patrick Cloke 2aa127c207
Revert pinning of setuptools (#8239) 2020-09-03 09:45:36 -04:00
Patrick Cloke 318245eaa6
Do not install setuptools 50.0. (#8212)
This is due to compatibility issues with old Python versions.
2020-09-01 08:16:58 -04:00
Erik Johnston 3b4556cf87
Fix `wait_for_stream_position` for multiple waiters. (#8196)
This fixes a bug where having multiple callers waiting on the same
stream and position will cause it to try and compare two deferreds,
which fails (due to the sorted list having an entry of `Tuple[int,
Deferred]`).
2020-08-28 17:12:45 +01:00
Patrick Cloke 2f4d60a5ba
Iteratively encode JSON responses to avoid blocking the reactor. (#8013) 2020-08-18 08:49:59 -04:00
Andrew Morgan e04e465b4d
Use the default templates when a custom template file cannot be found (#8037)
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
Richard van der Hoff 916cf2d439
re-implement daemonize (#8011)
This has long been something I've wanted to do. Basically the `Daemonize` code
is both too flexible and not flexible enough, in that it offers a bunch of
features that we don't use (changing UID, closing FDs in the child, logging to
syslog) and doesn't offer a bunch that we could do with (redirecting stdout/err
to a file instead of /dev/null; having the parent not exit until the child is
running).

As a first step, I've lifted the Daemonize code and removed the bits we don't
use. This should be a non-functional change. Fixing everything else will come
later.
2020-08-04 10:03:41 +01:00
Patrick Cloke 2c1e1b153d
Use the JSON module from the std library instead of simplejson. (#7936) 2020-07-28 10:28:59 -04:00
Oliver Kurz 4e118742ca
Allow to use higher versions of prometheus_client (#7780)
Fixes https://github.com/matrix-org/synapse/issues/7641

The package was pinned to <0.8.0 without an obvious reasoning with
7ad1d7635
in https://github.com/matrix-org/synapse/pull/5636
while the version selection looks to just try to exclude an arbitrary
next minor version number that might introduce API breaking changes.
Selecting the next minor number might be a good conservative selection.

Downstream distributions already reported success patching out the version
requirements.

This also fixes the integration of upgraded packages into openSUSE packages,
e.g. for openSUSE Tumbleweed which already ships prometheus_client >= 0.8 .

Signed-off-by: Oliver Kurz <okurz@suse.de>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-07-06 10:21:41 +01:00
Andrew Morgan 2f6afdd8b4
Explain the purpose of the "tests" conditional dependency requirement (#7751) 2020-06-30 10:11:36 +01:00
Oleg Girko 96bc110a68
Require parameterized package version to be at least 0.7.0. (#7680)
Older versions of `parameterized` package have no `parameterized_class` decorator. This decorator is used in tests.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2020-06-17 15:31:40 +01:00
Dagfinn Ilmari Mannsåker a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Richard van der Hoff 03619324fc
Create a ListenerConfig object (#7681)
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Quentin Gliech 616af44137
Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Richard van der Hoff e053c86a96
Make redis go faster with hiredis (#7439)
For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.
2020-05-06 17:36:46 +01:00
Erik Johnston 51f7eaf908
Add ability to run replication protocol over redis. (#7040)
This is configured via the `redis` config options.
2020-04-22 13:07:41 +01:00
Richard van der Hoff 5a246611e3
Type defintions for use in refactoring for redaction changes (#6803)
* Bump signedjson to 1.1

... so that we can use the type definitions

* Fix breakage caused by upgrade to signedjson 1.1

Thanks, @illicitonion...
2020-01-30 11:25:59 +00:00
Amber Brown 4e1c7b79fa
Remove the psutil dependency (#6318)
* remove psutil and replace with resource
2019-11-05 05:05:48 +11:00
Amber Brown 864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Amber Brown b617864cd9
Fix for structured logging tests stomping on logs (#6023) 2019-09-13 02:29:55 +10:00
Jorik Schellekens 6604b64fae
Check dependencies on setup in the nicer way. (#5989) 2019-09-11 14:00:37 +01:00
Amber Brown 7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Richard van der Hoff 412c6e21a8
Drop dependency on sdnotify (#5871)
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
2019-08-17 09:09:52 +01:00
Erik Johnston c560b791e1 Add process hooks to tell systemd our state.
Fixes #5676.
2019-07-22 14:52:18 +01:00
Amber Brown 7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Jorik Schellekens 38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d906

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
PauRE 856ea04eb3 Fix JWT login (#5555)
* Fix JWT login with register

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Add pyjwt conditional dependency

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Added changelog file

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Improved changelog description

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>
2019-06-27 12:02:41 +01:00
Andrew Morgan 5d6644efe0
Only import jinja2 when needed (#5514)
Fixes https://github.com/matrix-org/synapse/issues/5431

`jinja2` was being imported even when it wasn't strictly necessary. This made it required to run Synapse, even if the functionality that required it wasn't enabled. This was causing new Synapse installations to crash on startup.

Email modules are now required.
2019-06-21 16:52:09 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown 09e9a26b71
Remove Python 2.7 support. (#5425)
* remove 2.7 from CI and publishing

* fill out classifiers and also make it not be installed on 3.5

* some minor bumps so that the old deps work on python 3.5
2019-06-12 21:31:59 +10:00
Erik Johnston a766c41d25 Bump bleach version so that tests can run on old deps. 2019-06-11 12:34:18 +01:00
Richard van der Hoff efe7b3176e Fix federation connections to literal IP addresses
turns out we need a shiny version of service_identity to enforce this
correctly.
2019-06-10 15:58:35 +01:00
Richard van der Hoff d11c634ced clean up impl, and import idna directly 2019-06-10 15:55:12 +01:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Marcus Hoffmann 62388a1e44 remove urllib3 pin (#5230)
requests 2.22.0 as been released supporting urllib3 1.25.2

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2019-05-22 16:48:12 +01:00
Richard van der Hoff bab3eddac4
Pin eliot to <1.8 on python 3.5.2 (#5218)
* Pin eliot to <1.8 on python 3.5.2

Fixes https://github.com/matrix-org/synapse/issues/5199

* Add support for 'markers' to python_dependencies

* tell xargs not to strip quotes
2019-05-21 15:58:01 +01:00
PauRE f89f688a55 Fix image orientation when generating thumbnail (#5039) 2019-05-16 19:04:26 +01:00
Richard van der Hoff 836d3adcce Merge branch 'master' into develop 2019-05-03 19:25:01 +01:00
Richard van der Hoff e3281d7d26 pin urllib3 to <1.25 2019-05-03 18:33:10 +01:00
Silke Hofstra a137f4eac0 Add systemd-python to optional dependencies (#4339)
Using systemd-python allows for logging to the systemd journal,
as is documented in: `synapse/contrib/systemd/log_config.yaml`.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2019-04-16 20:41:17 +10:00
Richard van der Hoff 4abf5aa81a
Bump psycopg requirement (#5032) 2019-04-09 11:29:50 +01:00
Erik Johnston d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry
Add basic optional sentry.io integration
2019-02-18 17:22:45 +00:00
Richard van der Hoff c6e75c9f2d
Merge pull request #4450 from 14mRh4X0r/fix-dependency-message
Fix error message for optional dependencies
2019-02-13 16:12:49 +00:00
Erik Johnston ef2228c890 Basic sentry integration 2019-02-12 13:55:58 +00:00
Erik Johnston b201149c7e
Merge pull request #4420 from matrix-org/jaywink/openid-listener
New listener resource for the federation API "openid/userinfo" endpoint
2019-02-11 09:44:00 +00:00
Willem Mulder d528406cb8 Fix error message for optional dependencies
Signed-off-by: Willem Mulder <willemmaster@hotmail.com>
2019-01-23 18:44:57 +01:00
Amber Brown 6129e52f43
Support ACME for certificate provisioning (#4384) 2019-01-23 19:39:06 +11:00
Jason Robinson 899e60be80 Add parameterized Python module to test dependencies
Allows running parameterized tests. BSD license.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:32:41 +02:00
Richard van der Hoff 676cf2ee26
Fix incorrect logcontexts after a Deferred was cancelled (#4407) 2019-01-17 14:00:23 +00:00