Commit Graph

674 Commits (a5a799722db0c33dc61fb2c6c7282ff7e82eb2e9)

Author SHA1 Message Date
Andrew Morgan df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Andrew Morgan aeb40f355c
Ensure email validation link parameters are URL-encoded (#6063)
The validation links sent via email had their query parameters inserted without any URL-encoding. Surprisingly this didn't seem to cause any issues, but if a user were to put a `/` in their client_secret it could lead to problems.
2019-09-20 10:46:59 +01:00
Erik Johnston 05bae6b4fc Add opentracing span for HTTP push 2019-09-09 10:24:14 +01:00
Andrew Morgan 0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Andrew Morgan 4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Erik Johnston 96bdd661b8
Remove redundant return 2019-08-13 12:50:36 +01:00
Erik Johnston d02e41dcb2 Handle pusher being deleted during processing.
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
2019-08-01 13:44:12 +01:00
Erik Johnston 2e697d3013 Explicitly check that tombstone is a state event before notifying. 2019-07-31 16:32:03 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
J. Ryan Stinnett 9b1b79f3f5
Add default push rule to ignore reactions (#5623)
This adds a default push rule following the proposal in
[MSC2153](https://github.com/matrix-org/matrix-doc/pull/2153).

See also https://github.com/vector-im/riot-web/issues/10208
See also https://github.com/matrix-org/matrix-js-sdk/pull/976
2019-07-05 17:37:52 +01:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Neil Johnson f47969f42a
Improve email notification logging (#5502) 2019-06-20 15:14:26 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston dd927b29e1
Merge pull request #5388 from matrix-org/erikj/fix_email_push
Fix email notifications for unnamed rooms with multiple people
2019-06-17 13:54:35 +01:00
Erik Johnston 10383e6e6f Change password reset links to /_matrix. 2019-06-11 11:34:33 +01:00
Erik Johnston 2ebeda48b2 Add test 2019-06-07 12:15:33 +01:00
Erik Johnston 928d1ccd73 Fix email notifications for large unnamed rooms.
When we try and calculate a description for a room for with no name but
multiple other users we threw an exception (due to trying to subscript
result of `dict.values()`).
2019-06-07 12:15:28 +01:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Travis Ralston 8c5b1e30d4
Add a default .m.rule.tombstone push rule (#4867)
* Add a default .m.rule.tombstone push rule

In support of MSC1930: https://github.com/matrix-org/matrix-doc/pull/1930

* changelog

* Appease the changelog linter
2019-04-29 15:40:31 -06:00
Brendan Abolivier 91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier 20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Erik Johnston 5bec8d660d Make starting pushers faster during start up
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.

Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
2019-04-02 16:59:13 +01:00
Erik Johnston b82c9cf462 Add missing return 2019-02-22 15:27:40 +00:00
Erik Johnston f2891d2487 Correctly handle PusherConfigException 2019-02-22 15:18:19 +00:00
Erik Johnston a164134a53 Drop logging level of creating a pusher 2019-02-22 14:48:06 +00:00
Erik Johnston 1d9df51ff1 Correctly handle null data in HttpPusher 2019-02-22 14:47:48 +00:00
Richard van der Hoff e07384c4e1
Add prometheus metrics for number of badge update pushes. (#4709)
We're counting the number of push notifications, but not the number of badges;
I'd like to see if they are significant.
2019-02-22 10:57:15 +00:00
Richard van der Hoff 0abb094f1a
bail out early in on_new_receipts if no pushers (#4706) 2019-02-21 17:51:21 +00:00
Richard van der Hoff b2200a8690
Logging improvements for the pusher (#4691) 2019-02-20 11:36:50 +00:00
Amber Brown 3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Richard van der Hoff f5faf6bc14 Fix logcontext leak in EmailPusher 2018-11-19 17:07:01 +00:00
Erik Johnston b86d05a279 Clean up event accesses and tests
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
2018-11-02 13:44:14 +00:00
Amber Brown 0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
Richard van der Hoff c573794b22 Fix start_pushers vs _start_pushers confusion 2018-10-24 09:24:55 +01:00
Richard van der Hoff e564306e31 sanity-check the is_processing flag
... and rename it, for even more sanity
2018-10-24 09:23:33 +01:00
Richard van der Hoff 026cd91ac8 Run PusherPool.start as a background process
We don't do anything with the result, so this is needed to give this code a
logcontext.
2018-10-22 16:12:11 +01:00
Richard van der Hoff f749607c91 Make on_started synchronous too
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
2018-10-22 16:12:11 +01:00
Richard van der Hoff e7a16c6210 Remove redundant run_as_background_process() from pusherpool
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
2018-10-22 16:12:11 +01:00
Richard van der Hoff c7273c11bc Give pushers their own background logcontext
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
2018-10-22 16:12:11 +01:00
Richard van der Hoff 5110f4e425 move get_all_pushers call down
simplifies the interface to _start_pushers
2018-10-22 16:12:11 +01:00
Richard van der Hoff 04277d0ed8 Factor PusherPool._start_pusher out of _start_pushers
... and use it from start_pusher_by_id. This mostly simplifies
start_pusher_by_id.
2018-10-22 16:12:11 +01:00
Richard van der Hoff 3e8b02c939 Rename _refresh_pusher
This is public (or at least, called from outside the class), so ought to have a
better name.
2018-10-22 16:12:11 +01:00
Richard van der Hoff cc325c7069 Calculate absolute path for email templates 2018-10-19 14:01:59 +01:00
Richard van der Hoff 1519572961 Ship the email templates as package_data
move the example email templates into the synapse package so that they can be
used as package data, which should mean that all of the packaging mechanisms
(pip, docker, debian, arch, etc) should now come with the example templates.

In order to grandfather in people who relied on the templates being in the old
place, check for that situation and fall back to using the defaults if the
templates directory does not exist.
2018-10-17 16:46:02 +01:00
Neil Johnson 7de1989ea2 fix link for case that config.email_riot_base_url is set 2018-09-13 22:43:50 +01:00
Amber Brown 7c27c4d51c
merge (#3576) 2018-09-14 03:11:11 +10:00
Richard van der Hoff 66f7dc8c87 Fix logcontexts for running pushers
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.

Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Erik Johnston e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 07cad26d65
Remove all global reactor imports & pass it around explicitly (#3424) 2018-06-25 14:08:28 +01:00
Amber Brown a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Amber Brown c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Amber Brown 754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Amber Brown 1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown e987079037 fixes 2018-05-23 13:03:51 -05:00
Amber Brown 53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown 85ba83eb51 fixes 2018-05-22 16:28:23 -05:00
Amber Brown fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Adrian Tschira 933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira d9fe2b2d9d Replace some more comparisons with six
plus a bonus b"" string I missed last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Richard van der Hoff 950a32eb47
Merge pull request #3152 from NotAFile/py3-local-imports
make imports local
2018-04-30 01:28:13 +01:00
Adrian Tschira 57b58e2174 make imports local
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:41:41 +02:00
Richard van der Hoff fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff 2a13af23bc Use run_in_background in preference to preserve_fn
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff 9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
Richard van der Hoff 605defb9e4 Add missing consumeErrors
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
Richard van der Hoff 9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston 573712da6b Update comments 2018-02-21 11:29:49 +00:00
Erik Johnston 6ff8c87484 Batch inserts into event_push_actions_staging 2018-02-20 11:33:07 +00:00
Erik Johnston 012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston 4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Richard van der Hoff 03dd745fe2 Better logging when pushes fail 2018-01-29 15:49:06 +00:00
Richard van der Hoff e051abd20b add appid/device_display_name to to pusher logging 2018-01-29 15:04:16 +00:00
Richard van der Hoff 4528dd2443 Fix logging and add user_id 2018-01-22 20:15:42 +00:00
Richard van der Hoff 93efd7eb04 logging and debug for http pusher 2018-01-22 18:14:10 +00:00
Richard van der Hoff 2c6d63922a Remove pushers when deleting access tokens
Whenever an access token is invalidated, we should remove the associated
pushers.
2017-11-29 16:44:35 +00:00
Richard van der Hoff 1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs
Remove preserve_context_over_{fn, deferred}
2017-11-15 11:13:53 +00:00
Richard van der Hoff 7e6fa29cb5 Remove preserve_context_over_{fn, deferred}
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
David Baker 2a98ba0ed3 Rename redact_content option to include_content
The redact_content option never worked because it read the wrong config
section. The PR introducing it
(https://github.com/matrix-org/synapse/pull/2301) had feedback suggesting the
name be changed to not re-use the term 'redact' but this wasn't
incorporated.

This reanmes the option to give it a less confusing name, and also
means that people who've set the redact_content option won't suddenly
see a behaviour change when upgrading synapse, but instead can set
include_content if they want to.

This PR also updates the wording of the config comment to clarify
that this has no effect on event_id_only push.

Includes https://github.com/matrix-org/synapse/pull/2422
2017-11-08 10:35:30 +00:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
David Baker 9342bcfce0 Omit the *s for @room notifications
They're just redundant
2017-10-16 13:38:10 +01:00
David Baker 81a5e0073c pep8 2017-10-10 15:53:34 +01:00
David Baker ab1bc9bf5f Don't KeyError if no power_levels event 2017-10-10 15:34:05 +01:00
David Baker 0f1eb3e914 Use notification levels in power_levels
Rather than making the condition directly require a specific power
level. This way the level require to notify a room can be configured
per room.
2017-10-10 15:23:00 +01:00
David Baker c9f034b4ac There was already a constant for this
also update copyright
2017-10-10 11:47:10 +01:00
David Baker a9f9d68631 More optimisation 2017-10-10 11:38:31 +01:00
David Baker 707374d5dc What year is it!? Who's the president!? 2017-10-10 11:21:41 +01:00
David Baker 269af961e9 Make be faster 2017-10-05 13:27:12 +01:00
David Baker ed80c6b6cc Add fastpath optimisation 2017-10-05 13:20:22 +01:00
David Baker e433393c4f pep8 2017-10-05 13:08:02 +01:00
David Baker 985ce80375 They're called rooms 2017-10-05 13:03:44 +01:00
David Baker b9b9714fd5 Get rule type right 2017-10-05 13:02:19 +01:00
David Baker fa969cfdde Support for channel notifications
Add condition type to check the sender's power level and add a base
rule using it for @channel notifications.
2017-10-05 12:39:18 +01:00
David Baker 0c8da8b519 Use better method for word boundary searching
From ebc95667b8
2017-10-05 11:57:43 +01:00
David Baker cbe3c3fdd4 pep8 2017-10-05 11:43:10 +01:00
David Baker 6748f0a579 Fix notif kws that start/end with non-word chars
Only prepend / append word bounary characters if the search
expression starts or ends with a word character, otherwise they
don't work because there's no word bounary between whitespace and
a non-word char.
2017-10-05 11:33:30 +01:00
David Baker 436ee0a2ea Also include the room_id
as really it's part of the event ID
2017-09-18 15:58:38 +01:00
David Baker b393f5db51 Use .get - it's much shorter 2017-09-18 15:50:26 +01:00
David Baker a2562f9d74 Add support for event_id_only push format
Param in the data dict of a pusher that tells an HTTP pusher to
send just the event_id of the event it's notifying about and the
notification counts. For clients that want to go & fetch the body
of the event themselves anyway.
2017-09-18 15:39:39 +01:00
Erik Johnston bfde076022 Increase cache hit ratio for push
We don't update the cache in all code paths, which causes subsequent
calls to miss the cache
2017-07-14 16:11:26 +01:00
Erik Johnston 8d26385d76 Add more metrics to push rule evaluation 2017-07-13 14:37:30 +01:00
Erik Johnston f60218ec41 Push: Don't acquire lock unless necessary 2017-07-13 11:23:53 +01:00
Erik Johnston f502b0dea1 Perf: Don't filter events for push
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
Richard van der Hoff 3d31b39297 Merge pull request #2332 from matrix-org/rav/fix_pushes
Fix caching error in the push evaluator
2017-07-05 11:10:53 +01:00
Richard van der Hoff 73cfe48031 Fix caching error in the push evaluator
Initialising `result` to `{}` in the parameters meant that every call to
_flatten_dict used the *same* target dictionary.

I'm hopeful this will fix https://github.com/matrix-org/synapse/issues/2270,
but I suspect it won't. (This code seems to have been here since forever,
unlike the bug, and I don't really think it explains the observed
behaviour). Still, it makes it hard to investigate the problem.
2017-07-05 00:28:43 +01:00
Caleb James DeLisle 75eba3b07d Fix TravisCI tests for PR #2301 2017-06-23 15:15:18 +02:00
Caleb James DeLisle 1591eddaea Add configuration parameter to allow redaction of content from push messages for google/apple devices 2017-06-23 13:01:04 +02:00
Erik Johnston cca94272fa Fix typo when getting app name 2017-06-06 11:50:07 +01:00
Erik Johnston 66b121b2fc Fix wrong number of arguments 2017-06-06 11:46:38 +01:00
Erik Johnston 11c2a3655f Only load jinja2 templates once
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
2017-05-22 17:48:58 +01:00
Erik Johnston 6489455bed Comment 2017-05-22 16:22:04 +01:00
Erik Johnston d668caa79c Remove spurious log level guards 2017-05-22 16:21:06 +01:00
Erik Johnston 7fb80b5eae Check if current event is a membership event 2017-05-22 15:02:12 +01:00
Erik Johnston 2d17b09a6d Add debug logging 2017-05-22 15:01:36 +01:00
Erik Johnston 24c8f38784 Comment 2017-05-22 14:59:27 +01:00
Erik Johnston 25f03cf8e9 Use tuple unpacking 2017-05-22 14:58:22 +01:00
Erik Johnston 270e1c904a Speed up calculating push rules 2017-05-19 16:51:05 +01:00
Erik Johnston 1c1c0257f4 Move invalidation cb to its own structure 2017-05-19 11:44:11 +01:00
Erik Johnston cafe659f72 Store ActionGenerator in HomeServer 2017-05-19 10:09:56 +01:00
Erik Johnston 72ed8196b3 Don't push users who have left 2017-05-18 17:48:36 +01:00
Erik Johnston 760625acba Make get_if_app_services_interested_in_user faster 2017-05-18 16:34:44 +01:00
Erik Johnston c57789d138 Remove size of push get_rules cache 2017-05-18 16:17:23 +01:00
Erik Johnston 056ba9b795 Add comment 2017-05-18 11:45:56 +01:00
Erik Johnston 66d8ffabbd Faster push rule calculation via push specific cache
We add a push rule specific cache that ensures that we can reuse
calculated push rules appropriately when a user join/leaves.
2017-05-17 16:55:40 +01:00
Erik Johnston 78f306a6f7 Revert "Speed up filtering of a single event in push"
This reverts commit 421fdf7460.
2017-05-08 13:07:41 +01:00
Erik Johnston 421fdf7460 Speed up filtering of a single event in push 2017-04-28 09:52:36 +01:00
Erik Johnston acb58bfb6a fix up 2017-04-25 15:39:19 +01:00
Erik Johnston f7181615f2 Don't specify default as dict 2017-04-25 15:22:59 +01:00
Erik Johnston d9aa645f86 Reduce size of joined_user cache
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Daniel Dent 5058292537 Support authenticated SMTP
Closes (SYN-714) #1385

Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
2017-04-05 21:01:08 -07:00
Erik Johnston 9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed
Speed up cached function access
2017-03-31 10:10:56 +01:00
Erik Johnston 86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Erik Johnston a3810136fe Cache glob to regex at a higher level for push 2017-03-29 15:53:14 +01:00
Erik Johnston 6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Erik Johnston 8ffbe43ba1 Get current state by using current_state_events table 2017-03-10 17:39:35 +00:00
Erik Johnston 7455ba436a Ensure we pass positive ints to delay function 2017-02-22 12:08:14 +00:00
Erik Johnston 51adaac953 Fix email push in pusher worker
This was broken when device list updates were implemented, as Mailer
could no longer instantiate an AuthHandler due to a dependency on
federation sending.
2017-02-02 10:53:36 +00:00
David Baker 97d3918377 Merge pull request #1811 from aperezdc/unhardcode-riot-urls
Allow configuring the Riot URL used in notification emails
2017-01-18 14:38:49 +00:00
Erik Johnston 04006bb7f0 Get state at event rather than for room in push 2017-01-17 14:31:21 +00:00
Adrian Perez de Castro a3e4a198e3
Allow configuring the Riot URL used in notification emails
The URLs used for notification emails were hardcoded to use either matrix.to
or vector.im; but for self-hosted setups where Riot is also self-hosted it
may be desirable to allow configuring an alternative Riot URL.

Fixes #1809.

Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
2017-01-13 17:12:04 +02:00
Erik Johnston 52d12ca782 Add /room/<room_id>/joined_members API
This returns the currently joined members in the room with their display
names and avatar urls. This is more efficient than /members for large
rooms where you don't need the full events.
2016-12-08 13:32:07 +00:00
Erik Johnston 8ca05b5755 Fix push notifications for a single unread message 2016-10-18 10:57:33 +01:00
Erik Johnston f0ca088280 Reduce redundant database work in email pusher
Update the last stream ordering if the
`get_unread_push_actions_for_user_in_range_for_email` returns no new
push actions. This reduces the range that it needs to check next
iteration.
2016-10-18 10:52:47 +01:00
Erik Johnston 1d107d8484 Fix email push notifs being dropped
A lot of email push notifications were failing to be sent due to an
exception being thrown along one of the (many) paths. This was due to a
change where we moved from pulling out the full state for each room, but
rather pulled out the event ids for the state and separately loaded the
full events when needed.
2016-10-13 13:40:38 +01:00
Erik Johnston 49cf205dc7 _id field must uniquely identify different conditions 2016-09-19 10:34:01 +01:00
Matthew Hodgson 3f6ec271ba proposal for notifying on e2e events 2016-09-17 22:05:06 +01:00
Erik Johnston 6a5ded5988 Ensure we don't mutate the cache of push rules 2016-09-15 09:16:13 +01:00
Erik Johnston 3847fa38c4 Make bulk_get_push_rules_for_room use get_joined_users_from_context cache 2016-09-05 10:02:38 +01:00
Erik Johnston 44982606ee Merge pull request #1060 from matrix-org/erikj/state_ids
Assign state groups in state handler.
2016-09-01 14:20:42 +01:00
Erik Johnston c10cb581c6 Correctly handle the difference between prev and current state 2016-08-31 14:26:22 +01:00
Erik Johnston d80f64d370 Fix email notifs by adding missing param 2016-08-30 21:46:39 +01:00
Erik Johnston 55e17d3697 Fix push room names for rooms with only an alias 2016-08-30 11:19:59 +01:00
Erik Johnston bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston a234e895cf Fix room name in email notifs 2016-08-26 14:10:21 +01:00
Erik Johnston c7cd35d682 Typo 2016-08-26 11:23:58 +01:00
Erik Johnston 7356d52e73 Fix up push to use get_current_state_ids 2016-08-25 18:35:49 +01:00
Erik Johnston a3dc1e9cbe Replace context.current_state with context.current_state_ids 2016-08-25 17:32:22 +01:00
Erik Johnston 9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Erik Johnston f164fd9220 Move _bulk_get_push_rules_for_room to storage layer 2016-08-19 14:29:20 +01:00
Erik Johnston abcb9aee5b Make push Measure finer grained 2016-08-17 18:00:18 +01:00
David Baker 0bba2799b6 Merge remote-tracking branch 'origin/develop' into dbkr/contains_display_name_override 2016-08-16 16:46:37 +01:00
David Baker 1594eba29e s/underride/override/ in the rule_id too 2016-08-16 16:44:07 +01:00
David Baker 1c7c317df1 Move display name rule
As per https://github.com/matrix-org/matrix-doc/pull/373 and comment
2016-08-15 18:34:53 +01:00
Erik Johnston dc3a00f24f Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves. 2016-08-15 17:04:39 +01:00
Erik Johnston b260f92936 Ignore AlreadyCalled errors on timer cancel 2016-07-31 16:00:12 +01:00
Mark Haines 0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
David Baker ecd5e6bfa4 Typo 2016-07-28 10:04:52 +01:00
David Baker 0b640aa56b even more pep8 2016-06-24 11:47:11 +01:00
David Baker 870c45913e Use similar naming we use in email notifs for push
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
Mark Haines ded01c3bf6 Fix ``KeyError: 'msgtype'``. Use ``.get``
Fixes a key error where the mailer tried to get the ``msgtype`` of an
event that was missing a ``msgtype``.

```
 File "synapse/push/mailer.py", line 264, in get_notif_vars
 File "synapse/push/mailer.py", line 285, in get_message_vars
 File ".../frozendict/__init__.py", line 10, in __getitem__
    return self.__dict[key]
    KeyError: 'msgtype'
```
2016-06-17 13:49:16 +01:00
Mark Haines 0b2158719c Remove dead code.
Loading push rules now happens in the datastore, so we can remove
the methods that loaded them outside the datastore.

The ``waiting_for_join_list`` in federation handler is populated by
anything, so can be removed.

The ``_get_members_events_txn`` method isn't called from anywhere
so can be removed.
2016-06-07 15:07:11 +01:00
David Baker 72c4d482e9 3rd time lucky: we'd already calculated it above 2016-06-03 16:39:50 +01:00
David Baker fbf608decb Oops, we're using the dict form 2016-06-03 16:38:39 +01:00
David Baker 06d40c8b98 Add substitutions to email notif From 2016-06-03 16:31:23 +01:00
Matthew Hodgson 8d740132f4 Merge branch 'develop' into matthew/brand-from-header 2016-06-03 12:14:18 +01:00
Erik Johnston 6a0afa582a Load push rules in storage layer, so that they get cached 2016-06-03 11:10:00 +01:00
Erik Johnston 59f2d73522 Remove unnecessary sets 2016-06-03 11:05:45 +01:00
Erik Johnston 9c26b390a2 Only get local users 2016-06-03 11:04:31 +01:00
Erik Johnston 065e739d6e Merge pull request #811 from matrix-org/erikj/state_users_in_room
Use state to calculate get_users_in_room
2016-06-03 10:58:27 +01:00
Erik Johnston 696d7c5937 Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room
Add get_users_with_read_receipts_in_room cache
2016-06-03 10:58:24 +01:00
Matthew Hodgson 79d1f072f4 brand the email from header 2016-06-02 21:34:40 +01:00
David Baker 6bb9aacf9d Merge pull request #821 from matrix-org/dbkr/email_unsubscribe
Email unsubscribe links that don't require logging in
2016-06-02 17:44:55 +01:00
David Baker 745ddb4dd0 peppate 2016-06-02 17:38:41 +01:00
David Baker 7a5a5f2df2 Merge pull request #820 from matrix-org/dbkr/email_notif_string_fmt_error
Fix error in email notification string formatting
2016-06-02 17:26:06 +01:00
David Baker 1f31cc37f8 Working unsubscribe links going straight to the HS
and authed by macaroons that let you delete pushers and nothing else
2016-06-02 17:21:31 +01:00
Matthew Hodgson 2675c1e40e add some branding debugging 2016-06-02 17:21:12 +01:00
David Baker c71177f285 Merge remote-tracking branch 'origin/dbkr/email_notif_string_fmt_error' into dbkr/email_unsubscribe 2016-06-02 17:20:56 +01:00
David Baker 07a5559916 Fix error in email notification string formatting 2016-06-02 17:17:16 +01:00
David Baker 812b5de0fe Merge remote-tracking branch 'origin/develop' into dbkr/email_unsubscribe 2016-06-02 15:33:28 +01:00
Erik Johnston 356f13c069 Disable INCLUDE_ALL_UNREAD_NOTIFS 2016-06-02 14:07:38 +01:00
David Baker 3a3fb2f6f9 Merge branch 'dbkr/split_out_auth_handler' into dbkr/email_unsubscribe 2016-06-02 13:35:25 +01:00
Matthew Hodgson f84b89f0c6 if an email pusher specifies a brand param, use it 2016-06-02 13:29:48 +01:00
David Baker a15ad60849 Email unsubscribing that may in theory, work
Were it not for that fact that you can't use the base handler in the pusher because it pulls in the world. Comitting while I fix that on a different branch.
2016-06-02 11:44:15 +01:00
David Baker e793866398 Use user_id in email greeting if display name is null 2016-06-02 09:41:13 +01:00
Erik Johnston c8285564a3 Use state to calculate get_users_in_room 2016-06-01 15:25:25 +01:00
Erik Johnston 43db0d9f6a Add get_users_with_read_receipts_in_room cache 2016-06-01 10:54:32 +01:00
Erik Johnston e5b0bbcd33 Add caches to bulk_get_push_rules* 2016-05-31 13:46:58 +01:00
David Baker b007ee4606 Check for presence of 'avatar_url' key 2016-05-24 15:12:05 +01:00
Matthew Hodgson cb8a321bdd fix NPE in room ordering 2016-05-23 22:54:56 +01:00
Matthew Hodgson 88ea5ab2c3 consistency is the better part of valour 2016-05-23 19:33:45 +01:00
Matthew Hodgson 989bdc9e56 Tune email notifs to make them quieter:
* After initial 10 minute window, only alert every 24h for room notifs
 * Reset room state after 6h of idleness
 * Synchronise throttles for messages sent in the same notif, so the 24 hourly notifs 'line up'
 * Fix the email subjects to say what triggered the notification
 * Order the rooms in reverse activity order in the email, so the 'reason' room should always come first
2016-05-23 19:24:11 +01:00
Matthew Hodgson 6fb51eaf7b Merge pull request #793 from matrix-org/matthew/one-push-badge-per-convo
increment badge count per missed convo, not per msg
2016-05-18 13:56:38 +01:00
Matthew Hodgson e837df6adb increment badge count per missed convo, not per msg 2016-05-18 11:53:25 +01:00
Matthew Hodgson 43e1e0489c Merge pull request #786 from matrix-org/matthew/email_notifs_tuning
tune email notifs, fix CSS a bit, and add debugging details
2016-05-17 10:43:24 +01:00
Matthew Hodgson cbd2adc95e tune email notifs, fix CSS a bit, and add debugging details 2016-05-16 18:58:38 +01:00
David Baker 647781ca56 Fix emailpusher import
Try importing at the root level rather than conditionally importing, as per comment
2016-05-16 18:41:32 +01:00
Matthew Hodgson 782471b7e1 fix matrix.to URLs 2016-05-13 17:50:16 +01:00
David Baker b5e646a18c Make email notifs work on the pusher synapse
Plus general bugfix to email notif code
2016-05-13 11:36:50 +01:00
Mark Haines 7e23476814 move filter_events_for_client out of base handler 2016-05-11 13:42:37 +01:00
David Baker e04b1d6b0a Make pep8 happy 2016-05-10 14:23:16 +02:00
Matthew Hodgson 5599608887 Switch from CSS to Table layout for HTML mails so they work in Outlook aka Word
Remove templates-vector and theme templates with variables instead
    Switch to matrix.to URLs by default for links
2016-05-10 00:14:48 +02:00
Matthew Hodgson 53ca739f1f better mail subject lines 2016-05-05 15:55:44 +01:00
Matthew Hodgson 81c2176cba fix layout; handle app naming in synapse, not jinja 2016-05-05 15:54:29 +01:00
Matthew Hodgson 634efb65f1 pep8 2016-05-05 02:10:57 +01:00
Matthew Hodgson ce81ccb063 handle fragments correctly on mxc URLs.
switch to vector.im permalinks as matrix.to isn't ready yet.
merge overlapping notifications together.
give one message of context after a notification (in the unlikely event it exists, but it's possible thanks to throttling).
include name of app in mail templates
2016-05-05 02:00:33 +01:00
Matthew Hodgson 1cf5c379cb spell out emailpusher full path 2016-05-05 01:59:39 +01:00
David Baker 39d0a99972 Include no context
until we can de-dup between the context and other notifs
2016-05-04 14:52:49 +01:00
David Baker 9ef05a12c3 Add date header & message id 2016-05-04 14:52:10 +01:00
David Baker de22001ab5 pep8 2016-05-04 11:41:35 +01:00
Matthew Hodgson f1026418ea copyright 2016-05-04 11:38:01 +01:00
David Baker e6bffa4475 Unused import 2016-05-04 11:26:58 +01:00
David Baker 92f0f3d21d Catch all exceptions when creating a pusher 2016-05-04 11:24:07 +01:00
David Baker b084e4d963 Add constant for throttle multiplier 2016-04-29 20:14:55 +01:00
David Baker 35b7b8e4bc Remove unused function 2016-04-29 20:10:34 +01:00
David Baker 6b9b6a9169 Remove unused arg 2016-04-29 20:02:52 +01:00
David Baker e7a76b5123 Use the constant 2016-04-29 19:10:45 +01:00
David Baker 29c8cf8db8 Avoid `vars` builtin 2016-04-29 19:09:28 +01:00
David Baker d3da5294e8 Use named parameter format 2016-04-29 19:04:40 +01:00
David Baker 311b5ce051 pep8 2016-04-29 14:37:30 +01:00
David Baker 3facde2536 Remove rather pointless get function 2016-04-29 14:36:45 +01:00
David Baker 4364ea1272 Stop processing notifs once we've sent a mail 2016-04-29 14:31:27 +01:00
David Baker 40d40e470d Send mail notifs with a plaintext part too 2016-04-29 13:56:21 +01:00
David Baker b2c04da8dc Add an email pusher for new users
If they registered with an email address and email notifs are enabled on the HS
2016-04-29 11:43:57 +01:00
David Baker acded821c4 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-04-29 10:05:20 +01:00
David Baker e800ee2f63 May as well always include room link 2016-04-28 17:28:27 +01:00
David Baker cc0874cf71 Put back real delay before mailing 2016-04-28 17:00:40 +01:00
David Baker 68f8fc2f14 Support file messages & fix plain text 2016-04-28 16:59:57 +01:00
David Baker 4845c7359d Support image notifs 2016-04-28 15:55:53 +01:00
David Baker 60f86fc876 pep8 2016-04-28 15:16:30 +01:00
David Baker 937c407eef Only import email pusher if email notifs are on 2016-04-28 15:12:14 +01:00
David Baker ebbabc4986 Handle room invites in email notifs 2016-04-28 11:49:36 +01:00
David Baker 9dba1b668c Linkify plain text messages too 2016-04-28 10:55:08 +01:00
David Baker 424a7f48f8 Run filter_events_for_client
so we don't accidentally mail out events people shouldn't see
2016-04-27 17:50:49 +01:00
David Baker 4ed1e45869 Make html messages work 2016-04-27 17:18:51 +01:00
David Baker 8781083960 Better grammar for multiple messages in a room
Say who the messages are from if there's no room name, otherwise it's a bit nonsensical
2016-04-27 15:30:41 +01:00
David Baker fa12209c1b Hopefully all remaining bits for email notifs
Add public facing base url to the server so synapse knows what URL to use when converting mxc to http urls for use in emails
2016-04-27 15:09:55 +01:00
David Baker 7b4715bad7 More variable calculation for email notifs
Include name of the person we're sending to and add summary text at the top giving an overview of what's happened.
2016-04-25 18:27:04 +01:00