While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
Access it directly from the homeserver itself. It already wasn't
inheriting from BaseHandler storing it on the Handlers object was
already somewhat dubious.
There's been numerous issues with people playing around with their
application service and then not receiving events from their HS for
ages due to backoff timers reaching crazy heights (albeit capped at
< 1 day).
Reduce the max time between pokes to be 8.5 minutes (2^9 secs) which
is quick enough for people to wait it out (avg wait time being 4.25 min)
but long enough to actually give the AS breathing room if it needs it.
Richard van der Hoff