0e27501ee5
Fix UPSERT check ( #4459 )
2019-01-24 22:57:41 +11:00
e8c9f15397
Replace missed usages of FrozenEvent
2019-01-24 11:14:07 +00:00
58f6c48183
Use native UPSERTs where possible ( #4306 )
2019-01-24 21:31:54 +11:00
17898a5ba6
Merge branch 'erikj/fixup_rejecting_invites' of github.com:matrix-org/synapse into erikj/require_format_version
2019-01-23 20:13:07 +00:00
7c288c2250
Clarify the invite flows
2019-01-23 20:07:47 +00:00
886e5acc76
Store rejected remote invite events as outliers
...
Currently they're stored as non-outliers even though the server isn't in
the room, which can be problematic in places where the code assumes it
has the state for all non outlier events.
In particular, there is an edge case where persisting the leave event
triggers a state resolution, which requires looking up the room version
from state. Since the server doesn't have the state, this causes an
exception to be thrown.
2019-01-23 20:04:57 +00:00
2a8edbaf74
Merge branch 'erikj/get_pdu_versions' into erikj/require_format_version
2019-01-23 17:27:49 +00:00
be1065af59
isort
2019-01-23 11:48:16 +00:00
c5a296b10c
Add support for persisting event format versions
...
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.
We specify the format version rather than the room version as:
1. We don't necessarily know the room version, existing events may be
either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
events in the same room reported to be of different versions, which
sounds annoying.
2019-01-23 11:30:01 +00:00
90743c9d89
Fixup removal of duplicate `user_ips` rows ( #4432 )
...
* Remove unnecessary ORDER BY clause
* Add logging
* Newsfile
2019-01-23 19:45:18 +11:00
7f503f83b9
Refactor to rewrite the SQL instead
2019-01-22 16:31:05 +00:00
1c9704f8ab
Don't shadow params
2019-01-22 16:20:33 +00:00
2557531f0f
Fix bug when removing duplicate rows from user_ips
...
This was caused by accidentally overwritting a `last_seen` variable
in a for loop, causing the wrong value to be written to the progress
table. The result of which was that we didn't scan sections of the table
when searching for duplicates, and so some duplicates did not get
deleted.
2019-01-22 13:33:46 +00:00
277e50462d
Do not return in a deferred function
2019-01-22 12:40:26 +00:00
c433f61091
Ensure new filter is actually created
2019-01-22 12:06:36 +00:00
c9bfb058d8
Fix a bug with single-room search searching all rooms
...
* Create a new method for getting predecessor rooms
* Remove formatting change
2019-01-22 12:00:41 +00:00
25d64a846a
Fix typos
2019-01-22 11:00:04 +00:00
25dd56ace3
Fix race when persisting create event ( #4404 )
...
* Fix race when persisting create event
When persisting a chunk of DAG it is sometimes requried to do a state
resolution, which requires knowledge of the room version. If this
happens while we're persisting the create event then we need to use that
event rather than attempting to look it up in the database.
2019-01-18 23:17:04 +11:00
df3a661e4a
Search for messages across predecessor rooms
...
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
2019-01-18 11:19:20 +00:00
a35c66a00b
Remove duplicates in the user_ips table and add an index ( #4370 )
2019-01-12 06:21:50 +11:00
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. ( #4369 )
2019-01-09 22:26:25 +11:00
84b6fae1f5
Ensure synchrotrons can access is_support_user in the storage layer
2019-01-02 10:19:59 +00:00
0708f437cc
Log roomid along with Unknown room ( #4297 )
2018-12-24 10:49:42 +01:00
d2f7c4e6b1
create support user ( #4141 )
...
Allow for the creation of a support user.
A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
e93a0ebf50
Settings Fix deleting e2e room keys on xenial ( #4295 )
2018-12-14 05:10:31 +11:00
30da50a5b8
Initialise user displayname from SAML2 data ( #4272 )
...
When we register a new user from SAML2 data, initialise their displayname
correctly.
2018-12-07 14:44:46 +01:00
9a3e24a13d
drop undocumented dependency on dateutil ( #4266 )
...
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
2018-12-06 04:52:42 +11:00
158ffb92f1
Add an option to disable search for homeservers which may not be interested in it ( #4230 )
...
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
2018-12-04 12:01:02 +01:00
b5ac0ffa0a
add more detail to logging regarding "More than one row matched" error ( #4234 )
2018-12-04 11:57:39 +01:00
ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english ( #4253 )
...
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252 . Fundamentally the right answer is to drop support for Python 2.
2018-12-04 11:55:52 +01:00
704c5298f0
Drop sent_transactions
...
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2018-12-01 23:07:35 -06:00
f9b136a886
Neilj/fix mau initial reserved users ( #4211 )
...
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail
* towncrier
* white space
2018-11-28 20:33:41 +11:00
6c18cc4b50
Ignore __pycache__ directories in schema delta dir
...
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
2018-11-20 22:52:34 +00:00
835779f7fb
Add option to track MAU stats (but not limit people) ( #3830 )
2018-11-15 18:08:27 +00:00
bca3b91c2d
Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers
2018-11-09 18:35:02 +00:00
d44dea0223
pep8
2018-11-09 14:38:31 +00:00
4f93abd62d
add docs
2018-11-09 13:25:38 +00:00
5ebed18692
Lets convert bytes to unicode instead
2018-11-08 12:33:13 +00:00
b1a22b24ab
Fix noop checks when updating device keys
...
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.
The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
2018-11-08 12:18:38 +00:00
bc80b3f454
Add helpers for getting prev and auth events ( #4139 )
...
* Add helpers for getting prev and auth events
This is in preparation for allowing the event format to change between
room versions.
2018-11-06 00:35:15 +11:00
350f654e7b
Add unique indexes to a couple of tables
...
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
2018-11-02 10:36:13 +00:00
50e328d1e7
Remove redundant database locks for device list updates
...
We can rely on the application-level per-user linearizer.
2018-11-02 10:36:13 +00:00
0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups
...
Delete unreferenced state groups during purge
2018-10-30 16:58:22 +00:00
e0934acdbb
Cast to int here too
2018-10-30 11:12:23 +00:00
12941f5f8b
Cast bacjup version to int when querying
2018-10-30 11:01:07 +00:00
2f0f911c52
Convert version back to a string
2018-10-30 10:35:18 +00:00
64fa557f80
Try & make it work on postgres
2018-10-30 09:51:04 +00:00
563f9b61b1
Make e2e backup versions numeric in the DB
...
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
2018-10-29 21:01:22 +00:00
88e5ffe6fe
Deduplicate device updates sent over replication
...
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.
A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
2018-10-29 17:34:34 +00:00
ad88460e0d
Move _find_unreferenced_groups
2018-10-29 14:24:19 +00:00
664b192a3b
Fix set operations thinko
2018-10-29 14:21:43 +00:00
f4f223aa44
Don't make temporary list
2018-10-29 14:01:49 +00:00
b2399f6281
Make SQL a bit cleaner
2018-10-29 14:01:11 +00:00
03e634dad4
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups
2018-10-26 16:22:45 +01:00
4cda300058
preserve room visibility
2018-10-25 19:10:24 +01:00
cb53ce9d64
Refactor state group lookup to reduce DB hits ( #4011 )
...
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init
...
fix race condiftion in calling initialise_reserved_users
2018-10-25 16:33:40 +01:00
fcbd488e9a
add new line
2018-10-25 16:13:43 +01:00
f8fe98812b
improve comments
2018-10-25 14:58:59 +01:00
f7f487e14c
Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin
2018-10-25 14:40:06 +01:00
ea69a84bbb
fix style inconsistencies
2018-10-24 17:18:08 +01:00
07126e43a4
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init
2018-10-24 16:25:39 +01:00
7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8
...
Fix a number of flake8 errors
2018-10-24 11:23:35 +01:00
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase
...
Add v2 state resolution algorithm
2018-10-24 11:12:12 +01:00
ef771cc4c2
Fix a number of flake8 errors
...
Broadly three things here:
* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
them
* use `r""` for strings which include backslashes
Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
2018-10-24 10:39:03 +01:00
329d18b39c
remove white space
2018-10-23 15:27:20 +01:00
6105c6101f
fix race condiftion in calling initialise_reserved_users
2018-10-23 15:24:58 +01:00
e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) ( #4068 )
2018-10-20 11:16:55 +11:00
67f7b9cb50
pep8
2018-10-19 16:06:59 +01:00
056f099126
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups
2018-10-19 15:48:59 +01:00
47a9da28ca
Batch process handling state groups
2018-10-19 15:48:15 +01:00
fc0f13dd03
Fix incorrect truncation in get_missing_events
...
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.
We also shouldn't really be relying on depth, and should be checking the
room_id.
2018-10-16 21:10:04 +01:00
4a28d3d36f
Update event_auth table for rejected events
2018-10-16 16:28:42 +01:00
e238013c44
Add v2 state res algorithm.
...
We hook this up to the vdh test room version.
2018-10-16 16:16:13 +01:00
03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups
...
E2E backups
2018-10-15 10:19:43 +01:00
67a1e315cc
Fix up comments
2018-10-12 13:49:48 +01:00
306361b31b
Misc PR feedback bits
2018-10-12 11:48:56 +01:00
8ddd0f273c
Comments on get_all_new_events_stream
...
just some docstrings to clarify the behaviour here
2018-10-12 09:55:41 +01:00
dc045ef202
Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups
2018-10-09 10:05:02 +01:00
497444f1fd
Don't reuse backup versions
...
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.
Fixes https://github.com/vector-im/riot-web/issues/7448
2018-10-05 15:08:36 +01:00
ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin
2018-10-04 17:26:59 +01:00
4917ff5523
Add state_group index to event_to_state_groups
...
This is needed to efficiently check for unreferenced state groups during
purge.
2018-10-04 16:03:08 +01:00
17d585753f
Delete unreferened state groups during purge
2018-10-04 16:03:06 +01:00
9693625e55
actually exclude outliers
2018-10-03 10:19:41 +01:00
3e39783d5d
remove debugging
2018-10-02 23:44:14 +01:00
ae61ade891
Fix bug in forward_extremity update logic
...
An event does not stop being a forward_extremity just because an outlier or
rejected event refers to it.
2018-10-02 23:36:08 +01:00
7258d081a5
Fix bug when invalidating destination retry timings
2018-10-02 15:47:57 +01:00
5b68f29f48
fix thinkos
2018-09-29 02:14:40 +01:00
9c8cec5dab
Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache
2018-09-28 10:51:09 +01:00
51d33d5178
Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts
...
fix #3854 MAU transaction errors
2018-09-27 12:31:37 +01:00
ae6ad4cf41
docstrings and unittests for storage.state ( #3958 )
...
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
28781b65e7
fix #3854
2018-09-26 16:16:41 +01:00
fdd1a62e8d
Add a five minute cache to get_destination_retry_timings
...
Hopefully helps with #3931
2018-09-21 14:56:12 +01:00
1f3f5fcf52
Fix client IPs being broken on Python 3 ( #3908 )
2018-09-20 20:14:34 +10:00
bbab6ebfd9
Fix up changelog and remove spurious comment
2018-09-19 14:45:14 +01:00
392a54128c
pep8
2018-09-19 14:37:49 +01:00
b9158ac2bf
Remove get_destination_retry_timings cache
...
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.
Fixes #3798
2018-09-19 14:22:57 +01:00
85a43f4167
Return a 404 when deleting unknown room alias
...
As per https://github.com/matrix-org/matrix-doc/issues/1675
Fixes https://github.com/matrix-org/synapse/issues/2782
2018-09-17 13:19:00 +01:00
bc74925c5b
WIP e2e key backups
...
Continues from uhoreg's branch
This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
2018-09-13 17:02:59 +01:00
ed5331a627
comment
2018-09-13 16:10:56 +01:00
9cbd0094f0
pep8
2018-09-13 15:15:35 +01:00
9dbe38ea7d
Create indices after insertion
2018-09-13 15:05:52 +01:00
c857f5ef9b
Make purge history slightly faster
...
Don't pull out events that are outliers and won't be deleted, as nothing
should happen to them.
2018-09-13 12:48:10 +01:00
f30a303590
Merge pull request #3846 from matrix-org/neilj/expose-registered-users
...
expose number of real reserved users
2018-09-12 17:14:04 +01:00
8decd6233d
improve naming
2018-09-12 16:22:15 +01:00
0ddf486724
expose number of real reserved users
2018-09-12 11:58:52 +01:00
33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring
...
add some logging for the keyring queue
2018-09-12 20:43:47 +10:00
b041115415
Speed up lazy loading ( #3827 )
...
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats (#3840 )
2018-09-12 00:50:39 +01:00
cd7ef43872
clearer logging when things fail, too
2018-09-06 23:56:47 +01:00
84a750e0c3
ensure guests never enter mau list
2018-09-06 17:22:53 +01:00
3801b8aa03
try to make flake8 and isort happy
2018-09-06 11:35:19 -04:00
61b05727fa
guest users should not be part of mau total
2018-09-05 22:30:36 +01:00
0b01281e77
move threepid checker to config, add missing yields
2018-08-31 17:11:11 +01:00
09f3cf1a7e
ensure post registration auth checks do not fail erroneously
2018-08-31 15:42:51 +01:00
14e4d4f4bf
Port storage/ to Python 3 ( #3725 )
2018-08-31 00:19:58 +10:00
83caead95a
Merge branch 'develop' into e2e_backups
2018-08-24 11:44:26 -04:00
cd77270a66
Implement trail users
2018-08-23 19:17:19 +01:00
7a0da69eee
Add missing yield
2018-08-23 10:28:12 +01:00
764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles
...
Allow profile updates to happen on workers
2018-08-22 11:35:55 +01:00
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
...
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
bb81e78ec6
Split the state_group_cache in two ( #3726 )
...
Splits the state_group_cache in two.
One half contains normal state events; the other contains member events.
The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.
Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720 .
2018-08-22 00:56:37 +02:00
42a394caa2
allow session_data to be any JSON instead of just a string
2018-08-21 14:51:34 -04:00
8550a7e9c2
allow auth_data to be any JSON instead of a string
2018-08-21 10:38:00 -04:00
4d664278af
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler
2018-08-20 14:49:43 +01:00
782689bd40
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-17 14:15:48 +01:00
38f708a2bb
Remote profile cache should remain in master worker
2018-08-17 11:37:42 +01:00
bcfeb44afe
call reap on start up and fix under reaping bug
2018-08-16 22:55:32 +01:00
3f543dc021
initial cut at a room summary API ( #3574 )
2018-08-16 09:46:50 +01:00
2f78f432c4
speed up /members and add at= and membership params ( #3568 )
2018-08-15 16:35:22 +01:00
ef184caf30
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-15 14:25:46 +01:00
9ecbaf8ba8
adding missing yield
2018-08-14 16:55:28 +01:00
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
...
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
591bf87c6a
Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres
2018-08-14 20:56:23 +10:00
99dd975dae
Run tests under PostgreSQL ( #3423 )
2018-08-13 16:47:46 +10:00
f0cede5556
missing import
2018-08-12 19:14:31 -04:00
66a4ca1d28
404 nicely if you try to interact with a missing current version
2018-08-12 19:14:31 -04:00
edc427a351
flake8
2018-08-12 19:14:31 -04:00
fe87890b18
implement remaining tests and make them work
2018-08-12 19:14:31 -04:00
15d513f16f
fix idiocies and so make tests pass
2018-08-12 19:14:31 -04:00
b5eee511c7
don't needlessly return user_id
2018-08-12 19:14:31 -04:00
982edca380
fix flakes
2018-08-12 19:14:31 -04:00
234611f347
fix typos
2018-08-12 19:14:31 -04:00
9f0791b7bd
add a tonne of docstring; make upload_room_keys properly assert version
2018-08-12 19:14:31 -04:00
8d14598e90
add storage docstring; remove unused set_e2e_room_keys
2018-08-12 19:14:31 -04:00
cac0253799
rename room_key_version table correctly, and fix opt args
2018-08-12 19:14:31 -04:00
0abb205b47
blindly incorporate PR review - needs testing & fixing
2018-08-12 19:14:31 -04:00
69e51c7ba4
make /room_keys/version work
2018-08-12 19:14:31 -04:00
8ae64b270f
implement /room_keys/version too (untested)
2018-08-12 19:14:31 -04:00
6b8c07abc2
make it work and fix pep8
2018-08-12 19:13:09 -04:00
0bc4627a73
interim WIP checkin; doesn't build yet
2018-08-12 18:23:10 -04:00
53ace904b2
total WIP skeleton for /room_keys API
2018-08-12 18:23:10 -04:00
31fa743567
fix sqlite/postgres incompatibility in reap_monthly_active_users
2018-08-11 22:38:34 +01:00
b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
885ea9c602
rename _user_last_seen_monthly_active
2018-08-09 18:02:12 +01:00
ce6db0e547
Choose state algorithm based on room version
2018-08-09 14:58:47 +01:00
484a0ebdfc
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-09 10:16:29 +01:00
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
...
Support for room versioning
2018-08-08 17:10:53 +01:00
be59910b93
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users
2018-08-08 13:45:21 +01:00
990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
...
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
312ae74746
typos
2018-08-08 13:33:16 +01:00
ef3589063a
prevent total number of reserved users being too large
2018-08-07 17:59:27 +01:00
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
cd9765805e
Allow ratelimiting on workers
2018-08-07 10:50:28 +01:00
495cb100d1
Allow profile changes to happen on workers
2018-08-07 10:50:26 +01:00
ca9bc1f4fe
Fix occasional glitches in the synapse_event_persisted_position metric
...
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
a74b25faaa
WIP building out mau reserved users
2018-08-06 23:25:25 +01:00
e54794f5b6
Fix postgres compatibility bug
2018-08-06 21:55:54 +01:00
16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
96a9a29645
Pull in necessary stores in federation_reader
2018-08-06 15:23:57 +01:00
62ace05c45
Move necessary storage functions to worker classes
2018-08-06 15:23:38 +01:00
886be75ad1
bug fixes
2018-08-03 22:29:03 +01:00
e10830e976
wip commit - tests failing
2018-08-03 17:55:50 +01:00
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
897c51d274
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-03 13:40:47 +01:00
5e2f7b8084
typo
2018-08-02 22:47:48 +01:00
4ecb4bdac9
wip attempt at caching
2018-08-02 22:41:05 +01:00
74b1d46ad9
do mau checks based on monthly_active_users table
2018-08-02 16:57:35 +01:00
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
14fa9d4d92
Avoid extra db lookups
...
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
4a6725d9d1
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-02 11:04:18 +01:00
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
...
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
08281fe6b7
self.db_conn unused
2018-08-01 23:26:24 +01:00
c21d82bab3
normalise reaping query
2018-08-01 23:24:38 +01:00
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
...
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
7ff44d9215
improve clarity
2018-08-01 16:17:00 +01:00
da7785147d
Python 3: Convert some unicode/bytes uses ( #3569 )
2018-08-02 00:54:06 +10:00
303f1c851f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits
2018-08-01 13:42:50 +01:00
a6d7b74915
update docs
2018-08-01 13:39:14 +01:00
4e5ac901dd
clean up
2018-08-01 12:03:57 +01:00
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
6023cdd227
remove errant print
2018-08-01 10:27:17 +01:00
7931393495
make count_monthly_users async synapse/handlers/auth.py
2018-08-01 10:21:56 +01:00
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
...
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
0bc9b9e397
reinstate explicit include of EventsWorkerStore
2018-07-31 13:11:04 +01:00
fef7e58ac6
actually close conn
2018-07-30 22:29:44 +01:00
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
251e6c1210
limit register and sign in on number of monthly users
2018-07-30 15:55:57 +01:00
143f1a2532
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store
2018-07-30 09:56:18 +01:00
e9b2d047f6
make /context lazyload & filter aware ( #3567 )
...
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
...
Stop populating events.content
2018-07-26 22:54:08 +01:00
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
...
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
...
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
bd4b25f4d0
Remove some redundant joins on event_edges.room_id
...
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.
event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
21e878ebb6
Make EventStore inherit from EventFederationStore
...
(since it uses methods therein)
Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
bc7944e6d2
switch missing_types to be a bool
2018-07-25 23:36:31 +01:00
2565804030
Merge branch 'develop' into matthew/filter_members
2018-07-25 17:27:49 +01:00
7d9fb88617
incorporate more review.
2018-07-25 16:33:50 +01:00
78a691d005
Split out DB writes in federation handler
...
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
...
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
...
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
...
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
...
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
454f59b7ad
Merge branch 'develop' into matthew/filter_members
2018-07-24 14:03:37 +01:00
223341205e
Don't require to_delete to have event_ids
2018-07-24 14:02:40 +01:00
e22700c3dd
consider non-filter_type types as wildcards, thus missing from the state-group-cache
2018-07-24 13:59:07 +01:00
d19fba3655
Merge branch 'develop' into matthew/filter_members
2018-07-24 12:39:54 +01:00
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
30bfed5aa5
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-24 11:46:09 +01:00
97acd385a3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta
2018-07-24 11:32:13 +01:00
0fa73e4a63
Remove unnecessary if
2018-07-24 11:19:23 +01:00
ff5426f6b8
Speed up _calculate_state_delta
2018-07-24 10:55:11 +01:00
a678145010
Merge branch 'develop' into rav/logcontext_fixes
2018-07-24 10:43:30 +01:00
Amber Brown