f03c9d3444
Don't apply retention policy based filtering on state events
...
As per MSC1763, 'Retention is only considered for non-state events.', so don't filter out state events based on the room's retention policy.
2019-11-06 18:40:04 +00:00
eda14737cf
Also filter state events
2019-11-06 18:14:03 +00:00
a6ebef1bfd
Make numeric user_id checker start at @0, and don't ratelimit on checking ( #6338 )
2019-11-06 17:21:20 +00:00
5c3363233c
Fix deleting state groups during room purge.
...
And fix the tests to actually test that things got deleted.
2019-11-06 17:02:08 +00:00
71f3bd734f
Use correct type annotation
2019-11-06 17:00:18 +00:00
55bc8d531e
raise exception after multiple failures
2019-11-06 16:52:54 +00:00
1fe3cc2c9c
Address review comments
2019-11-06 14:54:24 +00:00
915903eada
Merge branch 'master' into develop
2019-11-06 13:51:11 +00:00
4059d61e26
Don't forget to ratelimit calls outside of RegistrationHandler
2019-11-06 12:01:54 +00:00
b33c4f7a82
Numeric ID checker now checks @0, don't ratelimit on checking
2019-11-06 11:55:00 +00:00
f697b4b4a2
Add failed auth ratelimiting to UIA
2019-11-06 11:08:58 +00:00
541f1b92d9
Only do `rc_login` ratelimiting on succesful login.
...
We were doing this in a number of places which meant that some login
code paths incremented the counter multiple times.
It was also applying ratelimiting to UIA endpoints, which was probably
not intentional.
In particular, some custom auth modules were calling
`check_user_exists`, which incremented the counters, meaning that people
would fail to login sometimes.
2019-11-06 11:08:58 +00:00
24a214bd1b
Fix field name
2019-11-06 11:04:19 +00:00
70d93cafdb
Update insert
2019-11-06 10:59:03 +00:00
feafd98aca
1.5.1
2019-11-06 10:02:23 +00:00
807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state ( #6320 )
...
Fixes a bug where rejected events were persisted with the wrong state group.
Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.
Fixes #6289 .
2019-11-06 10:01:39 +00:00
0e3ab8afdc
Add some checks that we aren't using state from rejected events ( #6330 )
...
* Raise an exception if accessing state for rejected events
Add some sanity checks on accessing state_group etc for
rejected events.
* Skip calculating push actions for rejected events
It didn't actually cause any bugs, because rejected events get filtered out at
various later points, but there's not point in trying to calculate the push
actions for a rejected event.
2019-11-05 22:13:37 +00:00
01ba7b38a7
Merge pull request #6336 from matrix-org/erikj/fix_phone_home_stats
...
Fix phone home stats
2019-11-05 18:29:57 +00:00
052513958d
Fix phone home stats
2019-11-05 17:44:09 +00:00
5570d1c93f
Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2
...
Fix exception when OpenGraph tag values are ints
2019-11-05 17:28:11 +00:00
81d49cbb07
Fix exception when OpenGraph tag values are ints
2019-11-05 17:22:58 +00:00
55a7da247a
Merge branch 'develop' into rav/url_preview_limit_title
2019-11-05 17:08:07 +00:00
e78167c94b
Apply suggestions from code review
...
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-11-05 16:46:39 +00:00
e9bfe719ba
Strip overlong OpenGraph data from url preview
...
... to stop people causing DoSes with malicious web pages
2019-11-05 15:51:18 +00:00
c16e192e2f
Fix caching devices for remote servers in worker.
...
When the `/keys/query` API is hit on client_reader worker Synapse may
decide that it needs to resync some remote deivces. Usually this happens
on master, and then gets cached. However, that fails on workers and so
it falls back to fetching devices from remotes directly, which may in
turn fail if the remote is down.
2019-11-05 15:49:43 +00:00
cb2cbe4d26
Only filter if a filter was provided
2019-11-05 15:28:36 +00:00
f141af4c79
Update copyright
2019-11-05 14:52:38 +00:00
8822b33111
Update copyrights
2019-11-05 14:46:35 +00:00
4086002827
Improve documentation for EventContext fields ( #6319 )
2019-11-05 13:23:25 +00:00
1dffa78701
Filter events_before and events_after in /context requests
...
While the current version of the spec doesn't say much about how this endpoint uses filters (see https://github.com/matrix-org/matrix-doc/issues/2338 ), the current implementation is that some fields of an EventFilter apply (the ones that are used when running the SQL query) and others don't (the ones that are used by the filter itself) because we don't call event_filter.filter(...). This seems counter-intuitive and probably not what we want so this commit fixes it.
2019-11-05 12:21:59 +00:00
ffe595381d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history
2019-11-05 10:27:41 +00:00
506a63de67
Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_groups
2019-11-04 18:22:41 +00:00
c2203bea57
Re-add docstring, with caveats detailed
2019-11-04 18:17:11 +00:00
e252ffadbc
Merge branch 'develop' into babolivier/msc2326_bg_update
2019-11-04 18:09:50 +00:00
0287d033ee
Transfer upgraded rooms on groups
2019-11-04 18:08:50 +00:00
4e1c7b79fa
Remove the psutil dependency ( #6318 )
...
* remove psutil and replace with resource
2019-11-05 05:05:48 +11:00
09957ce0e4
Implement per-room message retention policies
2019-11-04 17:09:22 +00:00
7134ca7daa
Change to not require a state_groups.room_id index.
...
This does mean that we won't clean up orphaned state groups (i.e. state
groups that were persisted but the associated event wasn't).
2019-11-04 13:36:57 +00:00
6a0092d371
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history
2019-11-04 13:29:35 +00:00
cc6243b4c0
document the REPLICATE command a bit better ( #6305 )
...
since I found myself wonder how it works
2019-11-04 12:40:18 +00:00
3b29a73f9f
Print out the actual number of affected rows
2019-11-04 09:56:11 +00:00
824bba2f78
Correctly order results
2019-11-04 09:56:11 +00:00
49008e674f
TODO
2019-11-04 09:56:11 +00:00
1586f2c7e7
Fix exit condition
2019-11-04 09:56:11 +00:00
1c1268245d
Lint
2019-11-04 09:56:11 +00:00
911b03ca31
Don't try to process events we already have a label for
2019-11-04 09:56:10 +00:00
07cb38e965
Use a sensible default value for labels
2019-11-04 09:56:10 +00:00
a46574281d
Use the right format for rows
2019-11-04 09:56:10 +00:00
c9a1b80a74
MSC2326: Add background update to take previous events into account
2019-11-04 09:56:04 +00:00
f496d25877
Merge pull request #6301 from matrix-org/babolivier/msc2326
...
Implement MSC2326 (label based filtering)
2019-11-01 17:04:45 +00:00
988d8d6507
Incorporate review
2019-11-01 16:22:44 +00:00
c6516adbe0
Factor out an _AsyncEventContextImpl ( #6298 )
...
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
2019-11-01 16:19:09 +00:00
5598445655
Update synapse/storage/data_stores/main/schema/delta/56/event_labels.sql
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-11-01 16:18:34 +00:00
fa7e52caf1
Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schema
...
fix hidden field in devices table for older sqlite
2019-11-01 10:52:46 -04:00
1cb84c6486
Support for routing outbound HTTP requests via a proxy ( #6239 )
...
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.
The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.
The proxy will then be used for
* push
* url previews
* phone-home stats
* recaptcha validation
* CAS auth validation
It will *not* be used for:
* Application Services
* Identity servers
* Outbound federation
* In worker configurations, connections from workers to masters
Fixes #4198 .
2019-11-01 14:07:44 +00:00
fe1f2b4520
Remove last usages of deprecated logging.warn method ( #6314 )
2019-11-01 12:03:44 +00:00
a2c63c619a
Add more data to the event_labels table and fix the indexes
2019-11-01 11:47:28 +00:00
669b6cbda3
Fix up comment
2019-11-01 11:32:20 +00:00
e3689ac6f7
Add unstable feature flag
2019-11-01 10:41:23 +00:00
57cdb046e4
Lint
2019-11-01 10:39:14 +00:00
c6dbca2422
Incorporate review
2019-11-01 10:30:51 +00:00
ace947e8da
Depublish a room from the public rooms list when it is upgraded ( #6232 )
2019-11-01 10:28:09 +00:00
c61db13183
fix hidden field in devices table for older sqlite
2019-10-31 22:52:55 -04:00
c3fc176c60
Update synapse/storage/data_stores/main/devices.py
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 22:49:48 -04:00
6f4bc6d01d
Merge branch 'develop' into cross-signing_federation
2019-10-31 22:38:21 -04:00
3b4216f961
Merge pull request #6254 from matrix-org/uhoreg/cross_signing_fix_workers_notify
...
make notification of signatures work with workers
2019-10-31 22:35:03 -04:00
42e707c663
rstrip slashes from url on appservice ( #6306 )
2019-10-31 17:32:25 +00:00
9c94b48bf1
Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify
2019-10-31 12:32:07 -04:00
f7e4a582ef
clean up code a bit
2019-10-31 12:01:00 -04:00
fb1a6914cf
Update log line to lie a little less
2019-10-31 15:45:48 +00:00
020add5099
Update black to 19.10b0 ( #6304 )
...
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
61be1a2926
Add state_groups.room_id index
2019-10-31 15:39:26 +00:00
f91f2a1f92
Docstrings
2019-10-31 15:26:00 +00:00
8f5bbdb987
Fix purge room API
2019-10-31 15:22:08 +00:00
cd581338cf
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history
2019-10-31 15:19:26 +00:00
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
...
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
3a74c03ffb
Expose some homeserver functionality to spam checkers ( #6259 )
...
* Offer the homeserver instance to the spam checker
* Newsfile
* Linting
* Expose a Spam Checker API instead of passing the homeserver object
* Alter changelog
* s/hs/api
2019-10-31 09:16:14 -06:00
64f2b8c3d8
Apply suggestions from code review
...
Fix docstring
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 15:44:31 +01:00
c6bcd38841
Fix /purge_room API.
...
It fails trying to clean the `topic` table which was recently removed.
2019-10-31 11:17:23 +00:00
54fef094b3
Remove usage of deprecated logger.warn method from codebase ( #6271 )
...
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
998f7fe7d4
make user signatures a separate stream
2019-10-30 17:22:52 -04:00
670972c0e1
Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify
2019-10-30 16:46:31 -04:00
bb6cec27a5
rename get_devices_by_remote to get_device_updates_by_remote
2019-10-30 14:57:34 -04:00
0467f33584
fix delete_existing for _persist_events ( #6300 )
...
this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
2019-10-30 18:05:00 +00:00
dcc069a2e2
Lint
2019-10-30 18:01:56 +00:00
fe51d6cacf
Add more integration testing
2019-10-30 17:28:41 +00:00
e7943f660a
Add unit tests
2019-10-30 16:15:04 +00:00
233b14ebe1
Add index on label
2019-10-30 15:58:05 +00:00
acd16ad86a
Implement filtering
2019-10-30 15:56:33 +00:00
7c8c97e635
Split purge API into events vs state
2019-10-30 15:23:37 +00:00
69f0054ce6
Port to use state storage
2019-10-30 14:46:54 +00:00
5db03535d5
Add StateGroupStorage interface
2019-10-30 14:46:49 +00:00
fa0dcbc8fa
Store labels for new events
2019-10-30 14:27:15 +00:00
7d7eac61be
Merge branch 'develop' into cross-signing_federation
2019-10-30 10:17:10 -04:00
bc32f102cd
black
2019-10-30 10:07:36 -04:00
d78b1e339d
apply changes as a result of PR review
2019-10-30 10:01:53 -04:00
ec6de1cc7d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store
2019-10-30 13:37:04 +00:00
a8d16f6c00
Review comments
2019-10-30 13:36:12 +00:00
e5c3a99091
Merge pull request #6291 from matrix-org/erikj/fix_cache_descriptor
...
Make ObservableDeferred.observe() always return deferred.
2019-10-30 14:06:34 +01:00
6e677403b7
Clarify docstring
2019-10-30 11:52:04 +00:00
326b3dace7
Make ObservableDeferred.observe() always return deferred.
...
This makes it easier to use in an async/await world.
Also fixes a bug where cache descriptors would occaisonally return a raw
value rather than a deferred.
2019-10-30 11:35:46 +00:00
a2276d4d3c
Fix log line that was printing undefined value ( #6278 )
2019-10-30 11:28:48 +00:00
7955abeaac
Fix small typo in comment ( #6269 )
2019-10-30 11:16:19 +00:00
46c12918ad
Fix typo in domain name in account_threepid_delegates config option ( #6273 )
2019-10-30 11:07:42 +00:00
b39ca49db1
Handle FileNotFound error in checking git repository version ( #6284 )
2019-10-30 11:00:15 +00:00
770d1ef673
Merge pull request #6280 from matrix-org/erikj/receipts_async_await
...
Port receipt and read markers to async/wait
2019-10-30 11:44:18 +01:00
ba4cc5541c
Merge pull request #6274 from matrix-org/erikj/replication_async
...
Port replication http server endpoints to async/await
2019-10-30 11:44:08 +01:00
72bc6294ed
Merge pull request #6275 from matrix-org/erikj/port_rest_events
...
Port room rest handlers to async/await
2019-10-30 11:44:02 +01:00
47f767269c
Add database table for keeping track of labels on events
2019-10-29 16:56:22 +00:00
a287f1e804
Don't return coroutines
2019-10-29 16:36:46 +00:00
38474707b9
Merge branch 'erikj/federation_server_async_await' of github.com:matrix-org/synapse into erikj/receipts_async_await
2019-10-29 15:53:17 +00:00
74c1e16106
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/federation_server_async_await
2019-10-29 15:52:39 +00:00
307e313ef4
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/port_rest_events
2019-10-29 15:51:12 +00:00
d6e40e7cbd
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_async
2019-10-29 15:42:58 +00:00
2c35ffead2
Port receipt and read markers to async/wait
2019-10-29 15:08:22 +00:00
09a135b039
Make concurrently_execute work with async/await
2019-10-29 15:02:23 +00:00
65cb307e19
Merge branch 'master' into develop
2019-10-29 14:40:57 +00:00
9ffcf0f7ba
1.5.0
2019-10-29 14:28:54 +00:00
3f33879be4
Port federation_server to async/await
2019-10-29 14:13:08 +00:00
9be41bc121
Port room rest handlers to async/await
2019-10-29 13:09:29 +00:00
e577a4b2ad
Port replication http server endpoints to async/await
2019-10-29 13:00:51 +00:00
561133c3c5
Merge pull request #6263 from matrix-org/erikj/caches_return_deferreds
...
Quick fix to ensure cache descriptors always return deferreds
2019-10-29 12:53:21 +01:00
e6c7e239ef
Update docstring
2019-10-29 11:48:30 +00:00
e419c44ba4
Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop
2019-10-29 11:41:27 +00:00
342d871d76
Make room directory search case insensitive
2019-10-29 10:24:02 +00:00
14504ad573
Add CI for synapse_port_db ( #6140 )
...
This adds:
* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)
as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:
1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)
Step `2` is done via a new script located at `scripts-dev/update_database`.
The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
d0d8a22c13
Quick fix to ensure cache descriptors always return deferreds
2019-10-28 13:33:04 +00:00
bcfc647e4d
Synapse 1.5.0rc2 (2019-10-28)
...
=============================
Bugfixes
--------
- Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247 ))
- Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256 ))
- Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262 ))
Internal Changes
----------------
- Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248 ))
- Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255 ))
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl225VkACgkQOSor00I9
eP/7wwf6AkQE1dGYDPpmS29XYocfD3EbjT8FojaHTz1gbkIt5rEF88QHzwRo4P/x
1S1kQZpEKxKrwmLMDmSbw9STK/76LKGk9k8VycVHfTdy8SICCiL50pVPpSTJ4nXV
905E7nY0zau3wunWJD4gEjqll967W0AtNKwCHO0D6vNP4pcQNhTkSjhnmypXCFGo
bvV9r0wq8mtq/FS1GX0Y2CEwbbV1MoPrvlXW37t/gBPk3P1QB/vPjakxO8LgeuZf
JzAamP5jOYZJpPCpZxwDm6ZNEQ7T4jORdA623wp1gh1nTG88cObCLXJe2jCnpjnJ
QVb1YlJiwcehXkclIoywgIzj13iIsw==
=7ctT
-----END PGP SIGNATURE-----
Merge tag 'v1.5.0rc2' into develop
Synapse 1.5.0rc2 (2019-10-28)
=============================
Bugfixes
--------
- Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247 ))
- Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256 ))
- Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262 ))
Internal Changes
----------------
- Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248 ))
- Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255 ))
2019-10-28 12:59:13 +00:00
c482d45822
1.5.0rc2
2019-10-28 12:48:18 +00:00
172f264ed3
Improve signature checking on some federation APIs ( #6262 )
...
Make sure that we check that events sent over /send_join, /send_leave, and
/invite, are correctly signed and come from the expected servers.
2019-10-28 12:43:23 +00:00
9aee28927b
Convert EventContext to attrs ( #6218 )
...
* make EventContext use an attr
2019-10-28 14:29:55 +02:00
da78f61778
Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys
...
delete keys when deleting backup versions
2019-10-25 11:28:11 -04:00
64c2cfda8a
Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop
2019-10-25 11:34:49 +01:00
87259b3a3a
Merge pull request #6255 from matrix-org/erikj/stats_config
...
Remove repeated calls to config.stats_enabled.
2019-10-25 11:34:38 +01:00
a71b8c87ec
Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop
2019-10-25 11:32:24 +01:00
a411f2b177
Fix /keys/query API on workers.
...
The necessary getters were added only to the master store and not the
worker stores.
2019-10-25 11:08:03 +01:00
f666d6f5d7
Remove repeated calls to config.stats_enabled.
...
Turns out that fetching variables from the config object is expensive,
so doing it once at startup avoids unnecessary work.
2019-10-25 10:28:36 +01:00
44ab048cfe
Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_logging
...
Reduce debug logging overhead
2019-10-25 10:05:44 +01:00
c40d7244f8
Merge branch 'develop' into cross-signing_federation
2019-10-24 22:31:25 -04:00
8ac766c44a
make notification of signatures work with workers
2019-10-24 22:14:58 -04:00
ff05c9b760
don't error if federation query doesn't have cross-signing keys
2019-10-24 21:46:11 -04:00
848cd388d9
delete keys when deleting backups
2019-10-24 21:21:51 -04:00
e4d98188da
Address codestyle concerns
2019-10-24 18:43:13 +01:00
8f4a808d9d
Delay printf until logging is required.
...
Using % will cause the string to be generated even if debugging
is off.
2019-10-24 18:31:53 +01:00
9eebc1e73b
use %r to __repr__ objects
...
This avoids calculating __repr__ unless we are going to log.
2019-10-24 18:18:56 +01:00
f85b9842f0
Don't encode object as UTF-8 string if not needed.
...
I believe that string formatting ~10-15 sized events will
take a proportion of CPU time.
2019-10-24 18:08:45 +01:00
39266a9c9f
Make user/room stats log line less verbose.
2019-10-24 17:55:53 +01:00
3aa2a90556
Move schema delta files to the correct data store.
...
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
8831b04a53
1.5.0rc1
2019-10-24 12:47:58 +01:00
2794b79052
Option to suppress resource exceeded alerting ( #6173 )
...
The expected use case is to suppress MAU limiting on small instances
2019-10-24 11:48:46 +01:00
Brendan Abolivier