f112cfe5bb
On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case. |
||
|---|---|---|
| .. | ||
| api | ||
| app | ||
| appservice | ||
| config | ||
| crypto | ||
| events | ||
| federation | ||
| handlers | ||
| http | ||
| logging | ||
| module_api | ||
| push | ||
| replication | ||
| rest | ||
| scripts | ||
| server_notices | ||
| state | ||
| storage | ||
| test_utils | ||
| util | ||
| __init__.py | ||
| server.py | ||
| test_distributor.py | ||
| test_event_auth.py | ||
| test_federation.py | ||
| test_mau.py | ||
| test_metrics.py | ||
| test_phone_home.py | ||
| test_preview.py | ||
| test_server.py | ||
| test_state.py | ||
| test_terms_auth.py | ||
| test_test_utils.py | ||
| test_types.py | ||
| test_visibility.py | ||
| unittest.py | ||
| utils.py | ||