Repository navigation
Conversation
Root cause: each server loads persist.dat once and then serves its queues from memory, and the file lock was only held around a single append. A second server on the same directory therefore delivered the same item a second time, and the first server's shutdown snapshot (written from its own memory) erased items the second server had acknowledged. The server now takes an exclusive lock on persist.lock before loading the log and holds it until exit. A second server logs that it is waiting and only loads and listens once the first has exited, so a rolling replace hands over the old server's final snapshot. The lock lives on a separate file because persist.dat is replaced by rename, and the OS releases it if the process dies. Fixes #173 Co-Authored-By: Claude Opus 5.5 <[email protected]>
Owner
Author
|
Note for reviewers: this is the third open PR for #173, alongside #178 and #179. Those PRs refuse to start a second server on the directory (a non-blocking The Stryker job here was cancelled at the 45-minute limit. That job is a full run (any 🤖 Generated with Claude Code |
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #173.
Root cause
Each server loads
persist.datonce at startup and then serves its queues from memory. The file lock was held only around a single append, so nothing stopped a second--persistserver on the same directory. The two servers delivered the same item twice. Then the first server's shutdown snapshot, written from its own memory, overwrotepersist.datand erased items the second server had acknowledged.Fix
FileStore.acquireOwnership()takes an exclusive lock on<PERSIST>/persist.lock, andclose()releases it.main.tstakes the lock before loading the log.Waiting for another server to release <dir>...and does not load or listen until the first exits. A rolling replace (start new, stop old) therefore hands over the old server's final snapshot.replace()renames overpersist.dat, which would drop a lock held on it. The OS releases the lock when the process dies, so a crash cannot leave it stale. A waiting server still exits straight away onSIGTERM.Verification
tests/shared_persist_dir_test.tsruns the issue's scenario with two realmain.ts --persistprocesses. It was red before the fix (the second server listened and served) and is green after, in about 0.25s.FileStoreunit test: a second owner waits until the first callsclose()."from-a"delivered once, server B serves only after A exits, and B's acknowledged"from-b"survives aSIGKILLand a restart.docker stopon the first.deno test: 381 passed, 1 failed. The failure ismutation_skip_test.ts(StrykerJS is not installed locally; CI installs it).🤖 Generated with Claude Code