Skip to content

Hold PERSIST directory ownership for the life of the server (#173) - #180

Closed
jonbaldie wants to merge 1 commit into
mainfrom
fix/173-persist-dir-ownership
Closed

jonbaldie wants to merge 1 commit into
mainfrom
fix/173-persist-dir-ownership

Conversation

@jonbaldie

Copy link
Copy Markdown
Owner

Fixes #173.

Root cause

Each server loads persist.dat once at startup and then serves its queues from memory. The file lock was held only around a single append, so nothing stopped a second --persist server on the same directory. The two servers delivered the same item twice. Then the first server's shutdown snapshot, written from its own memory, overwrote persist.dat and erased items the second server had acknowledged.

Fix

  • FileStore.acquireOwnership() takes an exclusive lock on <PERSIST>/persist.lock, and close() releases it. main.ts takes the lock before loading the log.
  • A second server on the same directory logs Waiting for another server to release <dir>... and does not load or listen until the first exits. A rolling replace (start new, stop old) therefore hands over the old server's final snapshot.
  • The lock is on a separate file because replace() renames over persist.dat, which would drop a lock held on it. The OS releases the lock when the process dies, so a crash cannot leave it stale. A waiting server still exits straight away on SIGTERM.
  • README Persistency section documents the behaviour.

Verification

  • New tests/shared_persist_dir_test.ts runs the issue's scenario with two real main.ts --persist processes. It was red before the fix (the second server listened and served) and is green after, in about 0.25s.
  • New FileStore unit test: a second owner waits until the first calls close().
  • The issue's replay against a binary compiled with the Dockerfile flags: "from-a" delivered once, server B serves only after A exits, and B's acknowledged "from-b" survives a SIGKILL and a restart.
  • Two Docker containers sharing one named volume: the second waits, then takes over after docker stop on the first.
  • Full local deno test: 381 passed, 1 failed. The failure is mutation_skip_test.ts (StrykerJS is not installed locally; CI installs it).

🤖 Generated with Claude Code

Root cause: each server loads persist.dat once and then serves its queues
from memory, and the file lock was only held around a single append. A
second server on the same directory therefore delivered the same item a
second time, and the first server's shutdown snapshot (written from its own
memory) erased items the second server had acknowledged.

The server now takes an exclusive lock on persist.lock before loading the
log and holds it until exit. A second server logs that it is waiting and
only loads and listens once the first has exited, so a rolling replace hands
over the old server's final snapshot. The lock lives on a separate file
because persist.dat is replaced by rename, and the OS releases it if the
process dies.

Fixes #173

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@jonbaldie

Copy link
Copy Markdown
Owner Author

Note for reviewers: this is the third open PR for #173, alongside #178 and #179. Those PRs refuse to start a second server on the directory (a non-blocking tryLockSync, then exit 1). This one makes the second server wait for the lock and then take over, so a rolling replace (start new, stop old) hands over the old server's final snapshot instead of failing. Merge one design and close the others.

The Stryker job here was cancelled at the 45-minute limit. That job is a full run (any main.ts change triggers one), and full runs take 35–47 minutes. #179 raises timeout-minutes to 60 for the same reason. Test, Mutasaurus and Production quality pass.

🤖 Generated with Claude Code

@jonbaldie

Copy link
Copy Markdown
Owner Author

Closed in favor of merged #178 which resolved #173.

@jonbaldie jonbaldie closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A second server on the same PERSIST directory delivers one item twice and can erase acknowledged items

1 participant