Skip to content

workdir: only read the overlay for the revision it describes - #141

Draft
12458 wants to merge 2 commits into
facebookexperimental:mainfrom
12458:workdir-overlay-gate
Draft

12458 wants to merge 2 commits into
facebookexperimental:mainfrom
12458:workdir-overlay-gate

Conversation

@12458

@12458 12458 commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Depends on #140. This branch is built on file-survey-reference-counts, so until #140 merges this PR also shows its commit. Review only the last commit (workdir: only read the overlay for the revision it describes).

Problem

DatabaseManager keeps one working-copy overlay per process. That's a parsed snapshot of the uncommitted edits in the checkout. It describes only the commit that's checked out, and only when the caller hasn't asked for git-only results. workdir_overlay_usable(git_sha) encodes that rule, and ensure_workdir_index_built uses it to decide whether to build or refresh the overlay.

The rule decides when the overlay gets built, but nothing checks it when the overlay is read. Each workdir_* helper calls ensure_workdir_index_built and then reads self.workdir_index regardless of the result. If the overlay isn't usable, ensure_workdir_index_built returns early without clearing it, so whatever an earlier query built is still there. Three things read the overlay this way:

  • the workdir_* lookup helpers: function and type lookup, callers, callees, grep, regex, and the deleted/dirty-count checks;
  • get_distinct_reference_counts_git_aware, the working-copy half of file_survey's reference counts;
  • generate_git_manifest, which merged the overlay's dirty and deleted paths into every manifest, for any commit.

A query at the checked-out commit fills the overlay. Later in the same process, the overlay is read for something it doesn't describe:

  • a query naming another commit or branch, which is common over MCP;
  • the same commit after HEAD has moved;
  • any query after set_git_only(true).

That query then mixes in edits made on top of a different revision. Indexed rows in files that are modified in the working copy get dropped, and functions that exist only in the working copy get added.

Example: HEAD is A, and a.c has an uncommitted edit adding a call to overlay_only(). A file_survey at A builds the overlay. A git-only query at A, or a query at A after HEAD has moved past it, still reports one caller of overlay_only and skips the indexed caller in a.c.

There's a related ordering bug in the reference counts: the manifest was generated before the overlay was refreshed for the current command. If a file was modified in an earlier REPL command and has since been restored, the manifest still carries its old working-copy hash, so its indexed callers were skipped.

Fix

  • ensure_workdir_index_built now returns whether the overlay may answer for git_sha.
  • Every overlay read goes through a new with_workdir(git_sha, f) helper, which returns an empty result when it may not, so the check and the read can't get out of step again.
  • generate_git_manifest checks workdir_overlay_usable before merging the overlay's paths. It still doesn't build the overlay itself, so generating a manifest doesn't walk the tree.
  • get_distinct_reference_counts_git_aware refreshes the overlay before generating the manifest, and reads the overlay whenever there are targets, not only when the index is scanned.

Testing

an_overlay_built_at_head_does_not_answer_for_another_revision builds the overlay at HEAD, then checks three cases:

  • a file restored in a later command;
  • the same commit in git-only mode;
  • the same commit after HEAD has moved on.

Each case fails without the corresponding change.

cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test all pass.

12458 and others added 2 commits November 19, 2026 23:04
Reference counting checks whether each relationship row comes from a
dirty or deleted file. Both checks can rediscover the repository and
read HEAD, so a kernel-scale survey repeatedly performs the same work.

Snapshot the excluded paths and overlay counts together before
scanning. Stream the LanceDB batches so the query can consume each
batch as it arrives. Exercise both clean and edited worktrees in the
reference-count test.

Co-authored-by: Chris J Arges <[email protected]>
The working-copy overlay is one slot per process and only describes the
checked-out commit when git-only mode is off. ensure_workdir_index_built
already declines to build or refresh it otherwise, but every workdir_*
helper, the file_survey reference counts and generate_git_manifest read
the slot regardless. An overlay filled by an earlier query at HEAD was
therefore applied to queries naming another commit, to the same commit
after HEAD moved, and to git-only queries: indexed rows in files modified
in the working copy were dropped and working-copy-only functions added.

ensure_workdir_index_built now reports whether the overlay may answer for
git_sha, and all reads go through with_workdir, which returns an empty
result when it may not. generate_git_manifest checks eligibility before
merging modified and deleted paths, still without building the overlay.

The reference counts also refresh the overlay before generating the
manifest. Previously a file modified in an earlier REPL command and since
restored kept its stale working-copy hash in the manifest, so its indexed
callers were skipped.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 20, 2026
@12458
12458 marked this pull request as draft November 20, 2026 05:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant