Skip to content

file_survey: snapshot the working-copy overlay once per request - #140

Open
12458 wants to merge 1 commit into
facebookexperimental:mainfrom
12458:file-survey-reference-counts
Open

12458 wants to merge 1 commit into
facebookexperimental:mainfrom
12458:file-survey-reference-counts

Conversation

@12458

@12458 12458 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

file_survey checks whether each indexed relationship came from a dirty or deleted file. Both checks can rediscover the Git repository and read HEAD for every row.

  • Snapshot the dirty and deleted paths and the overlay counts together once per request.
  • Stream LanceDB batches from both relationship tables.
  • Extend the reference-count test to cover modified and deleted files.

Benchmarks

  • Compared this commit (0f83543) with its parent on main (d233043), both release builds with Rust 1.98.1.
  • Fixture: single-ref full-tree Linux v7.3-rc4 index (93f51579, 1,060,300 functions, 1.1 GB), built with main's semcode-index.
  • Each build used its own copy of the database as a semcode-mcp stdio server. Each request was a file_survey call for drivers/net/wireguard/peer.c. The patched build ran first, so the baseline got the warmer page cache.
  • Clean worktree: 274.03 s on main, and 1.07 s, 0.71 s, 0.74 s with this PR.
  • One modified file: 274.62 s on main, and 1.01 s, 0.69 s, 0.70 s with this PR. The modification appended a caller of wg_peer_create and wg_peer_put to drivers/net/wireguard/device.c. Both builds raised those counts from 1 to 2 and from 21 to 22.
  • In both scenarios, the sorted functions_defined and types_defined matched the baseline exactly. The three patched responses were identical.
  • Host: 16-CPU Intel Core Ultra 7 265H, 30 GiB RAM, ext4.

Validation

  • cargo +stable test --locked --offline --quiet (621 passed)
  • cargo +stable fmt --check

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 20, 2026
@12458 12458 changed the title file_survey: avoid per-row working-copy lookups file_survey: snapshot the working-copy overlay once per request Nov 20, 2026
Reference counting checks whether each relationship row comes from a
dirty or deleted file. Both checks can rediscover the repository and
read HEAD, so a kernel-scale survey repeatedly performs the same work.

Snapshot the excluded paths and overlay counts together before
scanning. Stream the LanceDB batches so the query can consume each
batch as it arrives. Exercise both clean and edited worktrees in the
reference-count test.

Co-authored-by: Chris J Arges <[email protected]>
@12458
12458 force-pushed the file-survey-reference-counts branch from 0f83543 to f0ea480 Compare November 20, 2026 05:04
@12458
12458 marked this pull request as ready for review November 20, 2026 05:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant