Skip to content

fix(scanner): bound scans by file count; refuse the dependency scan above it and say so - #216

Merged
JordanCoin merged 4 commits into
mainfrom
fix/186-scan-bounds
Oct 11, 2026
Merged

JordanCoin merged 4 commits into
mainfrom
fix/186-scan-bounds

Conversation

@JordanCoin

@JordanCoin JordanCoin commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Measurements (macOS, /usr/bin/time -l, synthetic TypeScript repos, one import per file, default bound 100,000)

repo command before: wall before: max RSS after: wall after: max RSS
60k files codemap . 1.42 s 53 MB 0.37 s 51 MB
60k files codemap --deps 5.34 s 421 MB 5.02 s 421 MB (under the bound, runs as before)
60k files codemap --importers f0.ts 7.26 s 422 MB 4.33 s 424 MB
200k files codemap . 4.25 s 127 MB 4.13 s 70 MB (keeps 100k, counts 200k)
200k files codemap --deps 29.57 s 1,565 MB 3.07 s 113 MB (refused)
200k files codemap --importers f0.ts 14.52 s 1,634 MB 1.74 s 113 MB (refused)

Memory grows linearly with file count, about 8 MB per 1,000 files on the dependency path, which is how two concurrent scans of several large repos exhausted the machine in the report. Before this PR nothing refused, warned, or degraded. The default is 100,000 files: on these numbers that is roughly 800 MB and 10 s, the point where a default should stop and ask; 50,000 would have turned away repos a laptop handles in 5 s.

What changes

  • Walk bound. ScanFilesBounded keeps at most limits.DefaultMaxScanFiles (100,000) files and keeps counting, so the tree prints the whole count. ScanFiles is unchanged for callers.
  • Dependency scan refused above the bound. ast-grep walks the tree itself and its full JSON is held in memory, so the only place to bound it is before it starts. scanForGraphOutcomeWithFilters, which every graph build routes through (ScanForDeps, BuildFileGraph), counts first and above the bound records a scan-bound source (status unavailable) and runs nothing. --importers, blast-radius, find, hooks and MCP read the same coverage.
  • Honest wording. --importers says Importers of X: unknown, no dependency scan ran. rather than No files import X.; --deps says Dependency scan not run: repository is above the file bound. rather than No source files found.; MCP get_importers gets the same unknown wording.
  • Raise it. --max-files N per run, max_files in .codemap/config.json per repo; -1 removes the bound. README documents both.

Exact coverage lines on the 200k repo:

Coverage: partial — bounded: scanned 100000 of 200001 files (limit 100000; raise with --max-files or config max_files)
Coverage: unavailable — bounded: scanned 0 of 200001 files (limit 100000; raise with --max-files or config max_files)

JSON: Project.scan_bound {scanned, total, limit} (omitted when not bounded).

Tests

  • TestScanFilesBoundedStopsAtLimit, TestScanForDepsRefusesAboveBound (scanner/scan_bound_test.go): explicit bounds of 3 and 2 over tiny fixtures, never the default; bound of 2 over 3 TS files produces no analyses, coverage unavailable with the exact note, the graph has no edges; at the bound nothing changes.
  • TestImportersReportUnavailableSaysUnknown (scan_bound_cli_test.go): no "No files import" when nothing was scanned.
  • go build ./... && go test ./... && gofmt -l . clean.

Not changed

No edge cap (edges are bounded once files are), no runtime.MemStats sampler (the bound prevents the allocation; a sampler would only report after the fact), no wall-clock scan timeout (the existing ast-grep timeout and hook budget remain), no refusal to start the watch daemon (it inherits the bounded file list through ScanConfiguredFilesWithFilters).

Fixes #186

🤖 Generated with Claude Code

https://claude.ai/code/session_01NRaBgKRAZMX9hjbkkefA7T

…bove it and say so

Measured on 2026-10-11 with synthetic TypeScript repos (one import per file):
60k files, --deps 5.3 s and 421 MB RSS; 200k files, --deps 29.6 s and
1.57 GB RSS, --importers 1.63 GB. Growth is linear in files, so two
concurrent scans of large repos exhaust a laptop and codemap never said
anything (#186).

The walk now keeps at most limits.DefaultMaxScanFiles (50000) files and
counts the rest, so the tree reports "bounded: scanned N of M files".
ast-grep walks the tree itself and its whole JSON output is held in
memory, so the dependency scan is bounded before it starts: every graph
build routes through scanForGraphOutcomeWithFilters, which counts first
and above the bound records a scan-bound source with status unavailable
and the same note, and runs nothing. --importers then says the answer is
unknown instead of "No files import", and --deps says the scan did not
run instead of "No source files found". --max-files and config max_files
raise the bound; -1 removes it. Hooks and the watch daemon read the same
filters, so they degrade to the bounded result inside the 8 s budget.

Not added: an edge cap (edges are bounded by files once files are) and
a memory sampler (the bound is what prevents the allocation).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NRaBgKRAZMX9hjbkkefA7T
Copilot AI balanced review requested due to automatic review settings October 11, 2026 20:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JordanCoin
JordanCoin merged commit ee8c538 into main Oct 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

guard rails: no bound on files, graph size, or memory; a scan of a very large repo can exhaust the machine

2 participants