Repository navigation
fix(scanner): bound scans by file count; refuse the dependency scan above it and say so - #216
Merged
Merged
Conversation
…bove it and say so Measured on 2026-10-11 with synthetic TypeScript repos (one import per file): 60k files, --deps 5.3 s and 421 MB RSS; 200k files, --deps 29.6 s and 1.57 GB RSS, --importers 1.63 GB. Growth is linear in files, so two concurrent scans of large repos exhaust a laptop and codemap never said anything (#186). The walk now keeps at most limits.DefaultMaxScanFiles (50000) files and counts the rest, so the tree reports "bounded: scanned N of M files". ast-grep walks the tree itself and its whole JSON output is held in memory, so the dependency scan is bounded before it starts: every graph build routes through scanForGraphOutcomeWithFilters, which counts first and above the bound records a scan-bound source with status unavailable and the same note, and runs nothing. --importers then says the answer is unknown instead of "No files import", and --deps says the scan did not run instead of "No source files found". --max-files and config max_files raise the bound; -1 removes it. Hooks and the watch daemon read the same filters, so they degrade to the bounded result inside the 8 s budget. Not added: an edge cap (edges are bounded by files once files are) and a memory sampler (the bound is what prevents the allocation). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01NRaBgKRAZMX9hjbkkefA7T
…26-10-11 measurements) Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01NRaBgKRAZMX9hjbkkefA7T
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurements (macOS,
/usr/bin/time -l, synthetic TypeScript repos, one import per file, default bound 100,000)codemap .codemap --depscodemap --importers f0.tscodemap .codemap --depscodemap --importers f0.tsMemory grows linearly with file count, about 8 MB per 1,000 files on the dependency path, which is how two concurrent scans of several large repos exhausted the machine in the report. Before this PR nothing refused, warned, or degraded. The default is 100,000 files: on these numbers that is roughly 800 MB and 10 s, the point where a default should stop and ask; 50,000 would have turned away repos a laptop handles in 5 s.
What changes
ScanFilesBoundedkeeps at mostlimits.DefaultMaxScanFiles(100,000) files and keeps counting, so the tree prints the whole count.ScanFilesis unchanged for callers.scanForGraphOutcomeWithFilters, which every graph build routes through (ScanForDeps,BuildFileGraph), counts first and above the bound records ascan-boundsource (status unavailable) and runs nothing.--importers,blast-radius,find, hooks and MCP read the same coverage.--importerssaysImporters of X: unknown, no dependency scan ran.rather thanNo files import X.;--depssaysDependency scan not run: repository is above the file bound.rather thanNo source files found.; MCPget_importersgets the same unknown wording.--max-files Nper run,max_filesin.codemap/config.jsonper repo;-1removes the bound. README documents both.Exact coverage lines on the 200k repo:
JSON:
Project.scan_bound {scanned, total, limit}(omitted when not bounded).Tests
TestScanFilesBoundedStopsAtLimit,TestScanForDepsRefusesAboveBound(scanner/scan_bound_test.go): explicit bounds of 3 and 2 over tiny fixtures, never the default; bound of 2 over 3 TS files produces no analyses, coverage unavailable with the exact note, the graph has no edges; at the bound nothing changes.TestImportersReportUnavailableSaysUnknown(scan_bound_cli_test.go): no "No files import" when nothing was scanned.go build ./... && go test ./... && gofmt -l .clean.Not changed
No edge cap (edges are bounded once files are), no
runtime.MemStatssampler (the bound prevents the allocation; a sampler would only report after the fact), no wall-clock scan timeout (the existing ast-grep timeout and hook budget remain), no refusal to start the watch daemon (it inherits the bounded file list throughScanConfiguredFilesWithFilters).Fixes #186
🤖 Generated with Claude Code
https://claude.ai/code/session_01NRaBgKRAZMX9hjbkkefA7T