Repository navigation
feat(du)!: scan S3 directory usage by default - #422
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the change to make rc du scan S3 by default.
No open issues 🎉
What changed in this PR
This PR makes rc du scan S3 objects by default instead of reading RustFS background-scanner totals. It introduces a breaking default-behavior change while preserving snapshot access through --snapshot and snapshot-first behavior through --fallback.
Changes:
- Adds explicit snapshot selection and routes default usage requests directly to S3.
- Enforces directory-prefix boundaries and excludes zero-byte directory markers from scanned object and version counts.
- Adds CLI regression coverage and updates help, compatibility guidance, and migration examples.
| File | Description |
|---|---|
README.md |
Updates usage examples and explains live-scan costs. |
docs/reference/rc/ops.md |
Documents new defaults, directory semantics, migration, and backend limitations. |
docs/reference/rc/mc-compatibility.md |
Clarifies default S3 listing and snapshot capability requirements. |
crates/s3/src/ops.rs |
Excludes zero-byte directory markers from object and version totals. |
crates/core/src/ops.rs |
Updates the usage-scan request doc comment. |
crates/cli/tests/help_contract.rs |
Requires --snapshot in command help. |
crates/cli/tests/du.rs |
Adds integration coverage for scans, directory boundaries, counts, and fallback behavior. |
crates/cli/src/commands/mod.rs |
Updates the top-level du help description. |
crates/cli/src/commands/du.rs |
Implements scan-first routing, explicit snapshots, and directory-target validation. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BREAKING change to the default behavior of
rc du. A plain invocation now lists current objects through S3 and sums their sizes. The previous default queried the RustFS background scanner, so recently uploaded or deleted objects could be missing from the result and an S3-only endpoint required an explicit fallback.rc du --snapshot ALIAS[/BUCKET]to retain the previous low-cost snapshot query. Existing--fallbackinvocations keep their snapshot-first selection. A live scan needs S3 list access and can take many requests on large buckets.bucket/logsandbucket/logs/select the directorylogs/. With the slash omitted, one bounded listing checks for a current directory entry. A lone object or a missing directory returns the existing not-found code (5). An explicit trailing slash permits an empty result.The command reference includes migration examples and a known MinIO limitation when listing incomplete multipart uploads by directory.
Verification
cargo fmt --all --checkcargo clippy --locked --workspace --all-targets -- -D warningscargo test --locked --workspacelogsandlogs/, excluded the 29-byte sibling, and reported zero objects for a marker-only directory. Both rejected an individual object and a missing unsuffixed path. After an overwrite, both reported 24 bytes across three versions, including a zero-byte file.