Skip to content

feat(du)!: scan S3 directory usage by default - #422

Merged
cxymds merged 1 commit into
mainfrom
cxymds/fix/du-live-listing
Nov 18, 2026
Merged

cxymds merged 1 commit into
mainfrom
cxymds/fix/du-live-listing

Conversation

@cxymds

@cxymds cxymds commented Nov 18, 2026

Copy link
Copy Markdown
Member

Summary

BREAKING change to the default behavior of rc du. A plain invocation now lists current objects through S3 and sums their sizes. The previous default queried the RustFS background scanner, so recently uploaded or deleted objects could be missing from the result and an S3-only endpoint required an explicit fallback.

  • Use rc du --snapshot ALIAS[/BUCKET] to retain the previous low-cost snapshot query. Existing --fallback invocations keep their snapshot-first selection. A live scan needs S3 list access and can take many requests on large buckets.
  • Both bucket/logs and bucket/logs/ select the directory logs/. With the slash omitted, one bounded listing checks for a current directory entry. A lone object or a missing directory returns the existing not-found code (5). An explicit trailing slash permits an empty result.
  • Exclude zero-byte directory markers from scanned object and version counts. Ordinary zero-byte objects and nonzero payloads contribute normally. The v3 usage response continues to expose the source and separate object, version and delete-marker counts.

The command reference includes migration examples and a known MinIO limitation when listing incomplete multipart uploads by directory.

Verification

  • cargo fmt --all --check
  • cargo clippy --locked --workspace --all-targets -- -D warnings
  • cargo test --locked --workspace
  • Compared the built CLI with MinIO Client on local MinIO and RustFS servers using the same fixture. Both clients reported 17 bytes and two objects under logs and logs/, excluded the 29-byte sibling, and reported zero objects for a marker-only directory. Both rejected an individual object and a missing unsuffixed path. After an overwrite, both reported 24 bytes across three versions, including a zero-byte file.

Copilot AI balanced review requested due to automatic review settings November 18, 2026 02:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the change to make rc du scan S3 by default.

No open issues 🎉

What changed in this PR

This PR makes rc du scan S3 objects by default instead of reading RustFS background-scanner totals. It introduces a breaking default-behavior change while preserving snapshot access through --snapshot and snapshot-first behavior through --fallback.

Changes:

  • Adds explicit snapshot selection and routes default usage requests directly to S3.
  • Enforces directory-prefix boundaries and excludes zero-byte directory markers from scanned object and version counts.
  • Adds CLI regression coverage and updates help, compatibility guidance, and migration examples.
File Description
README.md Updates usage examples and explains live-scan costs.
docs/​reference/​rc/​ops.md Documents new defaults, directory semantics, migration, and backend limitations.
docs/​reference/​rc/​mc-compatibility.md Clarifies default S3 listing and snapshot capability requirements.
crates/​s3/​src/​ops.rs Excludes zero-byte directory markers from object and version totals.
crates/​core/​src/​ops.rs Updates the usage-scan request doc comment.
crates/​cli/​tests/​help_contract.rs Requires --snapshot in command help.
crates/​cli/​tests/​du.rs Adds integration coverage for scans, directory boundaries, counts, and fallback behavior.
crates/​cli/​src/​commands/​mod.rs Updates the top-level du help description.
crates/​cli/​src/​commands/​du.rs Implements scan-first routing, explicit snapshots, and directory-target validation.

Review effort: Balanced

@cxymds
cxymds merged commit e011e49 into main Nov 18, 2026
14 checks passed
@cxymds
cxymds deleted the cxymds/fix/du-live-listing branch November 18, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants