Skip to content

feat: add envvar RC_MULTIPART_THRESHOLD - #411

Open
qzhuyan wants to merge 1 commit into
rustfs:mainfrom
qzhuyan:feat/multipart-threshold-env
Open

qzhuyan wants to merge 1 commit into
rustfs:mainfrom
qzhuyan:feat/multipart-threshold-env

Conversation

@qzhuyan

@qzhuyan qzhuyan commented Nov 5, 2026

Copy link
Copy Markdown

No description provided.

@qzhuyan
qzhuyan force-pushed the feat/multipart-threshold-env branch from fc3ce21 to 430ed3c Compare November 13, 2026 16:35
@qzhuyan

qzhuyan commented Nov 13, 2026

Copy link
Copy Markdown
Author

rebased onto latest main.

@cxymds

cxymds commented Nov 18, 2026

Copy link
Copy Markdown
Member

@qzhuyan, I’d keep the 128 MiB use case from the README. Before merging, could you address the memory cost of raising the threshold? The default HTTP path reads the whole file into a Vec before sending the PUT. With a 5 GiB threshold, uploading a 4 GiB file would require at least 4 GiB of memory for that request alone. Single-PUT implementation

  1. For files above 64 MiB that still qualify for a single PUT, use a bounded buffer and a request body that the SDK can replay with identical bytes. One option is to copy the source into a private temporary file, calculate SHA-256 during the copy when requested, and stream that file with a known length. Keep it alive through SDK retries. This confines the extra disk I/O to the newly supported large single PUTs.
  2. Please add an HTTP-level regression test. With a 128 MiB threshold, a 65 MiB file should produce one PUT with the expected length and digest. A separate retry case could return 503 on the first attempt and replace the original source before the retry, then verify that both attempts sent identical bytes. Small fixtures are enough to check the exact threshold and one byte above it. Please also exercise the large single PUT in the RustFS smoke test, since fix: improve RustFS compatibility for cp, mv, and quota parsing #37 lowered the default after compatibility issues with aws-chunked uploads.
  3. For CI, please sync the branch with current main and rerun the checks. fix(replication): support permanent site decommissioning #421 already removed the two dependencies flagged by the advisory job, and the main branch advisory check passed.

If your use case only needs multipart uploads to start earlier, a smaller option would be to accept thresholds from 1 byte through 64 MiB and update the README and tests accordingly. I’d favor keeping the documented 128 MiB use case if raising the threshold matters for your workload.

@qzhuyan

qzhuyan commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

@cxymds , thanks for the comments.

Could you elaborate more about what do you mean on the 128 MiB use case from the README , is the one added in this PR or the usecases in existing documents?

My usecaes is basically transfer a large file (~100MB) without using the multi-parts transfer.

With AWS cli it is supported by profile arg

s3 =
    multipart_threshold = 128MB

For the concern of the memory cost, I think I could run one round of computing the SHA-256 (with iter parts) and 2nd round of streaming the file to the network with bounded buffer (buffer size: rc_s3::multipart::DEFAULT_PART_SIZE ) assuming the file won't be changed between this two ops, WDYT?

The reason I don't perfer the 'temporary file' is because

  1. use extra space
  2. requires write permssion
  3. Introducing write wear in embedded devices

@cxymds

cxymds commented Nov 19, 2026

Copy link
Copy Markdown
Member

PR allows the threshold to be set to 5 GiB, while the default HTTP implementation uses read_to_end to read the entire file of a single PUT into a Vec. Following this code path, when the threshold is set to 5 GiB, uploading a 4 GiB file requires at least 4 GiB of file buffering, and concurrent uploads will add to this. It is recommended to first use bounded memory for single PUTs of large files, and add a regression test that covers the actual upload path. The newly added CLI test runs with --dry-run, which covers threshold judgment and cannot detect this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants