Repository navigation
Conversation
fc3ce21 to
430ed3c
Compare
|
rebased onto latest main. |
|
@qzhuyan, I’d keep the 128 MiB use case from the README. Before merging, could you address the memory cost of raising the threshold? The default HTTP path reads the whole file into a
If your use case only needs multipart uploads to start earlier, a smaller option would be to accept thresholds from 1 byte through 64 MiB and update the README and tests accordingly. I’d favor keeping the documented 128 MiB use case if raising the threshold matters for your workload. |
|
@cxymds , thanks for the comments. Could you elaborate more about what do you mean on the My usecaes is basically transfer a large file (~100MB) without using the multi-parts transfer. With AWS cli it is supported by profile arg For the concern of the memory cost, I think I could run one round of computing the SHA-256 (with iter parts) and 2nd round of streaming the file to the network with bounded buffer (buffer size: The reason I don't perfer the 'temporary file' is because
|
|
PR allows the threshold to be set to 5 GiB, while the default HTTP implementation uses read_to_end to read the entire file of a single PUT into a Vec. Following this code path, when the threshold is set to 5 GiB, uploading a 4 GiB file requires at least 4 GiB of file buffering, and concurrent uploads will add to this. It is recommended to first use bounded memory for single PUTs of large files, and add a regression test that covers the actual upload path. The newly added CLI test runs with --dry-run, which covers threshold judgment and cannot detect this issue |
No description provided.