Repository navigation
feat(java): add an optional S3 storage adapter - #966
Conversation
|
Hi @SemyonSinchenko — this is the next independent Java slice after #960/#962: an optional S3 adapter over the now-merged storage API. It is intentionally isolated from #961 and has credential-free range/write contract coverage. I would appreciate your review when you have time. |
|
Updated this PR into the complete S3 transport vertical: it now includes the opt-in real-MinIO gate (staged conditional publication and range reads), while normal CI stays credential- and Docker-free. |
|
cc @SemyonSinchenko @SYaoJun — green CI, would appreciate a look when you have time. |
S3 range reads and staged writes need to satisfy the storage API without leaking AWS dependencies into metadata or core modules. Relates to apache#965. Rejected: a mandatory S3 or Hadoop dependency in the storage API. Not-tested: a real MinIO endpoint; the follow-up opt-in transport test remains separate.
Exercise staged conditional publication and range reads against an actual S3-compatible endpoint, while keeping the test opt-in for normal SDK verification. Relates to apache#965.
5d2fb5d to
fd746c8
Compare
|
@SemyonSinchenko @SYaoJun a review here would be appreciated whenever you have time. Rebased on current This follows the same shape as the local adapter merged in #960: it stays behind the storage API and keeps GraphAr layout and format concerns out of storage. |
@keksmd Thanks for your contribution! I am not very familiar with the Java codebase, so it would be helpful to also get a review from Sem or Xiaokang. I also ran this PR through an AI review tool and received some suggestions. I will paste them below for reference. I have a few questions about the overall design. I may be missing some context here, but I am trying to understand the motivation behind this PR. Why does GraphAr need to add S3 storage support? I noticed that there is already a Also, could you clarify the relationship between the new S3-related code and the existing It would be very helpful if you could add a high-level architecture diagram showing the relationships between these components and the overall design rationale. |
|
Thanks for the PR — the module is cleanly separated (optional dependency, caller-owned 1. The new module is not covered by CI
Please add the module to the trigger paths and the module list, e.g.: paths:
- 'maven-projects/storage-s3/**'
...
env:
SDK_MODULES: storage-api,storage-local,storage-s3(Also note 2. 3. Inconsistent license headers Happy to discuss any of these further. Thanks for the well-structured contribution. |
The workflow triggers on storage-local, io-api and core but compiled only storage-api, so every module merged after the storage API entered main without being built or tested upstream. Add storage-s3 to the trigger paths and list the shipped modules in SDK_MODULES.
create() deferred the existence check to the conditional publication, so a caller could stream a whole object before learning about the conflict. Head the object at open time to match the local adapter, and keep If-None-Match as the atomic backstop when the head request cannot answer.
S3InputFile and S3SeekableInput carried a collapsed single-line header while every other file in the SDK uses the multi-line form.
|
@SYaoJun thanks for the careful read. All three points are addressed in the three commits I just pushed, and the design questions are answered below. 1. CI coverage — fixed in You are right, and the gap was wider than this module: the workflow already triggers on Verified locally with exactly the module set CI will use, reading the surefire reports rather than the exit code: 2. Late failure in
3. License headers — fixed in Both files now carry the standard multi-line header. On the design questions
The pure-Java SDK is being built as a stack of narrow layers, each merged as its own slice: The rule the layers follow is that storage knows only URIs and byte streams — no GraphAr layout, no Parquet, no Hadoop, no Arrow, no query semantics. That is what keeps a backend swappable: So this PR does not duplicate anything in @SemyonSinchenko a second pair of eyes on the Java side would be welcome, as SYaoJun suggested. |
The step invoked javadoc:javadoc alone, so the reactor produced no jars and a module resolved its siblings from the repository instead. With one module in SDK_MODULES nothing was resolved across modules and the step passed; with the shipped modules listed, storage-local could not find graphar-storage-api. Run package in the same invocation so the reactor supplies the dependency.
|
Follow-up: widening The step invoked the bare goal Worth noting for a later cleanup, not changed here: the CI is green now, and the run actually exercised the new modules rather than passing them by: |
…d trip per read A head request that fails for any reason other than absence was read as 'object absent', so a permission or throttling error was deferred to close() under a different message; only NoSuchKeyException and a 404 now mean absent. A conditional put rejected with 412 lost the race for an existing object but surfaced as a plain IOException, so callers could not tell a conflict from a transient failure; it now surfaces as FileAlreadyExistsException, matching both the eager check and the OutputFile.create() contract. Reads were served by one GetObject range request per call, so byte-level readers of columnar headers and footers paid a round trip each. The input now keeps the last block in memory and serves subsequent reads from it, while a read at least as large as the block bypasses the buffer.
|
Hi @SYaoJun ! LGTM from my point of view. What do you think? |
|
Thanks @SemyonSinchenko for the review and the approval — the three points you raised are what made this module honest: typed failures on @SYaoJun your three points are addressed as well (CI now builds Let me know if anything else is needed here. |
Reason for this PR
Implements #965: an optional, dependency-isolated Java S3 storage adapter over the merged storage API.
What changes are included in this PR?
graphar-storage-s3, backed by caller-owned AWS SDK v2S3Client.s3://locations, uses HeadObject/HTTP Range GET for independent seekable reads, and stages sequential output locally before conditional publication.The production code does not start, configure, or close an S3 client. The MinIO test is intentionally opt-in: normal SDK CI has no Docker or credential requirement.
Are these changes tested?
Yes.
mvn --no-transfer-progress -f maven-projects/pom.xml -pl storage-s3 -am clean verify -Dspotless.check.skip=true GRAPHAR_MINIO_ENDPOINT=http://localhost:19000 mvn --no-transfer-progress -f maven-projects/pom.xml -pl storage-s3 -am -Dtest=MinioS3StorageIntegrationIT -Dsurefire.failIfNoSpecifiedTests=false testBoth pass locally, along with Spotless, pre-commit, and
git diff --check.Are there any user-facing changes?
Yes. Java users can construct
S3Storagewith their configured AWS SDK v2 client and access GraphArs3://data through the storage API.Checklist
make cpplint(if this PR contains C++ changes)pre-commit