Skip to content

feat(java): add an optional S3 storage adapter - #966

Merged
SYaoJun merged 7 commits into
apache:mainfrom
keksmd:codex/java-storage-s3-upstream
Sep 7, 2026
Merged

SYaoJun merged 7 commits into
apache:mainfrom
keksmd:codex/java-storage-s3-upstream

Conversation

@keksmd

@keksmd keksmd commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Reason for this PR

Implements #965: an optional, dependency-isolated Java S3 storage adapter over the merged storage API.

What changes are included in this PR?

  • Adds graphar-storage-s3, backed by caller-owned AWS SDK v2 S3Client.
  • Validates s3:// locations, uses HeadObject/HTTP Range GET for independent seekable reads, and stages sequential output locally before conditional publication.
  • Adds credential-free S3 contract coverage plus an opt-in test against a real MinIO endpoint for staged create, non-overwrite, overwrite, and direct-buffer range reads.

The production code does not start, configure, or close an S3 client. The MinIO test is intentionally opt-in: normal SDK CI has no Docker or credential requirement.

Are these changes tested?

Yes.

mvn --no-transfer-progress -f maven-projects/pom.xml -pl storage-s3 -am clean verify -Dspotless.check.skip=true
GRAPHAR_MINIO_ENDPOINT=http://localhost:19000 mvn --no-transfer-progress -f maven-projects/pom.xml -pl storage-s3 -am -Dtest=MinioS3StorageIntegrationIT -Dsurefire.failIfNoSpecifiedTests=false test

Both pass locally, along with Spotless, pre-commit, and git diff --check.

Are there any user-facing changes?

Yes. Java users can construct S3Storage with their configured AWS SDK v2 client and access GraphAr s3:// data through the storage API.

Checklist

  • I have self-reviewed this PR
  • I have run make cpplint (if this PR contains C++ changes)
  • I have run pre-commit
  • I have run the relevant tests

@keksmd

keksmd commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Hi @SemyonSinchenko — this is the next independent Java slice after #960/#962: an optional S3 adapter over the now-merged storage API. It is intentionally isolated from #961 and has credential-free range/write contract coverage. I would appreciate your review when you have time.

@keksmd

keksmd commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Updated this PR into the complete S3 transport vertical: it now includes the opt-in real-MinIO gate (staged conditional publication and range reads), while normal CI stays credential- and Docker-free.

@SemyonSinchenko
SemyonSinchenko self-requested a review August 24, 2026 14:38
@keksmd

keksmd commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

cc @SemyonSinchenko @SYaoJun — green CI, would appreciate a look when you have time.

S3 range reads and staged writes need to satisfy the storage API without leaking AWS dependencies into metadata or core modules.

Relates to apache#965.

Rejected: a mandatory S3 or Hadoop dependency in the storage API.

Not-tested: a real MinIO endpoint; the follow-up opt-in transport test remains separate.
Exercise staged conditional publication and range reads against an actual S3-compatible endpoint, while keeping the test opt-in for normal SDK verification.

Relates to apache#965.
@keksmd
keksmd force-pushed the codex/java-storage-s3-upstream branch from 5d2fb5d to fd746c8 Compare September 7, 2026 07:02
@keksmd

keksmd commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

@SemyonSinchenko @SYaoJun a review here would be appreciated whenever you have time. Rebased on current main, no conflicts, CI green.

This follows the same shape as the local adapter merged in #960: it stays behind the storage API and keeps GraphAr layout and format concerns out of storage.

@SYaoJun

SYaoJun commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

@SemyonSinchenko @SYaoJun a review here would be appreciated whenever you have time. Rebased on current main, no conflicts, CI green.

This follows the same shape as the local adapter merged in #960: it stays behind the storage API and keeps GraphAr layout and format concerns out of storage.

@keksmd Thanks for your contribution!

I am not very familiar with the Java codebase, so it would be helpful to also get a review from Sem or Xiaokang.

I also ran this PR through an AI review tool and received some suggestions. I will paste them below for reference.

I have a few questions about the overall design. I may be missing some context here, but I am trying to understand the motivation behind this PR.

Why does GraphAr need to add S3 storage support? I noticed that there is already a java-info directory containing a pure Java reader and writer. Could you explain why the new implementation is not built on top of the existing structure?

Also, could you clarify the relationship between the new S3-related code and the existing java-info implementation?

It would be very helpful if you could add a high-level architecture diagram showing the relationships between these components and the overall design rationale.

@SYaoJun

SYaoJun commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for the PR — the module is cleanly separated (optional dependency, caller-owned S3Client, conditional PutObject, eTag/version-pinned reads) and the unit/IT split is well done. A few points should be resolved before this is ready to merge:

1. The new module is not covered by CI
.github/workflows/java-sdk.yml does not trigger on, nor build, maven-projects/storage-s3:

  • paths: for both push and pull_request don't include maven-projects/storage-s3/**, so the workflow didn't even run for this PR's module.
  • SDK_MODULES: storage-api means the compile/test jobs never touch storage-s3 (this PR only stayed green because the root maven-projects/pom.xml changed).

Please add the module to the trigger paths and the module list, e.g.:

paths:
  - 'maven-projects/storage-s3/**'
...
env:
  SDK_MODULES: storage-api,storage-local,storage-s3

(Also note storage-local from #960 is currently missing from SDK_MODULES, so aligning this list fixes a pre-existing gap too.)

2. create() fails late, unlike the local adapter — inconsistent behavior under the same Storage API
LocalOutputFile.open() fails immediately when the file exists (CREATE_NEW), but S3OutputFile.open() only stages a temp file and defers the existence check until close()'s conditional PutObject (If-None-Match: *). A caller can end up buffering/writing the whole object before hitting the "already exists" error. Suggestion: issue a headObject fast-fail in create() at open time (covers the common case), and keep If-None-Match on publish as the atomic backstop.

3. Inconsistent license headers
S3InputFile.java and S3SeekableInput.java use a single-line /* ... */ ASF header while every other file in the module (and in storage-api / storage-local) uses the standard multi-line header. Please align those two files with the canonical header format.

Happy to discuss any of these further. Thanks for the well-structured contribution.

The workflow triggers on storage-local, io-api and core but compiled only
storage-api, so every module merged after the storage API entered main
without being built or tested upstream. Add storage-s3 to the trigger paths
and list the shipped modules in SDK_MODULES.
create() deferred the existence check to the conditional publication, so a
caller could stream a whole object before learning about the conflict. Head
the object at open time to match the local adapter, and keep If-None-Match
as the atomic backstop when the head request cannot answer.
S3InputFile and S3SeekableInput carried a collapsed single-line header while
every other file in the SDK uses the multi-line form.
@keksmd

keksmd commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

@SYaoJun thanks for the careful read. All three points are addressed in the three commits I just pushed, and the design questions are answered below.

1. CI coverage — fixed in ci(java): build the SDK modules that the workflow already watches

You are right, and the gap was wider than this module: the workflow already triggers on storage-local/**, io-api/** and core/**, but SDK_MODULES: storage-api meant none of them were ever compiled or tested upstream. That is my own regression from #962, where storage-api genuinely was the only SDK module at the time. The workflow now watches maven-projects/storage-s3/** as well and builds storage-api,storage-local,io-api,storage-s3.

Verified locally with exactly the module set CI will use, reading the surefire reports rather than the exit code:

graphar-storage-api ..... SeekableInputTest 3/0/0
graphar-storage-local ... LocalStorageTest 5/0/0
graphar-io-api .......... 35 tests across 7 classes, 0 failures, 0 errors
graphar-storage-s3 ...... S3StorageTest 2/0/0

2. Late failure in create() — fixed in fix(java): reject an existing S3 object before staging bytes

create() now heads the object at open time and throws FileAlreadyExistsException, the same type the local adapter surfaces from CREATE_NEW. The conditional PutObject stays as the atomic backstop, so a head request that cannot answer (throttled, denied) does not turn into a spurious conflict — it just falls through to publication. Covered by a new unit test asserting that nothing is published when create() is rejected.

3. License headers — fixed in style(java): use the canonical ASF header in the S3 input files

Both files now carry the standard multi-line header.


On the design questions

info is metadata only: it loads and saves *.graph.yml / *.vertex.yml / *.edge.yml into GraphInfo, VertexInfo, EdgeInfo. Its src/main/java tree is loader, saver, yaml, type — there is no code there that reads or writes a data chunk, so there is nothing in it for an S3 adapter to build on. The name ReaderGraphInfoLoader refers to a java.io.Reader, not to a data reader.

The pure-Java SDK is being built as a stack of narrow layers, each merged as its own slice:

info          metadata: parse and serialize GraphAr YAML
storage-api   bytes: URI, InputFile/OutputFile, SeekableInput/PositionOutput   (#958)
  storage-local   local filesystem backend                                     (#960)
  storage-s3      S3 backend                                                   (this PR)
io-api        format: batches, schema, read and write requests                 (#961, #971, #972)
core          layout: chunk ranges, offsets, adjacency resolution              (#968)

The rule the layers follow is that storage knows only URIs and byte streams — no GraphAr layout, no Parquet, no Hadoop, no Arrow, no query semantics. That is what keeps a backend swappable: S3Storage implements the same Storage interface as LocalStorage, so everything above it is unchanged whether a graph lives on a disk or in a bucket, and the AWS SDK stays an optional dependency that a user who does not need S3 never pulls in.

So this PR does not duplicate anything in info; it sits under it, and the reader vertical that will consume both is the layer still to come.

@SemyonSinchenko a second pair of eyes on the Java side would be welcome, as SYaoJun suggested.

The step invoked javadoc:javadoc alone, so the reactor produced no jars and
a module resolved its siblings from the repository instead. With one module
in SDK_MODULES nothing was resolved across modules and the step passed; with
the shipped modules listed, storage-local could not find graphar-storage-api.
Run package in the same invocation so the reactor supplies the dependency.
@keksmd

keksmd commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: widening SDK_MODULES made the Build Java Docs step fail, and I have fixed that in ci(java): bind the javadoc step to a lifecycle phase.

The step invoked the bare goal mvn -pl ${SDK_MODULES} -am javadoc:javadoc, so the reactor produced no jars and graphar-storage-local tried to resolve graphar-storage-api:jar:0.13.0-SNAPSHOT from the repository, where no snapshot is published. With a single module in the list nothing was ever resolved across modules, so the latent problem stayed invisible. Running package in the same invocation makes the reactor supply the dependency.

Worth noting for a later cleanup, not changed here: the verify step already runs javadoc:jar (attach-javadocs) for every module, so Build Java Docs duplicates it. Happy to drop the step in a separate PR if you agree.

CI is green now, and the run actually exercised the new modules rather than passing them by:

storage-api ...... SeekableInputTest 3/0/0
storage-local .... LocalStorageTest 5/0/0
io-api ........... 35 tests, 0 failures, 0 errors
storage-s3 ....... S3StorageTest 2/0/0
Build Java Docs .. all four modules SUCCESS

…d trip per read

A head request that fails for any reason other than absence was read as
'object absent', so a permission or throttling error was deferred to close()
under a different message; only NoSuchKeyException and a 404 now mean absent.
A conditional put rejected with 412 lost the race for an existing object but
surfaced as a plain IOException, so callers could not tell a conflict from a
transient failure; it now surfaces as FileAlreadyExistsException, matching
both the eager check and the OutputFile.create() contract.

Reads were served by one GetObject range request per call, so byte-level
readers of columnar headers and footers paid a round trip each. The input now
keeps the last block in memory and serves subsequent reads from it, while a
read at least as large as the block bypasses the buffer.

@SemyonSinchenko SemyonSinchenko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~LGTM! Nice work!

@SemyonSinchenko

Copy link
Copy Markdown
Member

Hi @SYaoJun ! LGTM from my point of view. What do you think?

@keksmd

keksmd commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @SemyonSinchenko for the review and the approval — the three points you raised are what made this module honest: typed failures on headObject, a FileAlreadyExistsException on the lost 412 race, and read-ahead so a byte-level reader no longer pays a round trip per read().

@SYaoJun your three points are addressed as well (CI now builds storage-api,storage-local,io-api,storage-s3, create() fast-fails at open time, canonical ASF headers), and the layer diagram in my earlier comment answers the design question about info. CI is green on f1d0bb2 with real test counts: S3StorageTest 5/0/0, io-api 35/0/0, and Build Java Docs now succeeds for all five modules.

Let me know if anything else is needed here.

@SYaoJun
SYaoJun merged commit dd17c9b into apache:main Sep 7, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants