Skip to content

feat(java): read and write Parquet through column vectors - #975

Merged
SemyonSinchenko merged 2 commits into
apache:mainfrom
keksmd:java-io-parquet-columnar
Oct 9, 2026
Merged

SemyonSinchenko merged 2 commits into
apache:mainfrom
keksmd:java-io-parquet-columnar

Conversation

@keksmd

@keksmd keksmd commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Reason for this PR

Closes #964. #961 replaced the row-oriented physical batch contract with ValueVector / VectorRecordBatch; this is the Parquet backend for that contract. It is the physical IO layer the pure-Java reader and writer (#947) sit on, and it keeps the layout-free boundary: no GraphAr semantics, no Arrow, no Row facade.

What changes are included in this PR?

New module graphar-io-parquet (wired into maven-projects/pom.xml and the Java SDK workflow):

  • ParquetPhysicalReader / ParquetBatchCursor: reads one row group per batch, applies physical projection and the strict half-open row range, and reports PROJECTION / ROW_RANGE as applied. Each batch is a VectorRecordBatch of ParquetValueVector columns.
  • ParquetValueVector: binary values come back as read-only ByteBuffers over a private copy and lists as immutable Lists, so a consumer never aliases the decoder's buffers.
  • ParquetPhysicalWriter: emits Offset Indexes with bounded pages. WriteMode.APPEND is rejected with UnsupportedOperationException because Parquet cannot append to a closed file; it is never treated as an overwrite.
  • io-api: ObjectValueVector (in-memory vector) and RecordBatches.ofRows / ofArrays (row-to-column transposer) so producers can build column batches without Arrow.

Non-goals kept out: GraphAr reader/writer/validator consumers, Arrow, changes to the decoding algorithm.

Are these changes tested?

Yes, from maven-projects/ with the same commands the workflow runs:

mvn --no-transfer-progress -pl storage-api,storage-local,io-api,io-parquet,storage-s3,core -am spotless:check
mvn --no-transfer-progress -pl storage-api,storage-local,io-api,io-parquet,storage-s3,core -am clean verify -Dspotless.check.skip=true
mvn --no-transfer-progress -pl io-api,io-parquet -am package javadoc:javadoc -DskipTests -Dspotless.check.skip=true

All green: io-api 79 tests, io-parquet 8 tests.

  • RecordBatchesTest: schema position, row/column counts, nulls, width mismatch rejection, list snapshot on read.
  • ParquetPhysicalReaderFixtureTest: canonical ldbc_sample fixture; projection and row-range reads retain applied PROJECTION and ROW_RANGE.
  • ParquetListRoundTripTest: nullable string lists round-trip, empty and null preserved.
  • ParquetPhysicalWriterOffsetIndexTest: written files carry readable Offset Indexes with the bounded-page assertion.
  • ParquetPhysicalWriterModeTest: APPEND throws and leaves the existing file byte-identical; OVERWRITE replaces it.
  • ParquetPhysicalIoEfficiencyTest: projection and row range only touch the needed column chunks / row groups.

pre-commit run --files on the changed paths: passed.

Are there any user-facing changes?

New public module org.apache.graphar:graphar-io-parquet and two new public io-api classes (ObjectValueVector, RecordBatches). No changes to existing public APIs.

Adds the graphar-io-parquet module: a physical reader that materializes each
Parquet row group as ValueVector columns inside a VectorRecordBatch, applying
projection and half-open row ranges and reporting what it pushed down, and a
physical writer that emits Offset Indexes with bounded pages.

Binary values are exposed as read-only ByteBuffers over a private copy and list
values as immutable lists, so a consumer cannot alias the decoder's buffers.

Parquet cannot append to a closed file, so WriteMode.APPEND is rejected with
UnsupportedOperationException instead of being downgraded to an overwrite.

io-api gains ObjectValueVector and RecordBatches, an in-memory vector and a
row-to-column transposer so producers can build batches without Arrow.

Closes apache#964
@SemyonSinchenko

SemyonSinchenko commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Did we even consider using anything else for parquet reading/writing except the hadoop ecosystem? For example, https://github.com/hardwood-hq/hardwood ? That is a question I would like to discuss first. The problem of hadoop tech stack for me is it is a) heavy b) leads to the dependency hell for consumers

@SemyonSinchenko

Copy link
Copy Markdown
Member

@SYaoJun @Thespica @yecol @keksmd @yangxk1 wdyt about the question above (hadoop vs something-else)?

@keksmd

keksmd commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Yes, considered, and I share the concern about the Hadoop tail. Current state and reasoning:

What this PR actually depends on. Our code has zero org.apache.hadoop imports. GraphAr storage (storage-api: InputFile/SeekableInput/PositionOutput) is what Parquet reads from and writes to, via ParquetInputFile/ParquetOutputFile adapters — no S3A, no Hadoop FileSystem. hadoop-common and hadoop-mapreduce-client-core are there only because parquet-hadoop needs Configuration and MapReduce input classes on the classpath at runtime. So the Hadoop dependency is an accident of parquet-java packaging, not a design choice.

Why parquet-java for the first slice.

  • Java 11 baseline of the SDK. Hardwood requires Java 21 (maven.compiler.release=21 in its pom; native build on 25).
  • Write to object storage: Hardwood's writer targets OutputFile.of(Path) only, object-store output is "coming soon" (README, 1.1.0.Beta1, 2026-08-31). Our writer goes through storage-api, S3 included, today.
  • Maturity as a format oracle: the C++ implementation is the reference, parquet-java is the canonical Java counterpart. For a first columnar slice I preferred boring-and-correct.

What is already backend-neutral. io-api (PhysicalReader/PhysicalWriter, RecordBatch/ValueVector, ReadRequest with projection/range/filter/limit and the applied-hints report) has no Parquet or Hadoop types. io-parquet is one implementation behind it; reader/writer never see parquet-java. Swapping or adding an engine is a module, not an API change.

What I would propose.

  1. Keep this PR as the parquet-java backend, but I can shrink the tail right here: replace hadoop-common + hadoop-mapreduce-client-core with the shaded hadoop-client-api (single jar, no transitive Guava/Jackson/etc.). If you want that in this PR, say so and I will do it.
  2. Follow-up issue: a Hardwood backend spike as a separate optional module, Java 21, behind the same PhysicalReader/PhysicalWriter, with a JMH comparison on topology (INT64 src/dst) and property chunks. Hardwood's typed ColumnReader/ColumnBatch maps well onto column vectors and would let us add primitive accessors to ValueVector without boxing; I would rather do that with numbers than by assumption.
  3. Decision needed from maintainers: is Java 21 acceptable for an optional module while the core SDK stays on 11? If yes, Hardwood is a real candidate for the default once its object-store write lands. If no, parquet-java + shaded Hadoop client is the realistic option.

Happy to open the follow-up issue if this direction works for you.

@SemyonSinchenko

Copy link
Copy Markdown
Member

@keksmd Thanks! Sounds to me. I will try to review the PR ASAP

Comment thread maven-projects/io-parquet/pom.xml Resolved
Follow the Parquet LIST spec: write and accept 3-level lists whose element
is REQUIRED or OPTIONAL, carrying null elements, so files from Spark, Arrow
and other writers read and our files are standard.

Validate producer values in ObjectValueVector against the field type and
nullability, naming column and index, and snapshot list values recursively
without rejecting null elements.

Read whole row groups without the offset-index path, release the file as
soon as the limit is reached, make the footer cache opt-in with an explicit
immutability contract, reuse the single-byte write buffer, and replace
hadoop-common and hadoop-mapreduce-client-core with the shaded Hadoop
client 3.4.2 (runtime jars 135 -> 19; 3.5.x targets Java 17).
@keksmd

keksmd commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

@SemyonSinchenko gentle ping: all review comments are addressed in eb0ad12, CI is green. Could you take another look when you have time?

@SemyonSinchenko SemyonSinchenko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~LGTM

@SemyonSinchenko
SemyonSinchenko merged commit d7fe558 into apache:main Oct 9, 2026
4 checks passed
@SemyonSinchenko

Copy link
Copy Markdown
Member

@keksmd sorry for the delay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE][Java] Adapt Parquet physical IO to columnar batches

2 participants