Repository navigation
feat(java): read and write Parquet through column vectors - #975
Conversation
Adds the graphar-io-parquet module: a physical reader that materializes each Parquet row group as ValueVector columns inside a VectorRecordBatch, applying projection and half-open row ranges and reporting what it pushed down, and a physical writer that emits Offset Indexes with bounded pages. Binary values are exposed as read-only ByteBuffers over a private copy and list values as immutable lists, so a consumer cannot alias the decoder's buffers. Parquet cannot append to a closed file, so WriteMode.APPEND is rejected with UnsupportedOperationException instead of being downgraded to an overwrite. io-api gains ObjectValueVector and RecordBatches, an in-memory vector and a row-to-column transposer so producers can build batches without Arrow. Closes apache#964
|
Did we even consider using anything else for parquet reading/writing except the hadoop ecosystem? For example, https://github.com/hardwood-hq/hardwood ? That is a question I would like to discuss first. The problem of hadoop tech stack for me is it is a) heavy b) leads to the dependency hell for consumers |
|
Yes, considered, and I share the concern about the Hadoop tail. Current state and reasoning: What this PR actually depends on. Our code has zero Why
What is already backend-neutral. What I would propose.
Happy to open the follow-up issue if this direction works for you. |
|
@keksmd Thanks! Sounds to me. I will try to review the PR ASAP |
Follow the Parquet LIST spec: write and accept 3-level lists whose element is REQUIRED or OPTIONAL, carrying null elements, so files from Spark, Arrow and other writers read and our files are standard. Validate producer values in ObjectValueVector against the field type and nullability, naming column and index, and snapshot list values recursively without rejecting null elements. Read whole row groups without the offset-index path, release the file as soon as the limit is reached, make the footer cache opt-in with an explicit immutability contract, reuse the single-byte write buffer, and replace hadoop-common and hadoop-mapreduce-client-core with the shaded Hadoop client 3.4.2 (runtime jars 135 -> 19; 3.5.x targets Java 17).
|
@SemyonSinchenko gentle ping: all review comments are addressed in eb0ad12, CI is green. Could you take another look when you have time? |
|
@keksmd sorry for the delay |
Reason for this PR
Closes #964. #961 replaced the row-oriented physical batch contract with
ValueVector/VectorRecordBatch; this is the Parquet backend for that contract. It is the physical IO layer the pure-Java reader and writer (#947) sit on, and it keeps the layout-free boundary: no GraphAr semantics, no Arrow, noRowfacade.What changes are included in this PR?
New module
graphar-io-parquet(wired intomaven-projects/pom.xmland the Java SDK workflow):ParquetPhysicalReader/ParquetBatchCursor: reads one row group per batch, applies physical projection and the strict half-open row range, and reportsPROJECTION/ROW_RANGEas applied. Each batch is aVectorRecordBatchofParquetValueVectorcolumns.ParquetValueVector: binary values come back as read-onlyByteBuffers over a private copy and lists as immutableLists, so a consumer never aliases the decoder's buffers.ParquetPhysicalWriter: emits Offset Indexes with bounded pages.WriteMode.APPENDis rejected withUnsupportedOperationExceptionbecause Parquet cannot append to a closed file; it is never treated as an overwrite.io-api:ObjectValueVector(in-memory vector) andRecordBatches.ofRows/ofArrays(row-to-column transposer) so producers can build column batches without Arrow.Non-goals kept out: GraphAr reader/writer/validator consumers, Arrow, changes to the decoding algorithm.
Are these changes tested?
Yes, from
maven-projects/with the same commands the workflow runs:All green: io-api 79 tests, io-parquet 8 tests.
RecordBatchesTest: schema position, row/column counts, nulls, width mismatch rejection, list snapshot on read.ParquetPhysicalReaderFixtureTest: canonicalldbc_samplefixture; projection and row-range reads retain appliedPROJECTIONandROW_RANGE.ParquetListRoundTripTest: nullable string lists round-trip, empty and null preserved.ParquetPhysicalWriterOffsetIndexTest: written files carry readable Offset Indexes with the bounded-page assertion.ParquetPhysicalWriterModeTest:APPENDthrows and leaves the existing file byte-identical;OVERWRITEreplaces it.ParquetPhysicalIoEfficiencyTest: projection and row range only touch the needed column chunks / row groups.pre-commit run --fileson the changed paths: passed.Are there any user-facing changes?
New public module
org.apache.graphar:graphar-io-parquetand two new public io-api classes (ObjectValueVector,RecordBatches). No changes to existing public APIs.