Repository navigation
feat: apply write defaults in Parquet and Avro writers - #942
Open
huan233usc wants to merge 12 commits into
Open
huan233usc wants to merge 12 commits into
huan233usc wants to merge 12 commits into
Conversation
huan233usc
force-pushed
the
feat/write-default-e2e
branch
7 times, most recently
from
September 18, 2026 22:45
b824448 to
ddb87cc
Compare
Extend Project with ProjectionOptions so the same field-id matching serves both paths: reads fill missing fields with initial-default and allow type promotion, writes fill with write-default and require exact types.
Add arrow::AlignBatchForWrite which projects an input batch onto the write schema via Project with write-default options: present columns are reordered by field id (recursively for structs), missing columns are materialized from write-default or null, and unknown input columns are rejected rather than silently dropped.
DataWriter aligns batches to the write schema when the input schema differs, via a new Write(data, input_schema) overload; schemas that already match keep the zero-conversion Write(data) path. Adds Parquet and Avro end-to-end tests where a column added with write-default reads back the default for rows written without it.
huan233usc
force-pushed
the
feat/write-default-e2e
branch
from
September 18, 2026 22:57
ddb87cc to
8e01c0e
Compare
14 of 21 tasks
8 tasks
Member
|
Is this ready for review? |
Contributor
Author
|
I plan to do some final passes and then will send for review -- thanks |
- Preserve sliced struct offsets and align collection children by field id. - Materialize unknown inputs as typed nulls. - Configure optional input schemas without extending the writer interface.
- Always align batches while retaining the declared or default input schema. - Forward equal-schema batches without importing or exporting Arrow C Data.
huan233usc
marked this pull request as ready for review
October 12, 2026 16:48
shangxinli
reviewed
Oct 17, 2026
| projection, | ||
| ProjectNested(*expected_field.type(), *source_field.type(), prune_source)); | ||
| ProjectNested(*expected_field.type(), *source_field.type(), prune_source, mode)); | ||
| } else if (mode == ProjectionMode::kWrite) { |
Contributor
There was a problem hiding this comment.
The kWrite branch checks types but never compares source_field.optional() against expected_field.optional(), so an optional input column projects onto a required write column and its nulls get written straight into it. Java rejects that at CheckCompatibility.java:184, "should be required, but is optional", reached from TypeUtil.validateWriteSchema at TypeUtil.java:498.
wgtmac
reviewed
Oct 18, 2026
| } | ||
|
|
||
| Status Write(ArrowArray* data) { | ||
| internal::ArrowArrayGuard data_guard(data); |
Member
There was a problem hiding this comment.
'AlignBatchForWrite' consumes and releases 'data' even when it returns an error, but this outer guard still owns the same array. A missing required field or unknown input field therefore releases it in the helper and again here.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writing an Arrow batch with an older schema currently requires callers to fill columns added by schema evolution. This PR adds optional
input_schematoDataWriterOptionsandWriterOptions, so Parquet and Avro writers can align batches to the target schema and populate omitted columns with Icebergwrite-defaultvalues.For a required column with
initial-default = 42andwrite-default = 7, an input batch containing onlyidwrites[[1, 7], [2, 7]]. Explicit nulls stay null; missing optional fields become null. Missing required fields without a write-default and unknown input field IDs are rejected.The shared alignment helper matches fields by ID and handles structs, lists, and maps recursively, preserving existing buffers, offsets, and validity. Matching schemas forward the original Arrow C Data batch.
Projectaccepts a read/write mode: reading uses initial-default and allows type promotion; writing uses write-default and requires matching types.Addresses the write-path portion of #637 and includes #941.
Validation: the complete
data_test(188 passed) andschema_test(564 passed) suites completed with no failures. Coverage includes Parquet and both Avro backends, identical-schema ownership transfer, sliced nested collections, field reordering, typed nulls, and rejected incompatible schemas.clang-format,clang-tidyon the changed implementations and tests, andgit diff --checkpassed.