Skip to content

fix(java): read a partial Parquet row range from files without an Offset Index - #982

Open
keksmd wants to merge 1 commit into
apache:mainfrom
keksmd:java-parquet-range-without-offset-index
Open

keksmd wants to merge 1 commit into
apache:mainfrom
keksmd:java-parquet-range-without-offset-index

Conversation

@keksmd

@keksmd keksmd commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Reason for this PR

Closes #981. Parquet files without an Offset Index, including the Arrow C++ edge chunks in testing/ldbc_sample, could not be read by a partial row range.

What changes are included in this PR?

  • When a partly selected row group has an Offset Index for every read column, it is still read page by page with readFilteredRowGroup.
  • When it does not, the row group is read whole and sliced to the half-open range. Row groups outside the range are still skipped.
  • ROW_RANGE is reported as applied only when every partly touched row group was read through its Offset Index, and as declined otherwise. The rows returned are identical either way.

Are these changes tested?

Yes. ParquetPhysicalReaderWithoutOffsetIndexTest reads a canonical ldbc_sample edge chunk that has no Offset Index (range, projection and limit). It also compares a multi-row-group file with an Offset Index against the same file with the page indexes stripped: the rows are identical and only the applied/declined sets differ. mvn -pl io-parquet -am clean verify: io-parquet 15 tests, io-api 85, 0 failures.

Are there any user-facing changes?

Reads that used to throw now succeed and report ROW_RANGE as declined. No API change.

Checklist

  • I have performed a self-review of my own code.
  • I have formatted my own code using make cpplint before submitting when changed files are in the cpp directory.
  • I have performed pre-commit run before commit the changed files.
  • I have added tests to prove my changes are effective.

@SemyonSinchenko SemyonSinchenko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~LGTM overall. I left a minor comment. +linting fail

}

private static List<BlockRange> rowGroups(List<BlockMetaData> blocks) {
private boolean indexedPartialRowGroups() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method re-implements the partial-overlap predicate (begin < end && (begin != rowGroup.start || end != rowGroup.end)) that nextRange() also encodes in negated form for the whole flag (begin == rowGroup.start && end == rowGroup.end). The two copies currently agree, but this predicate decides the public rowRangeApplied() result (and hence the ReadReport's applied/declined ROW_RANGE), while the copy in nextRange() decides the actual slice-in-memory fallback. If a future edit changes one site without the other, the report could claim the row range was physically pushed down while a group is in fact being sliced in memory. Extract the predicate into a shared helper used by both.

@SemyonSinchenko

Copy link
Copy Markdown
Member

Sorry for the delay @keksmd

…set Index

Parquet files written by Arrow C++ 9.0, including the canonical
testing/ldbc_sample edge chunks, carry no Offset Index. A row range that
covered only part of such a row group failed with "refusing JVM fallback",
so GraphAr adjacency ranges could not be read from them at all.

A partly selected row group without an Offset Index for a read column is now
read whole and sliced to the half-open range; row groups outside the range
are still skipped, so memory stays bounded to the touched row groups. The
read reports ROW_RANGE as declined in that case and as applied otherwise,
and returns the same rows either way, as the io-api read contract requires.

Rejected: keep refusing - breaks reading canonical C++-written fixtures
Rejected: report applied for row-group pruning - rows are sliced in the JVM
Directive: ROW_RANGE stays applied only when every partly touched row group is read through its Offset Index
@keksmd
keksmd force-pushed the java-parquet-range-without-offset-index branch from fb458f4 to 450c732 Compare November 12, 2026 19:23
@SemyonSinchenko

Copy link
Copy Markdown
Member

@keksmd should I merge this one?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG][Java] io-parquet refuses a partial row range of Parquet files without an Offset Index

2 participants