Repository navigation
Conversation
SemyonSinchenko
left a comment
There was a problem hiding this comment.
~LGTM overall. I left a minor comment. +linting fail
| } | ||
|
|
||
| private static List<BlockRange> rowGroups(List<BlockMetaData> blocks) { | ||
| private boolean indexedPartialRowGroups() { |
There was a problem hiding this comment.
This method re-implements the partial-overlap predicate (begin < end && (begin != rowGroup.start || end != rowGroup.end)) that nextRange() also encodes in negated form for the whole flag (begin == rowGroup.start && end == rowGroup.end). The two copies currently agree, but this predicate decides the public rowRangeApplied() result (and hence the ReadReport's applied/declined ROW_RANGE), while the copy in nextRange() decides the actual slice-in-memory fallback. If a future edit changes one site without the other, the report could claim the row range was physically pushed down while a group is in fact being sliced in memory. Extract the predicate into a shared helper used by both.
|
Sorry for the delay @keksmd |
…set Index Parquet files written by Arrow C++ 9.0, including the canonical testing/ldbc_sample edge chunks, carry no Offset Index. A row range that covered only part of such a row group failed with "refusing JVM fallback", so GraphAr adjacency ranges could not be read from them at all. A partly selected row group without an Offset Index for a read column is now read whole and sliced to the half-open range; row groups outside the range are still skipped, so memory stays bounded to the touched row groups. The read reports ROW_RANGE as declined in that case and as applied otherwise, and returns the same rows either way, as the io-api read contract requires. Rejected: keep refusing - breaks reading canonical C++-written fixtures Rejected: report applied for row-group pruning - rows are sliced in the JVM Directive: ROW_RANGE stays applied only when every partly touched row group is read through its Offset Index
fb458f4 to
450c732
Compare
|
@keksmd should I merge this one? |
Reason for this PR
Closes #981. Parquet files without an Offset Index, including the Arrow C++ edge chunks in
testing/ldbc_sample, could not be read by a partial row range.What changes are included in this PR?
readFilteredRowGroup.ROW_RANGEis reported as applied only when every partly touched row group was read through its Offset Index, and as declined otherwise. The rows returned are identical either way.Are these changes tested?
Yes.
ParquetPhysicalReaderWithoutOffsetIndexTestreads a canonicalldbc_sampleedge chunk that has no Offset Index (range, projection and limit). It also compares a multi-row-group file with an Offset Index against the same file with the page indexes stripped: the rows are identical and only the applied/declined sets differ.mvn -pl io-parquet -am clean verify: io-parquet 15 tests, io-api 85, 0 failures.Are there any user-facing changes?
Reads that used to throw now succeed and report
ROW_RANGEas declined. No API change.Checklist
make cpplintbefore submitting when changed files are in thecppdirectory.pre-commit runbefore commit the changed files.