Skip to content

Replace the Arrow scan with a port of the neo branch's scan - #693

Merged
evertlammerts merged 2 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:arrow-scan-port-integration
Oct 18, 2026
Merged

evertlammerts merged 2 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:arrow-scan-port-integration

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

Registered Arrow objects are wrapped in a pyarrow Scanner that evaluates pushed filters with pyarrow kernels on every batch and reads one shot streams far ahead of the query. That caused the kernel failures of #663 to #666, the memory growth of #691, wrong rows for struct columns in sliced batches, and a hang at exit with an unread generator reader.

This PR replaces that path with a port of the neo branch's scan: one table function, python_arrow_scan, over a private source adapter.

  • The engine offers each filter to the source, which takes only what it applies exactly (a pyarrow expression on Datasets, a polars one on LazyFrames); the rest stays above the scan. In memory objects never receive filters.
  • Only the columns a query reads are converted. Every array is validated before adoption; a stream whose schema differs from the declared one is refused.
  • Streams are read one batch per thread, without read ahead and without holding the GIL; a one shot reader read twice gets a clear error.
  • pa.Array, ChunkedArray, pandas and polars Series and polars LazyFrames are accepted everywhere; every Python error from a source is an InvalidInputException naming the object.
  • Interpreter exit waits for a pull inside Python and refuses new scans.

Notes:

  • No longer pushed, the engine filters instead: LIKE, NaN, nanosecond and infinite constants, date64 columns, and float predicates NaN can satisfy on Datasets and LazyFrames (parquet row group statistics leave NaN out). In memory row counts are estimates.
  • A struct below a struct in a sliced batch is refused until the engine folds the parent offset; it read wrong rows before.

@evertlammerts
evertlammerts added this pull request to the merge queue Oct 18, 2026
Merged via the queue into duckdb:v2.0-cyanoptera with commit 2d55a21 Oct 18, 2026
30 checks passed
@evertlammerts
evertlammerts deleted the arrow-scan-port-integration branch October 18, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant