feat(array): migrate encodings index reading to scalar probes

[jiengup]

This PR is still WIP. Any feedback or guidance is welcome.

Summary

• Tracking Issue: Migrate every encoding from OperationsVTable::scalar_at to probe_scalar #9971

OperationsVTable::scalar_at cannot retain preparation between row reads. The probe API introduced in #9843 retains encoding state, validity, and child probes across repeated reads; #9844 demonstrates this for RLE, RunEnd, and PCO.

This commit migrates eight simple implementations to explicit probe_scalar methods. Each uses ProbeState = (): none needs an encoding-owned cache. Encodings that need parent validity check it explicitly, and scalar_at remains a one-off probe shim during the transition. The probe benchmark now includes Bool and VarBinView.

Changes

Milestones

Each remaining migration will be a separate reviewable commit at the agreed milestone boundary.

• Simple: Bool, Constant, Decimal, Null, VarBinView, ByteBool, Sequence, PythonVTable
• Structural: Filter, Interleave, ScalarFnValidity, Union, Variant, FixedSizeList, List, ListView, VarBin, DecimalByteParts
• Map
• Patched
• PiecewiseSequence
• ScalarFn
• ALPRD
• BitPacked
• Delta
• OnPair
• ParquetVariant
• Sparse
• Zstd
• ZstdBuffers
• Make probe_scalar required; deprecate and remove OperationsVTable::scalar_at
• Address the public ArrayRef scalar and validity accessors on the later-release schedule in Migrate every encoding from OperationsVTable::scalar_at to probe_scalar #9971

ScalarFn has a probe_scalar override today, but it still calls scalar_at and is therefore unfinished. ScalarFnValidity is an additional internal target. TransposedBool is listed in the issue but has no implementation in the current tree.

Preparation and state

() means no encoding-owned cache; a repeated probe can still retain validity and child probes. Pending designs are marked TBD. Candidate reuse paths are listed to guide each milestone, not to commit to a cache before its benchmark and correctness review.

Encoding	Preparation to reuse	ProbeState
Bool	Generic retained validity	()
Constant	None; the stored scalar carries nullness	()
Decimal	Generic retained validity	()
Null	None; every value is null	()
VarBinView	Generic retained validity	()
ByteBool	Generic retained validity	()
Sequence	None; values are computed directly	()
PythonVTable	None; scalar access remains unsupported	()
Filter	TBD — source child probe	TBD
Interleave	TBD — selector and selected-value child probes	TBD
ScalarFnValidity	TBD — child scalar probe	TBD
Union	TBD — type-ID and selected-variant probes	TBD
Variant	TBD — core and shredded-value probes	TBD
FixedSizeList	TBD — element probe across the row’s range	TBD
List	TBD — offset and element probes	TBD
ListView	TBD — offset, size, and element probes	TBD
VarBin	TBD — offset probe	TBD
DecimalByteParts	TBD — high- and low-part child probes	TBD
Map	TBD — entry, key, and value probes across its ListView/Struct layout	TBD
Patched	TBD — lane offsets, patch search, and base/patch probes	TBD
PiecewiseSequence	TBD — decoded pieces or child probes	TBD
ScalarFn	TBD — argument child probes and function-input preparation	TBD
ALPRD	TBD — patch lookup and left/right child probes	TBD
BitPacked	TBD — patch and blocked-offset probes	TBD
Delta	TBD — decoded chunk and child probes	TBD
OnPair	TBD — row-offset, code, and length probes; dictionary preparation is already memoized	TBD
ParquetVariant	TBD — metadata, raw-value, and typed-value probes	TBD
Sparse	TBD — patch search and patch-child probes	TBD
Zstd	TBD — validity ranks and decoded frames	TBD
ZstdBuffers	TBD — decoded inner array and its probe	TBD

Verification

• cargo nextest run -p vortex-array -p vortex-bytebool -p vortex-sequence: 4,201 passed, 1 skipped.
• cargo nextest run -p vortex-python: 35 passed.
• Bool and VarBinView were benchmarked before and after with both one-off and repeated reads. VarBinView medians were stable or lower; Bool measurements varied between runs, with no sustained regression established.
• Lint and formatting commands were not run.

Described in #9843

AI Usage

• The implementation is generated by GPT-6-Sol harnessed by CodeX, the design and decision was done by the human and all the modification was reviewed by the human.

[codspeed]

Merging this PR will degrade performance by 19.67%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

• bench_compare_primitive[(10000, 2)]

⚠️ 18 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• chunked_dict_primitive_into_canonical[f64, (1000, 1000, 10)]
• bench_compare_sliced_dict_primitive[(3333, 10000)]
• chunked_dict_primitive_canonical_into[f64, (1000, 1000, 10)]
• optimize_lookup_predicate[ids=256, shape=in_list]
• random_i128[0.01]
• bench_dict_mask[(0.9, 0.9)]
• chunked_dict_primitive_into_canonical[u64, (1000, 1000, 10)]
• multipolygon_mixed_validity
• filter_run_end[len=4096_run=16_density=0.1]
• decompress[u32, (1000, 256)]
• …and 8 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 3 regressed benchmarks
✅ 2215 untouched benchmarks
🆕 8 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	density_sweep_dense_runs[0.001]	28.5 µs	47 µs	-39.26%
❌	Simulation	filter_powerlaw_by_mostly_true[250000]	98.1 µs	145.1 µs	-32.42%
❌	WallTime	filter_avx2[64]	2.1 µs	2.3 µs	-10.83%
⚡	WallTime	starts_with_avx512[8, VarBin]	46.3 µs	40.7 µs	+13.76%
🆕	Simulation	once[bool]	N/A	185.2 µs	N/A
🆕	Simulation	once[interleave]	N/A	520 µs	N/A
🆕	Simulation	once[list]	N/A	1.2 ms	N/A
🆕	Simulation	once[varbinview]	N/A	196.2 µs	N/A
🆕	Simulation	repeated[bool]	N/A	156.1 µs	N/A
🆕	Simulation	repeated[interleave]	N/A	547.4 µs	N/A
🆕	Simulation	repeated[list]	N/A	1.1 ms	N/A
🆕	Simulation	repeated[varbinview]	N/A	172.9 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing jiengup:migrate-every-encoding (7477dcb) with develop (ed119bf)}