Add lambda expressions

[mhk197]

Summary

Adds lambdas as arguments of higher-order scalar functions, such as list_transform(list, x -> x + $.a) in the next PR.

• Lambda is syntax: parameter names and a body. It is not an expression; it only appears in Expression::HigherOrder { scalar_fn, children, lambdas }, a scalar function applied to its children and then to its lambdas.

• Binding a higher-order node asks the function for each lambda's parameter dtypes (ScalarFnVTable::lambda_params), binds the body, and turns every outer reference in it ($, a field path such as $.a, or a variable of an enclosing lambda) into an extra input. The outer references become extra children of the node:

  list_transform($.l, x -> x + $.a)   binds to   list_transform([$.l, $.a], (x, a) -> x + a)
  

  The result is an ordinary BoundExpression::Scalar, so field pruning, partitioning and array rules see the outer references as plain children.

• BoundLambda is the rewritten lambda, (x, a) -> x + a with its dtypes, held in the function's options. BoundLambda::evaluate(inputs) runs the body over arrays.

Function hooks

ScalarFnVTable gains two defaulted methods, like reduce or validity, that only higher-order functions implement:

• lambda_params: the parameter dtypes of a lambda, given the dtypes of the value arguments;
• with_lambdas: the function's options holding the bound lambdas.

The defaults reject lambdas, so binding a HigherOrder node with an ordinary function fails with "does not take lambda arguments". DuckDB works the same way: a scalar function carries an optional lambda-binding callback and keeps the bound body in its bind data.

Details

• An input of a nullable dtype also accepts its non-nullable form: a layout reader stepping into a nullable struct's fields narrows nullability after binding, and evaluate casts such inputs back.
• ArrayRef::apply binds a higher-order node before applying it.
• Not in this PR: serializing higher-order nodes (it returns an error), building bound lambdas without syntax, and traversals into lambda bodies.

Testing

• cargo clippy -p vortex-array --all-targets --all-features -- -D warnings
• cargo nextest run -p vortex-array
• cargo test --doc -p vortex-array
• cargo +nightly-2026-09-10 fmt -p vortex-array

[codspeed]

Merging this PR will degrade performance by 21.33%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 9 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• random_i128[0.01]
• compress_fsst[(500, 4, 8)]
• density_sweep_dense_runs[0.001]
• decompress[u32, (1000, 256)]
• filter_run_end[len=4096_run=16_density=0.1]
• decompress[u16, (4000, 256)]
• take_varbin
• multipolygon_mixed_validity
• ilike_contains

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 12 regressed benchmarks
✅ 2199 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	filter_neon[8]	1.1 µs	2.1 µs	-50.44%
❌	WallTime	threshold_neon[80]	4.6 µs	8.2 µs	-43.8%
❌	Simulation	decode_primitives[i64, (1000, 32)]	24.1 µs	38.3 µs	-37.07%
❌	WallTime	threshold_neon[64]	4.2 µs	6.7 µs	-36.79%
❌	WallTime	threshold_neon[8]	2.8 µs	3.9 µs	-27.82%
❌	WallTime	filter_neon[80]	3.1 µs	3.9 µs	-20.25%
❌	WallTime	filter_neon[64]	2.9 µs	3.6 µs	-19.52%
❌	WallTime	threshold_avx2[8]	1.9 µs	2.4 µs	-17.58%
❌	WallTime	threshold_avx2[64]	3 µs	3.4 µs	-12.74%
❌	WallTime	bitpack_blocked_compress_avx2	6.7 µs	7.6 µs	-11.59%
❌	WallTime	threshold_avx2[80]	3.3 µs	3.7 µs	-11.48%
❌	WallTime	scalar_subtract_neon	10.9 µs	12.2 µs	-10.47%
⚡	Simulation	sum_v2_i64	225.1 µs	194.2 µs	+15.92%
⚡	Simulation	sum_i64	225.3 µs	194.3 µs	+15.92%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/lambda-expressions (676b887) with mk/variable-expressions (5625bb8)}

[github-actions]

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days