Repository navigation
feat(array): min/max for fixed-size lists, so UUID columns get zone-map stats - #10457
XiangpengHao wants to merge 4 commits into
Conversation
…ap stats `vortex.uuid` is stored as `FixedSizeList(U8, 16)`, and the min/max aggregate skipped fixed-size lists as a stop-gap (vortex-data#8243), so UUID zone maps and footers kept only null counts and `uuid = x` pruned nothing. Compute min/max for fixed-size lists of fixed-width elements (bool, primitive, decimal), ordered by the compare kernel's own row comparator so statistics and comparisons always agree. Non-nullable byte lists such as UUIDs compare as byte slices. Lists wider than 64 bytes per value get none, so embedding-style columns do not grow zone maps. DuckDB's COPY RETURN_STATS and the JNI writer now convert UUID and fixed-size-list bounds instead of failing on them. Signed-off-by: XiangpengHao <[email protected]>
Signed-off-by: XiangpengHao <[email protected]>
Merging this PR will degrade performance by 15.6%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | density_sweep_dense_runs[0.001] |
29.6 µs | 48 µs | -38.37% |
| ❌ | WallTime | threshold_neon[8] |
2.9 µs | 3.6 µs | -18.31% |
| ❌ | WallTime | threshold_neon[64] |
4.4 µs | 5.2 µs | -15.33% |
| ❌ | WallTime | threshold_neon[80] |
4.8 µs | 5.6 µs | -14.91% |
| ❌ | WallTime | bitpack_blocked_compress_avx512 |
4.8 µs | 5.6 µs | -14.26% |
| ❌ | Simulation | random_i8[0.01] |
108.5 µs | 123.7 µs | -12.35% |
| ⚡ | WallTime | threshold_avx512[8] |
2 µs | 1.8 µs | +11.96% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing XiangpengHao:xiangpeng/fsl-min-max (bb1551d) with develop (58d0490)
| @@ -473,14 +498,14 @@ impl AggregateFnVTable for MinMax { | |||
| Canonical::VarBinView(v) => accumulate_varbinview(args, partial, v, ctx), | |||
| Canonical::Decimal(d) => accumulate_decimal(args, partial, d, ctx), | |||
| Canonical::Extension(e) => accumulate_extension(args, partial, e, ctx), | |||
There was a problem hiding this comment.
we missed this before we need cannot just take min/max of stroage type need to check it okay (i.e. good for time, uuid, and a few others.)
There was a problem hiding this comment.
That makes sense, fixed, but not sure if it's the best place to do it.
… own Min/max of an extension array is computed on its storage, which is only meaningful when the type orders like its storage. Add `ExtVTable::orders_by_storage` (default `false`) and opt in timestamp, date, time and UUID. Other extension types, such as JSON, geometries, vectors and unknown foreign types, get no extrema in any encoding. Signed-off-by: XiangpengHao <[email protected]>
This branch has not been deployed
3 participants
Summary
vortex.uuidis stored asFixedSizeList(U8, 16), and the min/max aggregate skips fixed-size lists as a stop-gap (#8243, for #8189). UUID zone maps and file footers therefore keep only null counts, anduuid_col = xprunes nothing, which hurts sorted UUIDv7 keys most. This implements min/max for fixed-size lists, so UUID columns get the same zone-map and footer statistics, and the same pruning, as Parquet's UUID logical type.Changes
FixedSizeList(aggregate_fn/fns/min_max/fixed_size_list.rs): lists of fixed-width elements (bool, primitive, decimal), ordered lexicographically by the compare kernel's own row comparator (compare/nested.rs), so statistics and comparisons always agree. Null lists are skipped, null elements order first, floats use total order. Non-nullableu8lists (the UUID case) compare as byte slices, the same order as Postgres, DuckDB and Parquet.MAX_FIXED_SIZE_LIST_MIN_MAX_BYTES(64 bytes per value) get no min/max, so embedding-style columns don't grow zone maps for pruning they can't use. The result is the same for every encoding.vortex.uuidgets min/max through its storage; no UUID-specific code.COPY … RETURN_STATSconverts UUID bounds toUUIDand fixed-size-list bounds toARRAYinstead of failing on them; the JNI writer converts them tojava.util.UUIDandList.vortex/tests/uuid_pruning.rs(a UUID column over four zones;uuid = xkeeps only the zone that can hold it, footer min/max,can_prune),fixed_size_list::tests(ordering vsScalar::partial_cmpandcompare, nulls, floats, sliced and dictionary inputs, the cap at its boundary),vortex-uuidmin/max order, DuckDB conversion tests andcopy_return_stats.slt.JNIWriterTest.testUuidAndFixedSizeListBoundsis added but was not run locally (no JDK).API Changes
Files written with this change carry min/max for UUID and narrow fixed-size-list columns. Engines that receive file statistics (DuckDB
RETURN_STATS, the JNI writer's column statistics) now see bounds for those columns.