Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
165 commits
Select commit Hold shift + click to select a range
4a83a46
Add the ALP floating-point codec
sfc-gh-pgaur Dec 4, 2025
526b415
Wire ALP into the Parquet encoder and decoder
sfc-gh-pgaur Dec 4, 2025
2bac3ef
Add an ALP encoding benchmark
sfc-gh-pgaur Dec 4, 2025
c4a0f53
Add floating-point datasets for the benchmark
sfc-gh-pgaur Dec 4, 2025
a580d2f
Add the ALP sources to CMake
sfc-gh-pgaur Dec 4, 2025
303c11d
Rename the ALP headers from .hpp to .h
sfc-gh-pgaur Dec 6, 2025
f83358a
Draw the ALP diagrams in ASCII
sfc-gh-pgaur Dec 7, 2025
be8d267
Rename the ALP .cpp files to .cc
sfc-gh-pgaur Dec 7, 2025
642d61a
Use Arrow's Doxygen style in the ALP docs
sfc-gh-pgaur Dec 7, 2025
d419ec8
Follow Arrow naming style in ALP
sfc-gh-pgaur Dec 7, 2025
59f1a67
Follow Arrow style in the ALP tests
sfc-gh-pgaur Dec 7, 2025
ed0a119
Update ALP callers for the renamed methods
sfc-gh-pgaur Dec 7, 2025
6aeb406
Fuse FOR and decode loop
sfc-gh-pgaur Dec 7, 2025
e7bb7c0
Reduce memory allocation in the decompress call
sfc-gh-pgaur Dec 7, 2025
1db3182
Try SIMD for ALP decoding
sfc-gh-pgaur Dec 8, 2025
261498a
Revert "Try SIMD for ALP decoding"
sfc-gh-pgaur Dec 8, 2025
9f7882f
Point CMake at the renamed .cc files
sfc-gh-pgaur Dec 8, 2025
5cc4fa6
Move the ALP datasets to parquet-testing
sfc-gh-pgaur Dec 8, 2025
465ecde
Update the ALP benchmark data path
sfc-gh-pgaur Dec 9, 2025
dd4836c
Use Arrow file naming for the ALP sources
sfc-gh-pgaur Dec 15, 2025
7fb7006
Update ALP includes for the renamed files
sfc-gh-pgaur Dec 15, 2025
6059bbb
Read ALP sizes from the page header
sfc-gh-pgaur Dec 15, 2025
2790163
Drop redundant ALP namespace qualifiers
sfc-gh-pgaur Dec 16, 2025
1a6bbf6
Better pack the compression block header
sfc-gh-pgaur Dec 16, 2025
0014b5a
Rename CompressionBlockHeader to AlpHeader
sfc-gh-pgaur Dec 16, 2025
7b05d39
Reorder ALP vector metadata for packing
sfc-gh-pgaur Dec 16, 2025
241b899
Add the ALP specification drafts
sfc-gh-pgaur Dec 31, 2025
33e75ca
Add more ALP codec tests
sfc-gh-pgaur Jan 2, 2026
bfe0913
Move the element count to the ALP page header
sfc-gh-pgaur Jan 12, 2026
1526c60
Keep the ALP format version at 1
sfc-gh-pgaur Jan 12, 2026
0e4792f
Set kAlpVersion back to 1
sfc-gh-pgaur Jan 12, 2026
e065e69
Derive each ALP vector's element count
sfc-gh-pgaur Jan 12, 2026
69ff2b4
Update the ALP specification drafts
sfc-gh-pgaur Jan 12, 2026
134e9ef
Copy ALP exception arrays into aligned storage
sfc-gh-pgaur Jan 13, 2026
985cdd8
Use SafeCopy for the FOR subtraction
sfc-gh-pgaur Jan 13, 2026
b31cd27
Add Parquet-level ALP tests
sfc-gh-pgaur Jan 13, 2026
75ea9e5
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur Jan 14, 2026
8c1f5b7
Size ALP vector headers by kStoredSize
sfc-gh-pgaur Jan 14, 2026
8c1e72e
Cap the ALP page element count at INT32_MAX
sfc-gh-pgaur Jan 14, 2026
2f0a563
Rename the ALP layout to IntegerEncoding
sfc-gh-pgaur Jan 14, 2026
82060b7
Rename kBitPack to kForBitPack
sfc-gh-pgaur Jan 14, 2026
ab5c6b4
Make AlpHeader exactly 8 bytes
sfc-gh-pgaur Jan 14, 2026
d1bcd28
Move ALP vector data toward the start
sfc-gh-pgaur Jan 15, 2026
f740ec2
Read ALP metadata through a wrapper
sfc-gh-pgaur Jan 15, 2026
04b991a
Separate the ALP and FOR metadata
sfc-gh-pgaur Jan 15, 2026
3b73329
Drop the padding byte from the FOR metadata
sfc-gh-pgaur Jan 16, 2026
cb5d981
Put FOR-specific code behind a switch
sfc-gh-pgaur Jan 16, 2026
189d04c
Test empty input and the FOR metadata size
sfc-gh-pgaur Jan 17, 2026
fcba8d5
Store per-vector offsets in ALP pages
sfc-gh-pgaur Jan 26, 2026
0261c9a
Document the offset-based ALP layout
sfc-gh-pgaur Jan 26, 2026
58bb648
Add ALP sampling presets
sfc-gh-pgaur Jan 27, 2026
607d72f
Update ALP comments for the offset layout
sfc-gh-pgaur Jan 27, 2026
adf80b6
Add an ALP reference blob generator
sfc-gh-pgaur Mar 7, 2026
65e43d9
Add a tool that writes ALP Parquet files
sfc-gh-pgaur Mar 7, 2026
45d23e2
Drop the version byte from the ALP header
sfc-gh-pgaur Mar 7, 2026
6d3a520
Add ALP test files to parquet-testing
sfc-gh-pgaur Mar 7, 2026
25a7ede
Test reading ALP-encoded Parquet files
sfc-gh-pgaur Mar 7, 2026
4c2f127
Test reading Java-written ALP files
sfc-gh-pgaur Mar 7, 2026
4296be7
Test FLOAT ALP files across languages
sfc-gh-pgaur Mar 8, 2026
a1be2f7
Check the Status returned by ALP Decode
sfc-gh-pgaur Apr 22, 2026
77c0204
Bounds-check each ALP vector before reading
sfc-gh-pgaur Apr 27, 2026
2182a48
Rename ALP types and tidy the codec API
sfc-gh-pgaur Apr 27, 2026
9886e8a
Fix narrowing and call sites after the renames
sfc-gh-pgaur Apr 27, 2026
4ec9771
Batch the ALP size estimate for vectorization
sfc-gh-pgaur Apr 27, 2026
8e8ef6f
Use signed sizes and byte spans in ALP
sfc-gh-pgaur May 6, 2026
8d30342
Move decoder creation outside benchmark loop
sfc-gh-pgaur May 6, 2026
5ebc724
Rename alp_wrapper to alp_codec
sfc-gh-pgaur May 6, 2026
c1185a2
Return Status from ALP decode and load paths
sfc-gh-pgaur May 6, 2026
1443bde
Document the ALP Store preconditions
sfc-gh-pgaur May 6, 2026
4eb1c28
Use signed integer types in ALP
sfc-gh-pgaur May 6, 2026
10ba334
Put ALP out parameters last
sfc-gh-pgaur May 7, 2026
9259112
Remove the ALP specification drafts
sfc-gh-pgaur May 13, 2026
029349b
Rename ALP codec params to input and output
sfc-gh-pgaur May 13, 2026
362fcdf
Copy ALP header fields one at a time
sfc-gh-pgaur May 13, 2026
8ee30f0
Decode any power-of-two ALP vector size
sfc-gh-pgaur May 13, 2026
169527a
Make the ALP encode vector size configurable
sfc-gh-pgaur May 13, 2026
79809ff
Test Parquet ALP decoding at other vector sizes
sfc-gh-pgaur May 13, 2026
51a7e99
Use int64_t sizes in the AlpCodec API
sfc-gh-pgaur May 14, 2026
95fad0c
Round negative values correctly in FastRound
sfc-gh-pgaur Jun 3, 2026
1a8ac27
Use SafeLoad/SafeStore and classes in ALP
sfc-gh-pgaur Jun 8, 2026
f71025d
Make the ALP vector members private
sfc-gh-pgaur Jun 8, 2026
657d21f
Remove AlpMetadataCache and GetNumElements
sfc-gh-pgaur Jun 8, 2026
af4cca7
Drop private inheritance from ALP classes
sfc-gh-pgaur Jun 8, 2026
90fb841
Spell out LoadHeader's return type
sfc-gh-pgaur Jun 8, 2026
133f369
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur Jun 8, 2026
ff04d74
Validate header fields in LoadHeader
sfc-gh-pgaur Jun 8, 2026
4598521
Count AlpCodec input in elements, not bytes
sfc-gh-pgaur Jun 8, 2026
e53e420
Use uint8_t* in the AlpCodec API
sfc-gh-pgaur Jun 8, 2026
2d5d7fd
Drop trailing return types in ALP
sfc-gh-pgaur Jun 25, 2026
4015a9e
Cite the ALP paper for the tie-break rule
sfc-gh-pgaur Jun 25, 2026
d48496b
Pin AlpMode's underlying type to uint8_t
sfc-gh-pgaur Jun 25, 2026
d9c87f4
Tweak ALP docs, add an incremental-decode TODO
sfc-gh-pgaur Jun 25, 2026
edd18bb
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur Jun 25, 2026
a81b637
Drop the issue link from the decode TODO
sfc-gh-pgaur Jun 25, 2026
88dff38
Document how ALP encode reports bad input
sfc-gh-pgaur Jun 25, 2026
363d0b5
Use bit_util::CeilDiv for ALP vector counts
sfc-gh-pgaur Jun 25, 2026
d00cab8
Compare ALP test output with IsBitwiseEqual
sfc-gh-pgaur Jun 25, 2026
29d00cb
Type-parameterize the ALP integration tests
sfc-gh-pgaur Jun 26, 2026
058add5
Type-parameterize the AlpEncodingAdHoc tests
sfc-gh-pgaur Jun 26, 2026
44426ea
Widen the RandomData range to hit exceptions
sfc-gh-pgaur Jun 26, 2026
b0e8965
Remove the uncompiled ALP generator tools
sfc-gh-pgaur Jun 26, 2026
51cc19d
Add an incremental-encode TODO to AlpEncoder
sfc-gh-pgaur Jun 26, 2026
2042bac
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur Jun 26, 2026
6209f19
Use bit_util helpers for ALP bit math
sfc-gh-pgaur Jun 26, 2026
427c03e
Remove 'non-templated' from ALP comments
sfc-gh-pgaur Jun 26, 2026
968ca17
Inline GetBitPackedSize at its callers
sfc-gh-pgaur Jun 26, 2026
8a8038c
Assert kStoredSize against the member sizes
sfc-gh-pgaur Jun 28, 2026
ca187cc
Return Status from the AlpCodec encode API
sfc-gh-pgaur Jun 30, 2026
51a6ff0
Load ALP benchmark data with the CSV reader
sfc-gh-pgaur Jun 30, 2026
34aa9b4
Derive kStoredSize from the member sizes
sfc-gh-pgaur Jun 30, 2026
48e6526
Make ALP exception counts unsigned
sfc-gh-pgaur Aug 12, 2026
8c346f2
Enforce the [3, 15] ALP log_vector_size range
sfc-gh-pgaur Aug 12, 2026
3e0603b
Use array matchers in the ALP tests
sfc-gh-pgaur Aug 12, 2026
a2888f5
Document the ALP vector_size range
sfc-gh-pgaur Aug 14, 2026
a1235e7
Put EncodeAlp output parameters last
sfc-gh-pgaur Aug 24, 2026
e6e5cd4
Test that ALP rejects malformed page headers
sfc-gh-pgaur Aug 24, 2026
0865523
Use std::countr_zero in ALP
sfc-gh-pgaur Aug 24, 2026
5c9bf12
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur Aug 24, 2026
3e6ad01
Validate ALP vector metadata before decoding
sfc-gh-pgaur Aug 26, 2026
0d347fd
Correct inaccurate ALP comments
sfc-gh-pgaur Aug 26, 2026
e97584c
Type-parameterize the ALP compression tests
sfc-gh-pgaur Aug 27, 2026
80a99cd
Drop the ALP benchmark that needs fork data
sfc-gh-pgaur Aug 31, 2026
4bf8b21
Test ALP against the conformance file
sfc-gh-pgaur Aug 31, 2026
4926c77
Put ALP behind an opt-in writer flag
sfc-gh-pgaur Aug 31, 2026
7b7b520
Tighten the ALP tests and document the encoding
sfc-gh-pgaur Sep 1, 2026
cae830f
Validate the ALP count and offset chain
sfc-gh-pgaur Sep 1, 2026
cb34d5e
Take the ALP count from the page header
sfc-gh-pgaur Sep 1, 2026
ae23e6e
Encode an all-null ALP page as a bare header
sfc-gh-pgaur Sep 1, 2026
751f8b7
Build the ALP sources once, and under meson too
sfc-gh-pgaur Sep 1, 2026
843fb96
Serialize ALP as little-endian on any host
sfc-gh-pgaur Sep 1, 2026
9beaafc
Move AlpMode out of the public header
sfc-gh-pgaur Sep 1, 2026
f61722f
Test the ALP count and offset validation
sfc-gh-pgaur Sep 1, 2026
e7fec3d
Test batched ALP decoding
sfc-gh-pgaur Sep 1, 2026
44e6db1
Decode ALP pages one vector at a time
sfc-gh-pgaur Sep 1, 2026
1cdd4af
Move the ALP end-to-end tests to their own file
sfc-gh-pgaur Sep 1, 2026
50b5d52
Name the ALP headers _internal.h
sfc-gh-pgaur Sep 1, 2026
ac261dd
Fold the ALP vector-size test into the suite
sfc-gh-pgaur Sep 3, 2026
9bb9201
Reuse ALP decode scratch across vectors
sfc-gh-pgaur Sep 3, 2026
79984fb
Drop dead ALP code and unpause the benchmarks
sfc-gh-pgaur Sep 3, 2026
734f70f
Export AlpCodec and flatten its namespace
sfc-gh-pgaur Sep 3, 2026
ca0507c
Make AlpSampler's constants static
sfc-gh-pgaur Sep 3, 2026
0702557
Guard the ALP loop pragmas by compiler
sfc-gh-pgaur Sep 7, 2026
9d6e8ae
Export the nested VectorReader too
sfc-gh-pgaur Sep 7, 2026
b3d79de
Reject ALP values beyond the definition levels
sfc-gh-pgaur Sep 7, 2026
1455886
Size the sample buffer with integer division
sfc-gh-pgaur Sep 8, 2026
aa2e431
Describe the decode target type in prose
sfc-gh-pgaur Sep 8, 2026
4219b5b
Export the ALP decode member templates
sfc-gh-pgaur Sep 8, 2026
d9fa958
Cast the element count the ALP test passes
sfc-gh-pgaur Sep 8, 2026
4df7226
Export the internal ALP classes for tests
sfc-gh-pgaur Sep 8, 2026
c32fd46
Widen the shift the ALP test takes a modulus with
sfc-gh-pgaur Sep 8, 2026
9f2b73e
Rename the ALP roundtrip helper
sfc-gh-pgaur Sep 8, 2026
2e356bc
Read the stored size through its accessor
sfc-gh-pgaur Sep 8, 2026
01b6386
Skip the byte copies that have nothing to copy
sfc-gh-pgaur Sep 9, 2026
bbba931
Keep ALP bounds one ulp below the int limit
sfc-gh-pgaur Sep 10, 2026
df83dfb
Drop the parquet-testing submodule bump
sfc-gh-pgaur Sep 15, 2026
e0bcc57
Trim the ALP comments
sfc-gh-pgaur Sep 19, 2026
892b195
State the constraints the ALP tests check
sfc-gh-pgaur Sep 19, 2026
3aff81c
GH-48701: [C++][Parquet] Refactor and harden ALP encoding
wgtmac Sep 21, 2026
c0dd888
Restore ALP wire layout diagrams
sfc-gh-pgaur Sep 27, 2026
ed6ed63
Document the metadata layouts at the structs
sfc-gh-pgaur Sep 27, 2026
51e36be
Test the factory throw; keep the fallback TODO
sfc-gh-pgaur Sep 27, 2026
dff2ea5
Document that ALP is never chosen automatically
sfc-gh-pgaur Sep 28, 2026
642ee4e
Fix ALP builds across compilers
sfc-gh-pgaur Sep 29, 2026
2fb0c5f
Fix constrained ALP template exports
sfc-gh-pgaur Sep 29, 2026
e2275a1
Fix ALP vector metadata bounds on 32-bit targets
sfc-gh-pgaur Oct 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -550,6 +550,10 @@ arrow_add_object_library(ARROW_VENDORED ${ARROW_VENDORED_SRCS})

set(ARROW_UTIL_SRCS
util/align_util.cc
util/alp/alp_metadata.cc
util/alp/alp_compression.cc
util/alp/alp_sampler.cc
util/alp/alp_codec.cc
util/async_util.cc
util/atfork_internal.cc
util/basic_decimal.cc
Expand Down
4 changes: 4 additions & 0 deletions cpp/src/arrow/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,10 @@ arrow_components = {

arrow_util_srcs = [
'util/align_util.cc',
'util/alp/alp_metadata.cc',
'util/alp/alp_compression.cc',
'util/alp/alp_sampler.cc',
'util/alp/alp_codec.cc',
'util/async_util.cc',
'util/atfork_internal.cc',
'util/basic_decimal.cc',
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/util/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ add_arrow_test(bit-utility-test
rle_encoding_test.cc
test_common.cc)

add_arrow_test(encoding-utility-test SOURCES alp/alp_test.cc)

add_arrow_test(crc32-test
SOURCES
crc32_test.cc
Expand Down
461 changes: 461 additions & 0 deletions cpp/src/arrow/util/alp/alp_codec.cc

Large diffs are not rendered by default.

125 changes: 125 additions & 0 deletions cpp/src/arrow/util/alp/alp_codec_internal.h
Comment thread
prtkgaur marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#pragma once

#include <cstdint>
#include <span>

#include "arrow/result.h"
#include "arrow/status.h"
#include "arrow/util/alp/alp_compression_internal.h"
#include "arrow/util/alp/alp_metadata_internal.h"
#include "arrow/util/visibility.h"

namespace arrow::util::alp {

/// ALP compression and decompression codec.
template <AlpFloatingType T>
class ARROW_EXPORT AlpCodec {
public:
AlpCodec() = delete;

/// Creates an encoding preset by sampling `inputs`.
static Result<AlpEncodingPreset> MakePreset(std::span<const T> inputs);

/// Encodes `inputs` using a pre-computed preset.
static Result<int64_t> Encode(std::span<const T> inputs,
const AlpEncodingPreset& preset, int32_t vector_size,
std::span<uint8_t> output);

/// Samples a preset and encodes `inputs`.
static Result<int64_t> Encode(std::span<const T> inputs, int32_t vector_size,
std::span<uint8_t> output);

/// Returns the maximum encoded size in bytes.
static Result<int64_t> GetMaxCompressedSize(int64_t num_elements, int32_t vector_size);
};

/// Random access to the vectors of one compressed ALP page.
template <AlpFloatingType T>
class ARROW_EXPORT AlpVectorReader {
public:
/// Opens a page and validates its header and offset table.
///
/// The buffer referenced by `input` must outlive the reader.
static Result<AlpVectorReader> Open(std::span<const uint8_t> input, MemoryPool* pool);

/// Number of values the header declares.
int32_t num_elements() const { return num_elements_; }

/// Number of values in every vector but the last.
int32_t vector_size() const { return vector_size_; }

/// Number of vectors the buffer holds.
int32_t num_vectors() const { return static_cast<int32_t>(vector_offsets_.size()); }

/// Returns the number of values in `vector_index`.
Result<int32_t> VectorLength(int32_t vector_index) const;

/// Decodes one vector.
///
/// `output` must contain exactly `VectorLength(vector_index)` elements.
/// `TargetType` may be wider than `T`, but not narrower.
template <typename TargetType>
Status Decode(int32_t vector_index, std::span<TargetType> output);

private:
explicit AlpVectorReader(MemoryPool* pool)
: vector_offsets_(AlpAllocator<AlpFormatConstants::OffsetType>(pool)),
decode_view_(pool),
unpacked_integers_(
AlpAllocator<typename AlpTypedConstants<T>::EncodedUnsigned>(pool)) {}

/// Metadata and data bounds for one vector.
struct VectorLayout {
AlpInfo alp_info;
AlpForInfo<T> for_info;
/// First byte of the bit-packed values, after both metadata blocks.
const uint8_t* data{nullptr};
int64_t data_size{0};
int32_t num_elements{0};
};

/// Reads and bounds-checks one vector's metadata.
Result<VectorLayout> LoadVectorLayout(int32_t vector_index) const;

/// Returns the total encoded size of one vector.
Result<int64_t> VectorSizeInBytes(int32_t vector_index) const;

/// Start of the offset table; vector offsets are relative to it.
const uint8_t* body_{nullptr};
int64_t body_size_{0};
int32_t num_elements_{0};
int32_t vector_size_{0};
/// Byte offset of each vector.
AlpVector<AlpFormatConstants::OffsetType> vector_offsets_;
/// Reused exception arrays for vector decoding.
AlpEncodedVectorView<T> decode_view_;
/// Reused scratch space for unpacked integers.
AlpVector<typename AlpTypedConstants<T>::EncodedUnsigned> unpacked_integers_;
};

// Member templates need explicit export instantiations.
extern template ARROW_TEMPLATE_EXPORT Status
AlpVectorReader<float>::Decode<float>(int32_t, std::span<float>);
extern template ARROW_TEMPLATE_EXPORT Status
AlpVectorReader<float>::Decode<double>(int32_t, std::span<double>);
extern template ARROW_TEMPLATE_EXPORT Status
AlpVectorReader<double>::Decode<double>(int32_t, std::span<double>);

} // namespace arrow::util::alp
Loading
Loading