Repository navigation
fix: upgrade to apache-arrow 24.0.0 and enable CI run success - #922
Conversation
Signed-off-by: Jason <[email protected]>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #922 +/- ##
============================================
+ Coverage 76.80% 80.55% +3.74%
Complexity 615 615
============================================
Files 84 94 +10
Lines 8886 10742 +1856
Branches 1054 1060 +6
============================================
+ Hits 6825 8653 +1828
- Misses 1821 1849 +28
Partials 240 240
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Pre-commit Checks / Pre-commit Checks (pull_request_target)Pre-commit Checks / Pre-commit Checks (pull_request_target) the old configuration not install pre-commit, so it failed. After this PR merged, the problem will automaticly resolve. |
Signed-off-by: Jason <[email protected]>
Signed-off-by: Jason <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR upgrades GraphAr’s C++/CI integration to Apache Arrow 24.0.0 and updates the codebase and workflows to address Arrow 24 deprecations and CI failures (including initialization changes for Arrow compute/dataset usage).
Changes:
- Update C++ (and C++ tests) to use Arrow/Parquet APIs compatible with Arrow 24 (e.g.,
ReadTable()returningarrow::Result). - Add initialization calls for Arrow compute and dataset execution paths to avoid runtime/compile issues with newer Arrow.
- Update GitHub Actions workflows to install/use Arrow 24 across CI and adjust Python wheel publishing/build steps.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
cpp/test/test_builder.cc |
Switch Parquet ReadTable usage to Arrow 24 Result API in tests. |
cpp/test/test_arrow_chunk_writer.cc |
Switch Parquet ReadTable usage to Arrow 24 Result API in tests. |
cpp/src/graphar/reader_util.cc |
Add Arrow compute initialization before evaluating/inspecting filter expressions. |
cpp/src/graphar/high-level/graph_reader.cc |
Add dataset/compute initialization in EdgesCollection::Make. |
cpp/src/graphar/filesystem.cc |
Add dataset/compute initialization helper; migrate to Arrow 24 ReadTable() API. |
cpp/src/graphar/expression.h |
Ensure compute initialization happens before creating literals inline. |
cpp/src/graphar/expression.cc |
Add centralized compute initialization helper and call it from expression evaluators. |
cpp/src/graphar/arrow/chunk_writer.cc |
Initialize compute/dataset prior to using Acero plan-based sorting. |
cpp/src/graphar/arrow/chunk_reader.cc |
Initialize compute before Cast usage in reader utilities. |
cpp/src/CMakeLists.txt |
Add LibXml2 linkage for static Arrow builds on non-Apple/non-MSVC. |
cpp/CMakeLists.txt |
Update Arrow version threshold logic for selecting C++20. |
cpp/cmake/apache-arrow.cmake |
Bump default Arrow version to build from source to 24.0.0. |
.github/workflows/rust.yaml |
Install Arrow 24 packages in Rust CI job. |
.github/workflows/python.yml |
Install Arrow 24 packages and update checkout action version. |
.github/workflows/python-wheel-workflow.yml |
Adjust macOS wheel build environment/setup; pin PyPI publish action. |
.github/workflows/pre-commit.yml |
Switch trigger to pull_request, update checkout action, add pre-commit installation steps. |
.github/workflows/ci.yml |
Install Arrow 24 packages; update checkout action version; remove clang-format step from main CI job. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -41,9 +41,15 @@ jobs: | |||
| runs-on: ubuntu-slim | |||
| - name: Install Arrow (macOS) | ||
| if: matrix.os == 'macos' | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
| mamba install -y -c conda-forge arrow-cpp | ||
| # Note: CONDA_PREFIX may be unset unless conda is activated in this shell. | ||
| # setup-miniconda exports CONDA (base install prefix), which is sufficient here. | ||
| echo "CMAKE_PREFIX_PATH=$CONDA" >> "$GITHUB_ENV" | ||
| brew install apache-arrow | ||
| echo "CMAKE_PREFIX_PATH=/usr/local" >> "$GITHUB_ENV" | ||
| # Optional sanity check: ensure Arrow dylib isn't built for a newer macOS than deployment target. | ||
| if command -v otool >/dev/null 2>&1; then | ||
| ls -lah "$CONDA/lib" || true | ||
| if [ -f "$CONDA/lib/libarrow.dylib" ]; then | ||
| otool -l "$CONDA/lib/libarrow.dylib" | (grep -A3 -E 'LC_BUILD_VERSION|LC_VERSION_MIN_MACOSX' || true) | ||
| ls -lah /usr/local/lib || true | ||
| if [ -f "/usr/local/lib/libarrow.dylib" ]; then | ||
| otool -l "/usr/local/lib/libarrow.dylib" | (grep -A3 -E 'LC_BUILD_VERSION|LC_VERSION_MIN_MACOSX' || true) | ||
| fi |
| CIBW_MANYLINUX_AARCH64_IMAGE: ${{ matrix.os == 'linux' && format('manylinux{0}', matrix.manylinux) || '' }} | ||
| CIBW_ENVIRONMENT_WINDOWS: DISTUTILS_USE_SDK=1 MSSdk=1 | ||
| CIBW_ENVIRONMENT_MACOS: ${{ matrix.os == 'macos' && format('MACOSX_DEPLOYMENT_TARGET={0} CMAKE_OSX_DEPLOYMENT_TARGET={0} CFLAGS=-mmacosx-version-min={0} CXXFLAGS=-mmacosx-version-min={0} LDFLAGS=-mmacosx-version-min={0}', matrix.deployment-target) || '' }} | ||
| CIBW_ENVIRONMENT_MACOS: ${{ matrix.os == 'macos' && 'MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion) CMAKE_OSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion) CFLAGS=-mmacosx-version-min=$(sw_vers -productVersion) CXXFLAGS=-mmacosx-version-min=$(sw_vers -productVersion) LDFLAGS=-mmacosx-version-min=$(sw_vers -productVersion)' || '' }} |
| Status EnsureComputeInitialized() { | ||
| static bool initialized = false; | ||
| if (!initialized) { | ||
| RETURN_NOT_ARROW_OK(arrow::compute::Initialize()); | ||
| initialized = true; | ||
| } | ||
| return Status::OK(); |
| Status EnsureDatasetScannerInitialized() { | ||
| static bool initialized = false; | ||
| static Status init_status = Status::OK(); | ||
| if (!initialized) { | ||
| auto st = arrow::compute::Initialize(); | ||
| if (!st.ok()) { | ||
| init_status = Status::ArrowError(st.ToString()); | ||
| } else { | ||
| arrow::dataset::internal::Initialize(); | ||
| } | ||
| initialized = true; | ||
| } | ||
| return init_status; |
| static bool initialized = false; | ||
| if (!initialized) { | ||
| RETURN_NOT_ARROW_OK(arrow::compute::Initialize()); | ||
| arrow::dataset::internal::Initialize(); | ||
| initialized = true; | ||
| } |
| static bool initialized = false; | ||
| if (!initialized) { | ||
| RETURN_NOT_ARROW_OK(arrow::compute::Initialize()); | ||
| initialized = true; | ||
| } |
| Status GeneralCast(const std::shared_ptr<arrow::Array>& in, | ||
| const std::shared_ptr<arrow::DataType>& to_type, | ||
| std::shared_ptr<arrow::Array>* out) { | ||
| static bool initialized = false; | ||
| if (!initialized) { | ||
| RETURN_NOT_ARROW_OK(arrow::compute::Initialize()); | ||
| initialized = true; | ||
| } |
| Status CastTableWithSchema(const std::shared_ptr<arrow::Table>& table, | ||
| const std::shared_ptr<arrow::Schema>& schema, | ||
| std::shared_ptr<arrow::Table>* out_table) { | ||
| static bool initialized = false; | ||
| if (!initialized) { | ||
| RETURN_NOT_ARROW_OK(arrow::compute::Initialize()); | ||
| initialized = true; | ||
| } |
Reason for this PR
This commit fixes deprecation errors when compiling with Apache Arrow 24.0.0+.
close: #920
What changes are included in this PR?
ReadTable(Table**)witharrow::Resultversion in cppAre these changes tested?
yes
Are there any user-facing changes?
no