Skip to content

[To rc/2.5.0] Add multi-platform native releases and installed artifact tests - #1012

Merged
ColinLeeo merged 8 commits into
apache:rc/2.5.0from
ColinLeeo:develop
Oct 22, 2026
Merged

ColinLeeo merged 8 commits into
apache:rc/2.5.0from
ColinLeeo:develop

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Native SDKs, platform packages, and Python wheels need a single release workflow that verifies the files users actually download and install. This PR adds that workflow for 2.5.0 and replaces the separate Python wheel workflow.

Changes

  • Set the C++ and Python release versions to 2.5.0, with an explicit option to preserve those versions during Maven builds.
  • Export an installed CMake package and pkg-config metadata so consumers and C++ regression tests can use the installed SDK without building the library or CLI from source.
  • Use BUNDLED third-party dependencies for every native release package and as the CMake/Maven source-build default. Homebrew Formulae no longer require separate codec libraries. Validate Mach-O load commands in built bottles, staged SDKs and downloaded SDKs, rejecting external third-party dylibs even when present on the runner.
  • Build native SDKs and packages for Linux, macOS, and Windows on x86_64 and ARM64. Produce DEB runtime/development/tools packages, RPM runtime/development/tools packages, Windows ZIP packages, and Homebrew bottles with a merged Formula.
  • Build six CPython ABIs (3.9–3.14) on the five Python platforms: Linux and macOS on both architectures, plus Windows x86_64. Build the four Unix compatibility SDKs once per platform and reuse the native Windows SDK for its wheels.
  • Download and install the outputs in separate jobs. Run the existing C++ API tests through find_package, CLI operations against the four existing python/tests/resources TsFile files, Go's own vet/tests and supported race checks, and the full Python pytest suite against installed wheels. Also relocate and test every downloadable SDK archive.
  • Assemble all outputs only after the installation tests pass. Include source/version metadata, manifest.json, and SHA256SUMS, and require persistent JUnit reports for every Python wheel test.
  • Export share/tsfile/tsfile-build-config.json from the SDK and record the resolved CRT, compiler, configuration and runtime filename. Python reads it automatically, without a separate runtime environment variable.
  • Ship one native library per wheel, exclude SDK headers/import libraries/Cython build files, strip Unix wheel binaries before platform repair, retain license texts, and validate the content of every built and downloaded wheel. Preserve the build helper and Cython inputs in the sdist.
  • Add bounded retries for transient CMake/Maven dependency downloads.

The workflow is manually dispatched and uploads build artifacts; package-registry publication is a separate step. Python retains the existing five-platform scope, without Windows ARM64 wheels. CLI golden files and the existing CLI E2E fixtures are unchanged.

Commit structure

  1. b2ca340e0 — Bump C++ and Python release versions to 2.5.0.
  2. 3f45b0822 — Fix portable C++ builds and installed SDK consumption.
  3. 0ed880344 — Add the complete multi-platform native release workflow.
  4. d3a40dbb9 — Bundle third-party dependencies across all native packages and reject external macOS dylib references.
  5. 8db59d317 — Make release-version tests independent of the checkout version.
  6. 59aed648d — Fix native SDK linkage and CLI runtime paths, including static pkg-config dependencies, build-tree RPATH, Linux Debug parser macros, and installed-consumer regressions.
  7. f011e1a5e — Merge rc/2.5.0, resolve the python/setup.py conflict, and align Maven versions with the 2.5.0 release branch.

Validation

The current head, f011e1a5efece5ab4546de1a1c81839e67812074, includes the review fixes already merged on develop while retaining the 2.5.0 C++ and Python versions. Local packaging regression tests pass: 67 passed, 7 existing opt-in skips, and 175 subtests passed. The PR now merges cleanly into rc/2.5.0, and the full Apache PR CI is in progress.

Earlier local macOS ARM64 validation built the default bundled SDK and tested it after archive relocation: 222 C++ tests passed with 3 existing disabled cases; CLI checks on all four fixtures and Go vet/tests/race passed. The Mach-O check rejects previous Homebrew-derived SDKs and accepts the bundled SDK. Workflow contract, actionlint (with the existing newer runner labels allowed), formatting and dependency-policy checks passed.

The results below describe commit 0ed880344590ba575101a1fc64d971f8bb642e8f and its artifacts.

The previous commit, 0ed880344590ba575101a1fc64d971f8bb642e8f, passed the complete native release workflow: 60/60 jobs successful. An AlmaLinux mirror metadata outage interrupted the first RPM installation attempt; rerunning that task and dependent assembly succeeded without changing code or test coverage.

Validation Verified result
SDK configuration All 12 downloaded SDK variants contain versioned build metadata, the resolved CRT and the runtime filename; Windows SDKs record MultiThreaded
Native installation and SDK relocation C++, CLI and Go passed in all 24 package/SDK scenarios; Go race passed in the 22 supported scenarios
C++ reports 24 downloaded reports; 5,328 passed and 72 existing disabled cases across the reports; zero failures
Python reports All 30 downloaded reports; 18,019 passed and 41 existing platform/Python-version skips; zero failures
Wheel contents All 30 actual downloaded wheels contain one TsFile runtime, no SDK headers/import libraries/Cython build inputs or ELF debug sections; versions, ABIs, binary architectures and Windows PE dependencies verified
Source build compatibility Ordinary Python CI and MSVC Python CI both passed, including default dynamic-CRT builds; compiler inheritance fixed for isolated ANTLR dependency tests
Local packaging checks 68 tests: 63 passed and 5 existing opt-in skips; workflow contract, actionlint, formatting and Apache RAT passed
Source distribution Built an sdist and then a runtime-only wheel from that sdist; repaired/installed macOS ARM64 CPython 3.12 wheel passed 601 tests with 1 existing platform skip
Final bundle Downloaded all 61 manifest entries and verified SHA256, sizes, source commit, versions, SDK metadata, package contents and binary architectures

Actual wheel sizes in the final bundle (decimal MB):

Platform CPython 3.9–3.14 wheel size
manylinux_2_28 x86_64 2.21–2.23 MB
manylinux_2_28 aarch64 2.07–2.10 MB
macOS 12 x86_64 1.88–1.89 MB
macOS 12 arm64 1.62–1.63 MB
Windows x86_64 2.11–2.18 MB

The manylinux x86_64 CPython 3.9 wheel decreased from 12,374,013 bytes to 2,230,136 bytes, about 82% smaller, with 42 files instead of 841. License/notice texts are retained.

The verified final bundle contains 12 SDK archives, 30 wheels, 6 DEBs, 6 RPMs, 2 Windows ZIPs, 2 Homebrew bottles, 2 bottle metadata files and the merged Formula, plus manifest.json and SHA256SUMS. Its compressed size is 117,890,935 bytes. The CI artifact expires on October 28, 2026.

@ColinLeeo
ColinLeeo force-pushed the develop branch 2 times, most recently from ce7ebe6 to 095b086 Compare October 14, 2026 01:59
Fix MSVC language feature detection, runtime selection, native ARM64 builds, dependency downloads and relocatable library paths. Export headers, CMake and pkg-config metadata, static dependencies and native package components. Record the resolved CRT and runtime filename in build-tree and installed SDK metadata so independent consumers can select matching compiler options. Allow existing API tests to consume an installed SDK and include dependency-download regression tests. Inherit the selected compiler in isolated ANTLR4 dependency test configurations.
Build native SDKs, DEB/RPM packages, Windows ZIPs, Homebrew bottles and Python 3.9-3.14 wheels. Reuse SDKs and automatically read their MSVC runtime configuration. Ship runtime-only wheels with one native library, stripped Unix binaries and license texts; reject build-only files, duplicate libraries and ELF debug sections. Test every installed package, wheel and relocated SDK in fresh jobs using existing C++ and Go tests, CLI operations on Python resources and the full pytest suite. Gate checksummed bundle assembly on all test matrices and retire the separate wheel workflow.
Include bundled and system link dependencies in static pkg-config metadata, with per-configuration archive paths and C++ runtime flags. Preserve CMake's build-tree CLI RPATH while keeping relocated installations working in lib64 and multiarch layouts.

Align exported path-parser macros with the library's object targets to resolve Linux Debug UTF-8 failures. Exercise installed C and C++ consumers with real codec round trips and run the installation regressions in CI.
@ColinLeeo
ColinLeeo merged commit e34ca14 into apache:rc/2.5.0 Oct 22, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant