Skip to content

[CALCITE-7892] CI test jobs cost about 230 runner-minutes per commit and test the same configurations on every run - #5363

Open
vlsi wants to merge 1 commit into
apache:mainfrom
vlsi:ci-random-matrix
Open

vlsi wants to merge 1 commit into
apache:mainfrom
vlsi:ci-random-matrix

Conversation

@vlsi

@vlsi vlsi commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Jira Link

CALCITE-7892

Changes Proposed

The 11 hand-written test jobs in main.yml ran the same combinations on every commit: the Zulu JDK only, an English locale, two Guava versions, and JDK 24 as the newest JDK. They took about 230 runner-minutes per commit (see the dev@ thread "Calcite CI pending delays"), and adding a JDK meant copying a 30-line job.

.github/workflows/matrix.mjs now generates 6 test jobs per run with github-actions-random-matrix, which Apache JMeter, pgjdbc, and TestNG use. The axes are OS, test JDK (8, 11, 17, 21, 24), JDK vendor (zulu, temurin, corretto, liberica, microsoft), Guava (21.0, 27.1-jre, 31.1-jre, 33.4.8-jre), Avatica (the pinned release or main), time zone, and locale. Every run has Windows, macOS, JDK 8, JDK 24, the oldest and the latest Guava, Avatica main, and javadoc on JDK 8 and 21; the remaining rows cover value pairs no earlier row covers. A pull request keeps the same rows across pushes, each job name lists its parameters, and the first step of each job prints the properties the test JVM sees. To replay a run, pass the seed it printed to workflow_dispatch, or run RNG_SEED=<seed> node .github/workflows/matrix.mjs locally.

The new Gradle property testExtraJvmArgs passes JVM arguments to the test JVM only. CI sets -Duser.language, -Duser.country, and -Duser.timezone there, so Gradle itself keeps the runner's locale. A -D in it overrides the value passProperty sets.

The axes are the same ones the CI previously varied by hand, plus the JDK vendor and the locale; the choice follows past failures in JIRA (time zone: CALCITE-4790, CALCITE-4793, CALCITE-5493; locale: CALCITE-2816, CALCITE-6580; Guava: CALCITE-4789, CALCITE-4815; JDK vendor: CALCITE-5140).

User-visible changes:

  • New Gradle property testExtraJvmArgs: space-separated JVM arguments for the test JVM, for example ./gradlew test -PtestExtraJvmArgs="-Duser.language=de -Duser.country=DE".

Verification

Two trial runs on my fork, with 12 jobs and every value of every axis required at least once: run 40321987597 and run 40324575848. Across them, every job without -XX:hashCode=2 passed all 17087 :core tests: locales de_DE, fr_CA, ru_RU, and en_US, all four time zones, Guava 21.0 to 33.4.8-jre, every vendor, macOS and Windows, and Avatica main on JDK 8 and 11. The other failures led to the exclusions under Scope; the LintTest failures come from the trial branch's commit messages.

Scope

The trial runs found three defects that this PR keeps out of the matrix, so CI stays green; each needs its own fix:

  • With -XX:hashCode=2, 41 :core tests fail, for example SqlValidatorTest#testCastFails accepts CAST(DATE '1243-12-01' AS TIME). The cause is SqlTypeMappingRules.Builder#add, which uses a mutable set as a cache key. A follow-up can fix it and add the hash axis back.
  • In the tr_TR locale, InnodbAdapterDataTypesTest fails with Column parser not supported for type İNT: innodb-java-reader 1.0.10 uppercases column types in the default locale.
  • javadoc on JDK 24 reports reference not found in RelBuilder, SqlFunctions, and SqlValidatorScope, so javadoc runs on JDK 8 and 21, as before.

JDK 25 compiles the code but does not run tests: tests run on the JDK that runs Gradle, and Gradle 8.14 runs on JDK 24 at most. Oracle JDK is left out for the same reason: Oracle publishes current builds only for JDK 25, and setup-java takes older Oracle JDKs from the archive, which holds only their first GA build.

Windows now usually runs with JDK 8 or 24, since the Windows row tends to share a required JDK row; Windows with JDK 17 or 21 appears in a few percent of runs.

🤖 Generated with Claude Code

…and test the same configurations on every run

The 11 hand-written test jobs in main.yml ran the same combinations on
every commit: the Zulu JDK only, an English locale, two Guava versions,
and JDK 24 as the newest JDK. They took about 230 runner-minutes per
commit, and adding a JDK meant copying a 30-line job.

.github/workflows/matrix.mjs now generates 6 test jobs per run with
github-actions-random-matrix. It varies the OS, the test JDK (8 to 24)
and its vendor, the Guava version (21.0 to 33.4.8-jre), Avatica (the
pinned release or main), the time zone, and the locale. Every run has
Windows, macOS, JDK 8, JDK 24, the oldest and the latest Guava, and
Avatica main; the remaining rows cover value pairs that no earlier row
covers. A pull request keeps the same rows across pushes, and
workflow_dispatch replays a run from the seed it printed.

The new Gradle property testExtraJvmArgs passes JVM arguments to the
test JVM only. CI sets the locale and the time zone there, so Gradle
itself keeps the runner's locale.

Not changed: the ErrorProne, Checker Framework, and Druid jobs. The
avatica axis replaces the separate "Avatica main" job.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant