Skip to content

test: make BatchTestLocal less timing-sensitive and fix flaky macOS builds (#6674) - #6831

Merged
milamberspace merged 1 commit into
apache:masterfrom
rebelwithoutacause:fix-flaky-batchtestlocal
Oct 25, 2026
Merged

milamberspace merged 1 commit into
apache:masterfrom
rebelwithoutacause:fix-flaky-batchtestlocal

Conversation

@rebelwithoutacause

Copy link
Copy Markdown
Contributor

Description

:src:dist-check:batchServerBatchTestLocal fails intermittently on slow runners — every recent failure of the CI workflow on master and on unrelated PRs is a macOS matrix entry failing this task (e.g. runs 36885805256, 36881271396, 39191681575). #6674 reports the same failure on s390x.

Root cause

BatchTestLocal.jmx runs 2 threads with a 10 s ramp-up, so thread 2 starts ~5 s in, and the golden files assume thread 1 has finished by then. On slow runners its last samples overlap thread 2's start and record 2 active threads. The CI diff shows only grpThreads/allThreads (CSV) and ng/na (XML) changing from 1 to 2; labels, counters and row order are identical. BatchTest.kt compares byte-for-byte, so the task fails.

Changes

  1. Set <threadCounts>false</threadCounts> on all 18 result collectors in BatchTestLocal.jmx (each thread group has its own CSV/XML writers targeting the same two files).
  2. Raise the main thread group's ramp_time from 10 s to 20 s, doubling the margin before thread 2 starts. It is the only multi-threaded group; the others have 1 thread and are serialized.
  3. Remove the thread-count fields from BatchTestLocal.csv / .xml. No other content changes.

Testing

  • ./gradlew :src:dist-check:batchBatchTestLocal :src:dist-check:batchServerBatchTestLocal passes locally (JDK 21, Windows); server mode passed 3 consecutive reruns.
  • Each batch run takes ~11 s instead of ~6 s due to the longer ramp-up.
  • CI on fork: https://github.com/rebelwithoutacause/jmeter_/actions/runs/39911287011 — batchBatchTestLocal and batchServerBatchTestLocal passed on all macOS jobs across both attempts. The first attempt hit two unrelated flakes (batchTestSchedulerWithTimer duration check at 7262 ms vs 7000 ms, and a docs.oracle.com DNS failure in :src:functions:javadoc); both passed on re-run.

Limitations

The test is still timing-based. With forced heavy overlap (ramp_time = 1 s) the server-mode run failed 3 of 4 times because thread interleaving changes row order, which no output setting can mask. This PR removes the failure mode seen in CI and widens the margin; a fully deterministic test would need the plan restructured. Happy to do that if preferred.

Fixes #6674

@milamberspace milamberspace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — thanks for the careful root-cause analysis. The fix matches it exactly: threadCounts off on all 18 result collectors, ramp_time 10 → 20 s on the only multi-threaded group, and the golden files only lose the grpThreads/allThreads columns and the ng/na attributes, with labels, counters and row order unchanged. CI is green, including the macOS jobs where batchServerBatchTestLocal was failing (e.g. on #6827).

Smaller observations

  • With threadCounts off, the batch tests no longer cover the grpThreads/allThreads output of the CSV/XML writers. That's a fair trade for a stable build; if you do restructure the plan to make it fully deterministic as you offered, it would be worth bringing them back.

This review was drafted by an AI-assisted tool and
confirmed by an Apache JMeter maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache JMeter handles maintainer review:
Contributing guide.

@milamberspace
milamberspace merged commit 9fd4d23 into apache:master Oct 25, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build Failure BatchTestLocal

2 participants