Repository navigation
test: make BatchTestLocal less timing-sensitive and fix flaky macOS builds (#6674) - #6831
Conversation
milamberspace
left a comment
There was a problem hiding this comment.
LGTM — thanks for the careful root-cause analysis. The fix matches it exactly: threadCounts off on all 18 result collectors, ramp_time 10 → 20 s on the only multi-threaded group, and the golden files only lose the grpThreads/allThreads columns and the ng/na attributes, with labels, counters and row order unchanged. CI is green, including the macOS jobs where batchServerBatchTestLocal was failing (e.g. on #6827).
Smaller observations
- With
threadCountsoff, the batch tests no longer cover thegrpThreads/allThreadsoutput of the CSV/XML writers. That's a fair trade for a stable build; if you do restructure the plan to make it fully deterministic as you offered, it would be worth bringing them back.
This review was drafted by an AI-assisted tool and
confirmed by an Apache JMeter maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache JMeter handles maintainer review:
Contributing guide.
Description
:src:dist-check:batchServerBatchTestLocalfails intermittently on slow runners — every recent failure of the CI workflow onmasterand on unrelated PRs is a macOS matrix entry failing this task (e.g. runs 36885805256, 36881271396, 39191681575). #6674 reports the same failure on s390x.Root cause
BatchTestLocal.jmxruns 2 threads with a 10 s ramp-up, so thread 2 starts ~5 s in, and the golden files assume thread 1 has finished by then. On slow runners its last samples overlap thread 2's start and record 2 active threads. The CI diff shows onlygrpThreads/allThreads(CSV) andng/na(XML) changing from 1 to 2; labels, counters and row order are identical.BatchTest.ktcompares byte-for-byte, so the task fails.Changes
<threadCounts>false</threadCounts>on all 18 result collectors inBatchTestLocal.jmx(each thread group has its own CSV/XML writers targeting the same two files).ramp_timefrom 10 s to 20 s, doubling the margin before thread 2 starts. It is the only multi-threaded group; the others have 1 thread and are serialized.BatchTestLocal.csv/.xml. No other content changes.Testing
./gradlew :src:dist-check:batchBatchTestLocal :src:dist-check:batchServerBatchTestLocalpasses locally (JDK 21, Windows); server mode passed 3 consecutive reruns.batchBatchTestLocalandbatchServerBatchTestLocalpassed on all macOS jobs across both attempts. The first attempt hit two unrelated flakes (batchTestSchedulerWithTimerduration check at 7262 ms vs 7000 ms, and adocs.oracle.comDNS failure in:src:functions:javadoc); both passed on re-run.Limitations
The test is still timing-based. With forced heavy overlap (
ramp_time= 1 s) the server-mode run failed 3 of 4 times because thread interleaving changes row order, which no output setting can mask. This PR removes the failure mode seen in CI and widens the margin; a fully deterministic test would need the plan restructured. Happy to do that if preferred.Fixes #6674