Skip to content

fix(maestro): decide flow failure from status and success, not error_messages (1.4.1) - #43

Merged
jochen-testingbot merged 1 commit into
mainfrom
fix/maestro-success-ignores-stderr-noise
Oct 1, 2026
Merged

jochen-testingbot merged 1 commit into
mainfrom
fix/maestro-success-ignores-stderr-noise

Conversation

@jochen-testingbot

Copy link
Copy Markdown
Contributor

Problem

A customer's Maestro runs pass (every flow is status: DONE, success: 1), but testingbot maestro prints "Test failed" and exits 2. isFlowFailed treated any non-empty error_messages as a failure. The server fills error_messages from Maestro's stderr, and Maestro's log4j sometimes writes this at shutdown:

Thread-5 ERROR Unable to write to stream .../maestro.log for appender File
... Caused by: java.io.IOException: Stream Closed

That's a Maestro shutdown race with no effect on the result. Maestro still exits 0. testingbot/vm#69 stops the runner from forwarding it, and this PR makes the CLI robust to any stderr noise.

Change

  • isFlowFailed is now DONE && success !== 1, FAILED, or CANCELLED. error_messages no longer decides the result.
  • runPassed returns false for a CANCELLED run. Prod has 42 flows in the last 90 days that finished DONE, success: 1 after their run was cancelled, all with ["Cancelled by user"]. The old error_messages clause was the only thing failing those. This keeps that behaviour explicit.
  • hasAnyFlowFailed reuses isFlowFailed, so the "Fail reason" column doesn't appear for passed flows. The status summary also only prints the first error for a flow that failed.
  • Bump to 1.4.1 with a CHANGELOG entry.

Checked against prod data: no DONE, success: 1 flow in the last 90 days had a JUnit <failure>. So success is a reliable signal on its own.

Tests

jest: 756 passing. tsc --noEmit, prettier and eslint are clean. Updated "error_messages with success 1 = failed" to the opposite expectation. Added a cancelled-flow case, a passed-flow-with-noise case, and a cancelled-run-with-successful-flow case.

…messages (1.4.1)

error_messages also carries stderr noise from flows that passed, such as
Maestro's log4j 'Unable to write to stream ... maestro.log' at shutdown,
so a passing run exited 2. Cancelled flows, and cancelled runs whose flow
finished with success 1, still count as failed.
@jochen-testingbot
jochen-testingbot merged commit 6e992ee into main Oct 1, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant