Skip to content

Fix local pipeline shutdown deadlock and reentrant stopping (Fixes #9399) - #9401

Open
rmannibucau wants to merge 2 commits into
apache:mainfrom
rmannibucau:fix/deadlock-engine
Open

rmannibucau wants to merge 2 commits into
apache:mainfrom
rmannibucau:fix/deadlock-engine

Conversation

@rmannibucau

@rmannibucau rmannibucau commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #9399

Split out of #9374 per review: the engine-side stop/completion deadlock and reentrant-stopping fix, isolated from the transform-lifecycle changes (now #9402).

Problem

The local pipeline shutdown monitor can deadlock. A JVM thread dump of a pipeline shutdown showed a three-thread cycle:

  • TableInput completion holds the transform monitor and waits for the pipeline monitor in the transform-finished listener.
  • FilterRows completion holds the pipeline monitor and waits for the transforms-list monitor in transforms.size().
  • ExecSqlRow error shutdown holds the transforms-list monitor in stopAll() and waits for the TableInput monitor in stopRunning().

What changed

  • Iterate snapshots of the transform list in stopAll(), safeStop() and killAllNoWait() instead of invoking transform callbacks under the synchronized list monitor.
  • Atomically claim stopAll() before running callbacks, preventing duplicate concurrent shutdown and recursive stopping.
  • Update terminal state in BaseTransform.markStop() under its monitor, then invoke finished listeners outside it.
  • Snapshot finished listeners and invoke them without holding the listener-list monitor.
  • Move error-triggered stopping outside the pipeline monitor in the completion listener.
  • stopAll() sets the terminal stopped/paused/preparing/initializing state and fires the stopped listeners in a finally, so a throwing stop callback can no longer leave the pipeline without a terminal state.

Tests

  • PipelineTest / BaseTransformTest cover stop callback reentrancy, concurrent stop-all, snapshot iteration without holding the list monitor, and terminal-state reachability when a stop callback throws. 24 PipelineTest tests pass.

@bamaer

bamaer commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Thanks @rmannibucau, the split makes this much easier to review.

We read the code, built the PR and ran PipelineTest and BaseTransformTest (43 tests, all pass). With Pipeline.java and BaseTransform.java reverted to main, 5 of the new tests fail, including both deadlock tests, so they cover the fix. We have not run the integration tests.

Blockers

None.

Should fix

  • Integration test for error shutdown. This changes how every local pipeline stops, and only unit tests cover it. Please add a pipeline to integration-tests/database, for example 0049-execute-sql-row-error-stops-pipeline.hpl. It should run Table Input (a few thousand rows) → Filter rows → Execute SQL row, where the SQL fails on one row and there is no error hop. Call it from a workflow that expects the pipeline to fail, the same way main-0007-input-lookup.hwf handles 0007-database-lookup-should-fail.hpl. The workflow must finish with the failure result and must not hang. It won't reproduce the race every time, but it covers the shutdown path from Local pipeline shutdown deadlock and reentrant stopping #9399 end to end.

…ne, fixes apache#9399

Add an integration test covering the error-triggered local pipeline shutdown
that this PR fixes (apache#9401). The pipeline runs Table Input (3000 rows) to
Filter rows to Execute SQL row with no error hop; one INSERT (id 1500) violates
a CHECK constraint so the SQL error must stop the whole local pipeline instead
of hanging. The workflow runs the setup script then the pipeline, and finishes
with a failure result without hanging.
@bamaer

bamaer commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Thanks @rmannibucau for adding the IT. As written it fails on every run, with this PR's fix in place too. The pipeline action is the last action in main-0049-execute-sql-row-error-stops-pipeline.hwf, so its expected failure (result=[false]) becomes the workflow's result, and the runner reports the test as failed. We built this branch and ran run-tests-docker.sh PROJECT_NAME=database: 54 passed, 1 failed, and the failure is 0049-execute-sql-row-error-stops-pipeline.

Should fix

  • Route the pipeline action's result the way main-0007-input-lookup.hwf does for 0007-database-lookup-should-fail.hpl: the success hop (evaluation Y) goes to an Abort action, and the failure hop (evaluation N) carries on, here to a Success action. We ran a copy wired that way in the same run, and it passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local pipeline shutdown deadlock and reentrant stopping

2 participants