Repository navigation
Fix local pipeline shutdown deadlock and reentrant stopping (Fixes #9399) - #9401
rmannibucau wants to merge 2 commits into
Conversation
|
Thanks @rmannibucau, the split makes this much easier to review. We read the code, built the PR and ran Blockers None. Should fix
|
…ne, fixes apache#9399 Add an integration test covering the error-triggered local pipeline shutdown that this PR fixes (apache#9401). The pipeline runs Table Input (3000 rows) to Filter rows to Execute SQL row with no error hop; one INSERT (id 1500) violates a CHECK constraint so the SQL error must stop the whole local pipeline instead of hanging. The workflow runs the setup script then the pipeline, and finishes with a failure result without hanging.
|
Thanks @rmannibucau for adding the IT. As written it fails on every run, with this PR's fix in place too. The pipeline action is the last action in Should fix
|
Fixes #9399
Split out of #9374 per review: the engine-side stop/completion deadlock and reentrant-stopping fix, isolated from the transform-lifecycle changes (now #9402).
Problem
The local pipeline shutdown monitor can deadlock. A JVM thread dump of a pipeline shutdown showed a three-thread cycle:
transforms.size().stopAll()and waits for the TableInput monitor instopRunning().What changed
stopAll(),safeStop()andkillAllNoWait()instead of invoking transform callbacks under the synchronized list monitor.stopAll()before running callbacks, preventing duplicate concurrent shutdown and recursive stopping.BaseTransform.markStop()under its monitor, then invoke finished listeners outside it.stopAll()sets the terminal stopped/paused/preparing/initializing state and fires the stopped listeners in afinally, so a throwing stop callback can no longer leave the pipeline without a terminal state.Tests
PipelineTest/BaseTransformTestcover stop callback reentrancy, concurrent stop-all, snapshot iteration without holding the list monitor, and terminal-state reachability when a stop callback throws. 24PipelineTesttests pass.