Skip to content

Fix local pipeline shutdown deadlock and reentrant stopping - #9374

Closed
rmannibucau wants to merge 2 commits into
apache:mainfrom
rmannibucau:fix/deadlock-1
Closed

rmannibucau wants to merge 2 commits into
apache:mainfrom
rmannibucau:fix/deadlock-1

Conversation

@rmannibucau

@rmannibucau rmannibucau commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

(updated version with codex review)

Summary

Fix the local pipeline shutdown monitor deadlock and reentrant stopping, plus related transform completion/shutdown hangs found during a follow-up audit.

The original JVM thread dump showed this three-thread cycle:

TableInput completion:
  holds transform monitor
  -> waits for pipeline monitor in the transform-finished listener

FilterRows completion:
  holds pipeline monitor
  -> waits for transforms-list monitor in transforms.size()

ExecSqlRow error shutdown:
  holds transforms-list monitor in stopAll()/synchronizedList.forEach()
  -> waits for TableInput monitor in stopRunning()

The SQL exception in ExecSqlRow triggers shutdown; the deadlock is in shutdown/completion locking, not SQL execution itself.

The follow-up transform findings are completion-accounting and cancellation problems, not additional proven instances of that same monitor cycle.

Changes

Shared local engine

  • Iterate transform snapshots in stopAll(), safeStop() and killAllNoWait() instead of invoking transform callbacks under the synchronized transform-list monitor.
  • Atomically claim stopAll() before callbacks, preventing duplicate concurrent shutdown and recursive stopping.
  • Update terminal state in BaseTransform.markStop() under its monitor, then invoke finished listeners outside it.
  • Snapshot finished listeners and invoke callbacks without holding the listener-list monitor.
  • Move error-triggered stopping outside the pipeline monitor in the completion listener.

Async transform completion

  • Fix Html2Text, DetectLanguage and StanfordSimpleNlp so failed jobs cannot leak outstanding-job counts and strand EOF/admission waits.
  • Reserve work before submission; balance accounting and notify waiters on worker completion and rejected submission.
  • Make asynchronous failures observable instead of logging them and silently losing rows; preserve per-row error routing and serialized output.
  • Drain accepted work before disposal so workers do not publish after the transform has been disposed. Borrowed common pools are not shut down.

Language Model Chat

  • Correct submission accounting and distinguish borrowed common-pool execution from an owned private executor.
  • Remove redundant executor-statistics polling; shut down owned executors without waiting for a worker from a stop callback.
  • Preserve accepted-work draining and facade lifetime through disposal, provider-error output rows, and error-hop behavior.
  • Do not report a shutdown/submission race as a new processing error.

Execute Process

  • Make stopRunning() a nonwaiting cancellation request rather than waiting indefinitely for the execution thread's latch.
  • Observe cancellation during launch, process waiting and stdout/stderr draining, including after the direct child exits.
  • Use owned daemon stream readers rather than blocking shared common-pool workers.
  • Retain observed descendant handles while the process is alive for cancellation after the direct parent exits.
  • Move potentially blocking native process/stream cleanup off pipeline and transform execution threads; preserve normal output capture and exit-code behavior.

This branch does not contain the separate Table Input JDBC partitioning work from PR #9353.

Regression coverage

  • Existing bounded engine regressions cover transform-list access during stop callbacks, reentrant/concurrent stopping, safe-stop downstream draining, completion callbacks outside lifecycle monitors, and concurrent listener registration.
  • Async runtime tests cover successful draining, worker/output failures, submission failure, admission/completion waits, and error-row routing.
  • Language Model Chat tests cover owned/common executor behavior, mock output, provider-error output, worker-triggered stopping, disposal ordering, and rejection during shutdown.
  • Controlled-process tests cover cancellation before/during launch, blocked output draining, repeated stopping and retained descendant cleanup. Existing subprocess output/large-output tests remain covered.

Validation

Verified on this branch with Java 21 and Maven 3.9.12:

  • ./mvnw -B -ntp -pl plugins/transforms/execprocess,plugins/transforms/languagemodelchat,plugins/transforms/html2text,plugins/transforms/detectlanguage,plugins/transforms/stanfordnlp spotless:check verify -DskipITs=false -Pskip-uitest: passed, 108 tests, zero failures/errors/skips; Checkstyle and Javadoc checks passed.
  • JAVA_TOOL_OPTIONS='-Duser.language=en -Duser.country=US' ./mvnw -B -ntp -pl engine -am test -Dtest=PipelineTest,BaseTransformTest,HopRunSignalTest -Dsurefire.failIfNoSpecifiedTests=false -Pskip-uitest: passed, 43 tests (23 Pipeline, 19 BaseTransform, 1 signal test).
  • ./mvnw -B -ntp -N apache-rat:check: passed, zero unapproved/unknown licenses. The root configuration scans subprojects; no temporary .yca exclusion was needed in this isolated worktree.
  • Spotless apply/check and git diff --check: passed.

Validation limits

  • A full clean install was not performed for the expanded patch, and no assembled-client/production-workload replay was performed.
  • The earlier broader core/engine run was not fully green: a font-dependent SVG test failed and the signal test timed out under the workstation's French locale. The signal test subsequently passed with an English JVM locale, including the focused rerun above. The broader run is not claimed as a clean pass.
  • Noncooperative NLP computations or provider calls can still delay draining; Java interruption cannot guarantee their termination.
  • Descendant discovery is best-effort polling: a process spawned immediately before its parent exits can escape observation.
  • Native process/stream cleanup can still block its daemon cleanup worker; it no longer blocks the pipeline stop caller. These changes do not claim guaranteed termination of arbitrary external code or OS processes.

Contribution checklist

  • Run mvn clean install apache-rat:check (full clean install not performed; targeted verification and RAT results above).
  • Two focused commits: shared-engine fix, then audited transform lifecycle fixes and regression tests.
  • No separate issue filed.
  • I hereby declare this contribution to be licensed under the Apache License Version 2.0, January 2004.

@rmannibucau
rmannibucau marked this pull request as ready for review November 7, 2026 21:29

@bamaer bamaer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rmannibucau thanks for this PR, and for the thread dump and detailed write-up. The engine change fixes the deadlock: markStop() now notifies the finished listeners after releasing the transform monitor, and stopAll() / safeStop() / killAllNoWait() iterate over a copy of the transforms list. Each of these alone breaks the cycle.

Requesting changes for the three points below.

Blocking

1. Issue numbers and scope.
There is no issue for this PR, and the commits don't reference one. Please open an issue for each change and reference it in the commit messages (Fixes #nnnn : ...). The two commits are independent:

  • the engine deadlock fix (Pipeline, BaseTransform)
  • the transform lifecycle fixes (Execute Process, Language Model Chat, Html2Text, Detect Language, Stanford Simple NLP)

The preferred route is two PRs, one per issue. The engine fix is small and can be merged quickly, while the transform changes get a closer look. If they stay in one PR, please make it clear which change belongs to which issue, both in the commits and in the description.

2. Execute Process: Stop now sends SIGKILL immediately.
On main, a stop calls destroy() on the children and the process, waits up to 30 s, then calls destroyForcibly(). This PR calls destroyForcibly() straight away on every descendant and the process. A script that traps SIGTERM to clean up or roll back no longer gets that chance. stopRunning() can stay non-blocking: call destroy() first, and do the 30 s wait and the forced kill on the cleanup thread.

3. dispose() now waits without a limit.
Html2Text, DetectLanguage, StanfordSimpleNlp and LanguageModelChat now wait in dispose() for every submitted job, ignoring interrupts and with no timeout. On main, dispose() does not wait. For Language Model Chat this means a provider call that never returns now keeps a stopped pipeline running indefinitely. Please either cap the wait (log and abandon the remaining jobs) or make sure a provider timeout always applies.

Other comments

  • stopAll() now claims isAlreadyStopped before stopping the transforms. A concurrent second caller returns before setStopped(true) and the stopped listeners have run, and an exception thrown inside the loop leaves the pipeline without a stopped state. A try/finally around the body would cover the second case.
  • Execute Process calls process.descendants() every 100 ms for as long as the command runs. On Linux this scans /proc each time. A 1 s interval, as before, should be enough.
  • Execute Process creates a 2-thread reader pool and a cleanup thread for every row. On normal completion the cleanup thread only closes streams the readers already closed, so that part can run inline.
  • catch (Exception) around runtime.exec became catch (IOException), so a SecurityException or IllegalArgumentException now escapes processRow instead of going to the error hop.
  • DetectLanguage calls unloadLanguageModels() in both finish() and dispose().
  • Without error handling, an async failure is logged twice: once by the run loop after finish() throws, and again in dispose().
  • The job reservation and failure handling is the same in the three NLP transforms, and so are their test classes. A shared helper could be a follow-up.
  • Please mention these behaviour changes in the description: synchronous rows in the three NLP transforms now go to the error hop, and Execute Process no longer emits a row for a process stopped mid-run.

@rmannibucau

Copy link
Copy Markdown
Contributor Author

Superseded by PR #9401 (engine shutdown deadlock fix) and PR #9402 (transform lifecycle fixes), per review. This PR is closed to keep the two topics separate.

@rmannibucau rmannibucau closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants