Repository navigation
Fix local pipeline shutdown deadlock and reentrant stopping - #9374
rmannibucau wants to merge 2 commits into
Conversation
5e325cd to
3fa7da1
Compare
bamaer
left a comment
There was a problem hiding this comment.
@rmannibucau thanks for this PR, and for the thread dump and detailed write-up. The engine change fixes the deadlock: markStop() now notifies the finished listeners after releasing the transform monitor, and stopAll() / safeStop() / killAllNoWait() iterate over a copy of the transforms list. Each of these alone breaks the cycle.
Requesting changes for the three points below.
Blocking
1. Issue numbers and scope.
There is no issue for this PR, and the commits don't reference one. Please open an issue for each change and reference it in the commit messages (Fixes #nnnn : ...). The two commits are independent:
- the engine deadlock fix (
Pipeline,BaseTransform) - the transform lifecycle fixes (Execute Process, Language Model Chat, Html2Text, Detect Language, Stanford Simple NLP)
The preferred route is two PRs, one per issue. The engine fix is small and can be merged quickly, while the transform changes get a closer look. If they stay in one PR, please make it clear which change belongs to which issue, both in the commits and in the description.
2. Execute Process: Stop now sends SIGKILL immediately.
On main, a stop calls destroy() on the children and the process, waits up to 30 s, then calls destroyForcibly(). This PR calls destroyForcibly() straight away on every descendant and the process. A script that traps SIGTERM to clean up or roll back no longer gets that chance. stopRunning() can stay non-blocking: call destroy() first, and do the 30 s wait and the forced kill on the cleanup thread.
3. dispose() now waits without a limit.
Html2Text, DetectLanguage, StanfordSimpleNlp and LanguageModelChat now wait in dispose() for every submitted job, ignoring interrupts and with no timeout. On main, dispose() does not wait. For Language Model Chat this means a provider call that never returns now keeps a stopped pipeline running indefinitely. Please either cap the wait (log and abandon the remaining jobs) or make sure a provider timeout always applies.
Other comments
stopAll()now claimsisAlreadyStoppedbefore stopping the transforms. A concurrent second caller returns beforesetStopped(true)and the stopped listeners have run, and an exception thrown inside the loop leaves the pipeline without a stopped state. Atry/finallyaround the body would cover the second case.- Execute Process calls
process.descendants()every 100 ms for as long as the command runs. On Linux this scans/proceach time. A 1 s interval, as before, should be enough. - Execute Process creates a 2-thread reader pool and a cleanup thread for every row. On normal completion the cleanup thread only closes streams the readers already closed, so that part can run inline.
catch (Exception)aroundruntime.execbecamecatch (IOException), so aSecurityExceptionorIllegalArgumentExceptionnow escapesprocessRowinstead of going to the error hop.DetectLanguagecallsunloadLanguageModels()in bothfinish()anddispose().- Without error handling, an async failure is logged twice: once by the run loop after
finish()throws, and again indispose(). - The job reservation and failure handling is the same in the three NLP transforms, and so are their test classes. A shared helper could be a follow-up.
- Please mention these behaviour changes in the description: synchronous rows in the three NLP transforms now go to the error hop, and Execute Process no longer emits a row for a process stopped mid-run.
(updated version with codex review)
Summary
Fix the local pipeline shutdown monitor deadlock and reentrant stopping, plus related transform completion/shutdown hangs found during a follow-up audit.
The original JVM thread dump showed this three-thread cycle:
The SQL exception in
ExecSqlRowtriggers shutdown; the deadlock is in shutdown/completion locking, not SQL execution itself.The follow-up transform findings are completion-accounting and cancellation problems, not additional proven instances of that same monitor cycle.
Changes
Shared local engine
stopAll(),safeStop()andkillAllNoWait()instead of invoking transform callbacks under the synchronized transform-list monitor.stopAll()before callbacks, preventing duplicate concurrent shutdown and recursive stopping.BaseTransform.markStop()under its monitor, then invoke finished listeners outside it.Async transform completion
Html2Text,DetectLanguageandStanfordSimpleNlpso failed jobs cannot leak outstanding-job counts and strand EOF/admission waits.Language Model Chat
Execute Process
stopRunning()a nonwaiting cancellation request rather than waiting indefinitely for the execution thread's latch.This branch does not contain the separate Table Input JDBC partitioning work from PR #9353.
Regression coverage
Validation
Verified on this branch with Java 21 and Maven 3.9.12:
./mvnw -B -ntp -pl plugins/transforms/execprocess,plugins/transforms/languagemodelchat,plugins/transforms/html2text,plugins/transforms/detectlanguage,plugins/transforms/stanfordnlp spotless:check verify -DskipITs=false -Pskip-uitest: passed, 108 tests, zero failures/errors/skips; Checkstyle and Javadoc checks passed.JAVA_TOOL_OPTIONS='-Duser.language=en -Duser.country=US' ./mvnw -B -ntp -pl engine -am test -Dtest=PipelineTest,BaseTransformTest,HopRunSignalTest -Dsurefire.failIfNoSpecifiedTests=false -Pskip-uitest: passed, 43 tests (23 Pipeline, 19 BaseTransform, 1 signal test)../mvnw -B -ntp -N apache-rat:check: passed, zero unapproved/unknown licenses. The root configuration scans subprojects; no temporary.ycaexclusion was needed in this isolated worktree.git diff --check: passed.Validation limits
clean installwas not performed for the expanded patch, and no assembled-client/production-workload replay was performed.Contribution checklist
mvn clean install apache-rat:check(full clean install not performed; targeted verification and RAT results above).