Repository navigation
Fix transform completion hangs and process shutdown (Fixes #9400) - #9402
Conversation
|
Thanks @rmannibucau for splitting this out. The points from #9374 are addressed, and the failed-job accounting fixes are good. Blockers1. The 30 s cap also applies to the normal end of input, so slow rows are lost. The default parallelism is 1, so any batch of LLM calls that takes more than 30 s to finish draining hits this. With a mocked provider that takes 12 s per call and 4 input rows:
The same pattern is in Suggested fix: in Should fix2. Execute Process: under Suggested fix: keep What was verified: the code was reviewed and the PR's unit tests were run in all five modules (109 tests, all passing, checkstyle on). A scratch test reproduced point 1 on the PR and on |
…nd process shutdown Review feedback on PR apache#9402: - Finish: wait without a 30 s cap for in-flight jobs on the normal end-of-input path in Html2Text, DetectLanguage, StanfordSimpleNlp and LanguageModelChat, so slow rows are not dropped when the output is closed. A stop still breaks the wait. - Execute Process: run the graceful-then-forced process kill inline on the transform thread before processRow returns so a hop-run shutdown that exits the JVM cannot orphan a SIGTERM-ignoring command; only the process stream close stays on a background thread. Regression tests added for all of the above.
|
Thanks @rmannibucau, both points are fixed. Slow rows are no longer lost at the end of input, and a stopped process that ignores SIGTERM is now force-killed before One small regression from the new commit: Should fix: stopping Language Model Chat with queued jobs now logs errors. Suggested fix: remove What was verified: unit tests in all five modules (117, all passing, checkstyle on), plus scratch tests for the slow drain, the forced kill and the stop with queued jobs. Integration tests were not run. |
…wn after deadlock audit
…nd process shutdown Review feedback on PR apache#9402: - Finish: wait without a 30 s cap for in-flight jobs on the normal end-of-input path in Html2Text, DetectLanguage, StanfordSimpleNlp and LanguageModelChat, so slow rows are not dropped when the output is closed. A stop still breaks the wait. - Execute Process: run the graceful-then-forced process kill inline on the transform thread before processRow returns so a hop-run shutdown that exits the JVM cannot orphan a SIGTERM-ignoring command; only the process stream close stays on a background thread. Regression tests added for all of the above.
41f8a6c to
3ae7eb3
Compare
|
Rebased onto current main ( Review follow-up:
Validation on Java 21 / Maven 3.9.16:
Scope note: the live Ollama integration workflow was not run; stop/disposal ordering is covered by deterministic lifecycle regression tests rather than a live model service. An earlier broad local reactor test run encountered the unrelated existing SVG font-assumption test failure; the current GitHub build passes. |
mattcasters
left a comment
There was a problem hiding this comment.
Thanks @rmannibucau. Keeping the facades until dispose() is the right fix, and the regression test covers it. With Bart's earlier points done, LGTM.
One heads-up: #9434 also changes finish()/dispose() in DetectLanguage, so whichever is merged second needs a careful rebase that keeps the drain handling from here and the init()-built detector from there.
|
Thanks @rmannibucau, the facade fix and regression test look good, and the earlier points still hold after the rebase. LGTM. |
Fixes #9400
Split out of #9374 per review: the transform-lifecycle fixes, isolated from the engine shutdown deadlock fix (now #9401).
What changed
stopRunning()is a non-waiting cancellation request; the process is observed during launch/process-wait/output-drain. A stopped process is now killed gracefully (SIGTERM) then, only if it does not exit within 30 s, forcibly (SIGKILL) - matchingmain- and this happens on the cleanup thread sostopRunning()stays non-blocking. Descendant polling reverted to 1 s to stop rescanning /proc every 100 ms. On normal completion, stream cleanup runs inline (the readers already closed the streams) instead of spawning a cleanup thread per row. An exec failure (IOException/SecurityException/IllegalArgumentException) is routed to the error hop rather than escapingprocessRow.finish()/dispose()forever: draining waits up to 30 s, then logs and abandons the remaining jobs. An asynchronous failure is surfaced once (further calls do not log it again).unloadLanguageModels()is called only fromdispose(), not also fromfinish().Behaviour changes
Tests
Html2TextAsyncTest(9),DetectLanguageAsyncTest(9),StanfordSimpleNlpAsyncTest(9),LanguageModelChatLifecycleTest(6) andExecProcessTest(19) tests pass.