Skip to content

Fix transform completion hangs and process shutdown (Fixes #9400) - #9402

Merged
bamaer merged 3 commits into
apache:mainfrom
rmannibucau:fix/transform-lifecycle
Nov 16, 2026
Merged

bamaer merged 3 commits into
apache:mainfrom
rmannibucau:fix/transform-lifecycle

Conversation

@rmannibucau

Copy link
Copy Markdown
Contributor

Fixes #9400

Split out of #9374 per review: the transform-lifecycle fixes, isolated from the engine shutdown deadlock fix (now #9401).

What changed

  • Execute Process: stopRunning() is a non-waiting cancellation request; the process is observed during launch/process-wait/output-drain. A stopped process is now killed gracefully (SIGTERM) then, only if it does not exit within 30 s, forcibly (SIGKILL) - matching main - and this happens on the cleanup thread so stopRunning() stays non-blocking. Descendant polling reverted to 1 s to stop rescanning /proc every 100 ms. On normal completion, stream cleanup runs inline (the readers already closed the streams) instead of spawning a cleanup thread per row. An exec failure (IOException/SecurityException/IllegalArgumentException) is routed to the error hop rather than escaping processRow.
  • Html2Text / DetectLanguage / StanfordSimpleNlp: admission and completion accounting are balanced on worker completion and rejected submission so failed jobs no longer strand waits or leak outstanding-job counts. A stuck asynchronous provider can no longer block finish()/dispose() forever: draining waits up to 30 s, then logs and abandons the remaining jobs. An asynchronous failure is surfaced once (further calls do not log it again).
  • DetectLanguage: unloadLanguageModels() is called only from dispose(), not also from finish().
  • LanguageModelChat: submission accounting is balanced on worker completion and rejected submission; owned executors are shut down without waiting for a worker from a stop callback; draining is capped at 30 s like the NLP transforms.

Behaviour changes

  • Synchronous NLP rows that fail are now routed to the error hop instead of silently dropping the row.
  • Execute Process no longer emits a row for a process that was stopped mid-run.

Tests

  • New async/lifecycle tests spin up real or mocked providers/processes and assert observable behaviour: output drained on normal completion, no hang on stop, graceful-then-forced process kill, balanced reservation accounting and capped disposal. All Html2TextAsyncTest (9), DetectLanguageAsyncTest (9), StanfordSimpleNlpAsyncTest (9), LanguageModelChatLifecycleTest (6) and ExecProcessTest (19) tests pass.

@rmannibucau rmannibucau changed the title Fix transform completion hangs and process shutdown after deadlock audit (Fixes #9400) Fix transform completion hangs and process shutdown (Fixes #9400) Nov 10, 2026
@bamaer

bamaer commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Thanks @rmannibucau for splitting this out. The points from #9374 are addressed, and the failed-job accounting fixes are good.

Blockers

1. The 30 s cap also applies to the normal end of input, so slow rows are lost.
finish() now calls the same capped awaitJobs() as dispose(). When the queued jobs need more than 30 s to drain, finish() logs "Abandoning…" at basic level and calls setOutputDone() while workers are still running. Their rows reach the output after it is closed. In Language Model Chat, facadeMap.clear() also runs first, so the next queued job fails.

The default parallelism is 1, so any batch of LLM calls that takes more than 30 s to finish draining hits this. With a mocked provider that takes 12 s per call and 4 input rows:

rows when output is marked done after dispose()
main 4 4, errors 0
this PR 2 (at 30 s) 3, errors 1

The same pattern is in Html2Text.java:113, DetectLanguage.java:119 and StanfordSimpleNlp.java:112. Those use the shared common pool with up to cores*100 queued jobs, so Stanford parsing in particular can take longer than 30 s.

Suggested fix: in finish(), wait without a cap until the jobs are done or the transform is stopped, for example while (jobs > 0 && !isStopped()) lock.wait(1000);. Keep the 30 s cap for dispose() and the stop path only. With that change in LanguageModelChat.java:84, the test above returns all 4 rows and LanguageModelChatLifecycleTest still passes.

Should fix

2. Execute Process: under hop-run, the forced kill never runs.
The 30 s grace wait and destroyForcibly() now run on a daemon thread, and processRow returns as soon as the stop is seen. hop-run calls System.exit() when the pipeline ends, which ends that thread before the forced kill. A command that ignores SIGTERM keeps running after the pipeline has stopped. On main, stopRunning() waited until the forced kill. My earlier suggestion to put this on the cleanup thread missed this case, sorry.

Suggested fix: keep stopRunning() non-blocking, but run destroy(), the grace wait and destroyForcibly() on the transform thread before processRow returns. Only the stream closing goes to the background thread. (ExecProcess.java:331)

What was verified: the code was reviewed and the PR's unit tests were run in all five modules (109 tests, all passing, checkstyle on). A scratch test reproduced point 1 on the PR and on main, and checked the suggested fix. Point 2 is from reading the code and was not run. The existing integration tests (0062-execute-process, 0086-html2text, languagemodelchat) were not run.

rmannibucau added a commit to rmannibucau/hop that referenced this pull request Nov 10, 2026
…nd process shutdown

Review feedback on PR apache#9402:
- Finish: wait without a 30 s cap for in-flight jobs on the normal end-of-input path in
  Html2Text, DetectLanguage, StanfordSimpleNlp and LanguageModelChat, so slow rows are
  not dropped when the output is closed. A stop still breaks the wait.
- Execute Process: run the graceful-then-forced process kill inline on the transform
  thread before processRow returns so a hop-run shutdown that exits the JVM cannot orphan
  a SIGTERM-ignoring command; only the process stream close stays on a background thread.

Regression tests added for all of the above.
@bamaer

bamaer commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Thanks @rmannibucau, both points are fixed. Slow rows are no longer lost at the end of input, and a stopped process that ignores SIGTERM is now force-killed before processRow returns. I checked both with scratch tests, one of them against a real trap '' TERM script.

One small regression from the new commit:

Should fix: stopping Language Model Chat with queued jobs now logs errors.
After a stop, finish() returns at once because awaitJobsUncapped() exits when the transform is stopped. It then calls facadeMap.clear() while queued jobs are still in the executor, and those jobs fail on a null facade. With parallelism 1 and 5 queued rows, a stop now ends with 1 error. On main and on the previous commit it ended with 0.

Suggested fix: remove facadeMap.clear() from finish() (LanguageModelChat.java:88). dispose() already clears it after waiting for the jobs. With that change the stop ends with 0 errors and LanguageModelChatLifecycleTest still passes.

What was verified: unit tests in all five modules (117, all passing, checkstyle on), plus scratch tests for the slow drain, the forced kill and the stop with queued jobs. Integration tests were not run.

…nd process shutdown

Review feedback on PR apache#9402:
- Finish: wait without a 30 s cap for in-flight jobs on the normal end-of-input path in
  Html2Text, DetectLanguage, StanfordSimpleNlp and LanguageModelChat, so slow rows are
  not dropped when the output is closed. A stop still breaks the wait.
- Execute Process: run the graceful-then-forced process kill inline on the transform
  thread before processRow returns so a hop-run shutdown that exits the JVM cannot orphan
  a SIGTERM-ignoring command; only the process stream close stays on a background thread.

Regression tests added for all of the above.
@rmannibucau
rmannibucau force-pushed the fix/transform-lifecycle branch from 41f8a6c to 3ae7eb3 Compare November 15, 2026 10:59
@github-actions github-actions Bot added the AI label Nov 15, 2026
@rmannibucau

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (947ecbe21e); updated head is 3ae7eb3034.

Review follow-up:

  • Retained upstream synchronous Html2Text implementation and its existing tests; removed the superseded asynchronous test.
  • Kept LanguageModelChat facades alive until disposal so queued work after stop can still complete safely. Added a deterministic stop/queued-work regression test, verified red with the old cleanup and green with the fix.

Validation on Java 21 / Maven 3.9.16:

  • 140 focused tests across languagemodelchat, detectlanguage, execprocess, html2text and stanfordnlp: no failures, errors or skips; formatting and Checkstyle passed.
  • Client assembly succeeded.
  • Ran integration-tests/transforms/main-0127-html2text.hwf through the assembled hop-run.sh: passed both golden datasets, including three transform copies.
  • git diff --check passed.
  • GitHub build and UI checks are green; GitHub reports MERGEABLE / CLEAN.

Scope note: the live Ollama integration workflow was not run; stop/disposal ordering is covered by deterministic lifecycle regression tests rather than a live model service. An earlier broad local reactor test run encountered the unrelated existing SVG font-assumption test failure; the current GitHub build passes.

@mattcasters mattcasters left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rmannibucau. Keeping the facades until dispose() is the right fix, and the regression test covers it. With Bart's earlier points done, LGTM.

One heads-up: #9434 also changes finish()/dispose() in DetectLanguage, so whichever is merged second needs a careful rebase that keeps the drain handling from here and the init()-built detector from there.

@bamaer

bamaer commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Thanks @rmannibucau, the facade fix and regression test look good, and the earlier points still hold after the rebase. LGTM.

@bamaer
bamaer merged commit bf7c1da into apache:main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transform completion hangs and process shutdown after deadlock audit

3 participants