Repository navigation
Detect language: choose the languages to detect and a faster low accu… - #9434
Conversation
bamaer
left a comment
There was a problem hiding this comment.
Thanks @hansva, this looks good. Pipelines without a language list keep their current results, and the null/blank row and parallel-hang fixes come with tests.
Verified: built the plugin from 9fe3f8c and ran DetectLanguageTest (7) and DetectLanguageMetaTest (8), all green. Did not open the dialog in Hop GUI.
Should fix
- Add an integration test. The unit tests use mocks. A pipeline run would show that the language list, low accuracy mode and the blank-row pass-through work in a real run. Suggestion:
integration-tests/transforms/0126-detect-language.hpl(plus amain-0126-*.hwf). A Data grid with an English, French, German, empty and Spanish text goes into Detect language withENGLISH,FRENCH,GERMANlisted. Check it against a golden data set: three detected languages,UNKNOWNwith an empty confidence for the empty row, and a listed language orUNKNOWNfor the Spanish row.
mattcasters
left a comment
There was a problem hiding this comment.
The language selection and the low accuracy mode are useful additions. The blank-row and parallel-failure handling follows what you did in HTML to text, with tests, and building the detector in init() from the resolved languages is cleaner than the field initializer.
One thing I'd like changed before approving: dispose() now calls unloadLanguageModels(). Lingua keeps its models in static maps. On main they stayed loaded after the first run, so later runs got them for free, even though every transform built a detector with preloaded models. With this change, every run loads the models again in init() (withPreloadedLanguageModels()) and throws them away at the end. For a Detect language transform in a child pipeline that a Pipeline executor starts per row or group, or for repeated runs on Hop Server, that adds a lot of startup time to every run, and all spoken languages in high accuracy mode is the heavy case. It also removes the models from under another Detect language transform that's running at the same time, which then has to reload them lazily.
I'd keep the models loaded like before, or make unloading an option that's off by default.
+1 to @bamaer's integration test as well.
| // Lingua keeps its models in static maps: release what this transform asked for. Another | ||
| // Detect language transform that is still running loads them again on demand. | ||
| if (data.detector != null) { | ||
| data.detector.unloadLanguageModels(); |
There was a problem hiding this comment.
This clears Lingua's static model maps for these languages, so the next run (or the next child pipeline run of an executor) preloads everything again in init(), and another Detect language transform running at the same time loses them too. I'd rather not unload by default.
mattcasters
left a comment
There was a problem hiding this comment.
Thanks @hansva. I went back over my dispose() comment: main already unloads the models in finish(), so this isn't a regression, and waiting for the parallel jobs before unloading fixes the part that actually worried me. Keeping the models cached across runs can be a separate follow-up if executor-heavy setups need it.
Heads-up: #9402 also reworks finish()/dispose() in DetectLanguage, so whichever lands second needs a rebase.
+1 to @bamaer's integration test.
…cy mode and blank texts, apache#9433 Co-Authored-By: Claude Opus 5.5 <[email protected]>
…racy mode, fixes #9433
Adds the two options from #9433, both off by default so existing pipelines detect the same languages: Languages to detect (empty means every spoken language) and Low accuracy mode (trigram models only). The detector is built in
init()from those options instead of when the transform is created, and its models are unloaded indispose().Behavior change: a row whose text is null or blank used to be dropped. It now passes with
detected_language=UNKNOWNand no confidence, so pipelines that relied on the transform filtering out empty texts get those rows downstream.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
mvn clean install apache-rat:checkto make sure basic checks pass. A more thorough check will be performed on your pull request automatically.git rebase -i.addresses #123), if applicable.To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.
🤖 Generated with Claude Code