Repository navigation
feat(call): run background segmentation with LiteRT instead of MediaPipe - #6984
Draft
AndyScherzinger wants to merge 1 commit into
Draft
AndyScherzinger wants to merge 1 commit into
AndyScherzinger wants to merge 1 commit into
Conversation
MediaPipe Tasks reports usage statistics to Google through datatransport on Android in every version, without an option to turn it off. Run the selfie_segmenter.tflite model we already ship directly with a LiteRT 1.4.2 Interpreter instead. The helper keeps its API: the RGBA frame is scaled to the 256x256 model input, the person confidence is thresholded and scaled back to the frame size as the 0/1 mask the GPU blur shader expects, and frames are dropped while an inference is running, like MediaPipe's live stream mode. LiteRT 2.x depends on Play Asset Delivery, so Renovate is limited to 1.x. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Andy Scherzinger <[email protected]>
Contributor
📱 QA build
The QA build installs alongside a released Nextcloud app, so you can keep Downloading the file requires a GitHub account, so open this link on the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MediaPipe Tasks sends usage statistics (task name, running mode, device info, latency and frame counts) to Google via datatransport/Clearcut on Android in every version, including 0.10.26; there is no option to turn it off. This replaces it with LiteRT running the
selfie_segmenter.tflitemodel we already ship, so background effects work without that reporting.ImageSegmenterHelper(gplay) now runs the model with a LiteRTInterpreteron the CPU: the RGBA frame is scaled to the 256×256 model input and normalised, the person confidence is thresholded at 0.5 and scaled back to the frame size as the 0/1 maskBackgroundBlurGPUProcessorexpects (1 = background, blurred). Frames arriving while an inference runs are dropped, like MediaPipe's live-stream mode. The public API is unchanged, so the blur pipeline is untouched.com.google.mediapipe:tasks-visionis removed together with its Renovate lock and verification metadata;com.google.ai.edge.litert:litert:1.4.2is added (signed with the Google Maven key already trusted for androidx/Firebase). LiteRT 2.x is not used: it depends on Play Asset Delivery (com.google.android.play:ai-delivery), which F-Droid's scanner flags as non-free; Renovate is limited to<2.How the absence of telemetry was checked so far:
litertandlitert-apiAARs: no datatransport, Firebase, Clearcut or Play references and no network code in the classes; the native library (all four ABIs) imports no socket, DNS, HTTP or TLS functions, needs only libc/libm/libdl/liblog and only refers to NNAPI and system libraries. No manifest components or permissions. Native libraries are 16 KB page aligned.gplayReleaseRuntimeClasspath, master vs this branch): adds onlylitertandlitert-api; removestasks-vision,tasks-core, Flogger,protobuf-javaliteandchecker-compat-qual.genericReleaseRuntimeClasspathis unchanged.com.google.mediapipeorcom.google.android.play.coreclasses;selfie_segmenter.tfliteis stored uncompressed (needed for memory-mapping it).🖼️ Screenshots
🚧 TODO
protobuf-javalitesubstitution inapp/build.gradle.ktsno longer matches anything in the release classpaths🏁 Checklist
/backport to stable-xx.x🤖 AI (if applicable)