Skip to content

feat(call): run background segmentation with LiteRT instead of MediaPipe - #6984

Draft
AndyScherzinger wants to merge 1 commit into
masterfrom
feat/noid/litert-segmentation
Draft

AndyScherzinger wants to merge 1 commit into
masterfrom
feat/noid/litert-segmentation

Conversation

@AndyScherzinger

@AndyScherzinger AndyScherzinger commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

MediaPipe Tasks sends usage statistics (task name, running mode, device info, latency and frame counts) to Google via datatransport/Clearcut on Android in every version, including 0.10.26; there is no option to turn it off. This replaces it with LiteRT running the selfie_segmenter.tflite model we already ship, so background effects work without that reporting.

  • ImageSegmenterHelper (gplay) now runs the model with a LiteRT Interpreter on the CPU: the RGBA frame is scaled to the 256×256 model input and normalised, the person confidence is thresholded at 0.5 and scaled back to the frame size as the 0/1 mask BackgroundBlurGPUProcessor expects (1 = background, blurred). Frames arriving while an inference runs are dropped, like MediaPipe's live-stream mode. The public API is unchanged, so the blur pipeline is untouched.
  • com.google.mediapipe:tasks-vision is removed together with its Renovate lock and verification metadata; com.google.ai.edge.litert:litert:1.4.2 is added (signed with the Google Maven key already trusted for androidx/Firebase). LiteRT 2.x is not used: it depends on Play Asset Delivery (com.google.android.play:ai-delivery), which F-Droid's scanner flags as non-free; Renovate is limited to <2.
  • The generic and qa stubs only get their outdated comment fixed; enabling background effects in generic is a separate PR.

How the absence of telemetry was checked so far:

  • LiteRT 1.4.2 litert and litert-api AARs: no datatransport, Firebase, Clearcut or Play references and no network code in the classes; the native library (all four ABIs) imports no socket, DNS, HTTP or TLS functions, needs only libc/libm/libdl/liblog and only refers to NNAPI and system libraries. No manifest components or permissions. Native libraries are 16 KB page aligned.
  • Dependency tree (gplayReleaseRuntimeClasspath, master vs this branch): adds only litert and litert-api; removes tasks-vision, tasks-core, Flogger, protobuf-javalite and checker-compat-qual. genericReleaseRuntimeClasspath is unchanged.
  • Merged gplay manifest: identical to master.
  • gplay debug APK: no com.google.mediapipe or com.google.android.play.core classes; selfie_segmenter.tflite is stored uncompressed (needed for memory-mapping it).

🖼️ Screenshots

🏚️ Before 🏡 After
B A

🚧 TODO

  • Confirm mask polarity and quality on a device (derived from the shader, which blurs where the mask is non-zero)
  • Parity check against the MediaPipe build: mask quality, inference time
  • Network capture during a call with background blur, with a MediaPipe build as baseline that shows the Clearcut uploads
  • Follow-up: the protobuf-javalite substitution in app/build.gradle.kts no longer matches anything in the release classpaths

🏁 Checklist

  • ⛑️ Tests (unit and/or integration) are included or not needed
  • 🔖 Capability is checked or not needed
  • 🔙 Backport requests are created or not needed: /backport to stable-xx.x
  • 📅 Milestone is set
  • 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI

MediaPipe Tasks reports usage statistics to Google through
datatransport on Android in every version, without an option to turn
it off. Run the selfie_segmenter.tflite model we already ship directly
with a LiteRT 1.4.2 Interpreter instead.

The helper keeps its API: the RGBA frame is scaled to the 256x256 model
input, the person confidence is thresholded and scaled back to the
frame size as the 0/1 mask the GPU blur shader expects, and frames are
dropped while an inference is running, like MediaPipe's live stream
mode.

LiteRT 2.x depends on Play Asset Delivery, so Renovate is limited to
1.x.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Andy Scherzinger <[email protected]>
@AndyScherzinger AndyScherzinger added this to the 26.0.0 milestone Oct 30, 2026
@AndyScherzinger AndyScherzinger added the 2. developing Work in progress label Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📱 QA build

Download app-qa-debug.apk
QR code Open the QR code for this download
Commit 275b001
Version 6984
Available until 7 days after this build

The QA build installs alongside a released Nextcloud app, so you can keep
using your existing install while testing.

Downloading the file requires a GitHub account, so open this link on the
device you want to test on, or transfer the APK to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant