Skip to content

flow: scope Linux X11 recording to one window - #1966

Merged
abrichr merged 1 commit into
mainfrom
agent/1867
Nov 5, 2026
Merged

abrichr merged 1 commit into
mainfrom
agent/1867

Conversation

@abrichr

@abrichr abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member

Part of #1867

Summary

  • openadapt-flow record --backend linux now accepts --linux-app / --linux-window-title (and the generic --window / --window-title) and scopes the Capture recording to that one X11 window, the same way macOS and Windows already do.
  • Recording refuses up front, with nothing written, on Wayland, with no DISPLAY, or when the installed openadapt-capture has no Linux window capture (it needs 1.3.0 or later; flow's lock still pins 1.2.0, so a plain locked flow install gets the clear refusal instead of a late failure).
  • New packages/flow/scripts/qualify_linux_window_record.py opens two GTK windows under X11, records with --linux-window-title Chart, clicks through xdotool, and checks that the recording holds exactly the target's clicks at window-relative pixels (within 2 px), that every frame has the target window's size, and that meta.json names only the target.
  • The flow-linux-atspi CI job installs the in-tree Capture from its lock (editable types, capture, flow) plus xdotool, and runs the script under xvfb-run + openbox.
  • RECORDING.md, SURFACES.md and the CHANGELOG describe the Linux behaviour and refusals. Note record --linux-app matches a process-name substring while replay --linux-app uses the exact AT-SPI name, so for python3/java apps scope by title.

Why "Part of"
With clicks in the second window, Capture itself fails the recording: window-scoped input that is dropped leaves a gap in the native source ordinals, and the sealed journal verifier then raises "sealed native source journal has a missing ordinal". That is a Capture bug outside this Flow change, so the CI step passes --skip-other-clicks (with a comment). The "no event from the other window" check under real input there stays open until Capture is fixed.

Testing done

  • tests/test_desktop_record.py: 55 passed. With the base __main__.py and desktop_record.py in place, the 10 new tests fail.
  • Related flow suites (CLI config, backend factory/parity refusals, Linux backend and qualification, CI workflow contract, surfaces): 323 passed. Root CI tests (shards, concurrency, nightly, required check, locked installs): 60 passed.
  • Ubuntu 24.04 under Xvfb + openbox with Capture 1.3.0: target-only mode passed. The recorded clicks were [30,20], [60,30], [90,25], exactly the clicks sent; every frame was 560x360, the window's size; meta named only "Chart"; record exited 0. With clicks in the other window, record exited 1 with the missing-ordinal error above.
  • Emulated the new CI install step in a fresh venv: locked_install reported 143 locked packages compatible, with capture 1.3.0 editable.
  • ruff check/format clean; mypy clean on desktop_record.py.

Not tested

  • The updated flow-linux-atspi job on GitHub's runner itself (it runs on this PR).
  • Wayland recording (refused by design).

Allow --linux-app / --linux-window-title (and --window / --window-title)
during record on X11, refuse Wayland, an unset DISPLAY, and an installed
openadapt-capture without Linux window capture. Add a two-window Xvfb
qualification script and run it in the flow-linux-atspi CI job against
the in-tree Capture.
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review: APPROVE

From: Codex

Reviewed head fabda9576a18d2d43bf8983916aa48c9c7601d9a against literal parent 9f503bd8115710bd181b5d0cfaf6ece144a08fad.

Independent checks on isolated Xvfb displays with the real GTK fixture and in-tree Capture 1.3.0:

Focused recording tests: 55 passed on head, 47 on parent. Head Ruff passes. The canonical combined-lock install reports 143 compatible packages on both revisions.

No blocking findings within this partial PR's scope. CI 42026163133 is green, but the native Linux job is post-merge and was skipped on this PR; the app results above are from this independent run.

@abrichr
abrichr merged commit a23c2d8 into main Nov 5, 2026
53 checks passed
@abrichr
abrichr deleted the agent/1867 branch November 5, 2026 21:04
abrichr added a commit that referenced this pull request Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-needed Needs an independent review from another vendor's agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant