Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 6 additions & 24 deletions .github/workflows/packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -817,9 +817,7 @@ jobs:
# checked from file bytes); qualify_linux.py records a write workflow on a
# GTK3 chart app with SQLite, compiles it, runs it governed with the record
# check (clean, another patient, a look-alike) and repairs an update notice
# on its AT-SPI element identity; qualify_linux_window_record.py records
# that app through Capture scoped to its window, with a second window open.
# All results are uploaded.
# on its AT-SPI element identity. Both results are uploaded.
flow-linux-atspi:
name: flow native Linux (AT-SPI, Xvfb)
needs: scope
Expand Down Expand Up @@ -858,24 +856,19 @@ jobs:
set -e
sudo apt-get install -y \
at-spi2-core dbus-x11 gir1.2-atspi-2.0 gir1.2-gtk-3.0 libatk-adaptor \
libcairo2-dev libgirepository-2.0-dev openbox pkg-config xauth xdotool xvfb
libcairo2-dev libgirepository-2.0-dev openbox pkg-config xauth xvfb
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: true
cache-dependency-glob: |
packages/flow/uv.lock
packages/capture/uv.lock
cache-dependency-glob: packages/flow/uv.lock
cache-suffix: flow-linux-atspi
- name: Install from the locks
# Flow's lock, then Capture's: the in-tree Capture (X11 window capture)
# can need newer dependencies than the published one Flow's lock names.
- name: Install from the lock
run: >-
python ../../scripts/ci/locked_install.py --lock 'packages/flow[dev,linux,capture]'
--lock packages/capture
--editable packages/types --editable packages/capture --editable packages/flow
python ../../scripts/ci/locked_install.py --lock 'packages/flow[dev,linux]'
--editable packages/types --editable packages/flow
- name: Qualify LinuxBackend on the GTK3 fixture
timeout-minutes: 10
run: |
Expand All @@ -895,17 +888,6 @@ jobs:
env XDG_SESSION_TYPE=x11 NO_AT_BRIDGE=0 GTK_MODULES=atk-bridge \
python benchmark/repair_every_backend/qualify_linux.py \
--output runs/linux-atspi/record-compile-run.json
- name: Record one X11 window through Capture with a second window open
timeout-minutes: 10
run: |
# --skip-other-clicks: Capture's verifier rejects a recording whose
# dropped input left a hole in the source ordinals, so clicks in the
# second window fail the recording. The second window stays open.
xvfb-run -a -s "-screen 0 1280x800x24" dbus-run-session -- \
bash -c 'openbox >/dev/null 2>&1 & sleep 1; exec "$@"' bash \
env XDG_SESSION_TYPE=x11 GDK_BACKEND=x11 \
python scripts/qualify_linux_window_record.py --skip-other-clicks \
--output runs/linux-atspi/window-record.json
- name: Upload Linux AT-SPI qualification evidence
if: always()
timeout-minutes: 2
Expand Down
7 changes: 0 additions & 7 deletions packages/flow/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,6 @@

### Changed

- `record --backend linux` can scope the recording to one X11 window.
`--window`/`--window-title` and `--linux-app`/`--linux-window-title` select
it; before, they were refused. Wayland sessions, an unset `DISPLAY` and
openadapt-capture releases before 1.3.0 are refused before recording starts.
`scripts/qualify_linux_window_record.py` checks it under Xvfb with a second
window open: the recording holds only the target's clicks, in its pixel
space, and only its frames.
- No command falls back to the demo app on its own. `replay` needs a target: `--url`, `backend.url` in `--config`, or `--demo-app` to serve the bundled MockMed (the tutorial and quickstart commands pass it; `--drift` needs it). The variant router's first-screen probe captures nothing without a URL. `qualify accept`'s break-it check no longer runs every workflow's oracle against MockMed: it builds the lie from the workflow itself (a store that did not change, and a screen showing the workflow's own success text and every value the run typed). The reference Execute worker refuses, before running anything, an admission whose `target_url` it cannot drive, instead of resetting and reading that app as if it were the demo.

### Fixes
Expand Down
6 changes: 2 additions & 4 deletions packages/flow/docs/SURFACES.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,8 @@ openadapt-flow compile rec --out bundle --name my-task
openadapt-flow replay bundle --macos-app TextEdit \
--macos-window-title notes.txt

# Linux: the title scopes the local Capture window on X11; AT-SPI selects
# the replay target.
openadapt-flow record --backend linux \
--linux-window-title "Untitled Document 1" --out rec
# Linux: Capture records the local desktop; AT-SPI selects the replay target.
openadapt-flow record --backend linux --out rec
openadapt-flow compile rec --out bundle --name my-task
openadapt-flow replay bundle --linux-app gedit \
--linux-window-title "Untitled Document 1"
Expand Down
16 changes: 4 additions & 12 deletions packages/flow/docs/desktop/RECORDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,18 +165,10 @@ with `--rdp-readiness-text`, set `backend.rdp_readiness_text` in deployment
config, or pass the flag directly. If it is absent, `run` refuses before any
action; ordinary record/replay remains available.

Window-scoped capture is implemented on **macOS, Windows and X11 Linux**
hosts (`CGWindowListCreateImage` / Win32 region grab / X11 EWMH window list
and XComposite). On Linux, `--linux-app` and `--linux-window-title` are the
same selectors as `--window` and `--window-title`: the owner is matched
against the window's process name and the title against its `_NET_WM_NAME`,
both as substrings, and more than one match is refused. The process name of a
Python or Java app is the interpreter (`python3`, `java`), not the AT-SPI
application name that `replay --linux-app` matches, so scope such an app by
title. A native Wayland session, an unset `DISPLAY`, or an openadapt-capture
older than 1.3.0 is refused before recording starts. On any other host
`--window` is refused up front rather than silently falling back to
full-screen (which would record coordinates in the wrong pixel space). `--window` applies only to
Window-scoped capture is implemented on **macOS and Windows** hosts
(`CGWindowListCreateImage` / Win32 region grab); on any other host `--window`
is refused up front rather than silently falling back to full-screen (which
would record coordinates in the wrong pixel space). `--window` applies only to
the desktop backends — `--backend web` records the Playwright page and refuses
it.

Expand Down
51 changes: 28 additions & 23 deletions packages/flow/openadapt_flow/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ def _reject_unbound_record_target_flags(args: argparse.Namespace, backend: str)
"web": set(),
"windows": set(),
"macos": {"macos_app", "macos_window_title"},
"linux": {"linux_app", "linux_window_title"},
"linux": set(),
"rdp": {"rdp_window", "rdp_window_title", "rdp_readiness_text"},
"citrix": {"rdp_window", "rdp_window_title", "rdp_readiness_text"},
}
Expand All @@ -198,12 +198,15 @@ def _reject_unbound_record_target_flags(args: argparse.Namespace, backend: str)
"Scope the local recording with --window/--window-title, then "
"pass --agent-url to replay or run"
)
elif backend == "linux" and attr == "linux_allow_physical_input":
elif backend == "linux" and attr in {
"linux_app",
"linux_window_title",
"linux_allow_physical_input",
}:
reason = (
"it permits replay and run to send physical input; recording "
"sends none. Scope the recording with --window/--window-title "
"(or --linux-app/--linux-window-title), then pass it to "
"replay or run"
"the current Capture component has no Linux window-scoping "
"primitive. Record the local Linux desktop without this flag, "
"then pass it to replay or run"
)
elif backend == "rdp" and attr == "rdp_host":
reason = (
Expand Down Expand Up @@ -249,22 +252,26 @@ def _resolve_record_capture_window(

owner = getattr(args, "window", None)
title = getattr(args, "window_title", None)
if backend in ("macos", "linux"):
# On Linux, Capture resolves one X11 (EWMH) client window: the owner
# is matched against the window's process name and the title against
# its _NET_WM_NAME. No match, or more than one, stops the recording.
if backend == "linux" and (owner is not None or title is not None):
raise SystemExit(
"record --backend linux: --window/--window-title cannot be applied "
"because the current Capture component has no Linux "
"window-scoping primitive. Record the local Linux desktop without "
"these flags. Nothing was recorded."
)
if backend == "macos":
owner = _merge_record_window_selector(
owner,
getattr(args, f"{backend}_app", None),
getattr(args, "macos_app", None),
generic_flag="--window",
target_flag=f"--{backend}-app",
target_flag="--macos-app",
backend=backend,
)
title = _merge_record_window_selector(
title,
getattr(args, f"{backend}_window_title", None),
getattr(args, "macos_window_title", None),
generic_flag="--window-title",
target_flag=f"--{backend}-window-title",
target_flag="--macos-window-title",
backend=backend,
)
elif backend in ("rdp", "citrix"):
Expand Down Expand Up @@ -8508,9 +8515,8 @@ def _add_backend_flags(p: argparse.ArgumentParser) -> None:
metavar="APP",
help=(
"Exact AT-SPI application name for --backend linux (e.g. gedit). "
"During record it is the same selector as --window and scopes "
"Capture to one X11 window whose process name contains APP. "
"During replay/run it overrides backend.linux_app."
"Replay/run only: the current Capture path records the local Linux "
"desktop and refuses this flag. Overrides backend.linux_app."
),
)
p.add_argument(
Expand All @@ -8519,9 +8525,9 @@ def _add_backend_flags(p: argparse.ArgumentParser) -> None:
metavar="TITLE",
help=(
"Exact top-level window title for --backend linux. Zero or "
"multiple matches are refused. During record it is the same "
"selector as --window-title (a title substring) and scopes "
"Capture. During replay/run it overrides backend.linux_window_title."
"multiple matches are refused. Replay/run only: the current "
"Capture path records the local Linux desktop and refuses this "
"flag. Overrides backend.linux_window_title."
),
)
p.add_argument(
Expand Down Expand Up @@ -9192,9 +9198,8 @@ def build_parser() -> argparse.ArgumentParser:
"substring, e.g. --window Parallels / --window 'Citrix Workspace'). "
"Closes the coordinate-space gap with the pixel (rdp) replay "
"surface. Combine with --window-title to disambiguate. "
"Supported on macOS, Windows and X11 Linux hosts (on Linux the "
"owner is the window's process name); refused elsewhere, "
"including native Wayland. Ignored (rejected) for --backend web."
"Supported on macOS and Windows hosts; refused elsewhere. "
"Ignored (rejected) for --backend web."
),
)
p.add_argument(
Expand Down
62 changes: 4 additions & 58 deletions packages/flow/openadapt_flow/desktop_record.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,50 +56,9 @@
# (openadapt_capture.window_capture.resolve_window / capture_window). Kept in
# lock-step with that module: recording ONE window in its own pixel space needs
# a per-window capture primitive (macOS CGWindowListCreateImage / Windows
# Win32 + region grab / Linux X11 EWMH + XComposite). Elsewhere we refuse UP
# FRONT rather than start a full-screen capture that silently ignores the
# requested --window scope.
WINDOW_CAPTURE_PLATFORMS = ("darwin", "win32", "linux")


def _linux_window_session_refusal(environ: Any = None) -> Optional[str]:
"""Why a Linux session cannot record one window, or None for X11.

Mirrors ``openadapt_capture.window_capture_linux._require_x11_session`` so
the refusal comes before the operator is told to start demonstrating.
"""
import os

values = os.environ if environ is None else environ
session_type = values.get("XDG_SESSION_TYPE", "").strip().casefold()
if session_type == "wayland" or values.get("WAYLAND_DISPLAY", "").strip():
return (
"this is a Wayland session; Capture can scope a recording to one "
"window only under X11"
)
if not values.get("DISPLAY", "").strip():
return "DISPLAY is not set; Capture needs an X11 desktop"
return None


def _installed_capture_lacks_linux_windows() -> bool:
"""Whether the installed openadapt-capture predates Linux window capture.

openadapt-capture added X11 window capture in 1.3.0
(``openadapt_capture.window_capture_linux``). An older release would
refuse only after the operator was told to start. Returns False when
openadapt-capture is not installed: the recorder reports that itself.
"""
import importlib.util

try:
if importlib.util.find_spec("openadapt_capture") is None:
return False
return (
importlib.util.find_spec("openadapt_capture.window_capture_linux") is None
)
except (ImportError, ValueError):
return False
# Win32 + region grab). Elsewhere we refuse UP FRONT rather than start a
# full-screen capture that silently ignores the requested --window scope.
WINDOW_CAPTURE_PLATFORMS = ("darwin", "win32")


class _CaptureRecorder(Protocol):
Expand Down Expand Up @@ -356,22 +315,9 @@ def record_desktop_capture(
"window-scoped recording (--window) is not supported on this host "
f"(platform {sys.platform!r}); openadapt-capture implements "
"per-window capture only on "
f"{', '.join(WINDOW_CAPTURE_PLATFORMS)}. Record on a supported "
f"{' and '.join(WINDOW_CAPTURE_PLATFORMS)}. Record on a supported "
"host, or omit --window to capture the full screen."
)
if window is not None and sys.platform.startswith("linux"):
refusal = _linux_window_session_refusal()
if refusal is not None:
raise SystemExit(
f"window-scoped recording (--window) refused: {refusal}. "
"Nothing was recorded."
)
if recorder_factory is None and _installed_capture_lacks_linux_windows():
raise SystemExit(
"window-scoped recording (--window) refused: the installed "
"openadapt-capture has no Linux window capture; install "
"openadapt-capture 1.3.0 or later. Nothing was recorded."
)
if backend_kind is not None and backend_kind not in ("rdp", "citrix"):
raise ValueError(
"backend_kind execution hints are supported only for rdp or citrix"
Expand Down
Loading
Loading