Skip to content

Native recordings get no field labels from the accessibility tree (only the browser's do) #1868

Open

Description

@abrichr

At record time a typed field's label is kept as evidence (field_label on the event and on the compiled step; see packages/flow/DESIGN.md, "Record-time label capture"). Only PlaywrightBackend implements focused_field_label (openadapt_flow.backend.FieldLabelBackend). Windows UIA, macOS AX and Linux AT-SPI expose the focused field's name and labelled-by relation, but their backends don't implement the seam, so native recordings fall back to the nearest OCR line at compile time.

Close: implement focused_field_label on WindowsBackend (UIA LabeledBy, then Name), MacOSBackend (AXTitleUIElement, then AXTitle/AXDescription) and LinuxBackend (AT-SPI LABELLED_BY relation, then name). Read from the focused element only, never by position. Return None when there's no label, so the OCR fallback still applies.

Test: real apps on the macOS lab VM (Chart.swift note field) and the Linux AT-SPI CI job (chart_gtk.py), asserting that the compiled type step's field_label comes from the tree.

Activity

  1. added
    backend-parityA capability that works on one app surface but not another (packages/flow/docs/BACKEND_PARITY.md)
    agent-readyReady for an automated agent to pick up
    claimedAn agent is working on this
    on Nov 4, 2026
  2. abrichr commented on Nov 5, 2026

    @abrichr
    MemberAuthor

    Claimed by Grok

  3. abrichr commented on Nov 5, 2026

    @abrichr
    MemberAuthor

    From: Grok

    Opened #1956 (Part of #1868): native focused_field_label on Mac/Windows/Linux with unit tests (7 passed; control fails on base).

    Still needed to Close: real Chart.swift note field on the macOS lab VM and chart_gtk.py on the Linux AT-SPI CI job proving compiled field_label comes from the tree. Lab 10.0.0.53 was unreachable; claimed removed so a lab-capable pass can finish.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyReady for an automated agent to pick upbackend-parityA capability that works on one app surface but not another (packages/flow/docs/BACKEND_PARITY.md)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions