[claytondukes]

Summary

Three flags that let a customer capture a run the way LogZilla's certification harness does, so a model tested in their own lab compares like for like with a published row:

• --tag TAG: the saved files go under <save-dir>/<tag>/, one directory per model or hardware setup (needs --save-dir; letters, digits, ., _, -).
• --deploy-config PATH with --reload-wait SECONDS (default 30): before the first question, scp the ai.yaml to /etc/logzilla/settings/ai.yaml as root, ssh root@<host> logzilla settings reload ai, wait. A failed step stops the run with the remote's own message. With --save-dir, a copy is saved as ai-config.yaml with every *_KEY, *_TOKEN and API_KEY value replaced by <redacted>.
• --export-trace with --phoenix-url and --phoenix-project (or LZ_PHOENIX_URL / LZ_PHOENIX_PROJECT, or the --deploy-config file's TRACING_SERVER_URL / TRACING_PROJECT_NAME): after each answer, the question's trace is found on Phoenix (root spans inside the question's window, five seconds of slack, candidates exported and matched on the question text, never guessed) and saved as <timestamp>-<slug>-trace.json beside the result. A missing or ambiguous trace is an error for that question ("trace": {"status": ...} in the JSON, exit 2), because a run without its trace cannot be scored.

The JSON result gains a trace object when the export ran. The GraphQL queries are the ones the harness uses against Phoenix's schema (node(id: ID!), Project.spans(first: Int!, timeRange, filterCondition: "parent_id is None"), getTraceByOtelId). No new dependency: requests carries the GraphQL calls, the config copy is redacted line by line.

Verification

• python3 -m unittest discover -s tests -v on d34f6d2 -> Ran 22 tests, OK (the redaction incl. quoted keys, block scalars, continuation lines and the refused flow style; the tracing settings with trailing comments; the tag pattern and directory; the IPv6 scp target; the trace match: exported, missing, ambiguous, over the cap, a longer question not matched; the bounded ingestion wait and the lazily resolved project; a Phoenix failure as a recorded trace error; the lookup off the event-loop thread).
• python3 -m py_compile lz-ai-query.py; --help renders the new capture group and the example.
• Written against Python 3.8 syntax (no union types, no removeprefix, no walrus); checked by reading, not run on 3.8.
• Not exercised against a live server from this branch: the same three steps run in LogZilla's harness against the lab server with the same GraphQL queries.

Review rounds

Three Copilot rounds, every finding fixed RED-first (a failing test committed before the fix): round 1 seven threads (the redaction's quoted keys and block scalars, substring trace matching, YAML comments kept in values, the IPv6 scp target, no ingestion wait, a Phoenix failure discarding a captured answer, the project looked up before its first trace); round 2 two threads (secrets continued on the lines below the key, the lookup blocking the event loop); round 3 one suppressed finding (a flow-style secret, now refused before the deploy). Converged; the merge is LogZilla's call, the tool being customer-facing.

[Copilot]

🟡 Changes recommended

Unresolved defects can leave plaintext credentials in saved configuration and export unrelated traces, while project lookup and trace-export failures can abort otherwise valid runs.

7 open issues

• Redaction misses quoted keys and block scalar secrets · New
• Substring question matching selects unrelated traces · New
• Trailing YAML comments corrupt scalar setting values · New
• IPv6 scp destinations need bracketed host literals · New
• Single trace lookup misses asynchronously ingested spans · New
• Phoenix export failures discard successful answers · New
• Project lookup fails before new Phoenix project creation · New

What changed in this PR

Adds capture options to the query CLI so lab runs can save configuration, results, and Phoenix traces for comparison with LogZilla’s certification results.

Changes:

• Adds tagged output directories and remote AI configuration deployment with a reload wait and redacted configuration copy.
• Adds Phoenix trace lookup, export, and per-question trace status.
• Documents the capture workflow and adds tests for redaction, configuration parsing, tags, and trace matching.

File	Description
tests/​test_capture.py	Adds focused tests for capture helpers and trace matching.
README.md	Documents capture flags, workflow, and trace failure behavior.
lz-ai-query.py	Implements deployment, tagged output, configuration redaction, and trace export.
.env.example	Adds optional Phoenix server and project settings.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Continued YAML values can leak credentials into the saved configuration, and blocking trace lookups can cause WebSocket heartbeat timeouts.

2 open issues

• Multiline YAML secrets remain exposed in redacted config · New
• Blocking trace lookup can time out open WebSocket connections · New

7 resolved since last review

• Substring question matching selects unrelated traces
• Redaction misses quoted keys and block scalar secrets
• Project lookup fails before new Phoenix project creation
• Phoenix export failures discard successful answers
• Single trace lookup misses asynchronously ingested spans
• IPv6 scp destinations need bracketed host literals
• Trailing YAML comments corrupt scalar setting values

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Flow-style YAML bypasses the anchored key regex, allowing plaintext API credentials to be copied into ai-config.yaml despite the redaction guarantee.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Flow-style YAML secrets bypass redaction

lz-ai-query.py:711

A flow-style YAML document such as {"MODEL_TYPE": "anthropic", "API_KEY": "sk-live-secret"} never matches the anchored key regex, so this branch copies the credential verbatim into ai-config.yaml. This leaves plaintext secrets in the artifact despite the redaction guarantee. Please handle secret values structurally, or reject unsupported YAML forms before saving the copy, and add a regression case for a flow-style document.

2 resolved since last review

• Multiline YAML secrets remain exposed in redacted config
• Blocking trace lookup can time out open WebSocket connections

_{Review effort: Balanced}

[claytondukes]

Copilot suppressed finding fixed - e0663f2d020a in d34f6d2

[Copilot]

🔵 Needs a closer look

A critical redaction bypass leaves plaintext credentials in saved ai-config.yaml files when a secret value starts on the line after its key.

No open issues 🎉

_{Review effort: Balanced}