Skip to content

perf(ci): decline the stranded-prod check on a busy queue before it joins the deploy lock - #4796

Merged
devantler merged 1 commit into
mainfrom
claude/ci-stranded-check-queue-preflight-4598
Oct 12, 2026
Merged

devantler merged 1 commit into
mainfrom
claude/ci-stranded-check-queue-preflight-4598

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

Every hour a check looks for a production left on a release that never reached the main branch, and it lined up with the production deploys before it knew whether there was anything to repair. In one measured run it held that place for nine minutes while waiting for a machine, then worked for 18 seconds and found nothing to do. When pull requests are waiting to merge, that wait competes with real deploys.

What

The check now first asks, without lining up, whether pull requests are waiting to merge, and stops there if they are. If none are waiting it continues as before and asks again once it holds its place, so a pull request that arrives in between still stops it. If the first question cannot be answered, the run fails visibly and repairs nothing.

Fixes #4598

👉 After merge/promotion: nothing for the maintainer; the engineer compares the hourly runs before and after and records one run on a busy queue on the issue.

…oins the deploy lock

The hourly check joined prod-deploy before it had a runner or knew whether the
merge queue was busy. A lock-free preflight job now reads the same queue state
first; the locked job runs only when the preflight found the queue idle, and
keeps its own gate under the lock.

Fixes #4598

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 48 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: fb7d6b27-4ce9-440e-b773-2da26189b042

📥 Commits

Reviewing files that changed from the base of the PR and between abd5440 and 1d66a01.


📒 Files selected for processing (2)
  • .github/workflows/restore-stranded-prod.yaml
  • scripts/tests/test-restore-stranded-prod-workflow.sh

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T17:16:43.200679Z 1d66a01 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 1d66a01f71

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Evaluation at 1d66a01f71f35af3711e52c7bb5dd85bd8ca5420

This workflow runs only on a schedule or a manual start from main, so it cannot be exercised from the pull request without starting a job that holds the production lock. What I could check before merge:

  • The preflight's read, against the live queue. I ran the gate script read-only against this repository at about 17:15 UTC, while fix(ci): restore runner-isolation evidence collection #4795 was queued. It answered stranded=false ("1 queued, 1 merge-group run(s) unfinished") and exited 0. With this change that answer skips the locked job.
  • A job skipped by its condition does not wait in the lock. GitHub's concurrency documentation does not say this, so I looked for a real case. In run 38274081392, Deploy to Prod (same prod-deploy group, condition false on a pull request) was created and skipped at 16:06:16 UTC. At that moment the merge-group deploy in run 38272963849 held the lock (15:59:44 to 16:11:49 UTC). The skipped job did not wait for it.
  • Wiring. The workflow test now also requires: the preflight holds no lock and only read permissions, it runs the same gate script, the locked job depends on its answer, and the locked job keeps its own gate. Each of the 8 new mutations fails the named assertion; all 19 pass locally and in CI.
  • Unknown stays unknown. The gate script is unchanged, so a failed or malformed read still exits 1 and writes nothing. In the preflight that fails the job, and a job whose dependency failed is skipped.

Baseline for the comparison after merge (six scheduled runs today, 11:50 to 16:50 UTC): runner wait 2 to 9 s, execution 20 to 52 s. The 539 s wait on the issue is not typical, so the gain is on busy days. On an idle queue the run now uses two short jobs instead of one.

Still owed after merge, recorded on #4598: one scheduled run on a busy queue showing the locked job skipped, and the timings beside this baseline.

@devantler
devantler marked this pull request as ready for review October 12, 2026 17:28
@devantler
devantler added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit 3e152a1 Oct 12, 2026
33 checks passed
@devantler
devantler deleted the claude/ci-stranded-check-queue-preflight-4598 branch October 12, 2026 17:44
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Avoid reserving deployment capacity for scheduled no-op restore checks

1 participant