Small iOS gate, full nightly E2E on both platforms, and a cost-controlled iOS shard pilot

Status: Revised plan v9 — 29 October 2026. Implementation and cutover remain pending.
Repos: bitkit-e2e-tests, bitkit-ios, bitkit-android
Existing mainnet orchestration dependency: private bitkit-nightly

This supersedes v8's conditional broader local-money schedule and the earlier v7 implementation contract. Earlier comments and approvals describe earlier revisions; they are retained as history. The changes below preserve a small automatic PR gate while adding required coverage for risky changes, the full applicable E2E suite every night on both iOS and Android regardless of source changes, and a bounded iOS experiment for parallel execution.

Outcome

• Keep Appium and the shared E2E fixtures.
• Make the default iOS gate small, explicit, and reliable.
• Run deeper tests automatically when the changed behavior needs them; optional labels can request more coverage.
• Run the full applicable E2E inventory every night on both iOS and Android, even when app, E2E and backend revisions are unchanged.
• Compare one serial iOS heavy pack with two isolated shards before changing its default execution.
• Reduce duplicated work, repeated failures, and unnecessary paid runner time.
• Keep the Android local merge gate unchanged.

Current baseline and corrections to v7

Inspected revisions:

• The Mini migration is complete: iOS #923 moved E2E to namespace-profile-mac, with a separate temporary Namespace Linux backend. App compilation remains on macos-latest.
• @ios_gate and @ios_nightly select real tests. The hosted ios_nightly_ui shard already exists. Remove the old “empty green” baseline from the implementation plan.
• The send suite no longer initializes Bitcoin RPC unconditionally in its shared hook.
• Three sampled Namespace jobs started 19–22 seconds after job creation and overlapped on separate runners. This proves concurrency in that sample, not unlimited capacity or a queue-free service.
• Their first E2E attempts took about 55–58 minutes. Repeated Lightning HeaderMenu failures added roughly 14–19 minutes of retries. Investigate the app state and navigation helper before treating this as a driver problem: run 1, run 2, run 3.
• Standard GitHub-hosted compute is free for this public repository; Namespace compute and associated resources are billed separately. A sampled Namespace Mac was 6x14. Published rates are $0.06/min prepaid equivalent or $0.09/min overage: a 75-minute Mac job is about $4.50/$6.75 before backend/storage charges. These are list-price estimates, not an audited invoice. GitHub policy, Namespace pricing.

What stays automatic

Risk selection and freshness

Use a reviewed, versioned map from changed paths/behaviors to packs:

• Send/receive, amount and fee handling, Lightning/channel operations, wallet startup/recovery, and core dependency/configuration changes select the relevant money/recovery coverage.
• Hardware changes select hardware coverage.
• Backup, migration, persisted-state schema and upgrade-path changes select migration/restore coverage.
• Pubky/Paykit payment, proof and subscription changes select the relevant integration/staging coverage as well as shared-wallet coverage where affected.
• E2E helper, fixture and workflow changes exercise their affected consumers on both platforms.

Do not pretend existing packs cover new product behavior: record the gap and add the appropriate native, integration or E2E case. Shared or ambiguous paths select a conservative superset; an unmapped risk must not silently produce “no tests required.”

Keep a required e2e-risk-status aggregator on eligible iOS PRs. It passes only when selection succeeds and either explicitly requires no packs, or every required pack completes its expected tests on the tested revision. Infrastructure failures, blocked prerequisites and unexpected skips remain incomplete/failed coverage.

Evidence records the app merge commit/tree, PR head, E2E SHA and backend/fixture revision. New relevant app, base, test or fixture changes invalidate earlier acceptance. Never order commit hashes chronologically. Report against the current PR revision and an explicit run generation.

Stacked PRs, schedules and release branches

• Preserve the current rule that automatic iOS Appium PR runs start when a PR targets master; do not restore a full paid run on every intermediate stacked PR. Native checks retain their current triggers. Explicit pack dispatch remains available while stacked.
• Retargeted runs must test the verified merge candidate and publish current-head checks, preserving the guarantees introduced in iOS #1161.
• Preserve existing nightly staging, UI, mainnet and migration coverage during rollout, and expand the nightly execution plan to the full applicable suite on both platforms every night. Pin the latest master of each app and the E2E/backend revisions for that nightly batch. An unchanged input fingerprint or a previous green run must never suppress the next night's execution. A build or dependency failure is an incomplete nightly, not a reason to substitute an older passing result.
• Full nightly coverage does not depend on adopting parallelism: retain serial iOS execution if the pilot misses its targets. Android keeps its existing execution approach; the two-shard experiment is initially iOS-only. No additional full run on every master push is required.
• Retain existing release-* migration triggers during this rollout. Replacing them with an explicit candidate-validation mechanism is a separate, coordinated cutover with no coverage gap. Do not implement v7's blanket removal of automatic release migrations.

Full nightly inventory and completeness

“Full” means every applicable case/configuration in a reviewed inventory, not just the current @ios_gate, @ios_nightly, local PR grep or e2e-all label. Run the required backend jobs as one logical nightly batch per platform. A case need not be duplicated on every backend it can technically use, but every declared platform/backend obligation must execute that night.

Inventory at the inspected E2E revision

The following is the complete source-case inventory at 4c0cbef019220efed01ed3080c1f0ccc1fb627a3, including conditional, skipped and utility definitions. Ranges below enumerate individual IDs; they are not permission to use a loose substring grep. The destination column specifies the nightly mapping to implement and validate, not a claim that all jobs are wired or passing today.

Sources: test specifications, mainnet orchestration contract. Mainnet schedules and private configuration remain in bitkit-nightly; link their results into nightly accounting without copying secrets or moving orchestration into the public test repository.

Machine-readable manifest and result reconciliation

• Commit a versioned manifest in bitkit-e2e-tests with stable case ID, source file, platform, backend/network, build mode, required fixtures, executing workflow/pack, and applicability/disposition. Expand dynamic definitions and relevant migration source/target variants. Include an explicit row/disposition for both platforms, even when a case is inapplicable to one.
• Before cutover, validate the backend assignments above and map every applicable entry to an exact executable job, including unresolved mainnet support and migration source/target variants. Discover cases independently of the selected workflow greps and compare them with the manifest, so a new, untagged, conditionally registered or accidentally filtered-out case cannot go unnoticed. New specs and cases must update the inventory in the same change; this snapshot is not a permanent allowlist.
• Platform-inapplicable/retired cases require a specific reason and supporting reference. Unsupported automation or a missing fixture for an applicable feature is a coverage gap with an owner and tracking issue. Temporary quarantines need a reason, owner and review/expiry date; they do not count as passes or full coverage.
• Reassess existing skips individually: Blocktank-related transfer_1, transfer_max, multi_address_2, hardware_wallet_3, and settings_08 (whose source comment says address switching is unavailable). A .skip or stale comment alone is not an accepted exclusion.
• The expected execution key is nightly batch + platform + case ID + required backend/build/migration variant. Reconcile exact IDs and outcomes, not only counts. Report passed, failed, unexpectedly skipped, missing/not started, blocked, quarantined and documented inapplicable cases separately, including partial assertions or early returns.
• A platform's full nightly result passes only when every applicable expected obligation has a verified pass in that night's batch and inventory validation succeeds. Missing jobs, unexpected skips, zero matches for an expected nonempty pack, selection errors and unmet prerequisites make it failed/incomplete. An empty platform/backend combination is valid only when explicitly explained by the manifest.
• Start each night with fresh run-scoped retry state. A verified pass may satisfy later retries within the same batch/configuration; yesterday's pass, cached locks or a different platform/backend cannot satisfy tonight's obligation. Keep first-attempt failures visible even after recovery.
• Attempt all independent platform/backend jobs even when a sibling fails; report blocked dependent jobs explicitly. Publish a per-platform summary plus a combined coverage summary with source/artifact identities, expected/actual IDs, dispositions, timings, cost and links to every contributing run.
• Monitor the expected nightly batches as well as test results. A schedule that never starts or never completes must surface as missing/incomplete with an owner, not remain invisible because no failing job exists.

Small iOS smoke gate

Start with one standard GitHub-hosted macOS job using BACKEND=regtest, with no local Docker dependency. Target approximately 10–15 minutes of test execution, measured separately from queue/build/setup time; this is a design target, not an established runtime.

Initial candidate behaviors:

1. Create a wallet and reach usable Home.
2. Open Receive, validate its receiving details, and exercise basic amount validation.
3. Receive funds and send a small on-chain payment with explicit outcome checks.

Use onboarding_1, receive_1 and onchain_1 as starting references. They currently contain broader UI/activity permutations; extract focused smoke cases if needed and retain displaced assertions in native UI or broader E2E coverage. The permanent contract is an explicit set of stable case IDs, not “any three matching tests.”

Requirements:

• One attempt for the smoke verdict; any diagnostic rerun is separately identified.
• Verify the exact expected cases executed and passed; fail on zero matches, missing cases and unexpected skips.
• Validate recipient, amount and transaction identity using available independent backend/receiver evidence. A success screen and “balance decreased” alone are insufficient.
• Establish isolated wallet/address fixtures and prove concurrent smoke runs do not interfere through shared regtest services.
• Check dependency readiness with bounded waits and distinguish unavailable infrastructure from an assertion failure.
• Keep smoke required even when another pack overlaps its features.
• Keep the existing iOS gate required while smoke runs in shadow.

Two-shard experiment for heavier tests

Scope: initially iOS only; split the same heavier iOS test inventory across two independent Namespace jobs. Do not run the full inventory twice as the proposed production design.

• Build once per configuration; both shards download the same immutable app artifact.
• Pin the same app, E2E and backend revisions in serial and parallel comparisons.
• Each shard owns its simulator, Appium/WDA session, retry state and temporary Linux backend. Mining, RBF, LND and Trezor state must not be shared across shards.
• Keep tests sequential within each shard initially. Do not implement the first experiment by increasing WDIO workers inside the existing shared simulator/backend.
• Partition by measured duration and fixture dependencies, not test count. Keep related suites together until their state independence is established.
• Re-audit the pack inventory. For example, multi_address_3 now exercises RBF and belongs in the controlled-mining assessment; its old exclusion from money/local must not be copied unchanged.
• Require an exact disjoint partition: union of both shards equals the baseline inventory, with no accidentally omitted or duplicate cases. Isolate artifacts and retry locks by run/shard/attempt.
• Shard completion feeds one aggregate result; a green shard cannot hide a failed or missing sibling.

Bounded pilot and acceptance

Use manual dispatch or an experimental label initially. Run at most one comparison pair at a time, with at most two simultaneous trial shards; do not enable a serial-plus-parallel trial on every PR.

Capture test start/end times, setup, first-attempt outcomes, retries, total Mac and Linux unit minutes, cache/storage overhead, queue time and end-to-end completion time. Compare equivalent fresh fixtures; a retry on a previously modified chain is not a fresh control.

Proposed adoption targets:

• At least 30% less elapsed heavy-pack execution time.
• No more than roughly 10–15% additional billed compute for equivalent work.
• No worse first-attempt reliability or defect detection; no missing/duplicate tests.
• Demonstrated cross-shard isolation and concurrent-PR isolation.
• Clear failure artifacts and reliable teardown.

Use at least ten matched serial/parallel pairs across representative revisions and fixture conditions as an initial operational sample; this is not proof of a low long-term flake rate. Budget and hard job/step limits must be declared before dispatching the batch. Compare first attempts separately and charge all retries/cancellations to the experiment. If flaky failures prevent a fair comparison, fix them before expanding the pilot.

Keep the serial path available for rollback. A small smoke gate stays on one runner unless measurements justify splitting it.

Cost and reliability controls

1. Avoid redundant work without reducing nightly coverage. Preserve PR draft/path filtering, stacked-PR deferral, same-PR cancellation and build-artifact reuse. Optional labels add coverage; overlapping packs execute each required case/configuration once within a batch. Reuse matching build artifacts, not prior test results. PR change detection and unchanged-input deduplication must not suppress either platform's nightly suite; keep nightly concurrency groups separate from PR cancellation.
2. Reduce retry waste. Investigate the repeated Lightning failure first. Record first-pass and pass-after-retry separately. Keep retry policy identical between pilot arms; subsequently evaluate replacing blanket three-attempt execution with bounded, evidence-based retries. Repeating a deterministic failure is not a reliability fix.
3. Retain evidence after recovery. Publish a compact manifest for every run and upload diagnostics whenever any attempt failed, even if the final job is green. Capture best-effort evidence on hook/setup failures and cancellation. Retain failure media with bounded retention; avoid uploading full success videos by default.
4. Bound resource lifetime. Add explicit job/step timeouts using observed durations and an agreed margin. Always attempt backend teardown; use a bounded backend TTL for interrupted cleanup. The current four-hour backend lifetime is a ceiling to review, not a reason to let failed jobs idle.
5. Measure actual usage. Export Namespace instance usage by workflow, PR/nightly batch, platform, backend, shard and attempt. Track Mac/Linux unit minutes, paid cache/storage and cost per completed validation. Use the actual plan/profile rates; queueing, free GitHub compute and paid Namespace runtime are different measurements.
6. Control concurrency and spending. Keep a documented runner-profile/workspace concurrency cap and a daily/weekly usage alert. Increasing shards must not silently raise the account-wide cap. Reserve capacity/budget for the full nightly suites on both platforms. Budget pressure can defer optional experiments; it must not silently drop nightly coverage or turn missing required coverage green. Report any capacity/quota failure as incomplete coverage and alert the owner. Unchanged source inputs are never a reason to omit a nightly.
7. Repair cache reuse deliberately. iOS #1161's cache discussion records lost new per-PR SwiftPM saves on a miss. This mainly increases the GitHub-hosted build time before Namespace starts. Preserve retarget cache isolation; use a separate trusted cache-warming or execution/job design. Do not reintroduce a conditional cache-mode expression already rejected by GitHub.
8. Spend on useful coverage. Keep broad native/service assertions below E2E where possible. Review long multi-purpose scenarios and fixture setup before buying more parallel capacity. Physical-device and reviewer/agent journeys remain complementary acceptance work, not counted as automated CI passes.

Labels, selection and reporting

Retain useful labels: e2e-money-gate, e2e-local, e2e-staging, e2e-migration, e2e-all.

• Labels request extra runs; they cannot remove automatically required packs.
• Optional one-shot labels may be consumed, with documented re-add behavior. Required risk coverage reruns when its tested inputs change.
• e2e-local/e2e-all supersedes a separate money invocation when it contains the same required cases. Smoke remains independent.
• e2e-all fans out once to the applicable local, staging and migration packs. Snapshot the event's requested pack set before labels are consumed; do not let one workflow consume another's request.
• Keep workflow-scoped, PR-scoped concurrency groups. An unrelated label must not cancel a required run.
• Sticky results include tested app/base/E2E/backend identities, expected/executed/skipped case counts, first-attempt/final results, duration, usage and the Actions URL. Older runs cannot overwrite newer generations.
• Preserve existing feature tags and dual-tag during any rename. Do not require all current describe tags to move to case titles just for this redesign.
• Companion-branch fallback is based on branch existence, not tag compatibility. If an existing feature branch lacks the expected inventory, fail clearly and update the branch or choose an explicit ref. Do not silently switch to main and discard feature-specific tests.
• Fail-on-zero must be compatible with retry locks: a retry's skips count only as prior verified passes for the same run/shard/input identity, never as evidence that an empty selection succeeded.

Rollout and acceptance

1. Establish the baseline

• Commit and validate the complete per-platform/backend nightly manifest, including conditional cases, fixture preparation, private mainnet coverage and explicit inapplicable/retired/gap dispositions.
• Investigate the repeated Lightning failure.
• Add result manifests, per-case timing and first-failure artifact retention.
• Record current cost/queue/runtime distributions, budget/capacity for both full nightly suites, and a separate bounded iOS pilot budget.
• Implement and validate automatic PR risk selection without removing current coverage.
• Verify nightly completeness checks detect an omitted case/job, unexpected skip, missing fixture, stale prior-night lock and a schedule that never starts; verify unchanged revisions still produce a new full batch on both platforms.

2. Add smoke in shadow

• Implement focused smoke with explicit selection and independent payment checks.
• Verify detection failure, build failure, path miss, unexpected skip, empty selection, stale locks and full success.
• Exercise concurrent PRs and known-broken/fixed builds; prove smoke detects defects.
• Collect representative repeated runs with first-attempt results and timing. Three greens alone are not the reliability bar.
• Keep the existing gate required throughout this phase.

3. Run the iOS two-shard pilot

• Compare matched serial/parallel runs under the declared limits.
• Verify exact coverage equivalence, fixture isolation, cleanup and cross-PR concurrency.
• Meet the elapsed-time/compute targets without increased flakiness; otherwise keep serial execution.

4. Cut over gates without losing coverage

• Require e2e-smoke-status and e2e-risk-status in the same coordinated window that retires the old universal iOS e2e-status requirement.
• Both aggregators fail closed on selection/detection errors, failed builds, cancellations and missing required cases. Explicit “not required” is valid only after successful selection.
• Preserve native checks, automatic relevant PR packs, existing scheduled coverage and release-candidate acceptance.
• Activate full nightly E2E on both iOS and Android regardless of source changes, with complete platform/backend manifests and result reconciliation. Adopt the measured iOS heavy-pack execution mode independently; retain serial execution if the pilot fails its targets.
• Preserve retarget reporting and current-revision ownership for both new checks.
• Verify Android's unchanged gate against the same E2E revision.
• Document rollback to the existing Namespace gate and e2e-status; rollback does not depend on restoring the retired Mini.

No ruleset, workflow, runner profile or schedule changes are made by this issue update.