Skip to content

fix: return 503 when registry credentials can't be fetched from Platform [NOTASK] - #1293

Merged
pditommaso merged 9 commits into
masterfrom
fix/platform-unavailable-digest-503
Nov 20, 2026
Merged

pditommaso merged 9 commits into
masterfrom
fix/platform-unavailable-digest-503

Conversation

@pditommaso

@pditommaso pditommaso commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

When Wave can't fetch the registry credentials for a container request because Seqera Platform is unreachable, it now returns 503 instead of 400 "Container image ... does not exist or access is not authorized". Nextflow retries a 503, while it treats a 400 as fatal and aborts the run.

Rationale

On 2026-11-17 a node rotation in tower-prod dropped the pairing websockets between Wave and the self-hosted Platforms. A Harbinger Health run that had been going for 20.5 hours was then aborted:

  1. Wave found the ECR credential for the image, but every attempt to fetch its keys from the customer's Platform timed out (pairing-inbound-queue). Wave's client then raised HttpResponseException 408.
  2. RegistryProxyService.getImageDigest caught the error and returned null.
  3. ContainerController turned the null digest into a 400 "does not exist or access is not authorized", and Nextflow aborted the run.

The image existed and the credentials were valid. The failure was temporary, but Wave reported it as permanent.

Incident analysis: https://claude.ai/artifact/S9t2NkiVi42sFHwg9V1NcV

Changes

  • RegistryProxyService.getImageDigestOrUnavailable(image, identity): a variant of getImageDigest for the container request paths. When the lookup fails because Platform timed out (408), rate-limited (429) or returned a 5xx, it throws HttpResponseException(503). Any other failure still returns null, so a 401/403/404 from Platform or a missing image is still reported as a 400.
  • RegistryProxyService.isPlatformUnavailable(Throwable): walks the cause chain for an HttpResponseException. Only the Platform client raises that type.
  • ContainerController uses the new method for every digest lookup in the request path: the source image (getContainerDigest), the build target (checkBuild), the multi-platform target (checkMultiPlatformBuild) and the mirror target (checkMirror). If a target lookup returned null here, Wave would start a rebuild or re-mirror that needs the same Platform credentials, so it fails with a retryable 503 instead.
  • CredentialServiceImpl.findComputeCreds rethrows Platform 408/429/5xx errors instead of returning null, so ECR images that use the compute-environment credentials also get the 503. Other errors are still logged and return null.
  • getImageDigest still returns null on any error. Build completion (ContainerBuildServiceImpl.onJobCompletion) and the SOCI lookup keep using it, so a build that finishes while Platform is unreachable is not saved as failed, and the best-effort SOCI lookup cannot fail the request.

Impact

  • POST /v1alpha2/container returns 503 with the message Unable to fetch the registry credentials for image '...' from Seqera Platform - try again later when Platform can't be reached. Previously it returned a 400.
  • The Nextflow Wave client already retries 429/500/502/503/504, so no client change is needed.

Testing

  • Unit tests in RegistryProxyServiceTest cover the error classification, the 503 and success paths of getImageDigestOrUnavailable, and the null result of getImageDigest when Platform is unavailable.
  • CredentialsServiceTest covers findComputeCreds rethrowing a Platform timeout and returning null on other errors.
  • ContainerControllerTest expectations updated for the new method.
  • Compiled locally (compileTestGroovy). Tests are left to CI.

Additional Notes

The infrastructure side (ALB IP targets for prod Wave) is in a separate platform-deployment PR.

🤖 Generated with Claude Code

…orm [NOTASK]

A container request whose registry credentials could not be fetched from
Seqera Platform (e.g. a pairing connection drop) was reported as
"Container image ... does not exist or access is not authorized" (400),
which Nextflow treats as fatal. Now the request fails with a 503 when the
Platform lookup times out, is rate limited or returns a server error, so
the client retries it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@pditommaso
pditommaso requested a review from a team as a code owner November 19, 2026 20:25
munishchouhan
munishchouhan previously approved these changes Nov 20, 2026
…d [NOTASK]

Move the Platform-unavailable check into RegistryProxyService.getImageDigest
so every digest lookup (source, target, SOCI) returns 503 rather than
reporting the image as missing when Platform is unreachable.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@pditommaso
pditommaso requested a review from jordeu November 20, 2026 15:07
pditommaso and others added 2 commits November 20, 2026 16:45
Throwing from getImageDigest made a build completed while Platform was
unreachable be saved as failed, and could fail a request on the SOCI
lookup. getImageDigest returns null again, while the new
getContainerImageDigest fails with 503 and is used by the
ContainerController request paths (source, build, multi-platform and
mirror target).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ralabs/wave into fix/platform-unavailable-digest-503
@pditommaso
pditommaso requested a review from jordeu November 20, 2026 15:47
pditommaso and others added 2 commits November 20, 2026 16:48
…ASK]

findComputeCreds swallowed a Platform timeout, rate limit or server error,
so an ECR image using the compute-environment credentials was still
reported as missing (400). Rethrow those errors so the container request
fails with 503 as for the other credentials lookups.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@pditommaso

Copy link
Copy Markdown
Collaborator Author

@jordeu I addressed both of your comments, including the findComputeCreds fix, which is now in this PR. Could you take another look when you have a moment? Thanks!

@pditommaso
pditommaso merged commit 0e6e1f8 into master Nov 20, 2026
6 checks passed
@pditommaso
pditommaso deleted the fix/platform-unavailable-digest-503 branch November 20, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants