Repository navigation
fix: return 503 when registry credentials can't be fetched from Platform [NOTASK] - #1293
Merged
Merged
Conversation
…orm [NOTASK] A container request whose registry credentials could not be fetched from Seqera Platform (e.g. a pairing connection drop) was reported as "Container image ... does not exist or access is not authorized" (400), which Nextflow treats as fatal. Now the request fails with a 503 when the Platform lookup times out, is rate limited or returns a server error, so the client retries it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
munishchouhan
previously approved these changes
Nov 20, 2026
…d [NOTASK] Move the Platform-unavailable check into RegistryProxyService.getImageDigest so every digest lookup (source, target, SOCI) returns 503 rather than reporting the image as missing when Platform is unreachable. Co-Authored-By: Claude Opus 5.5 <[email protected]>
jordeu
reviewed
Nov 20, 2026
Throwing from getImageDigest made a build completed while Platform was unreachable be saved as failed, and could fail a request on the SOCI lookup. getImageDigest returns null again, while the new getContainerImageDigest fails with 503 and is used by the ContainerController request paths (source, build, multi-platform and mirror target). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ralabs/wave into fix/platform-unavailable-digest-503
…ASK] findComputeCreds swallowed a Platform timeout, rate limit or server error, so an ECR image using the compute-environment credentials was still reported as missing (400). Rethrow those errors so the container request fails with 503 as for the other credentials lookups. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…le [NOTASK] Co-Authored-By: Claude Opus 5.5 <[email protected]>
Collaborator
Author
|
@jordeu I addressed both of your comments, including the |
jordeu
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When Wave can't fetch the registry credentials for a container request because Seqera Platform is unreachable, it now returns 503 instead of 400 "Container image ... does not exist or access is not authorized". Nextflow retries a 503, while it treats a 400 as fatal and aborts the run.
Rationale
On 2026-11-17 a node rotation in
tower-proddropped the pairing websockets between Wave and the self-hosted Platforms. A Harbinger Health run that had been going for 20.5 hours was then aborted:pairing-inbound-queue). Wave's client then raisedHttpResponseException408.RegistryProxyService.getImageDigestcaught the error and returnednull.ContainerControllerturned thenulldigest into a 400 "does not exist or access is not authorized", and Nextflow aborted the run.The image existed and the credentials were valid. The failure was temporary, but Wave reported it as permanent.
Incident analysis: https://claude.ai/artifact/S9t2NkiVi42sFHwg9V1NcV
Changes
RegistryProxyService.getImageDigestOrUnavailable(image, identity): a variant ofgetImageDigestfor the container request paths. When the lookup fails because Platform timed out (408), rate-limited (429) or returned a 5xx, it throwsHttpResponseException(503). Any other failure still returnsnull, so a 401/403/404 from Platform or a missing image is still reported as a 400.RegistryProxyService.isPlatformUnavailable(Throwable): walks the cause chain for anHttpResponseException. Only the Platform client raises that type.ContainerControlleruses the new method for every digest lookup in the request path: the source image (getContainerDigest), the build target (checkBuild), the multi-platform target (checkMultiPlatformBuild) and the mirror target (checkMirror). If a target lookup returnednullhere, Wave would start a rebuild or re-mirror that needs the same Platform credentials, so it fails with a retryable 503 instead.CredentialServiceImpl.findComputeCredsrethrows Platform 408/429/5xx errors instead of returningnull, so ECR images that use the compute-environment credentials also get the 503. Other errors are still logged and returnnull.getImageDigeststill returnsnullon any error. Build completion (ContainerBuildServiceImpl.onJobCompletion) and the SOCI lookup keep using it, so a build that finishes while Platform is unreachable is not saved as failed, and the best-effort SOCI lookup cannot fail the request.Impact
POST /v1alpha2/containerreturns 503 with the messageUnable to fetch the registry credentials for image '...' from Seqera Platform - try again laterwhen Platform can't be reached. Previously it returned a 400.Testing
RegistryProxyServiceTestcover the error classification, the 503 and success paths ofgetImageDigestOrUnavailable, and thenullresult ofgetImageDigestwhen Platform is unavailable.CredentialsServiceTestcoversfindComputeCredsrethrowing a Platform timeout and returningnullon other errors.ContainerControllerTestexpectations updated for the new method.compileTestGroovy). Tests are left to CI.Additional Notes
The infrastructure side (ALB IP targets for prod Wave) is in a separate platform-deployment PR.
🤖 Generated with Claude Code