Skip to content

Implement GPU Operator high availability - #3103

Merged
aryangorwade merged 1 commit into
NVIDIA:mainfrom
aryangorwade:high-availability
Nov 19, 2026
Merged

aryangorwade merged 1 commit into
NVIDIA:mainfrom
aryangorwade:high-availability

Conversation

@aryangorwade

@aryangorwade aryangorwade commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Implements HA changes proposed in #582.

  • Add operator.replicaCount
  • Make operator pods release lease on shutdown
  • Add soft pod anti-affinity
  • Add pod disruption budget
  • Add cache warmup as a flag
  • Add helm template rendering test cases
  • Add an end-to-end test where there are 2 replicas and one takes over when leader is killed

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

Validated via manual testing on cluster.

Scenario Action Measure Expected Measured data What actually happened
Unplanned leader loss Kill the leader pod with SIGKILL Time to the first reconcile by the new leader Lease expiry—15s by default—plus cache sync. Sync grows with node/pod count without warmup; near zero with warmup. First reconciliation: 18.331/20.026 s; scaled: ≈21.008/16.010 s; post-acquisition cache-sync upper bound: ≈229/21 ms. Election delay exceeded the stated bound.
Planned handover Delete the leader pod gracefully Same About 2s with release-on-cancel; up to 15s without. First reconciliation: enabled 0.685/1.529/1.514 s; disabled 17.362/15.288/16.751 s. Enabled passed; disabled exceeded the stated 15-second maximum.
Node loss Stop the kubelet on the leader’s node Same; whether the pod is rescheduled Lease expiry drives failover, rather than the five-minute node eviction. Kubelet stop: no failover (52 s); leadership failover: 16.030 s; rescheduled: ≈344.752 s. Kubelet stop retained leadership; abrupt node loss triggered lease expiry driven failover.
Chart upgrade with two replicas helm upgrade with a new image Leader changes; time with no leader; two-leader check One handover; no interval with two leaders. Handover count: 1/1/1; empty-lease: 1.35/0.82/0.98 s; maximum sampled leaders at a time: 1. One handover per run.
Control-plane drain Drain the node running the leader Drain time; whether the budget blocks it Completes with two replicas; blocked with one replica if a budget exists. Two replicas: 18.056 s; one replica: blocked for entirety of 25 s timeout. Two-replica drain completed; a manual single-replica PDB blocked draining.
Long run under churn Two hours with simulated nodes and node/pod churn Involuntary leader changes; renew latency; memory per replica Zero changes; renewal latency under 10s; standby memory equals leader memory only with warmup. Changes: 0; maximum renewal: 35.078 ms; leader/standby RSS: 67.96/58.50 MiB (warm). Stability and latency passed; RSS supports the expected memory trend.
One-node control plane Install with replicaCount: 2 on a cluster with one control-plane node Pod status Both replicas schedule; no Pending pod. Running/Ready/Pending: 2/2/0; nodes: 1. Both replicas scheduled on the single node.
Throttled API server Long run with a low API priority level for the operator Lost leases Exposes the failure mode before a customer encounters it. Lost leases: 10; restarts/pod: 6; longest observed leaderless: 206.473 s. Leadership bounced; both pods repeatedly restarted; long intervals lacked leadership.
Warmup on and off Repeat unplanned leader loss with each setting Failover time; memory per replica Determines the warmup default once the cache is bounded. - -

The warmup on and off test that determines cache warmup default can't be completed because cache bounding hasn't been implemented yet. A test run was completed with warmup off:

  • Median controller-start delay: 1.51s at 2,500 nodes; 5.42s at 10,000.
  • Both replicas OOM-killed at 350Mi, at both scales.
  • No overlapping leaders or reconcile errors observed.

Acceptance criteria

One acceptance criterion was not fulfilled as written.

What was expected Measured data What actually happened
6 SIGKILL produces a new leader within the lease duration plus cache-sync time. Lease: 15 s; acquisition: 18.102/20.005 s; post-acquisition cache-sync upper bound: ≈229/21 ms Acquisition exceeded lease duration + cache sync time because of leader election observation and polling delay.

Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@aryangorwade
aryangorwade marked this pull request as ready for review October 22, 2026 02:48
@aryangorwade
aryangorwade requested a review from a team as a code owner October 22, 2026 02:48
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Walkthrough

The chart now supports configurable operator replicas, conditional PodDisruptionBudget rendering, generated soft pod anti-affinity, and optional leader-election warmup. Helm rendering tests cover default and multi-replica configurations. A reusable CI workflow creates a kind cluster with a simulated GPU node, installs three replicas, terminates the Lease holder, and checks successor election and ClusterPolicy reconciliation. CI runs this workflow for qualifying builds and makes release depend on it.

Priority: ⚪ Pending latest changes

Merge Risk: 🟡 Moderate · up to bdcab

Dependabot-authored main pushes can miss a release, and the new failover job does not enforce the stated takeover-time target. Resolve those conditions and the CI artifact and test-stability concerns before merging.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments: 2
.github/workflows/ci.yaml: 1

109-109: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Handle the Dependabot main push case.

.github/workflows/variables.yaml sets push_on_build to "false" when github.actor is dependabot[bot], even for a push event. In that case, ha-failover is skipped, and the implicit success() on release also skips the release job.

If Dependabot-authored pushes to main must release, allow the intentional ha-failover skip while still requiring successful E2E tests:

Suggested fix
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    if: >-
+      ${{
+        always() &&
+        !cancelled() &&
+        github.event_name == 'push' &&
+        github.ref == 'refs/heads/main' &&
+        needs.e2e-tests.result == 'success' &&
+        (
+          needs.ha-failover.result == 'success' ||
+          (
+            needs.ha-failover.result == 'skipped' &&
+            needs.variables.outputs.push_on_build == 'false'
+          )
+        )
+      }}
.github/workflows/ha-failover.yaml: 1

61-65: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Verify the kind and KWOK artifacts before use.

The workflow logs into GHCR before downloading and executing kind. The versioned URL is not an immutable digest, so a modified binary can read the Docker credential file. The KWOK manifests also come from the mutable v0.8.0 release tag and are applied directly with kubectl.

Verify the kind binary and both KWOK manifests with trusted SHA-256 checksums or immutable release references. Move the registry login to immediately before docker pull, after cluster creation and KWOK installation.

Move registry login after artifact setup
-      - name: Login to GitHub Container Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
       - name: Set up Helm
         uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
         with:
           version: v3.19.0
...
           kubectl wait --for=condition=Ready node/kwok-gpu --timeout=60s
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
       - name: Install three operator replicas
🧹 Nitpick comments: 3
.github/workflows/ha-failover.yaml: 1

181-181: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Check acquisition latency independently.

The workflow checks (( SECONDS - t0 <= 60 )) only after acquisition, patching, and reconciliation. This can pass when acquisition exceeds the stated lease-duration-plus-cache-sync acceptance bound but reconciliation finishes within the remaining 60 seconds. Assert the elapsed time immediately after wait_for 60 acquired against the acceptance bound, then keep reconciliation as a separate liveness check.

deployments/gpu-operator/chart_test.go: 1

125-125: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for the new template branches.

The current cases do not exercise user-provided podAntiAffinity, either disabled switch, or leader-election-warmup. Add cases for these paths and assert that the default render omits the warmup argument.

Suggested test cases
 import (
 	"os/exec"
+	"slices"
 	"strings"
 	"testing"
@@
 			check: func(t *testing.T, deployment *appsv1.Deployment, pdb *policyv1.PodDisruptionBudget) {
 				require.Equal(t, int32(1), *deployment.Spec.Replicas)
 				require.Nil(t, deployment.Spec.Template.Spec.Affinity.PodAntiAffinity)
 				require.Nil(t, pdb)
+				require.False(t, slices.Contains(deployment.Spec.Template.Spec.Containers[0].Args, "--leader-election-warmup"))
 			},
 		},
@@
 			},
 		},
+		"user-provided podAntiAffinity suppresses the generated one": {
+			values: `
+operator:
+  replicaCount: 2
+  affinity:
+    podAntiAffinity:
+      requiredDuringSchedulingIgnoredDuringExecution:
+      - topologyKey: topology.kubernetes.io/zone
+        labelSelector: {matchLabels: {app: gpu-operator}}
+`,
+			check: func(t *testing.T, deployment *appsv1.Deployment, _ *policyv1.PodDisruptionBudget) {
+				podAntiAffinity := deployment.Spec.Template.Spec.Affinity.PodAntiAffinity
+				require.NotNil(t, podAntiAffinity)
+				require.Len(t, podAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution, 1)
+				require.Empty(t, podAntiAffinity.PreferredDuringSchedulingIgnoredDuringExecution)
+			},
+		},
+		"anti-affinity and PDB can be disabled": {
+			values: "operator: {replicaCount: 2, podAntiAffinity: {enabled: false}, podDisruptionBudget: {enabled: false}}",
+			check: func(t *testing.T, deployment *appsv1.Deployment, pdb *policyv1.PodDisruptionBudget) {
+				require.Nil(t, deployment.Spec.Template.Spec.Affinity.PodAntiAffinity)
+				require.Nil(t, pdb)
+			},
+		},
+		"warmup renders the operator flag": {
+			values: "operator: {leaderElection: {warmup: true}}",
+			check: func(t *testing.T, deployment *appsv1.Deployment, _ *policyv1.PodDisruptionBudget) {
+				require.Contains(t, deployment.Spec.Template.Spec.Containers[0].Args, "--leader-election-warmup")
+			},
+		},
 	}
.github/workflows/golang-checks.yaml: 1

82-83: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Pin the Helm version for the chart tests.

The go-test job runs make coverage, which runs tests in deployments/gpu-operator. chart_test.go invokes helm template and asserts rendered resources. Without version, azure/setup-helm installs latest, so a Helm update can change CI results without repository changes. Use the existing v3.19.0 pin.

Proposed change
       - name: Install Helm
         uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
+        with:
+          version: v3.19.0

ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/gpu-operator/.coderabbit.yaml
  • Review profile: QUIET
  • Plan: Enterprise
  • Run ID: 783323a1-d0a4-45bf-aba0-10ddb9beb4ba
📥 Commits

Reviewing files that changed from the base of the PR and between aea3d72 and bdcab9d.

📒 Files selected for processing: 9
  • .github/workflows/ci.yaml
  • .github/workflows/golang-checks.yaml
  • .github/workflows/ha-failover.yaml
  • cmd/gpu-operator/main.go
  • deployments/gpu-operator/.helmignore
  • deployments/gpu-operator/chart_test.go
  • deployments/gpu-operator/templates/operator.yaml
  • deployments/gpu-operator/templates/poddisruptionbudget.yaml
  • deployments/gpu-operator/values.yaml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@aryangorwade aryangorwade self-assigned this Oct 22, 2026
@aryangorwade aryangorwade added the performance Issues/PRs that are related to performance label Oct 22, 2026
@aryangorwade
aryangorwade force-pushed the high-availability branch 2 times, most recently from b9995df to a6b9b70 Compare October 23, 2026 22:06

@kvalliyurnatt kvalliyurnatt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also need to make changes to must_gather.sh, currently in the script we do

OPERATOR_POD_NAME=$($K get pods -lapp=gpu-operator -oname -A)

To get the Operator Pod name and then below we do

echo "Get the GPU Operator Pod (status)"
$K get "${OPERATOR_POD_NAME}" \
    -owide \
    -n "${OPERATOR_NAMESPACE}" \
    > "${ARTIFACT_DIR}/gpu_operator_pod.status"

echo "Get the GPU Operator Pod (yaml)"
$K get "${OPERATOR_POD_NAME}" \
    -oyaml \
    -n "${OPERATOR_NAMESPACE}" \
    > "${ARTIFACT_DIR}/gpu_operator_pod.yaml"

echo "Get the GPU Operator Pod logs"
$K logs "${OPERATOR_POD_NAME}" \
    -n "${OPERATOR_NAMESPACE}" \
    --timestamps \
    > "${ARTIFACT_DIR}/gpu_operator_pod.log"

$K logs "${OPERATOR_POD_NAME}" \
    -n "${OPERATOR_NAMESPACE}" \
    --timestamps \
    --previous \
    > "${ARTIFACT_DIR}/gpu_operator_pod.previous.log"

Which will fail I think when there are more than one operator Pods

Comment thread cmd/gpu-operator/main.go Resolved
@aryangorwade
aryangorwade force-pushed the high-availability branch 2 times, most recently from 171ac38 to c384421 Compare November 3, 2026 01:50

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 0 new potential issues.

Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 0 new potential issues.

Devin Review

Comment thread cmd/gpu-operator/main.go Resolved

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 0 new potential issues.

Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 0 new potential issues.

Devin Review

Comment thread hack/must-gather.sh Resolved
Comment thread deployments/gpu-operator/values.yaml Resolved
@aryangorwade
aryangorwade force-pushed the high-availability branch 3 times, most recently from 7193332 to 5688d0f Compare November 6, 2026 00:40
@aryangorwade
aryangorwade force-pushed the high-availability branch 3 times, most recently from 4115258 to 1eaac3b Compare November 12, 2026 22:23
devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread cmd/gpu-operator/main.go Resolved
devin-ai-integration[bot]

This comment was marked as resolved.

@aryangorwade
aryangorwade force-pushed the high-availability branch 2 times, most recently from bde34b8 to 76298f1 Compare November 17, 2026 00:21

@rajathagasthya rajathagasthya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments.

Comment thread controllers/operator_metrics.go Resolved
Comment thread controllers/operator_metrics.go Resolved
Comment thread .github/workflows/golang-checks.yaml Resolved

@rajathagasthya rajathagasthya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work @aryangorwade!

Comment thread .github/workflows/golang-checks.yaml Resolved

@abrarshivani abrarshivani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @aryangorwade.

@@ -0,0 +1,17 @@
{{- $pdb := .Values.operator.podDisruptionBudget | default dict }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is being installed only via the helm install path and OLM bundle install is missing it. I wonder if this would be a documentation for that path or we need to add changes to OLM bundle

@aryangorwade aryangorwade Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add Added a section for HA in README.md, below Quick Start.

Operator high availability

Helm installations run one operator replica by default. Increase operator.replicaCount to run multiple replicas with leader election. Unless explicitly disabled, the chart also adds soft pod anti-affinity and a PodDisruptionBudget (PDB) requiring at least one available replica.

OLM installations continue to deploy one operator replica and do not include the Helm chart's PDB or pod anti-affinity. Helm values do not configure OLM installations. Administrators can manually increase replicas and add anti-affinity through the installed ClusterServiceVersion (CSV), then create a PDB separately, following deployments/gpu-operator/templates/poddisruptionbudget.yaml.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get @francisguillier on the same page regarding difference between HA in Helm vs OpenShift.

@aryangorwade
aryangorwade force-pushed the high-availability branch 2 times, most recently from 5d96681 to c915ea2 Compare November 19, 2026 01:28

@rahulait rahulait left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM. Thanks @aryangorwade

Run a standby operator pod so a single pod or node failure doesn't
stop reconciliation.

- Render the replica count from Helm values
- Add soft pod anti-affinity to spread replicas across nodes
- Add a PodDisruptionBudget
- Add a cache warmup flag so the standby is ready to take over
- Register operator metrics only on the leader
- Update must-gather to collect logs from both operator pods
- Add Helm template unit tests

Signed-off-by: Aryan Gorwade <[email protected]>
@aryangorwade
aryangorwade merged commit 6c1745d into NVIDIA:main Nov 19, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Issues/PRs that are related to performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants