Repository navigation
Implement GPU Operator high availability - #3103
Conversation
d3f3884 to
0bbdf3e
Compare
0bbdf3e to
bdcab9d
Compare
bdcab9d to
a0922d2
Compare
📝 Walkthrough
Priority: ⚪ Pending latest changes Merge Risk: 🟡 Moderate · up to
|
There was a problem hiding this comment.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments: 2
.github/workflows/ci.yaml: 1
109-109: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winHandle the Dependabot
mainpush case.
.github/workflows/variables.yamlsetspush_on_buildto"false"whengithub.actorisdependabot[bot], even for apushevent. In that case,ha-failoveris skipped, and the implicitsuccess()onreleasealso skips the release job.If Dependabot-authored pushes to
mainmust release, allow the intentionalha-failoverskip while still requiring successful E2E tests:Suggested fix
- if: github.event_name == 'push' && github.ref == 'refs/heads/main' + if: >- + ${{ + always() && + !cancelled() && + github.event_name == 'push' && + github.ref == 'refs/heads/main' && + needs.e2e-tests.result == 'success' && + ( + needs.ha-failover.result == 'success' || + ( + needs.ha-failover.result == 'skipped' && + needs.variables.outputs.push_on_build == 'false' + ) + ) + }}.github/workflows/ha-failover.yaml: 1
61-65: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winVerify the kind and KWOK artifacts before use.
The workflow logs into GHCR before downloading and executing
kind. The versioned URL is not an immutable digest, so a modified binary can read the Docker credential file. The KWOK manifests also come from the mutablev0.8.0release tag and are applied directly withkubectl.Verify the
kindbinary and both KWOK manifests with trusted SHA-256 checksums or immutable release references. Move the registry login to immediately beforedocker pull, after cluster creation and KWOK installation.Move registry login after artifact setup
- - name: Login to GitHub Container Registry - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 - with: - registry: ghcr.io - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - name: Set up Helm uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1 with: version: v3.19.0 ... kubectl wait --for=condition=Ready node/kwok-gpu --timeout=60s + - name: Login to GitHub Container Registry + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Install three operator replicas
🧹 Nitpick comments: 3
.github/workflows/ha-failover.yaml: 1
181-181: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCheck acquisition latency independently.
The workflow checks
(( SECONDS - t0 <= 60 ))only after acquisition, patching, and reconciliation. This can pass when acquisition exceeds the stated lease-duration-plus-cache-sync acceptance bound but reconciliation finishes within the remaining 60 seconds. Assert the elapsed time immediately afterwait_for 60 acquiredagainst the acceptance bound, then keep reconciliation as a separate liveness check.deployments/gpu-operator/chart_test.go: 1
125-125: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd regression coverage for the new template branches.
The current cases do not exercise user-provided
podAntiAffinity, either disabled switch, orleader-election-warmup. Add cases for these paths and assert that the default render omits the warmup argument.Suggested test cases
import ( "os/exec" + "slices" "strings" "testing" @@ check: func(t *testing.T, deployment *appsv1.Deployment, pdb *policyv1.PodDisruptionBudget) { require.Equal(t, int32(1), *deployment.Spec.Replicas) require.Nil(t, deployment.Spec.Template.Spec.Affinity.PodAntiAffinity) require.Nil(t, pdb) + require.False(t, slices.Contains(deployment.Spec.Template.Spec.Containers[0].Args, "--leader-election-warmup")) }, }, @@ }, }, + "user-provided podAntiAffinity suppresses the generated one": { + values: ` +operator: + replicaCount: 2 + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: topology.kubernetes.io/zone + labelSelector: {matchLabels: {app: gpu-operator}} +`, + check: func(t *testing.T, deployment *appsv1.Deployment, _ *policyv1.PodDisruptionBudget) { + podAntiAffinity := deployment.Spec.Template.Spec.Affinity.PodAntiAffinity + require.NotNil(t, podAntiAffinity) + require.Len(t, podAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution, 1) + require.Empty(t, podAntiAffinity.PreferredDuringSchedulingIgnoredDuringExecution) + }, + }, + "anti-affinity and PDB can be disabled": { + values: "operator: {replicaCount: 2, podAntiAffinity: {enabled: false}, podDisruptionBudget: {enabled: false}}", + check: func(t *testing.T, deployment *appsv1.Deployment, pdb *policyv1.PodDisruptionBudget) { + require.Nil(t, deployment.Spec.Template.Spec.Affinity.PodAntiAffinity) + require.Nil(t, pdb) + }, + }, + "warmup renders the operator flag": { + values: "operator: {leaderElection: {warmup: true}}", + check: func(t *testing.T, deployment *appsv1.Deployment, _ *policyv1.PodDisruptionBudget) { + require.Contains(t, deployment.Spec.Template.Spec.Containers[0].Args, "--leader-election-warmup") + }, + }, }.github/workflows/golang-checks.yaml: 1
82-83: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPin the Helm version for the chart tests.
The
go-testjob runsmake coverage, which runs tests indeployments/gpu-operator.chart_test.goinvokeshelm templateand asserts rendered resources. Withoutversion,azure/setup-helminstallslatest, so a Helm update can change CI results without repository changes. Use the existingv3.19.0pin.Proposed change
- name: Install Helm uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1 + with: + version: v3.19.0
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: NVIDIA/gpu-operator/.coderabbit.yaml
- Review profile: QUIET
- Plan: Enterprise
- Run ID:
783323a1-d0a4-45bf-aba0-10ddb9beb4ba
📒 Files selected for processing: 9
.github/workflows/ci.yaml.github/workflows/golang-checks.yaml.github/workflows/ha-failover.yamlcmd/gpu-operator/main.godeployments/gpu-operator/.helmignoredeployments/gpu-operator/chart_test.godeployments/gpu-operator/templates/operator.yamldeployments/gpu-operator/templates/poddisruptionbudget.yamldeployments/gpu-operator/values.yaml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
a0922d2 to
0ca7701
Compare
b9995df to
a6b9b70
Compare
kvalliyurnatt
left a comment
There was a problem hiding this comment.
I think we also need to make changes to must_gather.sh, currently in the script we do
OPERATOR_POD_NAME=$($K get pods -lapp=gpu-operator -oname -A)
To get the Operator Pod name and then below we do
echo "Get the GPU Operator Pod (status)"
$K get "${OPERATOR_POD_NAME}" \
-owide \
-n "${OPERATOR_NAMESPACE}" \
> "${ARTIFACT_DIR}/gpu_operator_pod.status"
echo "Get the GPU Operator Pod (yaml)"
$K get "${OPERATOR_POD_NAME}" \
-oyaml \
-n "${OPERATOR_NAMESPACE}" \
> "${ARTIFACT_DIR}/gpu_operator_pod.yaml"
echo "Get the GPU Operator Pod logs"
$K logs "${OPERATOR_POD_NAME}" \
-n "${OPERATOR_NAMESPACE}" \
--timestamps \
> "${ARTIFACT_DIR}/gpu_operator_pod.log"
$K logs "${OPERATOR_POD_NAME}" \
-n "${OPERATOR_NAMESPACE}" \
--timestamps \
--previous \
> "${ARTIFACT_DIR}/gpu_operator_pod.previous.log"
Which will fail I think when there are more than one operator Pods
171ac38 to
c384421
Compare
7c6443f to
6247fd2
Compare
6247fd2 to
5a91848
Compare
eb35a01 to
df9fc43
Compare
7193332 to
5688d0f
Compare
4115258 to
1eaac3b
Compare
1eaac3b to
77e91ad
Compare
bde34b8 to
76298f1
Compare
52d968b to
1250c4e
Compare
rajathagasthya
left a comment
There was a problem hiding this comment.
Excellent work @aryangorwade!
1250c4e to
e2af970
Compare
abrarshivani
left a comment
There was a problem hiding this comment.
LGTM. Thanks @aryangorwade.
| @@ -0,0 +1,17 @@ | |||
| {{- $pdb := .Values.operator.podDisruptionBudget | default dict }} | |||
There was a problem hiding this comment.
This is being installed only via the helm install path and OLM bundle install is missing it. I wonder if this would be a documentation for that path or we need to add changes to OLM bundle
There was a problem hiding this comment.
We should add Added a section for HA in README.md, below Quick Start.
Operator high availability
Helm installations run one operator replica by default. Increase
operator.replicaCountto run multiple replicas with leader election. Unless explicitly disabled, the chart also adds soft pod anti-affinity and a PodDisruptionBudget (PDB) requiring at least one available replica.OLM installations continue to deploy one operator replica and do not include the Helm chart's PDB or pod anti-affinity. Helm values do not configure OLM installations. Administrators can manually increase replicas and add anti-affinity through the installed ClusterServiceVersion (CSV), then create a PDB separately, following
deployments/gpu-operator/templates/poddisruptionbudget.yaml.
There was a problem hiding this comment.
Let's get @francisguillier on the same page regarding difference between HA in Helm vs OpenShift.
5d96681 to
c915ea2
Compare
rahulait
left a comment
There was a problem hiding this comment.
Overall, LGTM. Thanks @aryangorwade
c915ea2 to
ee8be9a
Compare
Run a standby operator pod so a single pod or node failure doesn't stop reconciliation. - Render the replica count from Helm values - Add soft pod anti-affinity to spread replicas across nodes - Add a PodDisruptionBudget - Add a cache warmup flag so the standby is ready to take over - Register operator metrics only on the leader - Update must-gather to collect logs from both operator pods - Add Helm template unit tests Signed-off-by: Aryan Gorwade <[email protected]>
ee8be9a to
fdfcdb3
Compare
Description
Implements HA changes proposed in #582.
operator.replicaCounthelm templaterendering test casesChecklist
make lint)make validate-generated-assets)make validate-modules)Testing
Validated via manual testing on cluster.
helm upgradewith a new imagereplicaCount: 2on a cluster with one control-plane nodeThe warmup on and off test that determines cache warmup default can't be completed because cache bounding hasn't been implemented yet. A test run was completed with warmup off:
Acceptance criteria
One acceptance criterion was not fulfilled as written.