Skip to content

Validate explicit operator PDB requests - #3269

Open
aryangorwade wants to merge 1 commit into
NVIDIA:mainfrom
aryangorwade:high-availability-pdb-validation
Open

aryangorwade wants to merge 1 commit into
NVIDIA:mainfrom
aryangorwade:high-availability-pdb-validation

Conversation

@aryangorwade

@aryangorwade aryangorwade commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

This PR builds on the GPU Operator HA change in #3103. Helm now reports an error when a user explicitly enables the operator PDB with one replica, while the default install still succeeds.


Devin Review

@aryangorwade
aryangorwade requested a review from a team as a code owner November 19, 2026 22:42

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread hack/must-gather.sh Resolved
Comment on lines +36 to +39
{{- $pdb := .Values.operator.podDisruptionBudget | default dict }}
{{- if and (dig "enabled" false $pdb) (le (int .Values.operator.replicaCount) 1) }}
{{- fail "operator.podDisruptionBudget.enabled=true requires operator.replicaCount to be at least 2" }}
{{- end }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Untested explicit PDB rejection

Removing this guard leaves TestOperatorHighAvailability and TestOperatorRejectsInvalidReplicaCount passing; neither explicitly enables the PDB with one replica. The testing conventions require a regression test that fails without the fix.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The Helm chart now supports configurable operator replicas, optional leader-election warmup, generated pod anti-affinity, and a conditional PodDisruptionBudget. Chart tests cover validation and rendered output, and the README documents Helm and OLM configuration. Operator metrics register through the manager after leadership is acquired. The must-gather script collects status, YAML, logs, and namespace leases for multiple operator pods.

Merge Risk: Low · up to 8f693

With multiple operator replicas, the must-gather script fails to collect namespace-scoped diagnostics and per-replica logs. Fix the namespace resolution before relying on it for HA installs. The default single-replica install is unaffected. Adding tests for explicit PDB validation and metrics lifecycle would guard the new behavior.

Security architecture risk: Low · up to 8f693

Multi-replica operation retains leader election and existing pod permissions. However, diagnostic collection combines repeated namespace names into one namespace argument, preventing collection of pod logs and lease information during normal HA operation. This weakens failover investigation; default single-replica installations are unaffected.

Retained concerns

  • Low · reliability · inferred: Multi-replica operation breaks namespace-scoped recovery diagnostics. OPERATOR_NAMESPACE collects one namespace result per matching pod without deduplication. With two replicas in the same namespace it becomes a value such as “gpu-operator gpu-operator”, which is passed as one quoted namespace argument to pod, lease, and log collection. Iterating over pod names does not repair that namespace identity. The assumption predates this PR, but configurable replicas newly make it a steady-state failure in supported HA installations, impairing failover investigation and recovery.
Security review details

Security Blast Radius

  • inferred — Additional replicas increase the number of operator processes using the existing gpu-operator service account rather than introducing a separate identity. Existing container restrictions remain in place: privilege escalation is disabled, the root filesystem is read-only, and all capabilities are dropped. Effective account permissions were not independently re-audited.

Trust Boundaries and Controls

  • observed — The changed diagnostic loop passes API-returned pod resource names as quoted log-command arguments and uses their names in quoted output paths. It does not evaluate those names as shell code. Its identified failure is namespace cardinality, not an observed command-injection path.

Resilience and Maintainability Implications

  • inferred — The PDB requires one available replica, but soft anti-affinity does not guarantee independent node failure domains. The values documentation explicitly acknowledges that co-located replicas can block draining their only eligible node and describes disabling the budget or returning to one replica.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Major · Resolve one namespace before collecting multiple replicas. · must-gather.sh:69

hack/must-gather.sh:69
Major · Functional Correctness · Quick win

Resolve one namespace before collecting multiple replicas.

With two operator pods in gpu-operator, .items[].metadata.namespace returns repeated namespace values, not one namespace. Kubernetes JSONPath preserves multiple selected results. (kubernetes.io)

Every -n "${OPERATOR_NAMESPACE}" then receives an invalid namespace. The new per-pod loop cannot collect either replica's logs, and namespace-scoped status, YAML, leases, and operand collection also fail. The script does not normalize this value before use. (raw.githubusercontent.com)

Resolve one namespace, validate that all selected pods belong to it, and scope OPERATOR_PODS discovery to that namespace.

Nitpick comments: 2

deployments/gpu-operator/chart_test.go: 1

38-38: Trivial · Maintainability & Code Quality · Quick win

Add a regression test for explicit PDB enablement.

Both inputs fail the replica-count check before the PDB check runs. Removing the new PDB validation would leave all supplied tests passing.

Add a case with replicaCount: 1 and podDisruptionBudget.enabled: true. Require the specific PDB validation error. Also check that explicit enablement succeeds with two replicas.

As per coding guidelines, “New behavior needs appropriate test coverage.”

Source: Coding guidelines

controllers/operator_metrics_test.go: 1

44-44: Trivial · Maintainability & Code Quality · Quick win

Test the manager-facing lifecycle methods.

If Start stops registering metrics, this test still passes because it calls register directly. It also passes if NeedLeaderElection returns false.

Exercise Start with an isolated registry and assert that NeedLeaderElection() returns true. Keep the gathered-metric assertions.

As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d666a21 and 8f693cd.

Files selected for processing: 14

  • .github/workflows/golang-checks.yaml
  • README.md
  • cmd/gpu-operator/main.go
  • controllers/operator_metrics.go
  • controllers/operator_metrics_test.go
  • controllers/upgrade_controller_test.go
  • deployments/gpu-operator/.helmignore
  • deployments/gpu-operator/chart_test.go
  • deployments/gpu-operator/templates/_helpers.tpl
  • deployments/gpu-operator/templates/operator.yaml
  • deployments/gpu-operator/templates/poddisruptionbudget.yaml
  • deployments/gpu-operator/templates/validations.yaml
  • deployments/gpu-operator/values.yaml
  • hack/must-gather.sh

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment thread deployments/gpu-operator/values.yaml Resolved
@aryangorwade
aryangorwade force-pushed the high-availability-pdb-validation branch from 8f693cd to f9c12b4 Compare November 20, 2026 00:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants