You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR builds on the GPU Operator HA change in #3103. Helm now reports an error when a user explicitly enables the operator PDB with one replica, while the default install still succeeds.
The reason will be displayed to describe this comment to others. Learn more.
🔍 Untested explicit PDB rejection
Removing this guard leaves TestOperatorHighAvailability and TestOperatorRejectsInvalidReplicaCount passing; neither explicitly enables the PDB with one replica. The testing conventions require a regression test that fails without the fix.
Was this helpful? React with 👍 or 👎 to provide feedback.
The Helm chart now supports configurable operator replicas, optional leader-election warmup, generated pod anti-affinity, and a conditional PodDisruptionBudget. Chart tests cover validation and rendered output, and the README documents Helm and OLM configuration. Operator metrics register through the manager after leadership is acquired. The must-gather script collects status, YAML, logs, and namespace leases for multiple operator pods.
Merge Risk: Low · up to 8f693
With multiple operator replicas, the must-gather script fails to collect namespace-scoped diagnostics and per-replica logs. Fix the namespace resolution before relying on it for HA installs. The default single-replica install is unaffected. Adding tests for explicit PDB validation and metrics lifecycle would guard the new behavior.
Security architecture risk: Low · up to 8f693
Multi-replica operation retains leader election and existing pod permissions. However, diagnostic collection combines repeated namespace names into one namespace argument, preventing collection of pod logs and lease information during normal HA operation. This weakens failover investigation; default single-replica installations are unaffected.
Retained concerns
Low · reliability · inferred: Multi-replica operation breaks namespace-scoped recovery diagnostics. OPERATOR_NAMESPACE collects one namespace result per matching pod without deduplication. With two replicas in the same namespace it becomes a value such as “gpu-operator gpu-operator”, which is passed as one quoted namespace argument to pod, lease, and log collection. Iterating over pod names does not repair that namespace identity. The assumption predates this PR, but configurable replicas newly make it a steady-state failure in supported HA installations, impairing failover investigation and recovery.
Security review details
Security Blast Radius
inferred — Additional replicas increase the number of operator processes using the existing gpu-operator service account rather than introducing a separate identity. Existing container restrictions remain in place: privilege escalation is disabled, the root filesystem is read-only, and all capabilities are dropped. Effective account permissions were not independently re-audited.
Trust Boundaries and Controls
observed — The changed diagnostic loop passes API-returned pod resource names as quoted log-command arguments and uses their names in quoted output paths. It does not evaluate those names as shell code. Its identified failure is namespace cardinality, not an observed command-injection path.
Resilience and Maintainability Implications
inferred — The PDB requires one available replica, but soft anti-affinity does not guarantee independent node failure domains. The values documentation explicitly acknowledges that co-located replicas can block draining their only eligible node and describes disabling the budget or returning to one replica.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
⚠️ Outside diff range comments (1)
Major · Resolve one namespace before collecting multiple replicas. · must-gather.sh:69
hack/must-gather.sh:69 Major · Functional Correctness · Quick win
Resolve one namespace before collecting multiple replicas.
With two operator pods in gpu-operator, .items[].metadata.namespace returns repeated namespace values, not one namespace. Kubernetes JSONPath preserves multiple selected results. (kubernetes.io)
Every -n "${OPERATOR_NAMESPACE}" then receives an invalid namespace. The new per-pod loop cannot collect either replica's logs, and namespace-scoped status, YAML, leases, and operand collection also fail. The script does not normalize this value before use. (raw.githubusercontent.com)
Resolve one namespace, validate that all selected pods belong to it, and scope OPERATOR_PODS discovery to that namespace.
Add a regression test for explicit PDB enablement.
Both inputs fail the replica-count check before the PDB check runs. Removing the new PDB validation would leave all supplied tests passing.
Add a case with replicaCount: 1 and podDisruptionBudget.enabled: true. Require the specific PDB validation error. Also check that explicit enablement succeeds with two replicas.
As per coding guidelines, “New behavior needs appropriate test coverage.”
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR builds on the GPU Operator HA change in #3103. Helm now reports an error when a user explicitly enables the operator PDB with one replica, while the default install still succeeds.