Skip to content

fix(k8s): show the logs of every GPU test pod in verify_gpu.sh - #1443

Merged
michael-balint merged 1 commit into
NVIDIA:masterfrom
0z5a:fix/verify-gpu-pod-logs
Nov 12, 2026
Merged

michael-balint merged 1 commit into
NVIDIA:masterfrom
0z5a:fix/verify-gpu-pod-logs

Conversation

@0z5a

@0z5a 0z5a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

scripts/k8s/verify_gpu.sh prints the logs of the test pods so the nvidia-smi output of each GPU can be checked. The loop counts with $i but indexes the pod array with $k, which is never set. The index is always 0, so the script prints the first completed pod's log once per GPU and never shows the others.

This PR iterates over the completed pods instead. The pod-count checks and exit codes after the loop are unchanged.

Testing

  • New case test_logs_are_read_once_from_every_completed_pod in tests/verify-gpu/test_verify_gpu.py (stub kubectl, as in the existing cases). On master @ 4eb47c33 it fails for 3 GPUs with ['cluster-gpu-tests-0', 'cluster-gpu-tests-0', 'cluster-gpu-tests-0']; with this change all 3 tests in the file pass (python3 tests/verify-gpu/test_verify_gpu.py). Run on x86_64 Linux, bash 5.2.21, Python 3.12.3.
  • bash -n scripts/k8s/verify_gpu.sh and git diff --check pass.
  • Not done: the script was not run against a real Kubernetes cluster or GPU.

AI disclosure: found and drafted with an AI coding assistant (Claude Code, claude-fable-5-2), including the loop change, the new test case and this description; the test was run as described above.

🤖 Generated with Claude Code

The loop that prints the test pod logs counted with $i but indexed the
pod array with $k, which is never set. The index was always 0, so the
script printed the log of the first completed pod once per GPU and
never showed the other pods.

Iterate over the completed pods instead, and add a regression case to
tests/verify-gpu/test_verify_gpu.py.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Signed-off-by: Dezhen Lu <[email protected]>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 16:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@michael-balint
michael-balint merged commit c8858fa into NVIDIA:master Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants