Skip to content

fix(slurm): honor slurm_oversubscribe in the batch partition - #1445

Merged
dholt merged 1 commit into
NVIDIA:masterfrom
0z5a:fix/slurm-oversubscribe-setting
Nov 14, 2026
Merged

dholt merged 1 commit into
NVIDIA:masterfrom
0z5a:fix/slurm-oversubscribe-setting

Conversation

@0z5a

@0z5a 0z5a commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

roles/slurm/defaults/main.yml documents slurm_oversubscribe ("Sets partition OverSubscribe. To avoid sharing nodes, set to EXCLUSIVE"), but the slurm.conf template reassigns it from slurm_manage_gpus on the line before the PartitionName line. A value set in config/group_vars/ was never used: OverSubscribe was always NO with slurm_manage_gpus: true and EXCLUSIVE otherwise.

This PR moves that expression into the role default and removes the set line from the template. The rendered default is the same as before for both values of slurm_manage_gpus; an explicit slurm_oversubscribe now reaches slurm.conf.

Fixes #1444

Testing

  • New tests/slurm-conf/test_partition.py renders the template in memory with Ansible's Templar (same approach as tests/slurm-lua/test_lua.py). On master @ 4eb47c33 it fails in the 6 subtests where the setting differs from the automatic value; with this change both tests pass (python3 tests/slurm-conf/test_partition.py). Run on macOS arm64, Python 3.12, ansible 10.7.0 (ansible-core 2.17.14, the version scripts/setup.sh pins).
  • git diff --check, scripts/deepops/check-template-syntax.py, scripts/deepops/ansible-lint-roles.sh (ansible-lint 26.1.1, ansible 10.7.0) and ansible-playbook --syntax-check playbooks/slurm-cluster/slurm.yml pass.
  • Not done: no playbook was run and no Slurm cluster was used.

AI disclosure: found and drafted with an AI coding assistant (Claude Code, claude-fable-5-2), including the default change, the new test and this description; the tests were run as described above.

The slurm.conf template reassigned slurm_oversubscribe from
slurm_manage_gpus right before the PartitionName line, so a value set in
the inventory was never used: OverSubscribe was always NO with
slurm_manage_gpus true and EXCLUSIVE otherwise.

Move that expression into the role default and drop the reassignment.
The rendered default is unchanged for both slurm_manage_gpus values, and
an explicit slurm_oversubscribe now reaches slurm.conf.

Add tests/slurm-conf/test_partition.py, which renders the template in
memory for the default and for explicit settings.

Signed-off-by: Dezhen Lu <[email protected]>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 18:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@dholt dholt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review approved; please hold merging until the day after the 26.11 tag. This keeps the release tree unchanged after QA; this approval is not a request to merge before then.

Thanks @0z5a for the focused fix and regression test. Reviewed commit 4bea04d43659ac545de6163e20a81a316b32f6bf.

Correct

No blocking findings. Moving the expression into role defaults preserves Ansible variable precedence. Full rendered configurations are byte-identical to the parent with slurm_manage_gpus: true (OverSubscribe=NO) and false (OverSubscribe=EXCLUSIVE). Explicit NO, EXCLUSIVE, YES and FORCE settings now reach the partition in both modes. I also checked FORCE:4 and the adjacent MaxTime/DefaultTime fields. Repository-wide search found no other OverSubscribe assignment or separate documentation requiring an update.

Secure / Readable / Elegant

No findings. The change removes the template-local override without adding execution paths or dependencies.

Altruistic

  • nit / non-blocking suggestion: tests/slurm-conf/test_partition.py:51 — consider wiring this test into CI in a follow-up. None of the current workflows invokes or discovers tests/slurm-conf; the existing Slurm workflow runs only the epilog suite, and the Molecule matrix excludes Slurm. Running python tests/slurm-conf/test_partition.py -v with the CI-pinned Ansible environment would preserve this regression coverage automatically.

Local validation: both test methods passed (10 parameter cases), role lint passed, all 12 maintenance playbook syntax checks passed after initializing the pinned Kubespray submodule, and git diff --check passed. These are offline checks, not live scheduler validation.

@dholt
dholt merged commit 3c584c6 into NVIDIA:master Nov 14, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

slurm: slurm_oversubscribe is ignored, slurm.conf always gets the value derived from slurm_manage_gpus

3 participants