feat(slurm): integrate Slurm execution for 1.0.0 release by nabinchha · Pull Request #1040 · NVIDIA-NeMo/DataDesigner

nabinchha · 2026-10-15T20:56:59Z

📋 Summary

Integrate the optional Slurm execution package into main for the planned 1.0.0 release. This consolidates the feature-branch implementation and documentation in one reviewable PR. Keep this PR open; do not merge until release review is complete.

🔗 Related Issue

N/A — integration of the existing Slurm feature branch and its previously reviewed changes.

🔄 Changes

✨ Added

Add the data-designer-slurm package and the version-matched data-designer[slurm] extra.
Add declarative Slurm profiles, serving image management, run planning and execution, status/cancel/retry/collection, and benchmark workflows.
Add Slurm examples, architecture notes, and user documentation.

🔧 Changed

Extend the CLI and generation invocation boundaries for the optional Slurm integration.
Update the publish workflow to build and upload the config, engine, Slurm, and interface packages at the same version.

🐛 Fixed

Incorporate feature-branch fixes for startup diagnostics, multi-model placement, shard recovery, partial outputs, and Parquet collection schema handling.

🔍 Attention Areas

Package metadata and publish order — confirm the exact-version dependency relationship and 1.0.0rc1 wheel/install flow.
Slurm run contracts and runtime — review public configuration and allocation behavior.

🧪 Testing

make test-slurm — 1,915 passed after rebasing onto main.
make check-slurm, uv lock --check, and uv pip check passed.
Unit and integration-style tests added for the Slurm package.
make test and final 1.0.0rc1 wheel/install validation are pending release review.

✅ Checklist

Follows commit message conventions.
All commits are signed off (DCO): nine existing feature-branch commits lack a Signed-off-by trailer; resolve with their authors before merge if required.
Architecture docs updated.
Release review complete; do not merge yet.

github-actions · 2026-10-15T20:59:38Z

Fern preview: https://nvidia-preview-pr-1040.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

* feat: add optional Slurm package Add the shared-namespace leaf package and publish it through the same-version data-designer[slurm] extra. Cover resolver, namespace, base-only isolation, and built-wheel installation behavior.\n\nCloses #852 Signed-off-by: Andre Manoel <[email protected]> * fix: address Slurm package review findings Signed-off-by: Andre Manoel <[email protected]> * fix: publish Slurm package Signed-off-by: Andre Manoel <[email protected]> * fix: scope workspace dependency extras Signed-off-by: Andre Manoel <[email protected]> * fix: publish Slurm before base package Upload the Slurm distribution before data-designer so the exact-version extra is resolvable as soon as the base package is published. Signed-off-by: Andre Manoel <[email protected]> * docs: generalize Slurm package wording Signed-off-by: Andre Manoel <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]>

* feat: define public generation contract * fix: complete public generation contract * docs: clarify public invocation limits * fix: handle mixed-case config extensions Signed-off-by: Andre Manoel <[email protected]> * fix: complete generation contract edge cases Signed-off-by: Andre Manoel <[email protected]> * test: make inline config regression self-contained Signed-off-by: Andre Manoel <[email protected]> * fix: classify malformed config paths Signed-off-by: Andre Manoel <[email protected]> * fix: normalize generation artifact errors Classify malformed inline configs independently of filename suffixes. Wrap unreadable result artifacts in the public storage error across counting and export paths. Signed-off-by: Andre Manoel <[email protected]> * fix: complete artifact error boundaries Signed-off-by: Andre Manoel <[email protected]> * fix: tighten generation result contract Signed-off-by: Andre Manoel <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]>

* feat: add lazy CLI extensions Signed-off-by: Andre Manoel <[email protected]> * fix: isolate CLI extension failures Signed-off-by: Andre Manoel <[email protected]> * refactor: order public CLI members first Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]> Signed-off-by: Nabin Mulepati <[email protected]> Co-authored-by: Nabin Mulepati <[email protected]>

* feat: add Slurm state record contracts Define strict, versioned execution, readiness, output, reconciliation, and scheduler records. Enforce cross-record identity, digest, shard-set, and terminal-state invariants with deterministic golden fixtures and built-wheel coverage. Closes #865 * fix: harden Slurm state contracts Reject partial winners and conflicting shard ownership. Require explicit versions, enforce evidence chronology and readiness precedence, and cover every state-contract path with validated fixtures. Refs #865 * fix(slurm): harden state contract invariants Hash persisted record bytes, align state identities with the resolved plan vocabulary, and preserve public model aliases. Add monotonic attempt and scheduler transition checks, including fixed accounting-lag deadlines and UNKNOWN precedence. Refs #865 * refactor(slurm): share contract primitives Make planning and runtime state consume one public immutable model family. Align shard, attempt, range, artifact, and resume workspace shapes so the configuration-plan branch can rebase without preserving duplicate Pydantic types.\n\nRefs #865

* feat: define Slurm config and plan contracts Add strict authored configuration, profile, image, client, benchmark, dependency-lock, and resolved-plan records. Validate cross-record identities and digests with sanitized single-node and multi-node golden fixtures. Closes #873 Signed-off-by: Andre Manoel <[email protected]> * fix: tighten Slurm contract validation Signed-off-by: Andre Manoel <[email protected]> * fix: align Slurm F3 contract seam * fix: bind client dataset to effective resume mode * fix: harden Slurm shared contracts Prevent secret material from entering persisted configuration and make nested contract collections immutable. Tighten client and benchmark semantic invariants with focused negative coverage. Signed-off-by: Nabin Mulepati <[email protected]> * fix: address Slurm contract review findings Signed-off-by: Andre Manoel <[email protected]> * fix: close Slurm argument validation bypasses Signed-off-by: Andre Manoel <[email protected]> * fix: close remaining Slurm contract gaps Signed-off-by: Andre Manoel <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]> Signed-off-by: Nabin Mulepati <[email protected]> Co-authored-by: Nabin Mulepati <[email protected]>

* feat: integrate Slurm plan and state Validate plan-aware shards, readiness, attempts, and finalization through a reusable immutable context. Add deterministic goldens and negative coverage for mismatch and failure paths. Closes #880 Signed-off-by: Nabin Mulepati <[email protected]> * fix: reject unsubmitted planned attempts Require scheduler-bound plan validation to start only after an attempt leaves the created state. Cover created attempts that already carry scheduler metadata and verify the integration validator imports from an isolated Slurm wheel. Signed-off-by: Nabin Mulepati <[email protected]> * test: harden Slurm integration join coverage Signed-off-by: Nabin Mulepati <[email protected]> * refactor: simplify plan state validator Signed-off-by: Nabin Mulepati <[email protected]> * refactor: remove validator wrapper functions Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* test: add deterministic Slurm fakes Add reusable scheduler, dependency, plugin, serving, clock, and state-transition fixtures for local and CI Slurm integration tests. Keep the harness deterministic and isolated from production lane implementations. Signed-off-by: Andre Manoel <[email protected]> * test: bind Slurm fakes to contracts Promote the merged planning and finalization goldens to package-wide fixtures, exercise dependency doubles with canonical records, and add site-neutral rendered script goldens for downstream lanes. Signed-off-by: Andre Manoel <[email protected]> * test: complete post-F3 Slurm fake coverage Bind rendered scripts and shared fixtures to the merged contracts, tighten scheduler command behavior, and cover retry terminal states and invalid serving restarts. Signed-off-by: Andre Manoel <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]>

* feat: add Slurm image registry foundation Add digest-bound client and serving inspection, atomic workspace-derived registry persistence, and verified alias/path resolution.\n\nPart of #867 Signed-off-by: Nabin Mulepati <[email protected]> * test: cover image inspection boundaries Exercise the production environment reader and verify restrictive atomic registry persistence discovered during review.\n\nPart of #867 Signed-off-by: Nabin Mulepati <[email protected]> * fix: align image registry persistence Persist the package-owned image registry as deterministic safe YAML at the migration plan's canonical path and declare the runtime YAML dependency explicitly.\n\nPart of #867. Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden image registry file boundaries Reject symlinked registry and lock files, use explicit UTF-8 persistence, and normalize malformed byte content into the package image error hierarchy.\n\nPart of #867. Signed-off-by: Nabin Mulepati <[email protected]> * test: verify Slurm image wheel integration Exercise the public image service from an isolated installed wheel and assert that the wheel metadata carries the runtime PyYAML dependency.\n\nPart of #867. Signed-off-by: Nabin Mulepati <[email protected]> * fix: verify registry file types before use Treat only a genuinely missing registry as empty and reject non-regular registry or lock nodes before reading, locking, or changing permissions.\n\nPart of #867. Signed-off-by: Nabin Mulepati <[email protected]> * fix: rollback stale image registrations Reverify image bytes while registry mutation locks remain held and restore the previous snapshot if final verification fails. Part of #867 Signed-off-by: Nabin Mulepati <[email protected]> * fix image registry review findings Require Data Designer in client image inspections, verify artifacts before publishing registry updates, harden storage directory boundaries, and normalize recursive YAML failures. * harden image registry contracts - Keep the image-registry foundation internal until the public facade lands\n- Require a complete client package inventory and unambiguous registry schema\n- Close SQSH identity races and exercise publication behavior through owned APIs * strengthen image registry test coverage Exercise persisted semantic invariants and exact-byte hashes. Split record, store, and service coverage by ownership. Signed-off-by: Nabin Mulepati <[email protected]> * address image registry review feedback Bind registry transactions to verified directory descriptors, centralize immutable image fact comparisons, and verify internal image modules from the installed wheel. Signed-off-by: Nabin Mulepati <[email protected]> * clarify image test helper names Use explicit get prefixes for value-producing test helpers while preserving action verbs for creation, inspection, and filesystem operations. Signed-off-by: Nabin Mulepati <[email protected]> * split registry filesystem helpers Move descriptor-bound filesystem operations into a focused internal module while keeping registry transactions and YAML semantics together. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat: add Slurm command client and renderer Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: tighten Slurm launcher boundaries Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: use native sacct output fields Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * test: require expanded Slurm array queries Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: sanitize Slurm command diagnostics Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: reject invalid visible GPU memory Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: hash rendered inputs without manifests Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: make Slurm output decoding stable Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: validate finite command timeouts Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden Slurm boundary inputs Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: reject malformed Slurm GRES lists Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: isolate submitted Slurm environments Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * test: verify launcher wheel import Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: pin batch host tool path Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * test: cover Slurm launcher boundaries Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: correlate Slurm query responses Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden Slurm command boundaries Preserve the caller's PATH as the only ambient lookup input for default Slurm commands while continuing to isolate all other environment variables. Normalize oversized numeric scheduler fields into the launcher parse-error boundary. Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: fall back from empty command path Use the platform default search path when the ambient PATH is absent or empty so bare Slurm executables remain resolvable. Part of #868 Signed-off-by: Nabin Mulepati <[email protected]> * fix: align Slurm launcher semantics * fix Slurm job observation contracts Support both ordinary jobs and array-task observations while preserving accounting-lag semantics. Bound scheduler numeric fields and diagnostics, and honor unthrottled plan arrays when concurrency is omitted. * refine Slurm launcher boundaries - Internalize implementation APIs and clarify transient types - Harden parsing, rendering, and process error normalization - Extend fakes and executable checksum coverage Part of #868 * fix Slurm selector filtering ownership * simplify Slurm launcher structure --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat: resolve Slurm serving deployments Add pure, package-owned vLLM compatibility and deterministic process, readiness, backend, and logical endpoint resolution. Validate topology, launch inputs, and network identities with single-node and multi-node golden coverage. Closes #866 Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden Slurm serving contracts Revalidate persisted launch policy inputs, require canonical process ordering, and normalize resolver errors. Reject vLLM CLI aliases and transports that could override compiler-owned topology or endpoints. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * fix: close Slurm serving validation gaps Preserve independent single-node expert-parallel replicas while rejecting cross-node EP. Block attached compiler-owned vLLM topology aliases, require canonical runtime versions, and cover the boundaries with regression tests. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden vLLM launch ownership Reject unmodeled topology, plugin, LoRA, speculative-model, and environment overrides at authored and persisted boundaries. Validate JSON arguments, secret references, readiness paths, and compatibility version lengths. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * fix: bind serving context to plan Derive serving placement and client endpoint records from the same resolved plan so callers cannot mix contracts across plans. Reject vLLM API authentication until the logical endpoint models credential forwarding. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * fix: align serving with migration contracts - enforce plan-owned environment and endpoint behavior - limit compatibility to tested vLLM release series - split shared and vLLM configuration from run records - cover multi-group ordering and launch delays Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * docs: clarify option flag extraction Signed-off-by: Nabin Mulepati <[email protected]> * refactor: share argument and duration helpers Use action-named shared helpers for option extraction and duration conversion while keeping vLLM flag canonicalization backend-specific. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * refactor: simplify vLLM compatibility contract Track the tested runtime range and contract revision without repeating baseline vLLM or package-owned behaviors as capability booleans. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * refactor: make vLLM resolution version agnostic Preserve inspected runtime versions as digest-bound provenance without hard-coded support gates or redundant compatibility records. Continue to validate persisted text and compiler-owned vLLM arguments. Refs #866 Signed-off-by: Nabin Mulepati <[email protected]> * refactor: treat runtime version as provenance Signed-off-by: Nabin Mulepati <[email protected]> * refactor: name vLLM serving module explicitly Signed-off-by: Nabin Mulepati <[email protected]> * refactor: remove serving context wrapper Signed-off-by: Nabin Mulepati <[email protected]> * refactor: centralize Slurm scalar types Signed-off-by: Nabin Mulepati <[email protected]> * refactor: use str for model aliases Signed-off-by: Nabin Mulepati <[email protected]> * refactor: clarify serving replica indices * fix: reject abbreviated vLLM flags * refactor: simplify Slurm serving contracts - make vLLM resolution consume placement inputs before plan assembly - keep serving internals backend-specific and remove speculative exports - clarify validation ownership, secret redaction, and digest semantics * fix: bind Slurm serving resolution inputs - Join placement and endpoint claims through one planner-owned context.\n- Normalize inconsistent resolved records to the serving error boundary.\n- Split resolver, record, and launch-policy tests by ownership. Signed-off-by: Nabin Mulepati <[email protected]> * remove empty serving package initializer * fix serving contract validation boundaries Derive deployment and endpoint inputs from one resolved plan and centralize topology arithmetic. Reject plaintext secrets in persisted builder and vLLM argument payloads, normalize resolver errors, and require canonical argv tokens. * fix option argument validation gaps --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat: compile deterministic Slurm run plans Signed-off-by: Andre Manoel <[email protected]> * fix: enforce Slurm planning invariants Reject unsupported multi-shard semantics and managed output collisions at the effective configuration boundary. Normalize config errors and cover the corrected Big Iron field disposition. Signed-off-by: Andre Manoel <[email protected]> * fix: harden Slurm plan resolution Preserve partial early-shutdown intent and hide secret values in validation errors. Reject unsafe sharding semantics, require exact deployment coverage, and bind sourced payload identity. Signed-off-by: Andre Manoel <[email protected]> * fix: bind Slurm effective plan inputs Preserve all authored early-shutdown controls and reject direct effective invocation drift. Bind shard partition digests to deterministic persisted JSON bytes. Signed-off-by: Andre Manoel <[email protected]> * fix: align Slurm planning boundaries * fix: unify Slurm builder artifact identity * fix: validate Slurm compiler inputs * fix: complete Slurm plan resolution defaults * fix: address plan compiler review feedback * fix: preserve safe validation paths Signed-off-by: Andre Manoel <[email protected]> * fix: preserve union validation paths Signed-off-by: Andre Manoel <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]>

* feat: add Slurm service foundation Signed-off-by: Andre Manoel <[email protected]> * fix: harden Slurm service contracts Signed-off-by: Andre Manoel <[email protected]> * fix: bind Slurm service results Signed-off-by: Andre Manoel <[email protected]> * fix: clarify Slurm service boundaries Separate run planning from attempt rendering so retries can reuse an immutable plan. Keep injected implementation seams private and align public service failures with the Data Designer error hierarchy. Signed-off-by: Andre Manoel <[email protected]> * fix: tighten Slurm service contracts * test: fix Slurm seam export guard * fix: define Slurm service boundary contracts * fix: expose Slurm service dependency contracts --------- Signed-off-by: Andre Manoel <[email protected]>

* fix: harden Slurm image publication Signed-off-by: Nabin Mulepati <[email protected]> * fix: close Slurm image publication races - Hash SQSH bytes outside the global registry lock.\n- Preserve recovery state when commit synchronization fails.\n- Bind lifecycle artifacts to verified directory descriptors. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat: add Slurm state writer Persist immutable run intent and shard manifests with restrictive descriptor-bound storage, locks, digest validation, and atomic publication. Validate monotonic attempt and readiness updates across concurrent writers, and cover crash recovery, unsafe filesystem state, tampering, and fresh-process reloads. Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden Slurm state persistence Bind loaded records back to persisted plan and location invariants, harden lock and file handling against replacement and hardlink races, and avoid duplicate immutable context reads. Add adversarial, concurrency, interruption-recovery, import, and performance regression coverage. Signed-off-by: Nabin Mulepati <[email protected]> * fix: recover interrupted state publication Repair a committed immutable record when an interruption leaves its package-owned temporary hard link behind. Add regression coverage for the exact post-publication interruption window. Signed-off-by: Nabin Mulepati <[email protected]> * fix: converge concurrent state recovery Treat a temporary link already removed by a recovery reader as a successful immutable publication. Cover the publisher-reader cleanup race directly. Signed-off-by: Nabin Mulepati <[email protected]> * test: cover Slurm state reader errors Exercise invalid and missing shard, attempt, ordinal, and readiness inputs through the public state-writer API. Signed-off-by: Nabin Mulepati <[email protected]> * fix: retry concurrent state replacement Let descriptor-bound readers retry valid atomic replacements, including replacements that race interrupted-publication cleanup. Continue rejecting unsafe record types and bound the retry loop. Signed-off-by: Nabin Mulepati <[email protected]> * refactor: compose Slurm state persistence Separate validated reads and descriptor-bound storage from the public mutation workflows. Close concurrent attempt publication, retry durability, exact-byte reading, and interrupted recovery races with focused regression coverage. Signed-off-by: Nabin Mulepati <[email protected]> * refactor: shorten state mutation workflows Keep error translation at public boundaries and move lock-held happy paths into small helpers. No state method now exceeds 30 lines. Signed-off-by: Nabin Mulepati <[email protected]> * refactor: expose state collaborators within package Signed-off-by: Nabin Mulepati <[email protected]> * refactor: harden shard-local state boundaries Keep attempt and readiness mutations scoped to their target shard, move persisted plan validation into the state layer, and share descriptor-safe filesystem primitives across state and image storage. Make the schema-v1 no-migration policy explicit and freeze compatibility fixtures. Signed-off-by: Nabin Mulepati <[email protected]> * fix: align state boundary invariants Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): add allocation-local client worker * fix(slurm): address client worker review feedback * fix(slurm): harden client worker boundaries * fix(slurm): harden client filesystem boundary

* feat: finalize Slurm shard winners Lease shard dataset workspaces under reviewed locks. Validate client and candidate records plus output file digests. Atomically publish one immutable winner and seal retries. Part of #869 Signed-off-by: Nabin Mulepati <[email protected]> * fix: unwind interrupted dataset locks Ensure an opened shard context closes when resume-lock acquisition is interrupted by any BaseException. Signed-off-by: Nabin Mulepati <[email protected]> * fix: rebind verified candidate files Reject candidate path replacement after digesting an opened output descriptor and before winner publication. Signed-off-by: Nabin Mulepati <[email protected]> * test: cover nested candidate paths Exercise valid nested output verification and rejection of a symlinked intermediate directory. Signed-off-by: Nabin Mulepati <[email protected]> * fix: validate winner retry timestamps Reject non-UTC publication timestamps before the existing-winner idempotency path can ignore them. Signed-off-by: Nabin Mulepati <[email protected]> * fix: classify invalid winners as corruption Normalize semantically invalid persisted winner chains as corruption across load, finalize, attempt, and workspace APIs. Signed-off-by: Nabin Mulepati <[email protected]> * refactor: compose shard winner finalization Signed-off-by: Nabin Mulepati <[email protected]> * refactor: compose finalization validation Signed-off-by: Nabin Mulepati <[email protected]> * fix: harden candidate publication boundaries Signed-off-by: Nabin Mulepati <[email protected]> * fix: bind finalization publication Signed-off-by: Nabin Mulepati <[email protected]> * fix: publish allocation results before success Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): wire M2 services and CLI Add package-owned execution, status, cancellation, and image service wiring. Resolve client dependencies and expose the M2 operations through the Slurm CLI. Part of #874 Signed-off-by: Andre Manoel <[email protected]> * fix(slurm): reject swapped SQSH file types Signed-off-by: Andre Manoel <[email protected]> * fix(slurm): clarify unavailable operations Signed-off-by: Andre Manoel <[email protected]> * fix(slurm): resolve review findings --------- Signed-off-by: Andre Manoel <[email protected]>

* fix(slurm): close one-node M2 integration * fix(slurm): address integration review findings * fix(slurm): address smoke and review findings * docs(slurm): align status reconciliation help * fix(slurm): honor nested runtime mounts * fix(slurm): allow exclusive GPU allocations * fix(slurm): accept exclusive job GPU visibility * fix(slurm): load plans through nested mounts * fix(slurm): validate logical client paths * fix(slurm): resolve client artifacts by mount * fix(slurm): map serving model paths * fix(slurm): avoid idle GPU reaper * fix(slurm): isolate shared-node service ports * fix(slurm): validate allocation-local endpoints

* feat: reconcile persisted Slurm state Persist normalized scheduler observations and compose fresh-process run, shard, attempt, readiness, generation, and winner status. Preserve bounded accounting lag and immutable terminal evidence for status and benchmark refresh consumers. Part of #869 Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): harden state reconciliation Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): bound preemption reconciliation Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): bound public preemption status Signed-off-by: Nabin Mulepati <[email protected]> * feat: add Slurm retry and deterministic collection (#915) * feat(slurm): add retry and deterministic collection Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): recover ambiguous submissions Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): preserve collection snapshots Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): bind retries and collection ownership Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): preserve scheduler path on retry Render retry allocations with the same profile-configured Slurm command path as initial generation attempts so cluster-local srun and scontrol binaries remain discoverable. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): isolate attempt observation clocks Apply persisted timestamp floors per scheduler identity so a clock-ahead sibling cannot expire another attempt's preemption-requeue window. Persist each active attempt with its own reconciled observation time. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): isolate plugin-aware client generation Signed-off-by: Nabin Mulepati <[email protected]> * refactor(slurm): expose client worker boundary Signed-off-by: Nabin Mulepati <[email protected]> * refactor(slurm): compose client process boundary Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): refresh client overlay inventory Signed-off-by: Nabin Mulepati <[email protected]> * test(slurm): exercise plugin worker handoff Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* harden Slurm release artifacts Add sanitized credential and infrastructure scans for source, fixtures, runtime archives, logs, and wheels. Include Apache license text in the Slurm wheel and redact sensitive scheduler diagnostics. Signed-off-by: Nabin Mulepati <[email protected]> * fix Slurm security audit gaps Signed-off-by: Nabin Mulepati <[email protected]> * harden adjacent secret redaction Signed-off-by: Nabin Mulepati <[email protected]> * make Slurm redaction fail closed Signed-off-by: Nabin Mulepati <[email protected]> * redact underscore secret options Signed-off-by: Nabin Mulepati <[email protected]> * harden Slurm security audit boundaries Signed-off-by: Nabin Mulepati <[email protected]> * fix control-obscured Slurm secrets Signed-off-by: Nabin Mulepati <[email protected]> * fix control-split secret redaction Signed-off-by: Nabin Mulepati <[email protected]> * fix whitespace-control secret suffixes Signed-off-by: Nabin Mulepati <[email protected]> * harden archive content classification Detect explicitly supplied ZIP and tar inputs from their content when filenames use opaque suffixes. Reject unsupported tar special members while continuing to allow ordinary directories. Signed-off-by: Nabin Mulepati <[email protected]> * close Slurm security boundary gaps Handle serialized authorization headers and Unicode whitespace without exposing credential suffixes. Keep public-artifact scanning fail-closed for non-regular files and unreadable ZIP members, and narrow test sentinels to complete values. Signed-off-by: Nabin Mulepati <[email protected]> * redact complete Authorization diagnostics Treat an Authorization field as credential-bearing through its line boundary even when its value is quoted. This keeps control-obscured suffixes from surviving the protected redaction pass. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): compose distributed runtime launch Layer multi-node topology and follower-failure handling onto the production allocation bootstrap introduced by #929. Preserve the one-node path while adding host-scoped steps, coordinated node workers, remote readiness probes, and distributed preflight coverage. Signed-off-by: Nabin Mulepati <[email protected]> * map distributed model paths Apply the resolved container-mount mapping to absolute model paths in both coordinated multi-node workers and remote-node serving commands. Cover differing host and container roots in the bootstrap manifest regression. Signed-off-by: Nabin Mulepati <[email protected]> * adapt retry bootstrap coverage Pass the one-node allocation layout required by the distributed runtime manifest after composing the merged retry coverage onto the new shared base. Signed-off-by: Nabin Mulepati <[email protected]> * fix distributed launch preflight Select the multi-process vLLM executor whenever pipeline or tensor parallelism spans more than one worker. Carry mapped absolute model paths through the node-worker contract so every assigned node verifies accessibility before launch. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): add profile setup and image lifecycle wiring * fix(slurm): address profile and lifecycle review * fix(slurm): require terminal cancellation evidence

* feat(slurm): wire retry and merge commands * fix(slurm): harden retry and collection actions * fix(slurm): preserve retry confirmation intent * fix(slurm): recover active retry previews * fix(slurm): address retry and merge review feedback

Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): improve CLI output and image logs Make operator-facing Slurm results readable while preserving opt-in JSON for automation. Surface bounded, redacted sbatch rejection details and print status/log guidance after successful generation submission. Follow image lifecycle logs by default in interactive terminals, with explicit flags to enable or disable streaming. Update the Slurm docs and cover both output modes and submission diagnostics in tests. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): harden CLI guidance and output Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): discover scheduler tools for jobs Capture the submit-host Slurm executable directory in the immutable run plan so controlled batch PATHs can invoke srun and scontrol without a profile override. Fail before submission with an actionable diagnostic when those tools are unavailable, and document the optional override. * fix(slurm): preserve scheduler command symlinks Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): isolate single-node serving replicas Run multiple vLLM replicas inside one GPU-owning Slurm step so overlapping steps cannot be assigned the same device. Preserve the existing local vLLM command for each process and cover TP=1 and TP=4 assignments. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): bind shared serving worker to node address Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): reserve CPUs across GPU sockets Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): keep local replicas on loopback Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): support multiple GPU partitions Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): validate every selected GPU partition Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): create profile parent directories Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): verify new profile directories Keep profile creation bound to verified directory descriptors and restore private permissions when a restrictive umask is active. Update the profile guide to reflect automatic parent creation. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): harden profile directory setup Normalize unsupported permission operations and sync newly created directory metadata before publishing the profile catalog. Cover replacement and error-boundary behavior. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): support Linux directory chmod Use an opened directory for permission repair when Linux rejects nofollow fchmodat. Handle a restrictive umask through a Linux path descriptor while keeping directory traversal bound to verified descriptors. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): show vLLM startup log path Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): honor JSON output with newer Typer Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): refresh serving log guidance across executions Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): align native retry and user-site policy Signed-off-by: Andre Manoel <[email protected]> * fix(slurm): preserve delayed replica startup budget Signed-off-by: Andre Manoel <[email protected]> * fix(slurm): cancel proxy requests and catch timeouts Signed-off-by: Andre Manoel <[email protected]> * fix(slurm): isolate user-site path inspection Signed-off-by: Andre Manoel <[email protected]> * test(slurm): guard proxy and readiness compatibility Signed-off-by: Andre Manoel <[email protected]> --------- Signed-off-by: Andre Manoel <[email protected]>

#1012) * fix(slurm): clarify config defaults and inert fields Signed-off-by: Nabin Mulepati <[email protected]> * docs(slurm): keep run config guidance concise Signed-off-by: Nabin Mulepati <[email protected]> * refactor(slurm): remove unused run config flags Signed-off-by: Nabin Mulepati <[email protected]> * docs(slurm): clarify benchmark model concurrency sweep Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): confirm image removal and allow artifact deletion Show an explicit removal result in human output. Allow opt-in deletion only for unshared package-owned OCI SQSH artifacts, with safe failure reporting and documentation. Signed-off-by: Nabin Mulepati <[email protected]> * feat(slurm): timestamp image job progress Signed-off-by: Nabin Mulepati <[email protected]> * feat(slurm): clarify image add progress Report deterministic image job phases and reduce idle status noise. Omit blank followed-log lines and show elapsed time after registration. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): simplify image details output Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): clarify image add and list results Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): clarify image command help Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): sanitize image list runtime Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): explain missing builder sources Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): show serving startup progress in allocation logs Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): point allocation logs to client generation progress Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): choose Parquet partitions at merge Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): preserve collection intent on retry Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

Avoid writing a duplicate Parquet export when the engine batches can be used directly as a bounded candidate. Keep the export path for schema drift or too many batch files. Signed-off-by: Nabin Mulepati <[email protected]>

Give each node-local vLLM replica its own tokenizer cache so concurrent startup cannot read or remove another replica’s partially downloaded vocabulary. Preserve an explicitly configured cache and clean generated caches after the replicas stop. Signed-off-by: Nabin Mulepati <[email protected]>

* feat(slurm): merge runs by ID with default output Signed-off-by: Nabin Mulepati <[email protected]> * refactor(slurm): name default merge directory Signed-off-by: Nabin Mulepati <[email protected]> * test(slurm): align merge destination coverage Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): read only committed attempt results Ignore client-owned partial result files when an attempt has no bound candidate. Failed shards can then be reconciled and reported without treating their incomplete output as a finalized result. Signed-off-by: Nabin Mulepati <[email protected]> * accept nonempty partial Slurm shards Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): analyze nonempty partial benchmarks Use actual winning rows for benchmark throughput and capacity estimates. Update the benchmark result contract and Slurm docs for partial winners. Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): merge reordered nested shard schemas Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): preserve saved collection plans Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): benchmark one array task per case Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): analyze saved benchmark manifests Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): validate benchmark versions lazily Signed-off-by: Nabin Mulepati <[email protected]> * docs: clarify saved benchmark rates Signed-off-by: Nabin Mulepati <[email protected]> * feat(slurm): default benchmark records adaptively Signed-off-by: Nabin Mulepati <[email protected]> * docs: fix Slurm example references Signed-off-by: Nabin Mulepati <[email protected]> * docs: clarify Slurm topologies and shards Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

Add opt-in shared-node GPU quotas and explicit GPU offsets for colocating distinct models, including multiple models on one GPU. Coordinate readiness before overlapping launches and support typed vLLM IO processor plugins. Signed-off-by: Andre Manoel <[email protected]>

Signed-off-by: Andre Manoel <[email protected]>

* fix(slurm): harden run lifecycle and diagnostics Validate built-in builder settings before allocation while deferring plugin and compute-local inputs. Protect image artifacts referenced by run plans and verify reused client overlays. Correct readiness, progress, benchmark, and multi-node behaviors found during branch review. Add regression coverage and update Slurm docs. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): address review findings Poll readiness across all started servers so a slow deployment cannot consume another deployment timeout. Protect images referenced by custom run IDs and retry transient batch metadata failures in progress reporting. Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): narrow artifact locks and validate plans Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* refactor(slurm): simplify preflight and readiness tests Signed-off-by: Nabin Mulepati <[email protected]> * test(slurm): reuse readiness command helper Signed-off-by: Nabin Mulepati <[email protected]> * refactor(slurm): simplify run preparation ownership Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

* fix(slurm): tag submitted jobs for telemetry Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): honor user deployment type during submission Signed-off-by: Nabin Mulepati <[email protected]> * feat(telemetry): recognize internal Slurm deployments Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): preserve deployment tags across job operations Signed-off-by: Nabin Mulepati <[email protected]> * fix(slurm): align fallback tag across initial and retry jobs Signed-off-by: Nabin Mulepati <[email protected]> --------- Signed-off-by: Nabin Mulepati <[email protected]>

Signed-off-by: Andre Manoel <[email protected]>

Signed-off-by: Nabin Mulepati <[email protected]>

nabinchha force-pushed the feat/slurm-execution branch from f404976 to a498342 Compare October 15, 2026 21:09

nabinchha changed the title ~~feat(slurm): integrate Slurm execution for 1.0.0rc1~~ feat(slurm): integrate Slurm execution for 1.0.0 release Oct 16, 2026

github-actions Bot mentioned this pull request Oct 19, 2026

Agentic CI: Issue & PR Triage Tracker #562

Open

andreatnvidia and others added 26 commits October 22, 2026 14:44

feat: add Slurm image lifecycle jobs (#896)

f25162e

feat: add one-node Slurm runtime (#909)

8fbca0c

feat: add allocation-local Slurm client worker (#911)

3be7063

* feat(slurm): add allocation-local client worker * fix(slurm): address client worker review feedback * fix(slurm): harden client worker boundaries * fix(slurm): harden client filesystem boundary

feat(slurm): add profile setup and image lifecycle wiring (#933)

6636e2c

* feat(slurm): add profile setup and image lifecycle wiring * fix(slurm): address profile and lifecycle review * fix(slurm): require terminal cancellation evidence

nabinchha and others added 27 commits October 22, 2026 14:44

fix(slurm): support Enroot 3.5 OCI imports (#988)

6e8ce60

Signed-off-by: Nabin Mulepati <[email protected]>

fix(slurm): leave host patterns empty by default (#1014)

1646120

Signed-off-by: Nabin Mulepati <[email protected]>

fix(slurm): reuse final batch files for candidates (#1023)

f4b3424

Avoid writing a duplicate Parquet export when the engine batches can be used directly as a bounded candidate. Keep the export path for schema drift or too many batch files. Signed-off-by: Nabin Mulepati <[email protected]>

fix: recognize locked dependency licenses

f522646

fix(slurm): deduplicate runtime inventory paths

98f19ef

Signed-off-by: Andre Manoel <[email protected]>

fix(slurm): sync checked examples with documentation

21b7595

Signed-off-by: Andre Manoel <[email protected]>

fix(slurm): speed up allocation artifact checks (#1071)

9ba05f5

Signed-off-by: Andre Manoel <[email protected]>

nabinchha force-pushed the feat/slurm-execution branch from 7bc52f4 to 9ba05f5 Compare October 22, 2026 20:47

dhruvnathawani mentioned this pull request Oct 23, 2026

feat(slurm): support hosted models through native DD provider settings #1080

Draft

12 tasks done

fix(slurm): remove vLLM release allowlist (#1086)

4cb13d9

Signed-off-by: Nabin Mulepati <[email protected]>

Navigation Menu

Repository navigation

feat(slurm): integrate Slurm execution for 1.0.0 release - #1040

feat(slurm): integrate Slurm execution for 1.0.0 release#1040
nabinchha wants to merge 62 commits into
mainfrom
feat/slurm-execution

nabinchha commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nabinchha commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Summary

🔗 Related Issue

🔄 Changes

✨ Added

🔧 Changed

🐛 Fixed

🔍 Attention Areas

🧪 Testing

✅ Checklist

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nabinchha commented Oct 15, 2026 •

edited

Loading