From f14af7626a50c575c4dac6c8ef4ba48ea7562d55 Mon Sep 17 00:00:00 2001 From: Doug Holt Date: Fri, 6 Nov 2026 05:39:37 -0700 Subject: [PATCH] docs(model-workflows): align cache file lists and note GPU smoke pre-pull - vllm-quickstart: fix 'verified on' date phrasing - Explain why serve (3 files) and fine-tune readiness (adds tokenizer_config.json) differ; note it in serve-model, shared-cache and finetune-readiness - Document that --gpu-smoke's 5s scheduler window requires a pre-pulled image - Comment that gpu_count() fails closed on unexpected Slurm output Signed-off-by: Doug Holt --- docs/model-workflows/finetune-readiness.md | 10 ++++++++-- docs/model-workflows/shared-cache.md | 4 ++++ docs/model-workflows/vllm-quickstart.md | 11 ++++++++--- scripts/validation/validate_finetune.py | 2 ++ skills/finetune-model/SKILL.md | 5 +++-- skills/serve-model/SKILL.md | 5 ++++- 6 files changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/model-workflows/finetune-readiness.md b/docs/model-workflows/finetune-readiness.md index 9c923bdbd..3403f4140 100644 --- a/docs/model-workflows/finetune-readiness.md +++ b/docs/model-workflows/finetune-readiness.md @@ -24,7 +24,9 @@ uses Python's standard library and the existing local cache validator. an image or verify its framework compatibility, contents, trust or registry accessibility. The operator must choose a reviewed fine-tuning image. - Supported cache layout: `config.json`, `tokenizer_config.json`, - `tokenizer.json`, and either `model.safetensors` or + `tokenizer.json` (the vLLM quickstart's serving example lists only + `config.json`, `tokenizer.json` and the weights; this check is stricter and + requires `tokenizer_config.json` too), and either `model.safetensors` or `model.safetensors.index.json` with a nonempty `weight_map`. Every indexed shard must exist, be nonempty, and resolve within the cache. Weight indexes larger than 8 MiB fail closed. The explicit required-file list must include @@ -74,7 +76,11 @@ srun --nodes=1 --ntasks=1 --gpus=1 --immediate=5 --time=1 \ This may pull the container image. The allocation has a one-minute scheduler time limit and a five-second immediate-allocation limit; the local command has -the configured timeout. A slow image pull can make the probe fail. Confirm job +the configured timeout. A slow image pull can make the probe fail, because the +five-second window is too short to pull an image: **pre-pull or pre-import the +pinned image on the target node** (for example with the site's normal +Enroot/Pyxis import step) before using `--gpu-smoke`, and treat a failure on a +cold node as inconclusive. Confirm job termination through the site's normal procedure after a timeout; do not blindly retry. Require a zero exit and exactly one GPU row. No training, dataset loading, model loading or shared-cache mounting is performed by this probe. In particular, diff --git a/docs/model-workflows/shared-cache.md b/docs/model-workflows/shared-cache.md index dae262854..0ff7526d4 100644 --- a/docs/model-workflows/shared-cache.md +++ b/docs/model-workflows/shared-cache.md @@ -92,6 +92,10 @@ export MODEL_REVISION=607a30d783dfa663caf39e06633721c8d4cfcd7e --revision "$MODEL_REVISION" --cache-dir "$HF_HUB_CACHE" ``` +The example list is what the vLLM quickstart needs. The fine-tuning readiness +check also requires `tokenizer_config.json`, so download it too if the same +snapshot will be used for fine-tuning. + For another model, explicitly list all needed configuration, tokenizer and weight files. For sharded weights, include the index **and every referenced shard**. A successful download of `config.json` alone does not make a usable model. diff --git a/docs/model-workflows/vllm-quickstart.md b/docs/model-workflows/vllm-quickstart.md index 527d97b17..62066367e 100644 --- a/docs/model-workflows/vllm-quickstart.md +++ b/docs/model-workflows/vllm-quickstart.md @@ -53,7 +53,7 @@ python3.12 -m venv "$HOME/.venvs/deepops-vllm" "$HOME/.venvs/deepops-vllm/bin/vllm" --version ``` -The example pins vLLM **0.31.0**, licensed under **Apache-2.0** (verified +The example pins vLLM **0.31.0**, licensed under **Apache-2.0** (verified on 2026-10-07 against the [tagged license](https://github.com/vllm-project/vllm/blob/v0.31.0/LICENSE) and [package metadata](https://pypi.org/project/vllm/0.31.0/)). This is the serving package's license, not the model's or every dependency's. Transitive packages @@ -67,7 +67,7 @@ package problem. vLLM loads a local snapshot, not a mutable Hub repository name. The example model, `openai-community/gpt2` at `607a30d783dfa663caf39e06633721c8d4cfcd7e`, is listed as **MIT** in its [pinned model card](https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md) -(verified 2026-10-07). Review its license terms before use; re-check the model's +(verified on 2026-10-07). Review its license terms before use; re-check the model's license and access requirements when selecting another model or revision. ```bash @@ -80,7 +80,12 @@ python3 scripts/validation/validate_model_cache.py --json \ --require-file config.json --require-file tokenizer.json --require-file model.safetensors ``` -Stop unless exit status is `0` and `ok` is `true`. For a different model, review +Stop unless exit status is `0` and `ok` is `true`. These three files are the +set this example was validated with. The fine-tuning readiness check +([finetune-readiness.md](finetune-readiness.md)) additionally requires +`tokenizer_config.json` because its validator insists on that layout; add it +here too if your model's tokenizer needs it, or if one cache snapshot will serve +both workflows. For a different model, review its complete tokenizer/configuration/weight file list, including every indexed shard. The validator only checks files you name. Never enable remote model code as a workaround; this example does not use `--trust-remote-code`. diff --git a/scripts/validation/validate_finetune.py b/scripts/validation/validate_finetune.py index cbc23347c..b48fb9d69 100644 --- a/scripts/validation/validate_finetune.py +++ b/scripts/validation/validate_finetune.py @@ -80,6 +80,8 @@ def parse_record(output): def gpu_count(tres): + # Fails closed: if Slurm changes the TRES format, no match returns None and + # the node is reported not ready instead of guessed. matches = re.findall(r"(?:^|,)gres/gpu=(\d+)(?:,|$)", tres) return int(matches[0]) if len(matches) == 1 else None diff --git a/skills/finetune-model/SKILL.md b/skills/finetune-model/SKILL.md index da43944d1..4e0b486e9 100644 --- a/skills/finetune-model/SKILL.md +++ b/skills/finetune-model/SKILL.md @@ -52,8 +52,9 @@ python3 scripts/validation/validate_finetune.py --json \ Off by default. Only with explicit authorization to allocate a GPU and run the reviewed image, repeat the command with `--gpu-smoke`. This can download the -container image and submits one bounded, single-node, single-GPU `nvidia-smi` -job after all readiness gates pass. It does not load the model or train. +container image, but its scheduler window is only five seconds, so pre-pull the +pinned image on the node first. It submits one bounded, single-node, single-GPU +`nvidia-smi` job after all readiness gates pass. It does not load the model or train. `gpu_smoke_ok: true` proves only the container GPU probe, not cache access inside the container, framework compatibility, learning or checkpoint recovery. diff --git a/skills/serve-model/SKILL.md b/skills/serve-model/SKILL.md index 7a750b7aa..b078c5090 100644 --- a/skills/serve-model/SKILL.md +++ b/skills/serve-model/SKILL.md @@ -29,7 +29,10 @@ or authorize a workload. 1. Follow quickstart steps 1–2. Require cache validator exit `0` **and** `ok: true` as the actual reader before startup. Config alone is not a complete model; - name every required weight shard and tokenizer file. + name every required weight shard and tokenizer file. The three files in the + example are the set it was validated with; `finetune-model` also requires + `tokenizer_config.json`. Add it for any model whose tokenizer needs it, or + when one cache snapshot serves both workflows. 2. Use quickstart step 3 unchanged for the public example: local snapshot and tokenizer, offline loading, one GPU, foreground process, loopback only. Never enable remote model code. `HF_HUB_OFFLINE=1` is not a firewall and belongs only