[rwgk]

— codex GPT-6 on behalf of Ralf —

Layer 5 of 8; last of the review group #3174 + #3175 + #3176, which lands as one merge. This PR is the group's full-CI target. Previous: #3175. Next: #3177. Stack map at the bottom.

The current tagging proposal can land shared prerequisite #3314 first, create the retained v12.9.11a0 milestone, then restack both alternatives onto that commit and remove the bootstrap.

What this PR does

The test side of running both bindings majors from main, plus the documentation that explains the result.

• test-wheel-linux.yml, test-wheel-windows.yml: matrix rows are annotated from the registry by the new ci/tools/select_test_matrix.py (replaces a jq program duplicated in both files). Rows whose toolkit matches a registered root test that root's wheels from this run; older-toolkit rows test Core against published floor bindings. Job names say what runs, for example pathfinder+bindings12+core12+meta12 / py3.12, CTK12.9.1 (wheels; bindings PR). A PR cannot silently lose a required local-wheel row. CI tools come from the workflow revision (.ci-control), so nightly can test an older artifact source with current tooling.
• ci-nightly.yml: extracts only ci/versions.yml from the artifact-source archive, reads that registry, and stops with a clear message if that source predates it (rollout recipe in ci/README.md).
• coverage.yml, ci-pixi-source-test.yml: registry-driven; the pixi smoke selects roots from the plan and runs the version preview bootstrap for the CUDA 12 root. Scheduled GPU source tests cover both registered environments; manual dispatch selects one. The container checkout explicitly trusts its workspace before the bootstrap.
• build-docs.yml: tooling from a .ci-control checkout, package data from the release source; CUDA 13 docs built against the checkout's bindings; build_docs.sh keeps .dev and rc versions out of the stable directory.
• CUDA 12 test maintenance ported from CUDA 13: NVML runner adaptations (Orin/Thor, PCI info, driver >= 13.4 refcount skip, flaky energy test removed) and serialization of the process-global coredump test (Mike's Fix coredump tests on free-threaded builds #3226, marker on the CUDA 12 copy).
• Docs: CONTRIBUTING.md (CI overview), ci/README.md (reading CI jobs and summaries, nightly rollout, historical reruns), both roots' install pages, CUDA 12 driver baseline correction.

After the group merges

• main builds, tests, and documents both majors; CUDA 12 bindings for Core come from the same run, not from the 12.9.x branch.
• CUDA 13 releases keep working (tag CI builds only the matching line). CUDA 12 releases from main need Release: select bindings artifacts from tagged package configuration #3177.
• Nightly needs its first successful post-merge main run before auto-detection works; until then dispatch it with a compatible run-id (recipe in ci/README.md). Docs reruns and dry runs for tags from before the registry need the 13.4.x branch tooling.

Where to look

• ci/tools/select_test_matrix.py: short and self-contained.
• .github/workflows/test-wheel-linux.yml: thin matrix wrapper plus suite execution (the Windows file mirrors it).
• ci/tools/run-tests: which bindings each suite installs.
• ci/README.md: "Reading CI Jobs and Summaries" and "Nightly Artifact Sources And Rollout".

All changes (35 files; 34 without the lockfile, +1,116 / -510)

Workflows: test-wheel-linux.yml, test-wheel-windows.yml, ci-nightly.yml, coverage.yml, ci-pixi-source-test.yml, build-docs.yml, .github/actions/doc_preview/action.yml (pin bump).

Tools: select_test_matrix.py (new), run-tests, lookup-run-id (orphaned --artifact option removed), tests (test_select_test_matrix.py new; test_merge_cuda_core_wheels.py; workflow-text tests removed).

CUDA 12 tests: cuda_bindings_12/tests/nvml/{test_cuda.py,test_init.py,test_pynvml.py,util.py}, cuda_bindings_12/tests/test_cuda.py.

Docs and environments: CONTRIBUTING.md, ci/README.md, ci/ci-pipeline.mmd, cuda_bindings_12/docs/{README.md,source/install.md,source/release/12.X.Y-notes.rst}, cuda_bindings_13/docs/{README.md,build_docs.sh,source/install.rst,source/support.rst}, cuda_bindings_13/pixi.toml + lock (docs feature builds bindings from source), cuda_core/docs/source/install.rst, cuda_python/docs/{README.md,environment-docs.yml}, toolshed/setup-docs-env.sh.

Review history and validation

• Mike's review (2026-10-26): dead --artifact option (removed), a CI-test policy (now ci/tools/tests/AGENTS.md; the suite shrank accordingly), and simplification suggestions (Python matrix selection and Python-written build environment adopted; composite action for the duplicated build blocks deferred; real v12.9.11a0 tag initially deferred until the group landed). Re-checked on 11-01 (summary).
• Full CI for the group passed at the pre-rebase head 34664b27 (run 40996807619, 105 jobs); the complete stack passed at cuda.bindings: backport CUDA 12 runtime maintenance fixes #3182's pre-rebase head 341d286d (run 40997121153, attempt 3, 105 jobs).
• Earlier head 632711bd254d83397f415532021c5392cd86cdcc (rebased onto main 57b27097, with the latest simplification commits): run 41373891909 finished with 103 of 105 jobs green; one GPU job (linux-64 WSL, RTX 4090, Python 3.12, CTK 13.4.2) failed in the cuda.bindings benchmark smoke step and took the aggregator with it. Not yet triaged; the CUDA 13 bindings sources are unchanged from main.
• Hosted artifact-reuse rehearsals CI: rehearse CUDA 12 source-change artifact reuse (do not merge) #3237 and CI: rehearse Linux-only artifact reuse without package builds (do not merge) #3238 passed against the 10-27 state of this PR.
• Local validation at 9bfd39eac8a0d1d57c8b53702f13874a691a37d9: 363 CI-tool tests and full pre-commit passed. Hosted CI passed on #3355 and the alternative stack ending at #3322.

Stack map: review scopes and merge units

Layer	PR	Scope	Merge unit
1	#3172	import cuda_bindings_12/, byte-identical to the 12.9.x branch	merged
2	#3173	integrate the import with the shared build tooling	merged
3	#3174	registry, packaging, version selectors, developer environment, guidance	group: #3174 + #3175 + #3176 land together
4	#3175	build both majors from main; planner; version preview bootstrap	group
5	#3176	test, document, nightly, coverage; first CI-complete state	group
6	#3177	releases driven by the tagged source's configuration	after the group
7	#3178	maintenance policy, manual backport workflow, retire backport_branch	after #3177
8	#3182	three CUDA 12 runtime fixes	independent of the CI layers; stacked only for ordering

Layers 3 to 5 are review boundaries, not working intermediate states: #3174 alone breaks CI (workflows still read the removed cuda.build.version), and #3175 alone fails actionlint (callers pass an input the test workflows only accept in #3176). The plan is to review them separately and land the reviewed group as one merge. #3176 is the full-CI target for the group.

The staged stack #3314-#3322 is the alternative in which every layer is mergeable on its own; both stacks reach the same end state, except that the staged stack keeps compatibility routes for older artifact sources and retained CUDA 13 tags.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[rwgk]

/ok to test 9b8ac76

[rwgk]

/ok to test 19865b0

[mdboom]

The following concerns I had from my last review are still not complete:

1. Push a real v12.9.11a0 tag and delete the synthetic-tag machinery.

There is a significant amount of complexity here to work around the fact that we don't have an existing v12.9.X tag on main. Adding a prerelease tag of some sort on main would allow us to remove all of it. I have not all read the verbose comments since I made this request but my agent tells me there is no mention of it (a decision to not do it, because X... for example). This is the one I think it the most important because it is truly temporary and unnecessary content.

3. De-duplicate the current/maintenance blocks in build-wheel.yml.

This is 345 lines of repeated boiler plate that we could remove.

2. Move the test-matrix jq into Python.

This was partially addressed. There is still some jq usage remaining we don't have to be 100%, but my klanker identified a couple additional simplifications:

Not worth moving

• The caller's matrix_filter. This is the jq -c '${{ inputs.matrix_filter }}' line.
  • Every caller passes the same one-liner shape: map(select(.ENV.MODE == "nightly-pytorch")). There are about 13 of them in ci-nightly.yml.
  • Moving this to Python means replacing the free-form jq expression with a different interface.
  • That could be a --mode argument or a Python filter language. Each caller would change, and you would still have the same mode strings.
  • That is more work and a bigger surface than the jq expression it replaces.
• The MATRIX_ENV line. echo "$MATRIX_ENV" | jq -r 'to_entries[] | "\(.key)=\(.value)"' >> "$GITHUB_ENV" is already a single standard line. A Python helper would add a file and a call without simplifying anything.

Worth simplifying

• The step-summary table. This is the jq -r '.include[] | "| \(.SUITES) | ..."' line, duplicated in both wheel workflows. It is cosmetic output. Per your inline note on simplification 6, it would be better to print the matrix JSON, as the plan summary now does. That would delete the jq and the duplication together.
• The yq | jq | python pipeline. Once the table is gone, select_test_matrix.py could read ci/test-matrix.yml and apply the architecture filter itself. The yq -o json ... map(select(.ARCH == ...)) step would then move into Python and the jq filter would be the only shell JSON handling left. That is a modest gain. It is optional and mostly a matter of taste.

I'd leave matrix_filter and MATRIX_ENV as they are and drop the summary table. That gives most of the benefit of your simplification 2 without the filter interface change.

(I did not finish reviewing this, because I understand #3182 and #3322 are superceding this).

[rwgk]

— codex GPT-6 on behalf of Ralf —

Thanks, Mike. I agree that a retained v12.9.11a0 tag would let us delete the temporary bootstrap helper, its tests, the metadata field, and the workflow calls.

The earlier response deferred this until the validated per-line configuration lands on main. The simplification is still unimplemented; the remaining question is its timing.

There is a tag-placement issue to handle: main’s CUDA 13 version selector still matches v*[0-9]*. Placing a CUDA 12 tag at main’s current head could therefore make CUDA 13 builds derive a CUDA 12 version.

One straightforward sequence is:

1. Land the per-line selectors.
2. Create the retained milestone on validated main history.
3. Immediately remove the bootstrap machinery.

Waiting until after integration is a conservative ordering choice, not a setuptools-scm requirement. If we want the deletion before integration, we could evaluate a safe earlier tag anchor or a small preparatory selector PR.

I’d like to settle that ordering explicitly so this temporary machinery has a concrete removal point.

[rwgk]

Thanks Mike. I've been organizing this work into safe, meaningful steps. I'd like to complete and validate the migration off 12.9.x first, then make these maintainability improvements separately. That keeps each review focused and makes regressions easier to isolate.

For these two suggestions in your follow-up review, I propose follow-ups:

De-duplicate the current/maintenance blocks in build-wheel.yml.

I agree this would improve maintainability. I propose extracting the shared implementation after migration, preserving the toolkit-switching and wheel-merging sequence.

The step-summary table. … it would be better to print the matrix JSON

Agreed. I propose replacing both tables with the matrix JSON while retaining the caller’s matrix_filter and simple MATRIX_ENV export, as you recommend.

I'd be happy to prepare two follow-up PRs now to address your requests, and stack them above #3182, so they can be reviewed alongside the migration and merged afterward. Would that work for you?