[GodlyDonuts]

Related to #3038.

Disable nvJitLink's intermediate cache when incremental linking and link-time optimization are both enabled. Repeated native round trips reproduced access violations on Windows with caching enabled, including through direct calls that bypass cuda-bindings.

Apply -no-cache at every incremental LTO stage, document the behavior, and add regression coverage for option generation. Other linking modes retain their existing caching behavior.

Validation:

• Cached native runs crashed in 7 of 8 processes; cache-disabled runs completed 1,000 round trips in each of 8 processes.
• Free-threaded Python 3.14 linker tests: 81 passed, 3 skipped with four workers, using the edited option-generation method injected into the existing wheel.
• Cython C++ translation and git diff --check passed.

Draft pending rebuilt-extension validation and confirmation against the original MCDM CI environment. This mitigates the reproduced failure; it does not establish that every crash in #3038 has the same cause.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cuda-python
• Branch: main
• File: .coderabbit.yaml
• Commit: 01037b1
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: f04b099d-1881-4d49-b3c8-7f585ab19249

Commits

Reviewing files that changed from the base of the PR and between ebab5f5 and 16f89fa.

Files selected for processing: 1

• cuda_core/cuda/core/_linker.pyx

Files with no reviewable changes: 1

• cuda_core/cuda/core/_linker.pyx

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Summary

Summary by CodeRabbit

• Bug Fixes
  • Incremental links that use link-time optimization now automatically disable intermediate caching when running with nvJitLink versions earlier than 13.6. Explicitly disabling caching continues to work across versions, while newer versions retain the existing caching behavior unless caching is explicitly disabled.
• Documentation
  • Clarified the version and linking conditions under which intermediate caching is disabled automatically.

The nvJitLink options builder emits -no-cache when no_cache is true or when incremental linking and link-time optimization are enabled on nvJitLink versions older than 13.6. The change also preserves the cached nvJitLink version on the cached-backend return path.

Changes: Incremental LTO cache handling

Layer / File(s)	Summary
Retain cached backend version cuda_core/cuda/core/_linker.pyx	The cached-backend return path no longer resets _nvjitlink_version to None.
Document and apply cache option cuda_core/cuda/core/_linker.pyi, cuda_core/cuda/core/_linker.pyx, cuda_core/tests/test_linker.py	The option documentation describes automatic cache disabling for incremental LTO links on nvJitLink versions older than 13.6. The options builder emits -no-cache when the version and linking options meet that condition, or when no_cache is true. The test covers versions 13.4, 13.5, 13.6, and 14.0 and all combinations of the relevant options.

Change: Bug fix

Merge Risk: Minimal · up to 16f89

The reviewed change has no identified issue requiring a fix before merge. Rebuilt-extension validation and confirmation in the original CI environment remain pending.

- [ ] Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[leofang]

@GodlyDonuts this comes out of blue. Could you please present your analysis on why you think the linker cache is to blame? For example, a stack trace when the segfault happens would be nice.

[GodlyDonuts]

The original CI failure happens during the first incremental LTO link, but that traceback alone doesn’t prove caching is the cause. I reduced the path to nvJitLink directly and tested it through both cuda-bindings and ctypes on Windows with nvJitLink 13.4.92.

With caching enabled, 7/8 processes failed across the two paths. With -no-cache on both incremental LTO stages, all 8/8 completed 1,000 round trips each. I also tried disabling the cache only for the first stage and still got a crash at the second cached stage, which is why the change applies whenever incremental=True and LTO is enabled.

The crash dump lands inside nvJitLink_130_0.dll and includes the Windows freed-memory pattern, so a native lifetime issue looks plausible, but I haven’t established the exact internal defect or which cache object is responsible.

This supports the workaround but actual confirmation against the CI failures is still pending.

[GodlyDonuts]

Here is the native stack excerpt from the local reproduction, captured with caching enabled and one worker thread. Personal paths and machine metadata have been removed.

Exception: C0000005.ACCESS_VIOLATION

rdx=feeefeeefeeefeee
rip=00007ffdaa2813a2

nvJitLink_130_0!_nvJitLinkGetLinkedPtx_13_1+0x1434f92:
00007ffd`aa2813a2 ff5210  call qword ptr [rdx+10h]

Call stack (top frames):
nvJitLink_130_0!_nvJitLinkGetLinkedPtx_13_1+0x1434f92
nvJitLink_130_0!_nvJitLinkAPI+0xf17
nvjitlink_cp310_win_amd64_7ffe2b9b0000+0x2ecf
cynvjitlink_cp310_win_amd64+0x1250
nvjitlink_cp310_win_amd64+0x36f9
nvjitlink_cp310_win_amd64+0x3939
python310!_PyEval_EvalFrameDefault+0x1a3b

The faulting instruction dereferences rdx, which contains a freed-memory pattern. This points toward a native lifetime issue, but doesn’t identify the responsible cache object. The large symbol offset also means the displayed GetLinkedPtx name should not be treated as identification of the actual internal function.
segfault-stack-sanitized.txt

[GodlyDonuts]

@claude review this PR

[leofang]

Thanks, @GodlyDonuts. I have shared your traceback with the compiler team and they are looking into it. I'll keep you posted, though at this point we cannot yet establish that the cache is to blame and accept this PR; the compiler team is against turning off the cache.

[leofang]

Thanks for your patience and data, @GodlyDonuts, our compiler team was able to confirm and fix a use-after-free bug. A future release of nvJitLink will fix it and in the meanwhile they agreed turning off cache is an acceptable workaround. Let's get this PR in!

[GodlyDonuts]

Thanks for confirming! Added the pre-13.6 guard, updated the docs and tests, and rebased onto main.

[coderabbitai]

Actionable comments posted: 1

---

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cuda-python
• Branch: main
• File: .coderabbit.yaml
• Commit: 01037b1
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 6606499f-9ff9-4d6a-a64d-737e1c06191a

Commits

Reviewing files that changed from the base of the PR and between 0e4a3d9 and ebab5f5.

Files selected for processing: 3

• cuda_core/cuda/core/_linker.pyi
• cuda_core/cuda/core/_linker.pyx
• cuda_core/tests/test_linker.py

Files skipped from review as they are similar to previous changes: 1

• cuda_core/cuda/core/_linker.pyi

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

[leofang]

/ok to test 16f89fa

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.