Skip to content

[BUG]: cuda.core tests fail with torch cu126 installed (NVRTC 12.6 options, torch CAI v2) #3333

Closed

Description

@0z5a

Is this a duplicate?

Type of Bug

Something else

Component

cuda.core

Describe the bug

With cuda-core 1.2.2 [cu12] and torch 2.14.1+cu126 in the same environment, cuda_core/tests has 10 failures that come from the tests, not from cuda.core. The same parametrizations are unchanged on main.

  1. test_program.py::test_cpp_program_with_various_options: ProgramOptions(no_cache=True), ProgramOptions(frandom_seed="12345") and ProgramOptions(ofast_compile="min") fail with NVRTC_ERROR_INVALID_OPTION. torch cu126 pins nvidia-cuda-nvrtc-cu12==12.6.85, and these options need NVRTC 12.9. Compiling an empty kernel with each option through cuda.bindings.nvrtc:

    Option NVRTC 12.6.85 12.8.93 12.9.86
    --no-cache unrecognized option unrecognized option OK
    --frandom-seed=12345 unrecognized option unrecognized option OK
    --Ofast-compile=min invalid argument min invalid argument min OK
    --pch unrecognized option OK OK

    The PCH parametrizations in the same list are already skipped below NVRTC 12.8; these three have no version gate.

  2. test_utils.py::TestViewCudaArrayInterfaceGPU::test_cuda_array_interface_gpu[torch-*] (7 cases) fail with BufferError: only CUDA Array Interface v3 or above is supported. torch still exports __cuda_array_interface__ with "version": 2, so StridedMemoryView.from_cuda_array_interface rejects every torch sample that _gpu_array_samples() adds. This test is listed in EPIC: Tests that never run in CI #2077 because CI has no array library installed for it; with torch installed it does run and these samples fail.

How to Reproduce

  1. pip install "cuda-core[cu12]==1.2.2" torch --index-url ... (torch 2.14.1+cu126), plus the test dependencies.
  2. Run pytest cuda_core/tests/test_program.py cuda_core/tests/test_utils.py from the cuda-core-v1.2.2 sources on a CUDA GPU.

Expected behavior

The three option cases skip on NVRTC older than 12.9, like the PCH cases skip below 12.8. The CAI test either leaves out torch samples or expects the BufferError for CAI v2 producers.

Operating System

Ubuntu 24.04, x86_64

nvidia-smi output

Tesla V100-SXM2-16GB, driver 580 (CUDA 13.0). The failures are not GPU-specific.

Activity

  1. leofang commented on Nov 2, 2026

    @leofang
    Member

    2 was fixed in #3033 I believe.

    1 just needs updated NVRTC. I don't think we have very extensive per-major-minor checks for everything provided by cuda-core. While we don't want to maintain an effectively lookup table in cuda-core, @Andy-Jost might have some thoughts on how we want to approach this in the long term.

  2. added
    cuda.coreEverything related to the cuda.core module
    on Nov 2, 2026
  3. Andy-Jost commented on Nov 2, 2026

    @Andy-Jost
    Contributor

    The option tests only check that cuda-core encodes each NVRTC option correctly. For instance, they check that ProgramOptions(no_cache=True) emits a flag that NVRTC accepts (--no-cache), not one it rejects (such as --no_cache).

    I agree we should not keep a per-option version table for that. A correct encoding only has to pass once, on a new-enough NVRTC, and CI already runs the newest NVRTC for each major version. I suggest we gate the option tests on the installed cuda-bindings version: they run when the NVRTC major.minor is at least the cuda-bindings major.minor, and they skip on an older NVRTC. The gate follows cuda-bindings releases, so it needs no updates when NVRTC adds options.

  4. added a commit that references this issue on Nov 3, 2026
    b462136
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cuda.coreEverything related to the cuda.core moduletriageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions