[BUG]: cuda.bindings._test_helpers removed without a shim — breaks released-cuda-core nightly since Aug 4

[rparolin]

Part of #2722.

The nightly Nightly cuda-core (released) jobs have failed since 2026-08-04 with a pytest collection error — zero tests run.

Symptom

Run 33038082047, jobs 98405282548 (linux-64) and 98405280561 (win-64):

________________________ ERROR collecting tests/system _________________________
tests/system/conftest.py:14: in <module>
    from cuda.bindings._test_helpers.arch_check import hardware_supports_nvml
E   ModuleNotFoundError: No module named 'cuda.bindings._test_helpers'
##[error]Process completed with exit code 2.

Confirmed identical on the Aug 20 nightly (job 96295460281), so this has been constant for 3+ weeks.

Root cause

This job is deliberately mismatched: it runs the released cuda-core test suite against a cuda-bindings wheel built from main, specifically to catch main-branch changes that break already-released cuda-core.

• test tree → released cuda-core v1.1.1, checked out at cuda-core-released/cuda_core/tests/
• library under test → cuda-bindings from main: cuda_bindings-13.3.2.dev244+g0477f61f6

PR #2384 (19e66499c20, 2026-08-04, "Fix #2377: Reorganize the test helpers (where appropriate) to cuda_python_test_helpers") deleted the module from the cuda-bindings package:

 cuda_bindings/cuda/bindings/_test_helpers/__init__.py           |   6 --
 cuda_bindings/cuda/bindings/_test_helpers/arch_check.py         |  70 -------
 cuda_python_test_helpers/cuda_python_test_helpers/arch_check.py |  93 +++++++
 cuda_core/tests/system/conftest.py                              |  28 -----

That PR correctly updated cuda_core/tests/system/conftest.py on main. But released cuda-core is frozen and still carries the old import. New bindings + old import + deleted module → collection error.

Verified: git ls-files cuda_bindings/cuda/bindings/_test_helpers/ returns nothing at HEAD.

Two things this is not

• Not a Python 3.14 issue. The linux-64 row runs Python 3.12.14 and fails with the identical message.
• Not a CUDA 13.3.0 API change. Nothing in the traceback reaches a CUDA call, and the released-cuda-core matrix contains only 13.3.0 rows — there is no 12.9.1 contrast to draw a conclusion from. Both rows fail; the job is 100% dead, not conditionally dead.

Fix

Preferred: restore cuda/bindings/_test_helpers/ as a thin deprecation shim re-exporting from cuda_python_test_helpers, for one release cycle:

# cuda_bindings/cuda/bindings/_test_helpers/arch_check.py
"""Deprecated shim. Moved to cuda_python_test_helpers in #2384 (2026-08-04).
Kept for one release so released cuda-core test trees keep importing.
Remove after cuda-core >= 1.2 is the oldest supported release."""
try:
    from cuda_python_test_helpers.arch_check import *  # noqa: F401,F403
except ImportError:  # test-helper package not installed in this environment
    def hardware_supports_nvml(*args, **kwargs):
        return True   # or vendor the real implementation here

Guard the re-export. A bare from cuda_python_test_helpers... makes the shim depend at import time on a test-only package; if that package is not installed in this job, the shim just swaps one ModuleNotFoundError for another. Either guard as above or vendor the handful of symbols the released conftest actually imports — today that is only hardware_supports_nvml. Worth confirming which package the job installs before choosing.

This addresses the general defect rather than the symptom: an underscore-private module that a released sibling package imports is, in practice, part of the contract with that sibling. Removing it needs the same deprecation cycle a public symbol would get.

Alternatives, both worse: pinning the job to released cuda-bindings defeats its purpose; patching the released conftest in CI hides the breakage from anyone running the released suite outside CI.

Prevention

Any PR deleting or moving a module under cuda/ should trigger the released-compat job at PR time rather than waiting for the nightly. Given a fixed CTK release train, gates have to be PR-time to matter.

Severity

No user-facing defect — _test_helpers is private and no shipped code path imports it. But this job's single purpose is to catch main-vs-released drift, and it has been sitting on a real instance of exactly that for three weeks without anyone noticing, because the nightly was already red. Anyone running released cuda-core's own suite against a nightly cuda-bindings wheel hits this today.

[mdboom]

My plan is to work on adding these shims as suggested here as a short-term stopgap.

However, the root problem is that we put the test_helpers in an uninstallable package and then use them directly from a checkout. This is fundamentally incompatible with the various combinations we are wanting to run in CI.

Follow-up work should be to make cuda_python_test_helpers a real installable package that flows through CI like the other packages do (it doesn't need to be / shouldn't be published on PyPI).