[BUG]: numba-cuda-mlir cu13 nightly — version gate drops the cuobjdump deselect along with the fixed #135 ones

[rparolin]

Part of #2722. Upstream skip-guard bug now filed: NVIDIA/numba-cuda-mlir#286.

Nightly numba-cuda-mlir (linux-64) / Python 3.12, CUDA 13.3.0 fails with 2 test failures. The CUDA 12.9.1 row passes.

Symptom

Run 33038082047, job 98405280699 — 2 failed, 4615 passed, 62 skipped, 301 xfailed. Both failures are the same test, two parametrizations:

_ TestLinkerDumpAssembly.test_nvjitlink_jit_with_linkable_code_lto_dump_assembly_warn

E   FileNotFoundError: [Errno 2] No such file or directory: 'cuobjdump'
E   RuntimeError: cuobjdump has not been found. You may need to install the CUDA toolkit
E                 and ensure that it is available on your PATH.
E   Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.

Root cause — a stale version gate, not a CUDA 13 regression

test-wheel-linux.yml:495 guards the entire deselect list on one version comparison:

if python -c "from packaging.version import Version; import sys; \
    sys.exit(0 if Version('${NUMBA_CUDA_MLIR_VER}') <= Version('0.4.1') else 1)"; then
  DESELECTS+=( ... 12 entries ... )
fi

The cu13 row now resolves NUMBA_CUDA_MLIR_VER: 0.5.0, so the guard is false and all twelve deselects are dropped. The log confirms the invocation carried none:

+ pytest -rxXs -v --ignore=tests/benchmarks --ignore=tests/doc_examples tests/

The gate conflates two unrelated problems that the workflow's own comment carefully distinguishes:

• Serial pytest: 8 tests fail with module 'numba_cuda_mlir.cuda.cudadrv' has no attribute 'driver' numba-cuda-mlir#135 — serial-pytest state contamination. Genuinely fixed upstream; retiring those 11 deselects at 0.5.0 is correct.
• The cuobjdump deselect — annotated in-repo as "not on PATH in the base ubuntu:24.04 container … No upstream fix yet." This is an environment condition with no version fix, and the same gate retired it anyway.

Why cu12.9.1 passes: that row still resolves numba-cuda-mlir ≤ 0.4.1, so the guard holds and the workaround stays applied. CUDA 13.3.0 is simply which row upgraded first — nothing in the traceback touches a CUDA API.

Fix

Preferred — fix the environment, not the symptom. Put cuobjdump on PATH in the container (install cuda-cuobjdump for the matrix CUDA major, or export the local CTK bin/). This removes the failure and re-enables the 4 currently-skipped test_dwarf_cubin.py tests — strictly more coverage:

SKIPPED [4] tests/infra/test_dwarf_cubin.py:41: cuobjdump not found (set CUDA_HOME or ensure cuobjdump is on PATH).

Minimum — split the gate so the environment workaround stops riding on an unrelated version check:

# Environment-conditional: cuobjdump is absent from the base ubuntu:24.04
# container. Version-independent; do NOT fold into the #135 gate below.
# Upstream skip-guard: NVIDIA/numba-cuda-mlir#286
DESELECTS+=(
  --deselect 'tests/numba_cuda_tests/cudadrv/test_nvjitlink.py::TestLinkerDumpAssembly::test_nvjitlink_jit_with_linkable_code_lto_dump_assembly_warn'
)

# NVIDIA/numba-cuda-mlir#135 — fixed in 0.5.0.
if [ ... <= 0.4.1 ]; then
  DESELECTS+=( ...the 11 contamination deselects... )
fi

Wider lesson

Version-gated test workarounds should be gated individually. A single shared gate will eventually retire a workaround whose cause was never fixed — which is exactly what happened here. Captured in #2722.

Severity

Low as a defect — 2 failures out of 4615, purely environmental, no product bug and no CUDA 13 incompatibility. Non-trivial as a cost, because it red-lights the whole nightly. Cheapest of the five in #2722 to fix.