[BUG]: pixi nightly GPU job has never passed — container missing NVIDIA_VISIBLE_DEVICES

[rparolin]

Part of #2722. Related to #2204 (the notification-visibility issue this failure originally prompted).

The nightly GPU job pixi run test (cu13, linux-64, GPU) in ci-pixi-source-test.yml fails on its first step, and has done so on every scheduled run in the retained window (7/7, Aug 21–27). As far as I can tell it has never passed.

Symptom

Run 33039410513, job 98409363225:

+ nvidia-smi
/__w/_temp/e57682ae-6d73-4df9-b0fd-fc623ab194cc.sh: line 1: nvidia-smi: command not found
##[error]Process completed with exit code 127.

Exit 127 — not a driver error. nvidia-smi produced no output because the binary does not exist inside the container.

Root cause

The GPU hardware is fine. The runner banner in the same log reports a physical L4 attached:

##[group]NVIDIA Managed Runner
GPUs UIDs: GPU-00766149-eda1-acb8-f0d3-5209bc616237
Runner group name: 'nv-gpu-general'

The container is the problem. A container does not automatically see the host GPU — on these runners the nvidia-container-runtime prestart hook keys off NVIDIA_VISIBLE_DEVICES to decide whether to bind-mount the driver libraries and CUDA CLI tools in. Without it the hook is a no-op, and you get a plain ubuntu:24.04 with no CUDA anything.

ci-pixi-source-test.yml:117-119 has no env: block:

container:
  options: -u root --security-opt seccomp=unconfined --shm-size 16g
  image: ubuntu:24.04

The resulting docker create in the log has no --gpus, no --runtime=nvidia, and docker inspect shows the container inherited exactly four variables: HOME, GITHUB_ACTIONS, CI, PATH.

This workflow is the only GPU container in the repo that omits it:

Workflow	Sets NVIDIA_VISIBLE_DEVICES	GPU step
test-wheel-linux.yml:129	yes	passes
coverage.yml:53	yes	passes
ci-pixi-source-test.yml	no	exit 127

Fix

    container:
      options: -u root --security-opt seccomp=unconfined --shm-size 16g
      image: ubuntu:24.04
      env:
        NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}

Then trigger one workflow_dispatch run to confirm, rather than waiting on the nightly.

There is an open PR touching this exact file right now — #2721 — but it only changes the CPU build-smoke job, so it does not fix this. It may be a convenient place to fold the two lines in.

Why this went unnoticed for 2.5 months

The job and its Ensure GPU is working step were both introduced by 774e988bb17 (#2185, 2026-06-09). The job carries:

if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'

so it is gated off on pull requests. The PR that added it structurally could not run it. It merged green because it never executed, and nothing has exercised it since except the nightly, where it has failed every time.

Note that "PR-triggered runs of this workflow succeed" is not evidence the job works — the GPU job is skipped entirely on PRs.

This is worth a process change beyond the fix: requiring one workflow_dispatch run before merging any schedule-only job would have caught this on day one. Captured in #2722.

Severity

Not release-blocking — wheel and source-build correctness are covered elsewhere. The cost is that pixi run test from a source build has never once been validated on a GPU, so the job's entire purpose is unmet.