CI: exclude bad Phoenix L40S node atl1-1-03-007-31-0 (uncorrectable ECC)

[sbryngelson]

What

Adds atl1-1-03-007-31-0 to the Phoenix GPU --exclude list. Net exclude is now atl1-1-03-007-29-0,atl1-1-03-007-31-0.

Why

A second L40S node in the same chassis as the already-excluded 007-29-0 (both atl1-1-03-007-*) is failing every GPU job with the same hardware fault:

Accelerator Fatal Error: call to cuCtxCreate returned error 214
  (CUDA_ERROR_ECC_UNCORRECTABLE): uncorrectable ECC error encountered

Confirmed across a CI sweep — 007-31-0 is the ECC source in the most recent master run (32601124944, 105 ECC hits) and a PR run, exactly as 007-29-0 was in the prior batch. Reruns don't help: SLURM has no GPU-health awareness and keeps rescheduling onto the same bad node.

Note: the PMIX ERROR: PMIX_ERR_NO_PERMISSIONS ... dstore_base.c lines in these logs are benign (they appear on passing syscheck/pre_process steps too, from the hpcx/2.19-cuda PMIx dstore); they are not the failure cause. The failure is the ECC fault above.

Both bad GPUs are being reported to PACE for repair/drain; this exclude is the CI unblock in the meantime.

[Copilot]

Pull request overview

This PR updates the Phoenix GPU CI SLURM submission script to avoid scheduling jobs onto a known-bad L40S node that is consistently failing GPU runs with uncorrectable ECC errors, helping keep the GPU CI lane reliable until the hardware is drained/repaired.

Changes:

• Extend the Phoenix GPU #SBATCH --exclude list to include atl1-1-03-007-31-0.
• Keep the existing exclusion of atl1-1-03-007-29-0, resulting in a two-node exclusion list for Phoenix GPU jobs.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.61%. Comparing base (d2ed786) to head (8391edd).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1750   +/-   ##
=======================================
  Coverage   61.61%   61.61%           
=======================================
  Files          84       84           
  Lines       21520    21520           
  Branches     3180     3180           
=======================================
  Hits        13259    13259           
  Misses       6082     6082           
  Partials     2179     2179           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.