Switch Frontier CI from the revoked hackathon QOS to normal

[sbryngelson]

Description

Frontier CI is currently failing on every job. The hackathon QOS that .github/scripts/submit-slurm-job.sh submits under was a temporary grant on CFD154, and that grant has ended — sbatch now rejects it outright:

sbatch attempt 1 of 3...
sbatch failed: sbatch: error: Batch job submission failed: Invalid qos specification
Non-transient sbatch failure — not retrying.

All 12 Frontier jobs (CCE and AMD, cpu and gpu, including the Case Opt jobs) failed this way in run 32414716605. retry-sbatch correctly classifies this as non-transient, so it fails immediately rather than burning retries.

The hackathon QOS was introduced for Frontier CI in c99ac2d ("Default Frontier templates to hackathon QOS/partition", #1407).

This switches both the frontier and frontier_amd blocks to qos="normal". Nothing else changes — partition (batch), -n 8 / -n 32, 01:59:00 walltimes, and --stepmgr are all untouched, as is Phoenix's embers.

Why normal specifically

It's the only QOS left on this allocation that permits more than one job at a time, which the CI matrix requires:

QOS	concurrency limit
normal	none (association caps submissions at 100)
debug	MaxSubmitPU=1 — one job in the queue, period
develop	MaxJobsPU=1, plus node=30 and a hard 2h MaxWall
jupyter	MaxJobsPU=1

develop was the pre-#1407 setting, but with MaxJobsPU=1 it would serialize the matrix one job at a time, and its 2h MaxWall leaves no headroom over our 01:59:00.

Known regression

Turnaround gets worse. Measured 1-node queue waits on Frontier over the three days before the grant lapsed:

QOS	n	median	p90
hackathon	989	0.53 min	1.2 min
normal	9602	5.1 min	5.2 h

hackathon gave CI a tight sub-2-minute start. normal has a 5-minute median but a long tail, and each run waits on the slowest of ~12 jobs. This unblocks the pipeline; restoring a priority QOS for CI is the real fix and needs an OLCF request against CFD154.

Type of change

• Bug fix

Testing

Validated the exact job shapes this script generates against the live Frontier scheduler with sbatch --test-only:

-A CFD154 -p batch -q normal -N1 -n8  -t 01:59:00 --stepmgr  → accepted
-A CFD154 -p batch -q normal -N1 -n32 -t 01:59:00 --stepmgr  → accepted

Both are rejected with Invalid qos specification under -q hackathon, confirming the diagnosis and the fix. Also submitted real 1-node jobs under -q normal to confirm concurrency: four submitted simultaneously all went PENDING → RUNNING in 23 s on four separate nodes, with no per-user limit rejections.

Full CI on this PR is the real check.

Checklist

• I added or updated tests for new behavior
• I updated documentation if user-facing behavior changed

CI-only configuration change; no test or documentation surface.

[Copilot]

Pull request overview

Updates the Frontier CI SLURM submission defaults to use a valid QOS now that the temporary hackathon QOS grant for allocation CFD154 has ended, unblocking Frontier-based GitHub Actions runs.

Changes:

• Switches Frontier (frontier) default QOS from hackathon to normal, with an inline rationale comment.
• Switches Frontier AMD (frontier_amd) default QOS from hackathon to normal.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.61%. Comparing base (21a4ebe) to head (f7c2286).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1746   +/-   ##
=======================================
  Coverage   61.61%   61.61%           
=======================================
  Files          84       84           
  Lines       21520    21520           
  Branches     3180     3180           
=======================================
  Hits        13259    13259           
  Misses       6082     6082           
  Partials     2179     2179           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.