Preserve eval failure origins

[vriesd]

Summary

• replace optional environment/error flags with typed provider, host, and evaluator attempt failures
• keep persistence outside attempt evidence and stop campaigns without synthetic records
• make required evaluator failures hard NOT VERIFIED reasons
• stop new paid work after evaluator integrity failures while draining in-flight attempts
• apply the same boundary model to conformance, paired benchmark, and reviewer eval runners
• retain distinct provider, host, evaluator, and persistence diagnostics in reports and cassettes

Verification

• bun run check: 589 pass, 1 intentional live skip
• bun run replay: 13/13 matched
• focused failure, scheduler, qualifier, persistence, benchmark, reviewer, and cassette tests green
• three-model interrogate, deslop, and comment review: no remaining findings
• source ceiling unchanged at 892 bytes headroom

Review guide

The large line count is mostly the three runner bodies moving under the same cleanup-preserving boundary. Review evals/failure-origin.ts first, then the schema and qualifier changes, then each runner adapter. Persistence failures remain campaign-level only and are schema-invalid as attempt outcomes.

[vriesd]

PASS+NOTES — independently verified exact PR head 3c112728ae15a4c0f0e51602debda17051818d6e (parent/base 51ea8903b6366ca0d64e792bf5556f8c641a9be7) in isolated temporary worktrees; no merge performed.

• Parent regression confirmed by source inspection: scenario.check(outcome) was inside the broad attempt catch that converted throws to host/environment failures; the new failure-origin suite was absent on the parent.
• Head frozen install and focused failure-origin/scheduler/qualifier/schema/persistence/report-store tests: 137 pass, 0 fail.
• bun run check: 589 pass, 1 intentional skip, 0 fail; bun run replay: 13/13 cassettes matched.
• Budgets: source 254,084 / 254,976 bytes (892 headroom); maintained prose 90,855 / 91,000 (145 headroom); decision records 47,911 / 48,000 (89 headroom).
• Exact-head GitHub CI passed: check, dependency-audit, lint-workflows, live-smoke, platform-persistence (macos-latest), and platform-persistence (windows-latest).
• Push preflight passed, including runtime/scheduler (78 pass), OpenCode surface (40 pass), and packed surface (1 pass).
• Diff review confirms evaluator failures are structurally distinct and hard qualification reasons; persistence failures stop without synthetic attempts and do not produce a finalized qualifying report.

Note: local optional gitleaks scanning was skipped because gitleaks is not installed; CI and all required deterministic checks passed.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c112728ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Classify the entire host bootstrap as host work

When fixture setup fails before the server starts—for example, mkdtemp, fixture writes, Git initialization, or package-cache copying—the new host boundary has not yet been entered. The raw exception therefore reaches each runner's evaluatorFailure fallback, is recorded as a non-retryable evaluator failure, and stops remaining paid work instead of being treated as a retryable host gap. Move the boundary around the complete bootstrap sequence, not only the port/server startup.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve a later persistence failure

When concurrent workers both throw, only the first exception is retained. In particular, if one attempt first throws during host cleanup and an already in-flight attempt subsequently throws EvaluationPersistenceError while publishing its ledger record, the persistence failure is silently discarded and the campaign reports only the host error. Draining the workers is insufficient here; retain the strongest failure or aggregate all thrown failures so a failed attempt write cannot be hidden.

Useful? React with 👍 / 👎.