Preserve eval failure origins

.agents/plans/03-assurance-hardening/decisions.tsv

@@ -23,3 +23,11 @@ ts	phase	decision	why	evidence	result
 2026-08-28T01:50:22Z	phase-3	kept zero-inode files ineligible	the same path and opened handle cannot be proven identical without a usable filesystem identity; accepting metadata similarity would weaken the phase guarantee	architecture judge; three reviewers; Node filesystem contract	fail closed, with macOS and Windows filesystem suites added to CI
 2026-08-28T01:56:55Z	phase-3	built deterministic filesystem race checkpoints	static symlink tests did not execute the replacement and mutation branches the assurance claim depends on	tests/test-results.test.ts	leaf replacement before and after open, parent replacement, growth, and truncation all fail closed
 2026-08-28T01:56:55Z	phase-3	completed filesystem report assurance	the final diff passed deslop, comment review, three-model interrogate, full product verification, and cassette replay	bun run check; bun run replay	572 pass, 1 intentional skip, 13 of 13 replays, 892 source bytes headroom
+2026-08-28T02:12:45Z	phase-3	merged stable report assurance	the exact head reproduced both parent regressions and passed independent shipping verification plus Windows and macOS CI	PR 50; merge 51ea890	merged to main
+2026-08-28T02:12:45Z	phase-4	selected narrow runtime, evaluator, and ledger boundaries	the independent judge chose explicit provider and host values, hard evaluator failures, and campaign-stopping persistence failures without synthetic attempts	three architecture candidates; independent phase2_ship_verify judge	candidate A base with B discriminated result and C integrity semantics
+2026-08-28T02:12:45Z	phase-4	captured failure-origin regressions before implementation	the runner and qualifier must not hide grader or store failures as environment gaps	four focused test files	four failures and one missing-module error as expected
+2026-08-28T02:50:17Z	phase-4	made failure origin structural across every eval runner	optional environment and error flags let one broad catch rewrite grader, provider, host, and store failures as host gaps	evals/failure-origin.ts; evals/run.ts; evals/benchmark-run.ts; evals/reviewer-run.ts	provider errors come from assistant error events, host operations are tagged, unknown processing defaults evaluator, store errors stop campaigns
+2026-08-28T02:50:17Z	phase-4	made evaluator integrity failures hard and non-retryable	an evaluator crash cannot become a smaller sample or spend reserve attempts	evals/analysis.ts; evals/harness.ts; focused failure and scheduler tests	NOT VERIFIED for required evaluator failures; paid queues drain in-flight work and start nothing new
+2026-08-28T02:50:17Z	phase-4	kept persistence outside durable attempt evidence	a store that cannot publish the attempt cannot truthfully publish a second attempt claiming the store failed	evals/report.ts; persistEvaluation; preservePrimaryFailure	persistence is a campaign stop only; primary errors retain cleanup diagnostics; fabricated persistence attempts are schema-invalid
+2026-08-28T02:50:17Z	phase-4	completed failure-origin verification	the corrected design passed deslop, comment review, three-model interrogation, full product verification, and replay	bun run check; bun run replay	588 pass, 1 intentional skip, 13 of 13 replays, 892 source bytes headroom
+2026-08-28T02:50:59Z	phase-4	closed final concurrent and provenance review findings	last-pass reviewers found a lost in-flight persistence error, silent host reads, one extra paired arm, and provider fidelity still labelled host	evals/harness.ts; evals/benchmark-run.ts; evals/cassette.ts; focused concurrency and cassette tests	589 pass, 1 intentional skip, 13 of 13 replays, no remaining Sol or 5.4 findings

docs/release-qualification.md

@@ -33,10 +33,9 @@ forces a decision about what its result is allowed to mean. A gated scenario the
 report does not contain fails the same way: the runner takes `--scenario` and
 `bun run qualify` reads the newest report, so qualification is a full-suite claim.
 
-An excluded attempt is not a smaller sample but a missing one: the runner drops one
-that aborted mid-flight, or asked where the scenario does not allow it, so a gated
-pair below the floor — or holding any abort — means re-running it, not reading the
-remainder as its rate.
+A non-product attempt never shrinks the required sample. Provider or host failure,
+or an unallowed ask, leaves an evidence gap. Evaluator failure is `NOT VERIFIED`;
+persistence failure stops without a finalized report. Re-run only external gaps.
 
 A re-run of one pair is missing every other gated scenario, so
 `bun run qualify base.json rerun.json` takes the pairs the later report measured and

evals/analysis.ts

@@ -79,6 +79,7 @@ export type DecisionReason = {
 		| "false-completion"
 		| "unsubmitted-review"
 		| "below-pass-rate"
+		| "campaign-integrity-failure"
 		| "campaign-stopped"
 		| "missing-attempt"
 		| "unscored-attempt"
@@ -479,6 +480,17 @@ export function deriveReleaseDecision(input: {
 			)
 			.map((cell) => cell.cellId),
 	);
+	if (
+		report.completion.status === "stopped" &&
+		report.completion.cause === "persistence"
+	) {
+		decisionReason(
+			reasons,
+			"hard",
+			"campaign-integrity-failure",
+			`Campaign stopped after a ${report.completion.cause} failure.`,
+		);
+	}
 	if (
 		promotionArtifact &&
 		!samePackedArtifact(expected.artifact, promotionArtifact)
@@ -508,6 +520,19 @@ export function deriveReleaseDecision(input: {
 		if (!requiredKeys.has(`${attempt.caseId}\u0000${attempt.caseVersion}`)) {
 			continue;
 		}
+		if (
+			attempt.outcome.kind === "failure" &&
+			attempt.outcome.origin === "evaluator"
+		) {
+			decisionReason(
+				reasons,
+				"hard",
+				"campaign-integrity-failure",
+				`Attempt ${attempt.attemptId} failed in ${attempt.outcome.origin} code.`,
+				attempt.caseId,
+				attempt.caseVersion,
+			);
+		}
 		if (attempt.outcome.kind !== "product") continue;
 		const evidence = attempt.outcome.evidence;
 		if ("falseCompletion" in evidence && evidence.falseCompletion) {