Seal regradable qualification bundles

[vriesd]

Summary

• retain normalized grader inputs, every attempt ledger and redacted transcript, expected provenance, exact artifact and canary, decision, policy, and the complete grader source closure
• rederive the complete product outcome, actors, instructions, host policy, usage, completion totals, and per-attempt independence before sealing
• publish content-addressed bundle objects with a manifest written last, immutable replay, crash recovery, stable path identities, strict role closure, and bounded secret scanning including tar members
• replace automatic digest-only decisions with a complete campaign artifact pending the exact canary

Verification

• bun run check: 599 passed, 1 intentional skip
• bun run replay: 13 of 13 cassettes reproduced
• pinned OpenCode 1.18.6 live smoke: passed
• real positive qualifier CLI: packed plugin, 8 passing replays expanded to the canonical 76 cells, evidence-derived canary, 12 fixed roles, 76 attempt/transcript pairs, complete source closure
• three-model interrogate plus comment/deslop review: green
• contribution commit and push preflights: passed

Review guide

Start with evals/qualification-bundle.ts for the seal and filesystem threat model, evals/grader-input.ts and evals/conformance-evidence.ts for regrading authority, then scripts/qualify-release.ts for cross-object binding. The full path is exercised by tests/qualification-cli.test.ts.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3298e60f17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve numeric output-token usage during redaction

Every successfully collected attempt puts outputTokens inside the value passed to redactTranscript, but redactSensitiveFields in evals/provenance.ts treats any key containing token as sensitive and rewrites this number to "[redacted]". persistV2Attempt then parses the transcript with RetainedScenarioEvidenceSchema, which requires usage.outputTokens to be a nonnegative integer, so the first live attempt fails persistence and aborts the campaign instead of producing a report.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Handle diagnostic transcripts before parsing retained evidence

When host setup, outcome collection, or another operation reaches runAttempt's catch block, that block deliberately creates a diagnostic transcript shaped like { evaluatorError: ... }; nevertheless, persistV2Attempt unconditionally parses every transcript as RetainedScenarioEvidence. In those error scenarios this parse throws inside persistEvaluation, converting the original typed attempt failure into a persistence stop and preventing the intended failure attempt and finalized diagnostic report from being written.

Useful? React with 👍 / 👎.