Skip to content

Qualify 8.1.3 with hardened release evals - #58

Open
vriesd wants to merge 12 commits into
mainfrom
codex/assurance-hardening-release-8.1.3
Open

Qualify 8.1.3 with hardened release evals#58
vriesd wants to merge 12 commits into
mainfrom
codex/assurance-hardening-release-8.1.3

Conversation

@vriesd

@vriesd vriesd commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Prepares 8.1.3 and hardens release qualification around canonical release sampling, retained failure evidence, exact regrading, and bounded environment reserves.
  • Changes the release gate to 76 primary cells plus 16 predeclared environment reserves, so one independently reproduced host or provider failure per provider/case stratum can be replaced without discarding the failed attempt.
  • Tightens eval prompts and canary/release docs so named assertions must bind to machine-readable JUnit evidence.

Verification

  • bun test tests/grader-input.test.ts tests/eval-reporting.test.ts tests/environment-reserves.test.ts
  • bun run typecheck
  • bun run check
  • .agents/skills/flow-contribution-check/scripts/preflight.sh push

Review Notes

OpenCode reviewer attempts with opencode/grok-4.6 and opencode/gpt-5.6-sol were attempted from a detached temporary worktree at ccc10c1 with a temporary XDG_CONFIG_HOME. Both failed before review with an upstream OpenCode gateway 500 on even a one-line prompt. No source files were edited by those reviewer attempts.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ccc10c1ef6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/harness.ts
const exited = new Promise<void>((resolve) =>
child.once("exit", () => resolve()),
);
signalProcessTree(child, "SIGTERM");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reap descendants before terminating the wrapper

On Linux containers where the detached wrapper owns a child process, sending SIGTERM to the entire process group kills the wrapper before it can wait for and reap that child. The child can consequently remain as a zombie, and the later polling and SIGKILL cannot remove it because this process is not its parent; the function then returns after its deadlines with the PID still present. This is reproducible with the newly added bun test tests/eval-reporting.test.ts, whose detached-wrapper test fails at the child-PID assertion, and repeated eval runs can accumulate these entries. Terminate descendants before the wrapper or otherwise arrange for them to be reaped before returning.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants