feat(*): add trajectory regression scaffold and CI gate

[forrestjgq]

Summary

Turn "bug fix to regression case" from a manual chore into a gated pipeline, and give trajectory regressions their own CI gate.

• raven trajectory replay --json: stable machine-readable report (schema_version 1) with defensive JSON coercion; on a halted replay the full JSON is still emitted before exit code 2.
• Case metadata contract: case.yaml requires non-blank issue/owner/why/re_record; residual-scan findings pass only with an explicit per-token human sign-off (reviewed_residuals: full-token sha256 plus a verbatim reason, nothing exempted automatically -- a machine-produced redaction report is not approval).
• raven trajectory regression validate [CASE_DIR|--all]: static commit gate -- whole-directory discovery so a broken case cannot hide, both schemas, cassette completeness down to the replay contract (valid JSON is not a usable payload: missing payloads via the extracted replayability check, corrupt inner fields via validate_recording), rejection of files the residual scanner would silently skip, and a 256 KiB / 1 MiB size budget; every parse boundary collects problems instead of raising, so one malformed case cannot abort the --all sweep.
• raven trajectory regression init <source> --name <case>: scaffold from a bundle directory, attempt id, trajectory report tarball, or bug report package (two-layer extraction; both passes accept only regular files/directories with root-contained names). Minimize lands in staging; residual findings are reviewed interactively (the full token is shown locally, files receive digest and reason); publication copies onto the destination filesystem under a .init- staging name, re-checks the case name, then renames on one device -- failures clean up only what the run created.
• Vacuous-pass guard (MIN_COMMITTED_CASES, raised by hand) and a trajectory CI job: replay the committed cases, run validate --all even after a pytest failure so both gates report, upload per-case failure reports (replay --json schema plus check failures, mode from each case's expect.yaml) as a 14-day artifact.
• Docs: the Trajectory Regression Case entry in CONTEXT.md extended in place; tests/trajectories/README.md documents the per-case layout and workflow.

Also: config_secrets_loaded (a field name of redaction.json itself, a guaranteed self-referential false positive in every cassette) joins the residual scanner's benign literals, and both sample cases gain real case.yaml metadata.

Type

• Fix
• Feature
• Docs
• CI / tooling
• Refactor
• Other

Verification

• uv run pytest tests/ -q (after rebasing onto latest main): 23149 passed; the 37 failures and 20 errors are environment-bound and reproduced identically on a pristine origin/main worktree without these changes.

• uv run pytest tests/test_trajectory_regressions.py tests/test_cli_trajectory_regression_commands.py tests/test_cli_trajectory_commands.py tests/test_trajectory_replay.py tests/test_trajectory_cassette.py tests/test_trajectory_redact.py -q: 282 passed (about 90 new tests across the five steps).

• uv run ruff check ., uv run ruff format --check ., make lint-imports, make lint-deps, make check-large-files, make check-source-language: pass. make lint-types reports one diagnostic in raven/agent/tools/web.py, a file this branch does not touch, pre-existing on main.

• CI workflow YAML parses (yaml.safe_load); the first real run of the new trajectory job happens on this PR (see this PR's checks).

• CLI assertion width-independence re-verified with COLUMNS=60.

• Relevant tests pass locally

• Relevant lint / type checks pass locally

• User-facing docs or screenshots are updated when needed

Risk

• All additive: the default replay output path is unchanged; regression init/validate are new subcommands; the one scanner change adds a fixed benign literal (any variation still flags).

• The new trajectory CI job becomes a required-passing gate for PRs; to lift it in an emergency, revert the CI commit (ci: add trajectory regression gate job and case-count guard) -- the remaining commits are independent.

• Tar extraction accepts only regular files and directories with root-contained relative names on both layers, with filter="data" kept as defense in depth.

• Security impact considered

• Backward compatibility considered

• Rollback path is clear for risky changes

Related Issues

N/A

[gloryfromca]

Blocking: the static validator lets malformed tool inputs disable the replay comparison.

I reviewed the full diff and covered the repository rules and canonical trajectory terms, the replay/cassette/CLI callers and branch history, backward compatibility, test-gate integrity, and the relevant architecture boundaries. The default replay path remains compatible and I found no evidence of weakened tests, but the inline false-green must be fixed before this gate can provide the contract it claims.

Verification: uv run pytest tests/test_trajectory_regressions.py tests/test_cli_trajectory_regression_commands.py tests/test_cli_trajectory_commands.py tests/test_trajectory_replay.py -q passed (244 tests); raven trajectory regression validate --all passed both committed cases; the source-language and large-file checks passed when run directly. I also reproduced the inline defect against a temporary copy of the committed tool-calling case.

[gloryfromca]

Blocking: require recorded LLM input fields before accepting a case.

The original tool-input blocker is fixed and its thread is resolved. I reviewed the current full diff; range-diff confirms the eight previously reviewed commits are patch-identical, then I inspected the new contract check, tests, and formatting commit. I covered AGENTS/CLAUDE and canonical trajectory terms, the replay/cassette/CLI callers and history, backward compatibility, test-gate integrity, and the relevant architecture boundaries. No tests were weakened, but the parallel LLM-input false-green inline still prevents the static gate from enforcing its stated contract.

Verification: uv run pytest tests/test_trajectory_regressions.py tests/test_cli_trajectory_regression_commands.py tests/test_cli_trajectory_commands.py tests/test_trajectory_replay.py tests/test_trajectory_cassette.py tests/test_trajectory_redact.py -q passed (286 tests); raven trajectory regression validate --all passed both committed cases; source-language, large-file, and diff checks passed. I reproduced both missing-field variants against temporary copies of the committed tool-calling case.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

The LLM-input contract fix rejects missing model, messages, and tools fields and a blank model, closing both zero-divergence reproductions. Range-diff confirms the previously reviewed series is patch-identical apart from the newer main base and this fix. I covered the repository rules and canonical trajectory terms, the full PR diff and new commit, replay/cassette/CLI callers and history, backward compatibility, test-gate integrity, and architecture boundaries; I found no weakened tests or remaining plain error. Both threads I opened are resolved.

Verification: the six relevant trajectory test files pass (290 tests); all committed cases pass raven trajectory regression validate --all; direct mutation checks reject each missing LLM field; source-language, large-file, and diff checks pass.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

This head is tree-identical to the prior clean revision (7f4891308a80feb42e99cd04fab16a2bd81993d3). Range-diff confirms every code and test patch is unchanged; only the formatting commit type moved from style to the repository-approved chore. The prior review coverage therefore still applies: repository rules and canonical terms, full diff, callers and history, backward compatibility, test integrity, and architecture boundaries. Both threads I opened remain resolved.

Verification: exact tree equality against 291e0c339763; patch series identical apart from commit metadata; git diff --check github/main...HEAD passes. The unchanged tree previously passed the six relevant trajectory test files (290 tests), committed-case validation, source-language, and large-file gates.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

This revision is tree-identical to the prior clean head (7f4891308a80feb42e99cd04fab16a2bd81993d3). Range-diff shows only a commit-body punctuation correction from a non-ASCII em dash to ASCII --; every code and test patch is unchanged. That correction satisfies the repository's ASCII-only commit-message rule. The prior coverage of repository rules, canonical terms, full diff, callers/history, backward compatibility, test integrity, and architecture therefore remains applicable, and both threads I opened remain resolved.

Verification: exact tree equality against 7401501d36f9; patch series identical apart from commit metadata; git diff --check github/main...HEAD passes. The unchanged tree previously passed 290 relevant tests and all case/repository gates.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

This head merges the current target branch. Range comparison shows all 11 feature patches are unchanged from the previously reviewed revision, and the current PR diff remains limited to the same 15 trajectory/CI files, so the merge introduces no new feature behavior to reassess.

Coverage this round included the repository rules and canonical terms, the full PR diff and revision history, callers/contracts, backward compatibility, test integrity, and the prior architecture conclusions through the unchanged-patch comparison. Both previously opened findings remain resolved.

Verification: the four affected trajectory test modules pass (252 passed); raven trajectory regression validate --all reports 2 cases and 0 problems; the source-language gate, large-file gate, and git diff --check pass. An initial invocation used stale singular test filenames and collected no tests; it was replaced by the successful run against the actual files above.

[0xKT]

**Not a blocker -- and the thing I most wanted to know came back right: this gate can fail, and it fails for the right reason. Three things stand, all of the same family: one rule with more than one reader.

The gate is not a gate that can only pass

I asked for this specifically, because a regression gate that never goes red is worse than no gate -- it buys confidence without paying for it. Two real regressions were built in production code and both were caught, with the replay naming the divergence:

replay halted: tool call #1: tool params -
  expected '{"note": "hi"}', got '{"injected_by_mutant": 1, "note": "hi"}'
replay halted: llm call #1: messages[1] -
  expected '...create the marker file", "role": "user"}', got '...create the marker file [M2-suffix]", "role": "user"}'

and a deliberately equivalent mutant (content.rstrip("."), on samples whose text ends in no period) correctly survived at "86 passed" -- the control that says the two kills were not a no-op edit passing for a catch. On the PR's own decision surface, 23 withdrawal mutants, 20 killed, and the three survivors each accounted for (one proven equivalent by a control that removes both escape checks and does go red; two byte-identical at base).

1. The .init-* rule is taught to one of three readers of the same directory

regression.py:418-421 introduces it, with a comment:

# ".init-*" is this feature's own publish-staging pattern (skipped so
# an interrupted init cannot read as a broken case); any other dot
# directory is still discovered - the gate stays whole-directory.
if path.is_dir() and not path.name.startswith(".init-")

The other two readers of that same root, both on CI paths this PR adds, do not know it:

# tests/test_trajectory_regressions.py:330  (ci.yml:261 parametrizes the replay over this)
return sorted(p for p in CASES_ROOT.iterdir() if (p / "expect.yaml").is_file())
# tests/test_trajectory_regressions.py:341  (the MIN_COMMITTED_CASES vacuity floor)
count = sum(1 for p in root.iterdir() if (p / "expect.yaml").is_file())

So a .init-* directory is replayed by ci.yml:261 and never validated by ci.yml:265 -- exempt from the residual scan, the schema checks, the cassette-completeness checks and the size budget, while the green pytest run makes it look covered. And it satisfies the floor that CONTEXT.md:1792-1794, added by this PR, says "keeps the suite from passing vacuously".

Measured, with the control the claim needs -- two byte-identical case directories carrying the same planted credential, differing only in name:

name	validate --all
control_plain_name	flags the planted token, 1 case, 1 problem, exit 1
.init-staged-abc123 (same bytes)	not among the cases at all

and on the real tree, adding one .init-leftover-* copy: validate --all -> "2 case(s), 0 problem(s)" exit 0, while the pytest gate goes 86 -> 87 passed -- the extra parametrized case is the directory validate refuses to look at.

Not a blocker because the directory does not appear on its own: trajectory_regression_commands.py:234-237 removes it on every ordinary error path, so reaching this needs a SIGKILLed init plus a git add of the cases root. The single-owner fix is for the test file to call discover_case_dirs.

2. The PR writes "nothing is exempted automatically" and adds an automatic exemption in the same change

CONTEXT.md (added here): residual-scan findings carry "a full-token sha256 plus a reason; nothing is exempted automatically".

redact.py:143 (added here):

-_BENIGN_LITERALS = frozenset({"completion_tokens_details", "prompt_tokens_details"})
+_BENIGN_LITERALS = frozenset({"completion_tokens_details", "prompt_tokens_details", "config_secrets_loaded"})

A literal in that set never becomes a finding, so it never reaches the human review the sentence describes.

To be precise about what is new: the mechanism is not -- _BENIGN_LITERALS existed at base with two entries. What is new is the sentence claiming it does not exist, in the document a reviewer consults to decide how much the residual scan can be trusted. The exemption itself looks right (config_secrets_loaded is a redaction-report field name, and every cassette carries one); it is the description that needs the one-line correction.

3. Nit: the manifest summary is written twice, and the test asserts its own copy

_manifest_summary (trajectory_commands.py:573-586) builds the manifest block of replay --json; tests/test_trajectory_regressions.py:357-362 re-implements the same comprehension for the CI failure artifact, hardcoding the same key tuple and the same isinstance(..., (str, int)) filter. test_dump_failure_report_writes_the_artifact_schema then asserts against the test file's own version, so the production one could drift without anything noticing.

Two pre-existing gaps this PR makes load-bearing, recorded rather than filed

Neither is introduced here -- both lines are byte-identical at de3870a5 -- but this PR is what promotes check_report onto a CI gate for the first time, so they matter more now than they did:

• regression.py:290: withdrawing the index half of the first-divergence match ((first.kind, first.index) -> (first.kind,)) leaves the suite fully green. Nothing asserts that an expectation naming llm call #1 is rejected when the replay first diverged at #3.
• regression.py:256: withdrawing the tool name_equals failure arm leaves the suite green -- no test makes a name_equals check fail. The sibling params_equal is covered (its mutant dies), which is the control showing the suite does reach that function, so this is a selection gap rather than unreached code.