Skip to content

feat(vis): add loop trace evaluation metrics - #2897

Open
yu-xin-c wants to merge 1 commit into
MoonshotAI:mainfrom
yu-xin-c:codex/trace-eval-infrastructure
Open

feat(vis): add loop trace evaluation metrics#2897
yu-xin-c wants to merge 1 commit into
MoonshotAI:mainfrom
yu-xin-c:codex/trace-eval-infrastructure

Conversation

@yu-xin-c

Copy link
Copy Markdown

Related Issue

Related to #2622

Problem

The visualizer can reconstruct turns, steps, tools, and context fill, but it cannot quantify the two loop shapes reported in the linked issue: long exact runs and rotating reuse of a small call alphabet. This makes candidate breaker policies difficult to calibrate against healthy traces before changing runtime behavior.

What changed

  • Add a deterministic, client-side loop trace evaluator that segments prompt-to-prompt phases and canonicalizes tool arguments.
  • Report phase-local exact-call repetition, longest immediate runs, peak repetition in a fixed-size rolling window, cancel/steer/compaction markers, and before/after steer histogram overlap.
  • Keep prompt text, tool arguments, tool outputs, and internal fingerprints out of the returned report.
  • Surface the metrics in the Timeline tab and allow copying the aggregate versioned JSON report.
  • Add synthetic regression traces for long exact runs, rotating small alphabets, healthy argument drift, steer response, phase boundaries, markers, and payload exclusion.

This PR is diagnostics-only. It does not classify a session as looping, choose alert thresholds, or change agent/runtime behavior.

Verification

  • pnpm --filter @moonshot-ai/vis-web test
  • pnpm --filter @moonshot-ai/vis-web typecheck
  • pnpm --filter @moonshot-ai/vis-web build
  • pnpm typecheck
  • pnpm exec oxlint --type-aware --quiet

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked a related issue, or explained the problem above.
  • I have added tests that prove my feature works.
  • Ran gen-changesets skill; vis packages are explicitly ignored and need no changeset.
  • Ran gen-docs review; this self-contained debug UI addition needs no docs update.

@changeset-bot

changeset-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 73c6063

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@yu-xin-c
yu-xin-c force-pushed the codex/trace-eval-infrastructure branch from 2a89721 to 73c6063 Compare August 14, 2026 15:42
@yu-xin-c
yu-xin-c marked this pull request as ready for review August 14, 2026 15:43

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 73c60638a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +172 to +175
if (record.type === 'turn.prompt') {
current.nextPromptLineNo = entry.lineNo;
mutablePhases.push(current);
current = createPhase(mutablePhases.length, entry.lineNo);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not split phases on rejected prompt records

In v1 wires, a prompt submitted while a turn is active is persisted before TurnFlow.launch rejects it with turn.agent_busy (packages/agent-core/src/agent/turn/index.ts:172-177,205-215); the existing regression trace at packages/agent-core/test/agent/turn.test.ts:2793-2803 confirms that the old turn's tool calls can follow that rejected prompt record. Unconditionally starting a phase here therefore resets exact-run and repetition-window metrics even though the agent never received a new prompt, artificially lowering loop measurements. Correlate boundaries with an actually started/new turnId, rather than treating every recorded prompt attempt as accepted.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant