Skip to content

fix(optimize): scope transcript-derived findings to the selected provider - #1003

Open
AndrewDongminYoo wants to merge 1 commit into
getagentseal:mainfrom
AndrewDongminYoo:fix/optimize-provider-scope
Open

fix(optimize): scope transcript-derived findings to the selected provider#1003
AndrewDongminYoo wants to merge 1 commit into
getagentseal:mainfrom
AndrewDongminYoo:fix/optimize-provider-scope

Conversation

@AndrewDongminYoo

Copy link
Copy Markdown

Summary

  • scanSessions() hardcoded discoverAllSessions('claude'), so every finding it feeds was computed from Claude transcripts regardless of --provider, while the header came from the already-filtered projects. optimize --provider codex printed a Claude read/edit ratio, Claude ghost skills and Claude MCP findings under a Codex header.
  • Skip the scan when the filter excludes Claude, and skip the detectors it feeds rather than handing them an empty scan — emptiness reads as "never invoked", so an empty scan reports every skill, agent and command as unused (19 → 38 on my machine).
  • projects-derived findings (MCP tool coverage, capability reliability, low-worth sessions, context bloat, outliers, model recommendations) already filter correctly and still run. cacheKey now carries the provider, since the provider decides whether the scan runs at all.

Fixes #1002

Testing

  • I have tested this locally against real data (not just unit tests)
  • npm test passes
  • npm run build succeeds

npm test: 199 files, 2,631 passed, 5 skipped, 0 failed. Two new cases in tests/optimize-fs.test.ts cover the positive (--provider claude still reports read-edit-ratio) and negative (--provider codex reports neither read-edit-ratio nor unused-skills) paths. I verified the negative case fails without the fix:

AssertionError: expected [ 'read-edit-ratio', …(2) ] to not include 'read-edit-ratio'
 ❯ tests/optimize-fs.test.ts:413

Against real data (30-day window, Claude + Codex + Copilot + Antigravity + Gemini + Warp on disk):

BEFORE
  --provider codex    347 sessions   75,929 calls   Health: F (20/100, 14 issues)
                      ─── 1. Claude edits more than it reads (16,368 reads / 10,510 edits)
                      ─── 10. 19 skills you never use

AFTER
  --provider codex    347 sessions   75,929 calls   Health: D (48/100, 4 issues)
                      ─── 1. 6 MCP servers correlate with retry-heavy edits
                      ─── 2. 79 possibly low-worth expensive sessions
                      ─── 3. 62 context-heavy sessions
                      ─── 4. 2 high-cost session outliers

  --provider claude   2,051 sessions   57,812 calls   Health: F (20/100, 14 issues)   unchanged
  (no flag)           2,409 sessions  134,068 calls   Health: F (20/100, 14 issues)   unchanged

The four findings that survive under --provider codex are genuinely Codex-derived; the Claude-only ones are gone. The default and --provider claude paths are byte-for-byte unchanged in finding count and health grade.

…ider

scanSessions() always ran discoverAllSessions('claude'), so every finding it
feeds was computed from Claude transcripts regardless of --provider, while the
header (sessions, calls, cost) came from the already-filtered projects. Under
--provider codex the two described different providers, and the Claude-derived
numbers read as the selected provider's.

Skip the scan when the filter excludes Claude, and skip the detectors it feeds
rather than handing them an empty scan: emptiness reads as "never invoked", so
an empty scan turned every skill, agent and command into a reported ghost.
Findings derived from projects (MCP tool coverage, capability reliability,
low-worth sessions, context bloat, outliers, model recommendations) already
filter correctly and still run.

The result cache key now carries the provider, since the provider decides
whether the scan runs at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

optimize --provider <non-claude> reports findings computed from Claude sessions

1 participant