Skip to content

fix: reject the tts-mcp#13 conventional-prefix golden entry - #125

Merged
bborbe merged 1 commit into
masterfrom
fix/golden-reject-tts13-prefix
Aug 26, 2026
Merged

fix: reject the tts-mcp#13 conventional-prefix golden entry#125
bborbe merged 1 commit into
masterfrom
fix/golden-reject-tts13-prefix

Conversation

@bborbe

@bborbe bborbe commented Aug 26, 2026

Copy link
Copy Markdown
Owner

What

Flips the tts-mcp#13 conventional-prefix entry from accepted to rejected in bench/golden.json, and bumps golden-curated-3golden-curated-4. 158 entries unchanged; the accepted/rejected split moves 138/20 → 137/21.

Why

The entry was promoted on 2026-08-18 because three models agreed on it (m3 + m2.7 + deepseek). Verification against the PR head 59f9fe8 shows the finding is not a defect:

  • tts-mcp declares no conventional-prefix requirement — the word appears in neither CLAUDE.md nor AGENTS.md
  • The repo's own released v0.2.0 and v0.1.0 CHANGELOG sections use the identical prose-bullet style

So it is a correct observation that names no defect — precisely the class the golden set's own rule rejects.

The reasoning error worth keeping

Three-model agreement was treated as triangulation. It wasn't: m3, m2.7 and deepseek share a conventional-commit training prior, so agreement amplified a shared bias rather than confirming a defect.

Model agreement is not independent evidence when the models share priors. Verification has to run against the repository, not a vote. This is recorded in adjudications so the correction sits next to the note that made the original claim.

Impact

Every score computed against golden-curated-3 credited models for reproducing a non-defect and penalised them for missing it. As a rejected entry it now functions as a precision probe — a model that flags it correctly loses precision.

Verification

  • make bench-test — 229 tests, OK (skipped=1)
  • bench/golden.json parses; entry 157 is rejected; counts confirmed 137 accepted / 21 rejected
  • Diff is 3 semantic lines plus the appended adjudication note — no reformatting

Promoted 2026-08-18 on three-model agreement (m3+m2.7+deepseek). That
reasoning was wrong: tts-mcp declares no conventional-prefix requirement
(absent from CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0
CHANGELOG sections use the same prose-bullet style. The finding is a
correct observation that names no defect, which the golden set's rule
rejects.

The three models share a conventional-commit training prior, so their
agreement amplified a shared bias instead of triangulating truth. Model
agreement is not independent evidence when the models share priors;
verification has to run against the repository, not a vote.

Every score computed against golden-curated-3 credited models for
reproducing a non-defect and penalised them for missing it. As rejected,
the entry works as a precision probe instead.

golden-curated-3 -> golden-curated-4; 158 entries unchanged, 138/20 -> 137/21.

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both plan concerns are confirmed as correctly addressed:

  1. Entry 11 (tts-mcp#13)state flipped from "accepted""rejected". Version bumped golden-curated-3golden-curated-4. Rejected count moved 20 → 21 ✓
  2. Adjudications — New entry appended at index 2067 (2026-08-26 CORRECTION…) is a single new string, not a duplicate ✓
  3. CHANGELOG.md — Single ## Unreleased bullet with fix: prefix, properly formatted ✓

No code files changed. No rule violations applicable. No concerns remaining.

{
  "verdict": "approve",
  "summary": "Clean precision correction: a single golden-set entry (tts-mcp#13) is flipped accepted→rejected with correct version bump, adjudication note, and CHANGELOG entry. 21 rejected entries confirmed (20→21 delta). No code, no violations.",
  "comments": [],
  "concerns_addressed": [
    "correctness: entry 11 flipped accepted->rejected, version golden-curated-3->golden-curated-4, rejected count 20->21 — verified",
    "correctness: adjudations array has one new entry (index 2067) with no duplication — verified"
  ]
}

@bborbe
bborbe merged commit 2934475 into master Aug 26, 2026
1 check passed
@bborbe
bborbe deleted the fix/golden-reject-tts13-prefix branch August 26, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant