fix: reject the tts-mcp#13 conventional-prefix golden entry - #125
Merged
Conversation
Promoted 2026-08-18 on three-model agreement (m3+m2.7+deepseek). That reasoning was wrong: tts-mcp declares no conventional-prefix requirement (absent from CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0 CHANGELOG sections use the same prose-bullet style. The finding is a correct observation that names no defect, which the golden set's rule rejects. The three models share a conventional-commit training prior, so their agreement amplified a shared bias instead of triangulating truth. Model agreement is not independent evidence when the models share priors; verification has to run against the repository, not a vote. Every score computed against golden-curated-3 credited models for reproducing a non-defect and penalised them for missing it. As rejected, the entry works as a precision probe instead. golden-curated-3 -> golden-curated-4; 158 entries unchanged, 138/20 -> 137/21.
There was a problem hiding this comment.
Both plan concerns are confirmed as correctly addressed:
- Entry 11 (tts-mcp#13) —
stateflipped from"accepted"→"rejected". Version bumpedgolden-curated-3→golden-curated-4. Rejected count moved 20 → 21 ✓ - Adjudications — New entry appended at index 2067 (2026-08-26 CORRECTION…) is a single new string, not a duplicate ✓
- CHANGELOG.md — Single
## Unreleasedbullet withfix:prefix, properly formatted ✓
No code files changed. No rule violations applicable. No concerns remaining.
{
"verdict": "approve",
"summary": "Clean precision correction: a single golden-set entry (tts-mcp#13) is flipped accepted→rejected with correct version bump, adjudication note, and CHANGELOG entry. 21 rejected entries confirmed (20→21 delta). No code, no violations.",
"comments": [],
"concerns_addressed": [
"correctness: entry 11 flipped accepted->rejected, version golden-curated-3->golden-curated-4, rejected count 20->21 — verified",
"correctness: adjudations array has one new entry (index 2067) with no duplication — verified"
]
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Flips the
tts-mcp#13conventional-prefix entry fromacceptedtorejectedinbench/golden.json, and bumpsgolden-curated-3→golden-curated-4. 158 entries unchanged; the accepted/rejected split moves 138/20 → 137/21.Why
The entry was promoted on 2026-08-18 because three models agreed on it (m3 + m2.7 + deepseek). Verification against the PR head
59f9fe8shows the finding is not a defect:tts-mcpdeclares no conventional-prefix requirement — the word appears in neitherCLAUDE.mdnorAGENTS.mdv0.2.0andv0.1.0CHANGELOG sections use the identical prose-bullet styleSo it is a correct observation that names no defect — precisely the class the golden set's own rule rejects.
The reasoning error worth keeping
Three-model agreement was treated as triangulation. It wasn't: m3, m2.7 and deepseek share a conventional-commit training prior, so agreement amplified a shared bias rather than confirming a defect.
Model agreement is not independent evidence when the models share priors. Verification has to run against the repository, not a vote. This is recorded in
adjudicationsso the correction sits next to the note that made the original claim.Impact
Every score computed against
golden-curated-3credited models for reproducing a non-defect and penalised them for missing it. As arejectedentry it now functions as a precision probe — a model that flags it correctly loses precision.Verification
make bench-test— 229 tests,OK (skipped=1)bench/golden.jsonparses; entry 157 isrejected; counts confirmed 137 accepted / 21 rejected