From 39b416014ecb24b538509b746870a05d904abfc3 Mon Sep 17 00:00:00 2001 From: Benjamin Borbe Date: Wed, 26 Aug 2026 09:42:05 +0200 Subject: [PATCH] fix: reject the tts-mcp#13 conventional-prefix golden entry Promoted 2026-08-18 on three-model agreement (m3+m2.7+deepseek). That reasoning was wrong: tts-mcp declares no conventional-prefix requirement (absent from CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0 CHANGELOG sections use the same prose-bullet style. The finding is a correct observation that names no defect, which the golden set's rule rejects. The three models share a conventional-commit training prior, so their agreement amplified a shared bias instead of triangulating truth. Model agreement is not independent evidence when the models share priors; verification has to run against the repository, not a vote. Every score computed against golden-curated-3 credited models for reproducing a non-defect and penalised them for missing it. As rejected, the entry works as a precision probe instead. golden-curated-3 -> golden-curated-4; 158 entries unchanged, 138/20 -> 137/21. --- CHANGELOG.md | 4 ++++ bench/golden.json | 9 +++++---- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 161f328..5daa10a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ Please choose versions by [Semantic Versioning](http://semver.org/). * MINOR version when you add functionality in a backwards-compatible manner, and * PATCH version when you make backwards-compatible bug fixes. +## Unreleased + +- fix: flip the `tts-mcp#13` conventional-prefix entry from `accepted` to `rejected` in the golden set (`golden-curated-3` → `golden-curated-4`; 158 entries unchanged, 138/20 → 137/21). Verified against the PR head `59f9fe8`: `tts-mcp` declares no conventional-prefix requirement (absent from both `CLAUDE.md` and `AGENTS.md`) and its own released `v0.2.0`/`v0.1.0` CHANGELOG sections use the same prose-bullet style — a correct observation that names no defect, which the golden set's own rule rejects. It was promoted on 2026-08-18 on the strength of three-model agreement (m3+m2.7+deepseek); that reasoning was wrong because all three share a conventional-commit training prior, so agreement amplified a shared bias rather than triangulating truth. **Model agreement is not independent evidence when the models share priors** — verification has to run against the repository, not a vote. Scores computed against `golden-curated-3` credited models for reproducing a non-defect and penalised them for missing it; the entry now works as a precision probe instead + ## v0.50.0 - feat: Ship the comprehensive security v1 rule base in `docs/security/security-review-guide.md` — 7 judgment-tier rules (SSRF, XSS, deserialization, open redirect, webhook verification MUST; mass assignment, insecure defaults SHOULD) and 2 invariant-linked authz rules (resource ownership, tenant isolation MUST) with `**Class**: security-invariant` and `@commits` triggers; extend `scripts/build-index.py` to emit a `class` index key and document the new field in `docs/rule-block-schema.md`; regenerate `rules/index.json` from 171 to 180 entries; record the cross-language detector layout decision (per-language `rules/security/{go,python,node}/` target, go-first v1 stays flat) diff --git a/bench/golden.json b/bench/golden.json index ef31971..a6fca76 100644 --- a/bench/golden.json +++ b/bench/golden.json @@ -1,5 +1,5 @@ { - "version": "golden-curated-3", + "version": "golden-curated-4", "created": "2026-08-09", "prs_version": "curated-1", "baseline": { @@ -2044,10 +2044,10 @@ "conventional-prefix" ], "rule_id": null, - "state": "accepted", + "state": "rejected", "line_when_seen": null, "excerpt": "**`CHANGELOG.md:12`** \u2014 All 9 bullets under `## Unreleased` violate the conventional-prefix requirement: none starts with `feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, or `perf:` (e.g. lin", - "provenance": "three-model agreement (m3+m2.7+deepseek, 2026-08-18 triangulation); deepseek-verified with runnable evidence; aliasing-checked genuinely new" + "provenance": "m3+m2.7+deepseek agreement 2026-08-18, PROMOTED IN ERROR; flipped to rejected 2026-08-26 after verification against the PR head (59f9fe8): tts-mcp declares no conventional-prefix requirement (absent from both CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0 CHANGELOG sections use the same prose-bullet style. A correct observation naming no defect -> rejected per the golden-set rule. Three-model agreement did not triangulate here: all three share a conventional-commit prior, so agreement amplified a shared bias rather than confirming a defect" } ], "adjudications": [ @@ -2062,6 +2062,7 @@ "2026-08-10 REJECTED grew 1 -> 20. The rule applied: a finding is rejected when it is a correct observation that names no defect - polish, taste, or a consequence the finding itself concludes is intended. The clearest cluster is github-releaser-agent#8, where four findings disqualify themselves in their own text ('worth a follow-up rather than in isolation', 'not a correctness fix', 'so not a deviation', 'raise as a dark-factory issue rather than here'). Before this, precision could not be lost by any configuration.", "2026-08-10 Judgment line held consistently, and it is a line: 'harmless today' hardening on correct code is rejected (unquoted shell expansion), while a latent defect where the code does something other than its shape implies is accepted (check() closing over the outer const). Reachability is not correctness. Comment-requests on correct code are rejected even when they argue regression risk.", "2026-08-18 PROMOTE 2 m3 findings deepseek-verified as real: recurring-task-creator#30 vulncheck (govulncheck reproduced GO-2026-6179/6180, CI genuinely red) and discord-assistant#5 apostrophe strip (reproduced exactly). Two more deepseek-verified candidates (unbounded uncaughtException swallow, supervise orphan) turned out to be RE-STATEMENTS of existing entries \u2014 the aliasing check caught them as duplicates and they were dropped rather than promoted twice. First entries backed by two models' independent agreement. glm-5.2 retired; glm-5.3[1m] only.", - "2026-08-18 TRIANGULATE: 1 m3 finding deepseek-verified as real and three-model-agreed (m3+m2.7+deepseek all flagged tts-mcp#13 CHANGELOG.md conventional-prefix violation; deepseek-verified with runnable evidence); promoted to golden-curated-3. Of 16 three-way agreement locations, 15 were rejected on defensible grounds or were re-statements of existing entries; only 1 was genuinely new. Agreement across models is necessary but not sufficient; aliasing checks remain essential to avoid silent double-counting." + "2026-08-18 TRIANGULATE: 1 m3 finding deepseek-verified as real and three-model-agreed (m3+m2.7+deepseek all flagged tts-mcp#13 CHANGELOG.md conventional-prefix violation; deepseek-verified with runnable evidence); promoted to golden-curated-3. Of 16 three-way agreement locations, 15 were rejected on defensible grounds or were re-statements of existing entries; only 1 was genuinely new. Agreement across models is necessary but not sufficient; aliasing checks remain essential to avoid silent double-counting.", + "2026-08-26 CORRECTION: the tts-mcp#13 conventional-prefix entry promoted on 2026-08-18 (index 11 above) was PROMOTED IN ERROR and is flipped accepted -> rejected here (golden-curated-3 -> golden-curated-4; 158 entries unchanged, 138/20 -> 137/21). Verified against the PR head 59f9fe8: tts-mcp declares no conventional-prefix requirement (absent from CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0 CHANGELOG sections use the same prose-bullet style, so the finding is a correct observation naming no defect -> rejected per the golden-set rule. The lesson corrects the note above: three-model agreement was treated as evidence, but m3, m2.7 and deepseek share a conventional-commit training prior, so agreement amplified a shared bias instead of triangulating truth. Model agreement is not independent evidence when the models share priors; verification must run against the repository, not a vote." ] } \ No newline at end of file