Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ Please choose versions by [Semantic Versioning](http://semver.org/).
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.

## Unreleased

- fix: flip the `tts-mcp#13` conventional-prefix entry from `accepted` to `rejected` in the golden set (`golden-curated-3` → `golden-curated-4`; 158 entries unchanged, 138/20 → 137/21). Verified against the PR head `59f9fe8`: `tts-mcp` declares no conventional-prefix requirement (absent from both `CLAUDE.md` and `AGENTS.md`) and its own released `v0.2.0`/`v0.1.0` CHANGELOG sections use the same prose-bullet style — a correct observation that names no defect, which the golden set's own rule rejects. It was promoted on 2026-08-18 on the strength of three-model agreement (m3+m2.7+deepseek); that reasoning was wrong because all three share a conventional-commit training prior, so agreement amplified a shared bias rather than triangulating truth. **Model agreement is not independent evidence when the models share priors** — verification has to run against the repository, not a vote. Scores computed against `golden-curated-3` credited models for reproducing a non-defect and penalised them for missing it; the entry now works as a precision probe instead

## v0.50.0

- feat: Ship the comprehensive security v1 rule base in `docs/security/security-review-guide.md` — 7 judgment-tier rules (SSRF, XSS, deserialization, open redirect, webhook verification MUST; mass assignment, insecure defaults SHOULD) and 2 invariant-linked authz rules (resource ownership, tenant isolation MUST) with `**Class**: security-invariant` and `@commits` triggers; extend `scripts/build-index.py` to emit a `class` index key and document the new field in `docs/rule-block-schema.md`; regenerate `rules/index.json` from 171 to 180 entries; record the cross-language detector layout decision (per-language `rules/security/{go,python,node}/` target, go-first v1 stays flat)
Expand Down
9 changes: 5 additions & 4 deletions bench/golden.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"version": "golden-curated-3",
"version": "golden-curated-4",
"created": "2026-08-09",
"prs_version": "curated-1",
"baseline": {
Expand Down Expand Up @@ -2044,10 +2044,10 @@
"conventional-prefix"
],
"rule_id": null,
"state": "accepted",
"state": "rejected",
"line_when_seen": null,
"excerpt": "**`CHANGELOG.md:12`** \u2014 All 9 bullets under `## Unreleased` violate the conventional-prefix requirement: none starts with `feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, or `perf:` (e.g. lin",
"provenance": "three-model agreement (m3+m2.7+deepseek, 2026-08-18 triangulation); deepseek-verified with runnable evidence; aliasing-checked genuinely new"
"provenance": "m3+m2.7+deepseek agreement 2026-08-18, PROMOTED IN ERROR; flipped to rejected 2026-08-26 after verification against the PR head (59f9fe8): tts-mcp declares no conventional-prefix requirement (absent from both CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0 CHANGELOG sections use the same prose-bullet style. A correct observation naming no defect -> rejected per the golden-set rule. Three-model agreement did not triangulate here: all three share a conventional-commit prior, so agreement amplified a shared bias rather than confirming a defect"
}
],
"adjudications": [
Expand All @@ -2062,6 +2062,7 @@
"2026-08-10 REJECTED grew 1 -> 20. The rule applied: a finding is rejected when it is a correct observation that names no defect - polish, taste, or a consequence the finding itself concludes is intended. The clearest cluster is github-releaser-agent#8, where four findings disqualify themselves in their own text ('worth a follow-up rather than in isolation', 'not a correctness fix', 'so not a deviation', 'raise as a dark-factory issue rather than here'). Before this, precision could not be lost by any configuration.",
"2026-08-10 Judgment line held consistently, and it is a line: 'harmless today' hardening on correct code is rejected (unquoted shell expansion), while a latent defect where the code does something other than its shape implies is accepted (check() closing over the outer const). Reachability is not correctness. Comment-requests on correct code are rejected even when they argue regression risk.",
"2026-08-18 PROMOTE 2 m3 findings deepseek-verified as real: recurring-task-creator#30 vulncheck (govulncheck reproduced GO-2026-6179/6180, CI genuinely red) and discord-assistant#5 apostrophe strip (reproduced exactly). Two more deepseek-verified candidates (unbounded uncaughtException swallow, supervise orphan) turned out to be RE-STATEMENTS of existing entries \u2014 the aliasing check caught them as duplicates and they were dropped rather than promoted twice. First entries backed by two models' independent agreement. glm-5.2 retired; glm-5.3[1m] only.",
"2026-08-18 TRIANGULATE: 1 m3 finding deepseek-verified as real and three-model-agreed (m3+m2.7+deepseek all flagged tts-mcp#13 CHANGELOG.md conventional-prefix violation; deepseek-verified with runnable evidence); promoted to golden-curated-3. Of 16 three-way agreement locations, 15 were rejected on defensible grounds or were re-statements of existing entries; only 1 was genuinely new. Agreement across models is necessary but not sufficient; aliasing checks remain essential to avoid silent double-counting."
"2026-08-18 TRIANGULATE: 1 m3 finding deepseek-verified as real and three-model-agreed (m3+m2.7+deepseek all flagged tts-mcp#13 CHANGELOG.md conventional-prefix violation; deepseek-verified with runnable evidence); promoted to golden-curated-3. Of 16 three-way agreement locations, 15 were rejected on defensible grounds or were re-statements of existing entries; only 1 was genuinely new. Agreement across models is necessary but not sufficient; aliasing checks remain essential to avoid silent double-counting.",
"2026-08-26 CORRECTION: the tts-mcp#13 conventional-prefix entry promoted on 2026-08-18 (index 11 above) was PROMOTED IN ERROR and is flipped accepted -> rejected here (golden-curated-3 -> golden-curated-4; 158 entries unchanged, 138/20 -> 137/21). Verified against the PR head 59f9fe8: tts-mcp declares no conventional-prefix requirement (absent from CLAUDE.md and AGENTS.md) and its own released v0.2.0/v0.1.0 CHANGELOG sections use the same prose-bullet style, so the finding is a correct observation naming no defect -> rejected per the golden-set rule. The lesson corrects the note above: three-model agreement was treated as evidence, but m3, m2.7 and deepseek share a conventional-commit training prior, so agreement amplified a shared bias instead of triangulating truth. Model agreement is not independent evidence when the models share priors; verification must run against the repository, not a vote."
]
}
Loading