diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 3a965e5..4556782 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,237 +1,88 @@ -# Knowledge flush — 13 insight(s) ingested (21 claimed, 2 dropped, 6 released) +# Knowledge flush — 21 insight(s) -Cross-Check: 1× independent adversarial `claude` CLI headless pass over the 5 new pages — it refuted the changed-files gate page's "prettier exits 0 on an empty match set" claim; re-measured against Prettier 3.7.4, confirmed the reviewer was right (unmatched operand exits **2**), and rewrote the page, report rows 3/5 and `log.md`. Other 5 claim groups verdicted sound. Limits: the reviewer's sandbox blocked repo reads, so source-quote-supports-directive and self-contradiction dimensions went unaudited (details in `## Cross-Check`). - -Queue drained under run id `20260827-125731-38371` (this session is the detached -`hooks/auto-flush.sh` run; its step-0 acquire resolved re-entrantly to -`already-owned`, not to a competing holder). 21 rows were claimed; 13 are -ingested below, 2 are retired as out-of-layer, and 6 are released back to -`pending` for a later flush because each needs its own page rather than a row, -and rushing six more pages in one pass would have lowered the bar on all of them. +Run: auto-flush headless child (run id `20260908-154254-28665`), queue drained from 14 session files (21 claimed rows). Outcome: 12 new pages, 8 merges into existing pages, 1 fold onto open PR #183, 0 drops. Lint after ingest: `wiki-structure-checks.js` 288 pages / 13 indexes / 0 findings; `wiki-lint-prohibitions.js` directives 75 / violations 0 (count unchanged, so the bats pin needs no bump); every touched page ≤ 120 body lines. ## Verified best-practice -Every external claim below was live-fetched this session and quoted in the page's -`Sources` block. Field evidence carries the repo, date, and the measured numbers. - -| # | Claim | Sources checked | How verified | Confidence | -|---|-------|-----------------|--------------|------------| -| 1 | `now()` is `transaction_timestamp()` (fixed at transaction start) while `clock_timestamp()` "changes even within a single SQL statement"; `RETURNING` yields computed defaults "without needing a separate database query" | postgresql.org `functions-datetime`, `dml-returning`, `transaction-iso` | Fetched; both key sentences quoted verbatim into the page | verified | -| 2 | A boundary recomputed in a follow-up step is a *second, later* `now`, widening a `<= boundary` set | Field: `rtb-unified` `packages/orpc/src/routers/batch.ts` — codifies "one `now` per decision" and passes `now` into the boundary helper; its result type omits the boundary, which is the shape that invites recomputation | Read the invariant and the signature in the cited file | field-tested | -| 3 | **[CORRECTED BY CROSS-CHECK]** The vacuous-pass shapes for `prettier --check` are: no operands (rc **0**), all operands ignore-filtered (rc **0**), and unsupported extensions with `--ignore-unknown` (rc **0**). A pattern/operand matching nothing exits **2** — it prints the success sentence *and* an unmatched-pattern error | prettier.io CLI + ignore docs; local measurement, Prettier 3.7.4 | The first draft generalised "empty match set ⇒ exit 0" from a field log where both messages appeared together. The independent reviewer flagged it; I then ran all seven cases against a real binary and rewrote the page around the measured table | verified (re-measured) | -| 4 | zsh does not word-split unquoted parameter expansions by default, so `cmd $FILES` arrives as **one** operand | zsh FAQ ch. 3 (`SH_WORD_SPLIT`) | Fetched; quoted ("By default, zsh does not have that behaviour: the variable remains intact") | verified | -| 5 | The zsh word-split operand exits **2**, but its log still carries the success sentence — so the log misleads even though the exit code does not | Field 2026-08-24 (`rtb-unified`, zsh) + local measurement 2026-08-27 | Field log showed both messages together; the local run reproduced it as `rc=2`. The page now says explicitly that this row fails loudly *unless* `--no-error-on-unmatched-pattern` is set. Probe placement re-confirmed: `.claude/tmp/` is `.gitignore`d, so a probe there passes at rc 0 | verified (re-measured) | -| 6 | TypeScript applies excess-property/contextual typing to fresh object literals, so a value of a type can be constructed with the type's name absent from the text | typescriptlang.org handbook, *Object Types* | Fetched; confirmed the check follows from the contextual type, not from a written annotation | verified | -| 7 | `tsc`'s program is `files` ∪ `include` ∪ transitive imports; `exclude` "only changes which files are included as a result of the `include` setting" and does not stop an imported file entering the program | typescriptlang.org TSConfig `#include`, `#exclude` | Fetched; the `exclude` sentence quoted (it sharpens the rule to "in the program", not "in `include`") | verified | -| 8 | Consequence of 6+7 measured | Field 2026-08-24/25 (`rtb-unified`): `grep "DealViewer"` reported 3 construction sites, actual 8 — the missed set included production wiring `routers/deal.ts:38`; `ContractScopeActor` 7→~22. Separately, `packages/orpc/tsconfig.json` `include: ["src/**/*"]` produced 3 production + 13 api-test errors and **zero** for `__tests__/routers/deal.test.ts`, whose 6 sites appeared only as 6 failing tests | Counts recorded from the cited runs | verified | -| 9 | cgroup v2: `memory.peak` is max usage since creation/reset; at `memory.max` "the OOM killer is invoked in the cgroup"; in `memory.events`, `max` counts times usage "was about to go over the max boundary" — **distinct** from `oom_kill` | docs.kernel.org cgroup-v2 admin guide | Fetched; all four quoted. This corrected the candidate, which had read a non-zero `max` as a kill; the page now states the distinction explicitly | verified | -| 10 | An `exec`'d process joins the container's cgroup and is invisible to the application's own semaphore | kubernetes.io `manage-resources-containers`, `assign-memory-resource`, `kubectl exec` reference + field 2026-08-26 (review-bot pod, `limits.memory: 3Gi`): `memory.current` 2.54 GiB, `memory.peak` 3.0 GiB (at the limit), `memory.events: max 5`, while `maxConcurrentAgents: 20` reported free slots | Docs fetched; pod numbers from the cited measurement | verified | -| 11 | Basename-keyed mutation backups collide across directories and restore cross-writes; an untracked file's `git diff` is empty whether restored or destroyed | Field 2026-08-21 (`rtb-unified`, NEWRTB-2936): restore wrote `schemas/deal.ts` into `routers/deal.ts` → `Cannot find module './common.js'`, `grep -c dealRouter` = 0; **both files were 154 lines**, so a line-count check passed; after re-keying, M9/M10 flipped SURVIVED→KILLED. Plus stryker mutant-states / pitest for the verdict vocabulary | Reproduced end to end in the cited run | field-tested | -| 12 | A negative assertion is vacuous when the fixture never supplies the triggering input | Field 2026-08-25 (`rtb-unified`): with `staleQueuedJobIds: []` the code early-returned; the widening the assertion claimed to catch survived 116/116 green | Mutation applied and observed | field-tested | -| 13 | A body-level (non-inline) review finding cites no file, so rebutting from an assumed file rejects real defects | Field 2026-08-19 (PR #327 r16): quote matched `report.py:393/416/425`, not the assumed `fill_plan.py:307` — sibling modules, one already fixed | Grep resolved the quote to the real site | field-tested | -| 14 | Unifying two duplicate allowlists defaults to the union and silently widens each side | Field 2026-08-25 (`rtb-unified` PR #965): folding `DISPLAYABLE_ERROR_CODES` into `USER_FACING_ERROR_CODES` would have added `UNAUTHORIZED` + `VALIDATION_ERROR`, exposing raw server messages as inline UI errors; caught only by computing the difference first | Difference computed before the merge | field-tested | - -Not upgraded: nothing was marked `verified` on field evidence alone. Two pages -carry `confidence: field-tested` (`mutation-harness-file-custody`, -plus the pre-existing `evaluating-review-feedback`), and no candidate was -recorded as `verified` without a fetched primary source. +Five research agents vetted each candidate against primary sources (WebFetch/curl of official docs and source code, plus local reproductions where a claim was runnable). Confidence per candidate: + +**New pages** +1. `infrastructure-ci-cd-secret-needing-gate-on-fork-prs` — **verified**. GitHub docs `events-that-trigger-workflows#pull_request_target` ("With the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository"; "Running untrusted code on the pull_request_target trigger may lead to security vulnerabilities"); GitHub Security Lab "Preventing pwn requests" ("Combining pull_request_target workflow trigger with an explicit checkout of an untrusted PR is a dangerous practice"); anthropics/claude-code-action `docs/security.md` ("Do not check out an untrusted ref into the workspace root"; "Preferred — check out the base ref (default)"). Local: dev-loop `wiki-agent-gate.yml` + 12 bats tests read directly. Note: the "instruction-shaped diff text is a finding" stance is this repo's gate prompt, not security.md — cited accordingly. +2. `infrastructure-ci-cd-review-bot-adopted-as-a-blocking-gate` — **verified**. alibaba/open-code-review `action.yml` (result written to `/tmp/ocr-result.json`; "Fail job on OCR error" keyed on exit code), `cmd/opencodereview/review_cmd.go` `reviewResultError` ("non-zero only for a run-level failure, or when every selected item failed. Any usable coverage — even incomplete — exits 0"), `scripts/github-actions/post-review-comments.js` (every `createReview` uses `event: "COMMENT"`, zero `REQUEST_CHANGES`/`setFailed` matches in 2657 lines), `sarif.go` exists. +3. `infrastructure-agent-orchestration-tool-retirement-knowledge-transplant` — **field-tested**. dev-loop PR #130 fetched via gh: 14 + ~30 skills inventoried, 8 gaps ported, skip list in log.md, lint 0 findings. No external canonical source for the methodology found; not upgraded. +4. `backend-python-language-dict-subclass-attribute-loss-on-copy` — **verified**. Python reference "A dictionary display yields a new dictionary object"; `copy` docs "copy.copy() normally returns an instance of the same type" (both quotes re-fetched by curl this run); local python3 3.14.6 reproduction (subclass attr lost under `dict()`/`{**}`; preserved under `copy.copy`; `getattr` default silent vs direct access `AttributeError`). +5. `backend-python-language-iterative-dfs-for-unbounded-graph-depth` — **verified**. `sys.getrecursionlimit`/`setrecursionlimit` docs; What's New 3.12 "The recursion limit now applies only to Python code. Builtin functions do not use the recursion limit, but are protected by a different mechanism" (re-fetched by curl this run; the agent's discuss.python.org source was replaced with the official page); local reproduction: recursive DFS on a 2000-chain raises `RecursionError`, iterative completes with matching cycle output. +6. `qa-document-verification-sweeping-pre-gate-citations-for-fabrication` — **field-tested**. dev-loop issue #156 fetched via gh (8 agents, ~440 fetches, 270/270 pages, 33 blockers on 24 pages, per-category breakdown; the page reports the breakdown rather than the monolithic "33 citation defects"). External LLM-citation-hallucination papers confirm the phenomenon, not the sweep methodology. +7. `qa-process-session-identity-leak-in-plugin-prose` — **field-tested**. Incident re-verified: installed marketplace copy + two cached plugin versions contain no occurrence of the leaked name. 12factor.net/config cited as analogy only ("strict separation of config from code"); the page itself does not contain the user's personal name. +8. `platforms-toolchains-regeneration-silently-drops-hand-edited-state` — **verified**. XcodeGen README ("Generate projects on demand and remove your .xcodeproj from git"), Docs/FAQ.md ("you can also check it in as a halfway step"), issues #515 (GUI-created shared schemes "will be overwritten the next time they generate") and #572 ("running xcodegen will overwrite that xcscheme file"). Field: 118 pbxproj lines + xcscheme deleted by xcodegen 2.45.4 on a one-key change. +9. `testing-quality-narration-based-ordering-assertions` — **field-tested**. Apple `terminationHandler` API doc confirms the invariant class; the mutation narrative is session-relayed (no repo located) and stated as such in the page. +10. `testing-mocking-autouse-fixture-shadows-function-under-test` — **verified**. pytest monkeypatch how-to ("monkeypatch.setattr must be called before the function which will use the patched function is called"; "All modifications will be undone after the requesting test function or fixture has finished"), fixtures how-to (autouse fixtures run as setup "even though neither test requested it"). +11. `testing-quality-cross-task-stub-assertions` — **field-tested**. The cited commit `de8c07c` and HANDOFF.md §5 item 18 were read directly in the source checkout (`git show`), confirming the placeholder-text → root-class rewrite. +12. `testing-strategy-real-cli-spot-check-for-new-execution-paths` — **field-tested**. Commits `e761ed3`, `b25ff04`, `656d763` and HANDOFF.md §5 item 19 read directly (`git show`): ENOENT cwd hang and permit-scope deadlock, both "found by the coordinator's real-CLI spot check". + +**Merges** +13. `security-data-commit-identity-in-public-repos` — **verified**. git-log pretty formats (`%ae`/`%ce`), git-filter-repo docs (`--mailmap` "rewriting author, committer, and tagger names and emails"; `--replace-message`), github.blog changelog 2019-12-19 ("we will automatically credit every commit author in the pull request as a co-author on the squash commit"). Note: the docs.github.com "about pull request merges" page does not state the trailer behavior; the changelog does. +14. `security-data-masking-verification` — **field-tested** (page stays field-tested). The OWASP API3:2023 fetch was not completed by the agent, so it is NOT cited. Evidence is the direct read of the mask implementation and the two bypassing read endpoints. +15. `infrastructure-agent-orchestration-session-completion-gates` — **verified**. Leonxlnx/unlazy `scripts/stop-hook.mjs` (`MAX_BLOCKS = 6`, ledger-hash no-progress counter) and CHANGELOG (1.0.0 instruction-only → 2.0.0 gate files + Stop hook) fetched; dev-loop `hooks/loop-gate.sh` Gate 2 + `tests/loop-gate.bats` read directly. The candidate's claim that loop-gate.sh is self-report-only was wrong (Gate 2 already exists); the page documents the mechanism, not the mischaracterization. +16. `debugging-methodology-probe-path-vs-operation-path` — **field-tested** row on a verified page (84/84 events, store-state polling via `useRunStore.getState()`). +17. `frontend-state-effects-usage` — **verified**. react.dev removing-effect-dependencies ("Object and function dependencies can make your Effect re-synchronize more often than you need"), useRef reference ("Changing a ref does not trigger a re-render"), Lenis `packages/core/src/lenis.ts` `updateClassName()` toggling `lenis-scrolling`/`lenis-stopped`. +18. `platforms-toolchains-flag-availability-at-the-execution-site` — **verified**. code.claude.com CLI reference documents `--max-turns`; local reproduction on claude 2.1.263 (`--help` has zero matches; `-p --max-turns 3 --output-format json` returns `is_error:false`). +19. `qa-document-verification-editing-a-gated-document` — **verified** by reproduction in this repo (`wiki-lint-prohibitions.js` prints `directives: 75`, bats pins the same literal). PR #151 numbers could not be re-fetched by that agent (401) and are cited as field evidence. +20. `infrastructure-agent-orchestration-worktree-isolated-workers` — **verified**. gitignore docs (`.gitignore` = distributed via clone; `$GIT_COMMON_DIR/info/exclude` = repository-local), git-worktree docs; local reproduction: from a linked worktree `git rev-parse --git-path info/exclude` resolves to the main `.git/info/exclude`, and one pattern silenced `git status` in both checkouts. + +**Fold** +21. Multi-line `CHECK:` truncation in `gate-check.sh` — **verified** by source read (`skills/loop-implement/scripts/gate-check.sh:117-118`, no wildcard case) and a scratch-dir reproduction (`exit=2: unexpected EOF while looking for matching quote` vs `exit=0 matched: WSGI_APP_OK` on one line). Folded onto PR #183 (see Open-PR check). ## Existing-layer check -Method: routed via `INDEX.md` → domain `index.md`; then built a full id+title -index of all 265 pre-existing pages and probed it with concept greps -(`clock_timestamp|clock skew`, `changed[- ]files|--ignore-unknown`, `tsconfig`, -`contextual typ|excess property`, `set difference|allowlist`, `cgroup`, -`basename|backup.*restore`, `2>&1`, `delta|baseline`) before deciding new vs merge. - -Pages read: testing-quality-source-text-wiring-assertions, testing-quality-tests-that-cannot-fail, backend-common-change-impact-call-site-enumeration, backend-common-change-impact-widening-a-closed-value-table, qa-process-evaluating-review-feedback, infrastructure-containers-host-cgroup-visibility, testing-quality-behavior-not-implementation - -Findings: - -- **Zero coverage** (→ new pages): changed-files-only gates, tsconfig/contextual - typing, allowlist set-difference, app-clock-vs-DB-timestamp, exec-into-a-running-container. - The concept greps returned no hits for these; the clock hits were incidental - (offline sync, token handling) and none compared an app clock to a DB column. -- **Already covered — one candidate all but retired.** The comment-stripping - insight is `source-text-wiring-assertions` step 2 verbatim ("Make the - assertion's subject the file with comments removed"), and its false-RED and - negative/count false-GREEN shapes are already edge rows. Only the *empty-slice* - consequence was new, so that alone was merged. -- **Line-cap conflict handled without breaking the invariant.** - `source-text-wiring-assertions` sits at exactly **120** body lines (the - documented cap). Rather than add a row and violate maintenance invariant 5, the - new nuance and the new field evidence were merged **in place** into an existing - edge row and an existing source bullet. Body count re-measured after editing: - still 120. -- **No conflicts found.** Nothing ingested contradicts an existing directive. - The one correction made was to a *candidate*, not to the wiki (item 9: the - `memory.events` `max` counter is approaches-to-limit, not kills). -- **Related links added both ways**: `tests-that-cannot-fail` ↔ - `mutation-harness-file-custody`; `widening-a-closed-value-table` ↔ - `compiler-as-call-site-inventory` (+ `errors-diagnostics-from-a-shared-code-path`); - `host-cgroup-visibility` → `exec-added-processes-and-the-memory-budget` - (from its existing self-monitoring row). -- **Indexes/log updated**: 4 domain indexes (+5 "load when" rows), `log.md` - appended. Root `INDEX.md` unchanged — no new domain. - -Gates run (the exact CI commands from `.github/workflows/test.yml`): -`node scripts/wiki-structure-checks.js wiki` → **pages: 270, indexes: 13, -findings: 0**; `node scripts/wiki-lint-prohibitions.js wiki` → **directives 72, -compliant 72, violations 0** (the 1 `info` is pre-existing in -`config/keys-ahead-of-their-consumer.md`, untouched); `bash scripts/check-versions.sh` -→ `ok: dev-loop 1.11.2`. The `bats tests/` job was **not** run — bats is not -installed on this machine, and this change touches only wiki markdown (no -scripts or hooks), so that suite's subject is unchanged. +Routing went INDEX.md → domain `index.md` → every page whose "load when" overlapped; second-domain indexes were read where the queue's domain tag was doubtful (eaa97ef1 tagged platforms → infrastructure; 5f1e60e7 tagged platforms → qa; 27598bf7/15c19bc4 untagged → testing, the domain that owns the test artifact per AGENTS.md routing step 1). + +Pages read: infrastructure-ci-cd-secrets-handling, infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-changed-files-only-gates, qa-process-llm-review-pipelines, infrastructure-agent-orchestration-session-completion-gates, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, infrastructure-agent-orchestration-worktree-isolated-workers, infrastructure-agent-orchestration-session-context-token-budget, qa-process-scope-purity-checks, backend-python-language-mutable-state-traps, databases-transactions-optimistic-vs-pessimistic-locking, security-data-masking-verification, security-data-pii-handling, backend-common-change-impact-call-site-enumeration, backend-common-change-impact-cross-module-consumer-census, debugging-methodology-probe-path-vs-operation-path, debugging-methodology-hypothesis-testing, debugging-methodology-reproduce-first, security-data-commit-identity-in-public-repos, security-secrets-secrets-in-code, qa-process-adversarial-change-review, qa-process-defect-class-resweep-after-review, security-dependencies-agent-skill-supply-chain, platforms-tools-plugin-mcp-server-registration, backend-common-llm-binding-instructions-for-agents, qa-document-verification-spec-document-gates, qa-document-verification-editing-a-gated-document, qa-deliverables-quantitative-claims-in-a-published-document, frontend-state-effects-usage, frontend-design-html-in-canvas, platforms-toolchains-compiler-sysroot-on-macos, platforms-toolchains-version-management, platforms-toolchains-environment-resync-removes-undeclared-packages, platforms-toolchains-flag-availability-at-the-execution-site, platforms-processes-non-interactive-cli-invocation, testing-quality-tests-that-cannot-fail, testing-quality-mutation-harness-file-custody, testing-async-async-testing, testing-mocking-captured-call-arguments, testing-quality-source-text-wiring-assertions, testing-mocking-what-to-mock, testing-data-test-data-and-isolation, testing-quality-stale-artifact-baselines, testing-strategy-test-level-choice, qa-process-completion-claims, testing-quality-checks-that-cannot-pass + +Overlaps and outcomes: +- **Merged (same trigger, extending directive):** commit-identity (author-only → committer + squash trailer + cross-repo audit + named rewrite tool), masking-verification (per-channel sweep → mask shape classification + direct-store-read call sites), session-completion-gates (phase gate items 1–6 → ledger gate items 7–8 + release valve), probe-path-vs-operation-path (auth probe case → multi-layer pipeline row), effects-usage ("read a value without re-running" row → rAF-loop ref-mirror row), flag-availability (cross-environment version drift → `--help`-as-source-of-truth row), editing-a-gated-document (author side → reviewer-side pinned-count row), worktree-isolated-workers (ignored state dirs → where the ignore pattern lives). +- **Created (new trigger, nothing covering it):** the 12 pages above. Closest neighbours checked and cross-linked rather than merged: tests-that-cannot-fail (at 115/120 lines; narration/mutation case is a distinct failure mode), what-to-mock and test-data-and-isolation (no autouse-shadowing case), environment-resync-removes-undeclared-packages (same abstract shape as xcodegen regen, different artifact → related), llm-review-pipelines (same OCR example, disjoint case), secrets-handling item 6 (general "no secrets on fork PRs" principle → the new page is the sanctioned exception pattern). +- **Conflicts flagged:** none. One candidate mischaracterized dev-loop's own hook as self-report-only (loop-gate.sh already has a ledger gate); the merge documents the pattern without repeating the mischaracterization. +- **Related links added both ways** for every new page and every merge (see log.md entry). Five reverse links are deferred because the target ids exist only on open PRs (listed under Open-PR check). +- **Domain routing overrides:** eaa97ef1 platforms→infrastructure/agent-orchestration (platforms owns OS portability, not hook design); 5f1e60e7 platforms→qa/process (a review-checklist page, sibling of adversarial-change-review; kept separate from the git-identity page because trigger, artifact, detection command and remediation all differ); 27598bf7 and 15c19bc4 → testing (the artifact changed is a test / test strategy; orchestration pages are linked, not the owner). ## Open-PR check -`gh pr list --repo choiyounggi/dev-loop --state open --search "head:knowledge/"` -returned **no open PRs**, and a second unfiltered `gh pr list --state open` -returned none either — the repository has zero open PRs at flush time. There -were therefore no in-flight sibling branches to diff against, and no -`git fetch origin ` / `git diff origin/main origin/ -- wiki/` -comparisons to run. +Open `knowledge/*` heads listed via `gh pr list` and fetched: #187 (`knowledge/choiyounggi-20260906-213635`), #186 (`…-20260906-013856`), #185 (`…-20260906-003745`), #183 (`…-20260904-133717`), #182 (`…-20260903-214027`), #181 (`…-20260903-203836`), #180 (`…-20260903-184706`), #179 (`…-20260903-172728`). Each candidate was diffed against every head's `wiki/` changes (138 page entries summarized, overlapping pages read via `git show origin/:`). + +Per-candidate verdicts: +- **fold:#183** — 5f41fd94 (multi-line CHECK truncation). #183 adds `testing/quality/gate-parsing-vs-command-execution.md`, which covers the same script and case family but not this failure mode. Not re-ingested here; the exact edge-case row + source bullet is posted as a comment on PR #183 for the owner to fold in (the page does not exist on main, so it cannot be edited on this branch). +- **new (no overlap)** — all other 20. Specific checks: c5ac7430 vs #187 `workflow-authored-pull-requests` (bot-PR/ruleset, different failure) and #182 `llm-review-pipelines` (byte-identical to main); acd8c8f6 vs #182/#187 (none touch exit-code gating); eaa97ef1 vs #183 `session-completion-gates` and #186/#179 `control-signals-vs-primary-artifacts` (related-only / orchestrator-side, no ledger-gate content; my additions append to section ends to minimize textual conflict with #183); 39d73731 vs #186 `model-coupled-guidance-aging-detector` (temporal staleness detector, not citation accuracy — no fold); a3e368c1 vs #187 `reproduce-first` (assertion-boundary of an existing test vs live layer probing — adjacent, cross-linked, not folded); 44ebf34c vs #187 `pointer-attracted-particle-fields` (sibling canvas page, different case); 7b689559 vs #186 "literal-constant re-assertion" row and #179 (unrelated); 1db78612 vs #181 `synthetic-corpus-measurement-floor` (pigeonhole floor, unrelated); 15c19bc4 vs #179 `verify-command-in-a-worker-brief` (venv path in a brief, different mechanism); 27598bf7 vs #181 `checkable-claims-in-an-adopted-plan` (plan numeric claims, different); 4b17b2af vs #179 `worktree-isolated-workers` (adds an item on the same directories but not where the ignore pattern lives); 0e9f9b6d vs #183 `non-interactive-cli-invocation` (related-id only); 158ef7ac, d30e0ceb, e9ed0c65, 600d2698, 70ebcc34, 5f1e60e7, f9f7e64e, c36cb938 — no open head touches a relevant page. +- **drop** — none. -Per-candidate verdict: **all 21 = `new`.** No `fold`, no `drop-as-pending-duplicate`. -(The 2 drops recorded below are out-of-layer drops, not pending-duplicate drops.) +Deferred `related:` links (target ids not on main yet; add after the PR merges): `frontend-design-pointer-attracted-particle-fields` (#187) ↔ effects-usage; `infrastructure-agent-orchestration-checkable-claims-in-an-adopted-plan` (#181) ↔ cross-task-stub-assertions; `infrastructure-agent-orchestration-verify-command-in-a-worker-brief` (#179) ↔ real-cli-spot-check; `qa-document-verification-model-coupled-guidance-aging-detector` (#186) ↔ sweeping-pre-gate-citations; `testing-quality-gate-parsing-vs-command-execution` (#183) ↔ session-completion-gates. ## Routing decision -**New pages (5)** - -| Page | Domain/category | From | Why not an existing page | -|------|-----------------|------|--------------------------| -| `application-clock-vs-database-timestamps` | databases / transactions | `2b27d15d` + `bea92fdd` | No page compares an app clock to a DB column. `transactions` chosen over `schema-design` because the decisive content is transaction-time semantics (`now()` = transaction start ⇒ stamp order ≠ commit order) and the fix is a lock/isolation choice | -| `changed-files-only-gates` | infrastructure / ci-cd | `ff041061` + `4b9af3a0` | Zero grep hits. Both candidates are the same defect (a gate green with an empty subject) from two directions, so they became one page rather than two | -| `compiler-as-call-site-inventory` | backend / common / change-impact | `702dcf4e` + `94d55f2f` | `call-site-enumeration` is the sibling case (callers of a changed signature, Python positional-vs-keyword) and is at 80 body lines; the TS mechanism is *constructors of a type* with its own workflow, so per "one case per page" it is a separate page, cross-linked | -| `mutation-harness-file-custody` | testing / quality | `6a9de235` + `41fa1c87` | `harness-reverse-controls` covers scoring a harness; nothing covers the harness's custody of the tree. Both candidates are that one case (keying, and the read window) | -| `exec-added-processes-and-the-memory-budget` | infrastructure / containers | `7b9e8788` | `host-cgroup-visibility` is cross-pod read mechanics and explicitly routes self-monitoring elsewhere; `resource-limits-and-probes` is manifest authoring. This is a runtime preflight before adding load | - -No new category was created — all five landed in existing categories. - -**Merged into existing pages (5 candidates)** - -| Candidate | Merged into | Shape | -|-----------|-------------|-------| -| `91ef5d53` | `testing-quality-tests-that-cannot-fail` | +1 never-fails row, +1 Instead-of row, +1 source | -| `f189f423` | `testing-quality-source-text-wiring-assertions` | In-place extension of 1 edge row + 1 source bullet (page at the 120-line cap) | -| `bb6d8539` | `backend-common-change-impact-widening-a-closed-value-table` | +Do-this 6 & 7 (incl. a set-difference ruling table), +1 Instead-of row, +1 source | -| `60a817ee` | `qa-process-evaluating-review-feedback` | +2 edge rows, +1 Instead-of row, +1 source | -| `7b9e8788` | `infrastructure-containers-host-cgroup-visibility` | Cross-link from its self-monitoring row to the new page | - -**Dropped — out of layer (2, retired)** - -- `094dedf3` — a Figma MCP `inspect_node` → `get_dev_ready` children-fetch - workaround. The server is a private, org-internal MCP plugin; the behavior is - not publicly verifiable and the directive does not transfer to any other reader. -- `e165a365` — an `/rtb:review` remote-fallback runbook naming - `~/.claude/tools/rtb-remote-review.sh` and an internal pod. The transferable - kernel ("a two-provider review gate degraded to one provider is not a passed - gate") is already the subject of `qa-process-llm-review-pipelines`; what remains - is machine-specific paths. - -**Released back to `pending` (6)** — each needs its own page, not a row, and is -better served by a dedicated pass than by being appended here: -`81dc1f98` (naming the carrier field/type when a plan says "wire A to B"), -`b9ae304a` (`VAR="$(cmd 2>&1)"` mixing stderr into a value used as a path), -`fdd0b3c6` (monitor markers anchored at line start; delta rather than absolute -state; first cycle records a baseline), -`c2adb2be` (positional-order assertions on rendered SQL predicates), -`815e8cb9` (grep only *active* `DATABASE_URL` assignments, and confirm which -dotenv file the tool loads, before a destructive DB command), -`f1146adb` (CI ticket-key extraction scoped by changed-file intersection rather -than by mention). - -## Decision Log - -**Intent.** Drain the harvested `★ Insight` queue into reviewable wiki knowledge -without lowering the wiki's evidence bar. The queue held 21 rows accumulated over -several days; the goal was correct routing and real verification, not a high -ingest count. - -**Alternatives considered and rejected.** - -- *Ingest all 21 in this pass.* Rejected: six of them each need their own page, - and writing six more pages in one pass would have produced thin, weakly-sourced - entries. They are released to `pending`, not dropped, so the next flush takes - them with a full budget. -- *Append the two TypeScript candidates to `call-site-enumeration`.* Rejected: - that page is the sibling case (callers of a changed signature, Python - positional-vs-keyword). AGENTS.md requires one case per page, so the - constructor-enumeration case became its own page, cross-linked both ways. -- *Add a row to `source-text-wiring-assertions` for the empty-slice nuance.* - Rejected: that page is at exactly the documented 120-line body cap, so adding a - line would violate maintenance invariant 5. The nuance was merged **in place** - into an existing edge row instead; body re-measured at 120. -- *Drop the comment-stripping candidate entirely as a duplicate.* Rejected: its - directive is already the page's step 2, but the empty-slice consequence - (vacuous **green**, not the documented noisy red) was genuinely absent. -- *Claim a cross-check exemption because this PR cannot merge itself.* Rejected — - see below; the check found a real error, which is the argument against exempting. -- *Push to `origin`* as the skill's snippet does. Not available: this contributor - has no write access to `choiyounggi/dev-loop` (403). Used the pre-existing - `fork` remote, which is how every prior knowledge branch here was published. -- *Branch name from `git config user.name`.* The skill's ASCII sanitisation of a - Korean name yields an empty string → `anon`, defeating the attribution the - branch name exists for. Used the gh login, matching existing branch names. - -**Where reviewers should look hardest.** - -1. `infrastructure/ci-cd/changed-files-only-gates.md` — rewritten after the - cross-check. The measured table is the load-bearing part; please sanity-check - it against your own Prettier version, since the exit codes are version-visible - behaviour rather than a documented contract. -2. `databases/transactions/application-clock-vs-database-timestamps.md` step 5–6 — - the claim that timestamp order is not commit order, and that the remedy is a - lock/isolation level rather than finer clock resolution. `[추정]` on the MySQL - `NOW()`/`SYSDATE()` row: taken from general MySQL semantics, not fetched this - session like the PostgreSQL pages were. -3. `widening-a-closed-value-table.md` Do-this 6–7 — this inserts a security-shaped - concern (allowlist widening) into a page whose original subject was value - tables. If that reads as two cases, it should be split. -4. The 2 dropped candidates — if you consider private-tooling runbooks in scope - for this wiki, they should be restored rather than retired. - -## Cross-Check - -Independent adversarial pass via `claude` CLI headless (separate process, no -shared context), prompted to refute rather than confirm, over the five new pages' -technical claims. - -**It found a real error, and the page was rewritten because of it.** The reviewer -challenged the claim that `prettier --check` exits 0 on an empty match set, -arguing an unmatched pattern errors by default and that exit-0 belongs to the -ignore-filtered case. I resolved it by measurement rather than by argument — -running all seven cases against Prettier 3.7.4 — and the reviewer was right: -an unmatched operand exits **2** (while still printing the success sentence), -whereas the genuine silent vacuous passes are no-operands, all-ignore-filtered, -and `--ignore-unknown`-with-unsupported-extensions. The page, this report's -rows 3 and 5, and the `log.md` entry were all corrected. - -Verdicts on the other five claim groups: **sound** (PostgreSQL clock semantics — -noted as if anything *understated*; zsh word-splitting; TS contextual typing; -`tsc` program membership incl. `exclude`-does-not-stop-imports; cgroup v2 -`max` vs `oom_kill` and `kubectl exec` cgroup placement). - -Stated limits of the check: the reviewer's sandbox denied it read access to -`~/.dev-loop/repo/wiki`, so it adjudicated the six claims as quoted in its prompt -and could **not** audit (b) whether each `Sources` quote supports the directive it -is cited for, or (c) whether any page contradicts its own edge-case rows. Those -two dimensions remain unreviewed by an independent party and are the residual -risk in this PR. A first attempt also returned only the session's Stop-hook -output rather than a verdict; that run was discarded rather than read as -"no findings". - -## Review notes - -- PR-only, as required: no merge, no push to `main`. -- Commit is under the contributor's own ambient git identity - (`최영기 `, gh `dch0202-rsquare`); no assistant identity - and no `Co-Authored-By` trailer. The branch uses the gh login because - sanitizing the Korean `user.name` to ASCII yields an empty string, which the - skill's snippet would have turned into `anon` — that would have defeated the - attribution the branch name exists for. -- Scope purity: only `wiki/**`, four domain indexes, `log.md`, and this report. - Two untracked leftovers from earlier flushes - (`.dev-loop/CROSSCHECK_FINDINGS.md`, `.dev-loop/fold-note-73.md`) were left - untouched and unstaged. +No new category or domain was created; every candidate fit an existing category. + +| Candidate | Target | +|---|---| +| c5ac7430 | infrastructure/ci-cd/secret-needing-gate-on-fork-prs.md (new) | +| acd8c8f6 | infrastructure/ci-cd/review-bot-adopted-as-a-blocking-gate.md (new) | +| f9f7e64e | infrastructure/agent-orchestration/tool-retirement-knowledge-transplant.md (new) | +| d30e0ceb | backend/python/language/dict-subclass-attribute-loss-on-copy.md (new) | +| e9ed0c65 | backend/python/language/iterative-dfs-for-unbounded-graph-depth.md (new) | +| 39d73731 | qa/document-verification/sweeping-pre-gate-citations-for-fabrication.md (new) | +| 5f1e60e7 | qa/process/session-identity-leak-in-plugin-prose.md (new; queue tag platforms overridden) | +| 158ef7ac | platforms/toolchains/regeneration-silently-drops-hand-edited-state.md (new; mobile considered, toolchains owns generator fidelity per the environment-resync precedent) | +| 7b689559 | testing/quality/narration-based-ordering-assertions.md (new) | +| c36cb938 | testing/mocking/autouse-fixture-shadows-function-under-test.md (new) | +| 27598bf7 | testing/quality/cross-task-stub-assertions.md (new; agent proposed infrastructure/agent-orchestration, routed to testing as the owner of the test artifact) | +| 15c19bc4 | testing/strategy/real-cli-spot-check-for-new-execution-paths.md (new; same reasoning) | +| 70ebcc34 | security/data/commit-identity-in-public-repos.md (merge) | +| 600d2698 | security/data/masking-verification.md (merge) | +| eaa97ef1 | infrastructure/agent-orchestration/session-completion-gates.md (merge; queue tag platforms overridden) | +| a3e368c1 | debugging/methodology/probe-path-vs-operation-path.md (merge) | +| 44ebf34c | frontend/state/effects-usage.md (merge) | +| 0e9f9b6d | platforms/toolchains/flag-availability-at-the-execution-site.md (merge; queue tag testing overridden) | +| 1db78612 | qa/document-verification/editing-a-gated-document.md (merge) | +| 4b17b2af | infrastructure/agent-orchestration/worktree-isolated-workers.md (merge) | +| 5f41fd94 | fold → PR #183 gate-parsing-vs-command-execution.md (comment) | + +Domain indexes updated: infrastructure (+3 rows, 2 rows refreshed), backend/python (+2), qa (+2, 1 refreshed), platforms (+1, 1 refreshed), testing (+4), security (2 refreshed), debugging (1 refreshed), frontend (1 refreshed). INDEX.md unchanged (existing domain route lines already cover the new pages). `tests/wiki-lint-prohibitions.bats` pin untouched (directive count still 75). diff --git a/log.md b/log.md index e2ee9ca..0139b4e 100644 --- a/log.md +++ b/log.md @@ -104,3 +104,4 @@ Append-only. Format: `## [YYYY-MM-DD] is a built-in class` for a plainly pure-Python class; choosing between registering the module in `sys.modules` and retargeting the assertion from a class to one of its methods; deciding whether an always-red check is the target's defect or the loader's | +| [dict-subclass-attribute-loss-on-copy](language/dict-subclass-attribute-loss-on-copy.md) | Building a new dict from a `dict` subclass instance that carries extra instance attributes (an ORM/driver row wrapper holding optimistic-lock version info, an ETag, or similar out-of-band metadata) via `dict(obj, **{...})` or `{**obj, ...}` before adding or overriding a key; downstream `getattr(row, "attr", None)` returns `None` instead of the real value with no error; choosing between `copy.copy` and in-place mutate-and-revert | +| [iterative-dfs-for-unbounded-graph-depth](language/iterative-dfs-for-unbounded-graph-depth.md) | Implementing a DFS-based graph or cycle-detection algorithm in Python (compiler diagnostics walking declared-entity relationships, a dependency graph) where the traversal depth scales with user/module content rather than a small fixed set; a recursive implementation risks `RecursionError` near Python's default 1000-frame limit; tempted to raise `sys.setrecursionlimit` instead | diff --git a/wiki/backend/python/language/dict-subclass-attribute-loss-on-copy.md b/wiki/backend/python/language/dict-subclass-attribute-loss-on-copy.md new file mode 100644 index 0000000..7be336e --- /dev/null +++ b/wiki/backend/python/language/dict-subclass-attribute-loss-on-copy.md @@ -0,0 +1,59 @@ +--- +id: backend-python-language-dict-subclass-attribute-loss-on-copy +domain: backend +category: language +applies_to: [python] +confidence: verified +sources: + - https://docs.python.org/3/reference/expressions.html#dictionary-displays + - https://docs.python.org/3/library/copy.html + - "Local reproduction (python3 3.14.6, 2026-09-08): dict(obj, **{...}) and {**obj, ...} on a dict subclass both yield type(...) == dict with the subclass's instance attribute gone; copy.copy(obj) yields the original subclass with the attribute intact" +last_verified: 2026-09-08 +related: [backend-python-language-mutable-state-traps, databases-transactions-optimistic-vs-pessimistic-locking] +--- + +# Dict Subclass Attributes Lost When Building a New Dict From It + +## When this applies + +You hold an instance of a `dict` subclass that carries extra instance +attributes alongside its key/value contents — an ORM/driver row wrapper +storing optimistic-lock version info, an ETag, or similar out-of-band +metadata — and you need to add or override a key on it before passing it +onward (to a serializer, a cache, an UPDATE builder). + +## Do this + +| Case | Do | +|------|----| +| The mutation must not be visible to other holders of the same reference | `copy.copy(obj)`, then mutate the copy — `copy.copy()` "normally returns an instance of the same type", preserving the subclass and its instance attributes; the top-level container is independent of the original | +| The original may be safely changed temporarily and must be restored before anything else observes it | Mutate in place — `obj[key] = value` — then revert in a `finally`: `obj.pop(key, None)` | +| The receiver is verified to never read the subclass's instance attributes | `dict(obj, **{key: value})` / `{**obj, key: value}` is fine — but the verification has to happen first, not be assumed | + +`dict(obj, **{...})` and `{**obj, ...}` are dictionary displays, and "a +dictionary display yields a new dictionary object" — always plain `dict`, +regardless of the runtime type of `obj`; `**` unpacking only copies key/value +pairs into that new object, never `__class__` or `__dict__`/`__slots__`. Reach +for them only in the third row's case. + +## Edge cases + +| Case | Then | +|------|------| +| Downstream code reads the attribute via `getattr(row, "x", None)` | The loss is silent — `getattr` returns the default with no exception, so the bug shows up as wrong behavior far from the copy site, not a stack trace at it | +| Downstream code reads the attribute directly (`row.x`) | The loss is loud — a plain `dict` raises `AttributeError: 'dict' object has no attribute 'x'` immediately, which is why this trap survives in codebases that always use `getattr` defensively | +| The subclass nests further mutable state (a cache, a list) that must also stay independent of the original | `copy.copy()` is shallow — nested mutable values still alias the original object; only the top-level container and its own attributes are independent | +| The subclass defines `__copy__` | `copy.copy()` calls it — confirm the override also copies instance attributes before relying on it as the fix | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Build `dict(row, **{key: value})` or `{**row, key: value}` from a `dict` subclass carrying instance attributes | `copy.copy(row)` then mutate the copy, or mutate `row` in place and revert in a `finally` | Dictionary displays always construct a plain `dict`; the subclass type and every attribute that lived only on the instance are silently dropped | + +## Sources + +- https://docs.python.org/3/reference/expressions.html#dictionary-displays — "A dictionary display yields a new dictionary object" +- https://docs.python.org/3/library/copy.html — slicing and `.copy()` methods "can create an instance of the base type when copying an instance of a subclass, whereas copy.copy() normally returns an instance of the same type" +- Local reproduction (python3 3.14.6, 2026-09-08): a `VersionedRow(dict)` with instance attribute `observed_version=42`; `dict(row, **{"_schema_gen": "abc"})` and `{**row, "_schema_gen": "abc"}` both produced `type(...) == dict`, `hasattr(..., "observed_version") == False`; `copy.copy(row)` produced `type(...) == VersionedRow`, `observed_version == 42`, and mutating the copy left `row` untouched; `getattr(dict_instance, "observed_version", "DEFAULT")` returned `"DEFAULT"` while `dict_instance.observed_version` raised `AttributeError` +- Field incident (a Python SQLite repository driver): `_read` returns a `_VersionedRow(dict)` carrying `.observed_version` for a conditional UPDATE; `dict(row, **{"_schema_gen": digest})` silently dropped `.observed_version`, turning every conditional UPDATE unconditional — caught by failing concurrency/retry-conflict tests (`1 != 10` on a concurrent-increment test), not by an exception diff --git a/wiki/backend/python/language/iterative-dfs-for-unbounded-graph-depth.md b/wiki/backend/python/language/iterative-dfs-for-unbounded-graph-depth.md new file mode 100644 index 0000000..7561e26 --- /dev/null +++ b/wiki/backend/python/language/iterative-dfs-for-unbounded-graph-depth.md @@ -0,0 +1,69 @@ +--- +id: backend-python-language-iterative-dfs-for-unbounded-graph-depth +domain: backend +category: language +applies_to: [python] +confidence: verified +sources: + - https://docs.python.org/3/library/sys.html#sys.getrecursionlimit + - https://docs.python.org/3/library/sys.html#sys.setrecursionlimit + - https://docs.python.org/3/whatsnew/3.12.html +last_verified: 2026-09-08 +related: [backend-python-language-mutable-state-traps, backend-python-language-dict-subclass-attribute-loss-on-copy] +--- + +# Iterative DFS for Graph Traversal Depth Driven by Input Size + +## When this applies + +You are implementing a DFS-based algorithm in Python (cycle detection, a +dependency/reference graph, compiler diagnostics walking declared-entity +relationships) where the graph's size is driven by user- or module-supplied +content rather than a small fixed set — the traversal depth is not capped by +anything in the language or the domain. + +## Do this + +1. **Use an explicit frame stack, not Python's call stack, whenever depth + scales with input.** `sys.getrecursionlimit()` documents that the limit + "prevents infinite recursion from causing an overflow of the C stack and + crashing Python" — a legitimate, non-infinite traversal that happens to be + deep hits the same wall as a bug would. +2. **Replace the recursive call with a stack of `(node, child_iterator)` + pairs.** Push a new frame when descending into an unvisited child; when the + current frame's iterator is exhausted, mark the node done (black) and pop. + This reproduces the recursive version's visitation order exactly, including + the point at which a node is finalized. +3. **Keep white/gray/black coloring (or an equivalent visited/in-progress/done + set) on the explicit stack version** — a gray node reached again is the + cycle; a black node reached again is safe cross-root memoization. The + iterative rewrite does not change this logic, only where the "call stack" + lives. +4. **When you are tempted to raise `sys.setrecursionlimit(N)` instead, rewrite + iteratively.** The documentation warns raising it "should be done with care, + because a too-high limit can lead to a crash" — it trades a catchable + `RecursionError` for an uncatchable interpreter crash once the C stack (a + separate, platform-dependent limit) is exhausted. + +## Edge cases + +| Case | Then | +|------|------| +| Target is CPython 3.12+ | The recursion limit "now applies only to Python code" (builtins are protected by a separate mechanism) — a pure-Python recursive DFS is still Python code, so the trap and the fix are unchanged | +| The graph is guaranteed small by a hard schema limit (not by convention) | Recursive DFS is fine — state the limit and why it holds, so a later change to the schema is the trigger to revisit | +| Traversal needs to return a value assembled bottom-up (not just visit/color) | Push return values on a parallel results stack keyed by node, popped and combined when the node's frame pops — the iterative shape still supports post-order aggregation | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Write a recursive DFS over a graph whose size is not bounded by the language/domain | An iterative DFS with an explicit `(node, child_iterator)` frame stack | Python's default recursion limit is 1000 frames; a chain of ~500+ nodes reaches it, and the failure mode is a crash on legitimate input, not a bug in the algorithm | +| Raise `sys.setrecursionlimit()` to work around a deep recursive DFS | Rewrite the traversal iteratively | A higher limit only postpones the crash and can turn a catchable `RecursionError` into an interpreter crash when the underlying C stack overflows | + +## Sources + +- https://docs.python.org/3/library/sys.html#sys.getrecursionlimit — "the maximum depth of the Python interpreter stack. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python" +- https://docs.python.org/3/library/sys.html#sys.setrecursionlimit — raising the limit "should be done with care, because a too-high limit can lead to a crash"; "If the new limit is too low at the current recursion depth, a RecursionError exception is raised" +- https://docs.python.org/3/whatsnew/3.12.html — "The recursion limit now applies only to Python code. Builtin functions do not use the recursion limit, but are protected by a different mechanism that prevents recursion from causing a virtual machine crash" +- Local reproduction (python3 3.14.6, 2026-09-08): default `sys.getrecursionlimit()` is 1000; a recursive DFS over a 2000-node chain graph raised `RecursionError: maximum recursion depth exceeded`; an iterative version with an explicit frame stack completed the same graph and returned matching cycle-detection output (`True`) against the recursive version on a 500-node cyclic graph within the default limit +- Field application (compiler diagnostics over declared-entity relationships): recursive DFS crashed at ~500 chained nodes; iterative white/gray/black rewrite with an explicit `frames`/`path_stack` list verified correct on synthetic 2000- and 5000-node graphs with `sys.setrecursionlimit(200)` forced low, cross-root memoization preserved, no correctness regression diff --git a/wiki/backend/python/language/mutable-state-traps.md b/wiki/backend/python/language/mutable-state-traps.md index def146f..a12b1cb 100644 --- a/wiki/backend/python/language/mutable-state-traps.md +++ b/wiki/backend/python/language/mutable-state-traps.md @@ -11,7 +11,7 @@ sources: - https://docs.python.org/3/library/contextvars.html - https://docs.python.org/3/library/asyncio-task.html last_verified: 2026-07-10 -related: [backend-common-concurrency-shared-state-and-pools, testing-strategy-import-time-side-effects] +related: [backend-common-concurrency-shared-state-and-pools, testing-strategy-import-time-side-effects, backend-python-language-dict-subclass-attribute-loss-on-copy, backend-python-language-iterative-dfs-for-unbounded-graph-depth] --- # Mutable State Shared Across Calls and Requests in Python diff --git a/wiki/databases/transactions/optimistic-vs-pessimistic-locking.md b/wiki/databases/transactions/optimistic-vs-pessimistic-locking.md index f29c42b..d176cd1 100644 --- a/wiki/databases/transactions/optimistic-vs-pessimistic-locking.md +++ b/wiki/databases/transactions/optimistic-vs-pessimistic-locking.md @@ -8,7 +8,7 @@ sources: - https://vladmihalcea.com/optimistic-vs-pessimistic-locking/ - https://www.postgresql.org/docs/current/explicit-locking.html last_verified: 2026-07-10 -related: [databases-transactions-isolation-level-selection] +related: [databases-transactions-isolation-level-selection, backend-python-language-dict-subclass-attribute-loss-on-copy] --- # Optimistic vs Pessimistic Locking diff --git a/wiki/debugging/index.md b/wiki/debugging/index.md index 521e247..47c1e88 100644 --- a/wiki/debugging/index.md +++ b/wiki/debugging/index.md @@ -12,7 +12,7 @@ Match your situation to a "load when" line; load only matching pages. | [reproduce-first](methodology/reproduce-first.md) | A bug is reported or behavior is wrong and you are about to investigate or fix; deciding what to capture when full reproduction is impossible (prod-only, timing-dependent, one-off crash) | | [isolate-by-bisection](methodology/isolate-by-bisection.md) | A bug reproduces but its location is unknown; it worked before / works in env A but not env B / fails with one input but not another — binary-searching versions (git bisect), code paths, data, or environment diffs | | [hypothesis-testing](methodology/hypothesis-testing.md) | You have a suspect cause and are about to "try a fix"; several suspects compete and you must pick what to test next; verifying that a fix that "worked" actually addressed the mechanism | -| [probe-path-vs-operation-path](methodology/probe-path-vs-operation-path.md) | A precondition probe (login status, health, connectivity) reports success while the operation it gates fails with an auth/permission error; a browser page-load login check gates direct API calls made with stored cookies; deciding what a preflight probe must exercise under refresh-token cookie auth | +| [probe-path-vs-operation-path](methodology/probe-path-vs-operation-path.md) | A precondition probe (login status, health, connectivity) reports success while the operation it gates fails with an auth/permission error; a browser page-load login check gates direct API calls made with stored cookies; deciding what a preflight probe must exercise under refresh-token cookie auth; a multi-layer pipeline (emit→transport→store→render) is under investigation and upstream layers probe clean while the user-visible symptom is unexplained | | [verify-the-fix](methodology/verify-the-fix.md) | You believe a bug is fixed and are about to close or ship it; the bug "cannot be reproduced anymore" after changes; a previously fixed bug came back; deciding what must pass (repro re-run, both directions, regression test) and what to clean up before closing | ## signals diff --git a/wiki/debugging/methodology/probe-path-vs-operation-path.md b/wiki/debugging/methodology/probe-path-vs-operation-path.md index fcb201c..94c031f 100644 --- a/wiki/debugging/methodology/probe-path-vs-operation-path.md +++ b/wiki/debugging/methodology/probe-path-vs-operation-path.md @@ -6,8 +6,8 @@ applies_to: [general, headless-browser, cookie-auth] confidence: verified sources: - https://github.com/velopert/velog-server/blob/master/src/lib/token.ts -last_verified: 2026-08-14 -related: [debugging-methodology-hypothesis-testing, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts] +last_verified: 2026-09-08 +related: [debugging-methodology-hypothesis-testing, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, debugging-methodology-reproduce-first] --- # A Passing Precondition Probe for a Failing Operation @@ -52,6 +52,7 @@ exercise. | The API layer itself refreshes when handed a valid refresh token (velog's `consumeUser` middleware) | Rotation is returned via `Set-Cookie`; a client that discards response cookies works once and fails on a later run — persist rotated cookies after every authenticated call | | Probe passes, operation starts, then fails auth mid-run | The access token expired during the operation; capture the operation's own error and re-authenticate there — tightening the preflight cannot cover a token that outlives it | | Both probe and operation fail after refresh | The refresh token itself is expired or revoked — re-run the interactive login flow, not the refresh path | +| A multi-layer pipeline (emit → transport → store → render) is under investigation, the transport layer is the prime suspect, and upstream layers (raw emit count, raw listener count) probe clean | Keep probing forward, one layer at a time, toward the layer the user-visible symptom lives at; at the final consuming layer, call its real production entrypoint (the app store's actual action, e.g. `store.getState().startRun(...)`) rather than a hand-rolled substitute — a healthy transport does not by itself explain a downstream symptom, and the true explanation may be a genuine, brief state transition that only the real call path shows | ## Instead of @@ -60,8 +61,10 @@ exercise. | Gate a direct API operation on a browser page-load login check | Gate on the API's identity query with the operation's own client and cookie jar | Page navigation triggers server-side token refresh that the browser persists; the check validates the refresh token while the operation depends on the stored access token | | Retry the operation because the probe "proves" auth is fine | Diff the probe's path against the operation's path first | The contradiction is the diagnosis: two paths, one credential expired on exactly one of them | | Fix the failure by loosening or removing the probe | Move the probe onto the operation's path | The probe was not wrong, it was answering a different question | +| Conclude the transport layer is at fault because emit and listen both probed 100% reliable | Continue to the final consuming layer and read its state through the real production call path before naming a root cause | A healthy transport rules out transport hypotheses; it says nothing about what the state/render layer did — polling the real store found a brief, correct state transition, not a bug | ## Sources - https://github.com/velopert/velog-server/blob/master/src/lib/token.ts — `setTokenCookie` sets `access_token` with `maxAge` 1 hour beside `refresh_token` with 30 days; `consumeUser` middleware refreshes on an expired/near-expiry access token and returns new cookies via `Set-Cookie`, so only clients that persist response cookies stay authenticated - Field reproduction 2026-08-14 (auto-velog pipeline, headless Chromium + stored cookie jar): page-load check returned `STATUS:LOGGED_IN`, the immediately following publish mutation failed with "Not logged in", and a direct `v3.velog.io/graphql` `currentUser` query with the same stored cookies returned null; re-running the login flow (rewriting the cookie store) made the same publish succeed +- Field reproduction 2026-09 (a desktop app's live-event pipeline, Rust emitter → IPC transport → client store → UI): emitter-level and raw-listener probes showed 0 failures across 84/84 events, disproving every transport hypothesis; polling the real store via its production entrypoint (`useRunStore.getState()`) then captured `taskStates: {"t-pm":"assigned"}` at +0.23s transitioning to all-`"accepted"` at +2.4s — the actual explanation of the "never reaches the UI" report was a correct, very brief state transition diff --git a/wiki/debugging/methodology/reproduce-first.md b/wiki/debugging/methodology/reproduce-first.md index 3f16748..8f780ad 100644 --- a/wiki/debugging/methodology/reproduce-first.md +++ b/wiki/debugging/methodology/reproduce-first.md @@ -10,7 +10,7 @@ sources: - https://sre.google/sre-book/effective-troubleshooting/ - https://github.com/mattpocock/skills/blob/main/skills/engineering/diagnosing-bugs/SKILL.md last_verified: 2026-08-24 -related: [debugging-methodology-hypothesis-testing, debugging-concurrency-intermittent-failures, debugging-signals-logs-and-correlation] +related: [debugging-methodology-hypothesis-testing, debugging-concurrency-intermittent-failures, debugging-signals-logs-and-correlation, debugging-methodology-probe-path-vs-operation-path] --- # Building a Reproduction Before Investigating a Bug diff --git a/wiki/frontend/index.md b/wiki/frontend/index.md index 88af713..5644de7 100644 --- a/wiki/frontend/index.md +++ b/wiki/frontend/index.md @@ -18,7 +18,7 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [client-vs-server-state](state/client-vs-server-state.md) | Deciding where/how to store a piece of UI data (fetched entities vs ephemeral UI vs theme/session vs filters/tabs); untangling a global store that has grown unmanageable | | [derived-state](state/derived-state.md) | About to store a value computable from existing state/props (filtered list, count, selected object); two copies of the same fact have drifted; tempted to set state from an effect | -| [effects-usage](state/effects-usage.md) | Writing or reviewing a useEffect (or framework-equivalent watcher); an effect chain causes render loops, flicker, or double-firing; deciding where non-render logic belongs (event handler vs effect vs module scope) | +| [effects-usage](state/effects-usage.md) | Writing or reviewing a useEffect (or framework-equivalent watcher); an effect chain causes render loops, flicker, or double-firing; deciding where non-render logic belongs (event handler vs effect vs module scope); a canvas/WebGL/rAF setup-teardown effect is keyed on an object a hook returns (theme, colors, viewport) that gets a fresh identity on every DOM mutation a scroll/animation library makes (e.g. Lenis toggling `` classes on scroll start/stop), re-running setup though nothing the loop needs changed | ## structure diff --git a/wiki/frontend/state/effects-usage.md b/wiki/frontend/state/effects-usage.md index 3c4dcdf..f9254d4 100644 --- a/wiki/frontend/state/effects-usage.md +++ b/wiki/frontend/state/effects-usage.md @@ -8,7 +8,9 @@ sources: - https://react.dev/learn/you-might-not-need-an-effect - https://react.dev/learn/synchronizing-with-effects - https://react.dev/learn/removing-effect-dependencies -last_verified: 2026-07-10 + - https://react.dev/reference/react/useRef + - https://github.com/darkroomengineering/lenis +last_verified: 2026-09-08 related: [frontend-state-derived-state, frontend-state-client-vs-server-state, frontend-data-fetching-race-conditions] --- @@ -57,6 +59,7 @@ Rules for the effects that remain: | Analytics "view" event fires twice in development | Expected under StrictMode dev remounting; production mounts once. Keep the effect as-is | | Effect must read a value without re-running when it changes | Split the effect so the frequently-changing value lives in its own effect, or move the logic into the event handler that owns the change | | The external subscription is a data store | Subscribing in a raw effect is replaceable with `useSyncExternalStore`, which handles the subscribe/read contract | +| The effect starts/stops an imperative loop (`requestAnimationFrame`, a canvas/WebGL draw loop) and also reads a value that changes identity without changing anything the loop needs (a theme/colors hook, or any hook whose return value is derived by watching ``/`` attribute or class mutations) | Key the setup/teardown effect only on the values that actually start or stop the loop. Mirror the frequently-changing value into a ref via its own tiny effect (`useEffect(() => { colorsRef.current = colors }, [colors])`) and read `colorsRef.current` inside the frame callback — the loop then survives identity churn instead of tearing down and re-seeding | ## Instead of @@ -66,9 +69,13 @@ Rules for the effects that remain: | Add `eslint-disable` on `exhaustive-deps` | Restructure: move the function into the effect, `useCallback` it, or split the effect | Every suppressed dependency reads stale values on later renders | | Guard an effect with a `didRun` ref to survive StrictMode | Write the cleanup that undoes the setup | The double-invoke exists to expose missing cleanup; the ref also masks real remount bugs in production | | Reset state via an effect that watches a prop | Remount with `key={prop}` | The effect version renders one frame of stale state, then re-renders | +| Key a canvas/animation setup effect on every hook value the frame callback reads (`useEffect(setup, [reduced, colors])`) | Key it only on the values that start/stop the loop; read the rest through a ref updated by its own effect | A class-toggling scroll library (Lenis adds/removes `lenis-scrolling`/`lenis-stopped` on its root element on every scroll start/stop) or any hook that derives state from DOM mutations returns a fresh object each time even when the underlying values are identical, so the whole effect — and whatever expensive setup it runs — re-fires on every toggle | ## Sources - https://react.dev/learn/you-might-not-need-an-effect — compute in render, event-handler logic, key-based resets, effect chains, app init, fetching - https://react.dev/learn/synchronizing-with-effects — external systems, cleanup contract, StrictMode dev remount -- https://react.dev/learn/removing-effect-dependencies — honest dependency arrays; change the code, never suppress the linter +- https://react.dev/learn/removing-effect-dependencies — honest dependency arrays; change the code, never suppress the linter; "Object and function dependencies can make your Effect re-synchronize more often than you need" +- https://react.dev/reference/react/useRef — "Changing a ref does not trigger a re-render"; the mechanism for reading a latest value inside a callback (including an animation-frame loop) without depending on it +- https://github.com/darkroomengineering/lenis — `packages/core/src/lenis.ts` `updateClassName()` adds/removes `lenis-scrolling`/`lenis-stopped` on the root element (`` by default) from the scroll start/stop lifecycle (read 2026-09-08) +- Field reproduction 2026-09 (a React canvas hero keyed on `[reduced, colors]` from a theme hook watching `` classes): `seedNodes` calls went 2 → 3 → 4 across one `lenis-scrolling` add/remove; holding colors in a ref and keying the lifecycle effect on `[reduced]` alone stopped the re-seeding, pinned by a regression test diff --git a/wiki/infrastructure/agent-orchestration/session-completion-gates.md b/wiki/infrastructure/agent-orchestration/session-completion-gates.md index 137c5bc..08e2d5e 100644 --- a/wiki/infrastructure/agent-orchestration/session-completion-gates.md +++ b/wiki/infrastructure/agent-orchestration/session-completion-gates.md @@ -7,7 +7,8 @@ confidence: verified sources: - https://code.claude.com/docs/en/hooks - https://csf.tools/reference/nist-sp-800-53/r5/ac/ac-5/ -last_verified: 2026-08-13 + - https://github.com/Leonxlnx/unlazy/blob/main/scripts/stop-hook.mjs +last_verified: 2026-09-08 related: [infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-worktree-isolated-workers, platforms-processes-tool-diagnostics-without-a-failing-exit-code, infrastructure-agent-orchestration-dispatching-after-a-completion-report] --- @@ -65,6 +66,30 @@ parked at an instructed pause and receiving the nudge every turn. own approval; a downstream scheduler that treats those phases as dependency- satisfying then dispatches work against an interface nobody reviewed. +7. **Pair the self-reported phase with a second gate that parses a + machine-verifiable evidence ledger** — a value the worker wrote about itself + is never the gate's only input. A phase field is free for the worker to set: + it is the actor being graded writing its own grade. Have the worker's checker + script run the actual verification commands and write their exit code and + output into a ledger file; have the Stop hook parse that ledger (never re-run + the commands itself — keep the parse/execute split, as `hooks/loop-gate.sh` + does via `gate-check.sh --status`) and block while any entry is unmet or + claimed without evidence: + +| Signal | Who can make it say "done" | Gate treats it as | +|--------|----------------------------|-------------------| +| Self-reported phase (`status.json` `.phase`) | The worker, by calling its own status-update script | Workflow position (items 1–6), not proof of correctness | +| Ledger evidence (`CHECK:`/`EXPECT:` result: exit code + matched output) | Only a run of the actual command — the worker cannot hand-write a passing entry | The completion gate the phase check cannot fake | + +8. **Give the ledger gate its own no-progress release valve, separate from the + harness's re-entry flag.** `stop_hook_active` (item 3) only prevents this + *same* stop event from re-blocking; it does nothing about a session that gets + blocked, tries again next turn, and stays blocked because the ledger never + changes. Hash the ledger's content each time the gate runs; block while the + hash changes between blocks (progress), and release after N consecutive + blocks with an unchanged hash (genuinely stuck) — track the count per session + so one stalled session cannot exhaust another's budget. + ## Edge cases | Case | Then | @@ -76,6 +101,7 @@ parked at an instructed pause and receiving the nudge every turn. | The worker cannot reach a terminal phase because the task is genuinely blocked | Provide a `failed` transition it may record itself; without one, the only escapes are fabricated completion or an eight-block override | | You are the worker and the nudge repeats every turn at an instructed pause | Read it as a gate-vs-prompt mismatch, not as work you skipped; inventing extra work to satisfy it writes code the brief did not ask for | | The gate's terminal set and the phase vocabulary live in different files | Cite both line numbers in the report — the fix belongs in the gate, and the coordinator is the one who can change it | +| The ledger gate (items 7–8) and the phase gate (items 1–6) are both present | Run both; a worker can be at a legitimate terminal phase (`impl_done`, awaiting review) while its ledger is still unmet — the ledger gate blocks independently of phase, because "instructed to pause" and "proved the work" are different claims | ## Instead of @@ -85,10 +111,13 @@ parked at an instructed pause and receiving the nudge every turn. | Treat an unrecognized phase value as unfinished | Treat it as terminal and log the value | A typo or a newly added phase would otherwise trap sessions until someone reads the hook | | Rely on the block message alone to stop a loop | Return early on the harness's re-entry flag first | The message does not bound repetition; the flag is what makes the gate fire once | | Advance your phase to a terminal value to stop a gate firing on you | Hold the instructed phase and report the gate-vs-prompt mismatch to the coordinator | The terminal values that would silence it are the review verdict; writing one makes the reviewed party its own approver, and the scheduler reads it as reviewed | +| Trust a self-reported phase/status field as proof the work is done | Add a gate that parses a ledger of actual command exit codes and output, written only by running the check | The worker can set a phase field to any value for free; it cannot fabricate a ledger entry without the command actually passing | ## Sources - https://code.claude.com/docs/en/hooks — `Stop`/`SubagentStop` input includes `stop_hook_active`; hooks check it and exit early to allow the stop. Claude Code overrides a Stop hook after it blocks eight times in a row without progress (cap adjustable via `CLAUDE_CODE_STOP_HOOK_BLOCK_CAP`) - https://csf.tools/reference/nist-sp-800-53/r5/ac/ac-5/ — NIST SP 800-53 r5 AC-5: "Separation of duties addresses the potential for abuse of authorized privileges and helps to reduce the risk of malevolent activity without collusion. Separation of duties includes dividing mission or business functions and support functions among different individuals or roles" — the phase that records a review verdict belongs to the reviewing role, not the reviewed one +- https://github.com/Leonxlnx/unlazy/blob/main/scripts/stop-hook.mjs — `MAX_BLOCKS = 6`; a per-session, ledger-content-hash-keyed no-progress counter releases the Stop-hook block after 6 consecutive blocks with no ledger change (read 2026-09-08); the project's CHANGELOG records a 1.0.0 "instruction-only" method replaced in 2.0.0 by "gate files, runnable checks, evidence, and an optional Claude Code Stop hook" +- Field reproduction 2026-09-08, dev-loop repo `hooks/loop-gate.sh` (Gate 2, lines 112–174): parses `.dev-loop/gates/*.md` via `gate-check.sh --status` (never executes CHECK commands from the Stop hook), blocks on UNMET/CLAIMED/malformed, releases after `MAX_GATE_BLOCKS=6` consecutive blocks with an unchanged ledger hash — in addition to Gate 1's self-reported-phase check; `tests/loop-gate.bats` "releases after 6 blocks without ledger progress" and "ledger progress resets the no-progress counter" pin the valve - Field reproduction 2026-08-13, dev-loop repo at `fa89dc2`: a worker parked at `impl_done` per `skills/orchestrate/templates/session-prompt.md:75` ("run `… status-update.sh {TASK} impl_done …` and wait") received "verification loop incomplete" on every turn, because `hooks/loop-gate.sh:55` accepts only `done|approved|merged|failed|""` while `skills/orchestrate/scripts/status-update.sh:6` lists `impl_done` as a first-class phase. The three values that would have silenced it are exactly the three `skills/orchestrate/scripts/ready-set.sh:74` counts as dependency-satisfying (`approved|merged|done`), and that file states the rule the fabrication would break: "A dependency counts as satisfied only at `approved` or higher, NOT at impl_done: a task that consumes an unreviewed interface has to be redone when rework changes that signature" - Field reproduction 2026-08-05, dev-loop repo at `95cf947`: `hooks/loop-gate.sh:55` lists `done|approved|merged|failed|""` as terminal, while `skills/orchestrate/templates/session-prompt.md:20` instructs a plan-phase worker to record `plan_ready` and "wait for an approval message. Do NOT write implementation code yet." A worker that followed its prompt exactly was blocked; the `stop_hook_active` early return at line 30 is what kept the block from repeating diff --git a/wiki/infrastructure/agent-orchestration/session-context-token-budget.md b/wiki/infrastructure/agent-orchestration/session-context-token-budget.md index c23e719..e799f8f 100644 --- a/wiki/infrastructure/agent-orchestration/session-context-token-budget.md +++ b/wiki/infrastructure/agent-orchestration/session-context-token-budget.md @@ -13,7 +13,7 @@ sources: - https://platform.claude.com/docs/en/build-with-claude/vision - https://www.anthropic.com/engineering/multi-agent-research-system last_verified: 2026-08-21 -related: [infrastructure-agent-orchestration-worktree-isolated-workers, infrastructure-agent-orchestration-usage-limit-paused-workers, backend-common-llm-context-window-budget] +related: [infrastructure-agent-orchestration-worktree-isolated-workers, infrastructure-agent-orchestration-usage-limit-paused-workers, backend-common-llm-context-window-budget, infrastructure-agent-orchestration-tool-retirement-knowledge-transplant] --- # Token Budget for Long-Lived Coordinator and Worker Agent Sessions diff --git a/wiki/infrastructure/agent-orchestration/tool-retirement-knowledge-transplant.md b/wiki/infrastructure/agent-orchestration/tool-retirement-knowledge-transplant.md new file mode 100644 index 0000000..6b5899b --- /dev/null +++ b/wiki/infrastructure/agent-orchestration/tool-retirement-knowledge-transplant.md @@ -0,0 +1,64 @@ +--- +id: infrastructure-agent-orchestration-tool-retirement-knowledge-transplant +domain: infrastructure +category: agent-orchestration +applies_to: [general] +confidence: field-tested +sources: + - https://github.com/choiyounggi/dev-loop/pull/130 +last_verified: 2026-09-08 +related: [infrastructure-agent-orchestration-session-context-token-budget] +--- + +# Retiring a Plugin or Tool That Carries Methodological Value + +## When this applies + +About to disable, uninstall, or stop loading a plugin, skill library, or tool +because of its token, context, or maintenance cost, and that tool's +skills/instructions encode development methodology rather than only mechanical +automation; deciding whether to just turn it off or preserve what it teaches +first. + +## Do this + +1. **Inventory before disabling.** Read every skill/instruction text the tool + ships in full — not just titles or descriptions — before deciding what to + keep. +2. **Gap-analyze at trigger granularity, not tool granularity.** Compare each + inventoried item against your own knowledge base by the specific situation + that invokes it. A skill can be covered for one trigger and missing for + another inside the same tool, so comparing whole tools against whole + knowledge bases hides partial gaps. +3. **Port only confirmed gaps (merge-before-create).** Create a new page/entry + only where the gap analysis found nothing covering that trigger. Where your + base already owns the trigger but the retired tool's version is deeper + (a sharper edge case, a measured failure mode), merge that content into the + existing entry instead of creating a second one. +4. **Record every already-covered item in a skip list.** Log the item and the + reason it was skipped (which existing page/id already owns its trigger) so + the same comparison is not silently re-run later and a duplicate is not + created by a future pass that forgets the first one happened. +5. **Verify the port before disabling the tool.** Run an automated structural + or duplication check (a wiki/docs lint, a link/id validator) over the + ported material and only disable the tool once that check passes — a port + that has not been checked is not yet complete. + +## Edge cases + +| Case | Then | +|------|------| +| The retired tool's item is deeper or more specific than your existing entry (a benchmark result, a worked failure-mode table) | Merge the stronger content into the existing entry rather than keeping both versions side by side | +| An item's trigger only partially overlaps an existing entry (some situations match, some don't) | Split the comparison at trigger granularity and port only the uncovered sub-trigger, not the whole item | +| No independent reviewer is available to check the port | Run a scripted structural/duplication check as the minimum bar before treating the port as complete | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Disable a plugin/tool outright because of its token or maintenance cost | Inventory its skills, gap-analyze them against your knowledge base at trigger granularity, and port only what's missing | Removing the tool without porting loses methodology it uniquely covered; nothing else in your base captures it once it's gone | +| Port a retired tool's material wholesale into your knowledge base | Compare at trigger granularity first and skip items your base already covers, recording them in a skip list | Wholesale porting creates duplicate entries that pollute the knowledge base and make future lookups less precise | + +## Sources + +- https://github.com/choiyounggi/dev-loop/pull/130 — merged PR: inventoried superpowers 6.3.0 (14 skills) and compound-engineering 2.63.1 (~30 skills + review-persona heuristics) in full before disabling either plugin for context budget; gap-analyzed at trigger granularity, found 8 real gaps, ported them as 8 new pages plus 3 amendments; already-covered items recorded in a skip list (log.md); verified with `node scripts/wiki-lint-prohibitions.js` (71/71 compliant, 0 violations) and `node scripts/wiki-structure-checks.js wiki` (260 pages / 13 indexes / 0 findings) before treating the port as done diff --git a/wiki/infrastructure/agent-orchestration/worktree-isolated-workers.md b/wiki/infrastructure/agent-orchestration/worktree-isolated-workers.md index 99d3c7c..9c0527f 100644 --- a/wiki/infrastructure/agent-orchestration/worktree-isolated-workers.md +++ b/wiki/infrastructure/agent-orchestration/worktree-isolated-workers.md @@ -7,8 +7,9 @@ confidence: verified sources: - https://git-scm.com/docs/git-worktree - https://code.claude.com/docs/en/hooks -last_verified: 2026-08-17 -related: [infrastructure-agent-orchestration-session-completion-gates, infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-shared-run-state, platforms-shells-command-text-inspected-before-execution, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts] + - https://git-scm.com/docs/gitignore +last_verified: 2026-09-08 +related: [infrastructure-agent-orchestration-session-completion-gates, infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-shared-run-state, platforms-shells-command-text-inspected-before-execution, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, qa-process-scope-purity-checks, testing-quality-cross-task-stub-assertions, testing-strategy-real-cli-spot-check-for-new-execution-paths] --- # Writing the Brief for a Worker Confined to Its Own Worktree @@ -47,6 +48,14 @@ wait loop keeps escalating with no error from the task itself. write commands from the brief inside a worktree and require them to complete without an escalation — one probe costs a minute and a bad brief costs every worker's first phase. +7. **Ignore a worker's tool byproduct directories (`.dev-loop/`, `.orchestration/`) + via `$GIT_COMMON_DIR/info/exclude`, not the tracked `.gitignore`.** From inside + any linked worktree, `git rev-parse --git-path info/exclude` resolves to the + *main* checkout's `.git/info/exclude` — one file every worker already shares — + so writing the pattern there once covers every worker. `.gitignore` is a tracked + file: a worker that edits it rides that change through to the final merge as a + permanent, unrequested edit to the user's repo, and N parallel workers each + patching the same `.gitignore` line produce spurious merge conflicts on top. ## Edge cases @@ -72,11 +81,14 @@ wait loop keeps escalating with no error from the task itself. | Disable the escape guardrail so the workers proceed | Rewrite the paths in the brief | The guardrail is what makes parallel workers safe to run against one repo | | Designate a shared scratch directory inside the repo for worker output | Place it outside the repo and pass its path as one named variable | A shared in-repo directory is both a guardrail trip and a write race between workers | | Reuse the coordinator's repo-relative path to a gitignored state directory in a worker's prompt template | Expand it to the absolute path at substitution time and note the directory is absent from the worktree | Ignored files exist only where they were created; the relative form silently resolves to a nonexistent path in every worker, and reads of absolute main-root paths pass the guardrail | +| Add a tool byproduct directory (`.dev-loop/`, `.orchestration/`) to the tracked `.gitignore` to silence `git status` in a worker | Append the pattern to `$GIT_COMMON_DIR/info/exclude` (`git rev-parse --git-path info/exclude`) once | `.gitignore` commits and merges into the user's repo, and every parallel worker patching the same line collides; `info/exclude` is shared across all linked worktrees and never committed | | Trust a Bash-hook guardrail as the only isolation for workers with native file tools | Pair it with a relative-paths-only instruction in the brief and a pre-merge `git status` of the main checkout | The hook inspects only the tool its matcher names; an Edit-tool write to an absolute main-checkout path passes silently — the worker need not be routing around anything for the escape to happen | ## Sources - https://git-scm.com/docs/git-worktree — linked worktrees are separate checkouts sharing one repository; each has its own working directory +- https://git-scm.com/docs/gitignore — patterns that "should be version-controlled and distributed to other repositories via clone" go in `.gitignore`; patterns "specific to a particular repository but which do not need to be shared with other related repositories" go in `$GIT_COMMON_DIR/info/exclude` +- Local reproduction 2026-09-08 (git, macOS): after `git worktree add ../wt1 -b wt1`, `git rev-parse --git-path info/exclude` from inside `wt1` resolved to the main checkout's `.git/info/exclude`; a `.dev-loop/` pattern appended there silenced `git status` for a `.dev-loop/marker` in the worktree and for a `.dev-loop/marker2` in the main checkout — one shared file, not per-worktree - https://code.claude.com/docs/en/hooks — tool-event hook matchers filter on the tool name ("`Bash` matches only the Bash tool"); a hook registered for Bash does not run on Edit/Write calls - Field observation 2026-08-17 (linkly run, worker under a `worktree_escape` Bash-hook guard): the worker modified two `examples/*.lnpl` files in the **main checkout** via its native Edit tool with absolute paths — no block, no log; discovered only when the coordinator's `git pull` failed on local changes (contents happened to match the merged branch, so no damage). The same paths written via Bash redirection would have escalated - Field reproduction 2026-08-05 (groundwork guardrails 1.0.0 `hooks/bash-guard.sh`, `worktree_escape` rule, macOS): from a linked worktree, `cp ./a /b` and `echo z > /f` were both stopped; `cat /f`, `ls /.orchestration`, and `grep -n x /f` all passed. The rule matches an absolute main-root mention together with a write verb (`rm|mv|cp|tee|mkdir|touch|install|dd`) or a redirect to an absolute path diff --git a/wiki/infrastructure/ci-cd/pipeline-structure.md b/wiki/infrastructure/ci-cd/pipeline-structure.md index 88a9f66..c8a33e7 100644 --- a/wiki/infrastructure/ci-cd/pipeline-structure.md +++ b/wiki/infrastructure/ci-cd/pipeline-structure.md @@ -8,7 +8,7 @@ sources: - https://12factor.net/build-release-run - https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows last_verified: 2026-07-10 -related: [infrastructure-containers-image-builds, infrastructure-ci-cd-secrets-handling] +related: [infrastructure-containers-image-builds, infrastructure-ci-cd-secrets-handling, infrastructure-ci-cd-secret-needing-gate-on-fork-prs, infrastructure-ci-cd-review-bot-adopted-as-a-blocking-gate] --- # Ordering and Structuring CI Pipeline Stages diff --git a/wiki/infrastructure/ci-cd/review-bot-adopted-as-a-blocking-gate.md b/wiki/infrastructure/ci-cd/review-bot-adopted-as-a-blocking-gate.md new file mode 100644 index 0000000..a0f4598 --- /dev/null +++ b/wiki/infrastructure/ci-cd/review-bot-adopted-as-a-blocking-gate.md @@ -0,0 +1,55 @@ +--- +id: infrastructure-ci-cd-review-bot-adopted-as-a-blocking-gate +domain: infrastructure +category: ci-cd +applies_to: [general] +confidence: verified +sources: + - https://raw.githubusercontent.com/alibaba/open-code-review/main/action.yml + - https://raw.githubusercontent.com/alibaba/open-code-review/main/cmd/opencodereview/review_cmd.go + - https://raw.githubusercontent.com/alibaba/open-code-review/main/scripts/github-actions/post-review-comments.js +last_verified: 2026-09-08 +related: [qa-process-llm-review-pipelines, infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secret-needing-gate-on-fork-prs] +--- + +# A Review-Bot CLI's Exit Code Trusted as a PR-Blocking CI Gate + +## When this applies + +Adopting a third-party AI/LLM code-review CLI (e.g. alibaba/open-code-review, +"OCR") or its official GitHub Action as a check meant to block a PR on +findings; the tool's own docs describe it as a "review" without stating +whether its exit code reflects findings versus a run failure; the shipped +Action already has a job-failure step and you must decide whether it gates +on severity. + +## Do this + +| Case | Do | +|------|----| +| Wiring a review CLI/Action into a check you want to block a PR | Read the tool's exit-code contract from its source before trusting it; "reports issues" and "exits non-zero" are separate facts. OCR's contract (`cmd/opencodereview/review_cmd.go`, `reviewResultError`): non-zero only for a run-level failure or when every selected item failed; any usable coverage, complete or partial, exits 0 regardless of how many findings it reported | +| The official Action already has a "fail the job" step (OCR's `action.yml` has "Fail job on OCR error", `if: env.OCR_EXIT_CODE != '0'`) | Confirm what feeds that condition before relying on it — it mirrors the CLI's own exit code, so it fails on a crash/incomplete-coverage run, not on findings or severity | +| Checking whether the Action's comment-posting logic can itself gate a review | Verify whether it ever submits a change-requesting review event, or only a comment event. OCR's poster (`scripts/github-actions/post-review-comments.js`) submits every review with `event: "COMMENT"` at every `createReview` call site; there is no request-changes or job-failing path in that script | +| The PR must actually block on findings above a severity threshold | Add your own CI step, after the official Action's steps and in the same job, that parses the tool's structured result and fails the job on the threshold you define. OCR's Action step writes its raw result to a fixed runner path (`ocr review --format json > /tmp/ocr-result.json`), so a follow-up step in the same job reads that file directly — no artifact download needed | +| Deciding which output format to parse | Use the JSON the shipped Action already produces (`--format json`) unless you call `ocr review` yourself with a SARIF flag; OCR implements SARIF v2.1.0 output (`cmd/opencodereview/sarif.go`) for tooling that consumes that format instead | + +## Edge cases + +| Case | Then | +|------|------| +| Calling `ocr review` yourself outside the shipped Action | The same exit contract applies — capture stdout to a file yourself and parse it for findings; `$?` reflects run health, not findings | +| The tool exposes routing inputs by severity/category (OCR has `route_severity_below`, `route_categories`) | These only move findings between inline comments and the PR summary; they do not fail the job — keep the severity-threshold step from Do this as the gate and use routing inputs only for comment placement | +| Your parsing step must tell "review crashed" apart from "review found nothing" | Check the upstream exit code first; a non-zero exit means the result file may be partial or absent, so run your severity parse only after confirming exit 0 | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Rely on a review CLI's exit code, or its official Action's default job outcome, to block a PR on findings | Add your own step that parses the JSON/SARIF result and fails on a severity threshold you define | Verified: OCR's exit code is a crash/incomplete-coverage signal only; every posted review uses `event: "COMMENT"`, never a change-requesting event | +| Assume a review-bot CLI that reports issues also fails its own process or job on them | Read the tool's exit-code contract in its source before wiring it into a gate | Reporting findings and gating a job are independent design choices a tool may or may not couple | + +## Sources + +- https://raw.githubusercontent.com/alibaba/open-code-review/main/action.yml — "Run OpenCodeReview" step (`ocr review "${ARGS[@]}" > /tmp/ocr-result.json`), "Fail job on OCR error" step (`if: env.OCR_EXIT_CODE != '0'`), "Post review comments" step (`if: env.OCR_EXIT_CODE == '0'`) +- https://raw.githubusercontent.com/alibaba/open-code-review/main/cmd/opencodereview/review_cmd.go — `reviewResultError`: "The exit contract is: non-zero only for a run-level failure, or when every selected item failed. Any usable coverage — even incomplete — exits 0" +- https://raw.githubusercontent.com/alibaba/open-code-review/main/scripts/github-actions/post-review-comments.js — every `createReview` call passes `event: "COMMENT"` (verified at three call sites); no request-changes or job-failing path found in the file diff --git a/wiki/infrastructure/ci-cd/secret-needing-gate-on-fork-prs.md b/wiki/infrastructure/ci-cd/secret-needing-gate-on-fork-prs.md new file mode 100644 index 0000000..91c9b79 --- /dev/null +++ b/wiki/infrastructure/ci-cd/secret-needing-gate-on-fork-prs.md @@ -0,0 +1,57 @@ +--- +id: infrastructure-ci-cd-secret-needing-gate-on-fork-prs +domain: infrastructure +category: ci-cd +applies_to: [github-actions] +confidence: verified +sources: + - https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target + - https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/ + - https://github.com/anthropics/claude-code-action/blob/main/docs/security.md +last_verified: 2026-09-08 +related: [infrastructure-ci-cd-secrets-handling, infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-review-bot-adopted-as-a-blocking-gate] +--- + +# A Secret-Needing Check Gate on Pull Requests From Forks + +## When this applies + +A CI check needs a repo secret (API key, OAuth token, cloud credential) to run +against PRs that can arrive from forks — most commonly an LLM/agent-run review +or gate. Plain `pull_request` withholds secrets from fork-originated runs, so +the check silently never authenticates and effectively no-ops instead of +failing loudly. + +## Do this + +| Case | Do | +|------|----| +| A fork-originated PR must run a secret-authenticated gate | Trigger on `pull_request_target`, not `pull_request` — it runs in the base repo's context, so secrets are present | +| `pull_request_target` would start the job for any PR, from anyone | Add a job-level `if:` that restricts to trusted actors before any step runs, e.g. `github.event.pull_request.head.repo.full_name == github.repository \|\| contains(fromJSON('["user-a","user-b"]'), github.event.pull_request.user.login)` | +| The check needs to read the PR's changes | Check out the base ref only (no `ref:` override on `actions/checkout`) and read PR content exclusively through `gh pr diff ` / `gh pr view `; the PR head stays out of the runner workspace | +| The PR diff or description contains instruction-shaped text | Tell the agent explicitly that such text is itself a finding to report, never an instruction to follow — untrusted content is data, not a prompt | +| A downstream step must act on the agent's verdict | Transport it as validated structured output (e.g. a `--json-schema` contract with required `verdict`/`summary`/`findings` fields) rather than a free-form comment or an ad hoc file the agent could skip writing | +| The agent crashes, times out, or emits malformed output | Treat missing or unparsable structured output as a failing gate, not a passing one (fail closed) | + +## Edge cases + +| Case | Then | +|------|------| +| The job must build or execute the PR's code itself, not just review its text | This checkout-avoidance pattern does not cover that case — use the two-workflow split instead: an unprivileged `pull_request` job builds/tests untrusted code with no secrets and uploads results as artifacts, a separate privileged `workflow_run` job (running on the base ref, with secrets) consumes those artifacts | +| No fixed trusted-account list is maintainable (open, high-volume external contributors) | Gate the job on a maintainer-applied label (e.g. `safe to test`) instead of a login allowlist, and re-check the label/approval on every new commit — a label applied once does not vouch for commits pushed afterward | +| The gate posts a comment back to the PR | Scope `permissions:` to exactly what that needs (e.g. `pull-requests: write`), not broader, since the job already runs with base-repo secrets | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Switch a fork-PR gate from `pull_request` to `pull_request_target` and keep `ref: ${{ github.event.pull_request.head.sha }}` in the checkout step | Drop the `ref:` override so checkout stays on the base ref; read PR content only via `gh pr diff`/`gh pr view` | `pull_request_target` plus an explicit head checkout is the "pwn request" pattern: untrusted PR content gets both the secret context and a writable workspace in the same job | +| Let a crashed or erroring agent step read as a passing check | Fail closed: missing/malformed structured output exits non-zero | A crash defaulting to green is a dangerous default-allow — the exact failure mode the gate exists to prevent | +| Rely on the `pull_request_target` trigger alone as the security boundary | Pair it with an explicit job-level `if:` trust check | The trigger only decides secret availability; without the `if:`, the job body still runs for a PR from anyone | + +## Sources + +- https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target — "With the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository"; "Running untrusted code on the pull_request_target trigger may lead to security vulnerabilities" +- https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/ — "Combining pull_request_target workflow trigger with an explicit checkout of an untrusted PR is a dangerous practice that may lead to repository compromise"; mitigations include the two-workflow split and label-gated re-checks +- https://github.com/anthropics/claude-code-action/blob/main/docs/security.md — "pull_request_target and workflow_run execute with the base repository's secrets"; "Do not check out an untrusted ref into the workspace root before this action"; "Preferred — check out the base ref (default)" +- Field reproduction (dev-loop `.github/workflows/wiki-agent-gate.yml` + `tests/wiki-agent-gate.bats`, 12 tests, each guard paired with a negative control; read directly 2026-09-08): trust `if:`, base-ref-only checkout, `gh pr diff` read path, `--json-schema` verdict, fail-closed enforce step. The "instruction-shaped text is a finding" stance is that repo's own gate prompt, not the claude-code-action doc diff --git a/wiki/infrastructure/ci-cd/secrets-handling.md b/wiki/infrastructure/ci-cd/secrets-handling.md index 0dea6c1..3d13d5c 100644 --- a/wiki/infrastructure/ci-cd/secrets-handling.md +++ b/wiki/infrastructure/ci-cd/secrets-handling.md @@ -12,7 +12,7 @@ sources: - https://cli.github.com/manual/gh_auth_login - https://docs.github.com/en/search-github/searching-on-github/searching-issues-and-pull-requests last_verified: 2026-08-06 -related: [infrastructure-containers-image-builds, infrastructure-ci-cd-pipeline-structure] +related: [infrastructure-containers-image-builds, infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secret-needing-gate-on-fork-prs] --- # Credentials Flowing Through Build and Deploy Pipelines diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 86c2ac3..48137f0 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -15,14 +15,15 @@ Match your situation to a "load when" line; load only matching pages. | [control-signals-vs-primary-artifacts](agent-orchestration/control-signals-vs-primary-artifacts.md) | An orchestrator is about to restart, discard, merge, or keep waiting on a worker based on a status file, a watcher's exit code, or a heartbeat; a monitor reports a worker dead while it is committing; a worker's status write produced no output and you must decide whether it landed; distinguishing alive-and-progressing from stalled from dead; several workers went quiet at once while every liveness check passes (usage-limit pause); a dispatch issued right after a worker's done signal fails runtime-unavailable | | [shared-run-state](agent-orchestration/shared-run-state.md) | Several agent/worker sessions coordinate through files in one repository (status directory, briefs, escalations, claim files); choosing the path layout for that state; starting an orchestration in a repo that may already have one running; a watcher woke on a task id it did not create; the default branch moved during a run; a coordinator is about to reset a task's status file while re-delivering a prompt | | [pane-delivery-confirmation](agent-orchestration/pane-delivery-confirmation.md) | An orchestrator drives another program through a terminal multiplexer (`tmux send-keys` + `capture-pane`) and must decide whether the input was consumed, retry, or escalate; a pane diff is being used as delivery evidence; the target echoes but never runs the input; deciding *where* in a pane capture to search for a collapsed paste marker whose position depends on payload size, or what to report when the pane's input-box chrome cannot be located at all | -| [session-completion-gates](agent-orchestration/session-completion-gates.md) | Writing a Stop/completion hook that blocks a worker session from ending while its phase is non-terminal; the gate fires on a worker that followed its own prompt; deciding the terminal phase set, the unknown-phase default, and how the gate bounds its own repetition; you are the worker the gate repeats on at an instructed pause and are deciding whether to advance your phase to silence it | +| [session-completion-gates](agent-orchestration/session-completion-gates.md) | Writing a Stop/completion hook that blocks a worker session from ending while its phase is non-terminal; the gate fires on a worker that followed its own prompt; deciding the terminal phase set, the unknown-phase default, and how the gate bounds its own repetition; you are the worker the gate repeats on at an instructed pause and are deciding whether to advance your phase to silence it; deciding whether a self-reported phase field is sufficient evidence of completion, or whether the gate needs a second check against a machine-verifiable execution ledger with its own no-progress release valve | | [dispatching-after-a-completion-report](agent-orchestration/dispatching-after-a-completion-report.md) | A worker reported completion and the orchestrator wants to hand that same terminal or runtime slot its next task; a start/dispatch call fails with a runtime-unavailable-class error moments after a completion report; a task reached a terminal `failed` status with no worker having worked on it; deciding a settled dispatch's next owner (transfer, release, or retain) and how to retry a failed start without spending the task's attempt budget | | [unattended-worker-questions](agent-orchestration/unattended-worker-questions.md) | A worker agent raises a question through its own interactive UI (a numbered chooser, a confirmation/trust/re-auth screen) with no human at that terminal; a worker is flagged stalled with a live terminal and no task-level error; a worker reports a decision it assumed rather than asked; designing the channel a worker uses to ask its coordinator for a decision | | [usage-limit-paused-workers](agent-orchestration/usage-limit-paused-workers.md) | Several workers billed to one account go quiet within minutes of each other while every liveness check passes; a worker's terminal shows a `You've hit your session/weekly/Opus limit · resets …` notice; deciding whether to restart, replace, or wait on a worker with no task-level error; writing the prompt that resumes a worker after a usage window resets | -| [worktree-isolated-workers](agent-orchestration/worktree-isolated-workers.md) | Authoring the brief/output contract for parallel workers each confined to its own git worktree; workers stall at the same phase with no task-level error; deciding where shared or produced artifacts live and which direction (read vs write) a worktree guardrail stops; a guardrail escalates on read-only access to another worktree; the isolation guard is a Bash-command hook while workers also edit files with native Edit/Write tools | +| [worktree-isolated-workers](agent-orchestration/worktree-isolated-workers.md) | Authoring the brief/output contract for parallel workers each confined to its own git worktree; workers stall at the same phase with no task-level error; deciding where shared or produced artifacts live and which direction (read vs write) a worktree guardrail stops; a guardrail escalates on read-only access to another worktree; the isolation guard is a Bash-command hook while workers also edit files with native Edit/Write tools; a worker's tool byproduct directory (`.dev-loop/`, `.orchestration/`) dirties `git status` and the fix under consideration is editing the tracked `.gitignore` | | [autonomous-decision-rulings](agent-orchestration/autonomous-decision-rulings.md) | An unattended agent hits a decision its plan does not answer and must choose between stopping to ask and proceeding; a run stalls on questions no human needed to see; deciding which decision categories require a human; recording autonomous decisions for audit; resuming after interruption/compaction without re-dispatching completed work | | [session-context-token-budget](agent-orchestration/session-context-token-budget.md) | Planning or running long-lived coordinator/worker agent sessions and deciding when to compact or clear context; a run's cost is dominated by cache reads; screenshots or large file reads are entering a long-lived session; choosing slot counts / per-phase token budgets for an orchestrated run | | [code-graph-as-orientation-layer](agent-orchestration/code-graph-as-orientation-layer.md) | A repository carries a locally built code knowledge graph (graphify `graphify-out/graph.json` or similar) and an agent is about to plan, decompose, or estimate the blast radius of a change; an orchestrator needs each parallel task's file set before dispatch; deciding whether a graph hit can stand as plan evidence; checking whether the graph is fresh enough to use | +| [tool-retirement-knowledge-transplant](agent-orchestration/tool-retirement-knowledge-transplant.md) | About to disable a plugin, skill library, or tool for token/context/maintenance cost that encodes development methodology; deciding whether to inventory and gap-analyze its skills against your knowledge base before turning it off; recording already-covered items in a skip list so re-comparison doesn't create duplicates | ## ci-cd @@ -31,6 +32,8 @@ Match your situation to a "load when" line; load only matching pages. | [pipeline-structure](ci-cd/pipeline-structure.md) | Creating or restructuring a CI pipeline; CI is slow, unreliable, or reports failures too late; deciding where a new check/stage belongs | | [secrets-handling](ci-cd/secrets-handling.md) | A build or deploy step needs credentials (registry, cloud, private packages, signing); reviewing how secrets flow through CI; a secret leaked (log/chat/commit) and deciding the response | | [changed-files-only-gates](ci-cd/changed-files-only-gates.md) | A CI step builds a changed-files list in the shell and passes it to `prettier`/`eslint`/a checker as operands; deciding whether a green gate means "no violations" or "nothing examined"; the list is empty because a base ref did not resolve; the script runs under zsh where an unquoted variable does not word-split; placing a probe file to prove the gate can fail | +| [secret-needing-gate-on-fork-prs](ci-cd/secret-needing-gate-on-fork-prs.md) | A CI check needs a repo secret (API key, OAuth token, cloud credential) to gate fork-originated PRs, including LLM/agent-run gates; plain `pull_request` silently withholds secrets so the gate never authenticates; switching to `pull_request_target` while still checking out the PR head; deciding how to transport and fail-close an agent's verdict | +| [review-bot-adopted-as-a-blocking-gate](ci-cd/review-bot-adopted-as-a-blocking-gate.md) | Adopting a third-party AI/LLM code-review CLI or its official GitHub Action as a PR-blocking CI check; the tool's docs don't state whether its exit code reflects findings versus a run failure; a shipped Action already has a "fail the job" step and you must confirm what triggers it; the tool exposes a severity/category routing input and you must decide whether it gates the job or only reroutes comments; adding your own severity-threshold step that parses the tool's JSON/SARIF output | ## config diff --git a/wiki/platforms/index.md b/wiki/platforms/index.md index 8200fae..827b518 100644 --- a/wiki/platforms/index.md +++ b/wiki/platforms/index.md @@ -67,9 +67,10 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [compiler-sysroot-on-macos](toolchains/compiler-sysroot-on-macos.md) | On macOS a non-Xcode compiler (Homebrew/MacPorts LLVM) fails with `'stdio.h' file not found`, `ld: library 'System' not found`, or a `-Wmissing-sysroot` warning naming an SDK directory that does not exist; a build works under `/usr/bin/clang` but not under the toolchain the project requires; choosing between `-isysroot`, `SDKROOT`, `CPATH`, and `LIBRARY_PATH`; separating a toolchain precondition from a code regression when only the compiled tests fail | -| [flag-availability-at-the-execution-site](toolchains/flag-availability-at-the-execution-site.md) | A diff adds a CLI flag, a new subcommand, or calls a new SDK/API method against a pinned dependency; reviewing such a diff and the tool/dependency version installed in CI, on a teammate's machine, or in the deploy image is not stated in the PR; a flag that worked on the author's machine fails, or silently no-ops, wherever it actually runs | +| [flag-availability-at-the-execution-site](toolchains/flag-availability-at-the-execution-site.md) | A diff adds a CLI flag, a new subcommand, or calls a new SDK/API method against a pinned dependency; reviewing such a diff and the tool/dependency version installed in CI, on a teammate's machine, or in the deploy image is not stated in the PR; a flag that worked on the author's machine fails, or silently no-ops, wherever it actually runs; deciding whether a flag/subcommand is valid at all when the only check performed is grepping the tool's own `--help` output | | [version-management](toolchains/version-management.md) | "Works on my machine" from tool-version drift; a project needs a pinned language/tool version (.nvmrc, .python-version, .tool-versions); making CI use the same versions as local; onboarding a machine reproducibly; a script/cron/CI step can't find a version-managed binary (shims absent in non-interactive shells); deciding where lockfiles fit in reproducibility | | [environment-resync-removes-undeclared-packages](toolchains/environment-resync-removes-undeclared-packages.md) | A package that was working (pytest, ruff, a scratch library) vanished after an unrelated dependency change and imports fail across unrelated test files; adding or dropping a dependency while a long job/test run/experiment is in flight; deciding whether dev-only tools belong in a dependency group or an ad-hoc `pip install`; deciding whether a manager's command prunes packages absent from the lockfile (`uv add` vs `uv remove` vs `uv sync` vs `uv run` exactness) | +| [regeneration-silently-drops-hand-edited-state](toolchains/regeneration-silently-drops-hand-edited-state.md) | A generator-owned file (XcodeGen's `.xcodeproj` regenerated from `project.yml`, or a similar codegen-plus-hand-edit setup) is about to be regenerated to make one small change; deciding whether to trust regenerated output before committing it; a GUI-added target, a shared scheme, or another hand-edit is missing after regeneration even though the build still succeeds | ## Planned (unseeded categories) diff --git a/wiki/platforms/processes/non-interactive-cli-invocation.md b/wiki/platforms/processes/non-interactive-cli-invocation.md index 3073b8d..fbeee19 100644 --- a/wiki/platforms/processes/non-interactive-cli-invocation.md +++ b/wiki/platforms/processes/non-interactive-cli-invocation.md @@ -12,7 +12,7 @@ sources: - https://man7.org/linux/man-pages/man1/timeout.1.html - https://en.wikipedia.org/wiki/Bracketed-paste last_verified: 2026-08-04 -related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing, platforms-processes-parsing-cli-structured-output, platforms-shells-env-var-off-switches, infrastructure-agent-orchestration-pane-delivery-confirmation, testing-strategy-signal-delivery-to-a-process-under-test, qa-deliverables-command-transcripts-in-a-document] +related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing, platforms-processes-parsing-cli-structured-output, platforms-shells-env-var-off-switches, infrastructure-agent-orchestration-pane-delivery-confirmation, testing-strategy-signal-delivery-to-a-process-under-test, qa-deliverables-command-transcripts-in-a-document, platforms-toolchains-flag-availability-at-the-execution-site] --- # Invoking a Prompt-Capable CLI from a Script or Agent Harness diff --git a/wiki/platforms/toolchains/environment-resync-removes-undeclared-packages.md b/wiki/platforms/toolchains/environment-resync-removes-undeclared-packages.md index 6bcf90b..6f0d23f 100644 --- a/wiki/platforms/toolchains/environment-resync-removes-undeclared-packages.md +++ b/wiki/platforms/toolchains/environment-resync-removes-undeclared-packages.md @@ -10,7 +10,7 @@ sources: - https://docs.astral.sh/uv/reference/cli/ - https://peps.python.org/pep-0735/ last_verified: 2026-08-07 -related: [platforms-toolchains-version-management, platforms-processes-background-services] +related: [platforms-toolchains-version-management, platforms-processes-background-services, platforms-toolchains-regeneration-silently-drops-hand-edited-state] --- # A Dependency Command Resyncing the Whole Environment and Deleting Undeclared Packages diff --git a/wiki/platforms/toolchains/flag-availability-at-the-execution-site.md b/wiki/platforms/toolchains/flag-availability-at-the-execution-site.md index 2027fb4..1c899ae 100644 --- a/wiki/platforms/toolchains/flag-availability-at-the-execution-site.md +++ b/wiki/platforms/toolchains/flag-availability-at-the-execution-site.md @@ -8,8 +8,9 @@ sources: - https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-changes.html - https://semver.org/ - https://protobuf.dev/programming-guides/proto3/ -last_verified: 2026-08-14 -related: [platforms-toolchains-version-management, platforms-tools-bsd-vs-gnu-cli, backend-common-integrations-externally-owned-defaults] + - https://code.claude.com/docs/en/cli-reference +last_verified: 2026-09-08 +related: [platforms-toolchains-version-management, platforms-tools-bsd-vs-gnu-cli, backend-common-integrations-externally-owned-defaults, platforms-processes-non-interactive-cli-invocation] --- # A CLI Flag, Subcommand, or API Method That Exists Locally but Not Where It Runs @@ -20,6 +21,9 @@ A change adds a CLI flag, a new subcommand, or a call to a new SDK/API method ag a pinned dependency. Also when reviewing such a diff and the PR does not state the tool/dependency version installed in CI, on teammates' machines, or in the deploy image — the author's shell proves the flag exists on the author's shell only. +Also when deciding whether a flag or subcommand is valid at all — before relying +on it in code or honoring a plan decision that specifies it — and the only check +performed so far is grepping the tool's own `--help` output for the name. ## Do this @@ -29,6 +33,7 @@ image — the author's shell proves the flag exists on the author's shell only. | Flag added to a command that runs on a user's/teammate's machine | State the minimum tool version in the change and add a preflight check that runs ` --version` and fails with a named-version error below the minimum, rather than letting the flag itself be the first thing that fails | | A new subcommand | Confirm the subcommand appears in ` help`'s own command list at the execution site before merging, not just that it runs on the author's machine — an absent subcommand is the loudest failure in this class (non-zero exit, "unknown command"), so catch it at review instead of at run time | | A new SDK/API method on a pinned dependency | Check the dependency version the lockfile that the deploy/runtime environment actually installs from resolves to, not the version under the author's local `node_modules`/`site-packages`; bump the pin in the same change that calls the new method | +| Deciding whether a flag/subcommand is valid at all, with only a `--help` grep performed so far | Verify with a real invocation: run the exact command with the flag and read the exit code, or a structured-output error field (e.g. `--output-format json`'s `is_error`). `--help` text and the argument parser are maintained separately, so an omission from `--help` is a documentation gap, not evidence the parser rejects the flag | Version pins exist for exactly this: a new flag, subcommand, or method is new *functionality*, and semantic versioning's own contract is that new backward-compatible @@ -43,6 +48,7 @@ one without the other lets them drift. | The flag/field exists syntactically but is a silent no-op in the older/deployed version | This is the case CLI flag parsers usually don't produce (unknown flags typically hard-error); it shows up instead in structured inputs — a new field on a protobuf-based request, an SDK constructor kwarg, a config-file key. Proto3 "preserves unknown fields... in the serialized output" (protobuf.dev) — but an older server's generated code has no accessor for a field it doesn't know, so the call returns 200 while the intended behavior never happens. Add an explicit assertion (response field present / behavior observed), not just an exit-code check | | The tool is version-managed, so the authoring shell and the CI shell resolve different binaries | Confirm which binary each shell actually resolves — pin file vs PATH lookup can diverge silently ([platforms-toolchains-version-management]) | | The same flag name means something different in the execution site's userland | A flag that exists at both sites is not the same guarantee as a flag that *behaves* the same at both — check the other userland's own docs, not just that the name is present ([platforms-tools-bsd-vs-gnu-cli]) | +| The tool's official docs site documents the flag but its own `--help` output omits it | Trust the docs site plus one real invocation over `--help`; treat the combination as the flag being valid rather than as a conflict to resolve before using it | ## Instead of @@ -51,9 +57,13 @@ one without the other lets them drift. | Trust that a flag works because it ran on your machine | Run the same command against the execution site's actual tool version (CI image, deploy image, teammate's pin) before merging | Your shell's version is one instance among several the diff must run under | | Add the flag and assume a missing one always errors loudly | Check whether the input is structured (protobuf field, SDK kwarg, config key) where unknown values are silently preserved or dropped rather than rejected | Hard-error-on-unknown is a CLI-parser convention, not a universal one | | Bump only the pin, or only add the flag, in separate changes | Land the version bump and the flag/method that depends on it together | A pin bump that lands later (or not at all) leaves the flag calling a version that doesn't have it | +| Remove a flag from a plan or implementation because it does not appear in `--help` | Verify with a real invocation (exit code / structured error field) and check the vendor's separate CLI reference before removing anything a plan already specified | `--help` completeness is not the parser's source of truth; a review round can catch the removal and require restoring it, costing a full rework cycle | ## Sources - https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-changes.html — concrete version-gated CLI surface: the `--copy-props` parameter is new to `aws s3` commands in CLI v2 ("The AWS CLI version 2 adds the `--copy-props` parameter"); `aws ecr get-login-password` is "available in the AWS CLI version 1.17.10 and later, and the AWS CLI version 2" — the same binary name, gated by a specific version - https://semver.org/ — clause 7: "Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backward compatible functionality is introduced to the public API" — the versioning contract that a new flag/method is a version fact, not just a code fact - https://protobuf.dev/programming-guides/proto3/ — "Proto3 messages preserve unknown fields and include them during parsing and in the serialized output" — unknown fields survive on the wire but are not exposed to code compiled without them, which is the silent-no-op mechanism for structured (non-CLI) inputs +- https://code.claude.com/docs/en/cli-reference — documents `--max-turns`: "Limit the number of agentic turns (print mode only)", example `claude -p --max-turns 3 "query"` — present in the official CLI reference though absent from `claude --help` output (v2.1.263) +- Reproduction 2026-09-08 (claude 2.1.263): `claude --help` contains zero occurrences of "max-turns"; `claude -p --max-turns 3 --output-format json "reply with exactly: ok"` returned `is_error:false`, `num_turns:1`, `result:"ok"` +- Field evidence 2026-09 (an orchestrated task in a Rust/CLI repo): a session concluded `--max-turns` was invalid based solely on its absence from `claude --help`, removed it from the implementation, and a review round required restoring it after a real invocation showed it worked diff --git a/wiki/platforms/toolchains/regeneration-silently-drops-hand-edited-state.md b/wiki/platforms/toolchains/regeneration-silently-drops-hand-edited-state.md new file mode 100644 index 0000000..4cdac5c --- /dev/null +++ b/wiki/platforms/toolchains/regeneration-silently-drops-hand-edited-state.md @@ -0,0 +1,74 @@ +--- +id: platforms-toolchains-regeneration-silently-drops-hand-edited-state +domain: platforms +category: toolchains +applies_to: [xcodegen, ios, general] +confidence: verified +sources: + - https://github.com/yonaskolb/XcodeGen + - https://github.com/yonaskolb/XcodeGen/blob/master/Docs/FAQ.md + - https://github.com/yonaskolb/XcodeGen/issues/515 + - https://github.com/yonaskolb/XcodeGen/issues/572 +last_verified: 2026-09-08 +related: [platforms-toolchains-environment-resync-removes-undeclared-packages, platforms-toolchains-version-management] +--- + +# Regenerating a Generator-Owned Project File That Is Also Hand-Edited + +## When this applies + +A code-generation tool writes a file the repository also commits (XcodeGen's +`.xcodeproj` from `project.yml`, or a similar codegen-plus-hand-edit setup) and +that committed file has also been edited outside the generator's spec — a target +or scheme added through the Xcode GUI, a manual patch to generated code. You are +about to run the regenerate command to make one small, unrelated change. + +## Do this + +1. Treat the spec (`project.yml`) as authoritative only for what it declares. The + generator performs a full regeneration, so anything present in the committed + output but absent from the spec has no representation there and is deleted, + not merged, on the next run. Confirmed on XcodeGen: a scheme created via the + Xcode GUI defaults to shared, and "if a user forgets to uncheck that when + creating a custom scheme, their schemes will be overwritten the next time + they generate" (issue #515); a second report independently confirms "running + xcodegen will overwrite that xcscheme file" (issue #572). +2. Right after running the regenerate command, check the delta before accepting + it: `git diff --stat `. A nonzero deletion count on a run + meant to add one key is the signal that the spec is missing something the + committed file carried — targets, schemes, or build settings added outside it. +3. When there is a loss, `git checkout -- ` to discard the + regenerated file, then re-apply only the specific change you wanted (hand-edit + the one key, or first add the missing target/scheme to the spec and regenerate + again) so the regeneration lands on a spec that actually has parity with what + is committed. +4. Close the gap at the source rather than repeatedly discarding regen output: + add any GUI-created target/scheme to `project.yml` so it becomes declared and + survives regeneration. XcodeGen's own FAQ frames a committed `.xcodeproj` + alongside `project.yml` as a "halfway step" — the fully supported path is + gitignoring the generated file entirely, which removes this failure mode by + removing the possibility of a hand-edit existing outside the spec. + +## Edge cases + +| Case | Then | +|------|------| +| Diff shows only additions/reorderings you expect, nothing removed | Regeneration is safe to accept as-is | +| The lost state is small and easy to redeclare (one scheme, one target) | Add it to the spec, regenerate again, confirm the diff is now clean, then commit | +| The repo already gitignores the generated project file | This failure mode does not apply — nothing hand-edited exists to lose; skip the diff-before-accept step | +| A CI job runs the generator on every push | The same undeclared-state loss happens silently there too; add a `git diff --stat` check as a CI gate, not only a local habit | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Run the regenerate command and commit whatever it produces | `git diff --stat` the regenerated file first, and `git checkout` it back if targets/schemes were deleted | The generator treats its spec as the sole source of truth; anything committed but undeclared is deleted, and the build still succeeds so the loss surfaces later, in CI | +| Assume the spec already describes everything in the committed generated file | Diff the two after any GUI/manual change and update the spec before the next regeneration | The spec does not absorb out-of-band edits automatically; it only ever reflects what was written into it | + +## Sources + +- https://github.com/yonaskolb/XcodeGen — "Generate projects on demand and remove your `.xcodeproj` from git, which means no more merge conflicts!"; `xcodegen generate` "will look for a project spec in the current directory called `project.yml`" +- https://github.com/yonaskolb/XcodeGen/blob/master/Docs/FAQ.md — "Can I still check in my project — Absolutely... But you can also check it in as a halfway step" +- https://github.com/yonaskolb/XcodeGen/issues/515 — GUI-created shared schemes "will be overwritten the next time they generate" +- https://github.com/yonaskolb/XcodeGen/issues/572 — "running xcodegen will overwrite that xcscheme file" +- Field reproduction (an iOS repo with committed `.xcodeproj` + `project.yml`, xcodegen 2.45.4, 2026-09): regenerating to add one localhost ATS exception deleted 118 pbxproj lines including the test target's `PBXNativeTarget` and the shared `.xcscheme` file; `git status`/`git diff --stat` surfaced it before commit diff --git a/wiki/platforms/toolchains/version-management.md b/wiki/platforms/toolchains/version-management.md index b925512..18be3fb 100644 --- a/wiki/platforms/toolchains/version-management.md +++ b/wiki/platforms/toolchains/version-management.md @@ -10,7 +10,7 @@ sources: - https://mise.jdx.dev/configuration.html - https://docs.npmjs.com/cli/v11/configuring-npm/package-json last_verified: 2026-07-10 -related: [platforms-processes-background-services, platforms-shells-portable-shell-scripts, platforms-toolchains-compiler-sysroot-on-macos, platforms-toolchains-environment-resync-removes-undeclared-packages] +related: [platforms-processes-background-services, platforms-shells-portable-shell-scripts, platforms-toolchains-compiler-sysroot-on-macos, platforms-toolchains-environment-resync-removes-undeclared-packages, platforms-toolchains-regeneration-silently-drops-hand-edited-state] --- # Pinning Tool Versions So Every Machine Runs the Same Toolchain diff --git a/wiki/qa/deliverables/quantitative-claims-in-a-published-document.md b/wiki/qa/deliverables/quantitative-claims-in-a-published-document.md index 0973ed1..049d2e4 100644 --- a/wiki/qa/deliverables/quantitative-claims-in-a-published-document.md +++ b/wiki/qa/deliverables/quantitative-claims-in-a-published-document.md @@ -8,8 +8,7 @@ sources: - https://google.github.io/styleguide/docguide/best_practices.html - https://www.writethedocs.org/guide/writing/docs-principles/ last_verified: 2026-08-06 -related: - [qa-deliverables-generated-artifacts-as-deliverable-source, qa-document-verification-spec-document-gates, testing-quality-spec-artifact-checks] +related: [qa-deliverables-generated-artifacts-as-deliverable-source, qa-document-verification-spec-document-gates, testing-quality-spec-artifact-checks, qa-document-verification-sweeping-pre-gate-citations-for-fabrication, qa-document-verification-editing-a-gated-document] --- # Numbers in a Document About to Be Published Outside the Team diff --git a/wiki/qa/document-verification/editing-a-gated-document.md b/wiki/qa/document-verification/editing-a-gated-document.md index 9a9552a..21b0d6b 100644 --- a/wiki/qa/document-verification/editing-a-gated-document.md +++ b/wiki/qa/document-verification/editing-a-gated-document.md @@ -9,8 +9,8 @@ sources: - https://docs.vale.sh/topics/scopes.md - https://docs.vale.sh/checks/existence - https://github.com/DavidAnson/markdownlint/blob/main/doc/md013.md -last_verified: 2026-08-14 -related: [qa-process-acceptance-criteria, qa-process-regression-scope, testing-quality-tests-that-cannot-fail] +last_verified: 2026-09-08 +related: [qa-process-acceptance-criteria, qa-process-regression-scope, testing-quality-tests-that-cannot-fail, qa-deliverables-quantitative-claims-in-a-published-document, qa-document-verification-sweeping-pre-gate-citations-for-fabrication] --- # Editing a Document That Automated Text Gates Check @@ -75,6 +75,7 @@ now fails on wording whose meaning did not change. | The failure surfaces in another task's or another agent's check log | Attribute before repairing: identify which file the failing pattern targets, since a cross-file gate makes your edit look like their regression | | The gate is genuinely wrong (it forbids a correct sentence with no replacement available) | Change the gate, with a control proving it still catches the defect it owns — do not reword a correct document into a worse one | | The gate is green on your machine and red only in CI after a prose edit (macOS-authored bats suites) | Treat the anchor inventory of step 1 as the authoritative local signal, not the test run: under bash ≤4.0 (macOS system bash 3.2) a failing mid-test `[[ ]]` does not fail the test, so the broken anchor passes silently on your platform ([testing-quality-tests-that-cannot-fail]) | +| A content-only PR fails CI and the failing gate is a corpus-derived exact count pinned in a test (a lint's printed `directives: N` vs. a test file's pinned `N`), authored on a machine that never ran the local suite | Diff the gate's own printed count against the pinned count before reading the changed content — content-only PRs are commonly authored without running the local test suite, so the mismatch first surfaces in CI and looks like a content defect but is a one-line pin bump alongside the content change | | You are authoring the gate rather than the document | This page covers the author's side; gate construction and its controls are a separate concern → [testing-quality-tests-that-cannot-fail]. For a phrase anchor specifically, compare whitespace-normalized text (collapse runs of whitespace, then `grep -qF`) so legitimate reflow of the document cannot break the gate | ## Instead of @@ -95,6 +96,8 @@ now fails on wording whose meaning did not change. - https://man7.org/linux/man-pages/man1/pgrep.1.html — "The running pgrep, pkill, or pidwait process will never report itself as a match" — self-exclusion is designed in because self-matching is the expected failure - Measured 2026-08-14 (macOS bash 3.2.57): `[[ "$s" == *"return to step 1"* ]]` does not match when `$s` carries the phrase split across a newline — the wrap genuinely breaks the anchor on every platform; and `set -e; [[ 1 -eq 2 ]]; echo REACHED` prints, so the broken mid-test anchor is invisible under macOS bats while bash ≥4.2 CI fails it - Field reproduction ×2 (dev-loop): PR #94 §O3 and PR #102 — reflowing SKILL.md prose split `return to step 1 of the dispatch` across two lines; `tests/orchestrate-review-pass.bats` asserted it as a single-line substring, CI red on ubuntu only, fixed by reflowing the phrase onto one physical line (commit 9cbc065) +- Reproduction 2026-09-08 (dev-loop repo): `node scripts/wiki-lint-prohibitions.js wiki` prints `directives: 75`; `tests/wiki-lint-prohibitions.bats` pins the identical literal `directives: 75` as an exact-match assertion — the gate's printed count and the pinned test count are two independently maintained copies of one corpus-derived number +- Field evidence (dev-loop PR #151): `wiki-lint-prohibitions.js` printed `directives: 72` while `tests/wiki-lint-prohibitions.bats` pinned 71; bumping the pin turned both CI jobs green with zero content changes ## Field context diff --git a/wiki/qa/document-verification/spec-document-gates.md b/wiki/qa/document-verification/spec-document-gates.md index 0cf00fb..72c5e9b 100644 --- a/wiki/qa/document-verification/spec-document-gates.md +++ b/wiki/qa/document-verification/spec-document-gates.md @@ -15,18 +15,7 @@ sources: - https://github.com/DavidAnson/markdownlint/issues/1206 - https://google.github.io/styleguide/docguide/best_practices.html last_verified: 2026-08-09 -related: - [ - qa-process-acceptance-criteria, - testing-quality-tests-that-cannot-fail, - platforms-environment-unicode-text-matching, - qa-deliverables-generated-artifacts-as-deliverable-source, - testing-quality-spec-artifact-checks, - testing-quality-checks-that-cannot-pass, - qa-deliverables-command-transcripts-in-a-document, - qa-deliverables-exclusivity-and-absence-claims, - qa-document-verification-generated-reference-drift-gates, - ] +related: [qa-process-acceptance-criteria, testing-quality-tests-that-cannot-fail, platforms-environment-unicode-text-matching, qa-deliverables-generated-artifacts-as-deliverable-source, testing-quality-spec-artifact-checks, testing-quality-checks-that-cannot-pass, qa-deliverables-command-transcripts-in-a-document, qa-deliverables-exclusivity-and-absence-claims, qa-document-verification-generated-reference-drift-gates, qa-document-verification-sweeping-pre-gate-citations-for-fabrication] --- # Automated Gates on a Specification Document diff --git a/wiki/qa/document-verification/sweeping-pre-gate-citations-for-fabrication.md b/wiki/qa/document-verification/sweeping-pre-gate-citations-for-fabrication.md new file mode 100644 index 0000000..4708053 --- /dev/null +++ b/wiki/qa/document-verification/sweeping-pre-gate-citations-for-fabrication.md @@ -0,0 +1,77 @@ +--- +id: qa-document-verification-sweeping-pre-gate-citations-for-fabrication +domain: qa +category: document-verification +applies_to: [general] +confidence: field-tested +sources: + - https://github.com/choiyounggi/dev-loop/issues/156 +last_verified: 2026-09-08 +related: [qa-document-verification-spec-document-gates, qa-document-verification-editing-a-gated-document, qa-process-defect-class-resweep-after-review, qa-deliverables-quantitative-claims-in-a-published-document] +--- + +# Retroactively Sweeping a Pre-Gate Corpus When a Citation-Verification Gate Lands + +## When this applies + +A knowledge base or wiki marks entries `confidence: verified` based on +citations the author added at write time (docs, URLs, quoted text), and you +are introducing — or just introduced — an automated citation-verification +gate: one that fetches every cited source and checks the page's claims +against it. Deciding whether that gate applies only to future edits or also +sweeps the corpus written before it existed. + +## Do this + +1. **Run the new gate over the full pre-gate corpus once, in addition to + future diffs.** Enumerate every page carrying the confidence level the gate + guards (`verified`), fetch every cited URL for each, and compare. Pre-gate + entries were verified only by the author's own session — nothing else ever + checked them, so a defect there stays invisible under a `verified` label + until something re-reads the source. +2. **Treat quotation marks as a verbatim guarantee.** Any text inside quote + marks attributed to a source must match the fetched source's text exactly, + not "convey the same idea." A close paraphrase inside quote marks is a + defect: the marks are the reader's signal that no interpretation happened. +3. **Classify each finding by its specific breakage kind** — the fixes differ: + +| Case | Do | +|------|----| +| Quoted text is not present verbatim in the fetched source | Blocker: replace with the source's actual wording, or drop the quote marks and restate as paraphrase — then re-check the underlying claim still holds without the quote | +| Cited source fetches fine but discusses a different subject than the sentence claims (an arXiv ID names a different paper; an OWASP page never covers the cited topic) | Blocker: find the correct source or delete the claim; a "roughly related" citation is removed, not kept | +| Numeric or identifier citation (arXiv ID, issue number, version number) not traceable to any fetched source | Blocker: treat as suspected fabrication — remove or replace it | +| Link 404s, but the content exists at a different path, branch, or commit | Blocker: replace with a pinned working link (commit SHA or default-branch path) — a re-guessed URL is the next auditor's dead link | +| Source is unreachable from the auditing environment (403/429, JS-rendered, paywalled) after one retry | Advisory only: mark "unverifiable from CI"; unreachability itself is not a content defect | +| A claim's only real support is internal/session-only evidence, and the cited external doc does not establish it | Downgrade the page's `confidence` to `field-tested` and keep the claim — field evidence still supports it | + +4. **Report the breakdown by kind, not a single defect count.** A "33 + blocker-level citation defects" headline can quietly include categories + that are not citation defects at all (frontmatter formatting bugs, + internal count mismatches within a page) — state each category's count + separately so the citation-specific total is not inflated by unrelated + lint failures found in the same pass. +5. **Parallelize the sweep by domain or category chunk** when the corpus is + large enough that serial fetching is impractical, and record the actual + agent count and fetch count achieved — that is the evidence the next + auditor uses to estimate cost before starting their own sweep. + +## Edge cases + +| Case | Then | +|------|------| +| The page's confidence was already `field-tested`, not `verified` | Still fetch its cited URLs if any exist and flag fabricated/dead ones — a lower confidence tier does not exempt a page from having its citations exist | +| A citation was correct when written but the source has since changed (a doc page moved a claim to a different section) | Blocker under "stale claim," logged separately from fabrication — the citation was true once, so the fix is re-pointing it, not removing it | +| Two independent pages cite the same broken source | Fix both in the same pass; recording only the first-found instance leaves the second silently wrong after the sweep closes | +| The gate itself cannot tell fabrication from an unreachable-but-real source | Retry once, then record as advisory "unverifiable" rather than auto-classifying the uncertain fetch as a blocker — matching the fail-closed-on-missing-anchor principle in [qa-document-verification-spec-document-gates] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Apply a new citation-verification gate only to future PRs going forward | Also run it once over the full pre-gate corpus | Nothing else ever checked the old entries, so a pre-gate defect stays invisible under a `verified` label indefinitely | +| Accept a quote because the cited source is topically related | Fetch the exact source and compare the quoted phrase against it character-for-character | Topical relation is not verbatim truth — a citation can be a real, fetchable, wrong document, and the mismatch is invisible without a fetch | +| Publish one combined "N citation defects" count that mixes fabrication with formatting bugs found in the same pass | Report counts per defect kind (fabricated quote / wrong source / dead link / stale claim / non-citation lint) | Each kind has a different fix and urgency, and a combined count overstates how many are true citation-content defects | + +## Sources + +- https://github.com/choiyounggi/dev-loop/issues/156 — wiki-wide retroactive audit: 8 parallel agents, ~440 URL fetches, 270/270 pages covered, 33 blockers on 24 pages (about 25 of them citation/sourcing defects — fabricated quotes, wrong-paper citations, dead links, unsupportable `verified` labels — the remainder frontmatter and internal-consistency bugs found in the same pass); the issue's verdict rule: a failed fetch gets one retry then becomes an advisory "unverifiable", never a guessed verdict diff --git a/wiki/qa/index.md b/wiki/qa/index.md index 18a4e9a..0ab6e7a 100644 --- a/wiki/qa/index.md +++ b/wiki/qa/index.md @@ -24,6 +24,7 @@ Match your situation to a "load when" line; load only matching pages. | [evaluating-review-feedback](process/evaluating-review-feedback.md) | Review findings arrived (human, bot, or reviewer agent) and you are deciding what to implement; a finding is unclear; a reviewer proposes robustness or features nothing uses; you disagree with a finding and are deciding how to respond | | [adversarial-change-review](process/adversarial-change-review.md) | Reviewing a diff ≥50 changed lines or one touching auth/payments/migrations/external APIs; a checklist review found nothing on a high-blast-radius change; deciding review depth for a risky diff; constructing failure scenarios (assumption violation, composition, cascade, abuse) | | [llm-review-pipelines](process/llm-review-pipelines.md) | Building or configuring an automated LLM code-review pipeline (CI review bot, review skill, PR-reviewer agent); review token cost grows with changeset size; deciding which stages run as deterministic code vs model judgment; keeping review rules as matchable data not prompt prose; model-written comments land on wrong lines; measuring a precision-vs-recall threshold change | +| [session-identity-leak-in-plugin-prose](process/session-identity-leak-in-plugin-prose.md) | Reviewing prose added or edited in a marketplace-distributed plugin/skill repo (SKILL.md, README, template, example, doc comment) authored inside a session whose identity-injection layer set the user's personal name into context; a plugin's distributed prose reads as personalized to one specific user; a code comment or string literal carries a session-local identifier in redistributed source; deciding what an identity-specific review check must catch beyond a general structural/adversarial pass | ## deliverables @@ -41,7 +42,8 @@ Match your situation to a "load when" line; load only matching pages. | [spec-document-gates](document-verification/spec-document-gates.md) | Deciding whether passing a doc gate is enough to accept the deliverable: a document passed its checklist but the requirement is still unmet; choosing what the gate must assert beyond keyword presence (table structure, MUST-vs-SHOULD demotion, closed-set completeness, cross-section consistency, agreement with a code constant the document copies); a gate over a table asserts the property the table itself claims; setting the release policy a gate verdict feeds (authoring or validating the check code itself → wiki/testing/quality/spec-artifact-checks.md, wiki/testing/quality/checks-that-cannot-pass.md) | | [generated-reference-drift-gates](document-verification/generated-reference-drift-gates.md) | Writing or reviewing the reference material that enumerates a closed vocabulary an agent will emit tokens from (DSL verbs, config keys, diagnostic codes, enum members) in a plugin/skill/SDK; deciding whether to hand-write that list or generate it from the owning constant; choosing what gates a generated document beyond a `--check` diff; a documented token compiles to a silent no-op instead of an error | | [retiring-a-provisional-marker](document-verification/retiring-a-provisional-marker.md) | Turning provisional markers (`[추정]`, TBD, "assumed", DRAFT) into settled statements in an ADR/RFC/spec that has been through review rounds, so the document also carries a review checklist and a round history referring to those markers; a checklist row stayed `[x]` on evidence you just deleted; a coordinator's marker count and yours disagree while you read the same file | -| [editing-a-gated-document](document-verification/editing-a-gated-document.md) | Editing or rewording a document that grep/regex gates or a lint config check; a gate fails on wording whose meaning did not change; reflowing prose a test asserts as a verbatim phrase (CI red on one platform only); describing what an upstream spec says without tripping a "do not redefine it" gate; a check matches the pattern your own document quotes; recording an audit verdict inside the document that was audited; deciding which checks to re-run after editing a gated document | +| [editing-a-gated-document](document-verification/editing-a-gated-document.md) | Editing or rewording a document that grep/regex gates or a lint config check; a gate fails on wording whose meaning did not change; reflowing prose a test asserts as a verbatim phrase (CI red on one platform only); describing what an upstream spec says without tripping a "do not redefine it" gate; a check matches the pattern your own document quotes; recording an audit verdict inside the document that was audited; deciding which checks to re-run after editing a gated document; reviewing a content-only PR whose CI fails on a corpus-derived count pinned in a test | +| [sweeping-pre-gate-citations-for-fabrication](document-verification/sweeping-pre-gate-citations-for-fabrication.md) | Introducing a citation-verification gate (fetches cited sources, compares quoted text) into a knowledge base whose existing entries were marked `verified` by author self-report only; deciding whether the new gate applies to future edits only or also sweeps the pre-existing corpus; a page's quote marks need checking against the fetched source rather than trusted at face value; classifying and reporting citation defects by kind | ## environments diff --git a/wiki/qa/process/adversarial-change-review.md b/wiki/qa/process/adversarial-change-review.md index b4c5a59..1f2f383 100644 --- a/wiki/qa/process/adversarial-change-review.md +++ b/wiki/qa/process/adversarial-change-review.md @@ -7,7 +7,7 @@ confidence: field-tested sources: - https://github.com/EveryInc/compound-engineering-plugin last_verified: 2026-08-22 -related: [qa-process-evaluating-review-feedback, qa-process-regression-scope, qa-process-post-release-verification] +related: [qa-process-evaluating-review-feedback, qa-process-regression-scope, qa-process-post-release-verification, qa-process-session-identity-leak-in-plugin-prose] --- # Constructing Failure Scenarios for a High-Risk Diff diff --git a/wiki/qa/process/completion-claims.md b/wiki/qa/process/completion-claims.md index 6ddcfc6..dd11635 100644 --- a/wiki/qa/process/completion-claims.md +++ b/wiki/qa/process/completion-claims.md @@ -7,7 +7,7 @@ confidence: field-tested sources: - https://github.com/obra/superpowers last_verified: 2026-08-22 -related: [debugging-methodology-verify-the-fix, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, testing-quality-tests-that-cannot-fail, qa-process-release-gates] +related: [debugging-methodology-verify-the-fix, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, testing-quality-tests-that-cannot-fail, qa-process-release-gates, testing-strategy-real-cli-spot-check-for-new-execution-paths] --- # Claiming Work Is Done, Fixed, or Passing diff --git a/wiki/qa/process/defect-class-resweep-after-review.md b/wiki/qa/process/defect-class-resweep-after-review.md index 892dc30..4026173 100644 --- a/wiki/qa/process/defect-class-resweep-after-review.md +++ b/wiki/qa/process/defect-class-resweep-after-review.md @@ -8,7 +8,7 @@ sources: - https://www.eecg.utoronto.ca/~yuan/papers/incorrect_fix_abstract.html - https://dl.acm.org/doi/10.1145/2025113.2025121 last_verified: 2026-08-11 -related: [qa-process-regression-scope, backend-common-change-impact-call-site-enumeration, testing-quality-tests-that-cannot-fail, qa-process-evaluating-review-feedback] +related: [qa-process-regression-scope, backend-common-change-impact-call-site-enumeration, testing-quality-tests-that-cannot-fail, qa-process-evaluating-review-feedback, qa-document-verification-sweeping-pre-gate-citations-for-fabrication, qa-process-session-identity-leak-in-plugin-prose] --- # Re-Sweeping the Reviewed Defect Class Across Your Own Remediation Diff diff --git a/wiki/qa/process/llm-review-pipelines.md b/wiki/qa/process/llm-review-pipelines.md index 82bb5ce..0351bdd 100644 --- a/wiki/qa/process/llm-review-pipelines.md +++ b/wiki/qa/process/llm-review-pipelines.md @@ -8,7 +8,7 @@ sources: - https://github.com/alibaba/open-code-review - https://github.com/alibaba/open-code-review/blob/main/skills/open-code-review/SKILL.md last_verified: 2026-08-24 -related: [qa-process-evaluating-review-feedback] +related: [qa-process-evaluating-review-feedback, infrastructure-ci-cd-review-bot-adopted-as-a-blocking-gate] --- # Structuring an Automated LLM Code-Review Pipeline diff --git a/wiki/qa/process/scope-purity-checks.md b/wiki/qa/process/scope-purity-checks.md index 04ac4f3..e62c94b 100644 --- a/wiki/qa/process/scope-purity-checks.md +++ b/wiki/qa/process/scope-purity-checks.md @@ -10,7 +10,7 @@ sources: - https://bazel.build/reference/test-encyclopedia - "Field reproduction 2026-08-14 (dev-loop, reviews/i83-insight-emission-r1.md): a permanent bats test asserting `git status` scope failed on an unrelated uncommitted sibling file; rewritten to commit-diff evidence → 521/521" last_verified: 2026-08-14 -related: [testing-quality-checks-that-cannot-pass, testing-quality-harness-reverse-controls, testing-quality-history-dependent-checks-on-shallow-clones] +related: [testing-quality-checks-that-cannot-pass, testing-quality-harness-reverse-controls, testing-quality-history-dependent-checks-on-shallow-clones, infrastructure-agent-orchestration-worktree-isolated-workers] --- # Proving Scope Purity from `git status` Output diff --git a/wiki/qa/process/session-identity-leak-in-plugin-prose.md b/wiki/qa/process/session-identity-leak-in-plugin-prose.md new file mode 100644 index 0000000..ae9b0a9 --- /dev/null +++ b/wiki/qa/process/session-identity-leak-in-plugin-prose.md @@ -0,0 +1,66 @@ +--- +id: qa-process-session-identity-leak-in-plugin-prose +domain: qa +category: process +applies_to: [general] +confidence: field-tested +sources: + - "Field incident (a marketplace-distributed plugin repo, task t3-skill review round 1): a skill's SKILL.md line 9 hard-coded the session's injected user name; caught in a four-lens review, fixed in rework, confirmed absent by a recursive grep returning 0 matches; re-checked 2026-09-08 against the installed marketplace copy and two cached versions — none contain the name" + - https://12factor.net/config +last_verified: 2026-09-08 +related: [security-data-commit-identity-in-public-repos, qa-process-adversarial-change-review, qa-process-defect-class-resweep-after-review, security-dependencies-agent-skill-supply-chain] +--- + +# Session-Injected Personal Identity Leaking Into Distributed Plugin Prose + +## When this applies + +Reviewing a diff that adds or edits prose meant for redistribution — `SKILL.md`, +`README`, a template, an example, or a doc comment — inside a marketplace-distributed +plugin or skill repo, authored in a session whose identity-injection layer set the +user's personal name (or another session-local identifier) into context. + +## Do this + +1. **Run this as the first check on a plugin-authoring diff**, before structural or + correctness review: grep the diff for the session's injected personal name and + any other session-local identifiers. The code works locally with the name + hardcoded, so nothing else in a normal review catches it. +2. **Route by what the diff touches:** + +| Case | Do | +|------|----| +| Diff touches prose meant for redistribution (SKILL.md, README, template, example, doc comment) | Grep for the injected name/identifiers before approving; generalize any hit | +| The name also appears in a code comment or a hardcoded string in the plugin's source (not only docs) | Same check, same fix — the file ships as source; a comment distributes identically to prose | +| Diff touches internal-only files never distributed (a private CLAUDE.md, a local scratch note, a personal habit log) | Leave personal names in place — they are expected content here | +| The injected name is also a common word/token with a legitimate generic meaning | grep still finds every occurrence — read each hit's context before flagging; a bare hit count without inspection over- or under-reports | + +3. When a hit lands in redistributed content, replace it with a generic + role/placeholder ("the user", `{{name}}`, a config key) and confirm the plugin's + identity-aware behavior comes from the runtime identity-injection layer, not from + the authored file. +4. **Re-grep the whole plugin tree for the same token after the fix**, not just the + flagged line — the review sampled one file; a sibling file (another skill's prose, + a shared template) can carry the same session-local value untouched. + +## Edge cases + +| Case | Then | +|------|------| +| The leaked value is a different session-local identifier (work email, absolute home directory path, a machine hostname) rather than a name | Same grep-before-approve check extends to it — the mechanism (session context bleeding into generic output) is identical, only the token type differs | +| grep returns zero hits but the diff includes a screenshot or a paraphrase of session output | A screenshot or reworded sentence can still carry the name where a text grep misses it — read the rendered diff, not only the grep result | +| The plugin is private and never published to a marketplace | The check does not apply — personalization is fine when the only consumer is the session that authored it | +| A structural/adversarial review already passed on this diff | Run the grep anyway — a general review is not looking for this token class and routinely misses it ([qa-process-adversarial-change-review]) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Approve a plugin-authoring diff after a general code-quality checklist with no identity-specific check | Grep the diff for the session's injected personal name/identifiers before approving | The code works locally with the name hardcoded — a general checklist review has nothing to trigger on this class | +| Hardcode a friendly per-user greeting or example directly into a skill's distributed prose | Keep the prose generic and let the runtime identity-injection layer supply the name at session start | A worker session inherits the coordinator's identity context and writes it into deliverables by default; every other install then misnames its user | +| Treat fixing the one flagged line as closing the defect | Re-grep the whole plugin tree for the same token after the fix | The reviewed line is a sample; a sibling file can carry the same session-local value untouched | + +## Sources + +- Field incident (a marketplace-distributed plugin repo, task t3-skill review round 1): a skill's SKILL.md line 9 hard-coded the session's injected user name; caught in a four-lens review, fixed in rework, confirmed absent by a recursive grep returning 0 matches. Re-checked 2026-09-08 against the installed marketplace copy and both cached versions — none contain the name. +- https://12factor.net/config — supports the general principle by analogy only: "Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code"; "A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials." Scoped to runtime config and credentials, not authored prose — which is why confidence stays field-tested. diff --git a/wiki/security/data/commit-identity-in-public-repos.md b/wiki/security/data/commit-identity-in-public-repos.md index 1c4f5f9..0f09d1d 100644 --- a/wiki/security/data/commit-identity-in-public-repos.md +++ b/wiki/security/data/commit-identity-in-public-repos.md @@ -7,8 +7,11 @@ confidence: verified sources: - https://git-scm.com/docs/git-commit - https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/setting-your-commit-email-address -last_verified: 2026-08-06 -related: [security-secrets-secrets-in-code, security-data-pii-handling] + - https://git-scm.com/docs/git-log + - https://github.com/newren/git-filter-repo/blob/main/Documentation/git-filter-repo.txt + - https://github.blog/changelog/2019-12-19-improved-attribution-when-squashing-commits/ +last_verified: 2026-09-08 +related: [security-secrets-secrets-in-code, security-data-pii-handling, qa-process-session-identity-leak-in-plugin-prose] --- # The Author Identity a Commit Carries into a Public Repository @@ -56,6 +59,36 @@ than the one your git config now holds. 5. **Verify after the first commit, before pushing** — `git log -1 --format=%ae` — because that is the last point at which the fix is a local amend. +6. **Audit every public repo under the account before remediating a single + commit** — identity leaks are config-level: the same ambient `user.email` + that leaked in the commit you noticed was in force for every commit you made + from that machine, in every repo, so the leak is wider than the one commit + you saw. + + ```sh + git -C log --format='%ae%n%ce' | sort -u + git -C log --format='%B' | grep -iE '|co-authored-by' + ``` + + Check `%ce` (committer email) as well as `%ae` (author email) — a rebase, + cherry-pick, or a forge-performed merge sets the committer independently of + the author, so an author-only check misses it. Grep the message body too: + GitHub credits "every commit author in the pull request as a co-author on + the squash commit", so a leaked address can appear only in a + `Co-authored-by:` trailer, on a commit whose `%ae`/`%ce` never shows it. + +7. **Rewrite with `git filter-repo --mailmap --replace-message ` + when the leak spans more than the one commit you already amended**, rather + than an amend/`rebase -i` loop over each affected commit — `--mailmap` + rewrites author, committer, and tagger names/emails from one mapping file + across full history in a single pass; `--replace-message` replaces matched + text in commit/tag message bodies, which is what reaches `Co-authored-by:` + trailers, since `--mailmap` only touches the structured identity fields. + Before force-pushing the rewritten branch, confirm both: a repeat of the + audit in item 6 returns zero matches, and `git diff ` is + empty (the rewrite changed identity metadata and message text, not file + contents). + ## Edge cases | Case | Then | @@ -63,6 +96,8 @@ than the one your git config now holds. | The wrong address is already pushed to a public repository | Treat it as disclosed: the address is served by the forge API, present in every clone, and in downloadable archives. Fix the identity going forward and, when the address must not be associated with the project, rewrite history and force-push before it is mirrored — the window is short and closes on the first fork or archive | | The repository is public and the commit is old | Rewriting shared history breaks every existing clone; correct forward and accept the historical entry unless the exposure is material | | Only the committer differs from the author (rebase, cherry-pick, a merge you performed) | Both identities are published; set both with `GIT_COMMITTER_EMAIL`/`-c user.email`, which feeds both unless the `GIT_*` variables are set | +| GitHub squash-merged a pull request whose commits carried the leaked address | The squash commit's message carries a `Co-authored-by:` trailer per original commit author even when the squash commit's own `%ae`/`%ce` is clean — grep message bodies (`git log --format=%B`), not only `%ae`/`%ce`, before treating a repo as clear | +| The leak spans many commits, multiple repos, or committer/trailer fields rather than one author field | Use `git filter-repo --mailmap --replace-message` (Do this, item 7) instead of amending commits one at a time — it rewrites author/committer/tagger fields and message text across full history in one pass, and its output is verifiable against the pre-rewrite tree | | The commit is produced by CI or a bot | Give it its own dedicated identity in the workflow environment, not a person's — a human address on machine commits misattributes authorship | | Your git config has no identity at all | Git falls back to the system user name plus a hostname-derived domain, which publishes the machine's hostname; set the identity explicitly rather than relying on the fallback | | The project requires a real address (DCO sign-off, CLA) | The no-reply form is not acceptable for sign-off; use a personal address you are willing to publish | @@ -80,4 +115,8 @@ than the one your git config now holds. - https://git-scm.com/docs/git-commit — "the information is taken from the configuration items `user.name` and `user.email`, or, if not present, the environment variable `EMAIL`, or, if that is not set, system user name and the hostname used for outgoing mail"; `GIT_AUTHOR_*`/`GIT_COMMITTER_*` take precedence over config - https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/setting-your-commit-email-address — "GitHub uses the email address set in your local Git configuration to associate commits pushed from the command line with your account"; a per-repository address "will override your global Git configuration settings in this one repository, but will not affect any other repositories" +- https://git-scm.com/docs/git-log — pretty formats: `%ae` is the author email, `%ce` the committer email — two distinct fields, both published +- https://github.com/newren/git-filter-repo/blob/main/Documentation/git-filter-repo.txt — `--mailmap`: "Use specified mailmap file ... when rewriting author, committer, and tagger names and emails"; `--replace-message`: "A file with expressions that, if found in commit or tag messages, will be replaced" +- https://github.blog/changelog/2019-12-19-improved-attribution-when-squashing-commits/ — "we will automatically credit every commit author in the pull request as a co-author on the squash commit" +- Field incident 2026-09 (two public repos under one account): one noticed author-field leak turned out to be 12 committer-field + 4 trailer leaks in one repo and 16 + 3 + trailers in the other (plus a second employer's address); all purged via `git filter-repo --mailmap` + `--replace-message` with byte-identical trees, verified by zero-match greps on the rewritten remotes - Field incident 2026-08-06 (`groundwork`, public repository, macOS): `git config user.email` resolved to an employer address while `git log -1 --format=%ae` showed the repository's history authored under a GitHub no-reply address — the mismatch was silent and would have been published by the next commit diff --git a/wiki/security/data/masking-verification.md b/wiki/security/data/masking-verification.md index 9142850..3f90e6a 100644 --- a/wiki/security/data/masking-verification.md +++ b/wiki/security/data/masking-verification.md @@ -7,8 +7,8 @@ confidence: field-tested sources: - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html - "Local reproduction (lnpl 0.2.0, 2026-08-05): one `--json` run held the raw planted card number at result.bindings while the trace/log channel showed `***`; an unmasked control field appeared in both" -last_verified: 2026-08-29 -related: [security-data-pii-handling, testing-quality-harness-reverse-controls, qa-exploratory-override-control-pairs] +last_verified: 2026-09-08 +related: [security-data-pii-handling, testing-quality-harness-reverse-controls, qa-exploratory-override-control-pairs, backend-common-change-impact-call-site-enumeration] --- # Verifying Masking Across Every Output Channel @@ -45,6 +45,19 @@ PASS" verdict from the platform itself. compares.** Ask which channels the check reads before accepting its PASS: a differential check that only compares masked-clean channels reports PASS while the raw value sits in the result payload. +5. **Before trusting a masking/projection function to hide a new key, read its + implementation and classify it.** A whitelist-shaped mask emits only + declared fields — an unknown key cannot pass through it by construction. A + blacklist-shaped mask (e.g. `{k: (MASK if k in masked_names else v) for k, v + in payload.items()}`) passes every key through unchanged and only + substitutes the *value* of fields it recognizes as sensitive; it has no + field-presence filtering at all. For a blacklist-shaped mask, grep every + call site that reads the underlying store directly — not just the call + sites inside your declared scope ([backend-common-change-impact-call-site-enumeration]) + — because a read path that bypasses the normal serialization/interpreter + layer (an HTTP handler calling the repository directly instead of routing + through `execute()`/`query()`) pipes the new key straight to the mask, and + the mask lets it through unmasked. ## Edge cases @@ -54,6 +67,7 @@ PASS" verdict from the platform itself. | Structured and human-readable output serialize separately | Run the channel sweep once per serialization mode (`--json` and default) | | Masking is type-driven (a `Password`-typed field) | Verify per channel anyway — the type triggers masking only in the layers wired to honor it | | A new output channel is added later (export, webhook, metrics label) | Re-run the sweep; channel enumeration is release-scoped, not one-time | +| The mask is blacklist-shaped (passes all keys through, substitutes values only for named-sensitive fields) and a reserved/internal key is being added to stored rows | Strip the new key explicitly at every call site that reads the store directly, in addition to relying on the mask — the mask itself provides no field-presence filtering | ## Instead of @@ -62,8 +76,10 @@ PASS" verdict from the platform itself. | Claim masking works after seeing `***` in the log | Grep the raw value in every enumerated channel of the same run | The log is one channel; the result payload in the same output file can carry the raw value | | Accept the tool's "masking PASS" verdict | Determine which channels the verdict compares, then sweep the rest yourself | The verdict is a claim about the compared channels only | | Check only that the raw value is absent | Also require the negative-control field present in each channel | An empty or uncaptured channel makes absence meaningless | +| Add a reserved/internal key to stored rows and assume the existing `mask_payload`-style function hides it from output | Read the mask's implementation first; for a blacklist-shaped mask, grep every direct-store-read call site and strip the key there explicitly | A blacklist mask only substitutes values for fields it already knows are sensitive — it has no concept of "unknown key", so a new key leaks verbatim through any channel that reaches it | ## Sources - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html — scoped to the logging channel only: lists "Authentication passwords" and "Bank account or payment card holder data" as data to be "removed, masked, sanitized, hashed, or encrypted" rather than recorded in logs. It does not address the result payload, pretty output, generated artifacts, or error-report channels, and does not itself prescribe the cross-channel sweep methodology — that comes from the local reproduction below, where the same masking check passed on the log channel while the raw value sat unmasked in the result payload - Local reproduction (2026-08-05, lnpl 0.2.0 runner): a `Password`-typed field fed the planted value `4111111111111111`; one `--json` output held the raw value at `.result.bindings.account.cardSecret` and `***` at `.trace.logs[0].payload.cardSecret`; the unmasked control field `label` appeared in both channels. Matches the originating QA case, where the platform's differential check reported "PASS 4/4 masking" while the raw card number sat in `result.bindings` +- Field incident 2026-09 (a Python interpreter + WSGI service): `mask_payload(payload, entity_node)` was `{k: (MASK if k in masked_names else v) for k, v in payload.items()}` — value substitution only, no field-presence filtering; the WSGI module's `_get_single`/`_get_list` read endpoints called the repository directly (bypassing the interpreter's `execute()`/`query()` path) and piped the raw row into `mask_payload`, so a newly added reserved key (`_schema_gen`) leaked verbatim into the JSON API response until stripped at those two call sites diff --git a/wiki/security/dependencies/agent-skill-supply-chain.md b/wiki/security/dependencies/agent-skill-supply-chain.md index a0a6eb5..739a50d 100644 --- a/wiki/security/dependencies/agent-skill-supply-chain.md +++ b/wiki/security/dependencies/agent-skill-supply-chain.md @@ -9,7 +9,7 @@ sources: - https://github.com/virgiliojr94/book-to-skill/blob/9c207f870adebe20ade4f7d2f11bc3d759c2fd88/SECURITY-NOTICE.md - https://github.com/different-ai/openwork/blob/fda0babb6c7600ed044757515beb374a3f4dc827/skills-lock.json last_verified: 2026-08-29 -related: [security-dependencies-supply-chain, security-agent-exposure-in-session-tool-exposure] +related: [security-dependencies-supply-chain, security-agent-exposure-in-session-tool-exposure, qa-process-session-identity-leak-in-plugin-prose] --- # Trusting Third-Party Agent Skills, Plugins, and MCP Servers diff --git a/wiki/security/index.md b/wiki/security/index.md index 3066214..5b9ae8a 100644 --- a/wiki/security/index.md +++ b/wiki/security/index.md @@ -67,5 +67,5 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [pii-handling](data/pii-handling.md) | A feature stores/processes personal data (emails, names, phones, addresses, government ids); reviewing a log/analytics/export/URL path that can carry PII; designing retention/erasure or handling an erasure request; choosing staging/test data for tables holding PII | -| [commit-identity-in-public-repos](data/commit-identity-in-public-repos.md) | About to commit to a repository whose history is public (or will be published) from a machine whose git identity was configured for something else — a work laptop, a shared build box, a container image; a repository's history uses a different author address than your current git config; deciding between a global, per-repository, and per-commit identity override; responding to an employer or personal address already pushed to a public repository; giving CI/bot commits their own identity | -| [masking-verification](data/masking-verification.md) | About to claim masking/redaction of a sensitive field works (logger filter, type-driven masking, serializer); a masking check passed on one output channel; reviewing an automated "masking PASS" verdict | +| [commit-identity-in-public-repos](data/commit-identity-in-public-repos.md) | About to commit to a repository whose history is public (or will be published) from a machine whose git identity was configured for something else — a work laptop, a shared build box, a container image; a repository's history uses a different author address than your current git config; a public repo's committer field or squash-merge Co-authored-by trailer carries a work/corporate address even though the author field looks clean; deciding between a global, per-repository, and per-commit identity override; auditing every public repo under an account for the same leak before remediating one noticed commit; responding to an employer or personal address already pushed to a public repository; giving CI/bot commits their own identity | +| [masking-verification](data/masking-verification.md) | About to claim masking/redaction of a sensitive field works (logger filter, type-driven masking, serializer); a masking check passed on one output channel; reviewing an automated "masking PASS" verdict; adding a reserved/internal key to a payload that already flows through an existing masking function and needing to know whether that function is whitelist- or blacklist-shaped before trusting it to hide the new key | diff --git a/wiki/testing/async/async-testing.md b/wiki/testing/async/async-testing.md index 49b91e1..a4730c1 100644 --- a/wiki/testing/async/async-testing.md +++ b/wiki/testing/async/async-testing.md @@ -11,7 +11,7 @@ sources: - https://testing-library.com/docs/dom-testing-library/api-async/ - https://martinfowler.com/articles/nonDeterminism.html last_verified: 2026-08-29 -related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-injected-clock-duration-assertions] +related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-injected-clock-duration-assertions, testing-quality-narration-based-ordering-assertions] --- # Testing Asynchronous Code Deterministically diff --git a/wiki/testing/data/test-data-and-isolation.md b/wiki/testing/data/test-data-and-isolation.md index 4df1135..7165bcd 100644 --- a/wiki/testing/data/test-data-and-isolation.md +++ b/wiki/testing/data/test-data-and-isolation.md @@ -11,7 +11,7 @@ sources: - https://nodejs.org/api/fs.html - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/env.html last_verified: 2026-08-29 -related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice, testing-strategy-import-time-side-effects, testing-data-artifact-leakage-from-a-suite, testing-quality-behavior-not-implementation, platforms-filesystems-permissions-and-exec-bits, backend-common-change-impact-call-site-enumeration, testing-data-harness-vs-run-path-fixtures, infrastructure-agent-orchestration-shared-run-state] +related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice, testing-strategy-import-time-side-effects, testing-data-artifact-leakage-from-a-suite, testing-quality-behavior-not-implementation, platforms-filesystems-permissions-and-exec-bits, backend-common-change-impact-call-site-enumeration, testing-data-harness-vs-run-path-fixtures, infrastructure-agent-orchestration-shared-run-state, testing-mocking-autouse-fixture-shadows-function-under-test] --- # Owning Test Data and Isolating Test State diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 4e65ac9..8a7e336 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -22,6 +22,7 @@ Match your situation to a "load when" line; load only matching pages. | [orm-generated-test-schema](strategy/orm-generated-test-schema.md) | Planning a regression test for a defect caused by the ORM model and the real database disagreeing (nullability, type, length, index); checking what a test profile's `ddl-auto`/schema source makes reproducible; choosing between a migration-built test database with `validate` and an `information_schema` comparison gate; reviewing a plan that promises to "add a test that reproduces" a drift-shaped defect | | [failing-test-first](strategy/failing-test-first.md) | Implementing new behavior or a bug fix and deciding the order of test and production code; a test passed on its first-ever run; production code exists that no test required; expected values are about to be copied from the code's own output; encoding a bug repro as a regression test | | [cross-layer-effect-tests](strategy/cross-layer-effect-tests.md) | About to mark a change tested when its tests exercise only the changed unit; the changed path fires callbacks/middleware/hooks/jobs/DB triggers; the change persists state in more than one step; several surfaces (API, CLI, job) expose the same changed logic; deciding whether unit-level coverage suffices for a change | +| [real-cli-spot-check-for-new-execution-paths](strategy/real-cli-spot-check-for-new-execution-paths.md) | A new execution path drives a real CLI process or real external environment alongside an existing deterministic/scripted/fake path, and that path's deterministic test suite is fully green but has never been exercised for real; deciding whether a green scripted suite is sufficient evidence for a new real-CLI/real-environment path | ## quality @@ -53,6 +54,8 @@ Match your situation to a "load when" line; load only matching pages. | [source-text-wiring-assertions](quality/source-text-wiring-assertions.md) | A test reads a source file as a string and asserts by regex that a call is present (cleanup in every handler, logging after each branch, teardown in each exit path) because the behavior has no reachable seam; such a guard is green while one of the call sites is gone; choosing between a count assertion, an order anchor, and a function-body slice; such a guard (a negative `not.toMatch`, or a token-count assertion) reddens on correct code and a comment is the only thing that changed | | [history-dependent-checks-on-shallow-clones](quality/history-dependent-checks-on-shallow-clones.md) | A test or gate resolves git history (`log --diff-filter`, `merge-base`, `rev-list`) and may run under a shallow CI checkout (actions/checkout default `fetch-depth: 1`); a history query reported every tracked file as added by one commit; deciding between a `--is-shallow-repository` skip-guard and deepening the fetch | | [expectation-sets-with-one-distinct-value](quality/expectation-sets-with-one-distinct-value.md) | Locking that a computed value is really carried into a response/DTO/event and every test case expects the same literal for that field; a constant-substitution mutant survives a green suite and you are about to add more assertions; choosing between a delete probe and a value-substitution probe for wiring; auditing fixtures for one-distinct-value expectation sets | +| [narration-based-ordering-assertions](quality/narration-based-ordering-assertions.md) | Writing a regression test for an ordering/timing invariant in concurrent or async code (e.g. "handler must be installed before launch") by recording self-reported narration strings at the code's existing hook/instrumentation call sites and asserting their relative order; proving such a test's power by mutation before trusting it | +| [cross-task-stub-assertions](quality/cross-task-stub-assertions.md) | A test asserts the wording/content of a cross-task placeholder or stub (a shell task's UI, a scaffolded module) that a later, different task is planned to replace; deciding what a test may assert about a stub without breaking when the stub's content is swapped for the real implementation | ## data @@ -70,6 +73,7 @@ Match your situation to a "load when" line; load only matching pages. | [destructive-operations-on-shared-daemons](mocking/destructive-operations-on-shared-daemons.md) | The code under test enumerates and deletes a machine-wide daemon's resources by name/pattern (tmux sessions, docker containers, systemd units, namespaces) and that daemon runs on the test machine; proving a sweep deletes the targets and spares bystanders; keeping a scope bug from destroying the dev environment instead of failing the test; giving a shell script a substitution seam for the tool it shells out to | | [what-to-mock](mocking/what-to-mock.md) | Deciding whether to mock/stub/fake a dependency or use the real one; mocks breaking on refactors; testing handling of a third-party's failure modes; the same mock setup is copy-pasted across tests | | [captured-call-arguments](mocking/captured-call-arguments.md) | Writing the spy/stub test that holds a fix to one argument of one wiring call (constructor, factory, server startup); such a test is green while a mutation of a *different* argument of the same call survives; the fix extracted the value into a resolver and you are choosing what to assert; deciding between asserting a constant's value and asserting that the call site passes it on; choosing how to record an argument you deliberately leave unpinned | +| [autouse-fixture-shadows-function-under-test](mocking/autouse-fixture-shadows-function-under-test.md) | A test calls `module.func(...)` directly to assert that function's own properties, while a conftest `autouse=True` fixture `monkeypatch.setattr`-replaces the same name; deciding whether an assertion target is the real function or the fixture's stub; capturing the original before the fixture applies | ## flaky diff --git a/wiki/testing/mocking/autouse-fixture-shadows-function-under-test.md b/wiki/testing/mocking/autouse-fixture-shadows-function-under-test.md new file mode 100644 index 0000000..bd9c9bb --- /dev/null +++ b/wiki/testing/mocking/autouse-fixture-shadows-function-under-test.md @@ -0,0 +1,66 @@ +--- +id: testing-mocking-autouse-fixture-shadows-function-under-test +domain: testing +category: mocking +applies_to: [python] +confidence: verified +sources: + - https://docs.pytest.org/en/stable/how-to/monkeypatch.html + - https://docs.pytest.org/en/stable/how-to/fixtures.html#autouse-fixtures-fixtures-you-don-t-have-to-request +last_verified: 2026-09-08 +related: [testing-mocking-what-to-mock, testing-data-test-data-and-isolation, testing-quality-tests-that-cannot-fail] +--- + +# A Test Asserting a Function's Own Properties While an Autouse Fixture Replaces It + +## When this applies + +A test calls `module.func(...)` directly to assert that function's own +behavior or properties, and a pytest `autouse=True` fixture in the same +conftest replaces `module.func` with `monkeypatch.setattr` for the whole +suite or file. + +## Do this + +1. **Determine which implementation the test's call site actually reaches + before trusting the assertion.** `monkeypatch.setattr` runs as fixture + setup, which completes before the test body executes, and it replaces the + name in the target's own namespace — so `module.func` inside the test body + resolves to whatever was set last: the fixture's stub, not the source + definition. +2. **Confirm by breaking the real function and requiring the test to go + red** ([testing-quality-tests-that-cannot-fail]). If the test stays green + while the real implementation is provably broken, the assertion is + exercising the stub, not the function. +3. **When the assertion must run against the real function, capture the + original reference before the fixture applies** — at module import time, in + a name the fixture does not target (`_REAL_FUNC = module.func` at the top + of the test file, executed during collection, before any fixture's setup + phase runs) — and assert against that captured reference instead of + `module.func`. + +| Case | Do | +|------|----| +| The test needs the real function's behavior and a co-located autouse fixture patches the same name | Capture the original at import time in a separate name; assert against the capture | +| The test's own purpose is to verify the fixture's isolation (that other code calls through the patched seam) | Assert via `module.func` as usual — this is the fixture's intended effect, not a defect | +| Unsure whether an existing test asserts the real function or the stub | Break the real function and rerun; green means the assertion never reached it | + +## Edge cases + +| Case | Then | +|------|------| +| The fixture patches at a different name than the test imports (`import module` vs `from module import func`) | `monkeypatch.setattr` must target the same binding the test reads; check both import forms — patching `module.func` does not affect a name already bound via `from module import func` in another file | +| The capture (`_REAL_FUNC`) is itself later monkeypatched by a different fixture | Capture at collection time is only safe from fixtures that run at test setup; a fixture that patches at collection/import time would still shadow it — verify by breaking the real function once more after adding the capture | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Assert on `module.func(...)` in a file where a co-located autouse fixture already `monkeypatch.setattr`s that name | Capture the original at import time and assert against the capture | `monkeypatch.setattr` finishes before the test body runs and replaces the name in its own namespace, so the two tests silently disagree about which implementation "the function" means | +| Trust that a green suite proves the real function is covered | Break the real function and confirm the specific test goes red | An isolation fixture and a test that verifies the function itself can coexist in one green suite while the second one asserts nothing | + +## Sources + +- https://docs.pytest.org/en/stable/how-to/monkeypatch.html — "monkeypatch.setattr must be called before the function which will use the patched function is called"; "All modifications will be undone after the requesting test function or fixture has finished" +- https://docs.pytest.org/en/stable/how-to/fixtures.html#autouse-fixtures-fixtures-you-don-t-have-to-request — autouse fixtures are requested automatically for every applicable test and run as setup "even though neither test requested it" +- Field evidence (a Python trading-bot repo, review finding F1 on a shared-throttle task, fix introducing `_REAL_THROTTLE_PATH`): a test asserting the throttle function's own property called it through the module attribute a co-located autouse fixture monkeypatched; capturing the reference before the fixture applied and asserting against that reference was confirmed by breaking the real function and observing the test go red diff --git a/wiki/testing/mocking/what-to-mock.md b/wiki/testing/mocking/what-to-mock.md index 17f7fa2..e5efc50 100644 --- a/wiki/testing/mocking/what-to-mock.md +++ b/wiki/testing/mocking/what-to-mock.md @@ -11,7 +11,7 @@ sources: - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import - https://nodejs.org/api/test.html last_verified: 2026-08-06 -related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation, testing-quality-tests-that-cannot-fail, testing-mocking-captured-call-arguments] +related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation, testing-quality-tests-that-cannot-fail, testing-mocking-captured-call-arguments, testing-mocking-autouse-fixture-shadows-function-under-test, testing-strategy-real-cli-spot-check-for-new-execution-paths] --- # Deciding Whether to Mock, Fake, or Use the Real Dependency diff --git a/wiki/testing/quality/behavior-not-implementation.md b/wiki/testing/quality/behavior-not-implementation.md index 68c3b49..5e9dee5 100644 --- a/wiki/testing/quality/behavior-not-implementation.md +++ b/wiki/testing/quality/behavior-not-implementation.md @@ -9,7 +9,7 @@ sources: - https://abseil.io/resources/swe-book/html/ch12.html - https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html last_verified: 2026-07-10 -related: [testing-quality-minimum-case-set, testing-mocking-what-to-mock, testing-quality-guard-shape-vs-consequence, testing-quality-value-preserving-refactor-assertions, backend-common-change-impact-call-site-enumeration, testing-quality-source-text-wiring-assertions] +related: [testing-quality-minimum-case-set, testing-mocking-what-to-mock, testing-quality-guard-shape-vs-consequence, testing-quality-value-preserving-refactor-assertions, backend-common-change-impact-call-site-enumeration, testing-quality-source-text-wiring-assertions, testing-quality-cross-task-stub-assertions] --- # Asserting Behavior Through the Public Interface diff --git a/wiki/testing/quality/captured-log-message-assertions.md b/wiki/testing/quality/captured-log-message-assertions.md index 69a09d6..4498cfe 100644 --- a/wiki/testing/quality/captured-log-message-assertions.md +++ b/wiki/testing/quality/captured-log-message-assertions.md @@ -9,7 +9,7 @@ sources: - https://docs.pytest.org/en/stable/reference/reference.html#pytest.LogCaptureFixture.messages - https://github.com/pytest-dev/pytest/blob/main/src/_pytest/logging.py last_verified: 2026-08-14 -related: [testing-quality-tests-that-cannot-fail] +related: [testing-quality-tests-that-cannot-fail, testing-quality-narration-based-ordering-assertions] --- # Asserting on Captured Log Messages diff --git a/wiki/testing/quality/cross-task-stub-assertions.md b/wiki/testing/quality/cross-task-stub-assertions.md new file mode 100644 index 0000000..861c181 --- /dev/null +++ b/wiki/testing/quality/cross-task-stub-assertions.md @@ -0,0 +1,54 @@ +--- +id: testing-quality-cross-task-stub-assertions +domain: testing +category: quality +applies_to: [general] +confidence: field-tested +sources: + - "Field evidence, a Rust + React orchestrated repo (commit de8c07c and its HANDOFF.md §5 item 18), verified by direct git show / read against the checkout 2026-09-08" +last_verified: 2026-09-08 +related: [testing-quality-tests-that-cannot-fail, testing-quality-behavior-not-implementation, infrastructure-agent-orchestration-worktree-isolated-workers] +--- + +# A Test Asserting a Cross-Task Stub's Wording Instead of Its Structural Contract + +## When this applies + +In a multi-task (orchestrated or sequenced) run, one task lands a placeholder +or stub view or component (a shell task's UI, a scaffolded module) and its +test asserts the stub's literal content — placeholder text, a specific string +— while a different, later task is planned to replace that stub's content with +the real implementation. + +## Do this + +1. **Identify what the plan/contract states is stable across the stub-to-real + transition** — a root DOM class, a component's exported type, a function's + signature — and assert that instead of the stub's wording. +2. **When no such stable contract is written down yet, add it to the + plan/contract document before writing the test**: state which element + (class name, id, exported symbol) the replacing task is required to + preserve, so the assertion has a named target instead of an implicit + assumption about what will still be true after the swap. +3. **Route the test through the same selector the consuming code or other + tests already use** to find the element (a `data-testid` or root class + already used for tab-switching/routing logic), so the assertion tracks the + same seam production code depends on rather than an independent guess. + +## Edge cases + +| Case | Then | +|------|------| +| The stub has no stable structural element yet (a bare text node, no wrapping container) | Add one (a root class/id) as part of landing the stub, specifically so downstream tests and the replacing task have a contract to target | +| The replacing task is not yet planned or known | Assert structure rather than wording for any content the plan flags as provisional — an unplanned replacement breaks a wording assertion just as surely as a planned one | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Assert a stub's placeholder text (`expect(screen.getByText("DAG view coming soon"))`) | Assert its root class or structural contract (`container.querySelector("section.dag-view")`) | The follow-up task that replaces the stub's content necessarily changes the wording; only the structural contract is what the plan guarantees stays stable | + +## Sources + +- Field evidence (a Rust + React orchestrated repo, commit `de8c07c` "assert stub root class instead of placeholder text in App.test.tsx", verified by `git show` against the checkout 2026-09-08): the tab-switch tests originally asserted the stubs' placeholder text; per the plan's §D4/D5 amendment establishing `section.dag-view`/`section.timeline-view` as the stable root across implementations, they were rewritten to `container.querySelector(...)` on those roots, so the later tasks that replaced the stub content did not break the integration seam +- Same repo, HANDOFF.md §5 item 18 (verified by direct read): records "a shell task's test asserting stub wording breaks at the follow-up replacement; assert cross-task stubs by root class / structural contract, not wording" as a standing pitfall for orchestrated runs diff --git a/wiki/testing/quality/mutation-harness-file-custody.md b/wiki/testing/quality/mutation-harness-file-custody.md index 8b00dd3..e457403 100644 --- a/wiki/testing/quality/mutation-harness-file-custody.md +++ b/wiki/testing/quality/mutation-harness-file-custody.md @@ -10,15 +10,7 @@ sources: - https://git-scm.com/docs/git-status - https://docs.python.org/3/library/pathlib.html last_verified: 2026-08-27 -related: - [ - testing-quality-tests-that-cannot-fail, - testing-quality-harness-reverse-controls, - testing-quality-surviving-mutant-equivalence-triage, - testing-quality-completion-predicates, - testing-data-artifact-leakage-from-a-suite, - qa-process-scope-purity-checks, - ] +related: [testing-quality-tests-that-cannot-fail, testing-quality-harness-reverse-controls, testing-quality-surviving-mutant-equivalence-triage, testing-quality-completion-predicates, testing-data-artifact-leakage-from-a-suite, qa-process-scope-purity-checks, testing-quality-narration-based-ordering-assertions] --- # A Hand-Rolled Mutation Harness That Edits and Restores Files In Place diff --git a/wiki/testing/quality/narration-based-ordering-assertions.md b/wiki/testing/quality/narration-based-ordering-assertions.md new file mode 100644 index 0000000..cda23c4 --- /dev/null +++ b/wiki/testing/quality/narration-based-ordering-assertions.md @@ -0,0 +1,58 @@ +--- +id: testing-quality-narration-based-ordering-assertions +domain: testing +category: quality +applies_to: [general] +confidence: field-tested +sources: + - https://developer.apple.com/documentation/foundation/nstask/1408746-terminationhandler +last_verified: 2026-09-08 +related: [testing-quality-tests-that-cannot-fail, testing-quality-mutation-harness-file-custody, testing-async-async-testing, testing-quality-captured-log-message-assertions] +--- + +# An Ordering Test Built From Narration Strings at Existing Hook Sites + +## When this applies + +Writing a regression test for an ordering or timing invariant in concurrent or +async code (e.g. "the termination handler must be installed before the process +launches") by recording self-reported narration strings ("handlerAssigned", +"launching") at the code's existing instrumentation/hook call sites and +asserting their relative order. + +## Do this + +1. **Inject a hook that reads the real object's state at the critical moment**, + rather than a hook that logs a narration string at a call site the fix might + move. For a handler-before-launch invariant, read the real flag + (`process.terminationHandler != nil`, or the language equivalent) inside the + hook that fires at launch — a separately-logged "handlerAssigned" marker's + position in the code is an assumption, not a guarantee. +2. **Prove the test can fail before trusting it**: reintroduce the exact + original bug by mutation (move the state-mutating line back to its old, + broken position) and require the test to go red — the same manual-mutation + discipline as [testing-quality-tests-that-cannot-fail] step 1, applied + specifically to ordering/timing invariants. +3. **Get independent confirmation of the red run** when the invariant is + safety-critical (crash/race conditions): have a second person or session run + the same mutation. A mutation performed and read only by the session that + wrote the test is exactly the untrusted case step 2 exists to catch. + +## Edge cases + +| Case | Then | +|------|------| +| The hook call sites are the only instrumentation available and moving them is impractical | Read the real object's field through a debugger/inspector API at the hook, rather than trusting the call site's position to still track the code path it once matched | +| The mutation moves the fix to a place with no hook nearby at all | Add a hook at the new location too, or assert via polling the real object rather than narration — no single call site's presence decides the test's power | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Assert the order of two logged narration strings recorded at existing hook call sites | Inject a hook that reads the real object's state at the critical moment | Moving the actual state-mutating line does not move the hook calls written to match the old code, so a narration-order test silently degrades into a test of the narration, not the invariant | +| Trust a new ordering/timing regression test because it passes on the first run | Reintroduce the exact bug by mutation and require red before trusting it | A test built from self-reported narration can keep passing after the real bug returns | + +## Sources + +- https://developer.apple.com/documentation/foundation/nstask/1408746-terminationhandler — `terminationHandler` is a Foundation API with race-sensitive semantics (one handler settable at a time; if the process already finished, the block runs promptly) — the general class of invariant this case guards +- Field evidence (a Swift process-launching component, 2026-09; session-relayed, general mechanism confirmed against the API doc above): a first version of the test asserted the order of two logged strings ("handlerAssigned"/"launching") recorded at existing hook call sites; it still passed after mutating the source to reproduce the exact original race, because moving the state-mutating line did not move the hook calls. Replacing it with a hook reading `Process.terminationHandler != nil` at the call site correctly failed under the same mutation, confirmed independently by both the authoring session and an auditor diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index fe33077..7c94ae5 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -16,7 +16,7 @@ sources: - https://git-scm.com/docs/git-checkout - https://git-scm.com/docs/git-restore last_verified: 2026-08-27 -related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-schema-additions-under-a-golden-gate, testing-quality-differential-run-agreement, testing-quality-completion-predicates, testing-quality-guard-shape-vs-consequence, testing-quality-injected-clock-duration-assertions, testing-quality-write-path-assertions, testing-quality-value-preserving-refactor-assertions, testing-quality-unasserted-return-fields, testing-quality-stale-artifact-baselines, backend-common-change-impact-call-site-enumeration, platforms-shells-portable-shell-scripts, qa-document-verification-spec-document-gates, testing-quality-surviving-mutant-equivalence-triage, testing-quality-source-text-wiring-assertions, testing-quality-default-values-under-test, testing-mocking-captured-call-arguments, testing-quality-captured-log-message-assertions, testing-strategy-failing-test-first, testing-quality-mutation-harness-file-custody] +related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-schema-additions-under-a-golden-gate, testing-quality-differential-run-agreement, testing-quality-completion-predicates, testing-quality-guard-shape-vs-consequence, testing-quality-injected-clock-duration-assertions, testing-quality-write-path-assertions, testing-quality-value-preserving-refactor-assertions, testing-quality-unasserted-return-fields, testing-quality-stale-artifact-baselines, backend-common-change-impact-call-site-enumeration, platforms-shells-portable-shell-scripts, qa-document-verification-spec-document-gates, testing-quality-surviving-mutant-equivalence-triage, testing-quality-source-text-wiring-assertions, testing-quality-default-values-under-test, testing-mocking-captured-call-arguments, testing-quality-captured-log-message-assertions, testing-strategy-failing-test-first, testing-quality-mutation-harness-file-custody, testing-quality-narration-based-ordering-assertions, testing-mocking-autouse-fixture-shadows-function-under-test, testing-quality-cross-task-stub-assertions] --- # Proving a Test Can Fail diff --git a/wiki/testing/strategy/real-cli-spot-check-for-new-execution-paths.md b/wiki/testing/strategy/real-cli-spot-check-for-new-execution-paths.md new file mode 100644 index 0000000..f4633bf --- /dev/null +++ b/wiki/testing/strategy/real-cli-spot-check-for-new-execution-paths.md @@ -0,0 +1,62 @@ +--- +id: testing-strategy-real-cli-spot-check-for-new-execution-paths +domain: testing +category: strategy +applies_to: [general] +confidence: field-tested +sources: + - "Field evidence, a Rust multi-worker orchestration repo (commits e761ed3, b25ff04, 656d763 and HANDOFF.md §5 item 19), verified by direct git show / read against the checkout 2026-09-08" +last_verified: 2026-09-08 +related: [testing-strategy-test-level-choice, testing-mocking-what-to-mock, infrastructure-agent-orchestration-worktree-isolated-workers, qa-process-completion-claims] +--- + +# A New Real-CLI Execution Path Whose Deterministic Suite Is Green but Never Run for Real + +## When this applies + +You add a new execution path that drives a real CLI process or real external +environment (a `RealCli` worker mode, a real-subprocess adapter) alongside an +existing deterministic/scripted/fake path, and the deterministic test suite +for it is fully green. + +## Do this + +1. **Treat a green deterministic suite as evidence about the scripted double + only.** Write and run one real-CLI spot check — an ignored/manually invoked + integration test, or a manual run — that actually exercises the real process + for every new such path, run by the coordinator or a human, before relying + on the path in an unattended run. +2. **Target the spot check at what a scripted/faked double cannot model**: + process spawn preconditions (working directory existence, binary + resolution) and shared-resource lifetime (semaphore/pool permit scope held + across the path's whole lifetime vs. released per use). +3. **When the spot check finds a defect, add a deterministic regression test + for the specific mechanism** (e.g. a unit test for the cwd-creation + helper) so the fix has ongoing coverage — and keep the manual spot check as + the gate for the *next* new real-path addition. The deterministic + regression proves this one fix, not that the class of defect is closed for + future paths. + +| Failure class | Why a deterministic/scripted suite cannot see it | +|----------------|----------------------------------------------------| +| Spawn precondition (missing cwd → ENOENT) | The scripted double never calls the real spawn syscall, so a directory that was never created never fails | +| Shared-resource lifetime (a semaphore permit held for a whole runner's life instead of one turn, under a hard concurrency limit) | The fake/scripted path either doesn't route through the real pool, or completes fast enough that the limit is never actually contended | + +## Edge cases + +| Case | Then | +|------|------| +| The real spot check is expensive (real API cost, long runtime) | Run it once per new execution path, not per commit; gate it behind an ignore/manual-only marker so CI stays deterministic while the coordinator retains the one-time obligation | +| The spot check passes on the first try | Keep it as ignored integration coverage for the specific path, and still require the same one-time manual run for the *next* new real path — passing today does not retire the obligation for future additions | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Ship a new real-CLI/real-environment execution path on a fully green deterministic/scripted suite | Have the coordinator manually run one real-CLI spot check against it first | An ENOENT spawn failure from a never-created cwd, and a semaphore-permit-scope deadlock, were both undetectable in principle by the deterministic suite in the field case and were found only by running the real path | + +## Sources + +- Field evidence (a Rust multi-worker orchestration repo, commit `e761ed3` "create RealCli worker cli-cwd before spawn (ENOENT hang)", verified by `git show` 2026-09-08): the `RealCli` arm pointed the worker config's cwd at a directory never created, so `Command::current_dir` failed every worker's CLI spawn with ENOENT; every worker reported blocked and the run hung forever under a zero escalation timeout. Commit message: "Found by the coordinator's real-CLI spot check" +- Same repo, commits `b25ff04` and `656d763` ("scope HarnessPool permit to a single turn" / "scope RealCli worker pool permit to a turn, not the runner"): a pool permit was held for a worker's whole runner lifetime; with the default limit-2 pool, only 2 of 5 workers could ever start, stalling the run. Commit message: "a latent M5 deadlock the coordinator's real-CLI spot check found twice" +- Same repo, HANDOFF.md §5 item 19 (verified by direct read): both defects passed the deterministic (`Scripted`) suite and surfaced only in a real-CLI two-sprint run; the recorded rule is "when a new `RealCli` path is created, the coordinator manually runs one real-CLI spot check" diff --git a/wiki/testing/strategy/test-level-choice.md b/wiki/testing/strategy/test-level-choice.md index 504305d..ce3aafb 100644 --- a/wiki/testing/strategy/test-level-choice.md +++ b/wiki/testing/strategy/test-level-choice.md @@ -9,7 +9,7 @@ sources: - https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html - https://docs.pytest.org/en/stable/how-to/skipping.html last_verified: 2026-08-04 -related: [testing-quality-minimum-case-set, testing-mocking-what-to-mock, testing-data-test-data-and-isolation, testing-strategy-import-time-side-effects, testing-strategy-cross-layer-effect-tests] +related: [testing-quality-minimum-case-set, testing-mocking-what-to-mock, testing-data-test-data-and-isolation, testing-strategy-import-time-side-effects, testing-strategy-cross-layer-effect-tests, testing-strategy-real-cli-spot-check-for-new-execution-paths] --- # Choosing the Test Level for a Behavior