A capacity witness that cannot flake, and the social card actually served

.claude/CLAUDE.md

@@ -193,7 +193,7 @@ directories under it are ignored (`node_modules/`, `.state/`, `.artifacts/`,
 | journey (465-action recipe)                            | **469 passed / 0 failed / 0 skipped**                                                          | 2026-08-15 |
 | smoke                                                  | **148 passed / 0 failed / 0 did not run**                                                      | 2026-08-14 |
 | mobile                                                 | **121 passed**                                                                                 | 2026-08-10 |
-| backend `go test -tags "test unittest" ./internal/...` | all 6 packages ok in ~11 s — service 336/337 specs (one pending), capability 37, middleware 46 | 2026-08-15 |
+| backend `go test -tags "test unittest" ./internal/...` | all 6 packages ok in ~10 s — service 337/338 specs (one pending), capability 37, middleware 46 | 2026-08-16 |
 | openreplay (9 tests)                                   | **13 passed / 0 skipped**                                                                      | 2026-08-11 |
 | frontend units (29 files)                              | **488 passed**                                                                                 | 2026-08-14 |
 
@@ -240,13 +240,15 @@ of this branch) and both report `ok` under the runner.
 
 What fails today is one SPEC, and it is the declared flake:
 `Capacity > never oversells the last place under simultaneous joins`
-(`capacity_test.go:326`) — the entry in the mutation runner's `KNOWN_FLAKY`,
+(`capacity_test.go:350`) — the entry in the mutation runner's `KNOWN_FLAKY`,
 which fails roughly one run in five under in-memory SQLite. Three consecutive
 re-runs of `./internal/service/` after it went green. The rest of that run:
 service 335 passed / 1 failed / 1 pending of 337, capability 37/37, middleware
 46/46, config 6/6, audit and storage `ok`. CI runs this command, so **that flake
 is a red CI run whenever it lands** — it is a test-side race to fix, not a
-runner quirk to route around.
+runner quirk to route around. As of 2026-08-16 no mutation VERDICT depends on it
+any more (see the capacity witness below), which removes the ambiguity from the
+manifest and changes nothing about CI.
 
 **API-to-UI coverage: 101 of 108 RPC declarations have a frontend caller.** The
 seven without one are accounted for in `docs/testing.md` —
@@ -887,24 +889,39 @@ don't count is precisely the shape that could hide a real one. Adding a line to
 `KNOWN_FLAKY` is a claim about the SUITE that wants justifying — never a way to
 quieten a mutation that is genuinely over-broad.
 
-⚠ **A flaky test can still be a genuine witness, and that trap fired within the
-hour.** That capacity spec hammers concurrent joins against a cap — which is
-precisely what `capacity.oversell-by-one` breaks — so under THAT mutation its
-failure is the evidence, and the first freeze had stripped it as noise. The
-rule: when a listed test really does witness a mutation, it belongs in that
-mutation's `expectReds`, where the excuse cannot reach it (the filter only ever
-looks at reds that are NOT expected). Excusing an extra prints that reminder
-every time.
-
-⚠ **And it flakes in BOTH directions, which no filter can excuse.** Observed
-2026-08-14: `capacity.oversell-by-one` came back MISMATCH with that same spec
-under "expected but stayed GREEN" — its witness had passed under the mutation.
-Same root cause (a join that errors out under SQLite contention seats one fewer,
-so the oversell never materialises), opposite symptom, and the `KNOWN_FLAKY`
-list cannot help: an expected red that does not arrive is exactly what a
-MISMATCH is for. Three re-runs of that one id came back EXACT. So a MISMATCH
-naming ONLY a `KNOWN_FLAKY` test in the "stayed GREEN" column wants a re-run
-before it is believed — the same courtesy the extras column already gets.
+⚠ **A flaky test can be genuine evidence and still be the wrong thing to judge
+by — closed 2026-08-16.** That capacity spec hammers concurrent joins against a
+cap, which is precisely what `capacity.oversell-by-one` breaks, so under THAT
+mutation its failure is the evidence and the first freeze had stripped it as
+noise. The fix then was to list it in the entry's own `expectReds`, where the
+excuse cannot reach it (the filter only ever looks at reds that are NOT
+expected). That bought a correct reading in one direction and a coin flip in the
+other: **it flakes in BOTH directions, and no filter can excuse the second
+one.** Observed 2026-08-14, `capacity.oversell-by-one` came back MISMATCH with
+that same spec under "expected but stayed GREEN" — its witness had PASSED under
+the mutation. Same root cause (a join that errors out under SQLite contention
+seats one fewer, so the oversell never materialises), opposite symptom, and an
+expected red that does not arrive is exactly what a MISMATCH is for.
+
+So the entry is judged by something that cannot flake.
+`Capacity > seats exactly the capacity, counting the roster after every join`
+makes the same END-STATE claim the race spec makes — the confirmed roster equals
+the cap, read back from the DB — by filling an event to exactly its capacity one
+join at a time and asserting the next arrival queues with the count unmoved.
+Three capacities, because a rule that is off by one is off by one at every cap,
+and the seated joins at 2 and 3 are the positive control against an event that
+simply queues everybody. It fails under the mutation in 0.012 s with no
+goroutines anywhere. Five consecutive `mutate.sh run capacity.oversell-by-one`
+came back EXACT.
+
+**The rule that survived: a verdict may only rest on witnesses that cannot
+flake.** The race spec is KEPT as a test — it pins `capacityMu`, which once
+genuinely oversold — and it is out of `expectReds`, so its red now lands in the
+extras column, is excused by `KNOWN_FLAKY`, and is PRINTED with its reason on
+every run of that entry. The evidence is still on screen; it is no longer on the
+ballot. ⚠ The flake itself is still open and still reds CI (see
+`just check::test -c backend` above) — this changes what the MUTATION is judged
+by, not the underlying test-side race.
 
 ## Container traps (Windows/macOS hosts) — read before touching compose
 
@@ -1120,6 +1137,34 @@ Two things measured while building that, both worth keeping:
   retries, and rolls the old tree back if the second rename fails, so
   `build/service` is never left missing. Anything else here that renames a
   directory on this mount needs the same treatment.
+- ⚠ **But one cause of that EPERM IS reproducible — a server still SERVING the
+  tree** (2026-08-16). `frontend-build.sh build` with the :8081 adapter-node
+  server up burned all five attempts and gave up; `prod-frontend.sh stop`, then
+  the very next attempt, succeeded — with the :8082 server still running. So
+  when the retries all fail, the question is who is serving it, not the
+  filesystem. The order is **stop, build, start**, which is what
+  `prod-frontend.sh start` already does; calling the helper directly against a
+  live server is the case that hits the wall.
+
+⚠ **Swapping the tree in does NOT reach a server that booted against the old
+one, and the bytes will still be right** (2026-08-16). adapter-node serves
+`build/service/client` through `sirv`, which builds its manifest — sizes and
+ETags included — ONCE at boot. After a swap it streams the file that is on disk
+NOW while advertising the length and ETag of the file that was there when it
+started. Measured on the tunnel's :8082 server after rebuilding the social card:
+`Content-Length: 85099`, `ETag: W/"85099-…"`, and 58,130 correct bytes on the
+wire, every request, `curl: (18) transfer closed with 26969 bytes remaining`.
+
+**This is a silent-green trap aimed straight at how you would check.** Hash what
+arrived and it MATCHES the file on disk — the body is genuinely the new card —
+so a `sha256sum` of the download says the deploy worked while every real client
+sees a truncated image and every cache honours an ETag that is a lie. The card
+exists for link-preview crawlers, which is exactly the audience that would have
+got the broken one. **Compare the HEADERS, and read curl's exit code, not just
+its output.** Any rebuild has to restart every server on that tree —
+`prod-frontend.sh start` for :8081 and `prod-serve.sh start <url> --no-build`
+for :8082 — because there are two of them and only one of them is ever the one
+you were thinking about.
 
 **6. An empty list is not an answer — say "I could not ask"** (fixed
 2026-08-13). The built :8081 server keeps ONE module-scope gRPC channel

.claude/skills/hackathon-e2e/mutations/manifest.jsonl

@@ -14,7 +14,7 @@
 {"id": "auth.require-user.admits-anon", "property": "RequireUser rejects the anonymous subject: endpoints that act on a PERSON must not accept 'anonymous' as an identity.", "arena": "go", "tier": "fast", "file": "components/backend/internal/middleware/auth.go", "find": "\tif sub == AnonSubject {\n\t\treturn \"\", nil, status.Error(codes.Unauthenticated, \"sign in to do that\")\n\t}", "replace": "\tif sub == AnonSubject && false {\n\t\treturn \"\", nil, status.Error(codes.Unauthenticated, \"sign in to do that\")\n\t}", "crossRef": ["act2.anonymous.register", "act4.team.anon", "act6.submit.anon"], "expectReds": ["middleware::Auth Middleware > RequireUser Function > refuses the anonymous subject", "service::TeamService with no credentials > tells an anonymous caller to sign in rather than whether a submission exists", "service::TeamService with no credentials > tells an anonymous caller to sign in rather than whether a team exists"]}
 {"comment":"─── Capacity: the FCFS rule and the fairness of the queue ───"}
 {"id":"capacity.queue-fairness","property":"Once anyone is waiting, a new joiner queues BEHIND them rather than sniping a freed seat.","arena":"go","tier":"fast","file":"components/backend/internal/service/capacity.go","find":"\treturn confirmed >= int(*maxParticipants) || waiting > 0","replace":"\treturn confirmed >= int(*maxParticipants)","expectReds":["service::Capacity > leaves a freed place to the organizer and keeps the queue unjumped","service::joinLandsWaitlisted > queues behind existing waiters even when a place is free"],"crossRef":["act2.cap.race","act2.cap.ui.queued"]}
-{"id": "capacity.oversell-by-one", "property": "The capped event seats exactly max_participants — the full test is >=, and > sells one seat too many.", "arena": "go", "tier": "fast", "file": "components/backend/internal/service/capacity.go", "find": "\treturn confirmed >= int(*maxParticipants) || waiting > 0", "replace": "\treturn confirmed > int(*maxParticipants) || waiting > 0", "expectReds": ["service::Capacity > hands out places first-come-first-served on a capped event", "service::Capacity > leaves a freed place to the organizer and keeps the queue unjumped", "service::Capacity > lets an organizer approve past capacity", "service::Capacity > never oversells the last place under simultaneous joins", "service::Capacity > reports the current state on an idempotent re-join", "service::joinLandsWaitlisted > waitlists once the confirmed roster reaches capacity"], "note": "The capacity concurrency spec is on the runner's KNOWN_FLAKY list AND is a genuine witness here: this mutation makes the cap off-by-one, which is exactly what that spec hammers. It is listed explicitly so the flake excuse — which only applies to UNEXPECTED reds — cannot swallow the evidence."}
+{"id": "capacity.oversell-by-one", "property": "The capped event seats exactly max_participants — the full test is >=, and > sells one seat too many.", "arena": "go", "tier": "fast", "file": "components/backend/internal/service/capacity.go", "find": "\treturn confirmed >= int(*maxParticipants) || waiting > 0", "replace": "\treturn confirmed > int(*maxParticipants) || waiting > 0", "expectReds": ["service::Capacity > hands out places first-come-first-served on a capped event", "service::Capacity > leaves a freed place to the organizer and keeps the queue unjumped", "service::Capacity > lets an organizer approve past capacity", "service::Capacity > reports the current state on an idempotent re-join", "service::Capacity > seats exactly the capacity, counting the roster after every join", "service::joinLandsWaitlisted > waitlists once the confirmed roster reaches capacity"], "note": "`service::Capacity > never oversells the last place under simultaneous joins` also goes red under this mutation and is deliberately NOT listed. It is the runner's one KNOWN_FLAKY entry, and it flakes in BOTH directions — it has stayed GREEN under this very mutation, which is a MISMATCH no excuse list can reach, because expectReds is by design where the flake filter does not apply. Every witness that decides a verdict has to be one that cannot flake. The property is carried instead by `seats exactly the capacity, counting the roster after every join`, which makes the same end-state claim (the confirmed roster equals the cap, read back from the DB) by walking the seats one join at a time. The race spec stays as a test — it pins capacityMu, which once genuinely oversold — and its red still PRINTS on this entry, excused with its reason, so the evidence is on screen without being on the ballot."}
 {"id":"capacity.unlimited-confirms","property":"Zero or unset capacity means UNLIMITED and keeps the approval model — everyone lands on the waiting list for an organizer to confirm.","arena":"go","tier":"fast","file":"components/backend/internal/service/capacity.go","find":"\t\t// Unlimited: the approval model — everyone starts on the waiting list.\n\t\treturn true","replace":"\t\t// Unlimited: the approval model — everyone starts on the waiting list.\n\t\treturn false","expectReds":["service::Capacity > clears the capacity back to unlimited with 0","service::Capacity > keeps the approval model for uncapped events","service::HackathonService > Join > allows authorized user to join hackathon","service::HackathonService > Join > returns success if user already joined (idempotent)","service::joinLandsWaitlisted > treats zero and negative capacity as unlimited","service::joinLandsWaitlisted > waitlists everyone when capacity is unset (the approval model)"],"crossRef":["act2.join.bob"]}
 {"comment":"─── Deadline windows, and the now-anchored overrides ───"}
 {"id": "window.never-closes", "property": "A hackathon with a windows row and a passed deadline refuses the action with FailedPrecondition.", "arena": "go", "tier": "fast", "file": "components/backend/internal/service/config_service.go", "find": "\t\tif closes == nil || !now.After(*closes) {\n\t\t\treturn false\n\t\t}", "replace": "\t\tif closes == nil || !now.After(*closes) || true {\n\t\t\treturn false\n\t\t}", "crossRef": ["act5.window.regclosed", "act6.window.sublate", "act4.window.preflate"], "expectReds": ["service::Window enforcement > refuses a join once the registration window has closed", "service::Window enforcement > stops honouring an override once it has expired"]}

.claude/skills/hackathon-e2e/mutations/run.mjs

@@ -680,15 +680,24 @@ const ARENAS = {
  * is a claim about the SUITE that someone has to justify — not a way to quieten
  * a mutation that is genuinely over-broad.
  *
- * ⚠ **A flaky test can still be a genuine witness**, and that trap fired within
- * an hour of this list existing. The capacity spec below hammers concurrent
- * joins against a cap — which is exactly what `capacity.oversell-by-one` breaks,
- * so under THAT mutation its failure is the evidence, not noise. The rule is:
- * when a listed test is a real witness for a mutation, it goes in that
- * mutation's `expectReds`, where the excuse does not apply (this filter only
- * ever looks at reds that are NOT expected). Excusing an extra therefore prints
- * a warning saying so, because "ignored" is the one word that could hide the
- * thing the mutation was written to find.
+ * ⚠ **A flaky test can be genuine evidence and still be the wrong thing to
+ * judge by**, and the capacity spec below has been both. It hammers concurrent
+ * joins against a cap — exactly what `capacity.oversell-by-one` breaks — so its
+ * red under THAT mutation is the evidence, not noise. The first fix was to list
+ * it in that entry's `expectReds`, where this filter cannot reach it (it only
+ * ever looks at reds that are NOT expected). That bought a correct reading in
+ * one direction and a coin flip in the other: an expected red that does not
+ * ARRIVE is a MISMATCH, and this spec flakes green under the mutation too (a
+ * join that errors out under SQLite contention seats one fewer, so the oversell
+ * never materialises). No excuse list can help there, by construction.
+ *
+ * The rule that survived: **a verdict may only rest on witnesses that cannot
+ * flake.** Write a deterministic witness for the same property, name THAT in
+ * `expectReds`, and leave the flaky one to be excused here — where its red is
+ * still printed, with its reason, on every entry it touches. That is what
+ * `Capacity > seats exactly the capacity, counting the roster after every join`
+ * is for. Excusing an extra prints a warning precisely because "ignored" is the
+ * one word that could hide the thing the mutation was written to find.
  */
 const KNOWN_FLAKY = [
   {
@@ -1082,8 +1091,12 @@ function main() {
           console.log(C.dim(`      declared in KNOWN_FLAKY: ${flakyReason(x)}`))
           console.log(
             C.dim(
-              `      if this test is a REAL witness for this mutation, put it in ` +
-                `expectReds — the excuse only applies to unexpected reds.`,
+              `      if this test is a REAL witness for this mutation, that is worth ` +
+                `knowing — but do NOT answer it by listing a flaky test in ` +
+                `expectReds. The excuse cannot reach a listed test, so the entry ` +
+                `then flakes in both directions (a red that stays green is a ` +
+                `MISMATCH). Write a deterministic witness for the same property ` +
+                `and name that one instead.`,
             ),
           )
         } else {

.claude/skills/lib/frontend-build.sh

@@ -123,10 +123,18 @@ build_locked() {
     #       Permission denied
     #
     # Observed 2026-08-13 mid-run and NOT reproducible a minute later with the same
-    # processes running and no open descriptors anywhere under the tree — so it is
-    # the filesystem, not a lock we could take or a handle we could close. An
-    # abort here is safe (the working tree is untouched) but it fails a build for a
-    # reason that clears itself, which is its own kind of flake.
+    # processes running and no open descriptors anywhere under the tree — so at
+    # least some of it is the filesystem, not a lock we could take or a handle we
+    # could close. An abort here is safe (the working tree is untouched) but it
+    # fails a build for a reason that clears itself, which is its own kind of flake.
+    #
+    # ⚠ ONE cause IS reproducible, found 2026-08-16: a server still SERVING this
+    # tree. `frontend-build.sh build` called with the :8081 adapter-node server up
+    # failed all five attempts; `prod-frontend.sh stop` and the very next attempt
+    # succeeded, with the :8082 server still running. So the retries are not the
+    # answer when a live server holds it — the order is stop, build, start, which
+    # is exactly what prod-frontend.sh's `start` already does. Calling this script
+    # directly against a running server is the case that hits the wall.
     local old="$OUT_PARENT/.service-old-$$" i
     rm -rf "$old"
     for i in 1 2 3 4 5; do