178 Playwright specs skip fleet-wide, and a skip is counted as a pass

[rubenvdlinde]

178 Playwright specs skip fleet-wide, and a skip is counted as a pass

Measured 2026-08-27 from the Playwright job logs of each app's most recent completed E2E run on development.

app	skipped	biggest clusters
buildiq	62	hydra-console 12, spec-coverage/buildiq-runtime 11, nldesign-theme-selection 6, chat-companion-streaming 6, docudesk-document-templates 5
decidiq	46	integration-registry 21, spec-coverage/resolution-minutes 5, rbac-authorization-workflow 4
dossiq	37	brp-kvk-initiator 6, pages 6, workflow-editor-canvas 3, related-case-linking 3
shillinq	21	budget-grid-view 5, budget-charts 4, receipt-extraction-consume 3, budget-known-costs 3
integriq	12	spec-coverage/connector-catalog 6, synchronization-workflow 2

Fleet totals for context: 3,041 specs passing, 0 failing — so this is the only material gap in E2E coverage.

Why these are not benign

The recurring shape is a skip conditioned on seeded data that the CI seeder never creates:

// decidiq/tests/e2e/integration-registry.spec.ts  (21 tests)
test.skip(!first, 'no meeting objects on this instance — seed at least one')

// shillinq/tests/e2e/budget-grid-view.spec.ts     (5 tests)
test.skip(!rendered, 'no LedgerGroup seeded for this administration')

Both seeders deliberately provision registers and schemas only — decidiq's ci-seed.sh header calls the register import "the ONLY legitimate seed source". So the object never exists unless an earlier spec happened to create one, which makes coverage depend on spec execution order and reports nothing-was-tested as success.

A skip cannot distinguish "not applicable here" from "the thing this needed was never created", and it reports the second as a pass.

The gate for this exists and now works

e2e-skip-blocking in the shared quality workflow reads the run's Playwright report — the only place a runtime test.skip(cond, reason) is visible, since the CI log prints counts with no reasons. It reports a spec file that executed zero tests, a skip deferring to a deploy state CI itself decides, and a skip carrying no reason.

It had never run in any app. Four independent breaks, each hiding the next:

1. check_e2e_skips.py existed only on .github main and was in none of the last 12 release tags → fixed by cutting v1.10.0
2. the workflow looked for it without the inner hydra-gates/ path segment composer installs it under → fix(quality): a missing skip-discipline gate must fail, not warn #595
3. a missing gate warned and exited 0, so opting in was a silent no-op → same PR
4. launchpad's lock pinned v1.9.0, predating the script → chore(deps): hydra-gates 1.10, so the skip-discipline gate can run launchpad#359

All four are now merged. launchpad is the first app with the gate genuinely enforcing.

Suggested path per app

1. Seed what the specs need, following each repo's documented fixture-marker convention — create through the established helper, track in a per-run ledger, delete in afterAll, and add any new schema to TEARDOWN_ORDER (a schema in the ledger but absent from that array leaks silently).
2. Where a skip really is legitimate — a genuinely absent optional app or external service — leave it, and make the reason say which service.
3. Then set e2e-skip-blocking: true for that app, so it can only stay clean.

Worked example: ConductionNL/decidiq#941 converts the 21-test integration-registry cluster using decidiq's existing governance-fixture helper (meeting was already in its TEARDOWN_ORDER).

[rubenvdlinde]

Correction: 178 is the raw skip count, not the violation count

I opened this counting every skipped spec. That overstates it, and the gate itself is the reason — it does not treat all skips as findings. classify() sorts a reason into no-reason (V3), deploy-state (V2), or allowed, and checks an ENVIRONMENT_ABSENCE list first so that a genuine environment gap is never reported as a deploy claim:

not reachable · no chat backend · unreachable · credential
llm provider · not provisioned · not installed · session does not survive

The clearest example is the cluster I listed as buildiq's largest. hydra-console.spec.ts skips 12 tests behind a real capability probe — it runs an actual query and fires only when OpenRegister reports the register missing — with this reason:

hydra-cache register is not provisioned on this instance — this suite is read-only against an externally seeded register and creates no data.

not provisioned is in that allow-list, so those 12 classify as allowed. They are not debt. The spec's own comment even says it is "not a blanket .skip", and it is right.

What is actually measured today

The gate now runs — but in 1 of 21 apps. Reading each app's latest completed E2E job:

launchpad	0/137 tests skipped (0.0%) across 30 spec files — V1 0 / V2 0 / V3 0
the other 20	no verdict: check_e2e_skips.py not found, step warns and exits 0

Only launchpad carries hydra-gates v1.10.0 (via ConductionNL/launchpad#359); the rest are on older pins that predate the script. So the per-app V1/V2/V3 numbers this issue needs do not exist yet for 20 apps.

Revised next step

Bumping conduction/hydra-gates to ^1.10.0 in the other 20 apps is what turns the raw counts above into real findings. Until then, treat the per-app totals in the original post as how many specs did not run, not as how many should have.

The two PRs already merged from this issue stand on their own either way, because both were V3 (no reason recorded at all, which is never allowed):

• fix(e2e): describe-level skips must record a reason integriq#1624 — two describe.skip blocks with no reason in the report
• fix(e2e): lift the FIXME reasons into annotations the report can carry dossiq#1413 — six test.fixme(title, fn) whose FIXME(#718) reasons lived only in comments