test: keep a live jspm outage from redding the required CI job

[vivek7405]

Problem

The required Unit + integration (node --test) CI job fetches from the live jspm CDN (api.jspm.io / ga.jspm.io), so a jspm outage or rate-limit reds CI on a pull request that has nothing to do with vendoring.

Proven on PR #1149 (a documentation-only change touching five prose files): the job failed on a blanket .webjs/ ignore is healed so the pin is committable in test/vendor-cli/vendor-cli.test.mjs:176, asserting assert.equal(code, 0) at L187 when webjs vendor pin exited 1. A re-run of the IDENTICAL commit (91205d71, no code change) then passed all ten checks. During the same window, a local webjs boot logged could not vendor 2 packages via jspm (status 401), and a direct curl to ga.jspm.io returned 200 again shortly after, so the outage was transient and external.

This is a recurrence of the class that #312 already established (De-flake the jspm install-order vendor test (live-CDN dependency), closed 2026-06-03, which reported the vendor tests flaking roughly 1 run in 4 across 18 PRs). #312 fixed ONE instance, in packages/server/test/vendor/vendor.test.js. The root cause, a required CI job depending on a third-party CDN's availability, survives. The cost is real: a false red blocks a merge, and because the project gates merges on green CI, it either burns an investigation (it burned one on #1149) or trains people to wave red checks through.

The surface is wider than the one failing test. Three third-party hosts are reached from the required job: api.jspm.io and ga.jspm.io (16 gated tests in packages/server/test/vendor/vendor.test.js, 5 gated in test/vendor-cli/vendor-cli.test.mjs), plus registry.npmjs.org from two tests that were never gated at all (vendor.test.js:1904 and :1930, via findOutdated, up to 10s each on an outage).

Implementation plan

Scope note. Almost all of this is test and CI infrastructure. The one exception is two small packages/server/src/vendor.js robustness fixes (a missing timeout and a missing cache reset), described at the end and carried in their own commit; nothing else in shipped source moves.

The core decision

Split the surface in two by FILENAME, and let the test runners enforce the split.

• *.live.test.{js,mjs} is the only shape allowed to touch a third-party host. Both runners exclude those files unless WEBJS_REQUIRE_NETWORK=1. A nightly non-required workflow sets that variable.
• Everything else is offline, driven by a shared jspm double.

The current NETWORK_OK / WEBJS_SKIP_NETWORK_TESTS gate is deleted entirely. It is the wrong shape: it is opt-OUT (so CI, which never sets it, always runs live), it is enforced by convention only, and it cannot express "this file is live" to the runner.

Why "keep the one live test gated in place" is not enough. After #1219 that test skips on a throw, a 5xx, or a 429, but still reds on a 4xx (packages/server/test/vendor/vendor.test.js:744), because the ground truth having just succeeded means a 4xx implies our own request is malformed. A WAF 403, a 404 on a moved route, or a rate limit expressed as a 403 therefore still reds an unrelated PR, which is the exact failure #1149 hit. Leaving it in the required job would leave the problem unfixed.

.github/workflows/ci.yml is not edited. Its on: block is shared by eleven jobs and it is concurrently edited by #1135 and #1257; the runner-level filter achieves the same thing with no rebase conflict.

Rejected

• env: WEBJS_SKIP_NETWORK_TESTS: 1 on the required unit job. It mutes 21 tests to fix 20 that should never have needed the network, deletes the one assertion worth keeping from every PR run, misses the two ungated registry.npmjs.org calls entirely, and is a third editor of a file fix: a flaky plain-link test aborts the whole Browser CI job #1135 and feat: let doctor gate CI without making every warning fatal #1257 already change.
• Committing .webjs/vendor/importmap.json into an in-repo app so its boot resolves offline. Explicitly rejected before: a committed pin file broke the Add e2e network probes for vendor-never-fetched + inert-route zero-JS elision #170 elision e2e and was reverted in Revert the blog vendor pin that broke the #170 elision e2e #196.
• A fixture file of recorded jspm responses with a periodic refresh. A refresh job nobody runs rots into a fixture asserting a graph jspm stopped producing, which is fix(server): stop pinning a yanked version in the vendor CDN test #1219's exact failure (@codemirror/lint@6.9.6 was yanked and redded main for every PR). A generated double has nothing to go stale.
• Making vendor pin treat a CDN failure as non-fatal in SHIPPED code. Pin failure is a real user-facing outcome (pinAll: refuses to write empty pin file when every install fails, L1115), and weakening the runtime to calm a test inverts the direction.

Files

New:

File	Purpose
test/fixtures/install-spec.mjs	splitInstall / packageName / subpath, lifted verbatim from test/e2e/fixtures/stub-jspm.mjs:98-130. Scope-aware and subpath-aware. No side effects.
test/fixtures/jspm-double.mjs	jspmDouble(opts) returning a fetch-compatible fn, plus withJspmDouble(opts, body). No import side effects.
test/fixtures/jspm-double-preload.mjs	Installs the double on globalThis.fetch at module load. The spawned-child arm.
packages/server/test/vendor/jspm-cdn.live.test.js	The live jspm contract tests, moved out of vendor.test.js.
test/vendor-cli/vendor-pin.live.test.mjs	One live webjs vendor pin run (no preload), so the real user command keeps live coverage.
test/repo-health/jspm-double.test.mjs	The double's own contract.
test/repo-health/live-cdn-callers.test.mjs	Enforces the filename policy.
.github/workflows/vendor-cdn.yml	Nightly, non-required, runs the live files with WEBJS_REQUIRE_NETWORK=1.

Modified:

• test/e2e/fixtures/stub-jspm.mjs: import the three install-spec helpers from the new shared module and re-export them, so test/repo-health/e2e-vendor-stub.test.mjs keeps passing verbatim. No behaviour change.
• test/vendor-cli/vendor-cli.test.mjs: preload wiring in runCli (L24), delete NETWORK_OK (L22) and all five gates (L65, L78, L85, L110, L165), fix the header (L7-8).
• packages/server/test/vendor/vendor.test.js: delete NETWORK_OK (L404) and all sixteen gates, convert thirteen tests to the double, move the live test out, fix the section comment (L399-403).
• scripts/run-node-tests.js: add the .live.test. filter beside the existing browserSeg and e2eSeg filters at L57-63.
• scripts/run-bun-tests.js: the same filter; drop ${SEP}vendor${SEP} from excludeSegs (L88) and fix both comments that call the vendor suite network-bound (L7 and L83-87).
• framework-dev.md: a policy section beside the existing fix(server): elision-OFF boot intermittently 404s a module, leaving components inert #1228 note at L102.

The double

Model api.jspm.io faithfully, because vendor.js's whole fallback ladder is built on its exact failure semantics (packages/server/src/vendor.js:383, :418-448, :548-626).

• POST https://api.jspm.io/generate: parse JSON.parse(init.body).install, answer 200 { map: { imports } }.
• Each install maps to https://ga.jspm.io/npm:<name>@<version><subpath>/double.js, keyed on <name><subpath>. Derive name, version, and subpath with test/fixtures/install-spec.mjs, never with the i.replace(/@[^@]*$/, '') shortcut the existing inline mocks use (vendor.test.js:588). That regex collapses dayjs@1.11.13/plugin/utc to dayjs, which would make partsByInstall.get(spec) miss, derivePinParts recover subpath: '', and pinAll report a spurious failure. The <name>@<version> substring must survive verbatim, since derivePinParts (vendor.js:702-704) parses the version straight out of the URL.
• Any install listed in opts.unresolvable fails the whole batch with 401 { error: 'Error: Not Found' }. That is real jspm behaviour (vendor.js:383) and it is what drives the per-package probe path.
• opts.transitives folds extra {spec: url} entries into the answer, so the flattened transitive pin path (vendor.js:1276-1291) becomes testable for the first time.
• opts.status forces a transient code (5xx or 429) for the transient-fallback tests.
• A GET of a URL the double minted answers bytes plus content-type: text/javascript, so downloadBundle (vendor.js:1079) and fetchIntegrity (vendor.js:1111) complete.
• The /double.js tail is load-bearing: real jspm never emits it, so a wiring assertion can key on it.

Refusal must be loud, and a throw is not loud enough. Every caller in vendor.js swallows a throw (vendor.js:441, :1096, :1122, :2213), so a broken preload would silently degrade to {} and a weak assertion would still pass. Instead:

• The double records every unexpected request on double.unexpected, and withJspmDouble asserts that array is empty in its finally.
• The preload prints [jspm-double] armed to stderr at load, prints each refusal, and sets process.exitCode = 1, so a refused CLI child fails runCli's assert.equal(code, 0).
• registry.npmjs.org is in the refusal set too.

This is a deliberate divergence from test/e2e/fixtures/stub-jspm.mjs:151-160, which passes through to the real network on refusal. Do not merge the two fixtures: that one must emit a real executable module for a browser, this one only needs jspm-shaped URLs and bytes.

Steps

1. test/fixtures/install-spec.mjs, lifted from stub-jspm.mjs; make stub-jspm.mjs import and re-export from it. Confirm test/repo-health/e2e-vendor-stub.test.mjs still passes unchanged.

2. test/fixtures/jspm-double.mjs per the section above, plus test/fixtures/jspm-double-preload.mjs reading its config from a WEBJS_JSPM_DOUBLE JSON env var (absent means resolve everything).

3. Wire the CLI test. In test/vendor-cli/vendor-cli.test.mjs, runCli (L24) prepends the preload flag. The flag is runtime-specific and neither runtime honours the other's, the test(e2e): stop the elision-off block depending on a live CDN #1229 lesson at test/e2e/e2e.test.mjs:85-89. Here the parent runtime IS the child runtime, since runCli spawns process.execPath and under bun test that is bun:

   const PRELOAD = resolve(__dirname, '..', 'fixtures', 'jspm-double-preload.mjs');
   statSync(PRELOAD); // a rename must fail loudly, not silently go live
   const preload = process.versions.bun
     ? ['--preload', PRELOAD]
     : ['--import', pathToFileURL(PRELOAD).href];

   The file:// form matters because runCli spawns with cwd set to the temp app dir, so a relative --import would resolve against the wrong directory. Then delete NETWORK_OK and every gate, and rewrite the header comment.

4. Prove the wiring for every call site, not one. Assert stderr matches /\[jspm-double\] armed/ inside runCli itself, so all seven call sites are covered by one line. Additionally assert in the L65 test that parsed.imports.picocolors ends with /double.js. The existing /^https:\/\/ga\.jspm\.io\/npm:picocolors@/ match (L75) is true against real jspm too, so it cannot detect the preload being dropped. Use match, never equality, on stderr elsewhere.

5. Convert the in-process tests in packages/server/test/vendor/vendor.test.js: L412, 420, 556, 1004, 1039, 1115, 1147, 1194, 1213, 1228, 1243, 1701, 1718. Every existing assertion about URL shape, bytes, integrity, prune, and pin-file removal survives verbatim, which is the point of a double that mints jspm-shaped URLs. Three need care:

   • L1115 (refuses to write empty pin file when every install fails) keeps asserting result.failed and a null pin file, driven by opts.unresolvable. Never relax it.
   • L1147 (partial success) traces batch-401, then two probes, then one survivor, at which point jspmGenerate short-circuits to a cached jspmResolveOne (vendor.js:611), so the total is three round trips, not four. Only assert a count if you pin it to that trace.
   • L420 (hits in-process cache) today only deepEquals two results, which passes even if a second round trip fired. With the double it can count calls and assert exactly one, so the conversion makes it discriminating for the first time.

   Add one assertion using opts.transitives that a flattened transitive lands in file.imports with a bundleFilenameWithSubpath-shaped bundle name. withJspmDouble calls clearVendorCache() on entry and exit.

6. Drop two gates that were never needed. L406 short-circuits before any fetch (vendor.js:549); L1950 already installs its own throwing fetch for its whole duration. Remove the gates, keep the bodies byte-identical.

7. Mock two ungated live calls the old gate missed. The updatePinned provider tests at vendor.test.js:1904 and :1930 reach registry.npmjs.org through findOutdated (vendor.js:1499), bounded by no gate and costing up to 10s each on an outage. Both assert only result.provider, so wrap them in the existing withMockedFetch (L466) answering a 404.

8. Move the live tests out. Create packages/server/test/vendor/jspm-cdn.live.test.js holding the #446 parity test (vendor.test.js:662-765) verbatim, plus one new test that re-anchors the premise the whole fallback ladder rests on: POST the same body with ['picocolors@1.1.1', 'this-package-truly-does-not-exist-xyz-789@99.0.0'] and assert !res.ok && res.status < 500 && res.status !== 429, i.e. jspm still fails the whole batch and still classifies an unknown package as permanent. Reuse the existing loud skip() helper (L693) for both. Without this, converting L556 and L1115 to the double would leave that premise asserted only by the double, which asserts our own model of jspm.

   Create test/vendor-cli/vendor-pin.live.test.mjs with one test: makeApp, then runCli(['vendor','pin']) with no preload, asserting a real ga.jspm.io URL and a sha384- integrity, so the actual user command keeps live coverage.

9. Promote skip to failure under WEBJS_REQUIRE_NETWORK. In the shared skip() helper, fail with the recorded reason instead of calling t.skip when the variable is set. This is the lever the nightly pulls; a normal run is unaffected.

10. Filter live files in both runners. In scripts/run-node-tests.js, add .filter((f) => process.env.WEBJS_REQUIRE_NETWORK || !f.includes('.live.test.')) beside the browserSeg and e2eSeg filters at L57-63. Add the same filter to scripts/run-bun-tests.js, and drop ${SEP}vendor${SEP} from excludeSegs (L88) so coherence.test.js, ensure-committable.test.js, prune-differential-property.test.js, prune-pin.test.js, and the converted vendor.test.js rejoin the Bun matrix. All five were read and are runtime-safe under Bun. No DENYLIST entry is needed, because the live file is filtered by name. Fix BOTH comments that call the suite network-bound (L7 and L83-87). If one of the five then fails under Bun for a genuine runtime reason, denylist that FILE with its real reason; do not restore the blanket segment. Bun coverage matters most here, since a globalThis.fetch swap is precisely where Node and Bun could diverge.

11. .github/workflows/vendor-cdn.yml: on: schedule (daily) plus workflow_dispatch, npm ci, then WEBJS_REQUIRE_NETWORK=1 node --test packages/server/test/vendor/jspm-cdn.live.test.js test/vendor-cli/vendor-pin.live.test.mjs. No pull_request trigger, so it can never be a required check. Add a final if: failure() step that opens or comments on a fixed-title tracking issue via gh (permissions: issues: write), because GitHub notifies only the workflow file's last committer and a nightly nobody watches is a green-forever placebo.

12. Guards.

    • test/repo-health/jspm-double.test.mjs, modelled on test/repo-health/e2e-vendor-stub.test.mjs: a /generate POST answers a map keyed by bare name with a /double.js URL per install; the install-spec case table from e2e-vendor-stub.test.mjs:52-72 holds for scoped and subpath forms; an install in unresolvable 401s the WHOLE batch, not just its own entry; a GET of a minted URL returns bytes with a JavaScript content type; an unminted jspm.io or registry.npmjs.org request is recorded on unexpected and never reaches the network; a malformed or absent body does not answer an empty map.
    • test/repo-health/live-cdn-callers.test.mjs: walk every test file under test/ and packages/*/test/, and assert that a live third-party host literal (api.jspm.io, ga.jspm.io, registry.npmjs.org) inside a fetch( argument, and any call to pinAll / updatePinned / auditPinned / findOutdated outside a withMockedFetch or withJspmDouble block, appears only in a *.live.test.* file. Shape the allowlist like run-bun-tests.js:39-59's DENYLIST ({ file, tests, why }) and exclude the guard's own path by exact repo-relative match. Do not assert on the literal NETWORK_OK: that certifies a spelling, not a property, and reds on a rename or a reformat.

13. Docs. A framework-dev.md section beside the fix(server): elision-OFF boot intermittently 404s a module, leaving components inert #1228 note at L102 stating the policy once: live third-party calls live only in *.live.test.*, both runners exclude those unless WEBJS_REQUIRE_NETWORK=1, the nightly vendor-cdn workflow is what stops a permanent skip from hiding and who is expected to watch it, and a new vendor test uses test/fixtures/jspm-double.mjs rather than the network. Plus the two stale header comments (vendor-cli.test.mjs:1-9, vendor.test.js:399-403). No docs site, AGENTS.md, skill, or scaffold change, since no public or agent-facing surface moves.

Counterfactuals (each must red when broken)

1. Delete the preload flag from runCli, and the [jspm-double] armed stderr assertion reds, as does the /double.js assertion.
2. Point the double at a name-only key (the i.replace(/@[^@]*$/, '') shortcut), and the subpath case in jspm-double.test.mjs reds.
3. Make the double answer a partial map instead of 401ing the batch, and the per-package-isolation plus pinAll total-failure tests red.
4. Remove the .gitignore exception-writing code, and the dogfood: webjs vendor pin writes to gitignored .webjs/vendor (opt-in pins not committable) #448 healing test reds (unchanged from today; the double does not touch the git half of that block).
5. Add a live-host fetch to any non-*.live.test.* file, and live-cdn-callers.test.mjs reds.
6. Drop WEBJS_REQUIRE_NETWORK from the nightly, and it runs zero live tests, which run-node-tests.js's existing zero-files guard plus the workflow's explicit file list surface.

Tests

• Unit: the two new repo-health guards above, plus every converted test keeping its existing assertions and three of them gaining a stronger one (the call count at L420, the transitive pin, the /double.js wiring).
• Bun parity: in scope and is steps 3 and 10. The vendor suite rejoins the Bun matrix and the CLI preload flag is runtime-selected, so a Node-only spelling of --import fails the matrix rather than silently skipping. No test/bun/*.mjs cross-runtime script is added, because no runtime-sensitive shipped source is touched.
• Browser / e2e: none, because nothing here renders or ships to a client. The browser-facing half of the same problem is already covered by test(e2e): stop the elision-off block depending on a live CDN #1229's e2e fixture.

Landmines for the implementing agent

• Verify the premise before implementing. Reproduce the failure deliberately (an --import stub answering 503 for api.jspm.io) and confirm test/vendor-cli/vendor-cli.test.mjs hard-fails TODAY, rather than assuming the mechanism from this description.
• Do not simply delete the live coverage. The vendor resolver's whole job is to talk to jspm, and dogfood: importmap per-package resolution yields inconsistent dep graph #446 / dogfood: webjs vendor pin writes to gitignored .webjs/vendor (opt-in pins not committable) #448 regressions were found by tests that really hit it. The goal is that a CDN hiccup cannot red a REQUIRED check, not that the contract goes untested.
• De-flake the jspm install-order vendor test (live-CDN dependency) #312 and fix(server): stop pinning a yanked version in the vendor CDN test #1219 are the prior incidents. Both are closed and both are directly reusable. Do not re-fix vendor.test.js's install-order test (De-flake the jspm install-order vendor test (live-CDN dependency) #312) or its conflicting-graph test (fix(server): stop pinning a yanked version in the vendor CDN test #1219); both already moved to deterministic mocks.
• vendor pin failing is sometimes CORRECT. pinAll: refuses to write empty pin file when every install fails (L1115) asserts a failure path on purpose, and pin names a found-but-uninstalled specifier (vendor-cli.test.mjs:133) asserts a non-zero exit. Drive those through opts.unresolvable; never relax the assertion.
• A double that answers everything is a double that proves nothing. An unknown install is a 401 or a recorded refusal, never an empty map, since an absent importmap entry is an unresolved-bare-specifier error that kills a whole page graph.
• A worktree without node_modules cannot run these tests. makeApp() symlinks the repo root node_modules into the temp app, so run npm run worktree:link first (fix(docs): the fresh-worktree node_modules remedy produces a broken worktree #1288). Related trap seen while diagnosing docs: explain the swr cache option on GET server actions #1149: through a linked root, @webjsdev/* resolves into the PRIMARY checkout, so a dirty sibling checkout silently changes local results.
• Invariants: no bundler or build step (no-build is permanent). packages/ stays plain .js with JSDoc, so the new helpers live under root test/ as .mjs. The pin file must stay committable (dogfood: webjs vendor pin writes to gitignored .webjs/vendor (opt-in pins not committable) #448). Do not commit a .webjs/vendor/importmap.json into any in-repo app (Revert the blog vendor pin that broke the #170 elision e2e #196).

Two vendor.js robustness gaps, fixed here

Both surfaced while mapping the network surface, both are directly about this issue (an unbounded third-party call in the required job, and cross-test contamination through the cache primitive this change leans on), so they are fixed in this change rather than deferred.

• fetchIntegrity (vendor.js:1111) has no AbortSignal, unlike every sibling fetch in the file (jspmCall 10s, fetchNpmJson 10s, fetchLiveIntegrity 10s). Default-mode pinAll calls it once per resolved URL (vendor.js:1306), and node --test applies no per-test deadline, so a hung CDN connection hangs the required job indefinitely rather than failing it. Give it the same INTEGRITY_FETCH_TIMEOUT_MS treatment its siblings have.
• clearVendorCache() (vendor.js:679) does not reset lastLiveResolveFailed, though it clears jspmCache and liveIntegrityCache. The flag is set by every transient path (vendor.js:474, :572, :592) and reset only inside resolveVendorImports (vendor.js:2295), so a pinAll test that never goes through that function leaks a true into whatever runs next. clearVendorCache is documented as the start-clean primitive and withJspmDouble leans on it, so it should clear the flag too.

Because these touch packages/server/src, they carry their own unit tests plus a test/bun/*.mjs cross-runtime assertion if the parity hook asks for one, and they ride a separate commit from the test-infrastructure work.

What changed during implementation

Two decisions moved from the plan above, both recorded on PR #1297 with the evidence.

The static live-caller scan was replaced by a runtime deny. The plan called for a repo-health test that scans test sources for a third-party host inside a fetch(. Three review rounds found three different ways it went blind, each hiding every call below it in the file, and each fix opened a new hole: a file-level exemption; then no regex-literal awareness, so /rel=["']modulepreload["']/ desynced the mask to end of file; then regex awareness that read the / in a closing tag inside a nested html`...` template as a regex opener. Deciding whether a / starts a regex means lexing JavaScript, and a hand-rolled lexer facing nested template literals full of markup will keep being wrong. A scan also cannot see the app-boot tests, which reach jspm transitively with no fetch( in their own source. Both runners now preload test/fixtures/deny-live-hosts.mjs, which answers 503 for the three hosts unless WEBJS_REQUIRE_NETWORK is set. No parsing, no blind spots inside the test process, and a third-party-dependent test fails on every run rather than only during an outage.

clearVendorCache was left alone. The plan listed it as a robustness fix. It is unobservable: resolveVendorImports is the flag's only reader and resets it on entry, and the pinned short-circuit above that returns ok: true outright. It was implemented, then reverted when the counterfactual showed the test passed just as happily with the fix removed. A comment now records why the line is absent. fetchIntegrity AND downloadBundle both gained the missing timeout.

Acceptance criteria

• A simulated jspm failure (unreachable host or forced non-200) does NOT fail the required Unit + integration CI job, including through a 4xx
• npm test passes with api.jspm.io, ga.jspm.io, and registry.npmjs.org all failing
• The #448 gitignore-healing property is still asserted, and still fails if the .webjs/vendor/ exception stops being written
• Live jspm contract coverage still runs somewhere (the nightly), and its failure is visible rather than silently skipped forever, including for the real webjs vendor pin command
• Unplugging the CLI preload reds a test, so the offline path cannot silently revert to the network
• WEBJS_REQUIRE_NETWORK=1 includes the live files AND turns their skips into failures
• A live third-party call added to any non-*.live.test.* file reds live-cdn-callers.test.mjs
• The Bun matrix runs the vendor suite again
• .github/workflows/ci.yml is unchanged by this work