fix: keep a live jspm outage from redding the required CI job

.github/workflows/vendor-cdn.yml

@@ -0,0 +1,138 @@
+name: Vendor CDN contract (nightly)
+
+# Runs the only tests that talk to the real jspm CDN (#1150).
+#
+# Why they are not in CI: the required `Unit + integration` job used to resolve
+# vendors live, so a jspm outage redded pull requests that had nothing to do
+# with vendoring. PR #1149, a five-file documentation change, was blocked that
+# way and passed on a re-run of the identical commit. Both test runners now
+# skip `*.live.test.*` unless WEBJS_REQUIRE_NETWORK is set.
+#
+# Why they still exist somewhere: deleting the live coverage was never the
+# goal. The vendor resolver's whole job is to talk to jspm, and the offline
+# double can only ever return what this repo already believes about the API.
+# Two things it cannot vouch for are checked here for real: that our merged
+# output equals jspm's own unified graph (#446), and that jspm still fails a
+# WHOLE batch permanently when one install is unresolvable, which is the
+# premise the entire per-package fallback ladder in vendor.js rests on.
+#
+# Why nightly rather than on a pull request: a live check on a PR is a live
+# check, whatever job it sits in. Moving it to a non-required job on the
+# `pull_request` trigger would still queue on every PR and still go red on an
+# outage; it would just be a red somebody is told to ignore, which is how a
+# real failure gets ignored too.
+#
+# WEBJS_REQUIRE_NETWORK selects the live files and lifts the test-run deny. It
+# does NOT promote their upstream-trouble skip into a failure, and that
+# separation is deliberate: it briefly did both, and since this job always sets
+# it, the transport-level skip was unreachable wherever the tests actually run.
+# A single jspm 503 or DNS blip at 04:20 UTC would then have redded the job and
+# filed the issue below, which is precisely the cry-wolf failure the paragraph
+# above argues against.
+#
+# A permanently skipping test still must not pass for a healthy one, so the run
+# is scanned for skips and annotates a warning instead. That is visible in the
+# run summary without waking anyone for an outage. `WEBJS_FAIL_ON_SKIP=1`
+# promotes a skip to a failure when you want to force the question by hand.
+#
+# There is deliberately no `pull_request` trigger, so this can never become a
+# required check and can never block a merge.
+
+on:
+  schedule:
+    # 04:20 UTC daily. Off the hour on purpose: GitHub queues scheduled jobs
+    # from every repository at :00, so an on-the-hour cron is the one most
+    # likely to be delayed or dropped.
+    - cron: '20 4 * * *'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  # Needed by the failure step below, which is what stops this from being a
+  # job nobody watches. GitHub notifies only the workflow file's last
+  # committer about a failed scheduled run.
+  issues: write
+
+concurrency:
+  group: vendor-cdn
+  cancel-in-progress: false
+
+jobs:
+  live:
+    name: Live jspm contract
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-node@v6
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - name: Run the live CDN tests
+        id: live
+        env:
+          WEBJS_REQUIRE_NETWORK: '1'
+        run: |
+          set -o pipefail
+          node --test \
+            packages/server/test/vendor/jspm-cdn.live.test.js \
+            test/vendor-cli/vendor-pin.live.test.mjs 2>&1 | tee live.log
+
+      - name: Warn if a live check only skipped
+        # Runs even when the step above failed, so a partial skip is still
+        # reported. A skip means jspm could not answer, which is upstream's
+        # problem, not a regression; it is surfaced rather than escalated.
+        if: always()
+        run: |
+          set -euo pipefail
+          # Counted from the markers the tests print THEMSELVES, not from the
+          # reporter. Which reporter `node --test` picks depends on the Node
+          # version and on whether stdout is a TTY, and guessing wrong here is
+          # silent: the count reads 0 in exactly the runs that skipped. Both
+          # live files print `[<file>] SKIP <reason>` from their own skip
+          # helper, which is ours and does not move.
+          # `grep -c` on a MISSING file prints nothing, so a bare command
+          # substitution yields the empty string, not 0, and an inequality
+          # against "0" is then true. The step is `if: always()`, so that
+          # reported a jspm skip for runs where the live tests never ran at all
+          # (a failed checkout, npm ci, or a cancellation). Default explicitly.
+          if [ ! -f live.log ]; then
+            echo 'no live.log; the test step did not produce output'
+            exit 0
+          fi
+          skipped=$(grep -c '] SKIP ' live.log || true)
+          skipped=${skipped:-0}
+          if [ "${skipped}" != "0" ]; then
+            echo "::warning title=Live jspm check skipped::jspm.io could not answer ${skipped} check(s). \
+          Not a regression, but if this repeats for days the live coverage has stopped running. \
+          Re-run with WEBJS_FAIL_ON_SKIP=1 to force it to fail instead."
+            grep '] SKIP ' live.log || true
+          fi
+          echo "skip markers: ${skipped}"
+
+      - name: Report a failure on the tracking issue
+        if: failure()
+        env:
+          GH_TOKEN: ${{ github.token }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          TITLE='Nightly live jspm contract check is failing'
+          # One issue, reopened and commented rather than duplicated, so a week
+          # of failures is one thread instead of seven issues.
+          NUM=$(gh issue list --state all --search "$TITLE in:title" \
+                  --json number,title \
+                  --jq "[.[] | select(.title == \"$TITLE\")] | first | .number // empty")
+          BODY="The nightly live jspm contract check failed: $RUN_URL
+
+          Either jspm changed something the resolver depends on, or the fixture
+          the parity test pins has stopped resolving. Neither blocks a merge:
+          nothing here runs in a required check. See the header of
+          \`packages/server/test/vendor/jspm-cdn.live.test.js\` for what the two
+          tests assert and why they are live."
+          if [ -n "$NUM" ]; then
+            gh issue reopen "$NUM" || true
+            gh issue comment "$NUM" --body "$BODY"
+          else
+            gh issue create --title "$TITLE" --label bug --assignee vivek7405 --body "$BODY"
+          fi

framework-dev.md

@@ -107,7 +107,35 @@ An ES module graph instantiates as a unit, so a failure at either point means `a
 
 So the OFF server boots with `test/e2e/fixtures/stub-jspm.mjs` preloaded, which answers the `api.jspm.io/generate` call from this repo's `node_modules` and points `dayjs` at a `data:` URL carrying those bytes. Stubbing the API call closes both holes at once, because the URL the browser fetches is whatever that map says. The ON server is left alone, since it resolves nothing.
 
-Two things to keep in mind when touching this. The stub serves only the packages listed in its `LOCAL_VENDORS` map and passes everything else through to the real API, so **a vendor added to the blog later needs an entry there.** That failure is not silent: one unserviceable install makes the stub refuse the whole batch, the real API answers, and the block's first test fails naming the CDN url it got instead of a `data:` one. The same test is what catches the wiring itself going away, so do not delete it to make a new vendor pass. And the preload flag is runtime-specific (`--import` on Node, `--preload` on Bun, neither honouring the other, and Bun ignoring `NODE_OPTIONS`), which is why it is passed as argv through `preloadArgs` rather than an env var; the `E2E (blog served on Bun)` CI job is what a Node-only spelling would silently skip.
+Two things to keep in mind when touching this. The stub serves only the packages listed in its `LOCAL_VENDORS` map and passes everything else through to the real API, so **a vendor added to the blog later needs an entry there.** That failure is not silent: one unserviceable install makes the stub refuse the whole batch, the real API answers, and the block's first test fails naming the CDN url it got instead of a `data:` one. The same test is what catches the wiring itself going away, so do not delete it to make a new vendor pass. And the preload flag is passed as argv through `preloadArgs` rather than an env var, because Bun ignores `NODE_OPTIONS` outright (measured: `NODE_OPTIONS=--import ... bun -e 0` loads nothing). The two flags are not symmetric, so do not reason from the Node side: `node --preload` is a hard `bad option` error, while `bun --import` currently works as an alias. Selecting per runtime anyway is what keeps this from depending on Bun continuing to accept a Node spelling.
+
+---
+
+### Live third-party calls live only in `*.live.test.*` files (#1150)
+
+No required check may FAIL because a third party is down. The required `Unit + integration` job used to resolve vendors against the live jspm CDN, so a jspm outage redded pull requests that had nothing to do with vendoring; PR #1149, a five-file documentation change, is the one that finally made the case (it failed on the `#448` gitignore-healing test and passed on a re-run of the identical commit).
+
+Plenty of required tests still TRY. No in-repo app carries a pin file, so every test that cold-boots one (`test/preload-subset.test.mjs`, the `test/docs/*` boot tests, `test/integration/blog-http.test.mjs`, `packages/server/test/elision/differential-elision.test.js`) asks `resolveVendorImports` to resolve its vendors on the first request. Under the deny those calls get a 503 without leaving the process, and each test still passes in a few seconds, because the resolve fails OPEN: an unreachable CDN yields a partial importmap and a warning, never a throw, and none of them assert on a vendor entry. That is what makes denying safe rather than disruptive, and it is why the deny prints one line per refused url: the list is there if that ever stops being true.
+
+The rule is carried by the FILENAME, so the test runners can enforce it rather than leaving it to discipline. `scripts/run-node-tests.js` and `scripts/run-bun-tests.js` both drop any `*.live.test.*` file unless `WEBJS_REQUIRE_NETWORK=1` is set. Everything else resolves against `test/fixtures/jspm-double.mjs`, an offline double that models jspm rather than merely answering it (a 5xx or 429 is transient and retries per package, a 4xx probes per install, and an unresolvable install fails the WHOLE batch, which is the premise `jspmGenerate`'s fallback ladder is built on).
+
+This replaced a `WEBJS_SKIP_NETWORK_TESTS` gate that could not work: it was opt-OUT, so CI, which never set it, always ran live; it was convention rather than something the runner could check; and two `registry.npmjs.org` callers were never covered by it at all. Leaving the one live parity test gated in place would not have been enough either, since after #1219 it still reds on a 4xx, and a WAF 403 or a moved route is exactly the shape #1149 hit.
+
+Four things to keep in mind when touching this.
+
+**A new vendor test uses the double, not the network.** `withJspmDouble(opts, body)` installs it, clears the vendor caches on both sides, and fails the test on any request the double was not asked to serve. Refusals are RECORDED rather than thrown on purpose: every fetch caller in `packages/server/src/vendor.js` catches, so a throw would be indistinguishable from the CDN being down and would quietly weaken whatever test hit it. The runtime deny answers 503 for the same reason, since that is the shape those call sites classify as transient.
+
+**The deny is at RUNTIME, and that was learned the hard way.** Both runners preload `test/fixtures/deny-live-hosts.mjs`, which answers 503 for jspm.io and registry.npmjs.org unless `WEBJS_REQUIRE_NETWORK` is set. It needs no parsing, and within the test process it has no blind spots (a spawned child is the exception, below). It covers the transitive callers a source scan structurally cannot see: the app-boot tests reach jspm through `resolveVendorImports` with no `fetch(` anywhere in their own source. A test that depends on a third party now fails on EVERY run rather than only during an outage, which arrives the day it is written instead of months later.
+
+The first three attempts were a STATIC scan over test sources, and each went blind a different way: a file-level exemption, so one `withMockedFetch` anywhere excused every live call in the file; then no regex-literal awareness, so `/rel=["']modulepreload["']/` desynced the mask to end of file; then regex awareness that read the `/` in `</li>` inside a nested `` html`...` `` template as a regex opener, swallowing the closing backtick. Each fix opened a new hole, because deciding whether a `/` starts a regex means lexing JavaScript, and a hand-rolled lexer facing nested template literals full of markup will keep being wrong. **Do not reintroduce it.** If the deny needs to be tighter, tighten the deny.
+
+**A spawned child does not inherit the deny.** `test/vendor-cli/vendor-cli.test.mjs` runs the CLI in another process, so it passes its own preload and asserts a `[jspm-double] armed` marker on every spawn, which reds all ten of its tests if the flag is dropped. A new test that spawns a process and vendors needs the same treatment.
+
+**The nightly is what stops a permanent skip from hiding.** `.github/workflows/vendor-cdn.yml` runs the live files with `WEBJS_REQUIRE_NETWORK=1`, which selects them and lifts the deny. It does NOT promote their upstream-trouble skip into a failure: a jspm outage is not a regression, and a job that reds on one is a job whose reds get ignored. `WEBJS_FAIL_ON_SKIP=1` promotes, by hand, and the nightly does not set it; a skip surfaces as a warning annotation instead. A genuine failure opens or comments on one fixed-title tracking issue, since GitHub notifies only the workflow file's last committer about a failed scheduled run.
+
+**Do not add a `pull_request` trigger to that workflow.** A live check on a PR is a live check whatever job it sits in; making it non-required would just produce a red somebody is told to ignore, which is how a real failure gets ignored too.
+
+**`.github/workflows/ci.yml` is deliberately not involved.** Eleven jobs share its `on:` block, so the filter belongs in the runners, where it also covers a local `npm test`.
 
 ---