T-ENG01: test the scripts, and validate what they export

[emmanuelgjr]

The project shipped OSCAL and STIX for other tools to consume and never read either back. It turns out 13 of the 25 frameworks were emitting OSCAL catalogs that no OSCAL tool would load — 750 control ids that are not valid OSCAL tokens — and nothing had noticed, because nothing looked.

Test suites — 3 tests → 50

File	Covers
stats.test.mjs	invariants on the numbers every headline renders from: per-list sums, mapped ≤ registries, draft_only really has no authored row, disk matches a fresh computation
generate.test.mjs	determinism (generate twice → byte-identical), committed output matches a fresh run, entry integrity, and that DRAFT never leaks into a stored enum field
reports.test.mjs	the three report CLIs through their CLIs — CSV column integrity under real commas and quotes, filters that actually filter, summary agreeing with the array it ships, an unknown --framework failing loudly
ingest.test.mjs	the framework ingest path — mostly that it rejects: no controls, no name, no control id, missing file
exports.test.mjs	OSCAL and STIX validated against schema, plus dangling-reference, uniqueness and completeness checks

Export validation

Uses ajv against three schemas in data/schemas/. They are deliberate subsets, written and owned here — the filenames and data/schemas/README.md say so. Passing proves the export has not lost its shape; it does not prove OSCAL or STIX conformance. The README records why fetching or vendoring the upstream schemas costs more than it returns, and how to run the real validators when that is the question.

Two fixes the tests found

1. oscalToken() in compliance-report.js. OSCAL ids are NCName-like: no leading digit, no whitespace. Real control ids routinely are neither — Art. 5 (DORA), SR 1.1 (ISA 62443), 4.1 (ISO 42001), V8 Data Protection (ASVS). They were emitted verbatim. The original is now preserved in the control title and a source-control-id prop, and the same coercion runs in both the catalog and component-definition renderers so the two still join.

Non-token ids across all 25 frameworks: 750 → 0.

2. ingest-framework.mjs now rejects an empty control set. It validated cleanly before, so a registry with zero controls could land in data/frameworks/ and count as a framework covering nothing.

Not fixed here — issue #35

576 mapping rows still hold prose in control_id, because some Markdown files put the identifier and the requirement in swapped columns and parseControlTable() falls through to the wrong one. That is the root cause of the invalid ids. It is a parser fix touching 576 rows across 25 frameworks and deserves its own review.

exports.test.mjs pins the per-framework counts as a regression fence — SOC 2 168, EU AI Act 118, OWASP NHI 103, and so on — so the problem cannot spread, plus a second test that fails if a baseline entry outlives the framework it names, so the fence cannot rot into dead permission.

CI

New Unit tests job: npm ci --ignore-scripts, the suite, then git diff --exit-code. The suite re-runs the generator to prove determinism, so it has to leave the tree clean. npm run ci now runs the suite too, between build and reports. Path filters gain scripts/*.test.mjs and package.json.

Verification

• npm run test:scripts — 50 pass, 0 fail
• validate.js 0 errors / 312 passed · stats:check green · markdownlint 0 errors
• npm run compliance and npm run incidents both regenerate clean
• New dependency: ajv + ajv-formats, devDependencies only

🤖 Generated with Claude Code