feat(utils): slug the URL path into output filenames - #30
Merged
Conversation
buildFilename keyed output names on hostname alone, so every page of one host collided and the _NN tiebreaker carried no page identity. runBatch records results in completion order rather than input order, so _NN did not even correspond to a URL's position in the list — the number a page received varied between runs of the same command. urlSlug derives a filename-safe slug from the path. Query strings and fragments are dropped, so tracking parameters do not produce a different filename for the same page. A root path yields an empty slug, which leaves single-page runs and origin-scoped crux/crux-history byte-identical to before. Two details that are not obvious: Percent-encoding is decoded and Unicode letters are kept, rather than slugging with [a-z0-9]. `pathname` percent-encodes non-ASCII, so an ASCII-only rule emits the UTF-8 bytes as text — /es/zapatos-de-niño becomes es-zapatos-de-ni-c3-b1o, and a non-Latin path becomes hex carrying no page identity at all. The cap is 80 bytes, not 80 characters, and truncation keeps the path's tail. 80 CJK characters is 240 bytes, which would exceed the 255-byte filename limit on its own. Keeping the tail matters because URLs are hierarchical: the shared part is the prefix, so keeping the head would collapse exactly the deep-category pages most likely to be audited together. A 6-char hash of the full path is appended once truncated, which makes the name stable across runs where _NN is not. _NN is retained as the last-resort tiebreaker. urlSlug is exported from web-perf-cli/utils so a consumer can predict a path.
consumer.ts imported nothing from web-perf-cli/utils, so the published declarations for that subpath were unchecked and urlSlug shipped uncovered. Asserts the return types and that urlSlug is not nullable, so a consumer never has to narrow it.
The filename formats are written out in both README.md and CLAUDE.md and have drifted before, so both carry the [-<slug>] segment and the rules behind it: query and fragment dropped, percent-encoding decoded, Unicode letters kept, an 80-byte cap that keeps the path's tail with a hash. States plainly that a root path adds no segment, since the practical question for anyone with an existing results/ directory is whether their filenames changed — for single-page and origin-scoped runs they did not. Adds urlSlug to the library API table now that it is exported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12. Implements the spec settled in the issue, as amended in this comment.
buildFilenamekeyed output names on hostname alone, so every page of one host collided and the_NNtiebreaker carried no page identity. Worse,runBatchrecords results in completion order, so_NNdid not correspond to a URL's position in the list — the number a page received varied between runs of the same command.Behaviour change
Output filenames gain a
[-<slug>]segment derived from the URL path:A root path adds no segment, so single-page runs and origin-scoped
crux/crux-historyproduce byte-identical filenames to before.sitemapis passed an origin and is unaffected. Applies tolab,psi,crux,crux-historyandlinks, plus the.clean.json,.summary.jsonand-output-ai.txtnames that derive from the same function.Two decisions amended from the original spec
Both were found by running the specified algorithm rather than reasoning about it.
Unicode is preserved instead of slugging with
[a-z0-9].pathnamepercent-encodes non-ASCII, so an ASCII-only rule emits the UTF-8 bytes as text:/es/zapatos-de-niñobecamees-zapatos-de-ni-c3-b1o, and/日本語/ページbecamee6-97-a5-e6-9c-ac-...— hex carrying no page identity, which defeats the purpose of the change. The slug now decodes first and keeps\p{L}\p{N}.The cap is 80 bytes rather than 40 characters, and truncation keeps the tail. A character cap is unsafe once Unicode is allowed: 80 CJK characters is 240 bytes, which exceeds the 255-byte filename limit on its own. Keeping the tail matters because URLs are hierarchical — the shared part is the prefix, so a head-truncating cap collapsed exactly the sibling pages most likely to be audited together (
.../zapatillas-running-hombreand.../zapatillas-running-mujerboth becamees-productos-calzado-deportivo-zapatilla). The 40-char cap was also leaving 126 bytes of measured headroom unused.A 6-char hash is appended when the slug is truncated, reversing the spec's "no hash suffix" call. The spec's own fallback is
_NN, which it concedes is unstable across runs given completion-order batching; a hash of the full path is stable. Short paths — the overwhelming majority — never see one._NNis retained as the last-resort tiebreaker.New export
urlSlug(url)fromweb-perf-cli/utils, so a consumer can predict an output path. Added to the README API table, andtype-tests/consumer.tsnow imports from that subpath — it previously imported nothing fromutils, so those published declarations were unchecked.Verification
npm run lint,npm test(609 passed, +15),npm run generate-types,npm run check-types— all pass.Verified against the real CLI, not just unit tests. A three-URL
psirun on one host completed out of order (lcp, then/, thencls— the exact instability the issue describes) and produced:examples/crux-save.js(origin scope) still writescrux-web.dev-2026-09-04-101714-phone.json, confirming the backward-compatibility claim live.Known churn
lib/utils.test.jshad one assertion passing a URL with a path; the other twelve use bare origins and were unaffected.