fix(import): homepage docs-route probe trailing slash + SPA catch-all - #39
Conversation
…t-404 detection Bare homepages whose /docs route 301s to a slash-stripped path were rejected as out-of-scope aliases, and SPA catch-all sites adopted the first well-known route even when it served the marketing shell. Scope comparison now strips trailing slashes and enforces segment boundaries; a soft-404 baseline title fetched from a garbage path rejects path candidates indistinguishable from a nonexistent page.
WalkthroughDocumentation route discovery now detects SPA soft-404 responses by comparing candidate page titles with a guaranteed-invalid path. Route checks follow redirects, enforce candidate scope, and return structured resolution results. Candidate paths that match the site’s catch-all response are skipped. The exported test helpers now include Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/import.js`:
- Around line 3578-3581: Update the path-candidate matching logic around strip
and the finalPath/candidatePath comparison to validate final.hostname first.
Accept only the candidate hostname or its permitted apex-to-www alias, then
retain the existing pathname match; reject unrelated hosts even when their paths
match, and add a regression test covering an unrelated hostname using /docs.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 88175e0d-2da5-4412-9e29-811cb7446d1c
📒 Files selected for processing (2)
src/commands/import.jssrc/commands/import.test.js
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
readmeio/ai(manual)readmeio/gitto(manual)readmeio/markdown(manual)readmeio/readme(manual)
minhthanhdang
changed the title
minhthanhdang
changed the title
minhthanhdang
changed the title
| /** | ||
| * Fetch a guaranteed-nonexistent path and capture its page title. On SPA | ||
| * catch-all sites every unknown path returns 200 with the same shell page; | ||
| * that title becomes the soft-404 baseline. Returns null when the site 404s | ||
| * properly (no baseline needed) or on any fetch failure. | ||
| */ | ||
| async function fetchSoft404Title(origin) { |
There was a problem hiding this comment.
good thinking to do this, neat trick.
2 participants
Context
Importing a bare homepage (e.g.
https://malveon.com,https://turbopaybd.com) produced a marketing-site skeleton instead of the doc site. Two root causes inresolveDocsBaseUrl's existence probe:https://site/docs/; many sites 301 to/docs(slash stripped).inCandidateScopecompared withstartsWith("/docs/"), so the correct route was rejected as an out-of-scope alias./docs/) was adopted over the real docs route ranked later (/developers/).Changes
inCandidateScopenow strips trailing slashes and compares on segment boundaries:/docsmatches the/docs/candidate, while/and/docsomethingstay rejected. The apex→www host hop for path candidates is preserved.<title>. A path candidate whose title equals that baseline is a catch-all response and is skipped with an explicit log line. Sites that 404 properly (or have no title) produce no baseline, so healthy sites are never falsely rejected.docsRouteResolvesreturns{ exists, soft404 }; the body is only read for path candidates when a baseline exists.Verification
npm test: 102/102 pass, including new coverage for slash-redirect adoption, redirect-to-homepage rejection, catch-all skip, all-candidates-fail, andinCandidateScopeboundary cases.step:skeleton https://malveon.com: adoptshttps://www.malveon.com/docs/and produces the full ~44-page doc skeleton (was: 9-page marketing skeleton).step:skeleton https://turbopaybd.com:/docs/and/doc/are logged asindistinguishable from a nonexistent path (SPA catch-all) — skipping;/developers/is adopted.Pairs with https://github.com/readmeio/ai-cli-runner/pull/108, which handles the follow-on single-stub fallback; a sandbox snapshot rebuild is required to ship this CLI change.
🤖 Generated with Claude Code