ENG-2219 Add an asset copy stage to Roam node materialization - #1401
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
cd7452d to
9c38544
Compare
9c38544 to
8f65942
Compare
8f65942 to
4620a4f
Compare
4620a4f to
45d86a6
Compare
45d86a6 to
06a464c
Compare
PR size/scope checkThis PR is over our review-size guideline.
Please split this into smaller PRs unless there is a clear reason the changes need to land together. If keeping it as one PR, please add a brief justification covering:
|
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| const resolvable = new Set( | ||
| collectAssetTokens(markdown).flatMap(lookupCandidates), | ||
| ); | ||
| const referenced = references.filter(({ filepath }) => | ||
| resolvable.has(filepath), | ||
| ); |
There was a problem hiding this comment.
🟡 Encoded aliases create unused uploads
When rows exist for both an encoded token and its decoded spelling, resolvable selects both. rewriteAssetLinks uses only the raw match, leaving the other upload permanently unused.
Learn more
The resolvable Set flattens every lookup candidate, so it cannot preserve rewriteAssetLinks' ordered choice. For markdown containing my%20folder/d.png and reference rows named both my%20folder/d.png and my folder/d.png, importNodeAssets mirrors both rows. rewriteAssetLinks then chooses the raw candidate first and never references the decoded row's upload. Select at most the first matching reference row per parsed token using the same candidate priority as rewriteAssetLinks, while still supporting multiple distinct tokens and deduplicating repeated references.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
The mechanism is right, but the case a real note produces costs nothing.
Two rows in one spelling family do both pass the filter, so mirrorAssetToRoamStorage is called twice. What that costs depends on whether the rows share a content hash.
Same hash is what one file referenced under two spellings gives you. The second call hits the asset registry and comes back reused, so there is one upload and one blob, and handledHashes counts it once. Covered by "counts one upload when two locators name identical bytes" in importNodeAssets.test.ts.
Different hashes would mean two genuinely different files whose paths differ only by percent-encoding, such as a file literally named my%20folder/h.png alongside my folder/h.png. That does leave one upload unreferenced, and it is not a data state we intend to design for.
No change.
06a464c to
1043e32
Compare
1043e32 to
61bedb2
Compare
61bedb2 to
4aa1ebe
Compare
c15631b to
ca53361
Compare
ca53361 to
6fba48b
Compare
6fba48b to
2e5ab33
Compare
2e5ab33 to
5ed5210
Compare
5ed5210 to
ed97690
Compare
ed97690 to
86fc5fc
Compare
86fc5fc to
a21fc03
Compare
a21fc03 to
157810b
Compare
157810b to
e1cd9c4
Compare
e1cd9c4 to
d895edd
Compare
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
|
|
||
| // Carried on success only. A node that failed to import has a stage of its own to | ||
| // report, and the assets it did or did not copy are not what the reader needs. | ||
| return result.success ? { ...result, assets: report } : result; |
There was a problem hiding this comment.
🟡 Failed asset copies never retry
When importNodeAssets reports a transient failure, materializeSharedNode still succeeds and records the source as current. isImportUpToDate skips later normal imports while that timestamp remains unchanged. The asset link stays broken until a forced refresh or source edit.
Learn more
Asset failures live in report.failed, so they do not make the materialization fail. The create and update paths therefore write the current source timestamp before this success result returns. On the next normal import, the freshness guard exits before importNodeAssets runs. The same guard also bypasses the new stage for unchanged pages imported before this feature shipped.
Example: An Obsidian node imports while one image download times out. Its vault link remains unresolved, but the node records the current source timestamp. Retrying the group import after connectivity recovers reports the node as skipped and never retries the image.
Recommended fix: Persist asset-materialization state or a stage version with the imported source identity. Only take the freshness shortcut when both content and assets are current; leave transient asset failures retryable while relying on the asset registry to reuse successful copies.
Was this helpful? React with 👍 or 👎 to provide feedback.
Reviewer brief
This finally ties in the components together, calling
importedNodeAssetsinmaterializeSharedNode.ts.Title collision detection is hoisted above asset creation.
Verification
Some new unit tests, but here is where all the behaviour test videos are landing.
Loom video
Code walkthrough:
https://www.loom.com/share/713b68611c90472b9306f02ad39bc6b5
Show an Obsidian node imported in Roam and Obsidian
https://www.loom.com/share/dfbc2b34f33d46d0827f4b471f6c5183
Show a Roam node imported in Roam and Obsidian
https://www.loom.com/share/40d6571412ac4c92b14a6427e058dc9e
Some more loom videos are needed to show that asset failures do not block the page sharing, those will be added, but the main path is shown to work.
Scope check
$scope-checkagainst ENG-2219 and the final diff.Done When: One addition. The title-collision check was extracted intotitleCollisionFailureand is now also made inmaterializeSharedNodebefore the asset stage runs, in addition to the checks the two import paths still make.Done Whenrequires no ordering guarantee between the collision check and the copy.roamAlphaAPI.file.deleteexists and takes a URL, but every user of a graph shares one asset registry keyed by content hash, so deleting a blob this import uploaded can strand another user's import that already resolved to it. Roam also exposes no way to list a graph's files, so an orphan cannot be found afterwards. The addition is ordering only: the failure message and the reported stage are unchanged.Local delegated full review
$dg-delegated-full-reviewwhen no other full-review workflow is available.https://linear.app/discourse-graphs/issue/ENG-2219/add-an-asset-copy-stage-to-roam-node-materialization