feat(tree): add persisted commit metadata - #28064
Noah Encke (noencke) merged 23 commits into
Conversation
Implements the design in packages/dds/tree/docs/wip/persisted-commit-metadata.md. Applications can attach arbitrary JSON-serializable metadata to the commit a transaction produces, via a new `persistedMetadata` field on `RunTransactionParamsAlpha`. The metadata is replicated to peers, persisted in the summary, and read back through `TreeBranchCommitMetadata.persistedMetadata` while walking the branch history. The metadata lives on the commit itself - on `GraphCommit` in memory and inline on commits in the `EditManager` summary - so it shares the commit's lifetime and needs no separate index to populate, reconcile, or prune. `GraphCommit`'s property is declared required so that every site which rebuilds a commit from its parts is a compile error rather than a silent drop. Both the op and summary formats gain a v7, written only when minVersionForCollab is 3.0.0 or later, so a document only ever contains metadata when every client that can open it understands and preserves it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Hi! Thank you for opening this PR. Want me to review it? Based on the diff (24671 lines, 114 files), I've queued these reviewers:
How this works
|
- Drop persisted metadata for every trimmed commit, including the newest one which survives internally as the trunk base. Previously a client could keep reading metadata that had already left the document, either through the reachable trunk-base sentinel or through a TreeBranchCommitMetadata obtained before trimming. - Snapshot the metadata at the transaction boundary by round-tripping it through JSON. This gives the commit a private copy, so later mutation of the caller's object can no longer change an already-created commit, and it guarantees the value read back locally is exactly what peers and future summaries see rather than diverging on values with no JSON representation. Throws a UsageError for values that cannot be represented as a JSON object. - Validate that persisted metadata is an object rather than accepting any value, by replacing Type.Any() in JsonCompatibleReadOnlyObjectSchema. - Add golden op and summary format tests that lock the serialized 'persistedMetadata' key at v7 and assert its absence before v7, so a rename on both the encode and decode sides can no longer pass unnoticed. - Add tests for trimming through a retained metadata object, value snapshotting and normalization, and runTransactionAsync. - Document why the metadata field is declared on the shared commit schema for all format versions, and warn callers of SharedTreeBranch.apply which cases must propagate metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Pushed 1. Trimmed commits kept their metadata. Eviction poisons 2. The commit aliased the caller's object, and could disagree with what was persisted. The metadata is now snapshotted at the transaction boundary via a JSON round trip. This fixes two things: mutating the object after 3. Also added golden op/summary format tests that pin the serialized Two review points I deliberately did not act on, both documented in code so they are explicit choices rather than oversights:
Suite is now 15304 passing / 1 failing, the failure being the pre-existing |
… metadata Addresses PR review feedback. - Rename the feature from "persisted" to "custom" metadata: `customMetadata` on RunTransactionParamsAlpha, the op format, the summary format and GraphCommit, and `custom` on TreeBranchCommitMetadata (matching the existing NodeSchemaMetadata.custom / FieldSchemaMetadata.custom convention). - Merge the metadata of all nested transactions into the single commit they produce rather than using only the outermost. Conflicting properties resolve to the outermost transaction, and a nested transaction that is rolled back contributes nothing. - Align JsonCompatibleReadOnlyObjectSchema with the existing PersistedMetadataFormat used for schema metadata, rather than Type.Any(). - Simplify or remove doc comments per review, and remove a duplicated doc comment left on SharedTreeBranch.apply. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Pushed Nested transaction merge - implemented. All transactions in the stack now contribute to the single commit they produce, outermost winning on conflicts. A nested transaction that is rolled back contributes nothing, which fell out of the existing commit/abort handling and seemed clearly right. Four new tests cover it. JSON validation precedent - the closest existing case is persisted schema metadata (\PersistedMetadataFormat), which validates at the schema level only via \Type.Record(Type.String(), JsonCompatibleReadOnlySchema). I've aligned \JsonCompatibleReadOnlyObjectSchema\ to that exact shape; it had been \Type.Any(), weaker than the format it describes. I kept the snapshot step on top, since it fixes a real aliasing bug rather than just validating. NaN - normalizing to One thing worth your call on naming: \persistedMetadata\ is already an established name in this package for exactly this concept - app-supplied \JsonCompatibleReadOnlyObject\ that gets persisted - on \FieldPropsAlpha, \NodeSchemaOptionsAlpha\ and \SimpleNodeSchemaBaseAlpha. So the rename does trade consistency with those options bags for consistency with \metadata.custom. I've made the change as requested since \custom\ clearly fits the read property; flagging it only in case the collision with the schema convention changes your mind for the write side. Validation: 15306 passing / 1 failing (the pre-existing Windows path-separator failure). API reports regenerated - the diff is exactly the two renamed lines per report, no incidental churn. No snapshot files changed, since none of them carry metadata. |
There was a problem hiding this comment.
Copilot reviewed 86 out of 86 changed files in this pull request and generated 2 comments.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Remove three tests fully subsumed by others: two "Reading" cases covered by the commit-association test, and the nested-transaction precedence case covered by the three-level merge test (which already includes a conflicting property). - Add coverage for two untested behaviours: the commit produced by reverting an annotated commit must not inherit its metadata, and an empty metadata object must survive replication rather than being treated as absent. Both were verified to fail when the corresponding behaviour is broken. - Replace per-test SharedTree setup with the existing `getView` helper for the cases that need neither replication nor persistence, and add a small `createConnectedViews` helper for the cases that do. - Move the op rollback test out of the persistence group and the async transaction test out of the nested transaction group. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Drop an assertion implied by the deep-equal above it. - Assert on the encoded peer branch structure directly rather than counting occurrences of the key in the serialized string. - Share codec construction between the encode and round-trip cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Did a pass over the tests themselves against completeness / convention / minimality / uniqueness. Pushed Removed as redundant (3). Two Added (2), both verified non-vacuous by breaking the behaviour and confirming the test fails:
Minimality. Most tests needed neither replication nor persistence but were each standing up a Organisation. The async transaction test was sitting under Gaps I considered and deliberately left: metadata over a shared branch ( Suite: 15305 passing, 1 failing (the pre-existing Windows path-separator failure). |
…erts Two related changes to the custom commit metadata API. Metadata is now a tree mirroring the transaction nesting, matching how `LabelTree` relates to a change's labels. `TreeBranchCommitMetadata.custom` remains the flattened view (outermost transaction wins on conflict), and the new `customTree` exposes the structure. The persisted form uses abbreviated keys (`m` for metadata, `c` for children) and omits both when empty, so the common un-nested case costs 6 bytes over a bare object rather than 27. Note that op compression only engages for batches over 600 KiB, so it does not help here. Reverts may now be performed inside a transaction, provided the revert is the transaction's only change. This exists so that a revert commit can be given its own metadata rather than inheriting the reverted commit's, which matters for values like a timestamp. The previous blanket restriction was there because the inverse is computed against, and applied to, the branch the transaction forked from; requiring that the transaction have produced no commits makes those two branches identical, so the inverse is computed against exactly the state it would be outside a transaction. Further changes in the transaction are then rejected. The squashed commit still reports the revert's CommitKind, so it remains redoable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Joshua Smithrud (Josmithr)
left a comment
There was a problem hiding this comment.
API and docs changes look good. I left one sanity-check question about one of the API changes. I did not review the code changes in detail.
…3.0.0
The v7 EditManager and Message formats were gated on `FluidClientVersion.v3_0`.
That string cannot exist on a pre-3.0 branch: `OldestSupportedClientVersion` is
`${1 | 2}.${bigint}.${bigint}` there, and `getConfigForMinVersionForCollabIterable`
validates every entry in the table against `isLtePkgVersion`, so merely listing
"3.0.0" would throw when the codec is built on a 2.x client.
Gating at 2.117.0 preserves the option of carrying these formats on a 2.117.0
minor cut from the last pre-3.0 commit on main, for consumers that cannot yet
absorb the 3.0 breaking changes. It costs nothing on 3.x: 3.0.0 > 2.117.0, and
selectVersionRoundedDown compares with semver, so 3.x clients still select v7 for
any minVersionForCollab at or above 2.117.0 whether or not 2.117.0 ever ships.
Both release lines must agree on this string. A client whose table maps 2.117.0
to v6 reads metadata but silently strips it when encoding.
Renames the corresponding snapshot directories and files, which are named from
the FluidClientVersion key. Their contents are unchanged: 2.117.0 and 3.0.0 both
select v7, so the encodings are identical.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Make `Commit.customMetadata` required-with-undefined to match `GraphCommit`, which lets the three `mintCommit` call sites in `editManager` pass the commit through directly instead of re-destructuring it. - Let `decodeCustomMetadataTree` accept `undefined`, removing the repeated ternary at its three call sites. - Hoist the v7 metadata check in `makeV1toV4andV6CodecWithVersion` into a const shared by the schema and `encodeSharedBranch`. - Clarify the 2.117.0 `FluidClientVersion` remarks and drop an unnecessary comment in `messageCodecVSharedBranches`. - Document what omitting the metadata options means. - Remove the hand-rolled wire/summary format suites; a better pattern will be established in a follow-up. Give `newCommit` and the resubmit machine's test commits distinct metadata so the existing assertions cover preservation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42b443d7-0621-42a1-b087-f4e4765046af
Saying that omitting the metadata option means the commit carries no metadata restates what the optional type already conveys. Also trims the `makeFactory` test helper doc, whose `@param` repeated the default visible on the next line. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42b443d7-0621-42a1-b087-f4e4765046af
|
🔗 Found some broken links! 💔 Run a link check locally to find them. See Checking for Broken Links for more information. linkcheck output |
Bundle size comparisonBase commit: Pending — |
…17 (#28104) ## Description Cherry-picks the SharedTree history and persisted-commit-metadata work onto the `release/client/2.117` branch. This is step 4 of the 2.117 release plan (branch created from `release/client/2.116` + version bump); release notes, changelogs and assert tagging are handled separately by the release engineers. Four commits, applied in merge order (each with `git cherry-pick -x`, so the original SHA is recorded in the commit message): | Commit on `main` | PR | What it provides | |---|---|---| | `22e5b4ee` | [#27932](#27932) | Renames the alpha `TreeBranchAlpha` interface to `UntypedTreeViewAlpha` (old name kept as a deprecated alias, so it is additive). Carries no feature, but #28012 is written against the new name, so it is a prerequisite rather than an optional cleanup. | | `3b665a2a` | [#28036](#28036) | `retainHistory` now retains the trunk in *summaries*, not just in memory. Without it, retained history is discarded at the next summary and never reaches clients that load from it. No persisted format change; the `retainHistory: false` default is untouched. | | `920e9469` | [#28012](#28012) | `UntypedTreeViewAlpha.branchHistory` (`length`, `getHead()` → `TreeBranchCommitMetadata { revision, getParent() }`), plus `rewindTo(revision)` and `revertTo(revision, options?)`. | | `90337165` | [#28064](#28064) | Persisted commit metadata: `customMetadata` on `RunTransactionParamsAlpha` and the revert options, read back as `.custom` / `.customTree`. Stored inline on commits in the `EditManager` summary under `EditManagerFormatVersion.v7` / `MessageFormatVersion.v7`, written only when `minVersionForCollab` is at least `2.117.0`. | A fifth commit adapts the picks to this branch — details below. ### Why a minor rather than a patch #28064 changes a persisted format, and the format gate is a version string baked into SharedTree's source. The config map that selects a format version rejects patch versions, so the gate has to be an `X.Y.0`. It is `FluidClientVersion.v2_117 = "2.117.0"`. ### Forward compatibility with 3.0 All four commits are already on `main`, and `main`'s tip was #28064 when this was prepared, so the two lines could be compared directly. Everything that ships here is identical to 3.0: - The whole persisted-format surface — `src/codec/` and `src/shared-tree-core/` — is byte-identical between this branch and `main`, including the `v2_117` → v7 gate. A mixed 2.117/3.0 session cannot silently disagree about the format. - Every alpha symbol added here (`TreeBranchHistory`, `TreeBranchCommitMetadata`, `branchHistory`, `rewindTo`, `revertTo`, `customMetadata`, `customTree`, `retainHistory`, `UntypedTreeViewAlpha`, `RunTransactionParamsAlpha`, `RevertOptionsAlpha`, `RevertToOptionsAlpha`) is identical on both lines, at the same API tier, with the same deprecation state. - `minVersionForCollab: '2.117.0'` stays valid on 3.0: `OldestSupportedClientVersion` *widens* from `` `${1 | 2}.${bigint}.${bigint}` `` to `` `${1 | 2 | 3}....` ``, so consumers who adopt 2.117 need no change in this area when they later move to 3.0. ## Adapting the picks to this branch `main` is on TypeScript 6 and past the 3.0 bump; this branch is TypeScript 5.4 and pre-3.0. Conflicts were all of one shape — an import list where `main` has accumulated names from commits that are *not* being backported — and were resolved by keeping only what this branch actually uses: - `src/index.ts` — kept `asTreeViewAlpha` (removed on `main` by 3.0 work) alongside the incoming `TreeBranchCommitMetadata` / `TreeBranchHistory`. - `src/shared-tree/treeCheckout.ts` — kept `StableId` and `findAncestor`, which #28012 uses; dropped `tagCodeArtifacts` (used on `main` only by schema-change telemetry, #27996) and `getDeltaChangeProfile` (introduced by #27989). Neither is backported here. - `src/test/shared-tree/schematizeTree.spec.ts` — kept `TreeBranchHistory`; dropped `UntypedTreeViewAlpha`, which was added to that import by the TypeScript 6 upgrade (#28052) rather than by #28012. One genuine TypeScript-version difference, in the fifth commit: - `customCommitMetadata.spec.ts` used `.filter((m) => m !== undefined)` and then read `.tag`, relying on **inferred type predicates**, a TypeScript 5.5 feature. On 5.4 the element type stays `T | undefined` and the compiler reports TS18048. The predicate is now spelled explicitly. This was the only TypeScript error in the entire build. The two conflicted `*.api.md` files are generated artifacts, so rather than hand-resolving them they were regenerated by a full clean build from the repo root. That correctly drops `asAlpha`, `codePointCount` and `utf16LengthForCodePoints` — whose exports come from #28011 and #28004, not backported here — and narrows `OldestSupportedServiceClientVersion` to `` `2.${bigint}.0` ``. ## Validation - Full `pnpm clean && pnpm build` from the repo root: succeeded, and left the working tree clean apart from the intended API report updates. No unexpected API report drift. - `@fluidframework/tree` test suite: **15301 passing, 488 pending, 1 failing**. The one failure is a pre-existing Windows-only issue in `snapshotCompatibilityChecker.spec.ts`, which builds the directory under test with `path.join(...)` but hardcodes forward slashes in the expected error string. It is untouched by these commits and passes on Linux CI. ## Reviewer Guidance The review process is outlined in [the pull request guidelines](../docs/content/Contributing/PR-Guidelines.md#guidelines). - The first four commits are unmodified cherry-picks; review effort is best spent on the fifth (`fix(tree): adapt the backported history work to the 2.117 line`) and on the conflict resolutions described above, since those are the only places this branch diverges from what was reviewed on `main`. - **Assert tagging is deliberately not included here.** The three new asserts are still untagged string literals, so `flub release prepare client` will report them. Running `flub generate assertTags` on a release branch allocates short codes from the branch's own high-water mark, which can disagree with `main` — on `main` today it would reassign `0xd33`, a code that shipped in 2.116 as `"compatibilityMode must be defined"`, to a tree assert. Happy to follow whatever sequencing the release engineers prefer. --------- Co-authored-by: jzaffiro <110866475+jzaffiro@users.noreply.github.com> Co-authored-by: yann-achard-MS <97201204+yann-achard-MS@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3af96e03-702e-49da-adaa-bc99ca75ee27 Copilot-Session: eedf6254-a76e-4efe-af90-1f7b43da8665 Copilot-Session: a037b96b-478f-4a65-9555-d7a970e7855e Copilot-Session: 42b443d7-0621-42a1-b087-f4e4765046af Copilot-Session: 797fba3e-0e9a-48db-86c6-530aa2837434
This reverts commit 9033716. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db28c8d7-8cb9-4efe-bb78-a1af7d1cdc0b
This reverts commit 9033716. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db28c8d7-8cb9-4efe-bb78-a1af7d1cdc0b
Description
Applications can now attach arbitrary, JSON-serializable metadata to the commit that a transaction produces, replicate it to collaborating clients, and persist it in the document.
Write it via the new
customMetadatafield onRunTransactionParamsAlpha:Read it back while walking the branch's history, via the new
customproperty onTreeBranchCommitMetadata:Because a commit may be produced by nested transactions, each of which may supply metadata,
customis the flattened combination of them all (outermost wins on conflicting keys). The structural view is available ascommit.customTree, aCustomMetadataTreemirroring the transaction nesting — the same relationshiplabels.treehas to a change's label set.The metadata lives directly on the commit — on
GraphCommit.customMetadatain memory and inline on the commits in theEditManagersummary. That is what makes its lifetime automatically match the commit's: once the commit is trimmed from the trunk, the metadata goes with it, so there is no separate index to populate, reconcile, or prune.Both the op format and the summary format gain a
v7(MessageFormatVersion.v7andEditManagerFormatVersion.v7), written only whenminVersionForCollabis2.117.0or later (gated byFluidClientVersion.v2_117). That floor is a declaration rather than an enforcement mechanism: a client too old for v7 fails cleanly with an unsupported-version error when it reaches v7 data, so adopting this requires deploying v7-capable readers everywhere before raising the floor. The changeset spells out the rollout sequence.A revert may now be performed inside a transaction provided it is that transaction's only change, which lets the revert be given its own metadata. Attempting any other change in such a transaction throws an error.
Reviewer Guidance
The review process is outlined on this wiki page.
The
GraphCommit.customMetadataproperty is required, not optionalThis is the load-bearing decision. Two places rebuild a commit from its parts rather than spreading it (
mintCommitandrebaseBranch), and an optional property would let both silently drop the metadata. Declaring it required turns each into a compile error, so the type system enumerates every site that has to make a decision.Where a rebuilt commit is the same logical commit as its source, the property is propagated.
undefinedis used only where a genuinely new commit is minted — the inverse commit produced by reverting, rollback commits, the synthetic root/trunk-base commits, and edits from the editor that a transaction has not yet annotated.The metadata tree mirrors transaction nesting
Each transaction in a nested stack contributes a node to a
CustomMetadataTree. The root node is the outermost transaction, and nested transactions add child nodes. Aborted nested transactions remove their node again, so they never contribute. The single commit produced by the stack carries the tree, from which the flattenedcustomview is derived.Compatibility
The
v7codecs are only selected whenminVersionForCollab >= 2.117.0, so nothing changes for existing clients by default. Evidence that this holds: regenerating the full snapshot corpus added newv2_117directories and modified no existing snapshot file, meaning the v3/v4/v6 output is byte-for-byte unchanged.If an application supplies metadata while configured below
2.117.0, the value is kept in memory for the local session but is neither replicated nor persisted. There is a test for this.The version-specific typebox schemas exclude the
customMetadatafield for pre-v7 formats, so no pre-v7 encoder can write it. The two formats then differ in how strictly they validate, and deliberately so: the summary schemas were alreadyadditionalProperties: false, so a payload falsely claiming to be pre-v7 while carrying the field is rejected there. The opMessageschema is left permissive, as it has always been — tightening the op envelope would risk rejecting ops over envelope properties unrelated to this feature, and belongs in its own PR — so such a payload is tolerated there instead. Both behaviors are covered by tests.Breaking Changes
None. All new API surface is
@alphaand additive.