feat(context): bound what a tool result contributes to a request - #2203
Open
mattzcarey wants to merge 32 commits into
Open
feat(context): bound what a tool result contributes to a request#2203mattzcarey wants to merge 32 commits into
mattzcarey wants to merge 32 commits into
Conversation
…to-a-capability # Conflicts: # examples/next/README.md # pnpm-lock.yaml
… context Sessions stored large content by truncating it and budgeted hydration by stored bytes, so a pointer row costing 8 MiB of memory was charged the ~100 bytes it occupies on disk. It also owned prompt assembly, which is not conversation storage. Storage. Every table is now WITHOUT ROWID with a composite key and no secondary index, ordered by a per-session `seq`, so a text append bills one row instead of two. The attachment reference table is (session_id, message_id, hash) and nothing else. An unchanged update writes nothing at all: no row, no FTS churn, no reference rewrite, no event. Offload. Media leaves the row at a size threshold wherever it appears, including `data:` URLs nested in tool output. Everything else, prose included, is offloaded largest-first only when the row cannot hold it, and a row that still does not fit raises SessionMessageTooLargeError. Nothing is truncated, and offloaded content reconstructs byte for byte. The aged-row maintenance pass applies that same policy, so a drained legacy row ends up exactly as if it had been written today. Memory. getRecentHistory charges each row its stored bytes plus, when reconstructing inline, the attachment bytes it re-inflates. That is the difference between a budget that bounds disk and one that bounds the isolate. Think and AIChatAgent both default to 32 MiB. Context. Blocks, frozen prompts, and the skill and search providers move to `agents/context`. Think declares them through a new configureContext() hook and reaches them through `this.context`; configureSession() keeps compaction and search. The Session handle stores messages and knows nothing about prompts. Hosts. AIChatAgent no longer loads the transcript in its constructor: the legacy lift and one bounded hydration run at start, the live array mirrors the change feed, and get-messages streams. Think reads pointers on its per-tool-result scan and no longer reads Sessions tables with raw SQL. Deleted: the synchronous storage aperture, the lossy eviction mode, sanitizeToolPairs, the token-counter plumbing, the per-field option thunks, and a duplicate copy of the sanitize helpers. Measured on a deployed worker with a real R2 bucket (examples/next/sessions-slam, 33 scenarios): one billed row per append, 1.6 MB and 5 MB text parts and a 3 MB tool output offloaded rather than cut, inline hydration stopping at 31.11 MiB against a 32 MiB budget, and 61.79 MiB of history streamed out of a 128 MiB isolate. Numbers are recorded in design/sessions.md. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
agent-think lives outside packages/ and examples/, so it was missed when the context system moved out of the Session handle. Its identity block is now declared through configureContext() and refreshed through this.context. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Renaming each lifted table to `*__lifted_v1` left every upgraded object storing its conversation history twice. A Durable Object tops out at 10 GB and gets uncomfortable well before that, so a 5 GB history plus its copy has nowhere to go. The copy already needs that space transiently, which is exactly why it must not be kept. Each source is now verified against its destination row by row, comparing the payload rather than just the key, and dropped only when every row arrived intact. A table that fails verification is left in place with a `session:migration:incomplete` event, so a partial lift keeps the only copy of its rows instead of destroying it. `assistant_sessions` and `assistant_fts` carry nothing the new schema needs, the registry being derived and the index rebuilt, so they are dropped outright. Think lifts `assistant_config` into its own table and now drops it too. AIChatAgent drops `cf_ai_chat_agent_messages` once every readable row has a copy, and its lift no longer reads the whole table into the isolate: order is read as ids alone, then bodies are fetched in windows bounded by rows and bytes, so a large transcript never lands in memory at once. Verified against real deployed storage by seeding a Think agent and an AIChatAgent on the pre-Sessions SDK and redeploying the identical worker built against this branch over the same objects. Think carried 5 rows and 67,184 bytes across with its branch topology and ids intact, AIChatAgent 4 rows and 53,765 bytes including a 53 KB inline image, both byte-identical, with no legacy or tombstone table left behind. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
…oviders Storage classified payloads by content type: media left the row at a 32 KiB threshold, everything else only to keep the row under its budget. Reading a PDF through pi settles that this is backwards. A document arrives as plain tool-output text with no media type, so the rule optimised the small case and ignored the large one. Deduplication, the other argument for it, does not depend on type either. Extraction into the attachment tables does not make the database smaller: chunk rows live in the same Durable Object, inside the same 10 GB. Only R2 reclaims space. And billing counts rows written, not bytes, so rewriting a 500 KB row costs the same single row as a tiny one while extracting it costs four. So there is now one rule and no content types in it. A payload is extracted when a bucket is configured and it reaches `r2ThresholdBytes`, which is the only extraction that reclaims anything, or when the row cannot otherwise hold it, largest first, into chunks. `inlineThresholdBytes` is gone and the R2 threshold is the single number. The maintenance pass returns immediately without a bucket, because inline is then the correct resting place. Separately, the skill-provider path is deleted: `R2SkillProvider`, `isSkillProvider`, the load and unload skill state on `ContextBlocks`, the `load_context` and `unload_context` tools, and the helpers that replayed that state from the transcript. Nothing ever registered a provider with a `load()` method. Think shipped its own Agent Skills through `agents/skills` before this path had a user and registers a plain readonly catalog block, so the tools never appeared and the history scan always returned immediately. A dead prompt line telling the model to use context-loading tools went with it. `agents/context` drops from 1,387 lines to about 880. Think's media eviction is untouched. It decides what the model sees, which is a different question from where bytes live. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Sessions is a message store, not a file store. A message can reference a file without being one, and hosts that handle files already have somewhere to put them: Think's Workspace spills to R2 at 1,500,000 bytes, the same threshold the session tier used, so the two were doing one job twice. With R2 gone there is no reason to extract a payload eagerly at all, because chunk rows live in the same Durable Object and never reclaim a byte. The rule is now one sentence with no configuration: a payload stays in its message row until the row cannot hold it, and then the largest payloads are chunked out until it fits. That takes the aged-row maintenance pass with it. Its only remaining job was draining rows into R2, so the pass, its scheduler, its backlog chaining, the `offload_candidate_bytes` column and the four core methods that stamped and read it are all gone. Think never used it: it disables the pass whenever its own media eviction is on and drives that from a truncated hydration read. Also removed: the R2 bucket port, key construction and cleanup, the declared-versus-unknown-length streaming split and `FixedLengthStream`, the `backend` and `r2_key` blob columns, and the in-memory bucket fakes three packages carried to observe a tier that no longer exists. `SessionsAttachmentOptions` goes from seven fields to three. Think's media eviction is untouched in behavior. It decides what the model sees, which is a different question from where bytes live, and it is what moves bytes to the Workspace. The sessions-slam example is deleted along with the measured table it fed, so the docs no longer quote numbers nothing can reproduce. Found in passing: ai-chat's test worker was inserting into a column renamed some time ago, which surfaced as seven swallowed unhandled rejections in a passing run. Adds design/context.md, recording that prompt assembly and history shaping belong in agents/context while retention stays with the host that owns the file store, and that shaping tool output at the boundary is the missing third piece (#2201). Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Media declared with a non-text media type and carried inline is stored
separately, addressed by SHA-256, and inlined again on read. The message
keeps a pointer and its mediaType, so a round trip is exact and a row
stays small however large its payloads are. Read with
{ attachments: "pointer" } to see references instead.
The rule is typed rather than sized. An image is extracted at any size
and text is never extracted at any size, so a message's stored shape
never depends on how large an image happened to be. Row chunking stays
as the independent size backstop for prose: the two never interact,
because media leaves before the row is measured.
This is not the layer that was removed. That one extracted only when a
row was over budget, which made it a rescue mechanism competing with
chunking and one that could fail with nothing to extract. 775 lines
replace 1,477.
Payload lifetime is derived from reference rows, taken from the stored
message rather than from what a given write extracted, so a pointer-mode
read written back keeps its payload alive. SessionRowStat.bytes charges
each message for what it points at, at inlined size, so a byte budget
still bounds real hydrated memory.
Cost, measured: a 200 KB image bills four rows and a 2 MiB image five,
against one when inlined. Text messages are unchanged at one row.
Also adds agents/context intake shaping. shapeMessage/shapeHistory cap
oversized tool results with a continuation hint and drop host-named
duplicate fields, on the read path so storage stays lossless. The limits
are a function of one message, so a shaped prefix stays byte-identical
across turns and prompt caching holds — which is why this landed here
and sliding history truncation stayed with the hosts.
Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Pi resizes images on the way into context. We deliberately do not, and this records why rather than leaving it as an open question: downscaling re-encodes a user's own bytes, which is a lossy transform of content nobody asked us to change, and it is the one kind of shaping a host cannot undo afterwards. Text caps and duplicate-field dropping stay. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Intake shaping is a separate concern from keeping attachments out of the message, and reviewing them together obscures both. It lands on a branch stacked on this one; nothing here depends on it. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
shapeMessage and shapeHistory apply a byte cap, a line cap, and a set of host-named fields to drop, to the tool parts of a message. Truncation always carries a continuation hint naming the offset to resume from, so an aggressive cap is a detour rather than a dead end — which is what lets pi cap roughly seventy times harder than Think does today. It runs on the READ path. Capping is lossy, so storage keeps the full result and the limit stays a policy that can change without having destroyed anything. That is the same line drawn everywhere else here: sessions is lossless, context shapes. Only tool parts are touched. A long assistant answer passes through, and so does an image — downscaling re-encodes a user's own bytes and buys less than it looks like, since providers scale images before tokenizing. dropFields exists because of a measurement: pi persists a raw provider payload beside the content it renders, and that duplicate alone was 2.6 MB and 1.65 MB in two of the three real messages that crossed the row budget. The module names no fields itself; only a host knows which of its own are duplicates. Nothing calls this yet — whether a host wants a cap, and at what size, is the host's decision to make. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
🦋 Changeset detectedLatest commit: 4931795 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 2 potential issues.
3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| depth: number | ||
| ): unknown { | ||
| if (depth > 8) return value; | ||
| if (typeof value === "string") return capText(value, limits); |
Contributor
There was a problem hiding this comment.
🔴 Multi-string results bypass intake limits
When one result contains multiple strings, shapeValue gives each string the full byte and line budgets. The request can remain arbitrarily large.
Prompt for agents
The limits documented by IntakeLimits and shapeMessage are per tool result, but shapeValue invokes capText independently for every nested string. Track shared remaining byte and line budgets while traversing each output/result value, preserving deterministic traversal and copy-on-write behavior. The continuation metadata must describe the actual cut point, and tests must cover objects and arrays containing multiple individually under-limit strings whose combined content exceeds each limit.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+117
to
+120
| let end = head.length; | ||
| while (end > 0 && byteLength(head.slice(0, end)) > maxBytes) end--; | ||
| const code = head.charCodeAt(end - 1); | ||
| if (code >= 0xd800 && code <= 0xdbff) end--; |
Contributor
There was a problem hiding this comment.
🔴 Large results exhaust Worker CPU
For oversized text, capText re-encodes the shrinking prefix after removing every UTF-16 unit. Multi-megabyte results can exhaust Worker CPU.
Prompt for agents
capText currently finds the byte boundary by decrementing a UTF-16 index and encoding the full prefix on every iteration, making truncation quadratic. Replace this with a linear or logarithmic boundary search, such as binary search over UTF-16 indices with surrogate-boundary adjustment or a single UTF-8 encoding followed by safe decoding. Preserve the maxBytes contract, valid surrogate pairs, exact byte-based nextOffset, and deterministic output. Add a multi-megabyte regression test or focused performance assertion.
Was this helpful? React with 👍 or 👎 to provide feedback.
…to-a-capability # Conflicts: # design/rfc-think-multi-session.md
…ions losing rows Three review findings, all real. The hydration budget was not a bound. `getRecentHistory` admitted rows whenever fewer than `minRecentMessages` had been taken, whatever they weighed, so a window of media-heavy messages hydrated far past the limit that was supposed to cap it. A floor that ignores size is not a floor under a budget, it is a hole in one. The parameter is gone from core, handle, Think and AIChat; the budget is a hard ceiling that always returns at least the newest message. Think's window can therefore be shorter than MODEL_RECENT_WINDOW when messages are unusually large. That is the intended trade: with a 32 MB budget it only ever binds on media, and 32 MB of text is far past what any model could read anyway. AIChat's legacy lift could delete history. It dropped the source table when `imported + skipped === order.length` — but a skipped row is one that could NOT be parsed or imported, so accounting for it and migrating it are not the same thing, and any malformed row was destroyed. It now drops only when every row actually landed, and says what it kept. Sessions stamped its schema version even when `migrateLegacy` reported an incomplete copy, so the lift never retried and the rows it left behind stayed unreachable. `migrateLegacy` now reports completeness and the version is stamped only on success. Both lifts are idempotent, so retrying costs reads and nothing else. Also resolves the conflict with main in rfc-think-multi-session.md, taking main's text, which records the same supersession plus the replacement RFC. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Contributor
🔴 agents import sizesMeasured 287 runtime imports as minified bundles. The primary size is gzip; raw minified size is included for diagnosis. An existing import growing by more than 10% is marked red. This report is informational.
Compared Changed imports (10)
All 287 current runtime imports
Reported by agent-think[bot]. |
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
`pathRowStats` charges a message for the payloads it points at, because that is what a read materializes. The incremental stats cache did not: it added the serialized POINTER json, so `stats().totalContentBytes` depended on whether the cache happened to be warm. Invalidate instead of tracking. Replicating the base64 charge in append and update would put the formula in three places and let them drift, which is how the two disagreed in the first place. A media write now drops the cache and the next `stats()` derives it — one recursive CTE read against writes that cost ~1000x more. Text writes, the hot path, keep the incremental path untouched. Narrow in practice: Think and pi read `totalContentBytes` from `getRecentHistory`, which derives from `pathRowStats` on every call and never from this cache, so no shipped behavior was wrong. But an API that returns a different number depending on cache warmth is a trap, and the regression test added here is the first thing that would have hit it. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Two findings from auditing Think against the Sessions API. `SessionStats` loses `totalContentBytes` and `pathLength`. Nothing read either — Think and pi take their byte total from `getRecentHistory`, which derives it from `pathRowStats` on every call and never from this cache. Both fields also measured the ACTIVE BRANCH only, excluding other branches, other sessions in the object, and the attachment tables, so as a "how big is this session" signal they answered a different question than the one anyone would ask them. A real size signal against the 10 GB ceiling has to sum the tables and deserves its own function. What remains is the token estimate that gates auto-compaction, which is the only field with a reader and the reason the cache exists at all. That also removes the cache invalidation added a commit ago: it existed solely to keep `totalContentBytes` honest once row stats began charging attachment bytes. Notably the field that diverged was the unused one — `tokenEstimate` is stamped from the message BEFORE extraction, so it always counted the payload and the compaction trigger was never wrong. `appendMessage` now returns the same inlined message whether it inserted or found a duplicate. It previously returned `getMessageRaw` on the duplicate and not-inserted paths, so `AppendResult.message` and the change feed carried `attachment:sha256:` pointers on some appends and inline content on others — one call, two shapes, depending on whether the row already existed. Think compensated for that with `_messageForCache`, which serialized every incoming message and substring-searched it for the pointer prefix before deciding whether to re-read, on the streaming hot path. Fixing the contract deletes the helper and its three call sites. AIChat's similarly named helper stays: it does v4 to v5 transformation, which is genuinely its own concern. The remaining duplication — both hosts reimplementing the change-feed cache mirror — is filed as #2205 rather than attempted here. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Finishing a fix I got half right. The previous commit made `appendMessage`'s duplicate paths inline but left the inserted path returning `prepared.message` — the caller's own object. So a caller that reads with `attachments: "pointer"` and writes the result back got pointer form on the insert and inline form on the retry: the same inconsistency as before, mirrored. It also invalidated the assumption the previous commit relied on to delete Think's `_messageForCache`. That helper existed because the feed could carry pointers; removing it was only safe if Sessions guarantees it never does. For pointer-form writes, it did not, and the pointers would have reached Think's live cache and then a model request. `appendMessage` and `updateMessage` now pass what they return and emit through `core.inlineMessage()`, so the guarantee holds for every write regardless of what the caller supplied. It is a no-op by reference when there are no pointers, so the ordinary write pays a walk and nothing else. The invariant now has one choke point instead of being maintained per branch, which is what went wrong twice here: each fix made another call site consistent rather than stating the contract in one place. Both tests fail without the change: insert-then-duplicate returning identical shapes with nothing pointer-shaped reaching a subscriber, and the same for updates. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
…to-a-capability # Conflicts: # packages/ai-chat/src/index.ts # packages/ai-chat/src/tests/worker.ts
`updateMessage` resolved attachments before checking the outcome, so a write against a row that no longer exists materialized every referenced payload and then returned null — megabytes loaded to be discarded, on a write guaranteed to fail. That path is also the one place a pointer legitimately cannot resolve: if the row is gone its payloads may have been collected with it. Bailing on `missing` before inlining settles both. Also drops `MAX_BOUND_PARAMS` and `buildInClauseStrings`, which arrived with main for the row-size-limit path this branch deletes. Claude-Session: https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4
Base automatically changed from
feat/move-sessions-into-a-capability
to
main
September 3, 2026 14:31
Contributor
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Comment on lines
+143
to
+145
| if (depth > 8) return value; | ||
| if (typeof value === "string") return capText(value, limits); | ||
| if (Array.isArray(value)) { |
Contributor
There was a problem hiding this comment.
🔴 Large tool images become invalid
When a tool returns an inline image above the limit, shapeValue truncates its encoded payload. The model receives a corrupted image.
Prompt for agents
Update packages/agents/src/context/intake.ts so shapeMessage recognizes inline media records before recursively shaping their fields. Data URLs and declared non-text base64 payloads must remain byte-for-byte unchanged, including media nested inside structured tool output. Reuse or align with the media detection rules in packages/agents/src/sessions/attachment-ingest.ts, and add tests for large url and data payloads.
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2196. Implements the read-path half of #2201.
What
shapeMessageandshapeHistorybound what a stored tool result contributes to a model request:dropFields, a set of host-named fields to stripWhy the read path
Capping is lossy — whatever is cut, the model cannot recover it. So storage keeps the full result and the cap applies to the copy being assembled into a request. A limit can then change next month without having destroyed the bytes in the meantime.
That is the same line drawn everywhere else in this design: sessions is lossless, context shapes. It is also the answer to the open question in
design/context.mdabout whether intake shaping runs before or after persist — attachment extraction is lossless so it runs before, in #2196; capping is lossy so it runs after, here.Why this is safe for prompt caching
A cache hit needs a byte-identical prefix, which is why sliding history truncation is deferred (#2200) rather than moved here.
These limits are a function of ONE message: how large that tool result is and which fields it carries. The same message shapes to the same bytes on turn 3 and on turn 300, whatever surrounds it, so the prefix is stable. There is a test asserting exactly that.
dropFieldscomes from a measurementAcross 251,895 real messages from local Claude Code and pi transcripts, three crossed the 1.5 MiB row budget. Pi persists a raw provider payload beside the content it renders, and that duplicate alone accounted for 2.6 MB and 1.65 MB of two of them. Dropping redundant payloads is worth more than the line caps, and costs the model nothing because it never needed both copies.
Which field is redundant is the host's call, so the module names none itself.
Scope
Tests
9 new tests covering the caps, the continuation hint, line-before-byte ordering,
dropFields, nested tool output, the pass-through cases, byte-budget cutting that never splits a surrogate pair, streaming, and prefix stability.https://claude.ai/code/session_01MXpoKqJrUFTqgmDrehDti4