API: Stream partial tool_calls argument deltas for qwen3_coder formats - #462
Open
tengotengo wants to merge 1 commit into
Open
API: Stream partial tool_calls argument deltas for qwen3_coder formats#462tengotengo wants to merge 1 commit into
tengotengo wants to merge 1 commit into
Conversation
Adds QwenToolCallDeltaStreamer, which converts TOOL-channel text into OpenAI-style tool_calls deltas as it is generated, so coding agents can render tool generation live instead of receiving one complete delta at end of stream. For well-formed calls the concatenated argument fragments are byte-identical to the end-of-stream parse, so a client that assembles deltas sees exactly what a non-streaming request returns. Malformed tag sequences are read the way the end-of-stream regexes read them: stray close tags and parameters outside a function block are dropped, and a nested function open belongs to the outer call. The streamer never raises into the collector. String values stream live once they provably cannot parse as JSON literals; values that coerce_param_value() may convert (objects, arrays, numbers, booleans, null, quoted strings) are emitted whole at parameter close. The authoritative end-of-stream parse stays the fallback whenever the streamer produced nothing, and an end-of-stream cross-check logs any divergence. That cross-check calls out the one case streaming cannot avoid: generation cut off in the middle of a call, which the end-of-stream parser drops whole while the client already holds the partial arguments. Only the qwen3_coder family is wired up for now; other formats keep the current end-of-stream behavior. Tests: byte-identity against the end-of-stream parse under random, one-character and whole-text chunkings, the stream-live decision table, malformed tag sequences, the verify() log branches, frame shape, a fuzz suite over tag-lookalike values and injected stray tags, collector wiring against a mocked backend, and live SSE contract checks against a running server. Fixtures take their wrapper tags from the format itself: with invented tags the text never reaches the tool channel and every comparison silently passes on empty input. The streaming helper in tests/_common.py keeps per-call state while folding deltas, which matters as soon as arguments arrive in more than one fragment. Closes theroyallab#460
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Streaming responses with a qwen3_coder-family format (
qwen3_coder,qwen3_5,step3_5,step3_7) now report tool calls while the model generates them:tool_callsdeltas withpartial
argumentsfragments, instead of one complete delta after generation finishes.Closes #460.
How
QwenToolCallDeltaStreamerconsumes the TOOL-channel text thatTagStreamParseralready producesand turns the pseudo-XML structure into OAI deltas: an open frame (
index,id,type,name,arguments: "{"), argument fragments as parameters close,"}"when the function closes.coerce_param_value()cannot reinterpret them later. Values it may convert (objects, arrays,numbers, booleans, null, quoted strings) are emitted whole when their parameter closes.
and parameters outside a function block are dropped, a nested function open belongs to the outer
call. The streamer never raises into the collector.
verify()compares the streamed assembly against the end-of-stream parse at the end of everyrequest. It is log-only, because fragments cannot be retracted.
Only the qwen3_coder family is wired up; harmony, muse_glimmer and the rest keep the current path.
Guarantee
For well-formed calls the concatenated
argumentsfragments per index are byte-identical to whatthe end-of-stream parser produces, under random, one-character and whole-text chunkings — a client
that assembles deltas sees exactly what a non-streaming request returns.
Three inputs diverge on purpose, and are documented in the module docstring and the docs page:
call, while the client already holds the partial arguments;
verify()names that causeexplicitly. Previously the same request returned
finish_reason: "tool_calls"with an emptytool_callslist.keeps the last one.
occurrence but read the remainder differently.
Testing
python -m unittest tests.test_toolcall_stream tests.test_tool_delta_streaming(61 tests).The whole repo suite — 182 tests — passes on Python 3.10 and 3.11.
ruff checkandruff format --diffclean.python tests/req_tool_delta.pyruns 16 contract checks against a running server(Qwen3.8-Flash-Next, ExLlamaV3): frame shape and ordering, streamed vs non-streaming equality,
tool_choicenone/required/named, reasoning-then-tool and content-then-tool ordering, clientdisconnect mid-call,
n>1,logprobs, non-streaming regression, truncated calls. Normal trafficproduces no
verify()errors in the server log.21 caught, 3 provably equivalent), so the suite fails when the guarantee breaks. One caveat
found that way: fixtures must take their wrapper tags from the format itself, otherwise the text
never reaches the tool channel and every comparison silently passes on empty input.
Also included
One-line fix in
tests/_common.py:tool_callsis a list of per-choice dicts, so the oldmembership test was never true and the entry of a call was recreated on every delta, keeping only
the last argument fragment. Invisible while calls arrived as a single delta; visible as soon as
arguments are streamed in fragments.
Relationship to #378
#378 adds an XML tool-call parsing path (it predates the current
toolcall_formatspackage) andtouches the same two files for that purpose. This change assumes the existing format parsers and
only changes how their input is streamed, so the two would need a rebase but address different
problems.