Skip to content

API: Stream partial tool_calls argument deltas for qwen3_coder formats - #462

Open
tengotengo wants to merge 1 commit into
theroyallab:mainfrom
tengotengo:feature/stream-tool-call-deltas
Open

API: Stream partial tool_calls argument deltas for qwen3_coder formats#462
tengotengo wants to merge 1 commit into
theroyallab:mainfrom
tengotengo:feature/stream-tool-call-deltas

Conversation

@tengotengo

Copy link
Copy Markdown

What

Streaming responses with a qwen3_coder-family format (qwen3_coder, qwen3_5, step3_5,
step3_7) now report tool calls while the model generates them: tool_calls deltas with
partial arguments fragments, instead of one complete delta after generation finishes.
Closes #460.

How

QwenToolCallDeltaStreamer consumes the TOOL-channel text that TagStreamParser already produces
and turns the pseudo-XML structure into OAI deltas: an open frame (index, id, type, name,
arguments: "{"), argument fragments as parameters close, "}" when the function closes.

  • String values stream live as soon as they provably cannot parse as a JSON literal, so
    coerce_param_value() cannot reinterpret them later. Values it may convert (objects, arrays,
    numbers, booleans, null, quoted strings) are emitted whole when their parameter closes.
  • Malformed tag sequences are read the way the end-of-stream regexes read them: stray close tags
    and parameters outside a function block are dropped, a nested function open belongs to the outer
    call. The streamer never raises into the collector.
  • The authoritative end-of-stream parse stays the fallback whenever the streamer produced nothing.
  • verify() compares the streamed assembly against the end-of-stream parse at the end of every
    request. It is log-only, because fragments cannot be retracted.

Only the qwen3_coder family is wired up; harmony, muse_glimmer and the rest keep the current path.

Guarantee

For well-formed calls the concatenated arguments fragments per index are byte-identical to what
the end-of-stream parser produces, under random, one-character and whole-text chunkings — a client
that assembles deltas sees exactly what a non-streaming request returns.

Three inputs diverge on purpose, and are documented in the module docstring and the docs page:

  1. A call generation was cut off inside. The regex needs a closing tag and drops the whole
    call, while the client already holds the partial arguments; verify() names that cause
    explicitly. Previously the same request returned finish_reason: "tool_calls" with an empty
    tool_calls list.
  2. Duplicate parameter keys. Both are streamed (they parse to the same object); the parser
    keeps the last one.
  3. A value containing a literal parameter close tag. Both sides end the value at the first
    occurrence but read the remainder differently.

Testing

  • Unit: python -m unittest tests.test_toolcall_stream tests.test_tool_delta_streaming (61 tests).
    The whole repo suite — 182 tests — passes on Python 3.10 and 3.11.
  • ruff check and ruff format --diff clean.
  • Live: python tests/req_tool_delta.py runs 16 contract checks against a running server
    (Qwen3.8-Flash-Next, ExLlamaV3): frame shape and ordering, streamed vs non-streaming equality,
    tool_choice none/required/named, reasoning-then-tool and content-then-tool ordering, client
    disconnect mid-call, n>1, logprobs, non-streaming regression, truncated calls. Normal traffic
    produces no verify() errors in the server log.
  • The tests were checked against deliberate breakage of each promised behaviour (24 mutations:
    21 caught, 3 provably equivalent), so the suite fails when the guarantee breaks. One caveat
    found that way: fixtures must take their wrapper tags from the format itself, otherwise the text
    never reaches the tool channel and every comparison silently passes on empty input.

Also included

One-line fix in tests/_common.py: tool_calls is a list of per-choice dicts, so the old
membership test was never true and the entry of a call was recreated on every delta, keeping only
the last argument fragment. Invisible while calls arrived as a single delta; visible as soon as
arguments are streamed in fragments.

Relationship to #378

#378 adds an XML tool-call parsing path (it predates the current toolcall_formats package) and
touches the same two files for that purpose. This change assumes the existing format parsers and
only changes how their input is streamed, so the two would need a rebase but address different
problems.

Adds QwenToolCallDeltaStreamer, which converts TOOL-channel text into
OpenAI-style tool_calls deltas as it is generated, so coding agents can
render tool generation live instead of receiving one complete delta at
end of stream.

For well-formed calls the concatenated argument fragments are byte-identical
to the end-of-stream parse, so a client that assembles deltas sees exactly
what a non-streaming request returns. Malformed tag sequences are read the
way the end-of-stream regexes read them: stray close tags and parameters
outside a function block are dropped, and a nested function open belongs to
the outer call. The streamer never raises into the collector.

String values stream live once they provably cannot parse as JSON literals;
values that coerce_param_value() may convert (objects, arrays, numbers,
booleans, null, quoted strings) are emitted whole at parameter close.

The authoritative end-of-stream parse stays the fallback whenever the
streamer produced nothing, and an end-of-stream cross-check logs any
divergence. That cross-check calls out the one case streaming cannot avoid:
generation cut off in the middle of a call, which the end-of-stream parser
drops whole while the client already holds the partial arguments.

Only the qwen3_coder family is wired up for now; other formats keep the
current end-of-stream behavior.

Tests: byte-identity against the end-of-stream parse under random,
one-character and whole-text chunkings, the stream-live decision table,
malformed tag sequences, the verify() log branches, frame shape, a fuzz
suite over tag-lookalike values and injected stray tags, collector wiring
against a mocked backend, and live SSE contract checks against a running
server. Fixtures take their wrapper tags from the format itself: with
invented tags the text never reaches the tool channel and every comparison
silently passes on empty input. The streaming helper in tests/_common.py
keeps per-call state while folding deltas, which matters as soon as
arguments arrive in more than one fragment.

Closes theroyallab#460
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream partial tool_calls argument deltas for coding agents (parity with vLLM/llama.cpp)

1 participant