feat(proxy): rescue tool calls emitted inside reasoning_content (#1) - #4
Merged
Merged
Conversation
The chat model sometimes emits a tool call as XML (<tool_call> / <function=…>) inside reasoning_content instead of the tool_calls delta. llama-server never recognizes it, so the turn ends with finish_reason=stop and the client never runs the tool — the "stuck thinking" pattern from the logs. SSEChunkLogger now transforms the forwarded stream: - detects tool-call XML in reasoning_content deltas, across chunk/event boundaries (stateful capture + partial-marker holdback); - strips the XML from the forwarded reasoning while keeping genuine prose; - re-emits it as a proper tool_calls SSE delta (args JSON-encoded, type-coerced); - rewrites a trailing finish_reason=stop to tool_calls when a call was rescued; - handles the no-wrapper and bare/empty <tool_call> variants and multiple calls. Logging still observes the original deltas; only the outbound bytes change. readany() now loops until a complete SSE event is ready (or true EOF) so it never returns b"" mid-stream — the forward loop treats empty as EOF, which would otherwise truncate responses whenever an upstream chunk split an event. Adds tests/test_toolcall_rescue.py (9 stdlib-unittest cases): passthrough, split-across-chunks, sub-event 13-byte chunk boundaries, prose-around-call, no-wrapper, bare-empty, arg coercion, and multiple calls per turn. Verified offline by replaying a real captured SSE stream at several chunk sizes, and live end-to-end against the restarted chat proxy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1.
What
SSEChunkLoggernow rescues tool calls the chat model emits as XML(
<tool_call>/<function=…>) insidereasoning_contentinstead of thetool_callsdelta. Previously llama-server didn't recognize them, the turn endedfinish_reason=stop, and the client never ran the tool (the "stuck thinking"pattern seen in the logs).
The proxy now transforms the forwarded SSE stream:
reasoning_contentdeltas across chunk/eventboundaries (stateful capture + partial-marker holdback);
tool_callsSSE delta (args JSON-encoded, type-coerced);finish_reason=stoptotool_callswhen a call was rescued;<tool_call>variants and multiple calls.Logging still observes the original deltas; only outbound bytes change.
Also fixes a latent stream-truncation bug surfaced while building this:
readany()now loops until a complete SSE event is available (or true EOF), so it never
returns
b""mid-stream — the forward loop treats empty as EOF and wouldotherwise truncate responses whenever an upstream chunk split an event.
Tests
Adds
tests/test_toolcall_rescue.py(9 stdlib-unittest cases): passthrough,split-across-chunks, sub-event 13-byte chunk boundaries, prose-around-call,
no-wrapper, bare-empty, arg coercion, multiple calls per turn.
Verified offline by replaying a real captured SSE stream at several chunk sizes,
and live end-to-end against the chat proxy (normal streaming intact; no false
rescue).