Skip to content

feat(proxy): rescue tool calls emitted inside reasoning_content (#1) - #4

Merged
Hrt-Htk merged 1 commit into
mainfrom
feat/issue1-rescue-toolcalls-in-reasoning
Jun 15, 2026
Merged

feat(proxy): rescue tool calls emitted inside reasoning_content (#1)#4
Hrt-Htk merged 1 commit into
mainfrom
feat/issue1-rescue-toolcalls-in-reasoning

Conversation

@Hrt-Htk

@Hrt-Htk Hrt-Htk commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Closes #1.

What

SSEChunkLogger now rescues tool calls the chat model emits as XML
(<tool_call> / <function=…>) inside reasoning_content instead of the
tool_calls delta. Previously llama-server didn't recognize them, the turn ended
finish_reason=stop, and the client never ran the tool (the "stuck thinking"
pattern seen in the logs).

The proxy now transforms the forwarded SSE stream:

  • detects tool-call XML in reasoning_content deltas across chunk/event
    boundaries (stateful capture + partial-marker holdback);
  • strips the XML from forwarded reasoning while keeping genuine prose;
  • re-emits it as a proper tool_calls SSE delta (args JSON-encoded, type-coerced);
  • rewrites a trailing finish_reason=stop to tool_calls when a call was rescued;
  • handles the no-wrapper and bare/empty <tool_call> variants and multiple calls.

Logging still observes the original deltas; only outbound bytes change.

Also fixes a latent stream-truncation bug surfaced while building this: readany()
now loops until a complete SSE event is available (or true EOF), so it never
returns b"" mid-stream — the forward loop treats empty as EOF and would
otherwise truncate responses whenever an upstream chunk split an event.

Tests

Adds tests/test_toolcall_rescue.py (9 stdlib-unittest cases): passthrough,
split-across-chunks, sub-event 13-byte chunk boundaries, prose-around-call,
no-wrapper, bare-empty, arg coercion, multiple calls per turn.

Verified offline by replaying a real captured SSE stream at several chunk sizes,
and live end-to-end against the chat proxy (normal streaming intact; no false
rescue).

The chat model sometimes emits a tool call as XML (<tool_call> / <function=…>)
inside reasoning_content instead of the tool_calls delta. llama-server never
recognizes it, so the turn ends with finish_reason=stop and the client never
runs the tool — the "stuck thinking" pattern from the logs.

SSEChunkLogger now transforms the forwarded stream:
- detects tool-call XML in reasoning_content deltas, across chunk/event
  boundaries (stateful capture + partial-marker holdback);
- strips the XML from the forwarded reasoning while keeping genuine prose;
- re-emits it as a proper tool_calls SSE delta (args JSON-encoded, type-coerced);
- rewrites a trailing finish_reason=stop to tool_calls when a call was rescued;
- handles the no-wrapper and bare/empty <tool_call> variants and multiple calls.

Logging still observes the original deltas; only the outbound bytes change.

readany() now loops until a complete SSE event is ready (or true EOF) so it
never returns b"" mid-stream — the forward loop treats empty as EOF, which would
otherwise truncate responses whenever an upstream chunk split an event.

Adds tests/test_toolcall_rescue.py (9 stdlib-unittest cases): passthrough,
split-across-chunks, sub-event 13-byte chunk boundaries, prose-around-call,
no-wrapper, bare-empty, arg coercion, and multiple calls per turn. Verified
offline by replaying a real captured SSE stream at several chunk sizes, and
live end-to-end against the restarted chat proxy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Hrt-Htk
Hrt-Htk merged commit 6df74c4 into main Jun 15, 2026
1 check passed
@Hrt-Htk
Hrt-Htk deleted the feat/issue1-rescue-toolcalls-in-reasoning branch June 15, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool calls emitted inside reasoning_content are never executed (forward them as real tool calls)

1 participant