Skip to content

fix(proxy): bound first-byte wait to recover a hung model-worker (#13) - #15

Merged
Hrt-Htk merged 1 commit into
mainfrom
fix/proxy-hung-worker-timeout
Jul 19, 2026
Merged

fix(proxy): bound first-byte wait to recover a hung model-worker (#13)#15
Hrt-Htk merged 1 commit into
mainfrom
fix/proxy-hung-worker-timeout

Conversation

@Hrt-Htk

@Hrt-Htk Hrt-Htk commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Closes #13.

Problem

A hung model-worker child makes the llama.cpp router accept a completion and never respond. _do_forward awaited the upstream with timeout=None, so the proxy (and any pi/Claude client behind it) hung indefinitely. The existing dead-worker recovery only triggers on a router 5xx body (_is_dead_worker_response), but a hung worker produces silence, not a 5xx — so recovery was never reached.

Fix

Bound the wait for response headers (nginx proxy_read_timeout model) and route a timeout into the existing recovery path:

  • FIRST_BYTE_TIMEOUT (default 60s, override $PROXY_FIRST_BYTE_TIMEOUT) — generous enough for a 15–25s cold model load.
  • Wrap the initial session.request(...) in asyncio.wait_for; on timeout raise the existing DeadWorkerErrorrecover_worker → retry → 503.
  • Guard the non-model path so a hung worker starving /v1/models etc. returns 503 instead of an uncaught error.
  • Post-header streaming (25s per-chunk keep-alive) is unchanged — this only bounds time-to-first-response, not generation length.

Test (TDD)

tests/test_dead_worker_timeout.py drives the real proxy_request against a scripted mock router:

  • hung (accept, never respond) — RED before the fix (hangs), GREEN after (503 + recover_worker called).
  • slow cold-load (headers delayed under the timeout) — not tripped (guardrail: slow ≠ hung).
  • healthy — passthrough.

Full suite: 23/23 pass.

Scope

Handles the no-headers-at-all hang (the 2026-07-19 incident). The headers-then-mid-stream-silence variant is still served indefinite keep-alives by the pre-existing loop (separate follow-up). Does not address the distinct empty-turn / degenerate-output problem (#14).

🤖 Generated with Claude Code

A hung worker child makes the llama.cpp router accept a completion and
never send a response; _do_forward awaited with timeout=None and hung
forever, so the existing dead-worker recovery (which only fires on a 5xx
body) was never reached. Bound the wait for response headers
(FIRST_BYTE_TIMEOUT, default 60s, override $PROXY_FIRST_BYTE_TIMEOUT) and
route a timeout into the existing DeadWorkerError -> recover_worker ->
503 path. Guard the non-model path too. Post-header streaming (25s
keep-alive) is unchanged.

Adds tests/test_dead_worker_timeout.py driving the real proxy_request
against a mock router: hung (RED->GREEN), slow cold-load (not tripped),
healthy passthrough.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Hrt-Htk
Hrt-Htk merged commit 122dfda into main Jul 19, 2026
1 check passed
@Hrt-Htk
Hrt-Htk deleted the fix/proxy-hung-worker-timeout branch July 19, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proxy hangs indefinitely on a hung model-worker (no first-byte timeout)

1 participant