fix(proxy): bound first-byte wait to recover a hung model-worker (#13) - #15
Merged
Conversation
A hung worker child makes the llama.cpp router accept a completion and never send a response; _do_forward awaited with timeout=None and hung forever, so the existing dead-worker recovery (which only fires on a 5xx body) was never reached. Bound the wait for response headers (FIRST_BYTE_TIMEOUT, default 60s, override $PROXY_FIRST_BYTE_TIMEOUT) and route a timeout into the existing DeadWorkerError -> recover_worker -> 503 path. Guard the non-model path too. Post-header streaming (25s keep-alive) is unchanged. Adds tests/test_dead_worker_timeout.py driving the real proxy_request against a mock router: hung (RED->GREEN), slow cold-load (not tripped), healthy passthrough. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #13.
Problem
A hung model-worker child makes the llama.cpp router accept a completion and never respond.
_do_forwardawaited the upstream withtimeout=None, so the proxy (and any pi/Claude client behind it) hung indefinitely. The existing dead-worker recovery only triggers on a router 5xx body (_is_dead_worker_response), but a hung worker produces silence, not a 5xx — so recovery was never reached.Fix
Bound the wait for response headers (nginx
proxy_read_timeoutmodel) and route a timeout into the existing recovery path:FIRST_BYTE_TIMEOUT(default 60s, override$PROXY_FIRST_BYTE_TIMEOUT) — generous enough for a 15–25s cold model load.session.request(...)inasyncio.wait_for; on timeout raise the existingDeadWorkerError→recover_worker→ retry → 503./v1/modelsetc. returns 503 instead of an uncaught error.Test (TDD)
tests/test_dead_worker_timeout.pydrives the realproxy_requestagainst a scripted mock router:hung(accept, never respond) — RED before the fix (hangs), GREEN after (503 +recover_workercalled).slowcold-load (headers delayed under the timeout) — not tripped (guardrail: slow ≠ hung).healthy— passthrough.Full suite: 23/23 pass.
Scope
Handles the no-headers-at-all hang (the 2026-07-19 incident). The headers-then-mid-stream-silence variant is still served indefinite keep-alives by the pre-existing loop (separate follow-up). Does not address the distinct empty-turn / degenerate-output problem (#14).
🤖 Generated with Claude Code