fix(proxy): raise first-byte timeout to 240s to cover large-context prefill (#13) - #16
Merged
Merged
Conversation
…refill (#13) Lifecycle testing exposed a false-positive in the #13 fix: this router withholds response headers until the worker emits its first token, so time-to-headers includes the full prompt prefill. A near-limit cold-cache prefill (131k tokens at ~800 tok/s) takes ~165s; the 60s bound tripped on large contexts and forced needless worker reloads (losing the prompt cache, which can cascade). Raise the default to 240s — above worst-case legitimate prefill — so a genuinely hung worker is still detected while normal large-context requests are not. Still overridable via $PROXY_FIRST_BYTE_TIMEOUT. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up calibration to #13 (found by lifecycle testing on the live server).
What went wrong
The #13 first-byte timeout (60s) false-tripped on large contexts. This router withholds response headers until the worker emits its first token, so time-to-headers includes the full prompt prefill. A near-limit cold-cache prefill (131k tokens @ ~800 tok/s ≈ 165s) exceeded 60s, so the proxy declared the (healthy, busy-prefilling) worker "hung", cycled
recover_worker(losing the prompt cache), and could cascade.Observed on 2026-07-19: requests at 59.6s and 60.7s just made it (200); the next crossed 60s → false DeadWorkerError → unload/reload → 400 on the cache-cold retry. Two spurious reloads in one session.
Fix
Raise
FIRST_BYTE_TIMEOUTdefault 60s → 240s — above worst-case legitimate cold prefill of the full 131k context. A genuinely hung worker is still detected (just later); normal large-context requests are no longer mistaken for hangs. Still overridable via$PROXY_FIRST_BYTE_TIMEOUT.Unit tests unchanged (they use a 1s override): 3/3 pass.
Relates to #13.
🤖 Generated with Claude Code