fix(llm): clamp requested ctx_size to a model's real context ceiling - #3003
fix(llm): clamp requested ctx_size to a model's real context ceiling#3003itomek wants to merge 10 commits into
Conversation
Lemonade reports a model's trained-context ceiling as max_context_window, but GAIA never checked it. It picked ctx_size from a flat per-device constant (65536 GPU/CPU, 32768 NPU) and reported that value back as "the context size" even when the model's real training window was smaller and llama.cpp silently capped it internally. A model trained at 40960 tokens would be requested and reported at 65536; prompts near the real ceiling failed with a 400 the user had no way to predict from GAIA's own "success" message. Reads max_context_window from Lemonade's catalog and health responses, clamps the ctx_size actually sent to /load, and reports the value really in force. Critically, this also covers the no-reload path: Lemonade will echo back recipe_options.ctx_size == whatever was last requested even when it capped the window lower internally, so a model already resident from an earlier process looked "already sufficient" and was never rechecked. Every reading of status.context_size is now cross-checked against the loaded model's ceiling, not just the ones following a reload. A registry floor (MODELS[...].min_ctx_size) that exceeds a known ceiling now raises loudly instead of silently picking one side. A model with no discoverable ceiling (undownloaded, older Lemonade) proceeds at the requested value with a warning naming the gap. The eval-only ctx_size_override exact-pin path is untouched.
github-actions
Bot
added
llm
…ntext A model already resident at its real ceiling (e.g. Qwen3-0.6B at 40960, GPU profile requesting 65536) was still triggering a full model reload to the exact same clamped value on every ensure_ready() call whose context came up short — the clamp fixed the *reported* number but not the redundant reload. Since the manager's memoization is per-process, this fired on every `gaia llm` / `gaia chat` / `gaia agent` invocation, not once, trading a wrong number for a wrong number plus a repeated model reload. Both call sites that could enter `_try_reload_with_ctx` now check whether the resident model is already at its known ceiling first and, if so, skip the reload and report the honest capped value directly (` _report_capped_at_ceiling`, reusing the same message shape the just-preloaded path already used for this exact situation). No load_model call, no "Reloading..." message that would have been untrue.
…ver one The ceiling-capped message reused print_context_message, which tells the user to stop the server and restart it with a bigger ctx_size. That remediation does nothing for a trained-context ceiling — no restart, flag, or gaia init raises a GGUF's n_ctx_train. The honest number was followed by instructions that can't work, sending the user to debug a healthy install. Added print_ceiling_message: states the shortfall as a property of the model and points at the only real remedy (a model with a larger trained context). print_context_message is untouched for its real callers, where the server-remediation text still applies.
|
Independently verified against a live Lemonade 11.6.0 with The starting state was the bug itself — the model resident with the server echoing The no-reload behaviour is the part worth checking closely, and it holds. After three consecutive runs the server still reports That mattered: an earlier revision of this fix clamped only on reload, which meant a model already resident from a previous process — the normal case, since Lemonade is a persistent daemon — would have reloaded on every 🔍 Technical detailsMeasured, post-fix, on
Pre-fix, on real Strix Halo hardware (Ryzen AI MAX+ 395 / Radeon 8060S), the same scenario printed Two internal warnings fire correctly on the corrected path: Separately confirmed that |
Verdict: Request changesThis PR makes GAIA report the context window it actually has instead of echoing back what it asked for — a real bug worth fixing, and the clamp itself is well built and well tested. But the two halves of the change disagree with each other, and the result is that some users lose the ability to run the model at all. The blocking issue: when a model's real context ceiling is smaller than the minimum GAIA has recorded for it, the startup path now says "fine, we'll use the smaller window and carry on" while the chat path treats the same situation as a fatal error and refuses to answer. The evidence run captured exactly this — a warning, then Also worth fixing before merge:
Real-world evidence
The evidence supports the clamp working as intended. It also is the source of the blocking finding above — the bundle explicitly flags the "no answer returned" behaviour change and the stale UI number, both reproduced live: Verified against the diff — the failure is not a stub artifact; the code paths line up. Whether a real GGUF reports a 🔍 Technical details🔴 Critical1. The clamp creates the condition that makes Pre-PR: the manager requests 65536, Lemonade echoes 65536, and the client's early return at This affects any model whose The two halves also disagree by construction: Suggested resolutions (maintainer call, hence no ```suggestion block):
Either way it needs a test that pins the manager and client agreeing on the same input, and the strix-halo lane needs to confirm what the real default GGUF reports for 🟡 Important2. The route reads 3.
Storing the model id the ceiling came from and invalidating when the resident LLM changes would fix it: _context_ceiling: Optional[int] = None
_context_ceiling_model: Optional[str] = None…and gating 🟢 Minor4. Possible extra catalog round trip per 5. Recheck branch reads 6. A known ceiling can be overwritten with Strengths
|
Approve with suggestionsThe ceiling-clamp logic is correct and the test coverage is thorough. One user-facing behaviour gap and one CLAUDE.md violation are worth fixing before merge. 🟡 Wrong error advice shown when reload first discovers the ceiling When On the next The fix: after 🟡 This adds a new 🔍 Technical detailsWrong fallback after reload failure — two call sites in
if cls._try_reload_with_ctx(client, status, min_context_size, quiet, cls._lock):
return True
cls._log.warning("... Restart Lemonade Server. ...") # wrong for ceiling case
if not quiet:
cls.print_context_message(cls._context_size, min_context_size, MessageType.WARNING)
Both should add a ceiling-aware bypass after the reload attempt: if cls._try_reload_with_ctx(client, status, min_context_size, quiet, cls._lock):
return True
# _try_reload_with_ctx may have just discovered the ceiling — check before
# falling through to server-restart advice, which can't raise n_ctx_train.
if cls._context_ceiling is not None and cls._context_size >= cls._context_ceiling:
cls._report_capped_at_ceiling(min_context_size, quiet)
# fall through to return True (caller's existing path already returns True here)
else:
cls._log.warning("... Restart Lemonade Server. ...")
if not quiet:
cls.print_context_message(cls._context_size, min_context_size, MessageType.WARNING)Broad-except location — except Exception as e: # pylint: disable=broad-except
self.log.debug(f"Could not query model catalog for {model_id!r}: {e}")
return NoneNarrow to the specific exception types that a network call to Lemonade actually raises. |
The floor-vs-ceiling conflict check in _ensure_model_loaded_locked raised LemonadeClientError when a model's MODELS registry floor exceeded its real trained-context ceiling. That contradicts LemonadeManager, which treats the identical situation as "proceed capped" and warns instead. Since min_ctx_size=GPU_CTX_SIZE (65536) is set on most registry entries including the default model, the raise landed on the default path: once the manager reloads at the clamped ceiling, every subsequent chat call hit the raise where the pre-PR build still answered. Clamp to the ceiling and warn instead, matching the manager's stance. Also fixes both LemonadeManager fall-through sites that told the user to "Restart Lemonade Server" after a reload failed because it had just discovered the model's real ceiling — that advice can't raise a GGUF's trained context, and the reload path already printed the correct message.
The catalog lookup's except Exception -> return None swallowed anything, not just the network/parse failures it was meant to tolerate. list_models() only ever raises LemonadeClientError (its subclasses included) since every HTTP/JSON failure inside _send_request is wrapped into one — narrow the catch to that.
/api/system/status read recipe_options.ctx_size straight off /health and reported it as context_size_sufficient, bypassing the ctx-size clamp the CLI already applies. On a machine where the CLI correctly warns "capped at 40960", the UI could report the raw 65536 echo and claim the context is sufficient — two different answers for the same running server. /health's all_models_loaded entries already carry max_context_window per loaded model, so this reuses resolve_effective_ctx_size (the same pure clamp LemonadeManager uses) with no extra network round trip.
_context_ceiling was class state with no model id attached, so a long-lived process (gaia.ui.server) that first loaded a small-context model and later switched to a roomier one kept reporting the small model's cap forever — the fast path returned "at known ceiling" before ever re-checking live status. Add _context_ceiling_model alongside _context_ceiling and gate the fast path on periodically reconfirming it, bounded by the same _RECHECK_INTERVAL that already rate-limits the chat-message hot path so this doesn't add an unconditional status call. Also guard the ceiling assignment in _try_reload_with_ctx against a falsy value discarding an already-discovered ceiling, and None-guard loaded_models in the recheck branch the same way the init branch already does.
Drop an import left unused after rewriting the raise-based test to a clamp-and-warn one, and apply black formatting.
_ensure_model_loaded_locked compared loaded_ctx against the raw MODELS registry floor before applying the floor-vs-ceiling clamp. A model resident exactly at its (below-floor) ceiling never matched that unclamped comparison, so every chat completion reloaded the model at the same ctx_size it was already running — undoing the point of the #2053 pre-flight probe. Moving the clamp ahead of the comparison lets an at-ceiling model short-circuit like LemonadeManager already does. Also dedupes the clamp warning per model so it doesn't repeat on every call once the early return is reachable again.
|
Addressed the review. Five findings fixed, two small guards added, one pushed back. The substantive one: the client and the manager disagreed about what to do when a model's real ceiling sits below its registry That reverses the "raises loudly" decision in the PR description above. Loud is still right, but a warning is loud enough here — refusing to run a model that physically cannot offer more context isn't a safer failure, just a worse one. Also fixed: the first-discovery reload path no longer follows an honest ceiling message with "restart Lemonade Server"; the Agent UI's status panel now reports the same clamped number the CLI does instead of the old inflated one; and the cached ceiling is tied to the model it came from, so a long-running server that switches to a roomier model stops reporting the stale capped value. Pushed back on the catalog round trip. That path is already rate-limited and already pays a 🔍 Technical details
One defect was caught during this pass and fixed before it shipped: the first cut of the clamp ran after the pre-flight probe's Verification on this branch ( The three failures are
Real-hardware pass on Strix Halo still outstanding — in particular what a real |
Verdict: Approve ✅This is a solid, well-targeted fix for #2992. The approach is correct: extract a pure clamp, thread it through every site that reads or caches Test coverage is strong: 307 new lines in A few observations (none blocking): 🟢 🟢 The UI path ( 🟢 🔍 Technical detailsDead-weight side-effect entry (
Older Lemonade degradation (
The "warn once per model" guarantee is per-instance. In |
1 participant
Closes #2992
GAIA picked its context window from a flat per-device number (65536 GPU/CPU, 32768 NPU) with no idea which model was actually loading, then reported that number back as fact. For a model whose real trained context is smaller — Qwen3-0.6B-GGUF's is 40960 — GAIA requested and reported 65536 while llama.cpp silently capped the real window at 40960. A ~40,900-token prompt worked; ~41,000 got a 400 the user had no way to predict from GAIA's own "success" message.
Lemonade already exposes the real ceiling as
max_context_window(catalog/api/v1/modelsand/api/v1/health). This clamps the ctx_size GAIA actually requests to that ceiling and reports the value really in force. It also covers the no-reload path: Lemonade echoes backrecipe_options.ctx_sizeas whatever was last requested even when it capped the window lower internally, so a model already resident from an earlier process looked "already sufficient" and was never rechecked — every reading ofstatus.context_sizeis now cross-checked against the loaded model's ceiling, not just ones following a reload.A model already sitting at its known ceiling now skips the reload entirely instead of running one that reproduces the identical result — since the manager's state is per-process, an unconditional reload here would otherwise fire on every single
gaiainvocation, not once. And the message for that case is now specific to the ceiling: it names the shortfall as a property of the model (its trained context) and points at the only real remedy — a bigger model — instead of the "stop the server and restart it" text meant for a genuinely-too-small server config, which can't do anything about a GGUF's trained context.Verified live against a running Lemonade 11.6.0 with
Qwen3-0.6B-GGUF(realmax_context_window=40960), before vs. after this fix, same repro:✅ Context size updated to 65536 tokens.(false — the real window was 40960)⚠️ This model's trained context is 40960 tokens (requested 65536); that is the maximum available regardless of configuration. Use a model with a larger trained context if you need more.Two explicit decisions, both tested:
MODELS[...].min_ctx_size) that exceeds a known ceiling raises loudly instead of silently picking one side.The eval-only
ctx_size_overrideexact-pin path (_ensure_pinned_load) is untouched — confirmed by a test assertingget_model_max_context_windowis never called on that path, so a pinned eval run still gets exactly what it asked for.Known adjacent issue, not introduced here (verified against
main): switching models can print a warning naming the previous resident model instead of the one about to load, because the pre-flight ctx check runs before model-switch logic picks the new target. Onmainthis same ordering silently reloads the wrong (soon-to-be-replaced) model and falsely reports success; this PR's honest check surfaces it as a warning instead, without the wasted reload — an improvement, but the underlying ordering issue is unfixed and out of scope here.Test plan
pytest tests/unit/test_lemonade_ctx_ceiling.py tests/unit/test_lemonade_manager_preload.py tests/unit/test_lemonade_model_loading.py tests/unit/test_llm.py tests/unit/installer/test_init_verify_ctx_display.py— 71 passedpytest tests/test_lemonade_client.py— 87 passed, 3 pre-existing failures (test_chat_completions,test_chat_completions_nonstream_includes_auth_header,test_validate_context_size_insufficient), confirmed present onmainviagit stashbefore this change — unrelated to this fixpython util/lint.py --all --fix— black/isort/pylint/flake8/bandit all pass (mypy warnings are pre-existing, non-blocking)Qwen3-0.6B-GGUFdownloaded — clamp, honest reporting, the skipped reload, and the model-specific message confirmed end-to-end (see before/after strings above)