kimik3-fp4-b300-vllm-agentic-dspark: add LMCache DRAM KV-offload arm - #2597
kimik3-fp4-b300-vllm-agentic-dspark: add LMCache DRAM KV-offload arm#2597sammshen wants to merge 7 commits into
Conversation
Add a dedicated config key kimik3-fp4-b300-vllm-agentic-dspark-lmcache with an LMCache 0.5.4rc2 DRAM KV-offload arm at TP8 conc 4/8/16, on top of the unchanged DSpark MTP serving stack of kimik3-fp4-b300-vllm-agentic-dspark (same image, script, and topology). A separate key means the changelog selects only the LMCache points; the resident and vllm-simple arms of the base key are not re-run. B300 sister of the MI355X arm in #2583.
sammshen
added
agentx
d62b9c9 to
eebc552
Compare
# Conflicts: # perf-changelog.yaml
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| wait_for_ready \ | ||
| --endpoint "http://127.0.0.1:${LMCACHE_HTTP_PORT}/healthcheck" \ | ||
| --log "$LMCACHE_LOG" \ | ||
| --pid "$LMCACHE_PID" \ | ||
| --sleep-interval 1 \ | ||
| --timeout 600 | ||
|
|
||
| # 100k-330k-token agentic prefixes make single retrieves large; use the |
There was a problem hiding this comment.
🟡 The lmcache arm calls wait_for_ready (benchmark_lib.sh) as a plain foreground statement, and that function's first line is set +x with no restore — so for this arm only, xtrace is silently disabled ~90 lines before the script's own explicit set +x at line 321. That swallows trace output for the DSpark SPEC_CONFIG build, MAX_NUM_SEQS, the CUDA_GRAPH_CAPTURE_SIZES loop, and COMPILATION_CONFIG assembly that the none/vllm-simple arms still trace. A one-line set -x right after the wait_for_ready call in the lmcache case (kimik3_fp4_b300_vllm_mtp.sh:241) restores parity.
Extended reasoning...
wait_for_ready() in benchmarks/benchmark_lib.sh (line 349) begins with set +x and never re-enables xtrace before it returns. In this script it is invoked as a plain foreground statement inside the new lmcache) case arm (kimik3_fp4_b300_vllm_mtp.sh:234-239), not in a subshell or pipeline, so the set +x leaks straight into the caller's shell and stays off for the rest of the script.
The script itself only toggles xtrace twice: set -x at line 3, and { set +x; } 2>/dev/null at line 321, immediately before the VLLM_CMD array is assembled. For the none/vllm-simple arms — which never call wait_for_ready inside the case — xtrace stays ON from line 3 all the way through the DSpark SPEC_CONFIG string construction, MAX_NUM_SEQS, the CUDA_GRAPH_CAPTURE_SIZES enumeration loop, and the COMPILATION_CONFIG assembly, and is only turned off at line 321 as intended. For the lmcache arm, the exact same block of code runs with xtrace already OFF, because wait_for_ready cut it roughly 90 lines earlier at line 234-239. This is a real, PR-introduced asymmetry: wait_for_ready for the vLLM server itself (wait_for_server_ready, called after line 321) never surfaces this because it runs after the explicit set +x anyway, so the discrepancy is unique to the new lmcache code path.
Concretely, this means the job log for an lmcache-arm run silently loses the trace lines that would show how SPEC_CONFIG, MAX_NUM_SEQS, CUDA_GRAPH_CAPTURE_SIZES, and COMPILATION_CONFIG were built — variables specific to this PR's own DSpark/MTP logic — while the none/vllm-simple arms keep full visibility into the same code. That directly undercuts the debuggability goal this same PR's SKILL.md update is pushing for (reading job logs carefully to root-cause failures), even though that SKILL.md section is really about server.log, not the launcher's own xtrace.
Proof, step by step:
- Script starts with
set -x(line 3) — xtrace is ON. - In the
lmcache)arm, after starting the LMCache server in the background, the script callswait_for_ready --endpoint ... --pid "$LMCACHE_PID" ...as a plain foreground call (line 234). wait_for_ready()'s first executed line isset +x(benchmark_lib.sh:349) — this runs in the current shell, so xtrace is now OFF for the rest of the script.- The script proceeds to build
SPEC_CONFIG, computeMAX_NUM_SEQS=$((2 * CONC)), run thefor ((num_seqs=1; ...))loop buildingCUDA_GRAPH_CAPTURE_SIZES, and assembleCOMPILATION_CONFIG— none of this is traced for the lmcache arm, whereas fornone/vllm-simplethese same lines run with xtrace still ON (sinceset -xwas never disabled before line 321). - Line 321's
{ set +x; } 2>/dev/nullis now a no-op for the lmcache arm (xtrace was already off) but is the actual first disable point for the other two arms.
Impact is limited to trace verbosity, not execution: the final resolved vLLM command is still written verbatim via printf '%q ' ... | tee "$RESULT_DIR/vllm_command.txt" regardless of xtrace state, and the LMCache command is captured via append_command before the call in question. So the load-bearing debugging artifacts survive; only the intermediate variable-construction trace is lost for this one arm, which is why this is a nit rather than a blocking issue.
Fix is a one-liner: add set -x immediately after the wait_for_ready call in the lmcache arm (or have wait_for_ready save and restore the caller's xtrace state internally, which would also fix this for any future caller).
| # The LMCache arm starts a second long-lived process that must not outlive this | ||
| # job (its L1 holds the whole host-DRAM budget). The vLLM server's lifecycle is | ||
| # left exactly as it was -- the job wrapper still owns it. | ||
| LMCACHE_PID="" | ||
|
|
||
| cleanup_lmcache_server() { | ||
| local exit_code=$? | ||
| trap - EXIT | ||
| set +e | ||
| stop_background_process_tree "$LMCACHE_PID" "LMCache server" | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup_lmcache_server EXIT | ||
|
|
||
| # ---- KV offloading ---------------------------------------------------------- | ||
| # The generated TOTAL_CPU_DRAM_GB budget is the aggregate host-DRAM pool for the |
There was a problem hiding this comment.
🟡 sweep:cleanup_(lmcache_server|agentic_services)()
This file's cleanup_lmcache_server() (kimik3_fp4_b300_vllm_mtp.sh:126-138) hand-rolls the capture-$?/trap-EXIT/set-+e/stop_background_process_tree/exit-$exit_code idiom to reap one auxiliary background PID; dsv4_fp4_mi355x_vllm_mtp.sh's cleanup_lmcache_server/cleanup_agentic_services do the same for one-to-three PIDs. A benchmark_lib.sh helper like register_cleanup_pid "$PID" "label" that accumulates tracked PIDs and installs the trap once would collapse each call site to a single line.
Extended reasoning...
Every agentic script that spawns an auxiliary long-lived process (an LMCache MP-server, a router, a Mooncake master, etc.) alongside the main vLLM/SGLang server needs to guarantee that process is killed when the job exits, with the original exit code preserved so CI still reports pass/fail correctly. The pattern used to do this — local exit_code=$?; trap - EXIT; set +e; stop_background_process_tree "$PID" "label"; exit "$exit_code", installed via trap cleanup_fn EXIT — is hand-authored independently in this PR's new cleanup_lmcache_server() (kimik3_fp4_b300_vllm_mtp.sh:126-138), in dsv4_fp4_mi355x_vllm_mtp.sh's cleanup_lmcache_server/cleanup_agentic_services, and in minimaxm3_fp4_mi355x_mtp.sh's cleanup_agentic_services (which iterates an LMCACHE_PIDS array). Several verifiers additionally found the same idiom repeated across roughly a dozen sibling scripts (minimaxm3's B200/B300 MTP variants, both qwen3.5 MTP variants, kimik3_fp4_mi355x_mtp.sh, and the deprecated kimik2.5_fp4_b200.sh).\n\nThe underlying primitive, stop_background_process_tree, is already centralized in benchmark_lib.sh:284 and every one of these scripts correctly calls into it — so this isn't a case of unshared kill logic. What's duplicated is the thin wrapper around it: capturing the pre-trap exit code, disarming the trap so it doesn't re-fire, relaxing set -e so the cleanup itself can't mask the original failure, and re-exiting with the preserved code. That's exactly the kind of small, easy-to-get-subtly-wrong boilerplate (e.g. forgetting set +e, or exiting 0 instead of the captured code) that benchmark_lib.sh exists to hold once. A helper such as register_cleanup_pid "$PID" "label" that appends to a global PID/label list and lazily installs a single shared EXIT trap would let each call site shrink to one line, while still accommodating the single-PID (this PR, dsv4's LMCache arm), multi-PID (dsv4's router+server+Mooncake), and array (minimaxm3's LMCACHE_PIDS) shapes seen today.\n\nOne verifier refuted this on scope grounds: the idiom recurs in roughly a dozen sibling scripts that this PR does not touch, so consolidating it here would make this script inconsistent with its un-refactored siblings, and the maintainers' apparent choice not to abstract it despite ~12 repetitions is itself signal that the per-script copy is the accepted convention. That's a reasonable caution against silently introducing a bespoke abstraction in a one-arm PR, but it doesn't change the underlying fact being reported: the duplication is real, growing (this PR adds yet another copy), and a shared helper is a plausible, low-risk win a maintainer could pick up in one pass across benchmark_lib.sh plus the handful of call sites, rather than being blocked on this PR alone. It's included as a quality/reuse observation, not a request to refactor unrelated scripts as part of this change.\n\nNothing is functionally wrong with the block as written — the trap correctly disarms itself, preserves the exit code, and lets set +e neutralize this script's set -eo pipefail for the duration of the cleanup call. This finding does not block merge; it's a maintainability nit best addressed as a standalone benchmark_lib.sh change touched by whoever next has occasion to edit one of these cleanup blocks.
| # The LMCache arm starts a second long-lived process that must not outlive this | ||
| # job (its L1 holds the whole host-DRAM budget). The vLLM server's lifecycle is | ||
| # left exactly as it was -- the job wrapper still owns it. | ||
| LMCACHE_PID="" | ||
|
|
||
| cleanup_lmcache_server() { | ||
| local exit_code=$? | ||
| trap - EXIT | ||
| set +e | ||
| stop_background_process_tree "$LMCACHE_PID" "LMCache server" | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup_lmcache_server EXIT |
There was a problem hiding this comment.
🟡 cleanup_lmcache_server (kimik3_fp4_b300_vllm_mtp.sh:134-135) is only trapped on EXIT, unlike both sibling lmcache scripts (minimaxm3_fp4_mi355x_mtp.sh:75-77, dsv4_fp4_mi355x_vllm_mtp.sh:242-244), which additionally trap INT and TERM via trap 'exit 130' INT / trap 'exit 143' TERM. Without that, an untrapped SIGINT/SIGTERM delivered to this script's own PID skips the EXIT trap entirely, so an external cancellation could leave the LMCache MP server running and holding the whole host-DRAM budget on the node. Fix by adding the same two trap lines after trap cleanup_lmcache_server EXIT.
Extended reasoning...
The bug: kimik3_fp4_b300_vllm_mtp.sh:135 registers trap cleanup_lmcache_server EXIT and nothing else. Both existing lmcache-using siblings — minimaxm3_fp4_mi355x_mtp.sh:75-77 and dsv4_fp4_mi355x_vllm_mtp.sh:242-244 — register the identical cleanup on EXIT and additionally do trap 'exit 130' INT and trap 'exit 143' TERM. This new arm drops those last two lines.
Why it matters: in bash, a signal (SIGINT/SIGTERM) delivered directly to a script's own PID while the script is blocked on a foreground child (here, wait_for_server_ready/the vLLM server) does not reliably invoke a plain trap ... EXIT handler — the shell can terminate from the signal without ever reaching the EXIT trap. The trap 'exit N' INT TERM idiom used by both siblings exists specifically to convert the signal into a normal exit, which does run the EXIT trap and its cleanup. I reproduced this directly: a minimal bash script with an EXIT-only trap that backgrounds a child and blocks on wait , when sent SIGINT or SIGTERM targeting its own PID, left the script running (or exited without invoking cleanup) and orphaned the backgrounded child — the fix (adding the INT/TERM traps) reliably ran the cleanup in the same harness.
Why nothing else catches this: the job wrapper here only owns the vLLM server's lifecycle (per the script's own comment), not LMCache's — that's the entire reason cleanup_lmcache_server was added as a dedicated trap in this PR. There's no other supervisor watching for an orphaned LMCache process once the script itself exits abnormally via signal.
Impact: the script's own comment states the LMCache server "must not outlive this job (its L1 holds the whole host-DRAM budget)" — at this config's dram-utilization: 0.63/TP8, that's ~1,889 GB of host DRAM. An external cancellation or timeout that signals the script's PID directly (the same class of external-termination path this PR's own SKILL.md update calls out, e.g. jobs with null step conclusions from external termination) can leave that MP server running, stranding the DRAM and blocking subsequent jobs on the node.
Proof walkthrough: (1) job is cancelled/times out; the runner sends SIGTERM/SIGINT to the script's PID while it's blocked in wait_for_server_ready on the vLLM child. (2) Only an EXIT trap is registered, so the shell can exit from the signal without invoking cleanup_lmcache_server. (3) stop_background_process_tree "" ... never runs. (4) The LMCache MP server (started earlier in the script, listening on 127.0.0.1:6555/8090) keeps running, holding its --l1-size-gb $TOTAL_CPU_DRAM_GB allocation. (5) The next job scheduled on that node now competes for or is blocked by that stranded DRAM allocation.
Fix: add the same two lines the siblings already use, immediately after line 135:
trap cleanup_lmcache_server EXIT
trap 'exit 130' INT
trap 'exit 143' TERMThis is a one-line divergence from an established two-script convention with a real (if narrow — normal completion and process-group signal delivery are unaffected) operational cost, so it's a nit worth fixing to match the siblings rather than a blocker.
| lmcache) | ||
| require_agentic_kv_offload_backend lmcache | ||
|
|
||
| # LMCache's own CUDA 12.9 build, not the generic PyPI wheel -- the | ||
| # release publishes a dedicated -cu129 asset set, the CUDA counterpart | ||
| # of the -rocm assets the MI355X sister arm installs. This is the | ||
| # upstream install line, which reads in uv form as: | ||
| # | ||
| # uv pip install lmcache==v$VERSION \ | ||
| # --extra-index-url https://download.pytorch.org/whl/cu129 \ | ||
| # --find-links .../expanded_assets/v$VERSION-cu129 \ | ||
| # --index-strategy unsafe-best-match | ||
| # | ||
| # --index-strategy has no pip counterpart because it does not need one: | ||
| # pip already resolves best-match across every configured index, which | ||
| # is exactly what unsafe-best-match restores in uv. torch is an | ||
| # unpinned LMCache requirement that the image already satisfies, so | ||
| # its tested build is left alone and the cu129 torch index is only | ||
| # consulted for CUDA wheels pip would otherwise miss. | ||
| LMCACHE_VERSION="0.5.4rc2" | ||
| LMCACHE_CUDA_INDEX="https://github.com/LMCache/LMCache/releases/expanded_assets/v${LMCACHE_VERSION}-cu129" |
There was a problem hiding this comment.
🟡 The new lmcache) arm never sets PYTHONHASHSEED, unlike the vllm-simple arm in this same file (line 150, PYTHONHASHSEED=42) and the ROCm lmcache sister arm in dsv4_fp4_mi355x_vllm_mtp.sh (PYTHONHASHSEED=${PYTHONHASHSEED:-0}). This is a pre-existing convention break that hurts run-to-run reproducibility of the sweep's prefix-cache/LMCache behavior; add export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" to the lmcache) case to match its siblings.
Extended reasoning...
What's inconsistent: every other KV-offload arm in the tree pins PYTHONHASHSEED. The vllm-simple case in this exact file (line 150) sets export PYTHONHASHSEED=42 with the comment "Identical prefixes must hash to identical block keys run-to-run." The directly analogous LMCache arm on the ROCm sister script, dsv4_fp4_mi355x_vllm_mtp.sh, sets export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" inside its own lmcache) case, and the deprecated kimik2.5_fp4_b200.sh lmcache case does the same. The new lmcache) case added here (lines 159-179 and following) is the only offload arm that sets neither, so it is reproducibly out of step with every sibling recipe that does the same kind of KV offload.
Why the strongest impact claim doesn't hold up: the original finding argued that unset PYTHONHASHSEED causes identical prefixes to hash to different block keys across the 8 TP ranks within a single run, silently corrupting that run's throughput number. That mechanism was directly refuted by one verifier and walked back by two others on review: in vLLM V1, prefix-cache block-hash computation happens once in the engine-core/scheduler process, not independently per TP worker, so there's no per-rank divergence within a run regardless of the seed. Separately, LMCache's own chunk-key hashing for its MP-server protocol is not simply Python's per-interpreter hash() — if it were, the MP server and its vLLM worker clients (which the sister arm also runs as separate processes) could never agree on a key at all, making LMCache non-functional rather than merely degraded, which is not what's observed in production use of this connector. So the "corrupts this run's throughput number" framing is not substantiated.
What is real: PYTHONHASHSEED is unset in this arm, and the vllm-simple comment itself scopes the concern to reproducibility "run-to-run," not within a single run. Leaving it unset means separate invocations of this LMCache arm (e.g. a rerun after a flake, or a later comparison run) don't get the same per-process hash seed as each other or as the sibling arms, which is a real — if narrow — loss of apples-to-apples reproducibility for exactly the kind of throughput comparison this sweep exists to produce. It is also a plain inconsistency: the PR explicitly copies the ROCm sister recipe's shape (same LMCache version line, same MP-server topology, same connector) but drops this one line from it without a stated reason.
Fix: add export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" inside the new lmcache) case, matching the ROCm sister arm's default and treatment.
Concrete walkthrough: run the lmcache arm today (no PYTHONHASHSEED in the environment) — each vllm serve invocation gets Python's default randomized siphash seed for that process. Run it again tomorrow (e.g. a rerun to double-check a throughput result) — a different random seed is used. Because the seed is unset in both cases, there's no guarantee the two runs' internal hashing state lines up with each other, unlike the vllm-simple arm two cases up in the same file where PYTHONHASHSEED=42 is fixed and both runs are guaranteed identical hashing. That is a real, if minor, reproducibility gap versus the established pattern, not the "silently wrong throughput this run" bug originally claimed.
Given the refuted within-run corruption mechanism, this is best treated as a convention/reproducibility nit rather than a blocking correctness bug.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31760278319 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31760609554 |
1 participant
What
Adds a dedicated config key
kimik3-fp4-b300-vllm-agentic-dspark-lmcachewith an LMCache DRAM KV-offload arm at TP8 conc 4 / 8 / 10 / 16, on top of the unchanged DSpark MTP serving stack ofkimik3-fp4-b300-vllm-agentic-dspark(same image, script, and topology). A separate key means the changelog selects only the LMCache points — the resident and vllm-simple arms of the base key are not re-run.B300 sister of the MI355X arm in #2583; same LMCache version (0.5.4rc2), same MP-server topology, CUDA install path and NVIDIA-side chunk size.
Changes
kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.4rc2" }, conc-list: [4, 8, 10, 16], spec-decoding: mtp, at the base key'sdram-utilization: 0.63(1,889 GB aggregate at TP8). 4/8/16 land on the base key's vllm-simple ladder at its top three offload points, so LMCache is directly comparable there; 10 fills the 8->16 gap.lmcache)case arm:v0.5.4rc2-cu129release assets — the CUDA counterpart of the-rocmassets the MI355X sister arm installs), not the generic PyPI wheel, via the upstream install line with--extra-index-url https://download.pytorch.org/whl/cu129.--index-strategy unsafe-best-matchhas no pip counterpart because it needs none: pip already resolves best-match across every configured index, which is exactly what that flag restores in uv.torchis an unpinned LMCache requirement the image already satisfies, so its tested build is left alone. A fail-fast import check follows.--chunk-size 768(that recipe's CUDA-path value),--separate-object-groups(one object group per sliding-window size for the hybrid KDA/MLA layout, which has more than one KV-cache group under MTP),--enable-extra-logging,--max-cpu-workers 8 --max-gpu-workers 1,--l1-size-gb $TOTAL_CPU_DRAM_GB, LRU eviction,--shm-name ""so L1 lives in ordinary process memory rather than being capped by /dev/shm.LMCacheMPConnector(lmcache.mp.port), keeping the DSpark--speculative-configuntouched.lmcachetoo.TOTAL_CPU_DRAM_GBverbatim per the agentic README.Note on
--chunk-size: the connector requires the chunk to be a multiple of every engine KV group'stokens_per_block. 768 is the published CUDA-path value; the ROCm sister arm needs 3072 because its hybrid layout registers 1536-token attention groups and a 3072-token KDA state group. If the B300 server log reports a mismatch, this is the one value to raise — the script comment says so and names the log lines to read.Validation
process_changelog.pyrun exactly as CI does (base = main) emits a 4-row matrix, nothing else:generate_sweep_configs.py test-configpasses for both the new key and the (unchanged) base key;bash -npasses on the modified script;utils/changelog_gate_tests/test_validate_perf_changelog.pyandutils/test_process_changelog.pypass (35 tests).