kimik3-fp4-mi355x-vllm-agentic-mtp: add LMCache DRAM KV-offload arm - #2583
kimik3-fp4-mi355x-vllm-agentic-mtp: add LMCache DRAM KV-offload arm#2583sammshen wants to merge 13 commits into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
Add an lmcache kv-offload-backend point at TP8 conc 10 on top of the existing DSpark MTP serving stack, mirroring the vllm-simple offload arm for a direct backend comparison. The benchmark script gains an lmcache case arm that installs the LMCache 0.5.4rc1 ROCm wheel (torch/ROCm stack untouched), starts one MP server per the Kimi-K3 recipe (chunk size 768 = K3 unified block size at 8 GPUs, --separate-object-groups for the hybrid KDA/MLA two-group KV layout, --enable-extra-logging, --max-cpu-workers 8 --max-gpu-workers 1), and wires vLLM to it via LMCacheMPConnector.
f206c4b to
328836b
Compare
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
A separate kimik3-fp4-mi355x-vllm-agentic-mtp-lmcache key lets the changelog select only the LMCache points instead of re-running the resident and vllm-simple arms of the base key. The base key returns to its upstream shape.
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
ApostaC
added
agentx
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
The LMCache MP server's L1 lives in /dev/shm and the script rejects budgets above 90% of free shm. mi355x-amds nodes mount ~1.5 TB of shm (cap ~1360 GB), so 0.50's 1499 GB budget failed the check in run 31644286169. 0.40 generates ~1199 GB, which fits with margin.
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
vLLM sizes the K3 unified attention block to 1536 tokens on the MI355X fp8-KV TRITON_MLA path (attention page >= mamba page), and the MP connector asserts chunk %% block == 0, so the recipe's CUDA-path 768 fails engine init (run 31644990546).
The connector requires the chunk to be a multiple of every engine KV group's tokens_per_block. On this stack the hybrid layout registers attention groups at 1536 and a KDA state group at 3072 (run 31645828378), so 1536 fails registration; 3072 is the minimum valid chunk.
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
Auto mode loads both transfer paths; pin server-driven STORE/RETRIEVE (as the MiniMax-M3 arm does) so the benchmark measures one deterministic path. The L1 stays shm-backed either way, so the /dev/shm capacity check still applies.
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
The default 300s read-lock TTL expires under a single GPU worker serializing huge K3 transfers: run 31648224111 logged 57k finish-read-on-non-read-locked-key warnings starting exactly at warmup+300s, followed by a GPU illegal-access crash mid-profile. Match the MiniMax-M3 arm's 7200s read TTL.
This reverts commit 20b4fda.
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
/stage-results |
|
Superseded by #2598, which carries the same commits on a branch in this repo rather than a fork. Same-repo PRs get secrets, so |
3 participants
What
Adds a dedicated config key
kimik3-fp4-mi355x-vllm-agentic-mtp-lmcachewith an LMCache DRAM KV-offload arm at TP8 conc 4 / 8 / 16, on top of the unchanged DSpark MTP serving stack ofkimik3-fp4-mi355x-vllm-agentic-mtp(same image, script, and topology). A separate key means the changelog selects only the LMCache points — the resident and vllm-simple arms of the base key are not re-run.Changes
kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.4rc1" }, conc-list: [4, 8, 16], spec-decoding: mtp.lmcache)case arm, modeled on the MiniMax-M3 lmcache arm:--no-depsplus its missing runtime deps, leaving the image's torch/ROCm stack untouched;--chunk-size 768(the K3 unified block size at 8 GPUs),--separate-object-groups(one object group per sliding-window size for the hybrid KDA/MLA layout, which has two KV-cache groups under MTP),--enable-extra-logging,--max-cpu-workers 8 --max-gpu-workers 1,--l1-size-gb $TOTAL_CPU_DRAM_GB, LRU eviction;LMCacheMPConnector(lmcache.mp.port), keeping the DSparkSPEC_ARGSuntouched;TOTAL_CPU_DRAM_GBverbatim per the agentic README, with an early /dev/shm capacity check (same as the MiniMax-M3 arm).LMCache + DSpark MTP + the rest of this recipe's feature set has been compatibility-tested;
--separate-object-groupsis the supported handling for the multi-KV-group layout that previously blocked LMCache-under-MTP arms.Validation
process_changelog.pyrun exactly as CI does (base = main) emits a 4-row matrix, nothing else:generate_sweep_configs.py test-configpasses for both the new key and the (unchanged) base key;bash -npasses on the modified script.