[AMD] [AGENTX] Qwen3.5 FP4 MI355X SGLang Agentic MTP / Qwen3.5 FP4 MI355X SGLang Agentic 数据集 MTP 基准配置 - #2605
[AMD] [AGENTX] Qwen3.5 FP4 MI355X SGLang Agentic MTP / Qwen3.5 FP4 MI355X SGLang Agentic 数据集 MTP 基准配置#2605ChangLiu0709 wants to merge 7 commits into
Conversation
Add baseline (no speculative decoding) configuration for Qwen3.5-397B MXFP4 on MI355X with AgentX trace replay. Mirrors the MTP variant's parallelism/concurrency matrix but omits EAGLE draft generation, serving as the non-speculative baseline for comparison. - configs/amd-master.yaml: new qwen3.5-fp4-mi355x-sglang-agentic entry - benchmarks: qwen3.5_fp4_mi355x_sglang.sh (non-MTP inner bench script) Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Not approving: this PR adds a new recipe without a perf-changelog.yaml entry, which AGENTS.md calls a non-negotiable invariant for every recipe addition. See the inline comment for details.
Extended reasoning...
Overview
The PR adds a new non-MTP agentic-coding config entry qwen3.5-fp4-mi355x-sglang-agentic to configs/amd-master.yaml plus a matching new benchmark script benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh, mirroring the existing -mtp sibling minus the EAGLE speculative-decoding flags.
Security risks
None. This is a benchmark configuration and shell script for launching an SGLang server in a controlled cluster environment; no new attack surface, credentials, or user input handling is introduced.
Level of scrutiny
This would normally be a low-scrutiny, mechanical recipe addition following an established pattern (the diff closely mirrors the neighboring -mtp entry). However, I independently verified the tail of perf-changelog.yaml and confirmed it has no entry for this PR or the new qwen3.5-fp4-mi355x-sglang-agentic config key — the last entries are for PR #2577/#2578. AGENTS.md states this changelog entry is a non-negotiable invariant for every recipe addition, so I'm treating this as blocking rather than a nit, despite a finder/verifier pair in this run disagreeing on whether it's a real issue.
Other factors
The bilingual title/description convention (also flagged inline) is a process nit and wouldn't alone block approval, but combined with the missing non-negotiable changelog entry, a human should confirm before merge.
| qwen3.5-fp4-mi355x-sglang-agentic: | ||
| image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260811 | ||
| model: amd/Qwen3.5-397B-A17B-MXFP4 | ||
| model-prefix: qwen3.5 | ||
| runner: cluster:mi355x-amds | ||
| precision: fp4 | ||
| framework: sglang | ||
| multinode: false | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| - { tp: 2, ep: 2, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20] } | ||
| - { tp: 4, ep: 1, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32, 40] } | ||
|
|
There was a problem hiding this comment.
🔴 This PR adds a brand-new recipe (qwen3.5-fp4-mi355x-sglang-agentic in configs/amd-master.yaml plus a new benchmark script) but does not append the required entry to perf-changelog.yaml. Per AGENTS.md's non-negotiable benchmark invariant, every recipe addition needs a new append-only tail entry — please add one describing this config before merging.
Extended reasoning...
AGENTS.md, under Non-negotiable benchmark invariants, states verbatim: 'Every change that can affect benchmark performance and every recipe addition or modification requires a new perf-changelog.yaml entry. The file is append-only and byte-sensitive... append only at the tail.' This is not a stylistic suggestion — it's called out as a non-negotiable invariant that gates recipe changes in this repository.
This PR introduces a brand-new top-level config key, qwen3.5-fp4-mi355x-sglang-agentic, in configs/amd-master.yaml (lines 377-391), along with its own new benchmark script benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh. This is unambiguously a 'recipe addition' as defined by the invariant — it's a new model/hardware/framework combination (Qwen3.5-397B MXFP4 baseline, non-MTP, on MI355X via SGLang) with its own search space (TP2/EP2 and TP4/EP1 arms) and its own image pin.
The PR diff touches exactly two files: the new benchmark script and configs/amd-master.yaml. perf-changelog.yaml is not part of the diff at all. I confirmed this directly by reading the tail of the current perf-changelog.yaml in the repo: the last two entries are for PR #2578 (dsv4-fp4-b200-sglang-agentic-hicache-mtp) and PR #2577 (dsv4-fp4-b300-sglang-agentic-hicache-mtp) — there is no entry referencing PR #2605 or the qwen3.5-fp4-mi355x-sglang-agentic config key anywhere in the file.
Step-by-step proof:
- Read AGENTS.md — non-negotiable invariant requires a perf-changelog.yaml entry for every recipe addition, appended at the tail.
- Read the PR diff — a new config key
qwen3.5-fp4-mi355x-sglang-agenticis added to configs/amd-master.yaml, and a new script implements it. This satisfies the definition of 'recipe addition.' - Enumerate the PR's changed files — only 2 files: the script and the config. perf-changelog.yaml is absent.
- Read the tail of perf-changelog.yaml on the base branch — ends with the PR Refresh DeepSeek-V4 B300 SGLang AgentX MTP #2577/Refresh DeepSeek-V4 B200 SGLang AgentX MTP #2578 entries, with no entry for this recipe or PR [AMD] [AGENTX] Qwen3.5 FP4 MI355X SGLang Agentic MTP / Qwen3.5 FP4 MI355X SGLang Agentic 数据集 MTP 基准配置 #2605.
- Conclusion: the required changelog entry is missing.
Nothing in the diff suggests this recipe is exempt (e.g. it isn't a config-only tweak to an existing key — it's a wholly new entry with a companion script). The fix is straightforward: append a new entry at the tail of perf-changelog.yaml with config-keys: [qwen3.5-fp4-mi355x-sglang-agentic], scenario-type: [agentic-coding], a short description of the TP2/EP2 and TP4/EP1 non-MTP baseline, and the PR link for #2605 — following the exact format of the adjacent entries already in the file.
ChangLiu0709
changed the title
Add spec-decoding: mtp to both search-space arms and update the benchmark script with EAGLE flags (num-steps=3, draft-tokens=4, topk=1) and golden simulated acceptance length (SGLANG_SIMULATE_ACC_LEN=3.39). Co-authored-by: Cursor <cursoragent@cursor.com>
ChangLiu0709
changed the title
Co-authored-by: Cursor <cursoragent@cursor.com>
Use conc-list [1, 2, 4, 8, 10, 12, 16] and kv-offloading: dram with hicache backend, matching the GLM-5.2 MTP configuration pattern. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
ChangLiu0709
changed the title
ChangLiu0709
changed the title
|
Claude finished @ChangLiu0709's task in 5m 15s —— View job Review — 3 blocking issues 🔴 / 审阅结果 — 3 个阻断性问题
Not approving. Two new blocking correctness issues (inline comments), plus one carried over from the previous review:
Suggested consolidation: apply the steps-5/draft-6/AL-3.61 changes directly to Resolved since last review: the PR description now includes the Simplified Chinese section (bilingual requirement satisfied). Image is public on Docker Hub, server launch formatting and 中文摘要：不予批准。① 新脚本永远不会被执行 — launcher 按 |
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| - { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 10, 12, 16] } |
There was a problem hiding this comment.
🔴 BLOCKING: This new config is an exact functional duplicate of qwen3.5-fp4-mi355x-sglang-agentic-mtp (line 403), and the new script benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh is dead code that will never run.
Why it matters: The launcher derives the script name from SPEC_DECODING, not the config key — runners/launch_mi355x-amds.sh:260 sets SPEC_SUFFIX=_mtp whenever spec-decoding: mtp, and lines 307–314 then resolve benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh, which exists and wins. Since this arm sets spec-decoding: mtp, this config runs the old _mtp.sh script (num-steps=3, draft-tokens=4, SGLANG_SIMULATE_ACC_LEN=3.39) — the advertised num-steps=5 / draft-tokens=6 / AL=3.61 settings live only in the unreachable new script. Net effect: two config keys run the byte-identical benchmark (same image, model, TP4/EP4, HiCache, conc-list, same script), doubling GPU sweep time and publishing duplicate results, while none of the PR's stated MTP changes take effect.
Fix: Apply the intended changes (num-steps 5, draft-tokens 6, SGLANG_SIMULATE_ACC_LEN=3.61) directly to benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh, and delete this duplicate config key plus the new qwen3.5_fp4_mi355x_sglang.sh script. (Per MODELS.md, agentic recipes ship spec-decode-only, so a separate non--mtp agentic key isn't needed — and a key named -agentic whose only arm is MTP is misleading either way.)
| search-space: | ||
| - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20] } | ||
| - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32, 40] } | ||
| - { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 10, 12, 16], spec-decoding: mtp } |
There was a problem hiding this comment.
🔴 BLOCKING: Both arms now declare kv-offloading: dram with the HiCache backend, but the script that actually runs (benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh — see routing comment above; the new qwen3.5_fp4_mi355x_sglang.sh has the same gap) contains no HiCache handling at all: no agentic_kv_offload_enabled / require_agentic_kv_offload_backend hicache check and no --enable-hierarchical-cache / --hicache-* server flags.
Why it matters: benchmark_lib.sh only validates that KV_OFFLOADING / KV_OFFLOAD_BACKEND / TOTAL_CPU_DRAM_GB are set (benchmarks/benchmark_lib.sh:78-104); it does not add any server flags. So the sweep will pass validation and launch SGLang without a host DRAM KV tier, while the results are labeled and published as HiCache DRAM offload — mislabeled benchmark data. Compare the GLM-5.2 recipe this PR says it mirrors: benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh:86-149 builds CACHE_ARGS (--enable-hierarchical-cache --hicache-ratio ... --hicache-write-policy ... --hicache-io-backend ... --hicache-mem-layout ...) gated on agentic_kv_offload_enabled and passes them to sglang.launch_server.
Fix: Add an equivalent CACHE_ARGS block (gated on agentic_kv_offload_enabled / require_agentic_kv_offload_backend hicache) to the script this config actually routes to, and include "${CACHE_ARGS[@]}" in SGLANG_CMD — or set kv-offloading: none if HiCache wasn't actually part of the tuned/validated runs.
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| - { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 10, 12, 16] } |
There was a problem hiding this comment.
🔴 Both new/updated search-space arms in configs/amd-master.yaml (qwen3.5-fp4-mi355x-sglang-agentic line 389 and qwen3.5-fp4-mi355x-sglang-agentic-mtp line 401) set spec-decoding: mtp, which makes runners/launch_mi355x-amds.sh route both to the pre-existing qwen3.5_fp4_mi355x_sglang_mtp.sh (num-steps=3, draft-tokens=4, acc-len=3.39) instead of the new qwen3.5_fp4_mi355x_sglang.sh this PR adds (num-steps=5, draft-tokens=6, acc-len=3.61, HiCache). The new script's suffix-less filename is unreachable dead code under this launcher convention; rename it with a _mtp suffix (replacing the stale script) so both recipes actually pick up the intended EAGLE settings.
Extended reasoning...
The bug. runners/launch_mi355x-amds.sh resolves which benchmark script to run purely from a filename convention: SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" ]] && printf '_mtp' || printf '') (line 260), then SCRIPT_FW="benchmarks/single_node/${SCENARIO_SUBDIR}${SCRIPT_BASE}_${FRAMEWORK}${SPEC_SUFFIX}.sh" (line 307-308), preferring SCRIPT_FW whenever it exists on disk (line 310). SCRIPT_BASE derives from EXP_NAME%%_*, which for both Qwen3.5 agentic configs is qwen3.5, giving SCRIPT_BASE=qwen3.5_fp4_mi355x.
Both arms this PR touches — the brand-new qwen3.5-fp4-mi355x-sglang-agentic (line 389) and the rewritten qwen3.5-fp4-mi355x-sglang-agentic-mtp (line 401) — set spec-decoding: mtp on their sole search-space entry. That forces SPEC_SUFFIX=_mtp for both, so SCRIPT_FW resolves to benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh in every case. I confirmed on disk that this file already exists and is untouched by this PR: it has --speculative-num-steps 3, --speculative-num-draft-tokens 4, and SGLANG_SIMULATE_ACC_LEN=3.39 (lines 108/110/79 of that file).
The new artifact never runs. This PR's actual new file, benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh (no _mtp suffix), contains the intended settings — --speculative-num-steps 5, --speculative-num-draft-tokens 6, SGLANG_SIMULATE_ACC_LEN=3.61 — plus the HiCache KV-offload wiring described in the PR summary. Per the launcher's naming convention, that file is only ever selected when SPEC_DECODING != mtp. Neither arm in either config satisfies that condition (both explicitly set spec-decoding: mtp), so the new script is unreachable dead code — it will sit in the repo but never execute in a real sweep.
Step-by-step proof:
configs/amd-master.yaml:389(newqwen3.5-fp4-mi355x-sglang-agentic): search-space arm is{ tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [...] }.generate_sweep_configs.pybuildsEXP_NAMEstarting with the model codeqwen3.5and exportsSPEC_DECODING=mtpfor this arm.launch_mi355x-amds.sh:260:SPEC_DECODING=="mtp"→SPEC_SUFFIX=_mtp.launch_mi355x-amds.sh:307-308:SCRIPT_BASE=qwen3.5_fp4_mi355x;SCRIPT_FW=benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh.- That file exists on disk (pre-existing, unmodified by this PR) → line 310 picks it as
BENCHMARK_SCRIPT. - The new file
qwen3.5_fp4_mi355x_sglang.sh(no_mtp) is only reachable viaSCRIPT_FWwhenSPEC_SUFFIXis empty, i.e.SPEC_DECODING != mtp— never true for either config's arm. - Result: both
qwen3.5-fp4-mi355x-sglang-agenticandqwen3.5-fp4-mi355x-sglang-agentic-mtpexecute the stale script with num-steps=3/draft-tokens=4/acc-len=3.39, not the PR's intended num-steps=5/draft-tokens=6/acc-len=3.61 with HiCache.
Impact. Every sweep run under either config key silently measures the old, unrelated speculative-decoding configuration rather than what the PR claims to add — acceptance-length and throughput numbers will not reflect the EAGLE MTP tuning or HiCache offloading the PR is meant to introduce, and the committed new script is permanently unreachable dead code as written.
Fix. Rename benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh to qwen3.5_fp4_mi355x_sglang_mtp.sh (replacing the stale pre-existing file with the same name), so the _mtp suffix the launcher expects actually resolves to the intended script. Alternatively, drop spec-decoding: mtp from both arms if a non-suffixed script were truly the goal, but that contradicts the PR description's explicit MTP intent, so the rename is the correct fix.
…55x-sglang-agentic
Append required perf-changelog.yaml entry for the new agentic-coding config and the consolidated agentic-mtp search space update. Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
Changes
New config:
qwen3.5-fp4-mi355x-sglang-agenticlmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260811amd/Qwen3.5-397B-A17B-MXFP4kv-offloading: dram,kv-offload-backend: { name: hicache })match-expectedmethod, non-eval mode)[1, 2, 4, 8, 10, 12, 16]Files changed
configs/amd-master.yaml— newqwen3.5-fp4-mi355x-sglang-agenticentry; updatedqwen3.5-fp4-mi355x-sglang-agentic-mtpsearch spacebenchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang.sh— inner bench script with EAGLE MTP, HiCache, and simulated acceptanceAuthors
@ChangLiu0709
@chunfangamd
中文说明
为 Qwen3.5-397B-A17B MXFP4 模型在 AMD MI355X 上新增代理编码（AgentX）基准配置：
qwen3.5-fp4-mi355x-sglang-agentic配置，使用 SGLang 原生 EAGLE MTP 投机解码（num-steps=5, draft-tokens=6, topk=1）qwen3.5-fp4-mi355x-sglang-agentic-mtp，从双臂（TP2/EP2 + TP4/EP1）合并为单臂 TP4/EP4 + HiCache 配置Made with Cursor