vulkan: use small Lightning Indexer CM for batches 4-15 - #6
Conversation
|
Hey @pepuscz very much appreciated for your work on this, the correctness numbers came our great, It did lead me down the next path of optimisations, ive got a new set of commits carried in the below branch : https://github.com/Nathanw1014/llama.cpp/tree/strix-halo-vulkan-indexer-gen2 https://github.com/Nathanw1014/strix-halo-llamacpp/releases/tag/dev-20260826-7c025e5 Your work was what let do these changes, id be keen for you to test it and see what further performance gains can be had. If you woul dlike to test, can you please test with and without the spec decoding:
|
|
Independent confirmation on a second rig — thanks @pepuscz, this patch (via the v0.7.0 release) checks out end to end on my machine. Rig: ASUS Flow Z13 (Ryzen AI MAX+ 395 / 8060S, 128 GiB), CachyOS, official v0.7.0 portable, UD-IQ3_XXS + DSpark drafter (Q2_K_S), One-variable A/B on the same v0.7.0 binary using the
+4.2% at 41k — right on your curve at an earlier depth than your first table point (+4.16% at 60k), with run-to-run variance of roughly ±1.5 t/s. End-to-end vs the v0.6.3-beta3-class build (same session, same config): prefill +8.4% at pp2048 (304.5 vs 281.0 t/s) and +8.1% at pp4096; decode flat from 9k to 41k context (25.2 → 25.3 t/s) where the old build slides 26.0 → 24.7. Everything I have is consistent with the depth-scaling model in the release notes. Your retrieval protocol replicated at 59k tokens (synthetic filler, five keys at 2/20/50/80/98% depth, temp 0 / top-k 1 / top-p 1, 256-token cap, cold KV, thinking off): 29.3 t/s on f16 KV (84.4% draft acceptance, all keys recovered byte-exact); 26.9 t/s on q8_0 KV. The absolute gap to your 36.5 t/s @ 120 W is fully accounted for by the power cap (+~1 t/s per 10 W on this SoC), your CPU 2.0 GHz/boost-off policy freeing package power for the GPU, and q8_0 KV — which your stack needs for the 491k allocation but measures ~9% slower than f16 at mid depths on my box. Full write-up of that replication and the config sweeps: our r/LocalLLaMA guide — DeepSeek V4 Flash 0731 at 27+ t/s decode on Strix Halo (v0.7.0 A/B and the retrieval replication are in the Aug 27 edit note). One data point that may be useful for the docs: on this fork, |
Independent v0.7.0 qualification resultWe qualified the unmodified official strix-halo-llamacpp v0.7.0 portable release, source 95c828e. No local source patch or Lightning Indexer override was applied. The matched profile used UD-IQ3_XXS, the Q2_K/Q8_0 DSpark drafter, full target and draft GPU offload (-ngl 999, -ngld 999), flash attention on (-fa on), n_max=4, q8_0 K/V, one slot, batch 2048, and microbatch 1024. The comparison baseline was v0.6.6 source 7b6c613 plus the 4-15 small-CM dispatch from this PR, with otherwise identical model and runtime parameters. Each row reused the same frozen prompt hash and allocation with cold KV state, thinking off, temperature 0, top-k 1, top-p 1, a 256-token output cap, and five byte-exact retrieval keys.
Units are tokens/s. All eleven v0.7.0 rows recovered 5/5 keys. Geometric-mean improvements were 4.10% for prefill and 7.74% for decode. For the requested -c 524288 speculative A/B at 491,520 actual prompt tokens, v0.7.0 with DSpark reached 163.226 prefill / 25.021 decode tokens/s; the same binary and configuration without the drafter reached 162.214 / 11.882. DSpark therefore delivered 2.1058x decode throughput, with 51 of 52 drafted tokens accepted. The reduced public dataset, production manifest, and chart are published here: https://github.com/pepuscz/strix-halo-deepseek-v4-flash/releases/tag/v1.4.0 |
…ecode CM shader Builds directly on pepuscz's PR Nathanw1014#6 and issue ggml-org#10. Their per-kernel table - 43.8 us scalar against 27.9 us small CM per 1k scanned rows at batch 5, and the same ratio at every depth - is what showed the indexer's cost is a per-tile constant rather than anything to do with tokens or bytes. Once the cost is per tile, the thing to minimise is the tile count, and that is what this changes. The finding is downstream of their work; only the choice of shader differs. The decode CM shader puts 16 HEADS in the coopmat N dimension and dispatches one workgroup per token, so it issues 4*n_batch tiles per 16 KV rows. That is the arithmetic minimum - (64 heads x n_batch tokens) / 16 columns - because 64 heads fill its 16 columns exactly, with no remainder at any batch. The small CM shader puts 16 TOKENS in N and pays a flat 64 tiles however small the batch is, so at batch 5 eleven of its sixteen columns are padding. Measured cost tracks tile count: at kv=131584 batch 1 costs 1.95 us per 1k scanned rows and batch 5 costs 26.28, i.e. 13.5x for 5x the work, and both shaders sit at 6.6-7.8 ns per tile. The decode CM shader body has no n_batch == 1 assumption - token is gl_WorkGroupID.y and it indexes q/w/mask/dst by it - so the old gate was an artefact of where it was written. gfx1151, kv=131584 (526k source tokens), us/run, shipped vs this: batch 2 2335.6 -> 419.0 (5.6x, was on the scalar path) batch 3 3492.3 -> 631.8 (5.5x, was on the scalar path) batch 4 3267.9 -> 840.8 (3.9x) batch 5 3465.6 -> 1119.4 (3.1x) <- DSpark n-max 4 verify shape batch 8 3704.9 -> 1764.1 (2.1x) batch 15 4467.9 -> 3231.6 (1.4x) Batch 16 and 32 are unchanged, which confirms the arms are isolated. PR Nathanw1014#6's route is kept as the opt-out arm rather than deleted, and is promoted from opt-in to default-on so that one variable is enough to reach it: default decode CM for the whole 2-15 window ..._DECODE_CM_BATCH=0 small CM for 4-15, scalar for 2-3 (PR Nathanw1014#6) ..._DECODE_CM_BATCH=0 SMALL_CM=0 scalar for 2-15 (pre-PR Nathanw1014#6 baseline) Verified all three route as documented and pass 29/29. Keeping it is not just courtesy: the tile-count argument is hardware independent, but decode CM re-reads the K tile once per token and that part is bandwidth dependent, so the crossover need not sit in the same place on other devices, and these numbers are from one gfx1151 box. A (head, token) packing shader was also tried and is strictly worse - it reaches 4b tiles only when the batch divides 16, and it divides the workgroup count by the batch. Kept out of tree. Numerics: decode CM sums the 64 heads in tiles of 16 rather than one at a time, so f32 accumulation order differs from the small CM path. Cleared by KLD A/B on trunc10 (-c 8192, -b/-ub small so every indexer dispatch routes through the window, 8 chunks wikitext-2, same binary, only the env var differing): -ub 8 mean KLD 0.000000, max 6.0e-5, RMS dp 0.000%, same-top 99.994%, PPL 10507558.3156 identical in both arms -ub 5 mean KLD 0.000000, max 8.8e-4 but 99.9% 4.9e-5 (one tail event, no argmax flip), RMS dp 0.000%, same-top 100.000% Same class as PR Nathanw1014#6's own measured numbers (max 5.5e-5, same-top 100.000%) and 45x tighter than the FA_WAVE32 change already shipped. Adds the eval coverage the 2-15 window never had (batches 4/8/15 at kv=256) and the perf grid these numbers came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3 participants
Overview
DSpark speculative verification can send 2–15 target tokens to the Lightning Indexer in one step. In measurements using Strix Halo llama.cpp v0.6.6, the cooperative-matrix shader lightning_indexer_cm_small_f16 was faster than the scalar lightning_indexer_f16 indexer for batches 4–15. The patch therefore keeps batches 2–3 on the scalar path and routes only batches 4–15 through small-CM.
Additional information
lightning_indexer_cm_small_f16is selected for DSpark verifier batches 4–15using:
v0.6.6 versus 4–15 small-CM patch
Benchmarked on the same dataset as Strix Halo DeepSeek V4 Flash repository.
Requirements