Add sm121 (GB10) to the wide_n_simt 4-bit GEMM dispatch rule - #2039
Open
yashb98 wants to merge 1 commit into
Open
Add sm121 (GB10) to the wide_n_simt 4-bit GEMM dispatch rule#2039yashb98 wants to merge 1 commit into
yashb98 wants to merge 1 commit into
Conversation
wide_n_simt already routes sm120 with >=48 SMs to the SIMT kernel at M<=6 with wide N. GB10 (sm121, 48 SMs) clears the same SM threshold but was excluded by the minor-version check, so it fell through to MMA. Measured on GB10: all 8 rerouted cells are faster with SIMT, 1.03x to 1.86x. Enumeration over 78,936 and 177,540 dispatch decisions confirms the change is confined to cc_maj==12 && cc_min==1 at M<=6, False->True only, with no other architecture affected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds an
sm121disjunct to thewide_n_simtrule incsrc/gemm_4bit.cu's 4-bit GEMM dispatch,mirroring the existing
sm120clause. This is the follow-up matthewdouglas asked for on #2030:"a simple change to the wide_n_simt rule" to cover sm121.
One hunk, three insertions, two deletions.
highbw_gddris untouched.Why
wide_n_simt's sm120 branch already gates oncc_maj == 12 && cc_min == 0 && num_sms >= 48.GB10 (sm121, 48 SMs) already clears that SM-count threshold on its own; only the minor-version
check excludes it. Same shape of gap #2030 closed one layer up in
_gemm_4bit_use_custom_cuda: a working sm120 branch next to a threshold sm121 already clears,with nothing routing it there.
Safety
Checkable, not argued. Two independent enumerations of the stock vs. patched dispatch decision,
by two independently-authored reimplementations of the boolean, over two non-overlapping grids:
All changes in both grids land on
cc_maj == 12 && cc_min == 1exactly, atM <= 6exactly, andthe boundary sweep confirms the SM threshold is exactly
>= 48(nothing below flips) and theminor version is exactly
1(the two decoy profiles never flip). A row-by-row replay of thesecond implementation against every one of the first implementation's 78,936 rows, and against
all 260 real-hardware rows from my earlier SIMT/MMA measurement, produced zero mismatches. Since
both grids are exhaustive over their enumerated space rather than sampled, this is a proof of
no-regression for those shapes, not evidence of one.
The diff was applied and re-applied cleanly in two separate, independent worktrees; the resulting
compiled
libbitsandbytes_cuda130.sodiffers by hash from stock (confirms the patch reaches thebinary), and its
CUDA Capabilities Selected: 121build log confirms it targets the right arch.Tests
Ran the 8 cells the enumeration flips False->True plus 5 control cells outside the patch's scope,
stock vs. patched, on real GB10 hardware (8 warmup + 25 reps, medians):
All 8 flipped cells are faster patched, 1.03x-1.86x.

The tightest-margin cell (M=4, N=8960, K=1536, 1.20x) was rechecked 3 independent times per
variant and reproduced tightly every time (stock 66.4/66.6/66.5us, patched 55.1/55.3/55.2us).
Of the 5 control cells (all M>6, outside the patch's scope), 4 are within 5% (noise). One is not
and I'm flagging it rather than dropping it:
float16, M=64, N=8192, K=8192read stock 447.1usvs. patched 378.5us (18% apparent patched speedup) on a shape where both variants run the
identical MMA path with no reason to differ. Rechecked 3x per variant: stock 430.1/431.4/432.1us,
patched 442.9/440.8/442.3us. The gap does not reproduce and reverses direction (patched ~2.5%
slower on recheck), so this was cold-start clock/thermal noise between two separate process
launches, not a patch effect.
Both rechecks side by side, in-scope cell on the left and the out-of-scope outlier on the right:
Raw per-cell JSON for every figure and table above, plus the harness, are available if useful.
Also ran the same functional/correctness suite the base PR ran, stock vs. patched, same two
detached worktrees:
test_functional.py -k 4bittest_linear4bit.py+test_autograd.pySame pass/fail/skip counts on both variants; the sorted (outcome, test id) sets are identical too
(0 differences across 2840 + 2157 pairs), so nothing merely balanced out in aggregate: no test
flipped outcome either direction. The one failure,
test_fsdp_state_dict_save_4bit, is the samepre-existing failure #2030's own PR body documented, with an identical root cause in both logs
(a
torchrun-launched subprocess not inheriting the worktree'ssys.path, unrelated to thispatch).
Honest limits
Single GB10, single machine. No second card to confirm these ratios, or the noise floor the
control-cell recheck surfaced, generalize beyond this box.
6 cells stay mispicked, and this patch structurally can't reach them. M in {4,6,8} x
{bfloat16,float16} at N=2048,K=2048: stock already (wrongly) picks SIMT there via the
undersubscribedclause, and MMA measures faster. Fixing that needs aTrue-removing change toan existing clause for
cc_maj==12 && cc_min==1, not anotherwide_n_simtdisjunct, so it is outof scope here, not touched, not proposed in this PR.
highbw_gddruntouched, and deliberately so. The same root-cause note that flaggedwide_n_simtalso namedhighbw_gddr, so to pre-empt the obvious question: atM == 4thatclause is broader than this one.
use_simttests it as(M == 4 && highbw_gddr)with no waveterm at all, whereas
wide_n_simtadditionally requiresmma_blocks >= num_sms. Extendinghighbw_gddrto sm121 would therefore also reroute sub-waveM=4shapes, which is a region Ihave not measured and which this PR's enumeration says nothing about. Whether GB10 wants SIMT
there too is a real question, just a different one, so I scoped it out rather than bundling an
unmeasured change in behind a measured one. Happy to go measure it as a follow-up if you want it.
Environment: GB10 (DGX Spark), sm_121, driver 580.142, torch 2.12.1+cu130, CUDA 13.0.88.