Skip to content

Add sm121 (GB10) to the wide_n_simt 4-bit GEMM dispatch rule - #2039

Open
yashb98 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
yashb98:wide-n-simt-sm121
Open

Add sm121 (GB10) to the wide_n_simt 4-bit GEMM dispatch rule#2039
yashb98 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
yashb98:wide-n-simt-sm121

Conversation

@yashb98

@yashb98 yashb98 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

What this does

Adds an sm121 disjunct to the wide_n_simt rule in csrc/gemm_4bit.cu's 4-bit GEMM dispatch,
mirroring the existing sm120 clause. This is the follow-up matthewdouglas asked for on #2030:
"a simple change to the wide_n_simt rule" to cover sm121.

-    // sm89 (>=60 SMs) and sm120 (>=48 SMs): at M<=6 with wide N, SIMT saturates
+    // sm89 (>=60 SMs) and sm120/sm121 (>=48 SMs): at M<=6 with wide N, SIMT saturates
     // bandwidth more efficiently than blocked MMA.
     const bool wide_n_simt =
         M <= 6 && mma_blocks >= num_sms &&
-        ((cc_maj == 8 && cc_min == 9 && num_sms >= 60) || (cc_maj == 12 && cc_min == 0 && num_sms >= 48));
+        ((cc_maj == 8 && cc_min == 9 && num_sms >= 60) || (cc_maj == 12 && cc_min == 0 && num_sms >= 48) ||
+         (cc_maj == 12 && cc_min == 1 && num_sms >= 48));

One hunk, three insertions, two deletions. highbw_gddr is untouched.

Why

wide_n_simt's sm120 branch already gates on cc_maj == 12 && cc_min == 0 && num_sms >= 48.
GB10 (sm121, 48 SMs) already clears that SM-count threshold on its own; only the minor-version
check excludes it. Same shape of gap #2030 closed one layer up in
_gemm_4bit_use_custom_cuda: a working sm120 branch next to a threshold sm121 already clears,
with nothing routing it there.

Safety

Checkable, not argued. Two independent enumerations of the stock vs. patched dispatch decision,
by two independently-authored reimplementations of the boolean, over two non-overlapping grids:

grid 1: 13 arch profiles x 3 dtypes x 23x11x8 M/N/K shapes = 78,936 decisions
    252 changed
    252 False -> True
      0 True  -> False

grid 2 (independent re-derivation, different implementation, different grid,
        plus a dedicated num_sms=40..56 x cc_min={0,1,2} boundary sweep with
        two decoy arch profiles to catch major/minor typos):
   177,540 decisions
      438 changed
      438 False -> True
        0 True  -> False

All changes in both grids land on cc_maj == 12 && cc_min == 1 exactly, at M <= 6 exactly, and
the boundary sweep confirms the SM threshold is exactly >= 48 (nothing below flips) and the
minor version is exactly 1 (the two decoy profiles never flip). A row-by-row replay of the
second implementation against every one of the first implementation's 78,936 rows, and against
all 260 real-hardware rows from my earlier SIMT/MMA measurement, produced zero mismatches. Since
both grids are exhaustive over their enumerated space rather than sampled, this is a proof of
no-regression for those shapes, not evidence of one.

The diff was applied and re-applied cleanly in two separate, independent worktrees; the resulting
compiled libbitsandbytes_cuda130.so differs by hash from stock (confirms the patch reaches the
binary), and its CUDA Capabilities Selected: 121 build log confirms it targets the right arch.

Tests

Ran the 8 cells the enumeration flips False->True plus 5 control cells outside the patch's scope,
stock vs. patched, on real GB10 hardware (8 warmup + 25 reps, medians):

dtype M N K stock patched ratio
bfloat16 4 4096 4096 83.6us 60.5us 1.38x
bfloat16 6 4096 4096 90.6us 80.1us 1.13x
bfloat16 4 8192 8192 350.9us 188.5us 1.86x
bfloat16 6 8192 8192 347.4us 268.0us 1.30x
float16 4 14336 4096 249.4us 167.4us 1.49x
float16 6 5120 13824 299.7us 290.0us 1.03x
bfloat16 4 8960 1536 66.5us 55.3us 1.20x
float16 4 6144 4096 95.4us 81.3us 1.17x

All 8 flipped cells are faster patched, 1.03x-1.86x.
image

The tightest-margin cell (M=4, N=8960, K=1536, 1.20x) was rechecked 3 independent times per
variant and reproduced tightly every time (stock 66.4/66.6/66.5us, patched 55.1/55.3/55.2us).

Of the 5 control cells (all M>6, outside the patch's scope), 4 are within 5% (noise). One is not
and I'm flagging it rather than dropping it: float16, M=64, N=8192, K=8192 read stock 447.1us
vs. patched 378.5us (18% apparent patched speedup) on a shape where both variants run the
identical MMA path with no reason to differ. Rechecked 3x per variant: stock 430.1/431.4/432.1us,
patched 442.9/440.8/442.3us. The gap does not reproduce and reverses direction (patched ~2.5%
slower on recheck), so this was cold-start clock/thermal noise between two separate process
launches, not a patch effect.

Both rechecks side by side, in-scope cell on the left and the out-of-scope outlier on the right:

image

Raw per-cell JSON for every figure and table above, plus the harness, are available if useful.

Also ran the same functional/correctness suite the base PR ran, stock vs. patched, same two
detached worktrees:

suite stock patched
test_functional.py -k 4bit 1976 passed, 864 skipped identical
test_linear4bit.py + test_autograd.py 2156 passed, 1 failed identical

Same pass/fail/skip counts on both variants; the sorted (outcome, test id) sets are identical too
(0 differences across 2840 + 2157 pairs), so nothing merely balanced out in aggregate: no test
flipped outcome either direction. The one failure, test_fsdp_state_dict_save_4bit, is the same
pre-existing failure #2030's own PR body documented, with an identical root cause in both logs
(a torchrun-launched subprocess not inheriting the worktree's sys.path, unrelated to this
patch).

Honest limits

Single GB10, single machine. No second card to confirm these ratios, or the noise floor the
control-cell recheck surfaced, generalize beyond this box.

6 cells stay mispicked, and this patch structurally can't reach them. M in {4,6,8} x
{bfloat16,float16} at N=2048,K=2048: stock already (wrongly) picks SIMT there via the
undersubscribed clause, and MMA measures faster. Fixing that needs a True-removing change to
an existing clause for cc_maj==12 && cc_min==1, not another wide_n_simt disjunct, so it is out
of scope here, not touched, not proposed in this PR.

highbw_gddr untouched, and deliberately so. The same root-cause note that flagged
wide_n_simt also named highbw_gddr, so to pre-empt the obvious question: at M == 4 that
clause is broader than this one. use_simt tests it as (M == 4 && highbw_gddr) with no wave
term at all, whereas wide_n_simt additionally requires mma_blocks >= num_sms. Extending
highbw_gddr to sm121 would therefore also reroute sub-wave M=4 shapes, which is a region I
have not measured and which this PR's enumeration says nothing about. Whether GB10 wants SIMT
there too is a real question, just a different one, so I scoped it out rather than bundling an
unmeasured change in behind a measured one. Happy to go measure it as a follow-up if you want it.

Environment: GB10 (DGX Spark), sm_121, driver 580.142, torch 2.12.1+cu130, CUDA 13.0.88.

wide_n_simt already routes sm120 with >=48 SMs to the SIMT kernel at M<=6 with wide N. GB10 (sm121, 48 SMs) clears the same SM threshold but was excluded by the minor-version check, so it fell through to MMA.

Measured on GB10: all 8 rerouted cells are faster with SIMT, 1.03x to 1.86x. Enumeration over 78,936 and 177,540 dispatch decisions confirms the change is confined to cc_maj==12 && cc_min==1 at M<=6, False->True only, with no other architecture affected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant