How far can you push an RTX 3080?
For the last 2 months I've been poking at an RTX 3080 below CUDA, figuring out a bunch of stuff NVIDIA doesn't document, and using it to run an LLM faster than llama.cpp. Some of it worked, a lot of it didn't.
Why one GPU
Some context: I used to have a 3090 with 24 GB of VRAM. I sold it (lowkey a dumb idea) and was left with a 3080 and 10 GB. So this started as me trying to get every last bit out of the card I still had.
When you write CUDA there are two compilers between you and the hardware. The second one, ptxas,
is closed source, and the machine code it spits out (SASS) isn't documented. NVIDIA also doesn't tell you how
long instructions take or what happens when one depends on another. A lot of GPU performance advice is
educated guessing as a result.
I wanted to see how much of that I could just measure. Rules: one card, no CUDA toolkit (the compilers and disassembler all come from pip), and no claim without a measurement behind it.
A lot of this has been mapped before. Scott Gray's maxas worked out control codes on Maxwell, Jia et al. dissected Volta and Turing with microbenchmarks, including the register banks, and tools like CuAssembler already edit SASS. What I did was re-measure it all on Ampere, on my card, and then see what I could do with it.
I also did a lot of this with Claude (Opus 5.5). It wrote probe kernels and scripts, read disassembly with me, and pushed back when a result looked too good. Without it this would have taken months.
Measuring the machine
Every SASS instruction is 128 bits. Most of that is what to compute. Bits 105 to 125 are the compiler telling the hardware how long to wait and what to wait for. I figured out the fields by editing them in compiled binaries and seeing what broke.
I figured if the stall count was too low the hardware would catch it and wait anyway, like a CPU does. So I took a chain of adds where each one needs the last result and set the stall below 4 cycles.
Wrong answer 20 times out of 20. Exactly half the adds just vanished, because each one read the register
before the previous write landed. The hardware doesn't check. Those 4 bits are the only thing keeping the math right.
A stall of 0 doesn't mean "don't wait". It's correct but costs about 33 cycles, which is part of why
-O0 code is so slow.
R1. The second one reads R1 when it issues, which
is stall cycles after the first. The first add's result only lands 4 cycles later. With stall 4 the
second add sees 1 and the answer is 2. With stall 2 it reads the old 0, and one add is lost.How long everything takes
After that it was a lot of measuring. The runner checks the disassembly of every timed loop before it runs, so I'm not accidentally timing something the compiler rewrote. Memory latency, for example:
A ceiling that turned out to be the compiler's fault
Each Ampere SM partition has 16 float lanes and 16 lanes that do either float or integer. If integer ops just borrowed the shared half, a loop that's half FMAs and half integer shifts should issue 4 instructions per cycle per SM. I measured 2.65.
Most of my mix ratios fit a simple model where float and integer never overlap at all: an FMA takes both halves
for a cycle, a shift holds the shared half for two. I wrote that down as how the hardware works. Two things
didn't fit: a couple of the mixes beat the model, and ptxas had reordered my loop (it moved the
FMAs to the front). But an additive cost model doesn't care about order, so I didn't think order could explain it.
Later I was probing the register file for a different reason. It has two banks (odd and even register
numbers), each bank gives one read per cycle, and an FMA reads three registers, so it needs a small operand
cache to run at full rate. Going back to the mix kernel with that in mind: the shift was evicting one FMA operand
from the cache, and ptxas had put the other two in the same bank. I patched the cubin to move
each FMA's chain register to the opposite bank. Same instructions, same order, different register numbers. The
50/50 mix went from 2.65 to 3.91. Moving it to the same bank on purpose gave 2.65 again. Float and integer
do overlap. My "no overlap" model was a register allocation problem.
ptxas.Beating the compiler, once
Memory comes in 32-byte sectors. In some quantized matrix-vector kernels, ptxas puts two loads that
each need half of the same sector way apart, so by the second load the sector's gone and gets fetched again. 3080lab optimize finds those pairs in the SASS, moves the second load up next to the first and redoes
the control bits. It checks that the rewritten binary disassembles and introduces no new modeled hazards. The
benchmark harness separately checks its output against the original.
I froze the tool and ran it on three suites. B and C were held out. I built the tool on A and kept adjusting it there afterwards, so A is a development number (57 kernels, 56 timed).
| suite | kernels | speedup (geomean) | wrong outputs |
|---|---|---|---|
| A, used while building | 56 | 1.036× | 0 |
| B, held out | 43 | 1.074× | 0 |
| C, held out, high register pressure | 20 | 1.055× | 0 |
After autotuning each kernel's configuration first, the rewrite added 0.3%, within measurement noise on all 12 shapes. The 1.074× comes from configurations an autotuner wouldn't keep. It's useful on a binary you can't recompile or retune, and not otherwise.
Then I tried writing a whole instruction scheduler to beat ptxas everywhere. It tied, 1.003×.
Picking the best of three orderings for every kernel only gets 1.005×, so there isn't much left there. It did catch a rule I'd missed (memory instructions need 4 cycles between them), which
had been silently breaking one test suite.
A real model, end to end
I took Qwen2.5-1.5B and loaded llama.cpp's own Q4_0 file, so both engines read the exact same bytes, and wrote an engine around it: about 200 fused kernels in one CUDA graph, int4 kernels for single-stream decode, tensor cores for batches, int8 tensor cores for prefill. It was already 1.28× llama.cpp on day one. Here's where it is now against the newest llama.cpp:
None of this speed comes from the binary rewrite; these decode kernels have nothing for it to fix. It comes from what the earlier measurements said the card is bound by: fuse what can be fused, keep many memory requests in flight, and use kernels shaped for each batch size.
Single stream is 514 tok/s. A token takes 1.95 ms, most of which is reading 928 MB of weights, and most of the win over llama.cpp is just launching fewer, bigger kernels. With more sequences the win gets bigger because each weight gets read once for all of them. Outputs match a token-by-token reference. On one text (the repo README), the int8 path's perplexity was within 0.1% of an fp16 reference using the same quantized weights.
Model, or just search?
By November I had a model that predicts kernel runtime, and I wanted to know whether it makes good decisions. I wrote down the test and what counts as a win before fitting anything, then measured 771 decisions:
| decision | best fixed rule | model | oracle |
|---|---|---|---|
| launch structure (fusion, CUDA graphs) | 1.159× | 1.229× | 1.231× |
| reordering SASS instructions | 1.000× | 1.0007× | 1.005× |
For launch structure, its choices achieved performance within 0.2% of the oracle on this benchmark. For instruction order there was only 0.5% to win in the first place.
Against autotuning
Autotuning is just timing options and keeping the fastest. When there are only 4 to 21 options, the model didn't beat a rule of thumb plus a couple of tries. The prefill GEMM is different: it has 212 tile shapes. On four Llama-3-8B layers (which the model had never seen) I timed every single one:
Then a bigger test: 32 workloads from Qwen2.5-7B, Gemma-2-9B, Phi-3-mini and Qwen2.5-14B, every one of the 212 tiles timed 3 times in random order (about 5 hours of GPU), against Optuna's TPE autotuner. The bar, picked before any timing: get within 5% of the best tile with 4× fewer measurements than TPE.
The model needs 2.4 measurements on average and TPE needs 22.6, so about 9× fewer (somewhere between 6× and 14× 95% interval). On 21 of the 32 the model's very first pick was already within 5%. TPE barely beat random guessing.
On 9 workloads it was under 4×, and on one Gemma layer TPE won. One tile was the fastest on 14 of the 32 workloads, and the model ranked it 12th almost every time. When that tile wins, the model spends about 11 tries getting there. Also, the model has to compile all 212 tiles before it can rank anything, which takes about 4 minutes of CPU. Counting total time from a cold start, TPE wins; the model only wins on GPU time. Full report.
What didn't work
- A general SASS scheduler. Parity with
ptxas, never better. Joint register allocation plus scheduling, and software pipelining, didn't help either. - Letting the model decide when to rewrite. No better than always rewriting, and only 5 of 9 right calls under register pressure.
- The decode megakernel. Wrong token, slower than the graph.
- Ranking close GEMM tiles. In blind tests the model got single trends right (occupancy 15 of 16), but picking between two good tiles was close to a coin flip (0.57).
- A correctness check that couldn't fail. My test inputs weren't quantized properly, so every output was NaN and the check was meaningless. Fixed it, reran it, everything passed.
- Selling the 3090. tbh this was really dumb even though I got a lot of money for it.
How I tried not to lie to myself
GPU timings are noisy and a 2% win is easy to imagine. Time two identical binaries against each other first, and anything smaller than that gap doesn't count. Alternate A and B instead of running all of A then all of B, because the card heats up. Commit the rules for a test before running it, so I can't move the goalposts later. And check the outputs too, since a fast kernel that gives the wrong answer is useless.
All of this is one 3080. I haven't checked any of it on another card, and the SASS rewrites are tested, not proven correct (the verification notes list the gaps). The code, the raw data and the day-by-day notebook are all on GitHub.