Quantization for Serving:
A Roofline View

Every decode token drags the entire model through HBM. A 70B model in FP16 moves 140 gigabytes per token; at 3.35 TB/s that caps you at 24 tokens per second per GPU before you have written a single line of kernel code. Quantization is not about making the model smaller — it is about moving fewer bytes per token. The roofline model turns that into an equation you can recompute: tokens/s ≈ bandwidth ÷ bytes-per-token. AWQ's INT4, FP8, and NVFP4 are just four different points on the same memory roof.

01 Decode is a memory problem

bytes per token is the metric

Take a dense transformer with P parameters. Generating one token performs roughly one multiply-add per linear weight — 2P floating-point operations for P parameters (attention, norms, routing, and embeddings add their own traffic on top). It also streams every parameter from HBM exactly once: P·b bytes, where b is bytes per parameter. Divide the two and P cancels:

arithmetic intensity = 2P FLOP / (P · b) bytes = 2/b  FLOP per byte

This is the whole game in one line, and it has a consequence people routinely miss: the arithmetic intensity of decode depends only on the number format, not on the model. FP16 (b = 2) sits at 1 FLOP/byte whether the model is 7B or 70B. INT4 (b ≈ 0.5) sits near 4. Nothing about architecture, nothing about kernels — just the format. Model size decides how many tokens per second you get; the format decides where you sit on the roofline.

Now put numbers on the machine. An H100 SXM carries 3.35 TB/s of HBM3 and 989 TFLOP/s of dense BF16 tensor throughput (NVIDIA's datasheet quotes 1,979 with 2:4 sparsity; dense is half). The ridge point — where the memory roof meets the compute roof — is 989 ÷ 3.35 ≈ 295 FLOP/byte. Decode at 1–4 FLOP/byte sits roughly two orders of magnitude below the ridge: unambiguously, structurally memory-bound. No kernel can fix that; only fewer bytes can.

The ceiling follows directly: tokens/s = bandwidth ÷ bytes-per-token. For a 70B model:

Check the arithmetic: 70×10⁹ params × 2 B = 140 GB; 3.35×10¹² ÷ 140×10⁹ = 23.9. INT4 with group size 128 stores 35 GB of 4-bit weights plus one FP16 scale per 128 weights — 70×10⁹/128 × 2 B = 1.09 GB of scales — for 36.1 GB total, hence 0.5156 bytes/param and a 92.8 tok/s ceiling, 3.88× the FP16 number. NVFP4 stores 35 GB of 4-bit values plus one 8-bit scale per 16 values — 70×10⁹/16 × 1 B = 4.375 GB — for 39.4 GB total. Note the wrinkle we will return to: NVFP4 moves more bytes per token than INT4 with group-128 scales. Its win is elsewhere.

Two consequences fall out. First, quantization is a serving decision first and an accuracy decision second: 4× fewer bytes is 4× the throughput ceiling with zero kernel heroics, and the accuracy question is the constraint on that speedup, not the objective. Second, these are ceilings, not promises. Real kernels land 10–30% below them — dequantization overhead, imperfect bandwidth utilization, attention and KV-cache traffic on top. When someone reports 70 tok/s on a 70B INT4 decode against a 93 tok/s ceiling, that is 75% of roofline: respectable, and it tells you the remaining headroom is in the kernel, not the format.

What does the one-line model leave out? Three things, none of which break it. First, attention: decode attention is O(seq) FLOPs per token and streams the KV cache, so it is memory-bound on exactly the terms we already count — the model absorbs it. (Prefill attention is the famous exception: quadratic FLOPs make it compute-bound, which is the original motivation for FlashAttention.) Second, the LM head: the unembedding matrix is part of P, and for a 128k vocabulary over an 8192-wide hidden state it is 128k × 8192 × 2 B ≈ 2.1 GB of traffic per token — small next to 140 GB, but not zero, and worth remembering when someone's measured ceiling undershoots the model by a few percent. Third, MoE: for sparse models P in the equation is active parameters, not total. A 2.8T-parameter MoE that activates 40B per token moves ~80 GB per token in FP16, not 5.6 TB — which is precisely why the giant sparse models are servable at all, and why quantization math for MoE starts from the active count.

02 Weight-only: AWQ and GPTQ

shrink the weights, keep the math

Weight-only quantization (W4A16) is the conservative play: weights go to INT4, activations stay in FP16, and the matrix math still happens in FP16. The dequantization — INT4 → FP16 — is fused into the GEMM itself, so the weights travel across the memory bus at 4 bits and are expanded in registers or shared memory just before the tensor cores consume them. You get nearly the full 4× byte reduction with none of the numerical risk of quantizing activations, which is where the outliers live.

The two dominant methods differ in how they choose the quantization parameters. AWQ (Lin et al., MIT, arXiv:2306.00978) starts from an empirical observation: roughly 1% of weight channels are salient, and salience is determined by activation magnitude, not weight magnitude — what matters is which weights see the big inputs. Protecting those channels with mixed precision would be hardware-hostile, so AWQ uses an equivalent scaling trick. For a per-channel scale s:

Y = X·W = (X / s) · (s·W)

Mathematically identical — but now the salient channel's weights are scaled up before quantization, so they occupy more of the INT4 grid and rounding costs them relatively less, while the activations are scaled down to compensate. The scales are found by grid search minimizing output MSE over a small calibration set (on the order of a hundred samples). No backpropagation, no Hessian, no retraining — and the transform folds into neighboring layers, so there is no runtime overhead.

GPTQ (Frantar et al., ICLR 2023) takes the heavier path: layer-wise quantization derived from Optimal Brain Quantization, using second-order information. It feeds calibration data (classically 128 samples of C4) through each layer, forms the Hessian H = 2X^TX of the layer's inputs, then quantizes weights column by column, using the inverse Hessian to update the not-yet-quantized weights and absorb each rounding error. It is slower to run — hours for 70B versus minutes for AWQ — and holds the Hessian in memory, but it is less dependent on any single heuristic about salience.

The kernel that made W4A16 a serving standard is Marlin (Frantar et al., IST-DASLab, arXiv:2408.11743): an FP16×INT4 GEMM that dequantizes on the fly, accumulates in FP32, and reports near-ideal ~4× speedups out to batch sizes of 16–32 tokens — well past the batch-1 regime most "decode kernels" target. Integrated into vLLM, Marlin showed ~2.8× end-to-end generation speedups at batch 16 on an A10 for Llama-2-7B. It runs from Ampere through Blackwell, which matters more than it sounds: on Hopper, where there is no INT4 tensor-core fast path, a well-built W4A16 kernel is still the highest decode ceiling available.

How Marlin gets within touching distance of the ideal 4× is a lesson in where dequantization cost actually hides. The weights are re-packed offline into the exact interleaved layout the tensor cores want, so the "dequantize" step at runtime is a few register operations, not a gather. Per-group scales are loaded once per tile and applied during the INT4→FP16 expansion in registers, before the FP16 tensor-core MMA — the dequantized values never round-trip through shared memory. Accumulation stays in FP32 with a single downcast at the end, and asynchronous global→shared copies overlap the dequant work. The batch-size story in the paper's title ("mixed-precision auto-regressive parallel inference") comes from parallelizing over the batch/sequence dimension: at batch 16–32 there is finally enough independent work to keep every SM fed while the memory bus streams the 4-bit weights once. That is why the speedup holds to medium batch instead of collapsing after batch 1 like naive dequantize-then-GEMM approaches.

The accuracy numbers, honestly. From the AWQ paper's Table 4 (WikiText-2 perplexity, lower is better):

AWQ at INT4 costs 0.09 perplexity points on 70B — noise. GPTQ is a hair behind at 0.10. But two honest caveats. First, the cliff is real below 4 bits: INT3-g128 costs 0.42 points on the same model, and small models degrade faster than large ones — the Marlin repo's GPTQ numbers show Llama-2-7B MMLU falling 41.80 → 40.07 (−1.73) while 70B falls only 65.43 → 64.81 (−0.62). Second, perplexity understates downstream damage: a 0.09 PPL delta can hide a point or more on reasoning benchmarks, which is why you should read the MMLU column, not just WikiText-2, before signing off on a format. And calibration domain matters — GPTQ's Hessian is fit to its calibration set, so calibrating on web text and serving code costs measurable quality; AWQ's scale-search is less sensitive, which is a genuine operational advantage.

Bottom line for serving: on any pre-Blackwell GPU, W4A16 is the highest decode ceiling you can buy — 92.8 tok/s for 70B on H100 — and the accuracy cost at 4 bits on 70B-class models is small enough that the decision is usually about kernel support, not quality.

03 Weight+activation: FP8

quantize the compute too

Weight-only quantization leaves half the roofline untouched: the compute roof. The GEMM still runs in FP16, so prefill — which at 200–400 FLOP/byte is compute-bound — gains nothing from INT4 weights. FP8 (E4M3 for weights and activations, E5M2 where dynamic range matters) quantizes both sides of the multiply, which buys two things at once: half the bytes and twice the math. On H100, dense FP8 tensor throughput is 1,979 TFLOP/s against 989 for BF16 — the datasheet's 3,958 figure is with 2:4 sparsity; halve it for dense. The compute roof doubles, the ridge point moves from 295 to 591 FLOP/byte, and prefill gets a genuine ~2×.

Why FP8 instead of INT8? The predecessor, SmoothQuant's W8A8, had to work around a real problem: activation outliers. A handful of activation channels in every LLM are 10–100× larger than the rest, and per-tensor INT8 quantization crushes everything else to fit them. SmoothQuant's fix was a per-channel smoothing factor that migrated the quantization difficulty from activations into weights. FP8's E4M3 — 4 exponent bits, 3 mantissa bits, max representable value 448 — handles the same problem in the format itself: the exponent gives dynamic range that INT8's uniform grid lacks, so outliers no longer dictate the scale for the entire tensor. The hack moves from the math into the number system, which is exactly where you want it.

The outlier phenomenon is worth stating precisely, because it governs when per-tensor FP8 is enough. The LLM.int8() work showed that past a certain scale, transformers develop a small set of activation outlier channels — well under 0.1% of feature dimensions — whose magnitudes run 10–100× larger than the rest, concentrated in a few layers. Under per-tensor INT8, one scale must cover both the outliers and the bulk, and the bulk loses. E4M3's representable range runs from subnormals near 0.002 up to 448 — over five orders of magnitude — so a single per-tensor scale can hold both without the smoothing transform. When a model defeats even that (some architectures concentrate outliers harder), the escalation ladder is per-channel scales, then per-token activation scales: each step costs metadata and kernel complexity, each step rescues the layers the coarser step clipped. Measure the amax distribution before choosing; the format decision is empirical, not philosophical.

The remaining problem is choosing scales, and here FP8 serving inherited its machinery from training. NVIDIA's Transformer Engine popularized delayed scaling: instead of computing the maximum absolute value (amax) of every tensor on every step — an extra reduction pass you would feel — you track amax over a history window and update the per-tensor scale every N steps. For inference the analogue is amax calibration: run a calibration set through the model once, record per-tensor amax values, and bake the scales in. vLLM's FP8 path (--quantization fp8) uses per-tensor scales in this style; SGLang and TensorRT-LLM offer the same, with per-channel and per-token variants for models whose outliers defeat per-tensor scaling. The rule of thumb: per-tensor is free and usually sufficient at 70B; per-channel costs bookkeeping but rescues outlier-heavy layers.

FP8's second dividend is the KV cache. Nothing about the KV cache requires 16 bits — it is just stored activations — and vLLM lets you pin it to fp8_e4m3, halving KV traffic per token. At 8k context on our 70B reference (2.68 GB of FP16 KV per token-step), that is 1.34 GB saved per step; at 128k context it is the difference between KV traffic rivaling weight traffic and staying well under it. We will put numbers on this in section 05. For long-context serving, FP8 KV is arguably the bigger win than FP8 weights.

Accuracy-wise, FP8 is the boring, dependable choice: E4M3's 3 mantissa bits preserve far more precision than INT8's uniform grid, and on 70B-class models vendor and community reports consistently land within a few tenths of an MMLU point of BF16. The failure mode is the same one SmoothQuant fought — models or layers with extreme activation outliers, where per-tensor E4M3's max of 448 clips. The fix is finer-grained scales, not a different format. If your model is an outlier farm, measure per-channel before you blame FP8.

One honest accounting note: on decode bytes alone, FP8 (70 GB/token, 47.9 tok/s ceiling) loses to INT4 (36.1 GB, 92.8 tok/s). FP8's case was never "fewest bytes" — it is simplicity (no dequantization kernels, no group scales, native tensor-core support on Hopper and Ada), prefill speed (2× the compute roof), and KV-cache halving. Different point on the roof, different reason to be there.

04 FP4 and microscaling

NVFP4's block scales

Four-bit floating point has a dynamic-range problem that no amount of wishing fixes. E2M1 — 1 sign bit, 2 exponent bits, 1 mantissa bit — spans −6 to +6, and its smallest nonzero magnitude is 0.5. That is a dynamic range of 12. A whole weight tensor, whose values routinely span four orders of magnitude, cannot live in a range of 12 under a single per-tensor scale: either the outliers clip or everything else rounds to zero. This is why naive FP4 quantization of LLMs failed, and why the fix had to be structural.

The structural fix is microscaling: divide the tensor into small blocks and give each block its own scale. NVIDIA's NVFP4 groups 16 contiguous E2M1 values under one FP8 scale in E4M3 format, plus a global FP32 per-tensor scale on top — a dual-scaling system. The block scale absorbs local dynamic range (an outlier poisons only its own block of 16, not the tensor), while the E4M3 scale — unlike a pure power-of-two scale — supports fractional adjustments like 1.5×, so the block's values land closer to their true magnitudes.

A correction worth making precisely, because the formats are easy to confuse: NVFP4 is E2M1 + E4M3 scale per 16 elements. The OCP-standard MXFP4 is E2M1 + E8M0 (an 8-bit exponent-only scale, powers of two only) per 32 elements. Smaller blocks and a more expressive scale format are NVFP4's whole accuracy argument over MXFP4: finer granularity, less rounding error per block. Both have native Blackwell tensor-core support — the tensor cores read the narrow values plus their 8-bit block scales, compute partial dot products per block, descale, and accumulate in FP32.

The quantization recipe, concretely: for each block of 16 values, take the block amax and choose the E4M3 scale that maps it near 6.0 — the FP4 maximum — so the block uses the full grid; quantize each value to the nearest E2M1 code; then pick the global FP32 per-tensor scale to absorb the tensor-level range the block scales shouldn't have to. In practice (NVIDIA's Model Optimizer flow), the scales are refined against a small calibration set to minimize layer-output error rather than trusting the amax heuristic blindly — the same "measure, don't assume" discipline as AWQ's scale search, one level finer. The cost of all this precision is metadata traffic: those 4.375 GB of block scales per 70B-token-step ride the memory bus alongside the weights, which is exactly the 0.5625 bytes/param we computed. There is no free granularity.

Now the bytes, honestly computed: 16 E2M1 values occupy 8 bytes, plus 1 byte of E4M3 scale — 9 bytes per 16 params, 0.5625 bytes/param, 39.4 GB per token for 70B. Compare INT4 group-128 at 0.5156 bytes/param and 36.1 GB. NVFP4 moves more bytes per token than INT4-g128. If decode were only about bytes, NVFP4 would never win — and on Hopper, which has no FP4 tensor cores, it doesn't. NVFP4's case is that it quantizes activations too (W4A4), and that Blackwell executes it natively: NVIDIA reports FP4 math at 2× FP8 throughput on GB200 (3× on GB300), i.e. 4× the BF16 math rate. The economics are W4A4: ~3.6× fewer bytes than FP16 plus ~4× the math — the format that finally moves both roofs at once. On Hopper you get the bytes without the math; on Blackwell you get both.

Who actually uses it? Kimi-K3's released checkpoint packs its routed experts as MXFP4 — a 2.8T-parameter sparse model is exactly where ~0.5 bytes/param turns an impossible memory footprint into a servable one — and NVIDIA's Model Optimizer publishes an official recipe, plus a converted checkpoint, casting them to NVFP4. In the open, community quants like the NVFP4 conversion of MiniMax-M2.7 (via NVIDIA's Model Optimizer, quantizing the MoE expert MLPs and calibrating with natural top-k routing so each expert's scales reflect the token distribution it really sees) show the pipeline working end to end, with single-digit KL divergence against the source checkpoint after calibration. vLLM has picked up NVFP4 execution paths for Blackwell, which is what makes the format deployable rather than merely publishable.

The accuracy frontier, stated carefully: public serving-accuracy data for NVFP4 is still thin, and vendor parity claims are model-specific — treat them that way. The pattern so far favors large, overtrained models, which have the redundancy to absorb 4-bit noise, and punishes small models, where every bit of precision is load-bearing. If you are evaluating NVFP4, the models most likely to survive are the ones with the most parameters to spare. That is not a coincidence; it is the same reason INT4's accuracy cost shrinks from 7B to 70B.

05 The roofline, drawn

four points on one plot

Everything so far collapses into a single picture. The blue line is the HBM bandwidth roof — 3.35 TB/s, slope 1 on this log-log plot. The red line is the BF16 compute roof at 989 TFLOP/s; they meet at the ridge, 295 FLOP/byte. The dashed orange line is the FP8 compute roof at 1,979 TFLOP/s, which only matters once you climb out of the decode gutter. The four dots are our formats, each sitting exactly on the memory roof at its 2/b operating point, each annotated with the decode ceiling we computed in section 01:

The H100 decode roofline for a 70B dense model at batch 1. All four formats sit on the bandwidth roof — decode never touches the compute roof — and each quantization step moves the operating point rightward along it, from 23.9 to 92.8 tok/s. The FP8 compute roof (dashed) is irrelevant to decode but is exactly what doubles prefill throughput.

The plot also shows why the two phases of serving want different formats. Decode lives at AI 1–4, two decades left of the ridge; prefill lives at AI 200–400, right of it and compute-bound. INT4 is a decode format — it does nothing for prefill, whose math still runs in FP16. FP8 is a both-phases format — 2× the bytes win in decode, 2× the math win in prefill. NVFP4 on Blackwell is the first format that moves both roofs by ~4× at once. Pick the format for the phase that dominates your workload: if you serve long generations, you live in decode; if you serve long prompts, you live in prefill.

Bytes per token has a second axis the roofline hides: the KV cache. Weights are per-token traffic; the KV cache is per-token traffic that grows with context. For our 70B reference (80 layers, 8 KV heads, 128-dim heads), one token of KV cache is 2 × 80 × 8 × 128 × 2 B = 327,680 bytes in FP16 — and every decode step re-reads the whole cache:

Per-token memory traffic at batch 1 and 8k context, split into weights, KV cache, and activations. Weights dominate at this context length — which is why weight quantization is the first lever — but KV traffic scales linearly with context while weight traffic does not. At 128k context, FP16 KV traffic reaches ~42 GB per token, over 30% of the FP16 weight traffic; FP8 KV halves it. The ceiling in the right column is simply 3.35 TB/s ÷ total bytes.

The KV cache is also a capacity axis: context length at fixed memory. With 10 GB budgeted for KV cache, FP16 fits 10×10⁹ ÷ 327,680 ≈ 30.5k tokens of context; FP8 KV fits ≈ 61k. That doubling is free with the format — no kernel work, no accuracy review beyond validating the cache itself. For long-context serving, quantizing the KV cache (FP8 today, 2-bit schemes like KIVI on the horizon) is the highest-leverage change per unit of engineering effort.

The full comparison, in one table:

And the frontier those choices trace — throughput on the x-axis, quality on the y-axis:

The decode throughput–accuracy frontier for 70B-class models on H100. The FP16 and INT4 points use published numbers (Marlin repo, GPTQ INT4-g128 on Llama-2-70B: MMLU 65.43 → 64.81); the FP8 and NVFP4 positions are illustrative, reflecting vendor-reported near-parity — exact values vary by model and calibration. The shaded region is the free lunch: roughly 4× the throughput for about half an MMLU point. Note NVFP4 sitting left of INT4 on throughput: on Hopper it pays the block-scale byte overhead without getting FP4 math in return.

06 Open questions

what to watch

Per-layer mixed precision, chosen by measurement. Not all layers are equally sensitive — early layers and the final norm-adjacent projections typically degrade first under quantization. Today this is folklore and manual lists; the obvious next step is automating it: run a sensitivity sweep per layer (or per block), keep the fragile 5% in FP8/FP16, drop everything else to FP4, and let the roofline tell you what the blended bytes-per-token buys. The tooling for this barely exists.

FP4-native training-to-serving pipelines. NVIDIA's NVFP4 pretraining work points at the endgame: if the model is trained in NVFP4, there is no quantization step at all — no calibration data, no accuracy cliff, no format conversion in the serving path. The checkpoint is the serving artifact. Watch whether the large labs start publishing training recipes in microscaling formats rather than BF16; that single change would obsolete half of this essay's calibration discussion.

The 2-bit frontier. At 0.25 bytes/param, arithmetic intensity hits 8 FLOP/byte and the 70B decode ceiling reaches ~187 tok/s on H100. Methods like QuIP# and AQLM have shown 2-bit weight quantization with survivable quality on large models, but serving kernels for 2-bit are still exotic and the accuracy story below 4 bits remains model-dependent. If 2-bit crosses from "paper result" to "vLLM flag," the roofline moves again — rightward, as always.

Sub-FP8 KV cache for million-token context. KIVI's 2-bit KV cache (per-channel quantization for keys, per-token for values, exploiting their different outlier structure) previews a world where the KV axis compresses 8× instead of 2×. At 1M context, KV traffic dominates everything — the weight roof stops mattering and the cache roof takes over. Long-context serving will be won or lost on KV formats, not weight formats.

Hardware fragmentation. NVFP4 is Blackwell-only today; AMD's and Intel's roadmaps have their own low-precision stories. If every vendor ships a different microscaling flavor, serving stacks eat the complexity — one more reason the boring, portable formats (INT4-g128, FP8) will outlive the hype cycle even where they lose on paper. The roofline is physics; the format zoo is politics. Bet on physics, hedge on politics.

A shared accuracy bar, reported at every operating point. Too many quantization results are still reported as perplexity alone, on WikiText-2, at batch 1 — a number that flatters formats and hides downstream damage. The honest report is a small battery (MMLU, GSM8K, HumanEval at minimum) run on the serving artifact — the actual quantized checkpoint behind the actual kernel — at each point on the frontier. Until that becomes the norm, treat any single-number accuracy claim the way you would treat a single-number latency claim: as the beginning of the investigation, not the end.