Fitting a Million
Tokens

A million tokens of KV cache on a 70B-class model weighs 328 gigabytes — more than twice the model’s own weights and four times what fits on an H100. Long context stopped being an attention problem the moment contexts outgrew HBM; it is now a memory-capacity problem with exactly three answers. Compress the cache, evict most of it, or move it off the GPU. This essay does the arithmetic for all three — and shows why production systems use all three at once.

01 The capacity math

count the bytes before anything else

The reason nobody serves a million-token context on a single GPU is not attention FLOPs. It is that the KV cache for that context is 328 GB — a number you can recompute on a napkin, and should, before believing any vendor’s context-length claim.

Every token you generate appends one key vector and one value vector per layer per KV head. The bytes per token are:

bytes/token = 2 × layers × KV-heads × head-dim × bytes/element

Nothing else enters. Take a 70B-class model with grouped-query attention: 80 layers, 8 KV heads, 128 dimensions per head, fp16. That is 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token — exactly 320 KiB. Times a million tokens: 327.68 GB (305 GiB). Cache alone, one sequence, before the weights, before the batch dimension, before anything else.

fp16 throughout. Weights for reference: 16 GB (8B) / 140 GB (70B). All figures per sequence; multiply by batch.

Two things fall out of the table. First, the weights are the smaller problem: 140 GB of fp16 weights already exceeds a single 80 GB H100, so the model needs at least two-way tensor parallelism before a single token of KV is stored anywhere. Second, everything scales with batch: eight concurrent 128K sequences on the 70B hold 8 × 42.9 = 343 GB of KV. Context length was never a model spec. It is a memory allocation, quoted in tokens because gigabytes wouldn’t fit on the slide.

vLLM’s PagedAttention stores this cache in 16-token blocks, and the internal fragmentation — at most 15 tokens per sequence — is noise next to hundreds of gigabytes. The block layout matters later, when quantization groups align to it.

There is a second wall hiding behind the first. The cache above is built during prefill — the compute-bound phase where the whole prompt is processed at once — and then streamed during decode, the memory-bound phase that generates one token at a time. Prefill at a million tokens is its own crisis: roughly 2 FLOP per parameter per token gives 2 × 70×10⁹ × 10⁶ = 1.4×10¹⁷ FLOP, which at ∼500 TFLOPS effective is 280 seconds — nearly five minutes before the first generated token. Long context has two walls, and this essay is about the memory one; the compute one belongs to a different essay.

One more number to carry into the rest of the essay: per-layer KV at 1M tokens in fp16 is 2 × 8 × 128 × 2 bytes × 10⁶ = 4.096 GB. A single layer’s cache for a single sequence is four gigabytes. Every prefetch, quantization, and eviction decision below is denominated in units of this number.

For calibration, the extremes of the head-count spectrum: multi-query attention (one KV head shared by all query heads) would cut the 70B’s per-token cost to 2 × 80 × 1 × 128 × 2 = 40,960 bytes — 40 KiB/token, 41 GB at a million. Nobody ships MQA at 70B scale anymore because the quality cost is real, but the number shows how much of the 328 GB is a choice: the reference point for every table in this essay is Llama-3-70B’s config (80 layers, 64 query heads, 8 KV heads, head-dim 128), and moving the KV-head slider is the cheapest architectural change in the book.

Weights are fixed at 140 GB; KV grows linearly with context. At 8K the cache is a rounding error; at 1M it is 2.3× the weights and 4× the HBM on an H100. The dashed line is the part nobody puts on the slide.

02 Quantize the cache

KIVI and friends

The cheapest bytes are the ones you never store. Quantizing the KV cache is the highest-return move in this essay, because decode is so memory-bound that the arithmetic to decompress it is nearly free.

KIVI (ICML 2024) starts from an observed asymmetry. Key caches have persistent outlier channels: a handful of channels whose magnitudes dwarf the rest — in the same channels, across nearly all tokens. Quantize keys per-token and those channels stretch every token’s scale and destroy precision; quantize per-channel, with groups of 32 running along the token axis, and each outlier is quarantined to its own scale. The paper’s ablation is brutal on this point: per-token key quantization at 2-bit explodes perplexity, per-channel does not. Value caches show no such channel persistence, so values are quantized per-token instead — which also matches how they are consumed, since the attention output is a weighted sum over token rows and per-row error stays per-row. The scheme is asymmetric 2-bit, tuning-free — no calibration data — and keeps the most recent ~128 tokens in fp16 as a residual window, because the tail of the sequence is where attention is most sensitive and it never gets quantized.

The group size of 32 is not an accident next to vLLM’s block size of 16: one quantization group spans exactly two PagedAttention blocks, so dequantization happens at block granularity inside the fused attention kernel with no layout change. The kernel loads 2-bit codes plus fp16 scales, dequantizes in registers, and never materializes a full-precision cache.

This trade is lopsided in our favor. Decode attention does roughly 2 FLOPs per byte of KV streamed; an H100’s ridge point sits near 300 FLOPs per byte (∼989 TFLOPS over ∼3.3 TB/s of HBM3). We are two orders of magnitude into the memory-bound region, which means there is two orders of magnitude of ALU headroom to spend on dequantization before the kernel stops being memory-bound. Cutting bytes 4× buys roughly 4× decode throughput; the extra multiply-add per element is lost in the headroom.

The table’s “effective bits” deserve their arithmetic shown. Asymmetric quantization stores, per 32-element group, an fp16 scale and an fp16 zero-point alongside the codes: 16 + 16 = 32 bits of metadata per 32 elements, i.e. exactly 1 extra bit per element. So 2-bit KIVI is really ∼3 bits/element (16 ÷ 3 ≈ 5.3×) and 4-bit is ∼5 bits/element (16 ÷ 5 = 3.2×). The residual fp16 window adds a fixed ∼128 tokens × bytes/token — about 40 MB on the 70B — which vanishes against gigabytes at long context but dominates at short ones. Compression ratios always improve with sequence length; quote them at your context length, not the paper’s.

Effective bits assume asymmetric quantization with an fp16 scale + zero-point per 32-element group; the fp16 residual window is excluded (negligible at 1M tokens). Accuracy claims are the authors’, on LongBench/GSM8K-class suites.

And then there is the boring option: FP8 (E4M3) KV, exactly 2× smaller, natively supported by Hopper, shipped in vLLM and TensorRT-LLM. No calibration, no group bookkeeping, no accuracy cliff to fall off. When a serving team tells you their long-context stack “just uses fp8 KV,” that is not a lack of sophistication — it is the part of the system they chose not to gamble on.

Two practical footnotes. First, not all layers are equally quantizable — practitioner lore, confirmed across several quantization efforts, is that early layers are the most sensitive, and mixed precision by layer (fp16 for the first layer or two, aggressive below) is the standard escape hatch when a uniform setting degrades quality. Second, E4M3’s appeal is operational as much as numerical: no calibration corpus, no per-model tuning, and dequantization the hardware already knows how to do. In practice it lands without per-tensor calibration on mainstream models — which is precisely why it shipped in production kernels while fancier schemes stayed in papers.

The streaming data structure behind KIVI deserves a closer look, because it is where the per-channel/per-token asymmetry bites. Values quantize per-token, so each new token forms complete groups the moment it arrives. Keys quantize per-channel with groups running along the token axis — so a single new token contributes one element to every group, and no group is complete until 32 tokens have accumulated. The residual window is therefore not just a quality trick; it is the staging buffer where keys wait in fp16 until they can form whole groups, and the quantization boundary must be snapped down to a multiple of the group size. Get this wrong and decode-time keys are silently never quantized at all (min equals max in a one-element “group,” scale collapses, round-trip is bit-exact fp16 wearing a 2-bit costume). KIVI ships custom CUDA kernels for this dance; the Triton reimplementations exist because everyone else wanted the same layout without the custom kernel.

03 Evict what doesn’t matter

attention sinks and H2O

Compression assumes every token deserves to be remembered. Eviction calls the bluff: most of the cache is never attended to again, so stop storing it.

StreamingLLM’s starting observation is strange enough to sound like a bug: in trained transformers, the first few tokens attract enormous attention mass even when they are semantically empty — a BOS token, a newline. The mechanism is softmax bookkeeping. When no key matches the query well, the probability mass has to go somewhere, and the model learns to park it on early tokens, which are visible to every later position. They become attention sinks. Keep the first 4 tokens plus a sliding window of the most recent 2,048, evict everything in between, and the model’s sink behavior is preserved — demonstrated on LLaMA-family models. The cache stops growing: 2,052 tokens is 672 MB on our 70B at any stream length, a 487× reduction that holds at ten million tokens as well as one.

H2O — the Heavy-Hitter Oracle — generalizes the idea. Instead of a fixed pattern, it accumulates attention scores as decoding proceeds; tokens that repeatedly attract mass are heavy hitters and are kept, alongside a recent window for locality. Everything else is evicted. It is dynamic and per-sequence where StreamingLLM is fixed, and the authors report full-cache-matching accuracy on their benchmark suite with roughly 20% of the KV retained. The cost is bookkeeping: per-token score tracking plus an eviction pass, which is its own kernel and its own set of edge cases.

There is a second, quieter reason eviction hurts less than it should. “Lost in the Middle” showed that even with the full cache resident, models retrieve worse from the middle of long contexts than from the ends — the attention mechanism itself is position-biased. Evicting the middle therefore discards the region the model was already worst at using. That does not make eviction safe for retrieval; it explains why summarization benchmarks barely notice it, and why you should distrust any eviction evaluation that only reports tasks the model was already good at.

Production honesty, to close the section: mainstream serving stacks — vLLM, SGLang, TensorRT-LLM — do not evict by default. The default is the full cache, paged, exactly as section 01 counted it. Eviction lives in plugins and research branches, because silently forgetting user content is a product decision, not an optimization. If you deploy it, the eviction policy is part of your model’s behavior contract and should be versioned like one.

One piece of folk wisdom falls out of the sink mechanism for free: in chat models, the system prompt occupies the sink positions. Operators learned empirically that pinning the system prompt — never evicting the first tokens — preserves behavior, and the sink theory says why: those positions were going to attract mass regardless of content, so you might as well put load-bearing content there. Four sink tokens is the paper’s number, and its smallness is the point: sinks are about softmax bookkeeping, not information, so a handful suffices.

Here is the honest part. Eviction is a bet that your task is recency-biased, and needle-in-a-haystack is that bet’s natural enemy: put the needle at 50% depth of a 128K context behind a 2K window and the needle’s KV is gone by construction — retrieval is ~0%, not degraded, gone. Summarization and local QA decay gracefully because their answers depend on local structure the window preserves. Most eviction papers report perplexity plus a LongBench subset; perplexity is nearly blind to eviction because local fluency survives intact. Demand needle-at-arbitrary-depth numbers before trusting any eviction scheme on a retrieval workload.

Left: where the mass goes — bright sink columns, a bright diagonal band of recent tokens, the evicted middle grayed out. Right: the shape of the tradeoff. Retrieval tasks fall off a cliff the moment the kept fraction stops covering the needle’s position; local tasks decay smoothly. The curves are schematic; the cliff is not.

04 Offload: the memory hierarchy answer

prefetch pipelines

If you refuse to lose information, the remaining move is to put the cache somewhere bigger than HBM: CPU DRAM (hundreds of gigabytes per socket) or NVMe (terabytes per drive). The trick that makes it survivable is prefetch — stream layer i+1’s KV over PCIe while the GPU computes layer i. LMCache implements exactly this as a vLLM plugin; Mooncake builds the disaggregated version, with a CPU-side KV pool shared across prefill and decode workers over RDMA.

Run the numbers for our 70B at 1M tokens in fp16. One layer’s KV is 4,096 bytes/token × 10⁶ = 4.096 GB. PCIe 5.0 x16 gives 64 GB/s in theory and roughly 50 GB/s sustained, so moving one layer takes ∼82 ms. The same 4 GB read from HBM3 at 3.3 TB/s takes 1.2 ms. The transfer is 66× slower than the read it replaces — and that ratio is the entire story of offload.

At batch 1 with prefetch, each layer costs max(compute, 82 ms) = 82 ms, so a token costs 80 × 82 ms ≈ 6.6 seconds against ∼140 ms HBM-resident (80 layers × ∼1.8 ms: 0.5 ms streaming weights plus 1.2 ms streaming KV). Offloading the full cache makes decode ∼45× slower. Prefetch does not save you here; there is nothing to hide behind.

Prefetch hides the transfer exactly when per-layer compute exceeds per-layer transfer. Per-layer compute at batch B is roughly 0.53 ms of weight streaming — amortized across the batch — plus B × 0.0117 ms of attention-plus-MLP arithmetic (5.85 GFLOP/token/layer (dense + attention at 1M context — the attention term grows with context length, so don't reuse this figure at 128K) at ∼500 TFLOPS effective). At 1M context in fp16, hiding an 82 ms transfer needs a batch in the thousands. Never, in practice. The transfer must get smaller, the compute bigger, or both. That is the whole design space:

PCIe 5.0 x16 at ∼50 GB/s sustained; compute at ∼500 TFLOPS effective; weights streamed once per layer. Recompute any row: transfer = per-layer KV bytes ÷ 50 GB/s.

Two rows of that table are the usable regime: shrink the context to 128K and quantize to 4-bit, and the transfer is 3.4 ms/layer — hidden at batch 1024, exposed at batch 256. This is why offload papers benchmark throughput, not latency.

InfiniGen attacks the transfer directly instead of the batch. Its observation is that attention is skewed: a few KV entries take most of the mass. It uses the previous layer’s attention pattern to predict which entries the next layer will need and prefetches only that salient subset from CPU — partial, query-aware prefetch. The granularity choice is the tradeoff: layer-wise prefetch (the whole next layer) is simple but wastes PCIe on tokens nobody attends to; token-wise prefetch (salient entries only) minimizes bytes moved but needs the prediction machinery, and its misses.

NVMe is another 7× below CPU DRAM — roughly 7 GB/s per Gen4 drive against ∼50 GB/s of usable PCIe to the host. It is viable only with deep prefetch queues or heavy saliency filtering. Treat NVMe as the archive tier for cold KV, not a prefetch target for hot decode.

Offload has an older sibling worth naming: FlexGen, which offloads not just KV but weights and activations across GPU, CPU DRAM, and disk, scheduled by a linear program that maximizes throughput under the hardware’s real bandwidths. It served OPT-175B on a single GPU — slowly per token, but with aggregate throughput no HBM-only setup could touch at that hardware budget. FlexGen is the proof that offload is a throughput architecture wearing a latency costume, and every prefetch pipeline since is a special case of its scheduling problem with the weights pinned to HBM.

The same bandwidth math governs disaggregated serving. Splitwise and DistServe split prefill (compute-bound) from decode (memory-bound) onto different workers — and the KV cache has to move between them. Mooncake exists because that handoff is the system: 328 GB per request at 1M tokens over 400 Gb/s RDMA (∼50 GB/s) is 6.6 seconds of transfer per request, before decode even starts. Disaggregation doesn’t remove the capacity wall; it relocates it to the network, where at least the bandwidth is someone else’s budget line. This is also why chunked prefill (Sarathi’s contribution, now standard in vLLM) matters: it keeps giant prefills from stalling decodes sharing the same GPU, which is the other half of making long context servable.

A final calibration: many deployed servers are still PCIe Gen4, at 32 GB/s per x16 slot — half the numbers in the table above. If your fleet is Gen4, double every transfer time and halve every break-even batch. The inequality doesn’t care about your roadmap; measure the sustained bandwidth with a microbenchmark before trusting a prefetch design, because “PCIe 5.0” on the spec sheet and 50 GB/s sustained are different numbers.

There is one more design in the space that inverts the pipeline: leave the KV on the CPU and run attention there. At batch 1 the arithmetic intensity of attention is identical wherever it runs — memory-bound on the KV stream — so skipping the PCIe transfer entirely and paying CPU compute instead can win when the GPU would otherwise sit stalled. It loses the moment batch grows, because GPU throughput scales while the transfer stays fixed, which is the same break-even inequality with the sides swapped. Prefetch-to-GPU and attention-on-CPU are two answers to one equation.

Same hardware, same model, same 128K context in 4-bit: at batch 1024 each layer computes for 5.2 ms while its 3.4 ms transfer hides underneath — zero exposed. At batch 8 the compute ticks are 0.6 ms slivers against the same 3.4 ms transfer, leaving ∼2.8 ms exposed per layer, ∼224 ms per token of pure PCIe stall. Batch size is the entire difference between “offload works” and “offload is a regression.”

05 Architectural escape hatches

MLA, GQA, and linear attention

Compression, eviction, and offload all accept the transformer’s KV cache as given and manage it. The fourth move is to change what gets cached.

Grouped-query attention is already assumed in every number above: 8 KV heads against 64 query heads stores 8× less than multi-head attention. Run the counterfactual — a 64-head MHA at a million tokens caches 2 × 80 × 64 × 128 × 2 bytes × 10⁶ = 2.6 TB. GQA is the reason the numbers in section 01 are merely alarming instead of impossible.

DeepSeek’s multi-head latent attention goes further. Instead of caching per-head keys and values, MLA compresses them jointly into a single 512-dimensional latent vector per token per layer, plus a 64-dimensional decoupled key that carries rotary position information. Every head’s key and value are linear projections of the shared latent — and at inference time those up-projections are absorbed into the query and output matrices, so attention is computed directly against the compressed representation with no explicit decompression step. The RoPE key stays separate because rotary embeddings cannot survive low-rank compression: position has to stay bound to the key. DeepSeek reports a 93.3% smaller KV cache than its MHA baseline (DeepSeek 67B). The percentage depends on the baseline; the mechanism — the cache stops being per-head entirely — is the durable part.

The radical version is to stop caching altogether. Mamba’s selective state-space layers keep a fixed-size recurrent state: roughly 10 MB for a 64-layer model (5,120 × 16 × 64 × 2 bytes), independent of context length — some 30,000× smaller than the 328 GB it replaces. Kimi’s KDA pushes the same frontier from the associative-memory direction with a delta-rule update. The price is known and unsolved: precise retrieval over long contexts — the needle tasks — is where linear recurrent models still trail attention. Constant memory is bought with a quality frontier, not a free lunch.

There is a quieter architectural lever between GQA and MLA: sliding-window attention with a rolling buffer, as in Mistral 7B. Each token attends only to the previous 4,096 tokens, and the cache is a ring buffer of exactly that size — 4,096 × 131,072 bytes = 537 MB for the 7B, fixed forever, at any context length. It is eviction as architecture: infinite context at constant memory, with precisely the retrieval cliff of section 03 baked into the model weights. The lesson generalizes: every “infinite context” claim cashes out as a statement about what the model is allowed to forget.

MLA, meanwhile, has escaped the lab. Beyond DeepSeek-V2/V3 it has been adopted by other frontier efforts — notably Kimi K2 and the K2 family — which is the real signal: a compression scheme that survives contact with someone else’s training stack is infrastructure, not a paper trick. Watch for the latent dimension to become a standard model-config field the way KV-head count did after GQA.

The reason MLA survived contact with other labs’ training stacks is that the quality held up: DeepSeek’s controlled ablation has MLA matching — slightly beating, in their numbers — MHA at the same scale, which is what turned a compression trick into a default architecture choice. That is the bar every escape hatch has to clear. GQA cleared it years ago by being nearly free; MLA cleared it by being better-compressed and no worse; linear attention has not cleared it yet, which is why it remains a frontier rather than a default.

06 Open questions

three things that would change the math

Learned, query-dependent compression. Static quantization spends the same bits on every token. The next step is a router that spends bits where the query will look — compression as a function of (key, value, query), not just (key, value). Early work conditions precision on predicted attention mass; nothing yet matches KIVI’s simplicity at its quality. The prize is large: attention mass is heavily skewed, so the gap between “average bits per token” and “bits where it matters” is most of the remaining headroom in section 02’s table.

OS-level KV paging. Every serving stack reimplements tiering in userspace — its own prefetcher, its own eviction, its own block layer. An experimental project of mine, Kairo, explores whether the kernel should know about KV explicitly: deadline-aware scheduling in blk-mq, the page cache treating KV pages as a first-class tier rather than anonymous memory. If that lands, “offload” stops being a serving-system feature and becomes a memory-management policy — and the prefetch timelines in section 04 get drawn by the OS scheduler, with real deadline semantics, instead of by each framework’s plugin.

CXL-attached memory pools. CXL.mem exposes terabytes of pooled DRAM with load/store semantics at on the order of 100 ns of extra latency and tens of GB/s per link — CXL 3.0 on a x16 slot is in the same bandwidth class as the PCIe transfers section 04 priced out, but without leaving the memory semantic. If the KV tier becomes a CXL pool shared across GPUs, the capacity wall moves from 80 GB to terabytes: prefetch becomes an optimization instead of a requirement, the batch-size break-even stops binding, and disaggregated prefill/decode can share one physical KV pool instead of shipping 328 GB per request over RDMA. The hardware exists; the software that treats it as a KV tier does not, yet.