The Model’s View of Memory

01 / Begin with the correction

“GPU memory” and HBM are not competing categories.

GPU memory describes memory attached to—or logically owned by—the GPU. HBM describes one physical DRAM technology that can implement that memory.

A datacenter GPU may expose tens or hundreds of gigabytes of device or global memory to software. On H100 and B200-class accelerators, that capacity is physically backed by HBM stacks mounted close to the GPU package. Other GPUs use GDDR. CUDA also exposes registers, shared memory, caches, mapped host memory, and managed allocations. So “GPU memory” can mean an address space, an allocation class, or the physical capacity on a card; HBM is specifically stacked, very-wide-interface DRAM.

CPU DRAM is also volatile semiconductor memory, commonly DDR5 DIMMs attached to the CPU’s memory controllers. NVMe is different again: it is a storage protocol and command interface, usually reaching NAND-flash SSDs over PCIe. It is persistent block storage—not byte-addressable working memory for a GPU kernel.

From the model’s perspective, the useful hierarchy is not “memory versus storage.” It is operand now, transferable soon, or recoverable later.

On-chip

Registers, shared memory and caches feed executing GPU instructions. Tiny, extremely local, managed by hardware and kernels.

Role: reuse tiles

HBM / device DRAM

The large working set directly consumed by GPU kernels: weights, KV blocks, activations and workspace.

Role: hot model state

CPU DRAM

Large host capacity for runtimes, request data, pinned transfer buffers, weight staging and KV offload.

Role: warm extension

NVMe SSD

Persistent checkpoints and a high-capacity cold tier for cached prefixes or evicted KV blocks.

Role: load / retain

02 / What the GPU can actually touch

A memory hierarchy is a data-movement contract.

The address may be visible, but performance follows the physical route. Every boundary introduces controllers, links, queues, translations, DMA setup, and synchronization.

Tier

Physical relationshipModel-state useCritical boundary

Registers / SRAM

Inside each streaming multiprocessorFragments, partial sums, attention tilesCompiler allocation, occupancy, bank conflicts

L2 + HBM

On GPU / beside GPU packageResident weights, active KV, activationsload/store traffic from GPU kernels

CPU DRAM

Across PCIe or coherent CPU–GPU linkPinned staging, KV overflow, CPU inferenceDMA, page pinning, NUMA and link contention

NVMe

PCIe storage endpoint, local or fabric-attachedCheckpoints, cold KV, restart and prefix cacheI/O commands, filesystem, block queues, flash access

NVMecheckpoint blocks

→

CPU DRAMbounce / page cache

→

PCIe DMAhost to device

→

HBMruntime layout

GPUDirect Storage can remove the CPU-DRAM bounce buffer for supported paths, allowing an NVMe device or NIC to DMA into GPU memory. It shortens the route; it does not turn NAND into HBM. The data still traverses PCIe and must be placed in device memory before normal kernels consume it at HBM rates.

03 / The practical comparison

Four tiers, four different jobs.

Tier	Volatile?	GPU consumption path	Strength	Failure mode in inference	Best state
On-chip SRAM	Yes	Native instructions	Lowest locality cost	Too small; spills/reloads	Tiles
HBM / GPU DRAM	Yes	GPU global-memory loads	TB/s-class local bandwidth	Capacity pressure; repeated scans	Hot
CPU DDR5 DRAM	Yes	DMA copy or mapped access over link	Capacity, flexibility, lower cost/GB	PCIe/link bandwidth and latency	Warm
NVMe SSD	No	I/O then DMA/staging	Persistence and TB-scale capacity	Queueing, flash latency, low random-read rate	Cold

Published peaks illustrate the scale separation, not a universal benchmark. NVIDIA lists H100 SXM with 80 GB and 3.35 TB/s of GPU-memory bandwidth, plus 900 GB/s NVLink and a 128 GB/s PCIe Gen5 headline. A DGX H100 aggregates 640 GB HBM3 and 24 TB/s across eight GPUs, while its CPU DDR5 and NVMe occupy separate capacity domains. Intel Xeon 6 supports up to twelve DDR5 channels: considerable host bandwidth, but still mediated when a discrete GPU needs those bytes.

Do not compare those headline numbers without checking directionality, payload efficiency, topology, NUMA placement, concurrency and access size. A sequential NVMe array can stream impressively; one missing 16 KB KV block on the decode critical path can still be ruinous.

04 / Weights: the fixed corpus

Weights are capacity at load time—
bandwidth at token time.

A model’s parameters look static because their values do not change during inference. But static does not mean motionless.

At startup, checkpoint tensors are read from storage, decoded or converted into the runtime’s layout, and placed into GPU memory. If the model is too large for one GPU, tensor, pipeline, or expert parallelism partitions the weights. Each rank stores only its shard, but the serving group collectively needs the complete model—or the active experts plus a mechanism to reach all possible experts.

During a forward pass, matrix-multiplication kernels repeatedly fetch weight tiles from HBM into caches, shared memory and registers. A weight may be reused across the tokens in a batch, which is why batching raises arithmetic intensity. At low-batch autoregressive decode, reuse is weak: the system can become weight-bandwidth bound, reading a large fraction of resident weights to produce one next token.

Raw weight capacityparameters × bits per stored weight ÷ 8

70B · BF16140 GB

Before scales, metadata, padding, embeddings, workspaces, KV cache or allocator reserve.

70B · 8-bit70 GB

Half the raw bytes, subject to quantization metadata and any higher-precision tensors.

70B · 4-bit35 GB

A capacity floor—not a promise that all kernels or layers use exactly four bits.

The important distinction

Weight placement traffic happens at load, reshard, failover or expert rebalancing. Weight read traffic happens inside every inference step as GPU kernels read their local shards from HBM. Tensor-parallel collectives usually transmit activations or partial results, not the entire weight shard each token.

05 / KV cache: the growing working set

The KV cache remembers this sequence.

Weights are shared across requests and fixed for a replica. KV state is created per sequence, grows with tokens, and must be read again during decode.

In causal self-attention, every layer projects a token into queries, keys and values. Earlier keys and values do not change, so caching them avoids recomputation. When a new token arrives, the attention kernel reads that token’s query and the stored keys and values for the visible history, computes attention, and appends one new K/V entry.

For a conventional cache, a useful uncompressed estimate is below. Architectures using Multi-Query Attention, Grouped-Query Attention, Multi-head Latent Attention, sliding windows, quantized KV, or linear/recurrent attention change one or more terms.

KV bytes = batch × sequence length × layers × 2 × KV heads × head dimension × bytes per elementThe factor 2 means one key tensor plus one value tensor. This is allocated state; allocator pages, fragmentation and metadata add overhead.

Worked GQA example

80 layers · 8 KV heads · head dimension 128 · BF16

320 KiB

per cached token, per sequence

8K context: 2.5 GiB
128K context: 40 GiB

Why concurrency wins the bill

The same hypothetical model serving 32 independent 8K-token sequences needs roughly 80 GiB of KV storage before paging overhead. At 128K, even a single sequence reaches 40 GiB.

tokens → linearly growing cache

If this were full multi-head attention with 64 KV heads instead of eight, the cache would be eight times larger: 2.5 MiB per token, 20 GiB at 8K, and 320 GiB at 128K for one sequence. This is why MQA and GQA matter: fewer K/V heads reduce both cache capacity and the bytes attention must read during incremental decoding.

PagedAttention addresses a different problem. It divides KV storage into blocks and maps logical sequences to non-contiguous physical pages, reducing fragmentation and enabling sharing. It can make capacity usable; it does not make attention stop reading the required K/V content.

06 / One request, two regimes

Prefill writes history.
Decode rereads it.

Phase A · Prompt ingestion

Prefill

The model processes many prompt tokens in parallel.

Weight tensors are read, but reused across a large token matrix.
Attention computes and writes K/V entries for every layer and prompt token.
Large matrix operations often produce higher accelerator utilization.
Long prompts create a burst of KV writes and capacity allocation.
Time to first token is sensitive to prompt length and compute throughput.

Phase B · Autoregressive generation

Decode

The model advances one token per active sequence per step.

Local weight shards are read again for each layer and step.
Existing K/V history is scanned by attention; one new entry is appended.
Small matrices and low reuse make memory bandwidth prominent.
Continuous batching amortizes weights across many sequences.
Time per output token is sensitive to HBM and collective latency.

The system therefore has two moving byte fronts: fixed weights repeatedly streamed through the compute pipeline, and request-specific KV state that grows and is revisited. Optimizing only capacity misses both.

07 / What “KV cache transmission” actually means

Most KV traffic is a read.
Some KV traffic is a transfer.

These are frequently conflated. Separating them explains when networking, PCIe, CPU DRAM and NVMe enter the hot path.

01 · LOCAL ATTENTION

HBM → GPU cores

On every decode step, attention reads resident K/V blocks from the same GPU’s HBM through its cache hierarchy. This is memory traffic, not a network transfer. Its volume grows with the attended context.

02 · PREFILL / DECODE DISAGGREGATION

GPU → fabric → GPU

A prefill worker builds the cache; a decode worker needs it. The serving layer transmits K/V blocks once across NVLink, InfiniBand or Ethernet/RDMA before generation continues. The transfer is roughly the cache size plus protocol and layout overhead.

03 · TIERED OFFLOAD

HBM ↔ DRAM ↔ NVMe

Cold or evicted blocks move to host memory or SSD. A reuse hit avoids recomputing prefill, but the blocks must return before attention consumes them. Value depends on reuse saved > fetch cost + stall risk.

04 · REQUEST MIGRATION

Worker A → worker B

Load balancing or failure recovery may relocate an in-flight sequence’s KV state. Migration preserves work but can put a large, latency-sensitive copy directly in the request path.

05 · PREFIX REUSE

Cache owner → requester

Shared system prompts or document prefixes can be matched and reused. Routing a request to the rank already holding the prefix can be cheaper than moving the cache or recomputing it.

06 · PARALLEL ATTENTION

Shard-local KV + collectives

With tensor or attention parallelism, each rank may own particular heads or sequence blocks. KV can remain sharded while activation reductions, exchanges, or context-parallel communication cross the fabric.

Do not say“The entire KV cache is transmitted across GPUs for every token.” That is not the default. A well-partitioned runtime keeps KV near the attention rank. Every step reads relevant KV from local HBM; bulk transmission occurs when ownership changes, stages are disaggregated, cache is remote, or the parallel algorithm requires exchange.

08 / Offload without magical thinking

Lower tiers buy capacity.
They borrow time.

Good fit

Cold prefix reuse

A large common prefix is expensive to prefill and likely to be requested again. Keeping it in CPU DRAM or NVMe can be worthwhile if asynchronous promotion finishes before the request needs it.

Good fit

Weight loading and standby

NVMe stores checkpoints; CPU DRAM stages, converts, pins or retains a warm copy. This improves startup and failover even though steady-state hot weights should remain near compute.

Danger zone

Per-layer weight streaming

Fetching large weight layers from CPU DRAM or SSD for every token can turn PCIe or storage into the throughput ceiling. Overlap helps only when transfer can stay ahead of compute.

Danger zone

Demand-paged KV on decode

A cache miss that blocks the next token pays transfer setup, queueing and bandwidth costs in the latency-critical loop. Random, fine-grained misses are far worse than planned bulk prefetch.

Offload decisionsaved recompute time + added concurrency value > transfer time + queueing + eviction cost + latency risk

09 / A model-centric design method

Ask where each byte must be
at the instant it is consumed.

Size the immutable corpus.Calculate weight bytes by precision, then add scales, padding, embeddings, runtime layout, workspace and replica count.
Size state per token.Use actual layers, KV heads, head dimension and cache dtype. Multiply by context, live sequences, beams and speculative branches.
Separate prefill from decode.Model prompt compute, cache writes, decode weight reads, historical KV reads and service-level objectives independently.
Map ownership.Record which GPU owns each weight shard and KV block. Name every collective, migration and cross-stage handoff.
Price the path, not the tier.Measure effective HBM, NVLink, PCIe, NIC, DRAM and storage throughput for the real block sizes and topology.
Reserve headroom.Account for activations, communication buffers, CUDA graphs, fragmentation, failures and admission-control slack.
Design misses explicitly.Choose what happens when HBM is full or a reused prefix is remote: wait, preempt, recompute, migrate, or reject.

The governing equation is not simply “does the model fit?” It is: can the system deliver the required weight and state bytes to the right compute rank before the token’s latency budget expires?

10 / Bottom line

Memory is where bytes live.
Inference is how they move.

HBM makes the working set fast. CPU DRAM makes the machine capacious. NVMe makes state persistent and recoverable. None substitutes for another without a movement plan.

Weights establish the fixed capacity floor, arrive from storage at load time, and become repeated HBM read traffic at inference time. Quantization reduces both footprint and potential traffic; parallelism distributes the corpus but introduces activation collectives and coordination.

KV cache is the dynamic, request-specific memory of prior tokens. It is written during prefill, grows during generation, and is reread by attention. GQA, MQA, compression, paging, sliding windows and linear attention change its size or layout. Disaggregation, migration, reuse and offload turn it into fabric, PCIe, DRAM or storage traffic.

The right mental model is a temperature map: keep bytes that are needed every microsecond in HBM, bytes worth reusing soon in CPU DRAM, and bytes valuable over longer horizons on NVMe. Then engineer promotion, eviction and ownership so that “available somewhere” becomes “ready here” before the next token is due.

Companion analysis → The Kimi K3 Paradox: why shrinking sequence state can coexist with rising GPU, HBM and fabric demand.

11 / Sources & method

Primary documentation and papers.

Published specifications are examples, not normalized benchmarks. The worked KV calculations are explicit architectural examples using the stated dimensions; real implementations add allocator, metadata and runtime overhead. Accessed 18 July 2026.

Evidence standard: hardware and software behavior is grounded in vendor documentation; architectural techniques are grounded in their papers. Quantitative examples are labeled and reproducible from the formulas shown.

“GPU memory” and HBM are not competing categories.

On-chip

HBM / device DRAM

CPU DRAM

NVMe SSD

A memory hierarchy is a data-movement contract.

Four tiers, four different jobs.

Weights are capacity at load time—bandwidth at token time.

The KV cache remembers this sequence.

Worked GQA example

Why concurrency wins the bill

Prefill writes history.Decode rereads it.

Prefill

Decode

Most KV traffic is a read.Some KV traffic is a transfer.

HBM → GPU cores

GPU → fabric → GPU

HBM ↔ DRAM ↔ NVMe

Worker A → worker B

Cache owner → requester

Shard-local KV + collectives

Lower tiers buy capacity.They borrow time.

Cold prefix reuse

Weight loading and standby

Per-layer weight streaming

Demand-paged KV on decode

Ask where each byte must beat the instant it is consumed.

Memory is where bytes live.Inference is how they move.

Primary documentation and papers.

Weights are capacity at load time—
bandwidth at token time.

Prefill writes history.
Decode rereads it.

Most KV traffic is a read.
Some KV traffic is a transfer.

Lower tiers buy capacity.
They borrow time.

Ask where each byte must be
at the instant it is consumed.

Memory is where bytes live.
Inference is how they move.