The thesis is directionally right.
The mechanism needs tightening.
Lower KV pressure does not mean lower system demand.
KDA attacks sequence-state growth. It does not erase 2.8 trillion parameters, MoE routing traffic, concurrency requirements, or the need to keep a production service highly utilized.
“More efficient” could raise aggregate hardware use.
Cheaper tokens and longer, more agentic workloads can increase total consumption. That is a credible rebound pathway—not a measurement of K3’s net market impact.
WideEP does not continually exchange expert weights.
Experts are placed or sharded across GPUs. During inference, routed token hidden states and results move through dispatch/combine collectives. The weights generally stay resident.
Do not collapse “memory” into one number.
Inference capacity is the sum of several pools with different scaling laws. KDA flattens one curve; K3 steepens others.
Model weights scale with parameter count × bits per weight. They do not grow with prompt length, but every serving replica needs access to them.
Conventional attention caches per-token keys and values. Linear attention replaces most of that unbounded history with fixed-size recurrent state, while K3’s hybrid layers retain some global attention.
Activations, routing buffers, CUDA graphs, workspace, batching slack, failures, and fragmentation consume the HBM left after weights and cache.
How heavy are K3’s weights?
At 2.8 trillion parameters, the arithmetic is unforgiving—even when each parameter is only four bits.
K3 applies quantization-aware training from supervised fine-tuning onward, using MXFP4 weights and MXFP8 activations, according to Moonshot’s launch material. A lower-bound calculation for raw 4-bit weights is:
Real checkpoints and runtime layouts also carry block scales, metadata, alignment, non-quantized tensors, embeddings, and possibly duplicated or padded structures. That makes “about 1.5 TB” a reasonable planning estimate, but not yet a verified K3 checkpoint size. At BF16, the same parameter count would be 5.6 TB before overhead.
Crucially, active parameters are a compute concept, not a storage exemption. Only 16 of 896 experts are selected per token, but the router must be able to reach any expert. Unless weights are streamed from slower memory—usually a latency disaster for high-throughput serving—the expert bank remains resident across the serving group.
Build your own weight-memory floor.
Change effective bits and overhead. This estimates weights only—no KV state, activations, buffers, replicas, or availability reserve.
A smaller memory of the sequence—not a smaller model.
Kimi Delta Attention is a linear-attention mechanism derived from delta-rule recurrent memory. Instead of keeping an ever-growing key/value record for every token in every attention layer, it updates a compact state as tokens arrive. That makes decode-state memory approximately constant for the KDA layers rather than linear in sequence length.
Kimi Linear used a 3:1 mix of KDA to global Multi-Head Latent Attention (MLA). The peer-reviewed claim attached to that research model is up to 75% less KV cache and up to 6× higher decoding throughput at a one-million-token context. “Up to” matters: the gain depends on context, batch shape, kernels, hardware, and the chosen baseline.
A finite state cannot preserve arbitrary token history exactly. Hybrid global-attention layers and learned update rules trade perfect storage for useful memory.
Moonshot says KDA creates new challenges for conventional prefix caching and contributed a KDA-aware prefill-cache implementation to vLLM.
CPU DRAM and NVMe can extend cache capacity in a disaggregated system, but “K3 must offload” does not follow from model size alone. A 64-GPU pool may have substantial aggregate HBM.
Wide expert parallelism moves tokens to weights.
The original passage gets the pressure right and the payload wrong.
Place the experts
Expert parallelism distributes whole experts across ranks; hybrid EP/TP can further shard an expert. Weight loading and expert placement happen at startup or during explicit rebalancing—not for every generated token.
Dispatch hidden states
The router selects experts. If an expert lives on another GPU, the token’s hidden-state vector, expert ID, and routing scale are sent across the fabric. Results are combined and returned to the token’s attention rank.
Pay the collective tax
WideEP uses optimized All-to-All dispatch/combine. NVIDIA describes message volume as scaling with top_k × tokens per rank × hidden dimension, rather than the full 2.8T weight footprint.
What 1.5 TB looks like on current accelerator platforms
Illustrative capacity math only. Production deployments need space for runtime state and may use different replicas, parallelism plans, or precisions.
11 GPUs fit 1.5 TB in aggregate, in theory. Eight H200s provide about 1.13 TB: not enough for that estimate.
64 GPUs → 9.0 TB aggregate HBM9 GPUs fit 1.5 TB in aggregate, in theory. One 8-GPU DGX B200 provides 1.44 TB—just below 1.5 TB before runtime overhead.
64 GPUs → 11.5 TB aggregate HBM6 GPUs fit 1.5 TB in aggregate, in theory. One 8-GPU HGX B300 provides about 2.30 TB.
64 GPUs → 18.4 TB aggregate HBMNVIDIA’s current reference table lists 141 GB HBM3e for H200, 180 GB for B200, and 288 GB for B300. Its GB200 NVL72 page lists 13.4 TB across a 72-GPU NVLink domain. Vendor “GB” values are decimal; software reporting and reservable capacity will differ.
Is this really Jevons’ paradox?
Possibly—but architecture alone cannot prove it.
What must be true
A rebound occurs when efficiency lowers the effective price of AI enough that the percentage increase in use outweighs the percentage decrease in resources per unit. A full Jevons paradox is the strong case: total resource consumption rises.
Why K3 is a credible trigger
Longer context, cheaper cache-hit input, faster million-token decode, and agents that work for hours can expand the unit of consumption from “one answer” to “one completed project.” The service may use fewer bytes per token while processing vastly more tokens and tool loops.
What the evidence says
Stanford documented a 280× fall in the cost of GPT-3.5-level inference from late 2022 to late 2024. The IEA projects data-centre electricity demand to roughly double from 2024 to 2030 even as task-level efficiency improves. That supports the broad rebound concern, but it does not isolate K3’s causal impact.
The bill is their product.”
KDA changes the constraint.
It does not abolish the machine.
The quoted thesis survives scrutiny after three edits.
- Say “up to 75% lower KV cache,” not a verified 10× reduction, until K3-specific evidence is public.
- Say WideEP exchanges routed activations, not weights, during the hot inference path.
- Present CPU/NVMe KV offload as an option, not an inevitable consequence of K3’s weight size.
With those corrections, the paradox becomes sharper: K3’s attention efficiency makes extreme context economically usable, while its enormous sparse capacity makes rack-scale deployment valuable. One optimization reduces bytes attached to the past; the architecture increases the infrastructure needed to make the whole model fast, concurrent, and reliable.
Primary sources first.
This analysis was researched on 18 July 2026, two days after K3’s announcement and before the planned full-weight release. K3-specific storage and deployment measurements may change when checkpoints, configuration files, kernels, and reproducible serving recipes are public. Calculations labeled “estimate,” “proxy,” or “illustrative” are derived from disclosed parameter counts and vendor capacity specifications.
- 01Kimi K3 launch / technical blog↗
- 02Kimi Linear paper↗
- 03MoonshotAI Kimi Linear repository↗
- 04Mooncake serving paper↗
- 05TensorRT-LLM parallelism guide↗
- 06TensorRT-LLM one-sided All-to-All↗
- 07NVIDIA HGX platform specifications↗
- 08NVIDIA GB200 NVL72↗
- 09IEA: Energy demand from AI↗
- 10Stanford AI Index 2025↗