K3 is disclosed—
not yet fully specified.
Moonshot’s launch post names the architecture, counts, high-level mechanisms and headline payoffs. It also says the full K3 technical report is forthcoming. A rigorous reading must separate K3 facts from evidence inherited from component papers.
The public record is unusually rich for a launch announcement but incomplete for exact reproduction. Moonshot states that K3 has 2.8 trillion parameters, native vision, a one-million-token context window, KDA, Attention Residuals, Stable LatentMoE, Gated MLA and quantile balancing. It reports effective activation of 16 among 896 experts and an approximate 2.5× improvement in overall scaling efficiency relative to Kimi K2.
Separate Kimi papers describe KDA and AttnRes in controlled models. Those experiments explain why the components are plausible; they are not automatically K3 end-to-end measurements. The exact K3 layer schedule, dimensions, state sizes, routing implementation and per-component ablations remain undisclosed as of publication.
Launch-level facts
2.8T total parameters; 1M context; KDA and AttnRes; Stable LatentMoE; effective 16-of-896 expert activation; quantile balancing; Gated MLA; SiTU; per-head Muon; MXFP4/MXFP8 QAT.
Tested elsewhere
Kimi Linear’s KDA/MLA hybrid and Attention Residuals have dedicated papers with controlled scaling, quality and efficiency results. They support mechanisms, not an unreported K3 ablation.
Await the report
Exact K3 state and layer dimensions, active parameter count, router equations, balance guarantees, training-token budget, component-level gains and production throughput.
Sequence. Depth. Capacity.
Then placement.
Each mechanism attacks a different scaling failure, and each hands a new systems problem to the next layer.
The architectural payoff is multiplicative only if all four stages remain healthy.
A compact sequence state lowers the long-context memory slope. Better depth-wise access can improve how efficiently the network uses its layers. Sparse experts enlarge representational capacity without executing every parameter. Balanced routing stops that sparsity from collapsing into hot experts, padded batches and idle accelerators.
But none abolishes the cost created by the others. A 2.8T model still needs its weight corpus placed across a large accelerator domain. Sparse activation saves arithmetic, not necessarily resident weight capacity. More experts make routing and all-to-all communication harder. Learned residual retrieval adds state and communication that must be engineered. K3 is best understood as a co-designed stack, not as a claim that scale has become cheap.
A bounded memory for most tokens—
exact attention where it matters.
Full causal attention preserves every prior key and value and revisits the visible history. Linear attention instead carries a fixed-size recurrent state. KDA makes that state more selective; the hybrid keeps periodic global attention as an escape hatch.
KDA extends Gated DeltaNet. Conceptually, each token proposes a key–value association, measures what the current state already predicts for that key, and writes a correction—the delta—rather than blindly accumulating another outer product. A learned forget gate controls retention. KDA’s defining change is finer-grained gating, giving the model more control over how limited recurrent memory is allocated.
The Kimi Linear paper couples KDA with Multi-head Latent Attention in a 3:1 layerwise hybrid. Moonshot’s K3 launch diagram likewise depicts KDA and Gated MLA in a repeating block, but exact K3 dimensions await the report. The division of labor is the key idea: KDA provides a context-length-independent recurrent state on most layers; global latent attention layers preserve token-addressable recall and reduce the risk that compression erases crucial details.
Query the state
The current query reads a compact matrix state rather than scanning a KV record for every earlier token.
Inspect existing memory
The key probes what the state already contains, creating a prediction for the association being written.
Write the delta
The update stores the difference between the new value and the state’s current prediction, reducing interference.
Gate retention
Fine-grained decay lets the model retire stale information and use finite state more selectively.
This is an explanatory delta-rule sketch, not the undisclosed K3 implementation equation. KDA’s paper adds a specialized diagonal-plus-low-rank transition structure and chunkwise algorithm for efficient parallel training.
Flatten the KV slope
A recurrent KDA layer carries fixed-size state instead of a KV history growing linearly with sequence length. Global attention layers still cache state, so the hybrid reduces—not eliminates—KV growth. In the 48B Kimi Linear model, Moonshot reports up to 75% lower KV-cache use.
Decode without a full scan
At very long context, a fixed-state recurrent update avoids rereading an ever-longer history on most layers. Kimi Linear reports up to 6× decoding throughput versus full MLA at one million tokens; this is component-model evidence, not a published K3 serving number.
Finite state can collide
Compressing an unbounded history into bounded state creates interference and imperfect recall. Delta correction and gating improve utilization, while periodic global attention restores a direct route to particular tokens.
Linear FLOPs do not guarantee speed
Recurrent dependencies, state layout, precision and chunking determine real utilization. KDA needs specialized kernels and complicates conventional prefix caching; Moonshot explicitly flags both integration concerns.
Do not merely accumulate depth.
Retrieve from it.
A standard residual stream treats every layer output as a unit-weight addition. Attention Residuals lets a layer choose which earlier representations it needs, using content-dependent softmax weights across depth.
In a PreNorm transformer, the residual stream repeatedly adds block outputs. The AttnRes paper argues that its magnitude grows with depth, while any individual layer’s contribution becomes diluted. The network has preserved information—but made precise access harder because old and new features are superposed through fixed additions.
AttnRes replaces that fixed accumulation with attention over previous layer outputs. The current representation produces selection weights; the layer receives a weighted mixture of earlier representations. This is not token-to-token attention. It is layer-to-layer retrieval for the same token position: memory across model depth, not sequence length.
Block AttnRes is the scale-enabling compromise.
Attending over every preceding layer output is expensive in a deep distributed model. Block AttnRes partitions layers into groups and retrieves block-level representations, retaining most reported gains while sharply reducing saved-state and communication overhead. The paper further introduces cached pipeline communication and a two-phase computation strategy.
On a 48B-total/3B-active Kimi Linear model trained for 1.4T tokens, the authors report improvements across all evaluated downstream tasks and more uniform signals through depth. That provides strong component evidence. It does not disclose how much of K3’s approximate 2.5× scaling-efficiency claim comes from AttnRes alone.
Compress the expert interface.
Spend the savings on diversity.
Standard MoE routes full-width hidden vectors into a small subset of feed-forward experts. LatentMoE projects tokens into a narrower shared latent space before routing and expert computation, then projects the result back.
The attraction is hardware-aware. In low-batch inference, experts often load substantial weights for few tokens and become HBM-bandwidth bound. Expert parallelism also dispatches and gathers activation vectors through all-to-all communication. If the expert pathway has width ℓ instead of model width d, the LatentMoE paper says routed parameter loads and communication fall by roughly d/ℓ.
Those savings can be banked as lower cost or reinvested: increase the expert count and the top-k selected experts while holding approximate inference cost fixed. More possible expert combinations increase conditional expressivity. K3 applies a framework Moonshot calls Stable LatentMoE; the launch post confirms the name and effective expert count, but has not yet published what “Stable” adds to the underlying architecture.
A dense residual representation enters the MoE block at width d.
The token moves through a smaller latent interface ℓ; selected experts compute in that space.
Weighted expert outputs return to model width and rejoin the dense path.
“Only 1.8% of experts activate” is mathematically accurate for 16 ÷ 896, but it does not imply that only 1.8% of K3’s 2.8T parameters are read or computed. Dense attention, embeddings, projections, norms, shared experts and any other always-on paths remain active. Nor has Moonshot published K3’s total activated-parameter count. The defensible statement is: K3 effectively selects 16 of 896 routed experts per token.
Many specialized subspaces
896 routed experts create enormous conditional diversity. Each token selects a tiny subset, so total parameter capacity can grow much faster than per-token expert compute.
Narrower dispatch payload
A latent expert interface can reduce token bytes sent through expert-parallel all-to-all and shrink each expert’s parameterization, improving accuracy per parameter and per FLOP.
Cold experts still occupy memory
Every expert must be available somewhere because future tokens may select it. Sparse execution reduces active work, while total weights still drive aggregate HBM capacity and placement.
Small GEMMs are difficult
More, narrower experts mean smaller per-expert batches. Kernel launch overhead, poor matrix utilization and weight-loading cost can erase theoretical FLOP savings without batching and specialized kernels.
A router decision is also
a scheduling decision.
At 896 experts, “choose the best 16” is not merely a modeling operation. It determines how many tokens, bytes and matrix multiplies land on every accelerator rank.
If router preferences concentrate on a few experts, those ranks become stragglers while others wait. Capacity limits may drop or reroute tokens; padding wastes computation; dynamic shapes complicate graph capture; network links see skewed traffic. Classical MoE systems therefore add auxiliary losses, per-expert biases or iterative control rules to encourage balance.
Moonshot says K3’s Quantile Balancing derives expert allocation directly from router-score quantiles, removing heuristic updates and a sensitive balancing hyperparameter. The key systems interpretation is rank-based calibration: allocation thresholds follow the observed score distribution, so expert loads can be disciplined without a hand-tuned feedback knob.
Preserve preference
The router produces affinities between tokens and experts. High scores represent the model’s learned specialization signal.
Normalize by distribution
Score ranks or quantile-derived cutoffs provide an allocation basis that is less hostage to raw-score scale across experts.
Meet physical capacity
Selections are converted into expert batches that can remain balanced enough for static execution and predictable collective traffic.
Balance buys determinism, but routing quality remains the constraint.
The ideal router sends every token to its most useful experts and gives every device equal work. Those objectives can conflict. Hard balancing can force semantically inferior assignments; soft balancing may leave a long-tail straggler. Quantile calibration is valuable because it removes an unstable control loop, not because the quality–systems tradeoff disappears.
The payoff compounds at large expert-parallel scale: tighter per-rank batch shapes, less padding, more predictable all-to-all volume, fewer stragglers and easier graph optimization. This is why quantile balancing belongs in the architecture discussion. At 896 experts, infrastructure behavior feeds directly back into achievable model quality and training stability.
The gains are coupled.
So are the costs.
| Mechanism | Scarce resource | Architectural payoff | Systems payoff | New cost / risk |
|---|---|---|---|---|
| KDA hybrid | Sequence memory | Selective finite-state memory plus periodic exact global recall | Lower KV growth and long-context decode traffic | State interference, specialized kernels, prefix-cache complexity |
| Attention Residuals | Useful depth | Content-dependent access to earlier representations | More capability from depth; healthier optimization | Saved activations, depth attention and pipeline communication |
| Stable LatentMoE | Conditional capacity | More expert diversity at controlled active compute | Narrower token dispatch and smaller expert interfaces | Huge resident corpus, small GEMMs, expert placement |
| Quantile Balancing | Parallel utilization | Specialization under disciplined allocation | Static shapes, less skew, predictable all-to-all | Potential preference-versus-balance tension |
Moonshot attributes an approximate 2.5× overall scaling-efficiency improvement over K2 to the structural changes plus refined training and data recipes.
“Scaling efficiency” in the launch post is not defined as a single hardware benchmark, and Moonshot does not apportion it among components. It should be read as a model-development claim about converting training compute into overall capability—not as 2.5× fewer GPUs, 2.5× faster inference, or 2.5× lower cost.
The deployment recommendation tells the other half of the story: Moonshot recommends supernodes with 64 or more accelerators because larger high-bandwidth communication domains help inference. KDA reduces sequence-state pressure, but Stable LatentMoE introduces an enormous distributed weight and routing problem. Efficiency at one axis enables ambition at another.
What would make the claims
fully auditable?
- Publish the exact block recipe.How many KDA, Gated MLA, MoE and AttnRes blocks; what dimensions, state sizes, head counts and latent ratios?
- Separate total, routed and activated parameters.Report dense-path, shared-expert and selected-expert contributions to per-token weight reads and FLOPs.
- Ablate each innovation.Hold data and compute constant while removing KDA, AttnRes, latent experts and quantile balancing individually and in combinations.
- Define scaling efficiency.State the fitted law, compute accounting, evaluation mixture and uncertainty behind the approximate 2.5× comparison with K2.
- Show the router frontier.Plot quality, expert-load skew, dropped or reassigned tokens, padding, network volume and straggler time.
- Report real long-context serving.Disclose KV/state bytes, prefix-cache behavior, prefill and decode throughput, TPOT, concurrency and fabric topology.
- Quantify AttnRes overhead.Give activation memory, pipeline bytes, FLOPs and latency for full versus block variants at K3 depth.
Innovation is not subtraction.
It is reallocation.
K3’s architecture reallocates memory, compute and communication toward places where they purchase more model capability.
KDA exchanges a token-by-token archive on most layers for selective recurrent state, while global latent attention preserves exact recall. Attention Residuals replaces indiscriminate accumulation across depth with learned retrieval. Stable LatentMoE narrows the expert interface so vastly more conditional capacity can fit within an active-cost envelope. Quantile Balancing turns that mathematical sparsity into executable, balanced work.
The four ideas are genuine architectural advances, but careful wording matters. Sixteen of 896 routed experts means 1.7857% of expert slots—not 1.8% of the whole model. Kimi Linear’s 75% KV reduction and 6× decode result are evidence for the component architecture—not published K3 measurements. “Stable” LatentMoE and Quantile Balancing remain only partially specified pending the report.
The most important payoff is not that K3 needs little hardware. It is that each token can draw on enormous learned capacity, survive extreme context and reuse intermediate representations without paying the naïve dense cost on every axis. That efficiency is what makes a 2.8T sparse model conceivable—and what makes the remaining weight-placement, expert-routing and fabric problem a first-class part of the model itself.
Primary sources first.
The K3-specific claims come from Moonshot’s official launch post. Component mechanisms and quantitative results come from their respective papers and repositories. Claims not yet disclosed in the K3 report are explicitly marked. Accessed 18 July 2026.
- 01Moonshot AI — Kimi K3: Open Frontier Intelligence↗
- 02Kimi Linear: An Expressive, Efficient Attention Architecture↗
- 03MoonshotAI/Kimi-Linear — code, checkpoints and kernels↗
- 04Attention Residuals↗
- 05MoonshotAI/Attention-Residuals repository↗
- 06LatentMoE: Toward Optimal Accuracy per FLOP and Parameter↗
- 07Gated Delta Networks: Improving Mamba2 with Delta Rule↗
- 08DeepSeek-V2 — Multi-head Latent Attention and MoE context↗
- 09Outrageously Large Neural Networks — sparse MoE foundations↗
- 10Switch Transformers — routing and load balancing↗