Architecture deep dive · 18 July 2026

Kimi Delta
Attention

Softmax attention keeps the past as a growing archive of keys and values. KDA keeps a fixed-size, editable associative memory—then pairs it with occasional global attention so compression does not have to solve every retrieval problem alone.

Delta rule · Channel-wise decay · Hybrid attention · FlashKDA
Fixed KDA stateIndependent of context length
3:1 hybridThree KDA layers, one MLA layer
Channel-wise decayDifferent memory lifetimes
Two execution modesChunked prefill, recurrent decode
01 / The one-sentence model

KDA turns attention from
searching an archive into updating a memory.

It does not preserve every earlier key and value. It learns how to compress their useful associations into a matrix, how strongly to overwrite a matching association, and how quickly each feature channel should fade.

In ordinary causal softmax attention, every new token compares its query with the stored keys of earlier tokens, weights their values, and appends another key/value pair to the cache. The history remains directly addressable, but both cache capacity and decode-time reads grow with sequence length.

Linear attention changes the order of operations. Instead of retaining all pairs, it folds each key–value association into a recurrent state S. A query reads from that state. The state has fixed dimensions, so a KDA layer’s recurrent memory does not expand from 4K to 128K to one million tokens.

The bargain is fundamental: a fixed state is a lossy summary. Multiple facts can collide in the same finite memory. KDA’s delta update and fine-grained forgetting are designed to use that capacity better; Kimi Linear’s periodic global-attention layers provide a second path for exact, content-addressed recall.

Terminology matters. KDA is the attention operator. Kimi Linear is the published model architecture that interleaves KDA and Multi-Head Latent Attention (MLA). Speed and cache results for Kimi Linear are not automatically results for every model that might use KDA.
02 / From additive memory to corrective memory

The delta rule asks:
“What does memory already predict?”

The easiest linear memory only adds. DeltaNet first reads the old answer at the incoming key, then writes the error. That correction is the central idea KDA inherits.

Plain linear attention: St = St−1 + ktvt⊤  ;  ot = St⊤qtEvery association is added. Similar keys accumulate and interfere; there is no targeted erase.
Delta rule: St = St−1 + βtkt(vt − St−1⊤kt)⊤Read the current prediction for kt, measure its error against vt, and write only a β-scaled correction. With normalized keys, this is online gradient descent on reconstruction loss.
01

Form a key

The token produces normalized k, a location or address in the associative state.

address
02

Read first

The old state predicts a value at that address: S⊤k.

current estimate
03

Compute error

The desired value minus the prediction reveals what is missing or wrong.

v − S⊤k
04

Correct

A learned scalar β controls how aggressively the rank-one update edits memory.

βk(error)⊤
Additive LA
Store: keep adding correlationsRisk: old and conflicting associations accumulate
DeltaNet
Store: correct the mapping at a keyGain: targeted overwrite rather than blind addition
Gated DeltaNet
Forget: one learned scalar decay per head/tokenGain: obsolete state can disappear
KDA
Forget: learned decay per key channelGain: many memory timescales inside one head
03 / KDA’s specific innovation

One head no longer has
one forgetting clock.

Gated DeltaNet multiplies a head’s prior state by a scalar α. KDA replaces that coarse knob with a diagonal vector gate: every key dimension can decay differently at every token.

Moonshot writes KDA’s recurrence as St = (I − βtktkt⊤) Diag(αt)St−1 + βtktvt⊤. Read right-to-left: first decay rows of the state independently; then apply the delta-rule erase/correction along the current key; finally write the new key–value association.

If one channel’s α stays near one, information encoded there persists. If another is lower, it turns over quickly. The gate is input-dependent, so the model can change these lifetimes with content rather than using a fixed schedule.

This is also KDA’s implicit positional mechanism. Products of successive decay matrices make the influence of an earlier key depend on the learned transitions between its position and the query. Kimi Linear therefore uses NoPE in its global MLA layers and delegates recency/position awareness to KDA. That is a design choice supported by the paper’s ablations—not proof that explicit positional encoding is universally unnecessary.

MechanismState at decodeUpdateForget controlDirect token retrieval
Softmax attentionAll cached K/V tokensAppendAttention weights at readYes
Plain linear attentionFixed matrixAdd correlationNoneNo; compressed
DeltaNetFixed matrixCorrect reconstruction errorTargeted delta erase/writeNo; compressed
Gated DeltaNetFixed matrixDelta ruleScalar per head/tokenNo; compressed
KDAFixed matrixDelta ruleChannel-wise α; scalar βNo; compressed
04 / Why “hybrid” is the honest answer

Compression handles the stream.
Global attention keeps the receipts.

Pure linear attention has historically struggled with exact copying and fine-grained retrieval. Kimi Linear repeats three full KDA layers followed by one global MLA layer.

Layer 1 · compressedKDAFixed recurrent state
Layer 2 · compressedKDAFixed recurrent state
Layer 3 · compressedKDAFixed recurrent state
Layer 4 · addressableGlobal MLAGrowing latent KV cache

Why layerwise?

The team reports choosing whole alternating layers over mixing head types inside a layer for infrastructure simplicity and training stability. A regular pattern also makes cache ownership predictable.

Why 3:1?

In a matched ablation, 3:1 achieved validation perplexity 5.65 versus 5.66 at 1:1, 5.70 at 7:1 and 5.77 for full MLA. More global layers cost inference; too few hurt generalization.

Why MLA?

Multi-Head Latent Attention compresses K/V representations but still retains per-token latent cache. Its periodic layers can directly compare a query against historical token positions that KDA has summarized.

The 75% intuition: in a repeating block, three of four attention layers replace length-growing K/V with fixed recurrent state. Only the MLA layer maintains token-by-token cache. The real byte reduction is “up to 75%,” not a law: state tensors, cache layouts, precision, allocator overhead and any non-repeating layers still count.
05 / The measured payoff

Long context stops taxing
every layer on every token.

Published hybrid cache claimup to 75%

Lower KV-cache usage from the 3:1 KDA/MLA design.

1M-token TPOT comparison6.3×

1.84 ms versus 11.48 ms for MLA in the report’s throughput-oriented setting.

Batch-one decode at 1M2.9×

Reported Kimi Linear advantage over MLA in Figure 7’s separate latency test.

128K long-context average54.5

Versus 52.2 for matched MLA across the report’s benchmark suite.

Published fact

A KDA head maintains a fixed dk × dv state; the experiments use dk = dv = 128. Decode therefore updates and reads a fixed state rather than scanning a KDA-layer cache proportional to context length.

Published fact

At 128K, the 1.4T-token Kimi Linear run scored 84.3 on RULER and 68.5 on RepoQA versus MLA’s 81.3 and 63.0. Its aggregate was higher, but it did not win every task: MLA led LongBench V2 and Frames.

Published fact

The later 5.7T-token released base checkpoint reports RULER scores of 95.4 at 128K and 94.8 at 1M. These are model-level results, not an isolated KDA operator ablation.

Systems inference

Freed HBM can admit more sequences, larger batches or longer prompts. Whether that becomes throughput, lower cost or simply headroom depends on weight footprint, MoE routing, scheduler, global-attention cache, fabric and latency target.

Systems inference

KDA reduces sequence-state growth; it does not shrink model weights. At short contexts, projections, experts and kernel launch overhead can dominate, so the advantage naturally becomes more visible as the ordinary attention scan grows.

Do not conflate

The report uses “6× throughput” for its optimized 1M-token configuration and also shows a 2.9× batch-one TPOT result. Both can be true under different batching/memory assumptions; neither should be quoted without its setup.

06 / From recurrence to GPU kernels

The math is sequential.
The implementation does not have to be.

Autoregressive decode wants the one-step recurrence. Training and prefill want thousands of tokens in parallel. KDA supplies both views of the same state update.

For decode, each token decays and edits the running state, then reads its output. Cost and state size are independent of earlier sequence length for KDA layers. For prefill, executing that recurrence token by token would underuse a GPU.

Moonshot derives a chunkwise algorithm: tokens inside a block are organized into matrix operations, while a compact transition carries state between chunks. The constrained diagonal-plus-low-rank structure binds the low-rank erase vectors to the key. According to the report, this avoids two secondary chunking operations and roughly three matrix multiplications compared with a general DPLR implementation; its benchmark approached 2× DPLR kernel speed up to 64K.

This is a hardware-algorithm co-design, not merely an asymptotic claim. The linear FLOP expression still contains head-dimension and chunk-size terms. Layout, numerical range, fusion, Tensor Core utilization and variable-length batching determine whether theoretical linearity becomes wall-clock speed.

Prefill / training

Chunk-parallel KDA

  • Process blocks of tokens with matrix multiplications.
  • Separate intra-chunk interactions from inter-chunk state.
  • Parallelize enough work to occupy the accelerator.
  • Carry only the final recurrent state across block boundaries.
Autoregressive generation

Fused recurrent KDA

  • Consume one new q/k/v and gate set.
  • Decay, correct and write the state in-place.
  • Read a fixed amount of recurrent memory per KDA head.
  • Avoid scanning a length-proportional K/V history.
2026 implementation update: Moonshot’s open-source FlashKDA uses CUTLASS kernels and integrates as an FLA backend. Its H20 benchmark reports 1.85–2.31× over the FLA chunk-KDA baseline for T=8192 across listed fixed/variable-length cases, and 1.17–1.43× over the listed scalar-gate GDN kernel. Those are forward-kernel microbenchmarks—not whole-model serving speedups.
07 / Prefix caching and state transport

A cached prefix becomes
a checkpoint, not an archive.

KDA can reuse a prefix by saving the recurrent state after that prefix. A Kimi Linear prefix checkpoint contains KDA states for linear layers and ordinary latent K/V blocks for its global MLA layers.

Shared prefixSystem prompt, policy, repository map or long document
→
Snapshot stateKDA final matrices + MLA prefix KV + convolutional state required by the runtime
→
Continue branchRestore the exact boundary state and ingest request-specific suffix tokens
What improves

Small, fixed KDA payload

For KDA layers, the prefix’s recurrent state does not grow with prefix length. Moving or restoring it can be far cheaper than transporting tokenwise K/V for those layers.

What remains

Global-layer KV

The hybrid’s MLA layers still need their cached per-token latent state, so a million-token prefix is not reduced to one tiny universal vector.

What is exact

Continuation from a boundary

Restoring the state produced by the exact same prefix reproduces the model’s continuation state, subject to implementation precision and deterministic execution.

What is not promised

Arbitrary editing or composition

The recurrent update is ordered and generally non-invertible. A compressed state cannot expose, delete or splice an arbitrary old token the way a token-indexed cache can.

Evidence boundary: the Kimi Linear release exposes initial and final recurrent states and vLLM-compatible caching interfaces. The operational prefix-checkpoint design above follows from those semantics. The report does not publish a universal prefix-cache hit-rate, migration-speed or cross-host serving benchmark.
08 / Costs, failure modes and design questions

Constant memory is not
infinite memory.

  1. Finite capacity creates interference.Many historical associations share a dk × dv matrix. Fine-grained decay allocates lifetimes more intelligently; it does not make the representation lossless.
  2. Exact recall still needs a path.The 3:1 hybrid preserves periodic global attention precisely because copying and fine-grained retrieval remain difficult for pure recurrent linear attention.
  3. Gating increases implementation complexity.Channel-wise cumulative decays can create numerical-range problems. Stable, fast chunkwise training requires careful formulations, precision choices and dedicated kernels.
  4. The state is sequential across chunks.Chunking exposes parallel work inside blocks, but state dependencies remain between blocks and during token-by-token decode. Sequence parallelism is not free.
  5. The hybrid keeps two memory systems.Serving software must manage recurrent matrices, convolution state and global-layer KV pages together—plus prefix reuse, migration, batching and quantization policies.
  6. Benchmarks are architecture-specific.Moonshot’s results use a 48B-total/3B-active MoE, matched recipes and particular hardware/runtime settings. A different head count, ratio, model width or kernel stack can move the crossover point.
  7. Later work identifies open design space.Gated DeltaNet-2 and FG²-GDN argue that KDA’s scalar β still couples erase/write strength or lacks per-channel adaptation. These are credible research extensions, not evidence that KDA’s published gains disappear.
09 / Bottom line

KDA’s payoff is not
“remember everything for free.”

It is a better allocation of memory: compress the continuous stream into editable fast weights, reserve expensive token-addressable storage for periodic layers, and execute each phase with the kernel shape it wants.

The delta rule makes the recurrent state corrective: overwrite a key’s wrong prediction rather than blindly adding another association. Channel-wise decay gives one head multiple learned lifetimes. The 3:1 hybrid accepts compression’s limits and retains global MLA for exact retrieval. The chunk/recurrent duality makes the same operator trainable in parallel and cheap to advance during decode.

The result is a credible way to decouple most sequence-state capacity and decode I/O from context length. It can turn HBM previously consumed by KV cache into concurrency and throughput. But weights, global-layer KV, recurrent states, expert communication and serving overhead remain. KDA changes the long-context scaling term; it does not repeal the rest of the system.

That is the durable architectural lesson: the best efficient attention may be a division of labor—learned compression where continuity matters, exact lookup where fidelity matters, and kernels designed around how each form of memory actually moves through hardware.

Companion field guide → See how weights and KV state move through HBM, CPU DRAM and NVMe from the model’s perspective.
10 / Sources & evidence standard

Primary reports, code and papers.

All quantitative claims are attributed to their published experimental setting. “Systems inference” labels mark consequences derived from the architecture rather than directly benchmarked by the cited work. Accessed 18 July 2026.

  1. 01Kimi Linear technical report — KDA, hybrid design, results↗
  2. 02MoonshotAI Kimi Linear — checkpoints and deployment↗
  3. 03FLA KDA operator implementation↗
  4. 04FlashKDA CUTLASS kernels and API↗
  5. 05FlashKDA H20 forward-kernel benchmark↗
  6. 06FlashKDA v1 kernel design deep dive↗
  7. 07Gated Delta Networks — gating plus delta rule↗
  8. 08DeltaNet / linear transformers as fast-weight programmers↗
  9. 09Gated Linear Attention — channel-wise gates and chunking↗
  10. 10RULER long-context benchmark↗
  11. 11DeepSeek-V2 — Multi-Head Latent Attention↗
  12. 12Gated DeltaNet-2 — decoupled erase and write↗
  13. 13FG²-GDN — finer-grained delta adaptation↗
  14. 14Comparative study of KDA and delta-rule architectures↗
Reading guide: equations are algebraically equivalent restatements of the papers’ recurrences. Illustrations of “memory channels” are conceptual, not visualizations of learned KDA activations. The prefix-caching section distinguishes exposed recurrent-state semantics from deployment-level inference.