Equation-first explainer · 18 July 2026

Kimi Linear,
decoded.

The slide’s dense formula describes a small, editable memory. At every token, KDA ages the old state, erases its wrong prediction, writes a correction, and reads with a query. This is why most Kimi Linear layers do not need a growing token-by-token KV cache.

KDA · Delta rule · Associative memory · Hybrid MLA · Long context
Fixed statedₖ × dᵥ per KDA head
Fine-grained αOne decay per key channel
Scalar βCorrection strength per token/head
3:1 scheduleKDA · KDA · KDA · global MLA
01 / What the two slides are saying

Attention becomes
memory maintenance.

Ordinary softmax attention asks: “Which earlier tokens should this token look at?” KDA asks a different question: “What compact mapping has the sequence taught me so far, and how should I edit it now?”

In a conventional attention layer, every past token contributes a key and value. During autoregressive generation those tensors are retained in a KV cache. A new query scores the historical keys and mixes their values. The archive grows with context length, and each decode step must interact with an increasingly large history.

Kimi Delta Attention (KDA) is recurrent linear attention. It compresses the past into a matrix S. The matrix behaves like “fast weights”: a transient associative map created from the current sequence, not a permanent learned model parameter. Give it a query-like direction and it returns a value-like vector.

That compression changes the scaling term. A KDA layer advances a fixed-size state instead of appending another cache record. But compression is not lossless. Kimi Linear therefore alternates three KDA layers with one global Multi-Head Latent Attention (MLA) layer, retaining a periodic exact token-addressable path.

Plain-English translation: KDA is a learned whiteboard, not a filing cabinet. It continually revises a finite summary of the stream. Global MLA is the filing cabinet Kimi keeps every fourth layer when exact historical lookup matters.
02 / Read the equation from left to right

One line contains
four operations.

Sₜ = (I − βₜkₜkₜᵀ) Diag(αₜ) Sₜ₋₁ + βₜkₜvₜᵀoₜ = Sₜᵀqₜ

State shape: Sₜ ∈ ℝdₖ × dᵥ. Output shape: oₜ ∈ ℝdᵥ. The paper L2-normalizes queries and keys; β is sigmoid-bounded, while the diagonal decay α is data-dependent and channel-wise.

Sₜ

The memory

A fixed matrix per head. Rows live in key-feature space; columns live in value-feature space. It stores compressed associations accumulated through the sequence.

αₜ

The lifetimes

A vector of dₖ retention factors. Diag(αₜ) scales key channels independently, letting different directions forget at different rates.

βₜ

The edit strength

A scalar for this token and head. Near zero means barely edit; near one means strongly replace the state’s answer along the current key direction.

q,k,v

Address and content

k selects where memory is corrected, v supplies the desired content, and q reads a mixture from the updated state to produce o.

I is not stored data. It is the identity matrix—the “leave everything unchanged” transform. Subtracting βkkᵀ from I creates a rank-one erase aimed specifically along k, rather than wiping the whole state.
03 / The update cycle

Age. Predict. Correct.
Then retrieve.

Step 1 · decay

Age the old memory

S̄ = Diag(αₜ)Sₜ₋₁

Each row/key channel gets its own retention. A low α rapidly clears that direction; an α near one preserves it. The meanings of channels are learned—not hand-labelled “syntax” or “facts.”

Step 2 · predict

Ask what k already returns

r = S̄ᵀkₜ

The decayed state makes a value prediction r at the current key. If the association is already correct, little needs to change.

Step 3 · delta edit

Write only the error

Sₜ = S̄ + βₜkₜ(vₜ − r)ᵀ

The target minus prediction is the delta. The outer product writes that correction along k while minimizing collateral change in orthogonal directions.

Step 4 · read

Query the new state

oₜ = Sₜᵀqₜ

A query aligned with stored key directions retrieves their associated value features. Similar queries can generalize; collisions can interfere.

The slide’s form(I − βkkᵀ) S̄ + βkvᵀ

Erase what the old state predicts at k; then add the desired value at k.

≡
The intuitive formS̄ + βk(v − S̄ᵀk)ᵀ

Keep the state, then write the reconstruction error. The two expressions are algebraically identical.

Why it is called the delta rule: the update is a gradient step on ½‖Sᵀk − v‖². It does not blindly add another key–value correlation. It trains the temporary memory online to predict v from k, correcting the prediction error each token creates.
04 / A two-dimensional worked example

β controls how far
memory moves toward truth.

Assume a normalized key k = [1, 0]. After α-decay, the state currently returns r = [0.20, 0.80], but the new target value is v = [1.00, −0.20]. Only the first row is corrected because k points exactly along the first key axis.

error = v − r = [0.80, −1.00]
new read = r + β(error)
= (1 − β)r + βv

At β = 1: new read = v exactly.
At β = 0: memory is unchanged.

Edit-strength simulator

old prediction r[0.20, 0.80]
target value v[1.00, −0.20]
updated read[0.60, 0.30]
remaining error norm0.640
The exact-overwrite result has conditions. If ‖k‖ = 1, β = 1, and we read back with that same k immediately after the update, then Sₜᵀk = v. Real heads superpose many non-orthogonal keys, use learned gates, and query with q rather than necessarily k; therefore the state is a compressed associative map, not a perfect dictionary.
05 / Why KDA is more expressive than Gated DeltaNet

One head can learn
many memory clocks.

Gated DeltaNet multiplies the whole head state by one scalar α. KDA replaces that single clock with dₖ channel-wise decay values.

With scalar decay, every direction within one head shares the same half-life. The head can remember broadly or forget broadly, but it cannot independently preserve one key-feature subspace while rapidly refreshing another. KDA’s diagonal gate changes that: Diag(α) scales the state’s rows independently.

This is finer-grained without becoming an arbitrary dense transition. The erase term remains rank one and tied to k. The resulting transition can be written as Diag(α) − βk(k ⊙ α)ᵀ: diagonal plus rank one (DPLR). It is more expressive than scalar weight decay, but constrained enough for a specialized chunkwise algorithm.

The authors also connect channel-wise decay to positional behavior. Different decay rates create different recency scales, analogous in spirit—not in mechanism—to RoPE assigning different rotation frequencies across dimensions. Kimi Linear consequently uses NoPE in its periodic global MLA layers and lets KDA carry the learned positional/recency bias.

Gated DeltaNet
Sₜ = αₜ(I − βkkᵀ)Sₜ₋₁ + βkvᵀ

one α per token/head

All key channels in the head age together. Simple and efficient, but coarse.

Kimi Delta Attention
Sₜ = (I − βkkᵀ)Diag(α⃗)Sₜ₋₁ + βkvᵀ

dₖ α values per token/head

Different learned directions can retain information over different horizons.

06 / What “3:1 hybrid” actually means

Three compressed layers.
One addressable layer.

Layer 1 · recurrentKDAFixed matrix state
Layer 2 · recurrentKDAFixed matrix state
Layer 3 · recurrentKDAFixed matrix state
Layer 4 · globalFull MLATokenwise latent KV

Why retain full attention?

Pure fixed-state models can struggle with exact copying and fine-grained selection because many token associations collide in one finite matrix. Global MLA can directly compare a query with historical token positions.

Why whole layers?

The team chose layerwise rather than headwise mixing for infrastructure simplicity and training stability. A regular repeating cache layout is also easier for serving systems to manage.

Why specifically 3:1?

It was empirical, not a mathematical constant. In the paper’s matched ablation, 3:1 had validation perplexity 5.65, versus 5.66 at 1:1, 5.70 at 7:1, 5.82 at 15:1 and 5.77 for 0:1 full MLA.

MechanismDecode-time historyMemory updateForgettingExact old-token lookup
Softmax / MLAGrowing per-token cacheAppend K/VChoose at read timeYes, through attention scores
Plain linear attentionFixed matrixAdd kvᵀNoneNo; history is compressed
DeltaNetFixed matrixWrite prediction errorTargeted erase/writeNo; history is compressed
Gated DeltaNetFixed matrixDelta ruleOne α per head/tokenNo; history is compressed
KDAFixed matrixDelta ruleChannel-wise α plus βNo; hybrid MLA supplies it
The “up to 75% KV-cache reduction” intuition: three of every four token-mixing layers replace length-growing cache with fixed recurrent state. It is not 75% less total GPU memory or 75% less computation: weights, recurrent states, convolutions, MoE activations, allocator overhead and the remaining MLA cache still exist.
07 / What the measurements establish

The slide is directionally right.
The setup still matters.

Matched training comparison1.4T

Tokens used for fair Kimi Linear vs MLA and GDN-H architecture runs.

Published KV-cache claimup to 75%

Reduction attributed to the repeating 3:1 KDA/MLA hybrid.

1M-context TPOT result6.3×

1.84 ms versus 11.48 ms for MLA in the report’s throughput setting.

128K benchmark average54.5

Versus 52.2 for matched MLA across the paper’s long-context suite.

Architecture result

With the same 1.4T-token pretraining recipe, Kimi Linear led the MLA baseline on most reported short-context tasks, including MMLU-Pro (51.0 vs 47.2). At 128K it scored 84.3 on RULER and 68.5 on RepoQA, versus MLA’s 81.3 and 63.0.

Not every benchmark

The hybrid did not win everything. MLA was higher on LongBench V2 and Frames; GDN-H or MLA led some SFT benchmarks such as MATH500, EvalPlus and LiveBench. “Outperforms full attention” is a summary of the aggregate tested recipe, not a universal theorem.

RL evidence

The report ran math RL with the same algorithm and hyperparameters. Kimi Linear’s training and evaluation curves were above MLA’s in that experiment. This supports compatibility with RL scaling; it does not isolate which KDA subcomponent caused the gain.

Release distinction

The 1.4T runs are controlled architecture comparisons. The public 48B-total/3B-active Base and Instruct checkpoints listed by Moonshot were trained on 5.7T tokens and support up to a 1M-token context. Those later checkpoint scores should not be presented as the 1.4T ablation.

Systems boundary

KDA removes the length-growing scan from KDA layers, not from the entire model. At short context, projections, experts and launches can dominate. At long context, avoiding cache reads becomes increasingly valuable; realized speed still depends on batching, HBM, kernels and the global MLA layers.

08 / How sequential math runs efficiently on GPUs

Recurrent for decode.
Chunked for prefill.

The one-token recurrence is ideal when generating the next token, but naively repeating it across a long prompt would serialize training and prefill.

Kimi’s algorithm expands the recurrence inside chunks. Many token interactions are reorganized into dense matrix multiplications, while a compact state transition links one chunk to the next. The transition’s specialized DPLR structure matters: KDA sets the low-rank erase directions from k instead of learning general independent vectors, reducing intermediate work and I/O.

The technical report says its constrained formulation removes two secondary chunking operations and roughly three matrix multiplications relative to a general DPLR algorithm, approaching 2× DPLR kernel speed in its benchmark. During autoregressive decode, the runtime returns to the simple recurrent update: read and edit a fixed dₖ × dᵥ state.

Moonshot has open-sourced KDA implementations in Flash Linear Attention and a CUTLASS-based FlashKDA project. FlashKDA’s API exposes q, k, v, channel gate g, scalar β, and optional initial/final states; its current kernel documentation lists dₖ = dᵥ = 128. This code makes the slide’s “hardware efficiency” claim inspectable, though kernel microbenchmarks are not whole-model serving benchmarks.

Two complexities coexist: KDA’s state size and recurrent decode work do not grow with prior sequence length, while global MLA still has a length-growing latent KV cache. “Linear attention” describes the KDA operator’s sequence scaling—not every byte and FLOP in the hybrid model.
09 / Where intuition can go wrong

Finite memory is efficient.
It is also finite.

Misread 01

“S contains every token.”

No. S contains a superposition of learned associations. It can generalize across similar keys, but it cannot expose an arbitrary old token as a discrete cache entry.

Misread 02

“α assigns semantic categories.”

No. α gates learned feature directions. We may describe “fast” and “slow” channels for intuition, but the network discovers distributed representations rather than named memory slots.

Misread 03

“β only writes.”

In the classical delta rule, the same β controls erasing the old prediction and writing the target. It is the learning rate of one corrective state update.

Misread 04

“75% means total memory.”

The figure refers to KV-cache savings in the hybrid. Model weights and other runtime state remain, and one in four token-mixing layers still maintains MLA cache.

Misread 05

“6.3× always.”

That is a reported 1M-context TPOT comparison under a particular setup. Batch size, hardware, precision, cache pressure and implementation move the crossover.

Misread 06

“The equation replaces attention quality.”

The hybrid architecture is evidence of the opposite: KDA handles efficient compression, while periodic global attention covers exact retrieval failure modes.

10 / The durable takeaway

KDA is online learning
inside inference.

At each token, the model briefly behaves like a learner: its temporary matrix predicts a value from a key, measures the error, and takes one gated gradient step.

That single perspective unifies the formula. α regularizes memory with feature-specific decay. kkᵀ targets the erase. v − Sᵀk is the correction. β is the online learning rate. q asks the revised memory for the representation needed downstream.

The systems payoff follows directly: because the learned state has a fixed shape, most Kimi Linear layers do not accumulate a KV record for every token. The quality strategy is equally important: periodic global MLA layers retain token-addressable history where compression is weakest. Kimi Linear is therefore not “linear attention beats attention.” It is a carefully engineered division of labor between compressed associative memory and exact global retrieval.

That is what the two slides mean—and why the equation matters. It is simultaneously a memory algorithm, a learned forgetting policy, a long-context scaling choice, and a GPU-kernel design constraint.

11 / Sources & evidence standard

Primary paper, model and code.

Mathematical interpretations are algebraic restatements of the published recurrence. Quantitative statements retain their experimental context; systems consequences not directly benchmarked are identified as such. Accessed 18 July 2026.

  1. 01Kimi Linear technical report — equation, architecture and results↗
  2. 02MoonshotAI Kimi Linear repository — release and deployment↗
  3. 03Official Kimi Linear A3B model collection↗
  4. 04Official model configuration — layer schedule and dimensions↗
  5. 05FlashKDA — CUTLASS kernels and state API↗
  6. 06Flash Linear Attention KDA implementation↗
  7. 07Gated Delta Networks — scalar decay plus delta rule↗
  8. 08Linear Transformers Are Secretly Fast Weight Programmers↗
  9. 09Gated Linear Attention — hardware-efficient gated recurrence↗
  10. 10DeepSeek-V2 — Multi-Head Latent Attention↗
Notation note: some implementations transpose the stored state layout for kernel efficiency. This article follows the paper’s S ∈ ℝdₖ × dᵥ notation. Conceptual colors and matrix cells are explanatory graphics, not visualizations of real learned activations.