Architecture deep dive · 18 July 2026

AttentionResiduals

Transformers learned to retrieve selectively across tokens. AttnRes applies the same idea across depth: each layer can choose which earlier representations deserve to survive, instead of inheriting one indiscriminate running sum.

Q
LAYER 01 · 34%
LAYER 02 · 04%
LAYER 03 · 26%
LAYER 04 · 02%
LAYER 05 · 34%
Depth attentionnot token attention
O(Ld) → O(Nd)Full to Block state
≈8 blockspaper’s practical region
<2%reported inference overhead
01 / The residual stream’s second job

A gradient highway is also
an information mixer.

Residual connections made very deep networks trainable. But the identity path also determines how every earlier computation is combined for every later layer.

In a PreNorm transformer, a block normalizes its input, computes attention or an MLP, and adds that output to the running hidden state. The identity path lets gradients bypass transformations, which is one reason deep transformers train reliably.

Unroll the recurrence, however, and a second role appears: the hidden state entering layer l is the embedding plus every prior block output, all with fixed coefficient one. The network can transform the aggregate, but the residual mechanism itself cannot say “retrieve layer 8 strongly, suppress layer 19, and keep the embedding visible.”

STANDARD UPDATEhₗ = hₗ₋₁ + fₗ₋₁(hₗ₋₁)

One immediate predecessor, one fixed identity addition.

UNROLLED THROUGH DEPTHhₗ = h₁ + Σᵢ₌₁ˡ⁻¹ fᵢ(hᵢ)

Every prior output enters with unit weight.

01 / NO SELECTIVE ACCESS

Everything is already mixed.

Later layers see one compressed stream, not individually addressable earlier outputs.

02 / DEPTH DILUTION

Each contribution becomes relatively smaller.

The paper argues PreNorm hidden-state magnitude grows with depth while individual layer influence is diluted.

03 / COMPENSATING OUTPUTS

Later layers must shout.

Deeper blocks can learn larger outputs to influence an increasingly large accumulated stream, affecting optimization.

Nuance: information is not necessarily erased from a residual stream; it becomes superposed. AttnRes changes the access path so a layer can retrieve earlier representations directly and content-dependently.
02 / Full Attention Residuals

Replace accumulation
with retrieval.

For each token independently, the current layer performs a small softmax over earlier layer outputs. The attention axis is model depth—not sequence position.

Let the token embedding and previous layer outputs be values vᵢ. Each destination layer owns one learned pseudo-query vector wₗ. It scores RMS-normalized earlier outputs, applies softmax across depth, and forms the layer input as a weighted mixture.

The pseudo-query is a learned parameter, not a query derived from the current sequential state. But the score is still input-dependent because the keys are the actual earlier representations for that token. Different tokens can therefore retrieve different mixtures at the same destination layer.

Score across depth

WEIGHTαᵢ→ₗ = softmaxᵢ(wₗᵀ RMSNorm(vᵢ))

RMSNorm prevents a large-magnitude source from winning merely because of its norm.

Retrieve representations

Embedding
Early attention
Middle MLP
Late attention
INPUThₗ = Σᵢ₌₀ˡ⁻¹ αᵢ→ₗvᵢ
Retainearlier outputs
→
Normalizeeach source
→
Scorepseudo-query · key
→
Softmaxover depth
→
Mixweighted values
Why this is lightweight mathematically: the learned addition is one d-dimensional pseudo-query per destination layer. Depth is usually below 1,000, far smaller than modern context lengths. The hard cost is preserving and moving the source representations—not the dot products alone.
03 / Time-depth duality

Attention solved recurrence in time.
Can it solve recurrence in depth?

The Kimi Team frames residual accumulation as a recurrence over layer index, analogous to the single compressed state of an RNN over time.

An RNN repeatedly compresses past tokens into one evolving state. A standard residual network repeatedly compresses past layer outputs into one evolving hidden stream. Token attention gave each sequence position direct, weighted access to earlier positions. AttnRes gives each layer direct, weighted access to earlier layer outputs.

The analogy is powerful but not exact. Sequence elements represent different token positions and may number in the millions. Depth elements are transformations of the same token position and typically number in the tens or hundreds. AttnRes does not extend context length, replace causal self-attention, or reduce the ordinary sequence KV cache.

Two independent memory axes: KDA or MLA determines what a layer can retrieve across tokens. AttnRes determines what a layer can retrieve across earlier transformations. A model such as Kimi K3 can use both because they solve different information-flow problems.
04 / Block AttnRes

Full retrieval is elegant.
Blocks make it deployable.

Full AttnRes saves every layer output. Block AttnRes sums locally within groups and attends only over completed block summaries plus the current partial block.

Suppose a model has L attention/MLP sublayers. Full AttnRes exposes all L outputs. Block AttnRes partitions them into N groups of S = L/N sublayers. Standard addition remains inside each block; softmax retrieval operates across the N block representations.

The embedding remains a source. Within the active block, later sublayers can also attend to the block’s partial sum. At a boundary, that partial sum becomes a completed block representation. The design interpolates smoothly: N = L recovers Full AttnRes; N = 1 approaches standard residual accumulation with the embedding isolated.

Full AttnRes

Each layer output remains individually addressable. Richest depth resolution; O(Ld) saved state and communication.

Block AttnRes

Outputs are summed within blocks; cross-block retrieval sees N summaries. The paper finds roughly eight blocks recover most Full AttnRes gains across tested scales.

MEMORY / COMMUNICATIONO(Ld) → O(Nd)
DEPTH ATTENTION COMPUTEO(L²) → O(N²)
RETAINED STATES / TOKEN≈ 8 in tested region
The real trade: blocks deliberately discard within-block addressability. Larger N preserves finer depth retrieval but consumes more state and fabric bandwidth; smaller N is cheaper but approaches ordinary residual mixing.
05 / Infrastructure is part of the mechanism

Depth retrieval creates
a data-movement problem.

TRAINING · ACTIVATIONS

Recomputation loses its easy win.

Vanilla training already keeps outputs for backpropagation, so Full AttnRes adds little state. Large training commonly discards and recomputes them; AttnRes needs selected representations kept alive.

PIPELINE · COMMUNICATION

Every later stage needs history.

A naïve pipeline forwards the growing set of block summaries repeatedly. Cross-stage caching retains previously received blocks and transmits only newly completed ones.

INFERENCE · MEMORY I/O

Every sublayer revisits block state.

The paper’s two-phase method batches inter-block scoring for a block and merges it with sequential intra-block work using online softmax, amortizing reads while preserving exact results.

Cross-stage caching payoffFor P physical stages and V virtual stages, the paper reduces peak per-transition growth from O(PV) to O(P), describing a V× improvement that can overlap with steady-state pipeline computation.

Long prompts multiply the state by sequence length.

AttnRes block representations are stored per token, so long-context prefill makes the activation footprint visible. Do not confuse these depth summaries with the ordinary self-attention KV cache: they are a separate state family, although both scale with the number of live tokens.

The paper reports memory-efficient prefill and a two-phase inference schedule, with less than 2% latency overhead on its typical evaluated inference workloads. This is an implementation result under specified shapes—not a promise that arbitrary runtimes gain AttnRes for free.

06 / What the paper actually shows

Better scaling—
with bounded claims.

SCALING-LAW RESULT1.25×

Block AttnRes matched the loss of a standard-residual baseline trained with 1.25× more compute in the authors’ scaling experiments.

LARGE MODEL48B / 3B

Kimi Linear model with 48B total and 3B activated parameters, pretrained for 1.4T tokens.

TYPICAL INFERENCE<2%

Reported latency overhead after infrastructure optimizations—not an architecture-independent bound.

BenchmarkBaselineAttnResAbsolute change
MMLU73.574.6+1.1
GPQA-Diamond36.944.4+7.5
BBH76.378.0+1.7
TriviaQA69.971.8+1.9
Math53.557.1+3.6
HumanEval59.162.2+3.1
MBPP72.073.9+1.9
C-Eval79.682.5+2.9
Interpretation: the strongest absolute gain in the repository’s table is GPQA-Diamond. That is consistent with the hypothesis that selective cross-depth reuse helps multi-step reasoning, but one model and one training recipe do not establish a universal causal law.
07 / Optimization payoff

Bound the stream.
Redistribute the gradients.

The reported benefit is not merely a better final benchmark. AttnRes changes how signal and learning pressure are distributed through the stack.

With standard PreNorm accumulation, the paper observes residual magnitudes growing across depth and deeper layers producing larger outputs to influence the stream. Under AttnRes, softmax weights sum to one, so the retrieved mixture is normalized rather than an ever-growing unit-weight sum.

The 48B experiment reports more bounded output magnitudes and a more uniform gradient-norm distribution across layers. This matters because model depth is useful only when layers receive meaningful learning signal and can contribute without competing against an expanding residual baseline.

AttnRes can also specialize retrieval by sublayer: attention and MLP blocks own different pseudo-queries and may prefer different earlier representations. The residual path becomes a learned routing surface across computation history.

The payoff is best phrased as representational and optimization efficiency across depth. Calling it simply “more memory efficient” is misleading: Full AttnRes consumes more live activation state, and Block AttnRes is the systems compromise that makes retrieval affordable.
08 / Kimi K3 evidence boundary

AttnRes is in K3.
The isolated K3 gain is unknown.

CONFIRMED

K3 uses Attention Residuals.

Moonshot’s K3 launch post names KDA and AttnRes as architectural backbones and describes AttnRes as selectively retrieving representations across depth.

COMPONENT EVIDENCE

The paper validates Kimi Linear.

Scaling experiments and the 48B/3B-active, 1.4T-token run establish a controlled component result outside the unreleased full K3 report.

NOT DISCLOSED

K3’s exact recipe and ablation.

Block count, depth, pseudo-query placement, state format, pipeline topology, per-component scaling gain, memory overhead and serving latency remain unspecified.

Moonshot attributes K3’s approximate 2.5× scaling-efficiency improvement over K2 to multiple architectural and recipe changes. It would be incorrect to assign that headline to AttnRes alone.
09 / Tradeoffs and open questions

A stronger access path
creates new obligations.

State versus resolution

Full AttnRes preserves individual layers; blocks compress them. The optimal N depends on model depth, hidden width, context length, fabric and quality sensitivity.

Softmax competition

Weights are positive and sum to one. That provides bounded mixing but forces sources to compete and cannot express arbitrary negative subtraction directly.

Interpretability limits

Depth weights reveal which representations are selected, but a high coefficient is not a complete causal explanation of the resulting behavior.

Runtime adoption

A “drop-in” architectural replacement still requires checkpoint training, block-state memory management, fused kernels, prefill scheduling and pipeline support.

Research question: does a learned depth router make some layers redundant, or does it let every layer become more specialized? Scaling, pruning and routing studies should measure both.
10 / Evaluation checklist

What should the next report disclose?

  1. Exact topology.Count attention and MLP sublayers, blocks, block boundaries, pseudo-queries and placement relative to pipeline stages.
  2. Quality-controlled ablations.Compare standard, Full and Block AttnRes at fixed data, parameters, tokens and training compute.
  3. State accounting.Report bytes per token, dtype, sequence length, activation recomputation, allocator overhead and interaction with ordinary KV cache.
  4. Communication accounting.Measure per-stage bytes, cache hit/reuse, overlap, bubble time and topology sensitivity.
  5. Serving curves.Show TTFT, TPOT and throughput across prompt length, batch, prefill/decode mix and block count—not one typical point.
  6. Retrieval behavior.Publish depth-weight entropy, source preferences by sublayer and token, stability across domains, and whether blocks collapse onto a few sources.
11 / Bottom line

Residuals preserved information.
AttnRes makes it addressable.

The architectural idea is simple: a deep model should not be forced to inherit its entire computational past through one unit-weight sum.

Full AttnRes turns prior layer outputs into a small memory over depth. A learned pseudo-query per destination layer scores normalized representations, and softmax constructs a token-dependent mixture. Block AttnRes keeps the central idea while compressing the source set into practical block summaries.

The payoff is selective reuse of intermediate computation, bounded hidden-state magnitude, healthier gradients, and better quality at a given training-compute budget in the reported experiments. The price is live depth state, memory traffic and cross-stage communication—costs addressed by blocks, caching and two-phase online-softmax execution.

AttnRes is therefore not “free attention added to the skip connection.” It is a redefinition of the residual stream as a retrieval system, with an information architecture and an infrastructure architecture that must agree.

The compact mental model: token attention asks, “which earlier tokens matter now?” Attention Residuals asks, “which earlier computations matter now?”

K3 architecture companion →

See how AttnRes combines with KDA, Stable LatentMoE and quantile-balanced routing.

KDA companion →

See how Kimi changes retrieval across sequence length rather than model depth.

12 / Sources & method

Primary evidence first.

The equations, complexity claims, benchmark values and infrastructure mechanisms come from the Kimi Team report and official repository. K3-specific statements come from Moonshot’s launch post. Broader interpretation is labeled as analysis. Accessed 18 July 2026.

  1. 01Kimi Team — Attention Residuals paper↗
  2. 02Official Attention Residuals repository↗
  3. 03Moonshot AI — Kimi K3 launch blog↗
  4. 04Kimi Linear architecture paper↗
  5. 05Identity Mappings in Deep Residual Networks↗
  6. 06On Layer Normalization in the Transformer Architecture↗
  7. 07Attention Is All You Need↗
  8. 08DeepCrossAttention — related cross-layer access↗
Evidence standard: author-reported results are presented with the model, token budget and comparison context. “Under 2%” and “1.25×” are not generalized beyond the paper’s implementation and experiments.