Training systems · 18 July 2026

MuonClipspeed, then stability

Muon reshapes matrix updates to learn more from each token. At trillion-parameter scale, that efficiency came with exploding attention scores. MuonClip’s answer was surgical: keep the optimizer, monitor each head, and clip the query/key weights at the source.

MOMENTUM MATRIX · UNEVEN DIRECTIONS
NEWTON–SCHULZ → APPROXIMATE UVᵀ
FLATTENED SINGULAR VALUES≈ 1
Muon + QK-Cliptwo mechanisms, one optimizer
5 NS stepscommon practical default
τ = 100Kimi K2 threshold
15.5T tokensreported zero loss spikes
01 / The optimizer is a stack

MuonClip solves
two different problems.

The name compresses an optimization method and a stability controller into one word. Understanding it requires pulling them apart.

Muon—MomentUm Orthogonalized by Newton–Schulz—changes the geometry of updates for matrix-shaped hidden-layer weights. It begins with momentum, then approximately replaces the update’s singular values with values near one. Moonshot’s scalable form adds weight decay and a shape-dependent scale chosen to match AdamW-like update RMS.

QK-Clip watches the maximum pre-softmax attention score for each head. If a head crosses a threshold, it rescales that head’s query and key projection weights after the optimizer step. This is not ordinary gradient-norm clipping, and it does not clip model-output vocabulary logits.

MuonClip is the complete Kimi K2 recipe: Muon’s token efficiency plus weight decay, consistent RMS matching, and QK-Clip’s attention stability. The components operate at different moments and on different signals.

LEARNING DIRECTION

Muon

Transforms momentum matrices before applying parameter updates. The goal is efficient feature learning across matrix directions.

STABILITY SIGNAL

QK-Clip

Observes per-head attention scores already computed during forward and conditionally rescales Q/K weights after the update.

PRODUCTION RECIPE

MuonClip

Combines Muon, decoupled weight decay, RMS-compatible scaling, and QK-Clip for very long, large-scale pretraining runs.

First correction to a common misconception: MuonClip does not “clip Muon updates.” Its clip is targeted at the parameters that generate attention logits.
02 / Muon from first principles

Momentum remembers.
Orthogonalization redistributes.

AdamW adapts each scalar coordinate using first and second moments. Muon respects the fact that a transformer’s hidden weights are matrices.

For a weight matrix W, Muon forms a momentum buffer Mt = μMt−1 + Gt. In common implementations a Nesterov-style lookahead may be applied. The key operation then takes the singular value decomposition M = UΣVᵀ and seeks the polar factor UVᵀ.

Why does that matter? A raw update can have a few large singular values and many tiny ones: it moves strongly in a narrow collection of matrix directions. Replacing the spectrum by values near one preserves the singular vectors but gives the weaker directions comparable scale. The update is semi-orthogonal—orthonormal columns for a tall matrix or rows for a wide one.

Computing an SVD for every hidden matrix at every training step would be expensive. Muon uses a short Newton–Schulz polynomial iteration built from matrix multiplications, which tensor cores execute efficiently. The widely used quintic coefficients are 3.4445, −4.7750, and 2.0315, commonly run for five steps after normalizing the matrix.

MOMENTUMMₜ = μMₜ₋₁ + Gₜ

Accumulate a smoothed matrix direction instead of reacting to one noisy minibatch.

IDEAL POLAR FACTORM = UΣVᵀ → O = UVᵀ

Keep singular vectors; flatten singular magnitudes.

QUINTIC NS STEPX ← aX + (bA + cA²)X

With A = XXᵀ after choosing the cheaper orientation.

Raw momentum spectrum

A few dominant modes can monopolize update energy.

Approximate Muon spectrum

The practical polynomial produces noisy values around one, not a perfect SVD.

Precise wording matters: practical Muon approximately orthogonalizes. The tuned five-step polynomial is optimized for useful speed and slope, not for exact convergence of every singular value to one.
03 / Making Muon scale

Orthogonalization alone
was not the production recipe.

Moonlight identified two additions that made Muon competitive beyond speedrun-sized models: weight decay and consistent update scaling.

Vanilla Muon could converge rapidly at first while some weights grew too large in longer training. Decoupled weight decay directly controls parameter growth. Moonshot’s experiments found Muon with weight decay outperformed both vanilla Muon and AdamW in the over-training regime they studied.

The second problem is dimensional: a semi-orthogonal n × m matrix has a predictable Frobenius norm, so its elementwise RMS changes with shape. A single raw learning rate would imply different effective update magnitudes for different projection shapes. Moonlight scales the orthogonalized update by 0.2√max(n,m) to match the RMS convention used by AdamW in its recipe.

This scale is not a universal mathematical constant. It is a transfer device: it allowed learning-rate and weight-decay choices tuned for AdamW to be reused in Moonshot’s setup. PyTorch now exposes “original” and “match_rms_adamw” adjustment modes because the reference Muon and Moonshot conventions differ.

ORTHOGONALIZED DIRECTIONOₜ = NS(Mₜ) · 0.2√max(n,m)

Kimi K2 algorithm’s Adam-RMS matching factor.

DECOUPLED DECAYWₜ = Wₜ₋₁ − η(Oₜ + λWₜ₋₁)

Update direction and explicit parameter shrinkage.

STATE ECONOMICSMuon: momentum buffer

Unlike Adam, base Muon does not require a second-moment buffer for its matrix group.

Hybrid parameter routing: Muon is intended for 2D hidden-layer matrices. Biases, normalization scales, embeddings, and the final language-model head are normally routed to AdamW or another standard optimizer—even when an embedding/head happens to be 2D.
04 / The scale-up failure

The loss spike begins
inside attention.

As Moonshot scaled Muon, a subset of attention heads developed rapidly growing pre-softmax QK scores.

For head h, attention forms queries and keys, then scores token pairs using qᵢᵀkⱼ with the model’s normal scaling and positional machinery. Softmax is shift-invariant but extremely sensitive to gaps. If query/key norms grow, scores become enormous, attention approaches a brittle one-hot selector, exponentiation becomes numerically demanding, and gradients can become pathological.

The Kimi K2 report’s 53B-total, 9B-active Muon experiment saw maximum attention logits exceed 1,000. The authors associate that regime with significant loss spikes and occasional divergence. They also report the issue occurring more often with Muon than AdamW in their experiments; that is an empirical observation, not a theorem that Muon must explode.

The max statistic matters because mean or RMS can remain ordinary while one head and one token pair runs away. A single pathological head can destabilize a training job that has already consumed enormous compute.

ILLUSTRATIVE MAX ATTENTION LOGIT
DESIRED BOUND
GLOBAL GRAD CLIP

Controls update norm

Useful broadly, but it does not directly bound the forward attention score or identify which head caused it.

LOGIT SOFT-CAP

Clamps downstream

Bounds the value passed onward, while large QK dot products are still formed before the cap.

QK-NORM

Normalizes activations

Effective in some architectures, but the K2 report says it is incompatible with MLA because full keys are not materialized at inference.

QK-CLIP

Rescales the source

Uses the observed maximum to shrink only the projection weights responsible for the exploding head.

05 / QK-Clip

Observe this step.
Correct the next one.

QK-Clip does not modify the current forward or backward pass. It uses a statistic from that pass to project selected weights back toward a safe region after Muon updates them.

For every attention head, the runtime retains Smaxh, the maximum input to attention softmax across the training batch and token pairs. Given threshold τ, it computes γh = min(1, τ/Smaxh). A head below threshold receives γ = 1 and is untouched.

For ordinary multi-head attention with balancing exponent α, query weights are multiplied by γα and key weights by γ1−α. Their dot product therefore scales by exactly γ. Kimi uses α = 0.5 in its explanation, so Q and K each receive √γ. Sharing the correction minimizes asymmetry between their paths.

If the observed maximum is 400 and τ = 100, then γ = 0.25; each side is scaled by 0.5, and their product is quartered. If the maximum is 80, γ = 1 and no clipping happens. Because this is weight rescaling, the change persists into subsequent batches.

PER-HEAD SIGNALSmaxh = maxB,i,j scorei,jh

The largest pre-softmax attention input observed for a head.

ADAPTIVE FACTORγh = min(1, τ / Smaxh)

Inactive below threshold; proportional correction above it.

BALANCED RESCALEWq ← γαWq · Wk ← γ1−αWk

The next QK score inherits a multiplicative γ, all else equal.

Threshold τ = 100 and α = 0.5. Move the signal to see when QK-Clip activates.

CLIPPING ACTIVE

Observed
400
Target
100
γ = 0.250COMBINED SCORE SCALE
√γ = 0.500QUERY WEIGHT SCALE
√γ = 0.500KEY WEIGHT SCALE
It is a feedback controller, not a hard per-example guarantee. The factor is calculated from the previous forward pass; different inputs after rescaling could still produce a different maximum. The report demonstrates empirical regulation around τ, not a formal global bound over all possible sequences.
06 / Multi-head latent attention changes the surgery

Shared components cannot be clipped
as if every head owned them.

Kimi K2 uses Multi-head Latent Attention. Its query and key construction includes head-specific and shared pieces, so naïve per-head scaling would spill across heads.

The report divides the relevant representations into content and rotary components. Head-specific content query qC and content key kC are each scaled by √γh. Head-specific rotary query qR is scaled by γh. The rotary key kR is shared and is left untouched.

This asymmetric-looking rule follows the score decomposition. The content term contains a head-specific Q and K, so the factor can be split across both. For the rotary term, only the query side can be adjusted independently without changing other heads, so it receives the full γ.

This is the engineering detail that turns a clean MHA equation into a deployable MLA mechanism. “Clip Q and K” is insufficient as an implementation specification; ownership and factorization determine which tensors can be changed without collateral effects.

CONTENT QUERY

qCh

× √γh

Head-specific; shares the content-score correction with kC.

CONTENT KEY

kCh

× √γh

Head-specific component can be adjusted without touching peers.

ROTARY QUERY

qRh

× γh

Owns the entire rotary correction because the other side is shared.

ROTARY KEY

kR

× 1

Shared across heads; left unchanged to avoid cross-head interference.

07 / One complete training step

Optimization first.
Projection second.

Forwardcompute loss + Smax per head
→
Backwardform gradients G
→
MomentumM ← μM + G
→
NSorthogonalize M
→
Muon updatescale + decay W
→
QK-Cliprescale violating heads

The clip is post-update but its telemetry comes from pre-update weights.

This ordering makes QK-Clip easy to reason about: the ordinary model forward and backward equations remain unchanged for the current step. The attention kernel exports one extra maximum statistic per head; the optimizer uses that statistic after its parameter update.

But ordering also creates state-consistency questions. If the parameter is clipped while Muon’s momentum buffer is not, the next step’s optimizer state does not undergo the same projection. That is exactly what the published algorithm specifies, but independent implementations must reproduce it intentionally rather than assuming generic parameter-clipping semantics.

Mixed precision adds another obligation: the max should represent the actual pre-softmax computation, reductions must not overflow, and rescaling must update the authoritative master parameter consistently with sharded or cached copies.

for each training step: loss, head_max = forward(batch, return_attention_max=True) loss.backward() for W in hidden_matrix_parameters: M[W] = momentum * M[W] + W.grad O = newton_schulz(M[W]) * rms_match(W.shape) W -= lr * (O + weight_decay * W) for each attention head h: gamma = min(1, tau / head_max[h]) rescale_head_specific_qk_weights(h, gamma) update_remaining_parameters_with_adamw()
Pseudocode, not drop-in code: real MLA tensor layouts, fused QKV projections, tensor parallel shards, optimizer-state precision, Nesterov momentum, and distributed reductions must follow the model implementation.
08 / Systems architecture

A matrix-aware optimizer
becomes a collective-communication problem.

Muon saves one Adam-like moment buffer but spends matrix multiplications and may require communication inside Newton–Schulz.

AdamW keeps first and second moment state for each optimized parameter. Base Muon keeps momentum but no elementwise variance buffer for its matrix group, which can reduce optimizer-state memory. Yet five Newton–Schulz steps perform repeated matrix multiplications on every eligible weight update, making each optimizer step heavier.

Under tensor or fully sharded parallelism, a logical weight matrix may be split across ranks. Orthogonalizing each shard independently is generally not equivalent to orthogonalizing the global matrix. Distributed implementations must choose a compatible partition dimension and use collectives for the Gram products or gather/redistribute data. NVIDIA’s current implementation explicitly accepts a tensor-parallel process group and warns that sharding orientation is the caller’s responsibility.

QK-Clip adds a much smaller statistic but a subtle reduction. The per-head maximum must cover the intended data-parallel batch and token pairs. Local-only maxima can make clipping depend on rank assignment; a global max requires collective communication. The rescaled weights then need consistency across replicas and shards.

COMPUTE

Five rounds of GEMMs.

Newton–Schulz is tensor-core friendly, but it is not free. Tall/wide orientation is chosen to operate along the smaller matrix dimension.

STATE

One matrix momentum.

Muon’s eligible group avoids Adam’s second moment; the separate AdamW group still carries Adam state.

COMMUNICATION

Global geometry matters.

Sharded Gram products, maxima, and parameter rescaling must preserve the semantics of the unsharded algorithm.

2D onlyMuon parameter class
5×typical NS iterations
1 max/headQK telemetry per step
2 optimizersMuon + AdamW routing
ConcernAdamWMuon / MuonClipEngineering consequence
Adaptive stateFirst + second momentsMomentum for Muon matricesPotentially lower optimizer-state bytes
Per-step mathElementwise operationsRepeated matrix productsHigher optimizer compute; accelerator-friendly
Parameter scopeBroadHidden 2D matricesRequires explicit parameter grouping
Parallel semanticsMostly elementwiseMatrix-global orthogonalizationShard orientation and collectives matter
Attention telemetryNot inherentPer-head max for QK-ClipFused attention must expose/reduce statistic
09 / What the evidence supports

Muon delivered efficiency.
QK-Clip preserved the run.

MOONLIGHT SCALING LAW≈2×

Author-reported compute efficiency for scalable Muon versus AdamW under compute-optimal training.

MOONLIGHT TRAINING5.7T

Tokens used to train the 3B-active / 16B-total MoE model with scalable Muon.

K2 PRETRAINING15.5T

Tokens in the Kimi K2 run using MuonClip.

K2 LOSS CURVE0 spikes

Reported for the full 1.04T-total, 32B-active training run.

The strongest claim is operational: an optimizer family that had exhibited attention-logit instability completed an enormous run smoothly.

In the mid-scale Muon run, maximum attention logits quickly exceeded 1,000. In Kimi K2, MuonClip used τ = 100; the maxima initially sat at the cap, then decayed into a normal operating range after roughly 30% of training. The paper reports no observable loss spikes.

Its “harmlessness” experiments found QK-Clip controlled maxima without degrading the studied loss trajectory or downstream performance. That supports QK-Clip in the tested Muon/Kimi configurations. It does not establish that τ = 100, α = 0.5, or the same intervention is optimal for every architecture, scale, optimizer, or attention kernel.

Moonlight’s ≈2× claim and K2’s zero-spike claim answer different questions. The former concerns compute/token efficiency in scaling-law experiments; the latter concerns stability at unprecedented scale. MuonClip combines the mechanisms, but QK-Clip should not be credited with Muon’s full efficiency gain.

DIRECTLY REPORTED

K2 completed cleanly.

1.04T total parameters, 32B activated, 15.5T tokens, τ = 100, no loss spike.

SUPPORTED IN ABLATIONS

Clipping was targeted.

Per-head intervention regulated maxima while preserving tested loss and downstream results.

NOT ESTABLISHED

Universal optimizer dominance.

The reports do not prove MuonClip always beats a thoroughly tuned AdamW or works unchanged for fine-tuning/RL.

Evidence boundary for Kimi K3: Moonshot’s currently public K3 launch material does not provide a MuonClip training ablation or full optimizer recipe. This article therefore anchors its hard claims in K2, Moonlight, Kimi Linear, and K2.5 disclosures rather than assuming continuity.
10 / Comparisons that clarify the mechanism

Clip the gradient, the logit,
or the weights?

Versus global gradient clipping

Gradient clipping limits the norm of the backward update. QK-Clip uses a forward statistic, identifies the offending head, and rescales the parameters that generate QK scores. The methods can coexist.

Versus attention soft-capping

A soft cap changes the attention function on every forward pass. QK-Clip leaves the forward equation unchanged and intervenes conditionally on weights after a threshold violation.

Versus QK-Norm

QK-Norm normalizes queries and keys continuously. QK-Clip is an event-driven parameter projection designed around MLA’s partially shared/factorized representation.

Versus AdamW

AdamW performs elementwise variance adaptation across a broad parameter set. Muon uses matrix-level spectral geometry for eligible weights, while an AdamW companion still handles the rest.

MuonClip is best understood as constrained optimization engineering: Muon proposes an efficient matrix direction; weight decay controls long-run parameter growth; QK-Clip projects a small, dangerous subset of weights back when a model-level invariant is violated.
11 / Tradeoffs and open questions

Stability has a price—
and a scope.

Extra optimizer FLOPs

Newton–Schulz adds several matrix products per eligible weight. Sample efficiency must outweigh slower steps in wall-clock and total training cost.

Head-wise weight distortion

Clipping is intentionally minimal, but it changes parameter norms discontinuously. Frequent activation could fight the optimizer or alter learned head specialization.

Threshold transfer

A fixed τ interacts with attention scaling, head dimension, positional encoding, activation precision, architecture, batch composition, and training stage.

State mismatch

QK weights are rescaled while momentum follows its published recurrence. The long-run dynamics of repeated projection deserve direct study.

Fine-tuning and RL

The flagship evidence is pretraining. Shorter, lower-data or policy-optimization regimes may prefer different parameter routing, learning rates, or no clipping.

Maximum as a statistic

Max is sensitive to a single outlier and distributed scope. Quantiles or persistent violation measures might offer different intervention tradeoffs.

12 / Reproduction checklist

What a serious implementation
must specify.

  1. Parameter groups.List exactly which attention, MLP, router, embedding, normalization, bias, and output-head tensors use Muon versus AdamW.
  2. Muon variant.Record momentum/Nesterov semantics, Newton–Schulz coefficients and steps, normalization, transpose rule, weight decay, and update-scaling convention.
  3. Logit definition.State whether Smax is measured before or after 1/√d scaling, positional terms, masks, soft-caps, and which batch/token axes participate.
  4. Distributed reduction.Clarify whether head maxima are local, data-parallel global, sequence-parallel global, or accumulated across microbatches.
  5. MLA tensor mapping.Identify the physical slices for qC, kC, qR, and shared kR; verify that per-head rescaling does not alter neighboring heads.
  6. Precision and master weights.Measure the max safely and apply clipping to the authoritative parameter so BF16/FP8 views, shards, and caches remain synchronized.
  7. Telemetry.Log per-layer/head maxima, clip frequency, γ distribution, weight norms, momentum norms, loss spikes, overflow events, and throughput.
  8. Ablation budget.Compare AdamW, vanilla Muon, scalable Muon, and MuonClip at matched model, data, FLOPs, tokens, batch, scheduler, and tuning effort.
13 / Bottom line

MuonClip separates
learning speed from stability control.

The architecture of the solution is more important than the branding: use a matrix-aware optimizer where it helps, then enforce a model-aware invariant where it fails.

Muon takes momentum matrices and flattens their singular spectrum using a few accelerator-friendly Newton–Schulz iterations. Moonlight added the weight decay and shape-aware RMS scaling needed for long, large-model training. That produced strong author-reported token and compute efficiency.

At Kimi K2 scale, a few attention heads still drove QK scores toward instability. QK-Clip watched the actual forward-pass maximum, calculated a per-head correction, and rescaled only the independently owned query/key projections after the update. It controlled the mechanism generating the explosion rather than merely hiding its output.

The resulting MuonClip stack carried a 1.04-trillion-parameter MoE through 15.5 trillion tokens with no reported loss spikes. That is compelling systems evidence—not proof of universal superiority, but a concrete demonstration that optimizer geometry, attention architecture, telemetry, and distributed implementation can be designed as one training system.

The compact mental model: Muon asks, “how should this matrix move?” QK-Clip asks, “did that movement make any attention head unsafe?” MuonClip closes the loop.

Logits companion →

Understand raw scores, softmax sensitivity, calibration, and the difference between attention and output logits.

Kimi architecture companion →

Explore KDA, Attention Residuals, Stable LatentMoE, and quantile-balanced routing.

14 / Sources & method

Primary evidence first.

Algorithm details and K2 results come from Moonshot’s technical report and official launch article. Muon design and scaling claims use the original implementation write-up, Moonlight report, and current framework documentation. Accessed 18 July 2026.

  1. 01Kimi Team — Kimi K2 Technical Report↗
  2. 02Moonshot AI — Kimi K2 launch article↗
  3. 03Official Kimi K2 repository and report↗
  4. 04Liu et al. — Muon Is Scalable for LLM Training↗
  5. 05Official Moonlight distributed Muon repository↗
  6. 06Keller Jordan — Muon design and implementation↗
  7. 07Reference Muon repository↗
  8. 08PyTorch — Muon optimizer documentation↗
  9. 09NVIDIA NeMo — Muon and distributed Newton–Schulz↗
  10. 10Kimi Team — Kimi Linear Technical Report↗
  11. 11Vaswani et al. — Attention Is All You Need↗
  12. 12Kingma & Ba — Adam↗
Evidence standard: “≈2×,” “15.5T,” “τ = 100,” and “zero loss spikes” are author-reported results tied to their experiments. Explanatory examples are labeled illustrative. No undisclosed Kimi K3 optimizer configuration is inferred.