Model mechanics · 18 July 2026

Logitsbefore probability

Every generated token begins as a row of raw scores. Those scores become a loss during training, a distribution during inference, and—at large vocabulary scale—a serious compute, memory, and networking problem.

NEXT-TOKEN SCORES · RAW, UNNORMALIZED
“ Paris”
8.4
“ Lyon”
5.1
“ France”
3.3
“ London”
1.7
↓ SOFTMAX(z / T) ↓
NORMALIZED DISTRIBUTIONΣ p = 1
z ∈ ℝVone score per token
p = softmax(z)scores to probabilities
∂L/∂z = p − ytraining signal
[B,T,V]the expensive tensor
01 / The raw score layer

A logit is a preference—
not yet a probability.

At token position t, a language model turns its final hidden vector into one real-valued score for every token in its vocabulary.

Let ht be the final transformer state with hidden width d. The language-model head—often called the output projection or unembedding—multiplies it by a matrix with one row per vocabulary token. The result is a vector zt of length V. Its components are logits.

A logit can be positive, zero, or negative. It need not lie in any fixed range, and the vector neither sums to one nor represents percentages. Only relative differences matter. Add 100 to every score and softmax returns exactly the same probabilities. The model is expressing a ranking and a set of margins, not an absolute unit of confidence.

VOCABULARY PROJECTIONzₜ = Wvocabhₜ + b

W has shape V × d. Each row scores one token against the current representation.

NORMALIZATIONpᵢ = exp(zᵢ) / Σⱼ exp(zⱼ)

Softmax converts arbitrary real scores into non-negative probabilities summing to one.

SHIFT INVARIANCEsoftmax(z + c) = softmax(z)

The score origin is arbitrary; pairwise logit gaps carry the decision information.

The useful mental model: the hidden state contains the model’s contextual computation. The vocabulary head asks every token embedding, “how compatible are you with this state?” Logits are the answers before normalization.
02 / A numerical example

Softmax turns gaps
into odds.

Suppose four candidate tokens receive logits [2, 1, 0, −1]. Exponentiation makes a one-point lead multiplicative.

The exponentials are approximately [7.389, 2.718, 1, 0.368]. Divide each by their sum, 11.475, and the probabilities become [64.4%, 23.7%, 8.7%, 3.2%]. A logit difference has a clean interpretation: if token A is 2 points above token B, its unnormalized odds are e² ≈ 7.39 times larger.

Implementations compute this stably by subtracting the maximum logit first. That does not change the answer because of shift invariance, but it prevents large exponentials from overflowing. Production code uses fused log-softmax or logsumexp paths rather than materializing fragile probabilities and then taking logarithms.

Hidden statehₜ ∈ ℝᵈ
→
LM headW hₜ
→
Logitsz ∈ ℝⱽ
→
Processorsmask / bias
→
Distributionsoftmax
→
Tokenargmax / sample
Do not apply softmax twice: training libraries such as PyTorch expect raw, unnormalized logits in cross-entropy loss. The loss combines log-softmax and negative log-likelihood in one numerically stable operation.
03 / Training consumes logits directly

The target token pulls up.
Every rival pushes back.

Next-token training does not need to sample. It asks how much probability the model assigned to the known continuation and backpropagates the error.

For target token y, cross-entropy is −log py, equivalently logsumexp(z) − zy. The target logit is rewarded; the log-sum-exp term accounts for every competing vocabulary item. If the target already dominates, the loss is small. If an incorrect token dominates, the loss is large.

The derivative is unusually simple: p − one_hot(y). For the target, the gradient is py − 1, pushing its logit upward. For each non-target, it is pi, pushing it downward. Label smoothing replaces the one-hot target with a slightly distributed target, reducing pressure toward infinitely separated scores.

PER-TOKEN LOSSL = −log pᵧ

Surprise measured in nats when natural logarithms are used.

STABLE FORML = logΣⱼ exp(zⱼ) − zᵧ

No explicit probability tensor is necessary.

LOGIT GRADIENT∂L/∂zᵢ = pᵢ − 1[i=y]

A dense competition signal across the vocabulary.

Target: −0.36If pᵧ = .64, gradient descent raises its logit.
Rival: +0.24The strongest wrong alternative is pushed down most.
Rival: +0.09Weak alternatives receive smaller corrections.
Σ gradient = 0Consistent with softmax’s shift invariance.
Sequence loss: training averages or sums this quantity over valid token positions, usually masking padding. Perplexity is the exponential of average negative log-likelihood—not an average “confidence” score.
04 / Temperature

One scalar reshapes
the entire distribution.

Temperature divides logits before softmax. Below 1 it magnifies gaps; above 1 it compresses them.

For positive temperature T, generation uses softmax(z/T). With the example [2, 1, 0, −1], T = 0.5 produces roughly [86.5%, 11.7%, 1.6%, 0.2%]. At T = 2, it becomes [45.5%, 27.6%, 16.7%, 10.2%]. The ranking never changes: dividing every logit by the same positive number preserves order.

Therefore temperature cannot change greedy argmax output by itself. It matters when sampling or when another processor uses the resulting probabilities. “Temperature zero” is normally a runtime convention for greedy decoding; implementations should not literally divide by zero.

Fixed logits: [2, 1, 0, −1]. Move the slider to see logit gaps sharpen or flatten after softmax.

Entropy: 0.948 nats

64.4%A · 2
23.7%B · 1
8.7%C · 0
3.2%D · −1
05 / From distribution to token

Sampling is a policy layer
built on top of logits.

A model supplies scores. The decoding stack decides which candidates remain eligible and how one is selected.

Greedy decoding chooses the maximum logit. Top-k retains a fixed number of highest-scoring tokens. Top-p, or nucleus sampling, keeps the smallest high-probability set whose cumulative mass crosses a threshold. Its candidate count expands when uncertainty is broad and contracts when one answer dominates.

Min-p removes candidates whose probability is too small relative to the leading token. Typical sampling favors tokens whose information content lies near the distribution’s entropy. These are not changes to model weights. They are runtime transformations of the next-token policy, repeated at every decoding step.

ARGMAX

Greedy

Deterministic and cheap. Can lock into repetitive or bland continuations because it never explores a runner-up.

FIXED CARDINALITY

Top-k

Predictable candidate count, but k ignores whether the model is certain or uncertain at a particular step.

DYNAMIC MASS

Top-p

Adapts set size to the distribution and removes the unreliable low-probability tail identified in nucleus-sampling research.

RELATIVE FLOOR

Min-p

Scales the cutoff relative to the best token, often pairing naturally with temperature changes.

TYPICAL PIPELINE: raw logits → hard masks / allowed tokens → bias & penalties → temperature → truncation → normalization → sample
Ordering matters. Applying top-p before temperature need not retain the same tokens as applying it afterward. Runtime libraries differ, so reproducibility requires the model, tokenizer, random seed, every processor, and processor order—not just “temperature 0.7.”
06 / Masks, penalties, and constraints

A logit can be edited
without retraining the model.

Generation systems commonly add biases, apply penalties, or set invalid candidates to negative infinity before sampling.

A hard vocabulary mask sets forbidden token logits to −∞, making their post-softmax probability exactly zero. Grammar-constrained decoding uses the parser state to construct a different valid-token set at each step. This can guarantee syntactic membership—such as well-formed JSON under the grammar—but cannot guarantee that fields are truthful, relevant, or semantically valid.

API “logit bias” adds a caller-specified offset to selected token scores. Repetition, presence, and frequency penalties discourage previously emitted tokens, but formulas vary across engines. Because tokenizers operate on subwords, banning the visible word “Paris” may require multiple token IDs, including space-prefixed and case variants.

Classifier-free guidance and contrastive methods can combine logits from conditional and unconditional passes. Speculative decoding compares draft and target distributions to preserve the target model’s distribution while accepting many draft tokens. In each case, logits are a convenient interface between learned computation and an explicit decoding policy.

Control boundary: constraints can make an output structurally valid, and bias can make a token more or less likely. Neither turns raw logits into factual certainty. The intervention changes selection behavior, not what the base model knows.
07 / One word, several axes

Not every logit is
a vocabulary logit.

OUTPUT / TOKEN LOGITS

Which token comes next?

SOFTMAX AXIS · VOCABULARY V

Produced by the language-model head. Used by cross-entropy, decoding, log-probability APIs, and distillation.

ATTENTION LOGITS

Which position should this query read?

SOFTMAX AXIS · SEQUENCE

Scaled QKᵀ scores, modified by causal or padding masks, then normalized across key positions for each head and query.

MOE ROUTER LOGITS

Which expert should process this token?

SOFTMAX AXIS · EXPERTS

A router scores experts; top-k selection and balancing logic turn those scores into dispatch decisions and network traffic.

CLASSIFIER / REWARD LOGITS

Which label—or preference—wins?

AXIS · LABELS OR OUTCOMES

Classification heads produce class logits. Preference models may output a scalar whose difference behaves like a log-odds score.

The shared idea is simple: logits are pre-normalization scores. Their meaning comes from what is being scored and which axis will be normalized—not from the word “logit” alone.
08 / Confidence and calibration

High probability does not mean
high epistemic certainty.

Softmax produces a mathematically valid categorical distribution. That does not guarantee its probabilities match real-world correctness frequencies.

Neural networks can be miscalibrated: predictions assigned 90% probability need not be correct 90% of the time. Temperature scaling can calibrate a classifier on a held-out distribution, but sampling temperature is usually chosen for output behavior, not formal calibration. A language model’s distribution is conditional on its training, tokenizer, prompt, conversation state, and decoding setup.

Token probability is also not word or answer probability. One word may span several tokens; a complete answer follows one of many possible surface forms. Sequence probability multiplies conditional token probabilities, or equivalently sums log-probabilities, and naturally shrinks with length. Comparing differently tokenized or differently sized answers requires care.

Log-probabilities are operationally preferable to raw probabilities: they add across sequences, avoid underflow, and expose relative likelihood. Even then, likelihood is not truth. A fluent false statement can receive higher probability than an awkward true one.

MYTH 01

“A negative logit is impossible.”

False. Logits are unconstrained real numbers. Negative infinity is also used deliberately as a hard mask.

MYTH 02

“The largest logit is the confidence.”

False. Softmax depends on every gap. [10, 9.9] is uncertain; [2, −5] is decisive despite a smaller maximum.

MYTH 03

“Temperature changes the favorite.”

Not by itself for T > 0. It changes concentration while preserving the ordering of logits.

MYTH 04

“Top-p alters the model.”

No. It truncates the runtime distribution. The same weights can serve many decoding policies.

MYTH 05

“Logits correspond to words.”

They correspond to tokenizer IDs: fragments, punctuation, bytes, spaces, or whole words depending on the vocabulary.

MYTH 06

“APIs return all logits.”

Usually not. Full vectors are enormous; services commonly expose only selected top log-probabilities.

09 / The systems cost

The last projection can be
a very large operation.

Vocabulary size turns a conceptually simple matrix multiply into a material compute, memory, and communication workload.

Consider an illustrative vocabulary of 128,000 tokens and hidden width 8,192. The output matrix contains 1,048,576,000 weights—about 2.10 GB in BF16, before metadata or alignment. Weight tying can share this matrix with the input embedding table, reducing parameter count, but the output multiply still must score the vocabulary.

During autoregressive decode, engines normally compute logits only for each sequence’s newest position. During training, every non-masked token needs a loss. A naïve BF16 logit tensor for batch 8, sequence 4,096, and vocabulary 128,000 occupies exactly 8 GiB. Fused and vocabulary-parallel cross-entropy exists partly to avoid keeping that entire dense tensor resident.

1.049Billustrative LM-head weights
≈2.10 GBBF16 weight bytes, decimal
8 GiBnaïve [8,4096,128k] BF16 logits
TENSOR PARALLELISM

Shard the vocabulary.

Each GPU owns rows of the output matrix and produces logits for its vocabulary slice. No single rank must store the whole head.

DISTRIBUTED SOFTMAX

Reduce max, then sum.

Stable normalization needs a global maximum and global exponential sum. Target-logit ownership and gradient terms also cross ranks.

SERVING OUTPUT

Do not ship V numbers.

Returning top-N log-probabilities adds selection, synchronization, serialization, bandwidth, and storage. Full logits are rarely economical.

PhaseTypical logit shapeWhat mattersCommon optimization
Training[B, T, V]Activation memory and cross-entropyFused / chunked vocab-parallel loss
Prefill for generation[B, V] at last valid positionTime to first tokenSkip unused positions’ LM head
Decode[active sequences, V]Per-step projection and samplingFused projection / distributed top-k
Log-prob scoring[B, T, selected V]Prompt and candidate likelihoodsGather only requested token scores
Precision matters near ties: quantization or low-precision accumulation can perturb small logit gaps and reorder candidates. Aggregate quality may remain strong while exact greedy reproducibility changes. Stable softmax often uses wider accumulation even when weights and activations are lower precision.
10 / Logits beyond next-token sampling

A compact interface for
learning, steering, and inspection.

Because logits preserve more structure than a hard prediction, they are reused throughout modern model development.

Knowledge distillation trains a student against the teacher’s softened output distribution. The teacher’s non-winning logits reveal similarities among alternatives—the “dark knowledge” lost in a one-hot label. Distillation temperature softens both distributions, with the loss rescaled so useful gradients survive.

Preference optimization and RLHF work with policy log-probabilities, likelihood ratios, and KL penalties derived from logits. Beam search accumulates log-probabilities across candidate sequences. Reranking compares answer likelihoods, while care is needed for length and tokenization biases.

Logit-lens probes apply the model’s unembedding to intermediate residual states, asking what token distribution is already decodable at each layer. This can expose evolving predictions, but it is a diagnostic projection—not a causal transcript of “what the model thinks.” Tuned-lens methods learn corrections for distribution shifts between intermediate and final representations.

Why logits are such a useful boundary: they keep graded information about every alternative, yet are simple enough to normalize, mask, combine, differentiate, shard, and inspect.
11 / An operator’s checklist

When generations look wrong,
inspect the whole pipeline.

  1. Confirm the tokenizer.Decode token IDs, including leading spaces and byte fallbacks; visible strings do not map one-to-one onto logits.
  2. Capture raw and processed scores.Separate model output from masks, biases, penalties, temperature and truncation.
  3. Verify processor order.Two engines with identical parameter names can produce different candidate sets if transformations are sequenced differently.
  4. Check the sampling contract.Record seed, RNG implementation, batch scheduling and determinism guarantees—not just temperature and top-p.
  5. Measure logit margins.A changed token may reflect a near tie amplified by precision, quantization or distributed reduction order rather than a large model regression.
  6. Account for distributed work.Track LM-head time, collective latency, top-N extraction, log-prob payloads and whether unused prefill positions are projected.
  7. Do not call probability truth.Evaluate task accuracy and calibration on representative data; likelihood alone cannot establish factuality.
12 / Bottom line

Logits are the hinge
between computation and choice.

Inside the model, logits are raw compatibility scores. Outside it, they become whatever the training or decoding system asks them to become.

Cross-entropy turns logits into a dense learning signal. Softmax turns their gaps into odds. Temperature changes concentration; truncation removes the unreliable tail; masks and biases impose policy. Distributed runtimes shard the enormous vocabulary axis and coordinate the statistics required for stable normalization.

The crucial discipline is to keep the layers separate. A logit is not a probability. A probability is not calibrated confidence. A high-likelihood string is not necessarily a true statement. And a decoding parameter is not a modification to the model’s learned weights.

Once those boundaries are clear, logits stop being mysterious. They become a precise interface—one that connects transformer representations to loss functions, inference behavior, controllability, and the physical realities of serving a large vocabulary.

The compact formula: hidden state → vocabulary logits → processed distribution → token. Training runs the arrow backward; serving must run it cheaply, once per generated step.

Memory hierarchy companion →

Follow model weights and runtime state through GPU SRAM, HBM, CPU DRAM and NVMe.

MXFP4 companion →

See how low-precision formats change model storage, arithmetic and sensitivity.

13 / Sources & method

Primary documentation first.

Definitions and implementation claims are grounded in official framework documentation and original papers. Numerical examples are independently calculated and labeled illustrative. Accessed 18 July 2026.

  1. 01PyTorch — CrossEntropyLoss documentation↗
  2. 02Hugging Face — Generation configuration↗
  3. 03Hugging Face — Logits processors and warpers↗
  4. 04Holtzman et al. — Nucleus Sampling↗
  5. 05Guo et al. — Calibration of Modern Neural Networks↗
  6. 06NVIDIA Megatron Core — Fused vocab-parallel cross-entropy↗
  7. 07NVIDIA Megatron Core — Language-model output and loss↗
  8. 08Vaswani et al. — Attention Is All You Need↗
  9. 09Beurer-Kellner et al. — Grammar-Constrained Decoding↗
  10. 10Press & Wolf — Tying Input and Output Embeddings↗
  11. 11Sanchez et al. — Classifier-Free Guidance for LMs↗
  12. 12Belrose et al. — The Tuned Lens↗
Scope note: processor formulas and ordering can differ across serving engines and versions. This article states general mechanisms and identifies implementation-dependent behavior rather than presenting one runtime as universal.