Open models · 18 July 2026

Open Weightsfrom files to tokens / second

What model weights actually contain, how an inference engine turns them into output, which hardware determines performance, and why access to the tensors opens an entirely new optimization surface.

WEIGHT SHARDS
INFERENCE ENGINEW × xroute · attend · sample
OUTPUT STREAMThemodelgeneratesonetokenatatime
Weightslearned tensors
VRAMcapacity gate
Bandwidthdecode engine
t/sneeds context
01 / Four direct answers

The executive version.
Everything else explains why.

HOW MANY T/S?

Common local models

8–150 t/s

A useful single-user range across quantized 7B–70B models. Production servers can produce thousands of aggregate t/s, while each user receives a fraction.

WHAT HARDWARE?

Start with memory

Fit, then feed.

Weights, KV cache and runtime buffers must fit. HBM bandwidth, low-precision compute and interconnect then determine how fast the model runs.

WHAT IS RELEASED?

The artifact

Tensors + recipe

Learned arrays, architecture configuration, tokenizer, chat template and license—usually not the original data or full production serving stack.

HOW DOES IT RUN?

The loop

Load → prefill → decode

The engine places tensors, processes the prompt, predicts one next token, reuses cached state and repeats until a stop condition is reached.

Open weight is not automatically open source. A downloadable checkpoint may have license restrictions and may omit its data, training pipeline and evaluation harness. Access to parameters and permission to use them are related—but different—questions.
02 / Anatomy of a release

A model repository is
a machine in pieces.

The checkpoint is the largest component, but weights alone are not enough. The runtime needs an architectural blueprint, a text codec and an exact conversation protocol.

01model-00001-of-N.safetensorsLearned tensors

Embeddings, attention projections, expert matrices, normalization parameters and output weights.

02model.safetensors.index.jsonShard map

Maps every named tensor to its file so loaders can stream, validate and place weights.

03config.jsonArchitecture blueprint

Layers, hidden dimensions, heads, experts, context length, data types and model identifiers.

04tokenizer.jsonText codec

The exact mapping between human-readable text and the token IDs consumed by the model.

05chat_template.jinjaConversation protocol

Formats system, user, assistant, tool and multimodal messages as expected during post-training.

06generation_config.jsonDecode defaults

Suggested temperature, top-p, stop tokens and generation controls—not learned knowledge.

07model code + kernelsExecution logic

Custom layers and optimized operations. A new architecture may need engine patches before it runs well.

08README + LICENSEOperational contract

Intended use, limitations, model-specific instructions and the legal permissions attached to the release.

What weights are not: a readable database, a collection of source documents, stored prompts, or guaranteed factual truth. Knowledge is distributed numerically across the network.
03 / The inference loop

How frozen numbers
become a live answer.

Inference does not search inside the weight files. It executes the network they define.

01TOKENIZEtext → IDs
02EMBEDIDs → vectors
03PREFILLprocess prompt
04FORWARDrun layers
05SAMPLEchoose token
06DECODErepeat + stream
PHASE A

Prefill

All input tokens are processed to create the initial state. Long prompts increase time to first token. Prefill is highly parallel and usually compute-heavy.

PHASE B

Decode

One output token is produced per model step. Low-batch decode often becomes memory-bandwidth-bound because the weights are streamed repeatedly.

THE BRIDGE

KV cache

Attention keys and values are retained so earlier tokens are not recomputed. This saves work but consumes memory as context and concurrency grow.

Token 101 depends on tokens 1–100. Generation for one response is therefore sequential across output tokens, even though thousands of operations inside each step run in parallel.
04 / Open weights as an optimization surface

Access to the tensors changes
what can be optimized.

An API customer can tune prompts and request parameters. An inference provider with open weights can transform the model artifact itself—and co-design its representation, execution graph and hardware placement.

CLOSED API ACCESS

Optimize around the model

  • Prompts and response length
  • Request concurrency
  • Application-side caching
  • Model and service-tier selection

The provider controls precision, kernels, placement, batching and cache policy.

→
OPEN-WEIGHT ACCESS

Optimize the model + system

  • Quantize and repack tensors
  • Shard layers or MoE experts
  • Fuse architecture-specific kernels
  • Compile fixed execution paths
  • Fine-tune or attach adapters
  • Choose KV-cache representation

The checkpoint becomes an input to a hardware-specific compilation and serving pipeline.

01

Change representation

Convert BF16 to FP8, INT8 or INT4; select group size and calibration; store scales; repack tensors into the blocked layout consumed by the target kernel.

02

Change placement

Split tensors across devices, place MoE experts for locality, replicate hot experts or consolidate a quantized model onto fewer GPUs.

03

Change execution

Fuse dequantization with matrix multiplication, fuse normalization and activation operations, capture stable graphs and select model-specific kernels.

04

Change cache policy

Use paged KV memory, lower cache precision, reuse common prefixes, or separate compute-heavy prefill from bandwidth-heavy decode.

05

Change the model

Apply pruning, distillation, low-rank adapters or task-specific fine-tuning—subject to license—and evaluate quality against the original.

06

Inspect failures

Trace layers, profile tensor shapes, measure expert imbalance and validate numerical drift—visibility unavailable through a remote endpoint.

WORKED EXAMPLE

Optimizing a hypothetical 70B dense model for one H100

This example uses capacity and bandwidth arithmetic to make the engineering process concrete. It is not a benchmark for a particular checkpoint.

STARTING CHECKPOINT70B parameters · BF16≈ 140 GB raw weights
INITIAL PLACEMENT2 × H100 80 GBtensor-parallel execution
WORKLOAD4K-token promptbatch 1 · 512 output
  1. 01

    Establish a reference

    Run a fixed prompt suite on BF16. Record task quality, TTFT, inter-token latency, output t/s, memory and power. Without a locked baseline, “optimized” has no measurable meaning.

  2. 02

    Calibrate and quantize

    Observe representative activations, select an INT4 scheme and convert large linear matrices. Sensitive tensors may stay at higher precision. Four-bit storage gives a 35 GB floor; scales and exceptions increase the operational image.

  3. 03

    Repack for the kernel

    Pack INT4 values into aligned tiles. A fused kernel expands values inside registers or shared memory and immediately performs the matrix multiplication—without writing a BF16 copy back to HBM.

  4. 04

    Consolidate placement

    If weights, KV cache and workspaces fit in 80 GB, move from two GPUs to one and remove tensor-parallel collectives. Or retain two GPUs and convert freed capacity into batch and concurrency.

  5. 05

    Spend the memory dividend

    Reserve runtime headroom, then allocate the rest to paged KV cache. More cache blocks allow more concurrent requests or longer contexts without CPU offload.

  6. 06

    Batch against an SLO

    Continuous batching combines tokens from multiple requests so a weight read serves more useful work. Raise the scheduling budget until aggregate throughput improves without breaking TTFT or ITL targets.

  7. 07

    Re-test the real workload

    Compare quantized output quality with BF16, including hard prompts and long contexts. Measure p50 and p99, not only the average, and preserve every configuration detail.

Why smaller weights can accelerate decode

At batch 1, a simplified ceiling divides aggregate HBM bandwidth by bytes of weights read per token. It ignores compute, KV traffic, communication and imperfect utilization, so real performance is lower.

BF16 · 2× H1006.7 TB/s ÷ 140 GB≈ 48 t/s ideal ceiling
INT4 · 1× H1003.35 TB/s ÷ 35 GB≈ 96 t/s ideal ceiling

The subtlety: weights became roughly 4× smaller, but consolidating from two GPUs to one also halves aggregate HBM bandwidth. The simplified ceiling therefore rises about 2×, not 4×. Keeping two GPUs may instead turn the memory dividend into greater aggregate throughput. Quantization creates options; it does not guarantee a multiplier.

WEIGHT CAPACITY~4× lower
GPU COUNT2 → 1 possible
IDEAL ROOFLINE~2× higher
ACTUAL RESULTmust be measured
Open weights do not guarantee optimization. They do not create a fast kernel, preserve quality under quantization, provide enough hardware or remove license conditions. They create the right to optimize and the visibility to measure.
05 / Memory is the admission ticket

Can the weights fit?

RAW WEIGHT BYTESparameter count × bits per weight ÷ 8
EXAMPLE · KIMI K3
2.8T × 4 ÷ 8
= 1.4 TB raw

The floor excludes block scales, higher-precision tensors, alignment, activations, communication workspaces, graph buffers, KV cache and safety margin. At BF16, the same parameter count would require 5.6 TB before overhead.

01 / CAPACITY

VRAM / HBM

Weights plus live state must fit. CPU or storage spill can make transfers the critical path.

02 / FEED

HBM bandwidth

Low-batch decode repeatedly streams weights. Idle tensor cores cannot outrun starved memory.

03 / MATH

Low precision

Native FP8, INT8, INT4 or MXFP4 kernels determine whether fewer bytes become speed.

04 / FABRIC

Interconnect

Tensor and expert parallelism exchange partial results and routed activations across devices.

06 / Tokens per second, decoded

One label.
Four measurements.

TTFT

Time to first token

Queueing, prompt prefill and first decode. The visible latency for long prompts.

ITL

Inter-token latency

Milliseconds between streamed tokens. Approximately 1,000 ÷ decode t/s.

USER

Per-sequence t/s

The rate experienced by one response—the useful conversational metric.

FLEET

Aggregate t/s

Total server output across all requests—the useful capacity and economics metric.

50 t/sper user
×
200active sequences
=
10,000 t/saggregate, if sustained
A complete benchmark sentence: “Model X produced 42 output t/s per sequence at batch 1, after a 4,096-token prompt, generating 512 tokens, using revision Y in 4-bit precision on hardware Z.” Missing nouns make the number ambiguous.
07 / Practical planning ranges

What can actually
be achieved?

Illustrative single-sequence decode ranges, not promises. Engine, context, quantization, thermals and hardware generation can move them substantially.

ModelWeightsPractical hardwareDecodeGood fit
7–8B · 4-bit5–6 GB16–24 GB consumer GPU50–150 t/sLocal assistant
13–14B · 4-bit9–11 GB16–24 GB consumer GPU30–90 t/sHigher-quality chat
30–35B · 4-bit18–24 GB24–48 GB GPU15–50 t/sCoding / specialist
70–72B · 4-bit40–48 GB48–80 GB or 2 GPUs8–30 t/sPrivate inference
70B · BF16~140 GB2× 80 GB+ data-center GPUs30–100+ t/sProduction serving
K3 · MXFP4≥1.4 TB raw64+ linked acceleratorsNot measured yetFrontier hosted inference
08 / The K3 case

Why the K3 “gold rush”
will begin in data centers.

2.8Ttotal parameters
16 / 896selected / total experts
MXFP4weights; MXFP8 activations
64+accelerators recommended

Sparse compute is not small storage.

Only a fraction of experts execute for each token, lowering arithmetic relative to a 2.8T dense model. But any expert may be selected, so the expert bank must remain reachable across the fabric. Expert parallelism places experts on devices and moves routed activations to resident weights.

What t/s should we expect?

No reproducible open-weight number exists before the promised July 27, 2026 checkpoint release. A prudent early planning target is tens of t/s per interactive sequence on an optimized 64–72 accelerator supernode, with aggregate throughput potentially in the thousands or tens of thousands. That is a projection, not a benchmark.

The honest headline: until weights and serving recipes are public, K3 speed is a range to engineer for—not a number to quote.
09 / The optimization stack

Speed is a stack of
compounding wins.

01

Quantize

Reduce weight bytes and memory traffic; validate quality and native kernel support.

02

Fuse kernels

Reduce launches, synchronization and intermediate memory movement.

03

Cache prefixes

Reuse shared system prompts, tool definitions and document prefixes.

04

Batch continuously

Fill accelerator steps with tokens from multiple requests while enforcing latency goals.

05

Page KV memory

Allocate cache in blocks to reduce fragmentation and support dynamic sequences.

06

Parallelize deliberately

Use tensor, pipeline and expert parallelism according to model shape and topology.

07

Decode speculatively

Let a draft model propose tokens and verify several with the target model at once.

08

Split prefill / decode

Scale compute-heavy prompt work and bandwidth-heavy generation independently.

09

Measure the workload

Tune with real prompts, output lengths, concurrency distributions and SLOs.

The compact mental model: the weights determine what the model knows. The inference stack determines what that knowledge costs.
10 / Sources & method

Facts, estimates,
and clean labels.

Primary model-maker, engine-maintainer and hardware-vendor documentation was prioritized. Speed ranges are planning estimates because t/s is workload-dependent. K3 statements reflect information available on 18 July 2026, before the promised weight release.

  1. 01Moonshot AI — Kimi K3 technical blog↗
  2. 02Hugging Face — Safetensors format↗
  3. 03Hugging Face — Tensor metadata↗
  4. 04Hugging Face — KV-cache strategies↗
  5. 05vLLM — Parallelism and scaling↗
  6. 06vLLM — Expert-parallel deployment↗
  7. 07NVIDIA — H100 specifications↗
  8. 08NVIDIA — DGX B200 specifications↗
  9. 09NVIDIA — GB200 NVL72↗
  10. 10AMD — Instinct MI350 series↗
  11. 11llama.cpp — Benchmark methodology↗
  12. 12TensorRT-LLM — Quantization↗
Publication rule: a measured result identifies model revision, precision, engine, hardware, input length, output length, batch or concurrency, and whether throughput is per sequence or aggregate. Everything else is labeled estimate or projection.