The Inference Factory — Reverse-engineering low-cost frontier-model serving

01 / The thesis

NVMe is visible. Decode utilization is decisive.

Disk-backed KV caching explains why repeated prompts can become cheap. It does not explain cheap output tokens. The missing advantage must live primarily in the decode path.

Reports of extraordinary inference economics tend to provoke a hunt for one hidden optimization. Perhaps the provider streams KV state from NVMe. Perhaps it has a private attention kernel. Perhaps the published model is not the model actually served.

That framing is attractive—and probably wrong. A modern inference service behaves less like a single GPU running a model and more like a factory whose output happens to be tokens. The economic unit is not one request. It is a continuously scheduled fleet, with model architecture, runtime, network, memory hierarchy, and traffic shape designed together.

Core hypothesis: the advantage is multiplicative systems co-design. Massive multi-tenant batching turns weight movement into shared infrastructure; every other optimization exists to keep that factory full.

A local benchmark commonly runs one sequence. Every generated token forces the device to stream much of the active model state from high-bandwidth memory. The operation is memory-bound, utilization is poor, and cost per token looks alarming. A provider with thousands of simultaneous sequences sees a different problem: expert weights can be reused across many tokens, GEMV becomes GEMM, communication can overlap computation, and idle gaps become schedulable inventory.

effective decode throughput = base saturated throughput × batch efficiency × low-precision gain × kernel-fusion gain × communication-overlap gain × realized speculative gain × slowest-rank expert efficiency

02 / The system

A serving stack built as one machine.

Prefill, decode, expert routing, and KV persistence have different resource profiles. Treating them as separate but coordinated production stages changes the economics.

STAGE 01

Prefix-aware admission

Hash exact token prefixes, route toward warm state, and begin asynchronous KV restoration before compute is assigned.

STAGE 02

Compute-dense prefill

Process cache misses in large chunks on a pool optimized for arithmetic throughput and long prompt sequence parallelism.

STAGE 03

Continuous decode

Merge live sequences, speculate future tokens, balance hot experts, and overlap dispatch, attention, and expert GEMMs.

HBMActive decode and hottest prefixes

DRAMWarm conversations and staging

NVMe / 3FSPersistent prefixes and cold sessions

03 / Six levers

Where the decode advantage can hide.

These mechanisms are individually familiar. Their production policies—and their interaction with real traffic—are the likely proprietary layer.

Utilization

Continuous batching as weight amortization

With enough concurrent sequences, each expert's weights serve many tokens per load. The scheduler delays work by imperceptible milliseconds to build much more efficient batches while preserving time-to-first-token and inter-token SLOs.

Speculation

Several tokens per full-model pass

Native multi-token prediction proposes a short future block. The target model verifies it in parallel. Confidence-aware draft length, workload grouping, and early abort policies determine how much theoretical acceptance becomes real throughput.

MoE placement

Duplicate the bottleneck, not the model

The slowest expert rank limits every MoE layer. Production traces expose stable code, math, language, and agent hotspots. Replicating popular experts and routing to the least-loaded copy converts parameter sparsity into execution efficiency.

P/D split

Different factories for different phases

Prefill is compute-heavy; decode is dominated by weight, KV, and network movement. Separate pools can use different parallelism, batch geometry, precision, scaling policy, and even accelerator type without forcing one compromise.

Memory

Persist prefixes, not live attention reads

NVMe is best used to restore compressed, reusable KV blocks and skip prefill—not as the synchronous source for every decoding step. Cache admission, promotion, and request routing matter more than the existence of an SSD tier.

Execution

Bytes removed are tokens gained

FP4 expert weights, FP8 KV, low-precision dispatch, fused RoPE and cache insertion, persistent graphs, and zero-copy RDMA attack the actual decode bottleneck: bytes moved and synchronization gaps, not headline FLOPs.

04 / Why SSD alone fails

Cache hits erase prefill. They do not generate the answer.

A prefix cache can turn a 128K repeated prompt into a fast state restore. Every new output token must still pass through the model.

DeepSeek publicly documents exact-prefix context caching on disk. Its 3FS storage system exposes an NVMe-backed distributed layer with RDMA-scale aggregate bandwidth. MLA and newer compressed attention mechanisms reduce the number of bytes associated with each cached token, making persistent KV practical.

But the output side remains expensive unless the decoder also improves. Reasoning traffic makes this distinction impossible to ignore: a cached prompt may be only a small fraction of the total work when the response contains thousands of generated tokens.

Likely architecture: HBM holds active state; DRAM holds warm sessions; NVMe holds reusable prefixes. Restoration is prefetched and overlapped. Synchronous SSD reads on every token would usually surrender the latency advantage.

05 / The experiment

Turn the hypothesis into falsifiable software.

The open-source simulator runs the same deterministic request trace through a generic baseline and an integrated inference factory. Every lever is visible and independently configurable.

Baseline GENERIC SERVING

Decode throughput 188 tok/s/GPU
Cache hit rate 0.0%
Speculative progress 1.00×
Expert efficiency 84.3%
Modeled gross margin −35.2%
Implied GPU payback 72.2 months

Factory CO-DESIGNED STACK

Decode throughput 1,929 tok/s/GPU
Cache hit rate 22.4%
Speculative progress 2.03×
Expert efficiency 95.2%
Modeled gross margin 77.5%
Implied GPU payback 12.0 months

Decode throughput — factory1,929 tok/s/GPU

Decode throughput — baseline188 tok/s/GPU

GPU-seconds consumed — factory13.8%

GPU-seconds consumed — baseline100%

The result is not proof that any provider operates this exact stack. It demonstrates that the reported order of magnitude is reachable without inventing a magical undisclosed accelerator. The combination of utilization, speculation, expert balance, precision, and cache reuse is sufficient under the stated calibration.

The model is deliberately conservative about speculative decoding: only 60% of the theoretical accepted-token improvement becomes throughput. It also charges capex depreciation and energy while excluding staff, storage capex, networking, taxes, and financing. The resulting margin is therefore a hardware-and-energy contribution margin, not a corporate accounting claim.

Open source · MIT

Interrogate the assumptions yourself.

Change arrival rates, cache budgets, expert replicas, draft depth, token prices, hardware cost, utilization, and precision gains. The simulator has no runtime dependencies and includes deterministic tests.

git clone https://github.com/manishklach/inference-factory-sim.git
cd inference-factory-sim
python -m pip install -e .
python -m inference_factory compare --config configs/factory.toml

View repository ↗

06 / Sources & boundaries

Public signals, explicit inference.

The architectural ingredients are public. Claims about private production policies remain hypotheses and are labeled accordingly.

The simulator is informed by public technical reports and repositories. It does not ingest leaked data, impersonate production traces, or claim privileged knowledge. Absolute numbers depend on calibration; comparative behavior is the primary object of study.