The $/MToken Model:
Inference Economics

Every token price you have ever seen is a GPU-hour price wearing a costume. A provider charging $0.14 per million input tokens is not performing magic; they are dividing a roughly $1.65 GPU-hour by roughly twelve million tokens per hour and keeping the difference. Build the cost model from the hour up — amortization, power, utilization — and the industry's wildest price gaps stop looking like strategy and start looking like arithmetic.

01 The cost stack

What a GPU-hour really costs

Start with the hour, because the hour is the only thing you actually buy. Nobody sells tokens; they sell time on silicon and divide. A fully-loaded H100-hour — the card, the host around it, the power, the building, the people — lands somewhere between $1.50 and $2.50 at scale. Here is the arithmetic, with every assumption flagged so you can substitute your own.

GPU amortization: $0.80/hr. An H100 SXM 80GB changed hands for roughly $25,000–$30,000 through 2024–25; take $28,000 as the midpoint. Amortize it over four years: $28,000 ÷ (4 × 365 × 24) = $28,000 ÷ 35,040 = $0.80/hr. Four years, not seven, because a Hopper bought in 2024 competes with Blackwell in 2025–26 and resale value falls off a cliff with each generation. Sensitivity, since you will ask: five-year life gives $0.64/hr; a $32,000 purchase price gives $0.91/hr. The amortization line is the single biggest component, which means purchase price and useful life move the model more than anything in the power bill. Push the sensitivity further: a three-year useful life (aggressive, but defensible if Blackwell supply is plentiful) takes GPU amortization to $28,000 ÷ 26,280 = $1.07/hr and the stack to ~$1.92/hr. A six-year life (buy late, run into the ground) gives $0.53/hr and ~$1.38/hr. Your $/MToken inherits whichever lifetime you believe — there is no physics here, only depreciation schedules.

Host and network amortization: $0.27/hr. The GPU is only about three-quarters of the capex. A DGX-H100-class 8-GPU node costs on the order of $300,000 all-in — $37,500 per GPU slot. Subtract the $28,000 card and you have $9,500 per GPU of host servers, InfiniBand or high-end Ethernet, racks, and power distribution. Over the same four years: $9,500 ÷ 35,040 = $0.27/hr.

Power: $0.08/hr. The H100's thermal design power is 700W; add roughly 200W of host share per GPU (CPUs, DRAM, NICs, fans, divided eight ways) for 900W. A decent datacenter runs at PUE ≈ 1.25, so 1.125 kW at the meter. At $0.07/kWh: 1.125 × 0.07 = $0.08/hr. Note how small this is — under five percent of the stack. Hyperscalers with $0.04/kWh power contracts pay about $0.045; a European neocloud at $0.12/kWh pays $0.135. Power matters at the margin, but "AI economics is a power story" is mostly wrong; it is an amortization story with a power footnote.

Facility and network opex: $0.30/hr. Floor space, cooling operations, backbone transit, spare parts, the unglamorous plumbing. Staff and ops at scale: $0.20/hr — SRE, network engineering, datacenter technicians amortized over ten thousand or more GPUs; this line collapses with scale and explodes without it.

Add it up: 0.80 + 0.27 + 0.08 + 0.30 + 0.20 = $1.65/hr fully loaded, per GPU-hour of clock time. An 8-GPU node: $13.20/hr. Now the correction that matters: $1.65 buys a clock hour, but tokens only come out of busy hours. A healthy inference fleet with diurnal demand averages perhaps 60% utilization. The cost per busy GPU-hour is therefore $1.65 ÷ 0.60 = $2.75. That $2.75 — not $1.65 — is the number that goes into token math, and it is why providers obsess over filling GPUs the way airlines obsess over filling seats.

One more turn of the crank: what happens when the hardware generation changes. Decode throughput scales with memory bandwidth — the 24 tok/s/GPU in section 02 is just 70GB ÷ bandwidth. A next-generation card with ~8TB/s of HBM (the public spec class of Blackwell B200, vs 3.35TB/s on H100) moves ~2.4× more tokens per second per GPU on the same memory-bound workload. If its fully-loaded hourly cost rises by a factor c, $/MToken changes by c ÷ 2.4. At c = 1.5, unit cost falls 37% overnight with zero software changes. This is why hardware transitions repricing the numerator are the closest thing inference economics has to free money — and why the framework in this essay outlives any particular card: recompute the $/hr, re-derive the tok/s, divide.

Not everyone pays $1.65. The stack compresses or expands with who you are:

The hyperscaler row is an estimate — their purchase prices and power contracts are not public — but the direction is right: vertical integration and scale buy roughly a third off the neocloud stack. The on-demand row is what you pay when you rent one H100 by the hour; the gap between $1.65 and $3.00 is the cloud's margin plus the cost of idle capacity you never see. Keep this table in mind in section 05: "should we self-host" is largely a bet on which row you can live in.

The diagram shows the five components of a fully-loaded H100-hour stacked to scale. The takeaway is the shape, not just the total: amortization (GPU plus host) is two-thirds of the stack, which means purchase price and hardware lifetime dominate inference economics, and power — the line item that gets the headlines — is under five percent.

02 Utilization is the whole game

Tokens per hour per GPU

The entire pricing model of the inference industry fits in one equation:

$/MToken = (hourly GPU cost × 10⁶) ÷ (tokens/sec/GPU × 3600 × utilization)

The numerator is the $1.65 from section 01 — boring, slow-moving, negotiated in power contracts and purchase orders. The denominator is where fortunes are made. Let us derive it for a concrete case: a 70B-parameter model in FP16 on an H100.

Decode — generating one token at a time — is memory-bandwidth bound. Each token requires reading every weight once: 70B parameters × 2 bytes = 140GB. The model does not fit on one 80GB H100, so split it across two (tensor parallel degree 2): 70GB per GPU. H100 HBM3 delivers about 3.35 TB/s of memory bandwidth, so one token takes 70 ÷ 3350 = 0.0209 seconds per GPU — 20.9 milliseconds. That is 47.8 tokens/second for the pair, or ≈ 24 tokens/second/GPU at batch size 1. (First-order estimate: it ignores KV-cache reads and attention compute, which matter more at long context — section 03.)

A vocabulary note, because it sharpens every discussion that follows. Training people talk about MFU — model FLOPs utilization, the fraction of peak FLOPS actually used. Inference serving lives and dies by MBU — memory bandwidth utilization, the fraction of peak HBM bandwidth doing useful weight and KV reads during decode. The 24 tok/s/GPU derivation assumed 100% MBU; production stacks land at 60–80%, with the rest lost to kernel launch overhead, imperfect batching, and attention recomputation. When someone tells you their serving stack is "highly optimized," ask for MBU, not MFU — MFU near zero is normal in decode, and quoting it signals either confusion or marketing.

Plug it into the equation at $1.65/hr and 100% duty cycle: 1,650,000 ÷ (24 × 3600) = $19.10 per million tokens. Read that again. An unbatched 70B costs roughly nineteen dollars per million tokens to serve. Nobody prices that way, because nobody serves that way — but the number is the honest cost of generating tokens one request at a time, and it explains why every serious serving stack is, at its core, a batching machine.

Now batch. With continuous batching (the Orca idea, productized in vLLM) and chunked prefill (Sarathi-Serve's contribution: slicing long prefills into decode-sized chunks so they ride along without latency spikes), the weight read is amortized across the batch. At batch 32, aggregate throughput is roughly 32 × 47.8 ≈ 1,530 tokens/second for the pair, or ~770 tokens/second/GPU — and the cost falls to 1,650,000 ÷ (770 × 3600) = $0.60/MToken. Thirty-two times cheaper, same hardware, same model. The batch is capped by KV-cache memory: a 70B with grouped-query attention (80 layers, 8 KV heads, head dim 128) stores 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes — 320KB per token. At 2K tokens of context per request, batch 32 needs 32 × 2048 × 320KB ≈ 21GB of KV cache, which plus 70GB of weights is right at the edge of an 80GB card. The memory wall is the batch wall is the cost wall. And the wall moves with context length — the batching ladder for this 70B at $1.65/hr:

Read the last row carefully: batch 128 at $0.15/MToken is real, but only for short contexts. Stretch those requests to 2K tokens and the KV cache alone wants 128 × 2048 × 320KB ≈ 84GB — more than the card holds. Long context does not just cost more per token; it lowers the batch ceiling, which raises every token's price. Context length is a throughput tax, and it compounds.

Move to a smaller model and the numbers get silly. A well-tuned 8B-class model in FP16 on an H100 sustains on the order of 2,000 tokens/second/GPU in production serving stacks (vLLM, SGLang, TensorRT-LLM all land in this neighborhood with good batching): 1,650,000 ÷ (2000 × 3600) = $0.23/MToken. From $19.10 to $0.23 — an 83× spread — without touching the numerator. Same $1.65 hour. Different denominator.

Three engineering levers move the denominator, in rough order of impact. Batching is first: continuous batching, chunked prefill, and disaggregated prefill/decode (Splitwise, DistServe) exist to keep the matrix units fed. Quantization is second: decode is memory-bandwidth bound, so FP8 weights halve the bytes per token and roughly double tokens/second/GPU — the arithmetic is that direct. Prefix caching is third: SGLang's radix-attention KV reuse means repeated system prompts and few-shot prefixes skip prefill entirely, converting expensive input tokens into cache lookups.

And then the utilization trap. Everything above assumed the GPU is doing useful work whenever the clock runs. Real traffic has diurnal curves, latency SLOs that cap batching (you cannot wait 500ms to fill a batch when P99 must be 200ms), and idle gaps between requests. A fleet at 60% average utilization pays the $2.75 busy-hour rate from section 01. The providers winning the price war are not the ones with the cheapest GPUs; they are the ones whose schedulers keep the denominator high at 3 a.m.

03 Prefill vs decode cost split

Not all tokens are equal

Input tokens and output tokens are priced differently everywhere — output typically 2–4× input — and the reason is physical, not marketing. Prefill (processing the prompt) is compute-bound; decode (generating tokens one by one) is memory-bandwidth bound. They live in different performance regimes on the same chip.

Prefill arithmetic for the same 70B FP16 on 2× H100: processing 1,000 prompt tokens costs roughly 2 × 70e9 × 1000 = 1.4×10¹⁴ FLOPs. An H100 delivers ~989 TFLOPS dense FP16; at a realistic 45% model FLOPs utilization (attention, communication, and kernel overhead take their cut) that is ~445 TFLOPS per GPU, ~890 for the pair. Time: 1.4e14 ÷ 8.9e14 = 0.157 seconds per 1,000 tokens — about 6,400 tokens/second for the pair, 3,200/GPU. (Using the pair's combined 890 TFLOPS; per-GPU prefill throughput ≈ 3,200 tok/s.) Compare decode at 24 tok/s/GPU: prefill moves tokens roughly 130× faster per GPU than decode at batch 1. Per-token GPU-time: ~0.3ms for a prefill token vs ~42ms for a decode token.

Now a typical chat mix — 1,000 input tokens, 500 output tokens. GPU-time: 1000 ÷ 3200 + 500 ÷ 24 = 0.31 + 20.8 = 21.1 GPU-seconds, of which decode is 98.7%. The input tokens are a rounding error in the cost. That is the entire explanation for input/output price asymmetry: providers are not charging for tokens, they are charging for GPU-seconds, and output tokens consume ~130× more of them. In practice the ratio lands at 2–4× rather than 130× because prefill is billed below its speed (it batches beautifully, and chunked prefill lets it ride for free alongside decode) and because pricing is also a competitive signal — but the direction and the order of magnitude come straight from the roofline.

This is also why the industry is disaggregating. Splitwise (Microsoft), DistServe, and Mooncake (Moonshot AI) split prefill and decode onto different GPUs — prefill instances tuned for FLOPs, decode instances tuned for memory bandwidth and KV-cache capacity — and schedule them independently. Once the phases live on different machines, they can be priced, scaled, and optimized separately, and the "input vs output" price gap becomes an explicit architectural choice rather than a blended guess. The hardware follows the phase, too: prefill wants FLOPs-heavy cards with modest memory, decode wants maximum HBM capacity and bandwidth per dollar. A fleet of identical 8-GPU H100 nodes is the wrong shape for disaggregated serving — expect the efficient operators to run heterogeneous pools, and expect their cost stacks to diverge from the single $1.65/hr number in section 01 accordingly.

The input side has its own cost driver: long context. Recall the 320KB-per-token KV cache for a 70B-class GQA model. A 128K-token context holds 128,000 × 320KB ≈ 41GB — more than half an H100's HBM — for the lifetime of that request. The token was cheap to prefill; it is expensive to keep. Providers handle this two ways: context-window pricing tiers, and KV-cache-specific pricing. Anthropic's prompt caching is the clearest public example — cache writes billed at 1.25×, cache reads at 0.10× — which is really a confession that a cache hit costs a tenth of a fresh prefill because the GPU-seconds were already spent. OpenAI's cached-input discounts tell the same story. When you see "input tokens $X, cached input $0.1X," you are looking at the prefill/decode split expressed as a price list.

The gap between the 130× physical ratio and the 2–4× commercial ratio is the interesting part: it is where batching efficiency, cache-hit rates, competitive positioning, and margin strategy all collide. Which brings us to the case study where someone published the actual numbers.

04 The DeepSeek case study

Reading the famous cost claims

In December 2024, DeepSeek published the V3 technical report with a number that detonated across the industry: 2.788 million H800 GPU-hours of training, costing $5.576 million at an assumed $2/hr rental price. The arithmetic is trivially verifiable — 2,788,000 × 2 = 5,576,000 — and that is precisely the problem. The number is a rental proxy for the final training run. It excludes every failed experiment, the data pipeline, the salaries of the researchers who invented MLA and Multi-Token Prediction, and the cluster those researchers sat on. SemiAnalysis counter-estimated roughly 50,000 Hopper GPUs under DeepSeek's control — an order of magnitude more capex than the headline implies (third-party estimate, flagged as such). The $5.576M is real arithmetic on a narrow definition; treat it as the cost of the last mile, not the journey.

Far more interesting for our purposes is what DeepSeek published during Open Source Week in February 2025: a full day of their inference statistics. This is the rare case of a frontier lab showing its serving math, and every number below is theirs, from their open-infra-index day-6 writeup. Over a 24-hour window, V3 and R1 inference ran on a peak of 278 nodes and an average of 226.75 nodes, each with 8 H800 GPUs. At their assumed $2/hr lease: 226.75 × 8 × 24 × 2 = $87,072/day. In that window they served 608 billion input tokens — 342B of them (56.3%) hitting an on-disk KV cache — and 168 billion output tokens. Per node: ~73,700 input tokens/second during prefill, ~14,800 output tokens/second during decode.

Now the waterfall they invited. "If all tokens were billed at R1's pricing" — $0.55/M input on cache miss, $0.14/M on cache hit, $2.19/M output — "the total daily revenue would be $562,027, with a cost profit margin of 545%." Check it: input revenue = 342,000M × $0.14 + 266,000M × $0.55 = $47,880 + $146,300 = $194,180; output revenue = 168,000M × $2.19 = $367,920; total $562,100 (rounding to their $562,027). Profit over cost: ($562,027 − $87,072) ÷ $87,072 = 5.45 → 545%. The arithmetic holds. So do the caveats, which they state plainly: actual revenue was "substantially lower" because web and app traffic is free, V3's pricing sits well below R1's, and only a subset of services is monetized.

Derive the unit economics from their stats. Blended cost: $87,072 ÷ 776,000M tokens = $0.112/MToken. Blended hypothetical price: $562,027 ÷ 776,000M = $0.724/MToken. Margin on revenue: 84.5%. And the throughput that makes it possible: 776B tokens ÷ (1,814 GPUs × 86,400 s) = 4,951 tokens/second/GPU blended — cache hits included. One subtlety: that 4,951 is flattered by the 56.3% cache-hit rate, because a cache-hit input token costs a disk read, not a prefill. Strip the 342B cache-hit tokens out and the fresh-compute throughput is roughly half. That is not a criticism — converting prefill into cache lookups is the entire point of the on-disk KV tier — but it means you cannot copy the 4,951 number onto a workload with no prefix reuse and expect the same economics. Throughput is always workload-specific; anyone quoting a single tok/s/GPU without the cache-hit rate is selling you something. Put the measured 4,951 on the hyperbola anyway: at ~5,000 tok/s/GPU, even a $1.65/hr stack serves at $0.09/MToken — and the $0.14 input price stops looking aggressive.

The hyperbola shows unit cost collapsing as effective throughput rises, at a fixed $1.65/GPU-hr. The four marked points are the section-02 derivations plus DeepSeek's measured 4,951 tok/s/GPU. The takeaway: every horizontal price line is crossed somewhere — $0.14/MToken input pricing is ruinous at 24 tok/s/GPU and comfortable at 5,000. Price is a throughput bet.

What engineering buys that throughput? Three things, all visible in their writeup. MLA (multi-head latent attention): instead of caching full key/value heads, V3 caches a 512-dim compressed latent plus 64-dim RoPE key per layer — 576 elements × 2 bytes × 61 layers = 70,272 bytes, ≈69KB per token, versus ~320KB for a comparable GQA 70B. That is a 4.6× smaller KV cache, which is 4.6× more batch headroom before the memory wall from section 02. FP8 end-to-end (matmuls and dispatch in FP8, MLA core in BF16) halves weight traffic in the memory-bound decode regime. And disaggregated prefill/decode with an on-disk KV cache tier — the 56.3% cache-hit rate means most input tokens never touch a GPU at all.

Honest caveats, because the headline margin invites over-reading. The $2/hr H800 lease is their assumption, applied to hardware you cannot buy at that price outside China. The 24-hour window is cherry-picked peak load; nighttime nodes get reallocated to training, which is clever capacity management, not free inference. Free web/app traffic sits in the cost denominator but not the revenue numerator. And nothing here amortizes the R&D that produced MLA, DeepEP, or DualPipe. The right reading: the engineering is real and the unit economics at scale are genuinely excellent — and the 545% is a peak-day, narrow-definition number, not a business P&L.

Turn their stats into the waterfall the brief asks for, on a blended per-million-token basis:

The waterfall starts from the blended $0.724/MToken hypothetical revenue, subtracts the $0.112/MToken measured infra cost, subtracts a $0.020/MToken overhead estimate (support, billing, idle capacity — the author's estimate, flagged), and lands at $0.592/MToken gross margin, about 82%. The takeaway: at 4,951 effective tok/s/GPU, the infra cost is a 15% line item, not the business — which is exactly why throughput engineering dominates inference strategy.

05 What this means for builders

Buy, build, or batch

The build-vs-buy decision is a crossover problem with one variable that matters: monthly volume at your achievable throughput. Work it with the numbers from this essay. Suppose you serve 500 million tokens a month — about 193 tokens/second average — at a blended API price of $1.50/MToken. API bill: 500 × $1.50 = $750/month. Self-hosting a 70B-class model needs tensor parallelism across at least 2 GPUs: 2 × $1.65 × 730 hours = $2,409/month on the neocloud stack. At this volume, the API wins by 3×, and it is not close.

Now find the crossover. Those 2 GPUs at a sustained, SLO-friendly 300 tokens/second/GPU move 2 × 300 × 2,592,000 seconds/month = 1,555 million tokens/month. Effective self-host cost at full tilt: $2,409 ÷ 1,555M = $1.55/MToken — essentially the API price. So the crossover sits near ~1.6 billion tokens/month: below it, buy; above it, build if you can keep the GPUs full. That last clause is doing heavy lifting. Most teams overestimate their utilization by 2–3× (diurnal curves, bursty agents, idle nights), which pushes the real crossover toward 3–5B tokens/month. Run your own duty-cycle numbers before believing the spreadsheet. The deeper table is unit cost versus achieved throughput on that fixed $2,409/month 2-GPU unit:

The table is the whole decision in one glance: self-hosting does not win on volume alone — at 300 tok/s/GPU you merely tie the API at any scale. It wins when your engineering sustains high throughput, because the hardware is a fixed cost and every token above break-even is nearly free. Which is, once again, the denominator game.

Whether you buy or build, the cost levers come in a fixed order. First, utilization — batching discipline, SLO-aware scheduling, filling the night shift with batch workloads. It is the only lever with 10–80× of headroom, per section 02. Second, quantization — FP8 (or INT8/FP4 where the quality budget allows) roughly doubles decode throughput because decode is memory-bandwidth bound and you just halved the bytes. Third, prefix reuse — cache your system prompts, tool definitions, and few-shot blocks; a 90% prefix-hit rate on agentic workloads converts most input tokens from $0.55/MToken work into $0.14/MToken lookups (using DeepSeek's price list as the shape of the curve).

A note on the "SLO tax": every lever above fights your latency budget. Larger batches mean queueing delay; aggressive quantization risks quality regressions you discover in production; disaggregated prefill/decode adds a network hop. The providers charging the lowest prices are not skipping this tradeoff — they have simply instrumented it. If you self-host, budget engineering time for the same instrumentation, or your $1.55/MToken model quietly becomes $4/MToken at 40% utilization.

06 Open questions

Where the model breaks next

Price wars vs cost floors. Input pricing at $0.10/MToken looks predatory until you run section 02's math at 5,000+ tok/s/GPU — then it is a 30–40% gross margin. The floor is not set by silicon; it is set by whoever sustains the highest effective throughput. Watch for the moment a provider's price implies throughput their architecture cannot deliver. That is when "strategy" becomes subsidy, and subsidies end.

Inference silicon reprices the numerator. TPUs (v6/Trillium and successors), Trainium, and Cerebras all attack the $1.65/hr stack from different sides: cheaper FLOPs per dollar of capex, better perf-per-watt, or wafer-scale memory bandwidth that moves the decode wall. If an accelerator halves the fully-loaded hourly cost at equal throughput, every hyperbola in section 04 shifts down by half. The essay's framework survives; its constants do not. Recompute annually.

Spot and preemptible inference markets. Batch workloads — distillation, evals, offline agents — do not need 3 a.m. latency SLOs. A real spot market for inference (interruptible capacity at 40–60% discounts, already emerging in nascent form) would let the utilization trap become someone else's arbitrage: run batch tokens on capacity that would otherwise idle. The open question is preemption granularity — killing a 30-minute agentic rollout mid-decode is not the same as killing a stateless batch job, and checkpointing KV cache across preemptions is still research. Watch for providers to productize this first as "batch API" tiers (discounted, slow, non-interactive) rather than true spot markets; the economics are identical, but the interface hides the preemption problem.

Jevons paradox, token edition. Every halving of $/MToken in history has been met with more than a doubling of tokens consumed — longer contexts, reasoning traces, agentic loops. DeepSeek's own stats show the shape: 776B tokens a day, most of them cache hits and machine-generated. The cost model says tokens get cheaper; the demand curve says we will spend the savings and then some. Budget for tokens the way you budget for bandwidth: the unit price falls, the bill rises.

What to watch: per-phase pricing as disaggregation spreads (prefill and decode billed as separate resources); KV-cache capacity becoming an explicit billed dimension alongside tokens; and providers quietly reporting effective tok/s/GPU the way clouds report PUE — because once the denominator is public, the price war becomes an engineering war, and that is a war worth watching.