TP vs PP vs EP vs DP:
The Communication Math

Every parallelism strategy is a deal with the network: split the work, pay in bytes moved. On 8 H100s serving a 70B model, tensor parallelism moves about 5 MB per token per step in all-reduces; pipeline parallelism moves 48 KB but leaves three-quarters of the pipe idle at batch 1; expert parallelism moves a similar 5 MB in all-to-alls and adds stragglers; data parallelism moves nothing and demands 140 GB of HBM per GPU that it does not have. The right strategy is the one whose tax is cheapest in your SLO's currency — and the arithmetic fits on a napkin.

01 Tensor parallelism: split every layer

the all-reduce tax

Tensor parallelism is the latency strategy, and its mechanism is the oldest in the book. Megatron-LM split every matrix multiply inside the transformer: the QKV projection goes column-parallel across ranks, the attention output projection goes row-parallel, and the MLP follows the same pattern — up-projection column-parallel, down-projection row-parallel. A row-parallel GEMM leaves each rank holding a partial sum, so before the residual connection the ranks must add their pieces together. That summation is an all-reduce, and it fires twice per layer on every forward pass: once after attention, once after the MLP.

The message is small. One all-reduce moves hidden × 2 bytes per token — 8,192 × 2 = 16 KB in fp16. A ring all-reduce moves roughly 2(N−1)/N × the message per rank in each direction (reduce-scatter plus all-gather), so budget 32 KB per all-reduce per token. Two all-reduces per layer, eighty layers:

2 × 2 × 8,192 × 2 B × 80 ≈ 5 MB per token per decode step, on every rank.

At 900 GB/s of theoretical NVLink bandwidth, 5 MB is 5.7 microseconds — a rounding error. On a 64 GB/s PCIe-class link it is 80 µs — still a rounding error next to milliseconds of compute. If bytes were the whole story, tensor parallelism would be free.

Bytes are not the story. Launches are. A single decode step fires 160 NCCL collectives — two per layer, eighty layers — and each launch costs on the order of 10 µs before a single byte moves. That is roughly 1.6 ms of pure launch overhead per token, against about 5.2 ms of compute from streaming 17.5 GB of sharded weights per GPU through 3.35 TB/s of HBM3. At batch 1, close to a quarter of every token's time is the network stack saying hello. This is the optimization that vLLM's custom all-reduce kernels and CUDA graphs actually target: not bandwidth, but the per-collective fixed cost.

TP still wins small batches, because it has no bubble and no straggler: every rank does identical work, and latency is one stage's compute plus the collectives. It also shards everything worth sharding — weights drop to 17.5 GB per GPU, and the KV cache splits with the attention heads, so per-GPU KV shrinks as TP degree grows. The natural limit is the NVLink domain: TP=8 is the default on an 8-GPU baseboard precisely because stretching TP across nodes puts those 160 collectives on a link with a fourteenth of the bandwidth and several times the launch latency. Multi-node TP is where the 5 MB stops being a rounding error.

Two refinements matter in practice. First, Megatron's later sequence parallelism shards the LayerNorm and dropout along the sequence dimension, cutting activation memory by the TP degree — it converts one of the all-reduces into a reduce-scatter/all-gather pair, same bytes, less memory. Second, the TP degree must divide the attention heads: Llama-70B has 64 query heads and 8 KV heads, so TP=8 splits both cleanly, while TP=16 would leave KV heads replicated and per-GPU KV memory would stop shrinking. The head count, not the interconnect, is often what caps the TP degree.

TP also shards the KV cache with the heads: 4 KB per token per layer across 80 layers is 320 KB per token total, or 40 KB per token per GPU at TP=8. And the whole analysis so far is decode. Prefill flips the bottleneck: with 4,096 prompt tokens, each all-reduce message is 4,096 × 16 KB = 64 MB, ring traffic 128 MB, and a full forward pass moves about 20 GB per rank — 22 ms on NVLink against roughly 72 ms of FLOPs-bound compute (2 × 70B × 4,096 ≈ 573 TFLOP, ÷8 GPUs ÷ ~990 TFLOPS). At prefill scale the bytes finally matter, which is why prefill parallelism choices differ from decode's.

NCCL does not even use the ring for these messages. The 2(N−1)/N figure is the ring algorithm’s traffic; for small messages NCCL switches to tree algorithms and low-latency protocols (LL, LL128) that trade bandwidth for fewer hops. At 16–32 KB, a tree all-reduce over NVLink completes in a handful of microseconds of wire time — which is precisely why the measured cost is almost entirely the ~10 µs launch. The protocol choice is another quiet confirmation: the stack gave up on bandwidth efficiency for these sizes long ago.

One asymmetry with training is worth stating: these 160 collectives are the inference count — forward pass only. Training needs the backward pass too, roughly doubling it to ~320 all-reduces per step plus the gradient sync, which is why training interconnect requirements are harsher and why inference can sometimes get away with fabrics training cannot.

The 160-launch count assumes no fusion, and production stacks fuse aggressively. TensorRT-LLM folds the all-reduce into the GEMM epilogue so the partial sums never make a separate trip, and vLLM's custom all-reduce bypasses NCCL entirely for small messages, using direct peer-to-peer writes over NVLink in a single kernel. Both exist because the profiler said the same thing the napkin says: at decode batch sizes, the collective's fixed cost is the whole game.

As batch grows the story bends but does not break. At batch 64 the messages are 1 MB — still launch-bound at ~10 µs against ~1 µs of wire time. The step stays memory-bound until batch ≈ 300, where 300 × 17.5 GFLOP of compute per GPU finally catches the 5.2 ms weight stream; past that, TP's comm fraction actually falls, because FLOPs-bound compute grows faster than the flat per-token launch tax. TP's worst regime is exactly where it is most used: small-batch, latency-sensitive decode.

02 Pipeline parallelism: split by layers

the bubble tax

Pipeline parallelism shards the model by depth instead of by width: with p = 4 stages, each GPU owns 20 consecutive layers. The only communication is point-to-point — each stage hands its activations to the next. Three boundary crossings per step at 16 KB per token: 48 KB total, two orders of magnitude fewer bytes than TP's 5 MB. PP's tax is not measured in bytes at all. It is measured in idle time: the pipeline bubble.

With p stages and m microbatches in flight, each stage works for m of the m+p−1 slots in a wave and idles for p−1 — the GPipe formula, an idle fraction of (p−1)/(m+p−1). Schedulers have spent years fighting it: interleaved schedules deal each rank several virtual stages, shrinking the bubble by roughly the virtual-stage count. (Leave 1F1B out of this story: PipeDream-Flush's one-forward-one-backward rhythm pipelines the backward pass to bound activation memory, and inference has no backward pass — inference pipelines are forward-only, so the bubble arithmetic above is the whole story.) But the formula is undefeated. The bubble only vanishes as m goes to infinity.

Decode is PP's worst phase, because a decode step's microbatch is a thin slice of the batch. At batch 1, m = 1 and the bubble is 3/4 = 75%: three stages idle while one works, and per-token latency becomes 4 × 5.2 ms = 20.8 ms — that's PP4+TP2 over 8 GPUs (17.5 GB per rank); pure PP4 with 35 GB per stage would be ~10.4 ms/stage, ~41.8 ms end-to-end — against TP8's 6.8 ms. At batch 64 with m = 8 microbatches of 8 tokens, the bubble falls to 3/11 ≈ 27%: an 11-slot wave of 5.2 ms stages takes 57.2 ms for 64 tokens, or 894 µs per token — still 8× TP8's 112 µs on the same hardware. PP on a single NVLink node is a strawman; its home is multi-node, where the alternative is not TP8 but cross-node collectives.

PP wins where microbatches are plentiful and bytes are scarce: prefill. A 4k-token prompt chopped into microbatches keeps m large and the bubble thin, while each stage's 35 GB of weights stays put — no 5 MB all-reduce per token. That is why phase-disaggregated serving pairs PP-heavy prefill pools with TP decode pools, in the style of Splitwise: match the parallelism to the phase. PP is also the only strategy that scales past a node's memory. A trillion-parameter model does not fit in 8 × 80 GB no matter how you shard within the node; stages are how it gets served at all.

The memory side of the schedule matters as much as the bubble. GPipe keeps m microbatches of activations alive per stage: very roughly 5 × hidden × 2 bytes per token per layer across 20 layers is ~1.6 MB per token, so a 4k-token microbatch pins ~6.5 GB per stage — times m = 8 microbatches, 52 GB of activations alone, which does not fit next to 35 GB of weights. (1F1B, the classic training fix for this, does not transfer: its one-forward-one-backward rhythm bounds activation memory by interleaving the backward pass, which inference does not have. In inference the forward activations of all m in-flight microbatches simply have to fit. Activation estimates are rough; the ratio is what matters.)

There is a tension the formula hides: bigger microbatches raise m and shrink the bubble, but each microbatch costs activation memory — the same 6.5 GB per stage per 4k-token microbatch. The schedule is a three-way trade between bubble fraction, activation memory, and microbatch count, and interleaving is the known Pareto move for the forward-only case. Anyone proposing a new pipeline schedule has to beat all three at once.

One subtlety in the PP4+TP2 hybrid from the tables: with TP inside the stage, the 16 KB activation at each stage boundary is sharded across the two TP ranks, so the handoff needs a small all-gather before the point-to-point send — a third collective type hiding inside the "p2p only" story. And PP composes naturally with chunked prefill in the Sarathi-Serve style: slicing prompts into chunks keeps m high independent of request count, which is exactly the regime where the bubble starves.

PP also changes the failure domain. A dead GPU in TP8 stalls the whole serving unit — every collective waits for every rank. A dead stage kills a PP pipeline just as dead, but DP replicas degrade gracefully: lose one of eight replicas and you lose an eighth of throughput. When the parallelism strategy is also the redundancy strategy, DP's simplicity is an operations feature, not just a math result.

Four stages, four microbatches. Blue blocks are compute; red hatching is the bubble — 3 idle slots out of 7 per stage, 43% of stage-time doing nothing, and only slot t3 has all four stages busy. The takeaway is that PP's efficiency is entirely a function of m, the microbatches in flight: decode (m ≈ 1) is its worst phase and chunked prefill its best.

03 Expert parallelism: split the experts

the all-to-all tax

Expert parallelism exists for exactly one architecture: mixture-of-experts. Each MoE layer routes every token to its top-k experts, and the experts live on different GPUs. Moving a token's hidden state to its experts and the results back takes two all-to-alls per MoE layer — dispatch and combine. No rank ever holds the whole layer; each holds a few experts' worth of weights.

The volume will look familiar: 2 all-to-alls × top_k × hidden × 2 bytes per token per MoE layer. With top-2 routing and h = 8192 that is 64 KB per layer; across 80 MoE layers, about 5 MB per token — the same bytes as TP. The difference is pattern, not volume. An all-to-all is hundreds of small point-to-point messages with incast at the receivers and none of the ring's bandwidth doubling. In practice, budget two to three times the ideal bytes-over-bandwidth time for small-message all-to-all on 8 GPUs.

The uglier tax is imbalance. Uniform routing is a fiction; real routers develop hot experts, and the step waits for the slowest GPU — a straggler tax TP never pays. Training copes with capacity factors (GShard: each expert accepts (tokens/experts) × capacity-factor tokens and drops the overflow), but inference cannot drop tokens, so production systems over-provision experts or rebalance continuously. DeepSeek-V3 attacked all three problems at once: auxiliary-loss-free load balancing, which replaces the load-balancing auxiliary loss with per-expert bias terms so the main objective stays clean; routing that caps each token's experts to a handful of nodes; and a dual-micro-batch schedule that overlaps the all-to-alls with attention and MLP compute. Their operating point — 671B total parameters, 37B active per token, 256 routed experts with top-8 routing plus a shared expert — works because the communication is hidden, not because it is small.

EP is never deployed alone. Pure EP would replicate the attention weights on every rank — 23 GB times 8 of waste on our 70B-class model. Production MoE serving pairs EP for the experts with TP or DP for attention, exactly as DeepSeek's own deployment does. The rule of thumb is blunt: EP if and only if MoE. Never pay the all-to-all tax for a dense model.

The DeepSeek-V3 scale makes the tax concrete. With h = 7,168 and top-8 routing, dispatch moves 8 × 7,168 × 2 B ≈ 112 KB per token per MoE layer; dispatch plus combine is ~224 KB, and across roughly 58 MoE layers (all but the first few) that is ~13 MB per token per step — more than double our 70B-class example, on a model that activates only 37B of its 671B parameters per token. The bytes scale with routing width, not with total parameters.

EP degree is its own knob: EP=8 puts one expert per GPU and every token’s dispatch touches all 8 ranks; a higher degree shards experts finer but grows the all-to-all peer count, and all-to-all latency scales with peers, not bytes. DeepSeek runs EP across dozens of nodes for 256 experts — which only works because device-limited routing keeps each token’s fan-out small. Push EP degree past the routing fan-out and you pay for peers you never use.

The compute side has its own mechanism: experts run as grouped GEMMs — CUTLASS grouped kernels in the Megablocks style — batching every expert's matmul for a layer into one kernel launch with per-expert token counts. Throughput depends on padding those groups to Tensor Core-friendly tiles, which is another reason skewed routing hurts: a hot expert is not just a network straggler, it is a giant grouped-GEMM tile while other SMs idle. In the adversarial case every token routes to one expert and EP degenerates to single-GPU compute paying 8 GPUs' worth of all-to-all overhead.

That is why DeepSeek's dual-micro-batch schedule matters more than their link speed: split the batch in two and run one's all-to-all while the other computes attention and MLP. The 13 MB does not shrink; it disappears behind math. Any EP deployment that cannot overlap dispatch and combine pays the full sticker price, and on most fabrics that price is two to three times the bytes-over-bandwidth ideal. Note the scaling asymmetry, too: EP's per-token bytes are batch-invariant, but its efficiency improves with batch, because bigger messages amortize the same fixed costs TP suffers from.

04 Data parallelism: replicate everything

the memory tax

Data parallelism is the null strategy: every GPU holds the full model and the batch is sharded across replicas. Communication per token: zero bytes. No all-reduce, no bubble, no all-to-all. The tax is paid once, upfront, in HBM — N times the model size — and for a 70B model in fp16 that bill is 8 × 140 GB, i.e. 140 GB per GPU against 80 GB of HBM3. DP8 for 70B in fp16 is not a strategy; it is infeasible. In FP8 the weights shrink to 70 GB and it technically fits, with about 10 GB left for KV cache — which caps the batch long before anything interesting happens.

DP's kingdom is small models. Llama-3.1-8B in fp16 is 16 GB: eight replicas across 8×H100, more than 60 GB of HBM free per GPU for KV cache, zero inter-GPU traffic, throughput scaling linearly with replica count. When the model fits comfortably, DP is the highest-throughput, simplest-to-operate strategy available — no NCCL tuning, no bubble math, no stragglers.

In production, DP is the outer loop and TP the inner: TP=8 inside the NVLink domain, DP=N across nodes or replicas — the standard multi-node vLLM layout. Note the asymmetry with training, where DP must all-reduce gradients every step (the tax ZeRO was invented to tame). Inference DP synchronizes nothing, because there are no gradients. The only shared state is the request queue.

The FP8 DP8 case deserves its number, because it shows how thin the margin is: 70 GB of weights leaves ~10 GB for KV cache, and at 320 KB per token that caps the context-batch product hard — about 10 GB / (320 KB × 2,048 tokens of context) ≈ 15 sequences maximum. The replica count is not the binding constraint; the KV cache is. DP scales throughput only until the per-replica batch cap bites.

DP also composes with the two busiest patterns in serving. Each replica runs its own continuous-batching scheduler in the Orca style, so the fleet scales by adding replicas with zero cross-replica coordination. And multi-tenant LoRA serving — Punica, S-LoRA — packs dozens of adapters onto each replica, batching adapter matmuls across requests: DP replicas for scale, adapter batching for tenancy. The outer loop stays dumb on purpose.

There is one more DP trick worth naming: prefix-aware routing. SGLang's cache-aware scheduler routes requests sharing a prompt prefix to the same replica so the KV cache is reused instead of recomputed. With DP, the fleet stops being just a throughput multiplier and becomes a cache-sharding problem — the replicas are not interchangeable, and the router is part of the parallelism strategy.

05 The worked comparison

70B on 8×H100, one decode step

Assumptions, stated so you can redo the math: dense 70B with h = 8192, 80 layers, fp16; the EP column uses an MoE with the same hidden size, 80 MoE layers, 8 experts, top-2 routing; 8×H100 80GB with 900 GB/s NVLink, a 64 GB/s PCIe-class link, 3.35 TB/s HBM3; NCCL launch ≈ 10 µs per collective (implementation-dependent — vLLM's custom kernels do better). Compute baseline: each GPU streams 17.5 GB of weights per step, 5.2 ms, memory-bound — the step stays memory-bound until batch ≈ 300, where the arithmetic finally catches the streaming. KV cache rides along sharded with the heads and is omitted; weight streaming dominates at these batch sizes.

* Ideal; small-message all-to-all runs 2–3× worse in practice.

* Plus straggler risk on skewed routing.

Three things pop out. First, on NVLink the bandwidth tax is negligible for everyone — microseconds against milliseconds of compute. The taxes that matter are second-order: launch latency for TP and EP, bubbles for PP, HBM capacity for DP. Second, PP's 48 KB looks virtuous until the bubble multiplies latency by four at batch 1. Third, DP's zero is real but unaffordable: the cheapest communication is the kind you cannot provision memory for.

One more scenario the decode tables hide: prefill. A 4k-token prompt on TP8 moves ~20 GB of all-reduce traffic per forward pass — 22 ms on NVLink against ~72 ms of compute, a 30% tax that NCCL can only partly overlap with the next layer's GEMMs. PP's prefill story is the mirror image: 48 KB × 4,096 ≈ 197 MB of point-to-point traffic and a bubble thinned by dozens of microbatches. This is the quantitative reason phase-disaggregated serving exists: the parallelism that wins decode is not the parallelism that wins prefill, and forcing one choice for both phases overpays on one of them.

Sensitivity check: put TP8 on the 64 GB/s link — the cross-node case. Bandwidth time rises to 80 µs per token, still small; but cross-node collective launch is measured in tens of microseconds, so 160 × ~30 µs ≈ 4.8 ms of launch against 5.2 ms of compute, and per-token latency roughly doubles to ~10 ms before a single byte is blamed. The cliff is real, and it is made of launches, not bytes.

Per-token step time split into compute (green) and communication (orange) at batch 1 (left, ms) and batch 64 (right, µs). On NVLink the bytes are invisible — the orange is NCCL launch overhead for TP/EP and bubble idle for PP. DP8's bar is the memory tax made visible: 41.8 ms of weight streaming per token, and the ✗ marks 140 GB of weights against 80 GB of HBM. The takeaway: single-node decode belongs to TP; PP's bar only shrinks with microbatches it does not have at batch 1.

Four questions, in order. A latency SLO under ~10 ms/token forces TP inside one NVLink domain — nothing else has zero bubble. An MoE model forces EP for the experts, with TP or DP for attention. Multi-node or prefill-heavy work goes to PP. If the weights fit on one GPU, DP replicas are the simplest throughput. Everything else — which is most production serving — lands on the hybrid leaf: TP=8 intra-node, DP across nodes. The takeaway: choose from the SLO downward; the hardware answers follow.

06 Open questions

what the math doesn't settle yet

NVLink domains keep growing — 72 GPUs in a single domain is now a shipping configuration. When the whole model fits inside one domain, the cross-node cliff disappears and the TP-vs-PP decision moves: the question stops being "can the network survive the collectives" and becomes "how many launches can the step afford." Nobody has published a serious TP-72 decode analysis yet; the launch tax says it will not be free.

Co-packaged optics may move the cliff instead of removing it. If inter-node bandwidth ever approaches NVLink, PP's founding reason — avoiding cross-node collectives — weakens, and the decision becomes purely bubbles versus launches. Until then, every scale-out design is a bet on where the optical transition lands.

All-to-all is the collective least likely to survive contact with scale. EP's tax grows with expert count and routing skew, not with link speed, which is why the most interesting MoE systems work is in routing and overlap — DeepSeek's dual micro-batch, node-limited routing — rather than in fatter pipes. Watch whether next-generation fabrics add anything for many-to-many patterns; so far the answer is mostly no.

Finally, phase disaggregation — Splitwise, DistServe, Mooncake — splits the problem the tables above assume is whole. Prefill pools run PP over chunked prompts; decode pools run TP over thin batches; and a fifth column appears in the tax table: migrating KV cache between pools. Mooncake's KVCache-centric design treats that migration as the first-class cost. The parallelism math is not getting simpler. It is getting per-phase.

And the interconnect vendors are not standing still. NVLink Switch fabrics keep stretching the coherent domain, and the UALink consortium is pushing an open standard so accelerators from different vendors can share one fast fabric. A bigger domain widens TP's home turf and shrinks PP's cross-node niche. The napkin math survives all of it; only the constants change.

One sensitivity analysis is worth doing on the way out: double HBM bandwidth, as HBM4 promises, and the 5.2 ms weight stream becomes ~2.6 ms while the 1.6 ms launch tax does not move — TP’s comm fraction climbs from ~24% toward 40% of the step. Faster memory makes the network relatively more expensive, not less. Every table in this essay should be recomputed the year your memory bandwidth doubles.