Cold Starts:
The First-Token Problem

Benchmarks measure warm servers. Production measures cold ones: a new replica needs the weights in HBM before it can emit token one, and 140 GB over PCIe 5.0 x16 takes 2.2 seconds minimum. Everything after that is optimization — and most serving stacks leave a factor of 30 on the table.

01 The cold-start waterfall

Account for every millisecond. The download is the long pole; everything else is a tax stack.

Define the term precisely, because vendors won't. A cold start runs from the moment the scheduler places a replica to the moment the first token leaves it. Not "model load time" — the whole waterfall. For a 70B FP16 model on 8× H100 with weights sitting in object storage and nothing cached, it looks like this:

Container and image start: 2–5 s. Assuming the image is already on the host. If it isn't, add the pull — gigabytes of CUDA base image at disk speed — and your cold start just became a container-distribution problem. Pin your images on GPU hosts; this stage should be boring.

Runtime init: 5–15 s. Python startup, import torch (a couple of seconds on its own), CUDA context creation on each of 8 GPUs, then NCCL init across them. NCCL's first-time initialization — IB device discovery, topology detection, communicator setup — is routinely the single slowest thing in this stage, single to double-digit seconds. It happens exactly once per process, which is why nobody notices it in benchmarks and everybody pays it in production.

Weight download: 30–120 s. The long pole. 140 GB has to travel from object storage to local disk, and the number that matters is your sustained per-connection throughput times your parallelism. A single HTTPS stream might give you a few hundred MB/s; sharded checkpoints downloaded with parallel range GETs across 8 shards can push low single-digit GB/s. At 2 GB/s aggregate, 140 GB is 70 seconds. This stage alone is usually longer than everything else combined.

Deserialize: 2–8 s. Parsing the safetensors headers, mmap-ing the buffers, dtype and shape checks. Small, but only because safetensors was designed for it — more on that in section 02.

HBM upload: 2.2 s floor, 5–15 s typical. Here's the floor arithmetic the lede promised. PCIe 5.0 runs at 32 GT/s per lane with 128b/130b encoding:

32 GT/s × (128/130) ÷ 8 bits = 3.94 GB/s per lane → ×16 lanes = 63 GB/s usable

140 GB ÷ 63 GB/s = 2.2 s

That is the physical floor for moving 70B FP16 weights into one GPU's HBM, and it assumes zero contention, zero protocol overhead, and the bytes already sitting in host memory. In practice, 8 GPUs share PCIe switches and root-complex bandwidth, uploads contend with NCCL traffic if you're careless about ordering, and the typical number lands at 5–15 s. On PCIe 4.0 hosts the floor doubles to 4.4 s — know which generation your fleet is.

CUDA graph capture: 10–60 s. vLLM captures CUDA graphs at engine startup for every batch size in cudagraph_capture_sizes. Each capture replays the entire model forward pass under stream capture. For a 70B model this is tens of seconds of pure startup tax, and it is strictly serial with everything after weight upload.

Kernel autotune and compile: 0 to 300+ s. Triton kernels autotune their launch configs on first invocation; regions compiled with torch.compile run Inductor codegen and autotuning on first touch — single-digit minutes for a 70B-class model is a reasonable estimate. Both cache to disk (TRITON_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR), so this stage is either the longest one or zero, depending on whether your image bakes the caches in. Most teams discover which one the hard way.

KV cache pre-allocation: 1–3 s. vLLM grabs its configured fraction of HBM (default gpu_memory_utilization=0.9, minus weights and overhead) in one large allocation and initializes the paged block tables. Fast, but on the critical path.

First prefill and health gate: 2–5 s. The first real forward pass plus whatever your readiness probe needs to see before the load balancer sends traffic.

Add it up: 60–240 s typical, ~3 s floor. The floor is the weight-movement physics (2.2 s PCIe plus ~1 s of unavoidable init) with everything cached locally and pre-compiled. The gap between 3 s and 3 minutes is copies, downloads, and cold caches — not compute.

Two refinements before we optimize. First, the floor depends on where the bytes start. The ~3 s floor assumes weights resident in host DRAM (~100+ GB/s: 140 GB in ~1.4 s, plus 2.2 s PCIe). If they're on local NVMe Gen4 at 7 GB/s, add 140 ÷ 7 = 20 s of read — the floor becomes ~22 s before you've done anything wrong. If they're on NVMe Gen5 (~14 GB/s), ~10 s. "Weights on local disk" is not one tier; it's a 10–20 s spread, which is why the tiered stores in section 03 distinguish DRAM from SSD rather than lumping them as "local."

Second, the waterfall as drawn is fully serial, and it doesn't have to be. Download shard N+1 while uploading shard N to HBM (double-buffered across the NVMe→HBM path), and run runtime init concurrently with the download: the 12 s init bar disappears inside the 75 s download bar entirely. Pipelining doesn't change the arithmetic much when one stage dominates — max(download, upload) instead of their sum saves ~8 s here — but overlapping the independent stages (process start, NCCL init, tokenizer load) with the download is free latency. The waterfall's serial shape is a choice, not a law.

The waterfall makes the priorities obvious. The download bar is the only one measured in minutes; the PCIe floor is the dashed line nobody reaches because the bytes start three network hops away. Every optimization in this essay is either shortening the orange bar or moving the dashed line left.

02 Faster weight movement

Safetensors, mmap, and sharding. Every byte is moved three or four times; cold-start engineering is copy elimination.

Trace a single weight byte from object storage to HBM and count the copies: object store → local NVMe → page cache → userspace buffer → pinned host memory → HBM. Some stacks add a deserialization bounce on top. Each transit is a tax, and the format you store weights in decides how many you pay.

Safetensors won because it deletes copies. The format is almost insultingly simple: an 8-byte little-endian header length, a JSON header mapping tensor names to dtype, shape, and byte offsets, then the raw buffers concatenated. The loader mmaps the file, so the kernel's page cache is the buffer — no userspace copy, no parsing pass over the data. Tensors are just views at offsets. Compare with the old .bin pickle path: unpickling reconstructs every tensor through Python objects, executes arbitrary code as a side effect, and is GIL-bound and single-threaded. Loading 70B through pickle could take minutes and transiently needed ~2× the model size in RAM. Safetensors reduced "deserialize" from a stage to a rounding error, which is why every serious serving stack standardized on it. It won on safety (no arbitrary code execution) and kept the win on speed.

What the loader does with the mapped bytes matters too. vLLM's weight loading walks each shard's tensors and hands them to per-layer weight_loader functions, which slice each tensor to the TP rank's shard and issue the host→device copy — that copy is the PCIe transit from the floor arithmetic. Dtype casts and quantization-aware repacking happen in this pass where possible, fused into the copy rather than as a separate sweep. The failure mode to watch: any loader stage that materializes the full unsharded tensor on a single GPU before splitting (some naive device_map="auto" paths do this) briefly needs 140 GB on one device and serializes the one stage sharding was supposed to parallelize.

On the download side, parallelism should match the sharding. Eight shards downloaded over eight connections — HuggingFace Hub's snapshot_download fans out per file, S3 clients use multipart range GETs — gives you 8× the per-connection throughput without any cleverness. The natural pairing (shard count = TP width = connection count) is why 8-shard 70B checkpoints are the sweet spot: the parallelism you need for the network is the parallelism you already have for the GPUs.

Two host-level details decide whether you actually hit the PCIe floor. First, the host→device copy wants pinned (page-locked) memory: pageable host memory can't be a DMA source, so the driver either copies through an internal pinned bounce buffer or you pin it yourself. Staging 140 GB through pinned buffers in chunks with cudaMemcpyAsync keeps the DMA engines fed; one giant synchronous copy per tensor starves them between launches. Second, NUMA locality: on dual-socket hosts each GPU hangs off one NUMA node, and copying from the other node's memory crosses the inter-socket link — halving effective bandwidth and adding jitter. Pin the staging buffers on the NUMA node local to the target GPU (numactl, or the loader's NUMA-aware allocator). These are the details that separate "we measured 63 GB/s" from "we measured 28 GB/s and blamed PCIe."

Sharded checkpoints parallelize everything. A 70B model ships as 8 shards of ~17.5 GB (model-00001-of-00008.safetensors, …). Under tensor parallelism with 8 GPUs, rank i loads shard i — each GPU pulls its own bytes over its own PCIe link, and the aggregate upload bandwidth is 8 × 63 = 504 GB/s. The theoretical floor for the whole model drops to 140 ÷ 504 ≈ 0.28 s. In practice the limiter moves upstream to NVMe read bandwidth or host memory bandwidth, but the shape of the win is real: sharding turns one 63 GB/s pipe into eight. Downloads parallelize the same way — eight range-GET streams instead of one.

GPUDirect Storage removes the CPU bounce. The standard path stages 140 GB through host DRAM twice (disk → page cache → pinned buffer → HBM). GDS lets the NVMe drive DMA straight into GPU memory via cuFile, skipping host memory entirely. The honest caveat: it needs the nvidia-fs kernel module, a compatible filesystem and drive topology, and explicit integration — as of this writing it is rarely wired into serving stacks end to end. The win is real but the plumbing is the friction. Measure your actual copy chain before assuming GDS is the bottleneck; on many hosts the download dwarfs the bounce.

Quantization is a cold-start lever, not just an inference lever. Every stage that moves bytes gets faster when there are fewer bytes. INT4 70B is 35 GB: the PCIe floor drops to 35 ÷ 63 ≈ 0.56 s, the download shrinks 4×, deserialization shrinks 4×. FP8 is 70 GB for a 1.1 s floor. And because dequantization is fused into the matmul kernels (AWQ, GPTQ, Marlin, FP8 paths), there is no extra pass — the smaller bytes go straight into the compute you were already doing. If your cold-start SLO is aggressive, quantization buys more than any loader optimization.

The playbook, stated once: count the copies, then delete them. mmap instead of read+parse. Shards instead of one stream. DMA instead of bounce buffers. Fewer bytes instead of faster pipes.

03 Broadcast at fleet scale

RDMA weight distribution. One replica's cold start is a latency problem; a hundred replicas' is a thundering herd.

Scale the waterfall up and the physics changes. One replica pulling 140 GB from object storage is slow. A hundred replicas scaling up at once pull 14 TB through the same storage uplink — and now you're not waiting on your bytes, you're waiting on everyone else's. Object stores have enormous aggregate bandwidth but finite per-prefix request rates (S3 documents ~5,500 GET/s per prefix) and your VPC's NAT and gateway have opinions about 14 TB arriving at once. Tail latency explodes precisely when you need the capacity most: traffic spikes, deploy rollouts, region failovers.

The fix is to stop treating the object store as the fan-out point. In a star topology, the 140 GB crosses the storage uplink N times for N replicas. In a broadcast tree, it crosses once, and each node forwards to its children. This is the same binomial broadcast NCCL uses for collectives, applied to weights instead of gradients: with 200 Gb/s (25 GB/s) RDMA links, one hop moves 140 GB in 140 ÷ 25 ≈ 5.6 s, and a 3-level tree feeds 8 workers with chunked pipelining keeping every link busy — end-to-end on the order of ~10 s instead of eight serial downloads. The total bytes on the wire barely grow with N; the depth grows as log N.

Below the tree, add a second tier: local NVMe caches. The second replica scheduled on a host where a peer already downloaded the model reads from the peer's NVMe at 7–14 GB/s — 10–20 s, zero network. Host-local sharing turns the fleet's disks into a CDN edge cache. The scheduling constraint this creates — place new replicas where the bytes already are — is exactly what startup-time-aware schedulers optimize for.

The most complete articulation of this stack is ServerlessLLM (SoCC 2024). Its checkpoint store treats the GPU server's own storage hierarchy — DRAM, SSDs, remote — as a multi-tier cache for model checkpoints, in a chunked format designed for sequential reads rather than safetensors' mmap-friendly layout. The reported numbers: 3.6–8.2× faster checkpoint loading than the safetensors and PyTorch baselines on large models, 4.4× on LoRA adapters, and 10–200× better latency than KServe and Ray Serve on workloads modeled from the public Azure trace. The follow-on sllm-store makes the mechanism explicit: a custom binary format optimized for sequential reads, O_DIRECT I/O bypassing the page cache, a pinned host-memory pool for DMA-friendly GPU transfers, and parallel multi-threaded loading — claiming 6–10× over the safetensors loader. Two design choices matter more than the speedups: the scheduler does startup-time-optimized placement (schedule where checkpoints are warm), and it supports live migration of running inference to rebalance without reloading. The cold start is treated as a scheduling input, not a fixed cost.

The pipeline arithmetic is worth seeing once. Split 140 GB into 1,400 chunks of 100 MB. With chunked pipelining, the tree's end-to-end time is roughly one hop time (5.6 s at 25 GB/s) plus the time to drain the last chunks — the depth of the tree stops mattering and only the link bandwidth does. That's the "order of ~10 s" figure: not 3 hops × 5.6 s = 17 s serial, but ~5.6 s plus pipeline fill, because every link is busy simultaneously. The practical requirements are chunk-level checksums (a corrupt chunk shouldn't poison a subtree) and backpressure (a slow leaf shouldn't stall its siblings) — the same machinery BitTorrent worked out twenty years ago, now running over RDMA.

And the placement insight generalizes beyond any one system. "Checkpoint present on host" belongs next to "GPU free" and "zone" as a scheduling signal, and the cheapest implementation is a DaemonSet that pre-warms the NVMe cache on every GPU host when a new model version lands — before the autoscaler ever fires. Pre-warming converts the 75 s download bar into the 20 s NVMe-read bar for every replica that lands on a warmed host, which, on a large fleet, is nearly all of them.

The tree turns a thundering herd into a pipeline. Each link carries the model once, RDMA keeps the CPU out of the data path, and depth grows logarithmically while the star's uplink load grows linearly. The NVMe tier underneath does the same thing at host scope: bytes already on the machine beat bytes anywhere else, so the scheduler should treat "checkpoint present" as a first-class placement signal — which is precisely what ServerlessLLM's startup-time-optimized scheduler does.

04 Everything after the weights

Capture, compile, warm. The weights are in HBM and you're still not serving — here's the rest of the tax stack.

Teams that fix the download often stop too early. The weights land in HBM and the replica still can't take traffic, because a serving engine isn't a weight buffer — it's a compiled, captured, pre-allocated runtime. Each of these stages is "small." Together they're 30–90 seconds, and they're all serial.

CUDA graph capture: 10–60 s. Graphs exist to kill kernel-launch overhead during decode, where batches are small and launches dominate. vLLM captures them at engine startup for every batch size in cudagraph_capture_sizes: each capture replays the full model forward under CUDA stream capture, then the graph is replayed per step. For a 70B model that's tens of seconds of mandatory startup work — and it must finish before the first request, because a request arriving mid-capture either waits or takes the slow non-graph path. The operational move is replay-warm: after capture, run a real prefill through each captured graph before the replica reports healthy. An uncaptured graph is a latency landmine wired to your first user.

torch.compile: minutes, once. If your stack compiles model regions with Inductor, the first touch of each new shape runs codegen plus autotuning — single-digit minutes for a 70B-class model is a fair estimate, dominated by autotuning GEMM configs across dozens of shapes. This is a compiler running inside your cold start. The mitigations are all about never paying it twice: persist TORCHINDUCTOR_CACHE_DIR into the image, ship ahead-of-time (AOTI) artifacts, and keep the shape set small so the cache actually hits. If your cold starts are mysteriously 5 minutes on fresh hosts and 40 seconds on warm ones, look here first.

Triton autotune: minutes, once. Same shape, different layer: kernels like fused attention and MoE dispatch tune their tile configs and warp counts on first launch per shape. vLLM pins a number of these configs and ships tuned defaults, but any kernel or shape outside the pinned set autotunes live. Persist TRITON_CACHE_DIR in the image alongside the Inductor cache — the two caches are the difference between "cold" and "lukewarm."

KV cache pre-allocation: 1–3 s. vLLM allocates its HBM budget up front — gpu_memory_utilization (default 0.9) minus weights and overhead — as one large block pool, then initializes the paged-attention block tables over it. The budget arithmetic for our running example: 8× H100 80 GB = 640 GB HBM; 0.9 × 640 = 576 GB usable; minus 140 GB of weights and ~10 GB for CUDA graphs and workspace leaves ≈ 426 GB for KV. A 70B-class model with 80 layers and grouped-query attention (8 KV heads × 128 dim) stores 80 × 2 × 1024 × 2 bytes ≈ 320 KB per token — so the pool holds roughly 1.3M tokens of KV cache. It's fast, but note the ordering constraint: it can't run until weights are placed, because only then is the free budget known. Another serial stage.

Health-check gating: 2–5 s, and the one teams get wrong. A Kubernetes readiness probe that returns 200 when the HTTP server is up — but before graphs are captured and warmed — routes real traffic into a replica that isn't ready. The first user request then eats the capture cost, and your p99 has a cold-start-shaped spike that no dashboard attributes correctly. Gate readiness on a synthetic prefill: flip the probe only after a real forward pass through the warmed graphs succeeds. The 2–5 s this costs is the cheapest latency you'll ever buy.

Note the ordering constraints that make this stack serial. vLLM's engine init runs: tokenizer → model skeleton on meta device → weight loading → KV pre-allocation (budget known only after weights land) → graph capture (needs the full model resident) → warmup. Each stage's input is the previous stage's output, so they add. The one escape hatch is enforce_eager: disabling CUDA graphs deletes the entire capture stage from the cold start — at the price of paying kernel-launch overhead on every decode step forever. It's a real knob with a real tradeoff curve: for low-QPS services where TTFT matters more than per-token latency, eager mode can be the right call, and it makes the cold start dramatically shorter.

Operational rule: bake the caches into the image. Triton autotune cache, Inductor cache, and pre-captured graph metadata are all deterministic artifacts of (model, GPU type, software version). A host that has seen this exact combination before should never compile anything.

05 Architectures that dodge it

Warm pools and snapshots. When the waterfall can't be shortened enough, stop walking it.

Keep-warm pools: pay GPUs to idle. The blunt instrument, and the only one with a truly bounded p99. A warm replica holds weights in HBM, graphs captured, caches hot: spawn latency drops from 90 s to ~3 s (process attach + first prefill). The cost is arithmetic: 8× H100 at roughly $2.50/GPU-hour (estimate) is $20/hour, $480/day, about $14,600/month per warm replica. Whether that's expensive depends on what a 90-second p99 costs you — for interactive products it's usually cheap. The refined version is predictive scale-up: hold a small warm pool and grow it ahead of demand using queue depth, diurnal patterns, and deploy events. The knee of the curve is where the marginal idle dollar stops buying meaningful p99 improvement — that's where predictive scaling lives, not at zero and not at full warmth.

The curve is the whole argument for predictive scaling in one picture. Latency collapses over the first 16 idle GPUs and then flattens — past ~32, you're paying $1,825/month per GPU (at the estimated rate) for single-digit seconds. Size the standing pool for the knee and let prediction, not panic, cover the rest.

VM snapshots: fast at the wrong layer. Firecracker can snapshot and restore a microVM in on the order of a hundred milliseconds — but that's CPU and RAM. GPU state doesn't snapshot: CUDA contexts live in the driver, tied to the physical device, and there has historically been no CRIU equivalent for GPUs. A restored VM still faces the entire GPU waterfall.

The frontier: cuda-checkpoint. NVIDIA's utility (driver 570+, CUDA 12.8 era) finally gives CUDA processes a suspend/resume primitive that works with CRIU: suspending locks CUDA APIs, drains in-flight work, copies device memory back to host allocations, and releases the GPU; restore reverses it. This is the first credible path to snapshotting a warm GPU process. The catch is physics, not software: restoring means moving 140 GB back over PCIe — the 2.2 s floor doesn't disappear, it relocates from "load" to "restore." And multi-GPU state (NCCL communicators across 8 devices) is still hard. Frontier, not product — but it's the thing to watch.

Disaggregated loading. Borrow the Splitwise/DistServe instinct — split the system along its real bottleneck — and separate weight-loading from request-serving. A loader tier streams weights into fresh GPUs (broadcast tree, NVMe caches, the works) and hands warm devices to the serving pool. The serving scheduler never sees a cold GPU. It's more machinery, but it converts a latency problem into a capacity-planning problem, which is a strictly easier problem.

Two workload shapes make all of this non-optional. The first is spot and preemptible capacity: if your fleet turns over hourly, the 90-second cold start isn't a rare event, it's your steady-state tax on every replica-hour, and warm pools stop being an optimization and become the price of using cheap GPUs. Size the pool against preemption rate, not just traffic. The second is multi-tenant LoRA serving: adapters themselves cold-start in seconds (megabytes, not gigabytes), but they ride on a base model that still pays the full waterfall — which is why ServerlessLLM's 4.4× adapter-loading speedup only matters once the base model's cold start is already handled. The base is the problem; everything else is a rounding error on top of it.

06 Open questions

What to watch. The waterfall is understood; the primitives around it aren't finished.

Will GPU snapshot/restore standardize? cuda-checkpoint plus CRIU is the first time "hibernate a GPU process" has been more than a research demo. For it to matter operationally, clouds would need to offer GPU hibernate/resume as a primitive — snapshot the warm process to host memory or local NVMe, restore onto any compatible device. The open engineering is all in the restore path: 140 GB still has to cross PCIe, NCCL state across 8 GPUs has to be reconstituted, and driver-version skew between snapshot and restore hosts has to be tolerated. Watch whether the next driver generations make checkpointed state portable, not just pausable.

Peer-to-peer weight meshes as an orchestrator primitive. Today every team hand-rolls some version of "don't all pull from S3 at once" — host-affinity scheduling, pre-warming DaemonSets, custom downloaders. None of it is a first-class scheduler primitive. The BitTorrent-shaped answer — content-addressed weight chunks, peer exchange over RDMA, the scheduler placing replicas where chunks already live — wants to exist inside the orchestrator, next to the image-pull machinery that solved the same problem for containers a decade ago. Kubernetes solved image distribution with layering and caching; weight distribution is still DIY.

Cold-start SLOs as a first-class serving metric. Benchmarks report warm TTFT because warm is easy to measure. Production p99 is set by the cold path: scale-ups, deploys, failovers, spot preemptions. Serving benchmarks should spawn replicas, not just query them — report cold TTFT alongside warm, the way storage benchmarks report both cached and uncached reads. Until the number is measured, nobody optimizes it, and section 01's waterfall stays invisible.

Formats co-designed for loading. Safetensors won on safety and simplicity, not on load speed — its layout is mmap-friendly, which is good, but it isn't chunked, checksummed per chunk, or designed for partial fetch. ServerlessLLM's loading-optimized chunked format points at the gap: a format designed for "get these bytes onto 8 GPUs as fast as possible" would look different from a format designed for "don't execute arbitrary code." Expect the next format war to be fought on cold-start seconds.

Scale makes this structurally worse. A 1T-parameter FP16 model is 2 TB: the PCIe 5.0 floor alone is 2,000 ÷ 63 ≈ 32 s, before downloads, captures, or compiles. Every technique in section 05's table graduates from "optimization" to "requirement" — sharded loading, quantized weights, broadcast trees — because the floor itself exceeds most latency budgets. The teams that treat cold start as a distribution problem now will be the ones whose 1T-parameter serving still has a p99 worth quoting.

Weight distribution as a cloud primitive. Containers got this solved a decade ago: layered images, content-addressed blobs, registry caches in every region, and pull-through caching built into the orchestrator. Model weights have none of it — no standard chunking, no content addressing, no "model registry with RDMA pull" offered by any cloud. The first provider to ship weights-as-a-primitive (versioned, checksummed, broadcast-capable, placement-aware) removes an entire category of undifferentiated engineering from every serving team. Until then, section 03's tree is something you build yourself.