Dense GPU racks do not fail suddenly. They degrade. Thermal energy accumulates across requests, across shifts, across weeks — in the substrate, in the power delivery, in the packaging. Current observability stacks measure point-in-time temperatures and call the cluster healthy. This essay maps the degradation loop, its failure modes, and what a thermally-aware control plane would need to know.
When a GPU cluster exhibits a reliability problem, the first thing operators do is look at temperature dashboards. They check GPU die temperature, inlet air temperature, coolant temperature at the manifold, and power draw. If everything looks within spec at that moment, the cluster gets a green status and the problem is filed as intermittent.
This is the wrong way to reason about thermal state. A GPU can be at 65°C right now while having been at 85°C for the previous six hours, during which its HBM3 stack was operating at the edge of its thermal envelope, its package solder joints were cycling through repeated expansion and contraction, and its power delivery components were running at sustained high current density. The dashboard reports healthy. The device is not healthy.
The distinction between current temperature and accumulated thermal history is not academic. It determines how a device will behave for the next six hours. It determines how many thermal cycles remain before a solder bump in the CoWoS substrate begins to develop a crack. It determines whether the NVLink connectors in the tray are slowly oxidizing at their mating surfaces. None of that is visible in a point-in-time temperature reading.
Thermal state is not a scalar. It is a history. And current observability stacks throw away the history.
This essay is about why that matters for inference infrastructure specifically, what the accumulation mechanisms actually are in a dense GPU rack, and what a control plane that understood thermal debt would need to do differently.
Thermal debt is the accumulated gap between the heat generated by a system over time and the heat that has been dissipated. In a perfectly steady-state system, generation equals dissipation and debt is zero. In practice, systems never operate in perfect steady state — power draw spikes during prefill, cooling capacity varies with ambient conditions and coolant flow, and the thermal mass of the components introduces lag between heat generation and observable temperature change.
Every moment in which heat generation exceeds dissipation adds to the thermal debt. Every moment in which dissipation exceeds generation draws it down. The net integral over time is what determines the actual thermal stress on the components.
thermal_debt(t) = ∫₀ᵗ [P_gen(τ) - P_dissipated(τ)] dτ
where P_gen is instantaneous power draw
P_dissipated is cooling capacity at that moment
positive debt = accumulated thermal stress
The thermal mass of a component is what governs how quickly debt translates into temperature elevation. A thin die heats up quickly; a thick copper heatspreader with a large thermal mass buffers the temperature against short-duration spikes. But buffering does not eliminate the debt — it defers the temperature elevation. When the thermal mass saturates, the temperature catches up to the accumulated debt, and the device enters a high-temperature regime that may persist long after the power spike has ended.
In a dense AI GPU rack, this plays out at multiple timescales simultaneously. At the millisecond scale, prefill bursts create instantaneous thermal spikes. At the second-to-minute scale, consecutive long-context inference requests can prevent the die from returning to baseline between jobs. At the hour-to-shift scale, ambient temperature rise in the rack aisle degrades cooling efficiency and raises the baseline for everything. At the week-to-month scale, repeated thermal cycling causes physical degradation of packaging structures.
The accumulation mechanisms in a modern GPU rack are layered and interact in ways that are not obvious from examining any single component in isolation.
The most immediate accumulation source is the power asymmetry between prefill and decode in transformer inference. Prefill is compute-bound — the GPU runs at near-peak utilization, driving power draw to TDP or beyond. Decode is memory-bound — the GPU cycles through long sequences of high-bandwidth memory reads while running at far lower arithmetic utilization and lower power.
This means every request that starts with a long prefill phase creates a short, intense thermal spike followed by a long, moderate decode phase. The spike is short enough that the die temperature does not fully normalize before the next request arrives. Across a batch of requests, the baseline temperature climbs. In a serving system running continuous batching with back-to-back prefill injections, the thermal floor for the die never returns to idle levels during the shift.
At the rack level, thermal debt accumulates heterogeneously. The GPU trays at the top of a rack receive inlet air that has already been heated by the trays below it. The GPUs in those top positions run consistently hotter than their physically identical counterparts at the bottom of the rack, for no reason other than position. Over time, those top-position GPUs accumulate more thermal cycles, degrade faster, and fail at higher rates — but the fleet management system sees them as identical hardware with identical health profiles because it does not track thermal history per position.
In liquid-cooled installations, coolant temperature at the supply manifold is typically regulated to a setpoint, but in practice the actual coolant temperature at the cold plate on any given GPU depends on the length of the coolant path, the flow resistance of the loop, and the heat being deposited by all the upstream components. A coolant loop serving 16 GPUs in a tray will deliver noticeably warmer coolant to the last GPU on the loop than to the first. That last GPU will run hotter under identical workload, accumulate thermal debt faster, and experience more packaging stress.
The thermal debt in a GPU rack is not confined to the GPU die itself. The voltage regulators (VRMs) that step down rack power to the voltages required by the GPU operate at high current density and generate significant heat. Under heavy AI workloads, VRM junction temperatures can approach their thermal limits even when the GPU itself is within spec. VRM thermal degradation causes voltage regulation quality to degrade over time — which introduces noise into the GPU's power rails, which can cause subtle timing errors and reliability issues that are extremely difficult to attribute to a thermal root cause after the fact.
The most serious thermal debt accumulation in a modern AI GPU is in the HBM3 stack and the CoWoS interposer that connects it to the GPU die.
HBM memory stacks are assembled by bonding multiple DRAM dies on top of each other using through-silicon vias and micro-bumps. The resulting 3D stack has very limited thermal conductivity in the vertical direction. Heat generated by the bottom DRAM die (which is closest to the interposer) must travel upward through multiple dies before it can be conducted to the cooling solution, or horizontally out through the stack edges. This means the bottom dies in the HBM stack run significantly hotter than temperature sensors typically measure — those sensors are usually located on the top of the stack or on the interposer, not inside the stack where the highest temperatures occur.
The temperature you measure on an HBM stack is not the temperature where the damage is happening. The hottest point is inside the stack, beyond the reach of surface sensors.
The CoWoS interposer connecting the HBM stacks to the GPU die has a different coefficient of thermal expansion from the silicon stacks above it and the organic substrate below it. Every thermal cycle — every time the system heats up and cools down — the interposer and its neighbors expand and contract by slightly different amounts. Over thousands of cycles, this differential expansion drives shear stress into the micro-bump connections between the HBM stack and the interposer. Those micro-bumps are only 40-50 microns in diameter. The accumulated shear stress eventually causes them to crack.
The failure mode when micro-bump connections begin to degrade is not an immediate hard failure. It is a progressive increase in resistance at those connection points, which causes elevated heating at the affected bumps, which accelerates the degradation further — a positive feedback loop that runs below the detection threshold of any current monitoring system until the connection fails entirely.
| Component | Primary thermal stress mechanism | Observable before failure? | Current monitoring coverage |
|---|---|---|---|
| GPU die (GPC/SM) | Junction temperature, hotspot cycling | Yes — die temp sensors | Good (NVML, DCGM) |
| HBM stack (internal dies) | Internal vertical thermal gradient, stack hotspots | Partially — top-of-stack only | Weak (surface sensors miss internal temps) |
| CoWoS micro-bumps | Differential CTE shear, accumulated cycle count | No — resistance increase is sub-threshold until failure | None in production |
| VRM components | Junction temperature, electromigration under sustained current | Partially — output ripple changes | Weak (rarely monitored) |
| NVLink connectors | Oxidation at mating surfaces from thermal cycling | Partially — error rate increases before failure | Moderate (link error counters) |
| PCB traces (power delivery) | Electrothermal fatigue under high sustained current | No | None |
What makes thermal debt particularly dangerous in AI serving infrastructure is that the accumulation mechanisms are not independent — they form a closed feedback loop where degradation from one component creates additional thermal stress on adjacent components.
The most pernicious element of the loop is the feedback from throttling back into thermal accumulation. When a GPU throttles under thermal pressure, it reduces clock frequency to lower power draw and reduce heat generation. This works at the millisecond scale. But at the request scale, throttling means each inference step takes longer. A 128K context decode step that takes 100ms at nominal clock takes 135ms under throttle. That means the GPU spends 35% more time per token in a high-thermal state, because the job duration has increased. The time-at-temperature integral goes up even though the instantaneous temperature goes down. Thermal debt accumulates faster under throttle than it would have if the same request had been processed at full speed and released the GPU sooner.
Thermal debt in the GPU die creates a downstream failure mode in the NVLink fabric that is rarely analyzed as a thermal consequence.
NVLink connectors operate at extremely high data rates — NVLink 4.0 runs at 100 Gb/s per lane, and modern GPU systems have multiple links active simultaneously. At these frequencies, the signal integrity of each link is sensitive to the quality of the mating surfaces at the connector. Thermal cycling causes the connector housing to expand and contract, and if the thermal cycles are large enough or frequent enough, the mating force at the contact points can decrease over time as the mechanical spring elements in the connector relax.
Reduced mating force means increased contact resistance, which means signal degradation at the affected pins. The NVLink protocol handles this gracefully at first — it detects CRC errors and retransmits. But retransmissions increase the effective latency of fabric transfers, which adds to the tail latency of any operation that requires cross-GPU communication. In a multi-GPU inference setup, that means all-reduce operations during distributed inference, KV cache transfers in disaggregated serving, and checkpoint operations all start taking longer — not by a lot, but by an amount that is correlated with thermal history in a way that current attribution tools will not detect.
The symptom is a rack that was meeting latency SLAs in January and is missing them in July, without any identifiable hardware failure having occurred. The thermal debt accumulated over six months of production serving has been silently eroding the fabric's effective bandwidth, and the observability stack shows everything as healthy because no link has failed — they are just slow.
The gap between what AI cluster observability stacks currently measure and what would be needed to track thermal debt is wide. It is not a gap in instrumentation resolution — modern GPU drivers and management tools like NVIDIA DCGM expose a rich set of metrics. It is a gap in the questions being asked of those metrics.
Current monitoring approaches treat temperature as a threshold signal: is the current temperature above the alarm threshold? If yes, take action. If no, mark healthy. This is appropriate for detecting acute thermal events — a coolant leak, a fan failure, a workload that is dramatically exceeding TDP. It is completely inappropriate for detecting accumulated thermal stress, because accumulated stress is a function of the integral of temperature over time, not the current value.
A second gap is in physical location granularity. When a DCGM alert fires for GPU temperature, the alert refers to the temperature at a specific sensor location — usually the die hotspot sensor or the memory temperature sensor. It does not refer to the temperature inside the HBM stack, or the temperature at the CoWoS interposer, or the temperature at the junction between the VRM and its heatsink. Those temperatures can differ from the monitored sensor by 10-20°C under load, and they are the temperatures that matter for degradation mechanisms.
The third gap is temporal: observability systems that aggregate metrics at one-minute granularity will miss sub-minute thermal spikes entirely. A prefill burst that drives the die to 90°C for 800ms before the thermal mass absorbs the spike will not appear in a 60-second averaged metric. But 800ms at 90°C is a real thermal event that contributes to thermal debt and cycling stress, even if it never appears in any dashboard.
Building a control plane that can manage thermal debt requires a different kind of telemetry pipeline than current systems provide. The required capabilities are:
For each GPU in the cluster, the control plane should maintain a running integral of time-above-threshold for the key monitored parameters — die hotspot, HBM temperature, VRM junction temperature — with a sliding window of at least 24 hours. This integral is the closest available proxy for thermal debt accumulation and can be computed from existing telemetry sources with a simple aggregation step.
The specific threshold matters less than the consistency of its application. Pick a value — say, 75°C for die temperature — and track the cumulative minutes-above-threshold per device per day. A device that has been above 75°C for 400 minutes today has accumulated twice the thermal debt of a device that has been above 75°C for 200 minutes, and the scheduler should treat them differently.
Packaging fatigue is driven by cycle count, not absolute temperature. A device that has experienced 10,000 thermal cycles between 40°C and 80°C is in a different state than a device that has experienced 1,000 such cycles, even if both devices are at 65°C right now. The control plane should track thermal cycle counts — each time the device crosses the threshold on the way up is one cycle — and use that count as a degradation proxy in scheduling decisions.
Detecting the prefill burst thermal events that contribute to thermal debt requires telemetry at sub-minute resolution. Five-second sampling is a reasonable starting point — it will catch the envelope of most prefill spikes without overwhelming the telemetry infrastructure with per-second data for a large fleet.
The control plane should maintain a thermal profile not just per device but per rack position. A device that is persistently hotter than its nominal thermal peers at the same utilization level is either in a disadvantaged rack position, in a degraded cooling zone, or beginning to develop a thermal resistance problem at the die/heatspreader interface. That signal should influence placement decisions for high-priority workloads.
Once a control plane has thermal debt scores per device, a range of scheduling interventions become possible that are not in the design space of current serving systems.
The most immediate intervention is thermal-debt-aware placement. When a new high-priority request arrives, route it to the device with the lowest current thermal debt score, not simply the least-loaded device by utilization. A device at 60% utilization with a high debt score may deliver worse latency over the next hour than a device at 70% utilization with a low debt score, because the high-debt device is closer to a throttle event.
A second intervention is preemptive cooldown scheduling. If a device's thermal debt score is high but no throttle event has occurred yet, the scheduler can choose to stop routing new requests to that device for a short window — say, five minutes — allowing it to cool down and pay down thermal debt before the throttle event occurs and forces a longer recovery. Five minutes of planned light load is cheaper than the latency impact of an unplanned throttle event during a high-value request.
The third intervention is maintenance signal generation ahead of hard failure. If a device's thermal cycle count is approaching a modeled fatigue limit for its packaging generation, the control plane can flag it for preventive maintenance before it fails in production. This requires building thermal cycle budget models for the specific packaging technology in use — those models exist in the reliability literature for CoWoS and similar interposer technologies — and integrating them into the fleet management system.
AI infrastructure discussions tend to focus on the things that can be benchmarked and graphed in real time: tokens per second, latency percentiles, GPU utilization, memory bandwidth. Thermal debt is none of those things. It accumulates silently, degrades performance gradually, and fails hardware in ways that are difficult to attribute after the fact. That is precisely why it tends to stay invisible until the cluster is already in trouble.
The observability gap here is not a hard problem. The sensor data exists. The telemetry infrastructure exists. What is missing is the conceptual layer that frames temperature as a history rather than a current value, and that treats accumulated thermal cycles as a resource that gets consumed over a device's lifetime rather than an event that either happens or does not.
Dense AI racks are operating at power densities that would have been considered aggressive in a high-performance computing context five years ago. The packaging technologies that make that density possible — CoWoS interposers, HBM3 stacks, high-current VRMs — all have known thermal cycle fatigue limits that are being tested by continuous heavy inference workloads. The clusters that manage this well will not be the ones that buy the most cooling infrastructure. They will be the ones whose schedulers understand that every request they run today is spending a fraction of the cluster's thermal budget — and that budget needs to be tracked, managed, and replenished deliberately, not discovered empty after the first wave of unexpected hardware failures.
Manish KL writes about AI infrastructure, memory systems, and accelerator architecture. Related essays: AI Cluster Reliability Beyond Fault-Tolerant Parallelism · The Next AI Cluster Failure Won't Look Like a GPU Failure · How to Measure GPU Underutilization on H100 and H200
© 2026 Manish KL