All essays
RSS
Systems Architecture Note
FIG. 01–03 / MEMORY & INFERENCE

The Memory Mirage
Why “HBM Isn’t Needed” Isn’t What It Sounds Like

Read literally, the claim sounds absurd — fast memory obviously still exists. Read as an architecture thesis, it says something narrower and more interesting: future inference systems may lean on a small hot tier plus software-managed flash streaming instead of one giant pool of premium memory.

Core Translation
Premium memory stops being a durable requirement — it becomes an optimization choice, made per workload.
Practical Read
Training still wants HBM. The real skepticism is aimed at the long-run inference stack, not the lab.

01Start With the Simplest Interpretation

When someone says “you don’t need DRAM anymore” and then pushes further to “HBM isn’t needed,” the underlying model isn’t deletion — it’s relocation. The claim is about which memory tier gets to matter, not whether fast memory disappears from the system entirely.

The Hidden Sentence

“The future inference box may keep only a small, hot working set in very fast memory — and push almost everything else into flash, trusting software to stream it fast enough that nobody notices.”

Quantized weights KV-cache compression Flash / SSD streaming Smaller active working set Scheduling & prefetch Custom runtimes
Bottom line — this is mostly an argument about inference economics. It's much harder to make the same case for frontier training, where bandwidth and power efficiency still reward HBM heavily.

Training-Centric World

  • Large parameter updates keep the memory fabric under constant pressure.
  • Bandwidth matters enough that premium memory stays structurally valuable.
  • The system optimizes for maximum throughput, not minimum serving cost.

Inference-Centric World

  • Useful models can be quantized, sharded, prefetched, and staged in smaller chunks.
  • Cheap latency-per-token can matter more than peak absolute throughput.
  • Flash and software absorb more of the capacity burden than memory-heavy intuitions assume.
FIG. 01 — Stack ComparisonSCALE N.T.S.
A — PREMIUM MEMORY TRAINING STACK GPU / Accelerator HBM Near Compute GOAL: max bandwidth, dense training throughput B — STORAGE-TIERED INFERENCE STACK Cheap Accelerator / ASIC Small Fast Tier Large Flash / SSD Tier GOAL: low-cost inference via streaming + compression vs different bottlenecks different economics NOTE — stack (A) optimizes for throughput density; stack (B) optimizes for cost-per-served-token. Both stacks can coexist inside the same company, even the same model family.

02Why Kernel Drivers Even Came Up

Drivers aren't the thesis. They're the plumbing that makes the thesis credible. If the architecture depends on data moving efficiently from storage into a much smaller hot tier, low-level software matters because it's what hides latency and smooths over residency changes.

Runtime Tiering

The serving stack decides what stays resident, what gets prefetched, and what streams later without wrecking latency.

Driver / I/O Path

Faster NVMe access, pinned memory, direct paths, and better paging make flash-backed inference less painful.

Model Compression

Quantization, pruning, speculative decoding, and cache compression shrink how much premium memory must be active at once.

FIG. 02 — Data Flow PathSCALE N.T.S.
Flash / SSD weights · context · cold state Software / Driver Layer prefetch · paging · residency · compression · scheduling the plumbing that tries to hide a leaner memory stack Active Fast Tier small hot working set Every arrow here is a latency budget the driver stack has to protect.

03The Tier Nobody Mentions by Name

Most of the debate collapses two ideas into one: cost per gigabyte, and distance from compute. Laying out the full hierarchy makes the trade-off explicit — and shows why "small hot tier + large flash tier" is a coherent middle path, not a contradiction.

FIG. 03 — Memory Hierarchy, Cost vs. ProximitySCALE N.T.S.
← CLOSER TO COMPUTE / HIGHER COST PER GB FARTHER FROM COMPUTE / LOWER COST PER GB → Registers & Cache KB–MB scale near-instant HBM tens of GB extreme bandwidth DRAM hundreds of GB flexible workhorse Flash / SSD TB scale cheap, slow, streamable the thesis shrinks this whole span into "small fast tier + large flash tier" stays untouched HBM doesn't vanish — it gets a narrower, more deliberate job.

04Architectural Implications

If premium memory is a temporary bottleneck rather than a lasting requirement, some observers may be over-extrapolating how central that memory tier stays as AI systems mature.

If This Thesis Is Right

  • HBM demand may be narrower than today's broad narrative assumes.
  • AI may not increase every memory tier equally, or permanently.
  • Inference optimization may favor NAND, storage controllers, or custom silicon over premium memory volume.
  • Value could shift from raw bandwidth toward orchestration, compression, and storage-aware serving.

Main Technical Rebuttal

  • Frontier training still heavily rewards high-bandwidth memory and power-efficient data movement.
  • High-throughput inference often still benefits from keeping larger working sets close to compute.
  • Software tricks reduce memory pressure — they don't repeal physics.
  • The real question isn't whether flash helps; it's whether flash can replace enough premium memory to change the architecture meaningfully.
// NOTE This explainer is intentionally interpretive rather than documentary. It translates a likely architecture claim into plain English and keeps the framing generic rather than company-specific.