The Next Inference Bottleneck Is Not FLOPs.
It Is the Shape of the Decode Loop.

Once inference moves toward real-time token rates, the bottleneck stops being raw compute. What takes over is data movement, orchestration overhead, decode serialization, KV movement, and tiny idle gaps between GPU work. The GPU may be fast, but the system around it keeps creating bubbles.

SRAM is already inside the GPU — it exists in the form of fast on-chip staging structures: registers, shared memory, L1-level paths, and cache-local execution resources around the SMs. The goal isn't necessarily to add a new SRAM chip next to the GPU. The goal is to make the inference loop use existing on-chip SRAM better.

The issue isn't whether we have enough FLOPs. The issue is whether the hot inference loop is tiled, quantized, sparse, parallel, and resident close to compute.

Large models are not just compute-heavy — they are movement-heavy. Every layer has to move weight tiles through the memory hierarchy: from HBM, through cache, into the compute-side local staging path where tensor cores can consume them. FP4 and NVFP4-style quantization reduce the number of bits moved per weight and activation tile, directly reducing pressure on the memory hierarchy.

NVIDIA's Blackwell architecture introduced native FP4 support in its 5th-generation Tensor Cores. NVFP4 — NVIDIA's custom FP4 format — uses block-wise micro-scaling: values are grouped into blocks of 16, each sharing a high-precision FP8 (E4M3) scaling factor, plus an additional per-tensor FP32 scale. This two-level scheme preserves dynamic range and reduces quantization error while keeping the data compact.

NVIDIA TensorRT Model Optimizer and LLM Compressor both offer streamlined workflows for quantizing models to NVFP4 via post-training quantization (PTQ). As of early 2025, NVFP4-quantized versions of DeepSeek-R1, Llama 3.3 70B, and Llama 3.1 405B are available on Hugging Face.

Research into Quantization-Aware Distillation (QAD) for NVFP4 has also emerged to recover accuracy in smaller models where PTQ alone shows larger degradation. Approaches like RaZeR (Redundant Zero Remapping) and ARCQuant further push NVFP4 accuracy by optimizing the quantization grid itself. The format's fine-grained block-wise isolation prevents high-magnitude outliers from inflating scaling factors across entire tensors — a key advantage over coarse-grained integer quantization.

A trillion-parameter model doesn't have to activate all trillion parameters for every token. That's the whole power of MoE. The model can preserve large total capacity while routing each token through only a subset of experts — changing the problem from "run the whole trillion-parameter model per token" to "route each token through the right small subset of total capacity."

The concrete scale of this efficiency gain is striking: DeepSeek-V3 has 671 billion total parameters but activates only 37 billion during inference, achieving GPT-4 level performance at a fraction of the compute cost. Mixtral-8x7B has 46.7 billion total parameters but only 13 billion active during inference — matching LLaMA-2-70B while running six times faster.

Since early 2025, nearly all leading frontier models use MoE designs — DeepSeek-R1, Kimi K2, Mistral Large 3, Llama 4, and Qwen3 all rely on sparse activation. The architecture has moved decisively from research prototype to production backbone.

The memory constraint is real: full DeepSeek-R1 requires roughly 13,719 GB/s of memory bandwidth at scale — achievable only on data-center systems. At batch size 1, requirements drop to ~1,040 GB/s. This is why inference-optimal MoE scaling (balancing expert count against serving efficiency) is an active research area — models with fewer, larger experts can be more serving-efficient even if they require more training compute.

In classic full attention, the KV cache grows with context length: O(N) memory complexity means that as the context becomes longer, both the storage and movement costs of KV data keep increasing. Sliding-window attention changes this to O(C) — constant rather than growing — by restricting each token's attention to a fixed local window of size C.

This matters enormously during decode. The model might be quantized, the active expert set might be sparse, but if the KV path keeps growing without bound, the memory system still gets crushed at long context. SWA is the layer that says: do not let context length turn every decode step into an ever-growing memory problem.

Gemma 3's technical report shows that a 1:3 global-to-local ratio with a sliding window of 1024 tokens reduces KV cache memory overhead from 60% (global-only baseline) to under 15% at 32K context. Mistral-7B pioneered this approach with a 4096-token window, storing only a rolling buffer of recent KV pairs rather than the full history.

The dominant practical pattern in 2025 is hybrid attention: most layers use local/sliding-window attention for efficiency, while a smaller subset of layers use global attention to maintain long-range coherence. This lets models have large theoretical context windows while keeping actual KV memory manageable during decode.

Classic autoregressive decoding is painfully serial: generate token 1, then token 2, then token 3. Each step depends on the previous. This creates a narrow workload. Modern GPUs are enormous parallel machines — if the runtime only gives the GPU one thin token path at a time, compute sits underutilized even when the model is large.

Parallel token or block decoding makes decoding wider. Instead of producing exactly one next token per step, the runtime creates a larger unit of token work. Approaches include speculative decoding, multi-token prediction (MTP), blockwise decoding, parallel candidate branches, and diffusion-style token blocks.

Speculative Decoding

The core idea: a smaller, faster draft model proposes several future tokens cheaply and in parallel; a larger target model verifies them all in a single forward pass. If the draft tokens match what the target would have generated, you get multiple tokens for the latency cost of one. The output is mathematically identical to standard autoregressive decoding — it is a lossless speedup.

Speculative decoding has achieved 2–3× speedups in production settings. A 2025 paper at ICLR 2026 demonstrated up to 2.37× speedup even at batch size 256, challenging the conventional wisdom that speculative decoding only works in small-batch settings. The key finding: memory bandwidth, not compute, remains the dominant bottleneck even in large-batch inference.

Multi-Token Prediction (MTP) — Gemma 4's Approach

On May 5, 2026, Google released MTP drafter models for the entire Gemma 4 family: the 31B dense flagship, the 26B A4B Mixture-of-Experts model, and the on-device E2B/E4B edge models. Using speculative decoding powered by MTP, the drafters deliver up to 3× faster inference with zero degradation in output quality. The Gemma 4 MTP drafter is not independent — it shares the input embedding table with the target model and builds directly on its last-layer activations.

MoE models present an interesting nuance: because different tokens may activate different experts, verifying drafted tokens can require loading additional expert weights from memory, partially offsetting drafting gains at batch size 1. At larger batch sizes, there is more overlap in activated experts across sequences, improving reuse.

The field is rapidly developing variants: Medusa adds multiple MLP heads to reuse last hidden states; EAGLE builds on both last hidden state and preceding tokens for autoregressive drafting; FastMTP uses a shared-weight MTP head with language-aware vocabulary compression. The meta-pattern is clear — do not only optimize one-token-at-a-time decoding, make decoding wider.

A normal inference runtime involves repeated CPU-to-GPU orchestration: launch kernel → wait → sync → launch next kernel → wait → update state → launch again. At very high token rates, these gaps compound. The GPU may be fast, but the system around it keeps creating small stalls.

A GPU-resident persistent mega-kernel tries to keep the hot loop on the device. Instead of the CPU orchestrating every decode step, the GPU owns more of the loop continuously:

This matters because the goal is not just high average throughput — it's avoiding tiny stalls in the inner inference loop. Once the model is quantized, sparse, and block-parallel, the runtime must not keep bouncing through CPU orchestration on every decode step.

FlashInfer — The Current State of the Art

FlashInfer, which won best paper at MLSys 2025, is a library and kernel generator for LLM serving that provides high-performance GPU kernels for attention, GEMM, and MoE operations. NVIDIA is now actively releasing its most performant TensorRT-LLM kernels into FlashInfer for easy integration into vLLM, SGLang, and custom inference engines.

FlashInfer's JIT system builds on CUDA/Cutlass templates derived from FlashAttention-2/3, injecting user-defined transformations directly into templates before compilation to produce fused attention kernels specialized for the specific variant and cache layout. After the initial compilation, kernels run in the microsecond launch latency range. Its load-balanced scheduling decouples plan and run stages, alleviating the load imbalance issue that arises from variable-length inputs in heterogeneous batches.

When someone asks "why build yet another SRAM hardware system for inference?" the answer is:

We may not need another chip. We need better use of the chips we already have.

The path to extremely high token throughput is not only about adding memory — it's about changing the shape of inference. Each layer of the stack addresses a different bottleneck, and they compound:

SRAM in this framing is not necessarily a new chip. It is the on-chip staging layer that makes the loop fast when the runtime is designed correctly. The real thesis is not "SRAM as a separate hardware religion." The real thesis is deterministic, tiled, sparse, quantized, parallel, GPU-resident inference.