Mostly yes — a lot of serving speed does come from reusable kernels. But it's more accurate, and more useful, to think of them as reusable kernel families rather than one universal kernel that accelerates every model on contact.
A kernel doesn't contain model knowledge or weights. It implements an operation — nothing more:
attention(Q, K, V) grouped_GEMM(tokens, expert_weights, routing)
The model supplies the tensors, the dimensions, the layouts, the parameters. The kernel just does the arithmetic — fast, and in the right shape. That distinction is the whole story: reuse depends entirely on whether two models feed the kernel compatible shapes, not on whether they're related in any other sense.
01 Attention Kernels
the most reusable layer in the stack
Standard attention implementations travel remarkably well across models that share compatible attention semantics:
- Multi-head attention
- Grouped-query attention
- Causal masking
- Paged KV caches
- Common head dimensions
- Supported FP16 / BF16 / FP8 formats
- Prefill and decoding
Libraries such as FlashInfer provide reusable kernels for paged/ragged KV caches, prefill, decoding, MLA, and sparse attention.
Concretely, the same paged-attention kernel can serve an entire lineage of Llama-like models without modification:
The weights change. The number of layers changes. But the mathematical attention operation stays compatible, so the kernel doesn't need to know or care which model it's running.
Where reuse breaks
That compatibility is fragile in specific, predictable ways. Any of the following can force a custom kernel:
- Different head dimension
- Different KV layout
- Unusual positional encoding
- MLA instead of GQA
- Sliding-window or sparse pattern
- Quantized KV with different scale metadata
- KDA / recurrent state instead of ordinary attention
A newly released architecture that trips any of these wires doesn't inherit the existing kernel ecosystem for free — it needs a custom kernel, or a new specialization bolted onto an existing library.
02 MoE Kernels
reusable primitive, model-specific assembly
Mixture-of-experts models route each token through only a subset of the network, which splits the serving problem into two distinct stages:
The expensive part — the expert computation itself — is usually implemented as a grouped GEMM: many matrix multiplications, of different effective batch sizes, executed together rather than one at a time.
Grouped GEMM is a genuinely reusable primitive: NVIDIA CUTLASS provides reusable grouped-GEMM and MoE building blocks specialized by tile size, data type, scaling format, and GPU generation.
But "reusable primitive" isn't the same as "drop-in fused kernel." A complete fused-MoE kernel has to match a long list of model-specific details before it will run correctly:
- Number of experts
- Experts selected per token
- Expert hidden dimensions
- SwiGLU / GELU or another activation
- Routing and normalization semantics
- Weight precision and scale layout
- Tensor / expert parallelism
- Token permutation format
- GPU architecture
This is exactly why FlashInfer supports standard top-k routing plus specialized routing patterns tied to specific model families like DeepSeek and Llama 4 — the underlying machinery is shared, but some routing paths still need family-specific support built in.
A Reusability Hierarchy
from pure hardware primitive to bespoke architecture
Laid out end to end, kernel reusability forms a fairly clean spectrum — the closer you get to raw hardware instructions, the more universal the kernel; the closer you get to a specific architecture's quirks, the more bespoke it has to be.
Even a kernel sitting comfortably on the reusable end of that spectrum usually isn't served as one generic binary. The runtime typically selects or generates a specialization tuned to the exact serving configuration:
GPU = H100 dtype = FP8 head_dim = 128 page_size = 16 expert_dim = 2048
This selection happens through templates, JIT compilation, or runtime autotuning — the "reusable kernel" is often really a reusable generator of many narrow, hardware-tuned variants.
What Open Weights Actually Buy You
Put the pieces together and the practical payoff of open weights becomes concrete. If the released model uses a supported, well-trodden architecture, optimizing it is close to a solved problem:
architecture configuration +
supported vLLM / SGLang / TensorRT-LLM backend +
existing FlashAttention / FlashInfer / CUTLASS kernels
= optimized serving without retraining
For a conventional Llama-like model, most of the important kernels already exist off the shelf. A quantized clone can often reuse them directly, after converting the weights into the layout the kernel expects.
Conventional architecture
Llama-shaped attention, standard MoE routing → existing kernel libraries apply almost immediately. Optimization is largely a configuration problem.
Novel architecture
Kimi's KDA changes the operation itself. Ordinary attention kernels simply don't apply — Kimi had to ship specialized KDA kernels and integrate them into the serving runtime directly.
Standard attention and MoE architectures reuse mature kernel libraries extensively. Fine-tuned or quantized clones usually inherit that compatibility for free. Novel attention, routing, state layout, or quantization schemes commonly require integration, specialization — or an entirely new kernel.
The takeaway isn't that kernels are universal — it's that kernel families track architectural families. Stay inside a well-supported shape and the ecosystem does most of the optimization work for you. Step outside it, the way Kimi did with KDA, and the convenience disappears: you inherit none of the tooling, and you're back to writing the serving stack from close to first principles.