Reference Diagram
Executive Technical Summary
Problem
Long-context inference turns KV cache into a dominant memory object. Decode repeatedly reads historical KV state while appending new KV for every generated token.
Observation
Tensor compute remains critical, but end-to-end serving can become bottlenecked by memory residency, data movement, and exposed transfer latency.
KVFabric thesis
Future systems may need a dedicated inference-memory fabric/control plane around accelerators to schedule, stage, compress, and move KV state.
One-line mental model: GPUs do math. KVFabric studies how future systems may orchestrate inference memory.
Inference Pipeline Mapping
| Stage | What happens | Primary compute/memory placement | KVFabric relevance |
|---|---|---|---|
| Input/tokenization | Prompt text becomes token IDs. | CPU/runtime, tokenizer memory. | Not the main acceleration target, but request metadata becomes useful for scheduling. |
| Prefill | Full prompt is processed once. Q/K/V are produced for each layer and KV cache is written. | GPU/ASIC tensor cores; HBM stores weights, activations, active KV. | KV write path: allocate KV blocks, select placement, prepare future residency metadata. |
| Decode | Each new token repeatedly reads prior KV cache and appends new KV state. | Attention kernels on accelerator; KV reads from HBM/SRAM/CXL depending on residency. | Main target: prefetch, SRAM staging, cold KV compression, CXL demotion, overlap scheduling. |
| Sampling/output | Logits are sampled into next token. | GPU/CPU depending on runtime. | Lower priority; output timing can still provide scheduling hints for next-token prefetch. |
Conceptual KVFabric Accelerator Blocks
Runtime Command Queue
Receives reserve, prefetch, compress, materialize, and release requests from the LLM runtime. This is the software/hardware boundary of the control plane.
Policy Microcontroller
Runs residency and movement policies: hot/warm/cold classification, deadline-aware prefetching, compression eligibility, and CXL demotion decisions.
KV Metadata SRAM
Stores compact per-block metadata: layer/head/token range, tier, compression state, access count, last access step, deadline, and reuse hints.
DMA / Prefetch Engines
Move KV blocks between HBM, SRAM staging buffers, CXL memory, and host DRAM. The goal is to hide movement behind compute.
Compression / Rehydration
Compresses cold KV state and rehydrates selected blocks near compute. This trades capacity and bandwidth against latency/quality assumptions.
SRAM Staging Buffers
Small, fast staging area for active or imminent KV blocks. Useful for next-token attention reads and avoiding exposed HBM/CXL stalls.
Conceptual Block-Level Layout
Memory Hierarchy Specification
| Tier | Role | Typical contents | Policy concern |
|---|---|---|---|
| SRAM | Ultra-fast staging | Current/next KV blocks, attention scratchpad, DMA buffers | Very limited capacity; must be deadline-aware. |
| HBM | High-bandwidth active memory | Model weights, active KV, activations | Expensive and finite; pressure rises with context and concurrency. |
| CXL / pooled memory | Capacity expansion tier | Warm/cold KV, compressed KV, less urgent blocks | Higher latency; useful only with prefetch and overlap. |
| Host DRAM | Large spill/control tier | Evicted KV, metadata backing, traces, runtime queues | Can rescue capacity but may add exposed latency if accessed synchronously. |
Overlap-Aware Execution Model
KVFabric’s value is not simply moving KV to cheaper memory. The key is whether movement and rehydration can be hidden behind accelerator compute.
Design target: reduce exposed transfer latency, not merely total transfer volume. A movement policy is only valuable if it improves the critical path or improves density at acceptable latency cost.
Runtime API Surface
A future runtime integration would expose KV intent to the orchestration layer. The API below is conceptual but makes the hardware/software contract concrete.
What This Is — and Is Not
Current repository scope
- Architecture prototype
- Systems simulator
- Control-plane model
- Policy experimentation framework
- Memory hierarchy study
Not currently implemented
- No production silicon
- No final hardware
- No RTL/FPGA datapath
- No real GPU kernels
- No production runtime integration
Roadmap: Simulator to Accelerator Architecture
| Stage | Scope | Status |
|---|---|---|
| Stage 1 | Simulator, memory tiers, workload models, residency policies | Current |
| Stage 2 | Trace-driven replay, policy comparison, overlap refinement, visualization | In progress |
| Stage 3 | Runtime API contract, integration shims, vLLM/SGLang trace adapters | Planned |
| Stage 4 | Cost-aware tiering, multi-tenant workloads, hardware-grounded profiles | Planned |
| Stage 5 | FPGA/datapath experiments, compression engines, DMA offload studies | Future research |
| Stage 6 | Production die studies, final hardware architecture, silicon implementation research | Long-term exploration |
Key Technical Questions
Granularity
What is the optimal KV block size for staging, compression, and metadata overhead?
Policy
When should the system prefer SRAM residency, HBM residency, CXL demotion, or compression?
Overlap
How much transfer and rehydration time can be hidden under token compute?
Economics
When is trading latency for HBM capacity economically beneficial?
Runtime hooks
What minimal API must serving runtimes expose to make KV orchestration practical?
Accelerator design
Which blocks are worth hardware acceleration: DMA scheduling, compression, metadata lookup, or staging?
Recommended Repo Integration
Add this specification as:
For the pinned X post or README hero, use the simplified line: “GPU does math. KVFabric studies how future systems may orchestrate inference memory.”