GhostKV:
Attention Without
Storing Full KV

Inference is colliding with memory movement, not just compute.

As context windows grow to 128K, 1M, and beyond, LLM inference increasingly bottlenecks on the cost of moving KV tensors — not on FLOPs alone.

Every decode step must load, decompress, and attend over all stored KV state. In a 128K-token context at 32 layers and 128 heads with half-precision 128-dim keys/values, the raw KV cache approaches ~67 GB. Even with 4-bit quantization, that's ~17 GB — and it must be partially traversed every step.

The common instinct: compress KV more aggressively. GhostKV asks a different question:

Most tokens become semantically dormant.

In long-context transformers, empirical analysis of attention patterns reveals a consistent structure: a small fraction of historical tokens — typically sink tokens at position 0–5, instruction tokens, and recent tokens — dominate total attention mass. The majority are nearly inactive.

StreamingLLM observed that attention sinks are critical. H2O and SnapKV showed that only a small per-head KV budget is needed for high quality. GhostKV goes further: instead of keeping a fixed budget of the "most important" tokens, it converts dormant tokens into compact witnesses that can prove their own irrelevance — or trigger reconstruction when they can't.

GhostKV introduces attention witnesses.

Instead of storing full key/value tensors for cold tokens, GhostKV converts them into ghost records: compact structures that retain just enough information to answer one question per decode step:

A ghost record has three components:

From full KV to ghost state

Standard transformers store tensors shaped KV[layer][token][head][dim]. GhostKV introduces a lifecycle with four states. Fresh tokens begin as exact KV and transition through the hierarchy as they age and their attention mass declines.

Bounded attention elimination

GhostKV tries to prove a token cannot matter before decompression or reconstruction occurs. For query vector Q and ghost token Gᵢ, the runtime computes a conservative upper-bound estimate of the maximum possible attention score:

// Runtime pseudocode — per ghost token per decode step
for Gᵢ in ghost_tokens:
  bound = sketch_sim(Q, Gᵢ.sk) + Gᵢ.ε_res + Gᵢ.σ_anchor
  if bound < θ_elim:
    // Provably safe to skip. Never loaded, decompressed, or transferred.
    softmax_denom += exp(ELIM_CONSTANT)  // add negligible bounded constant
    continue
  else:
    resurrect(Gᵢ)  // DMA fetch from DRAM/CXL

Walking through one decode step

Consider a model generating token T+1 in a 32K context. The query Q encodes "What was the final conclusion?" — semantically rich with long-range retrieval intent.

Sketch → Bound → Resurrect → Attend

Traditional KV-cache systems load and decompress first, then discover many tokens were irrelevant. GhostKV inverts this order — the filtering happens near memory, before data moves to the GPU.

Token resurrection

Tokens that survive the elimination bound are "resurrected" — their full or near-full KV is reconstructed before being handed to the attention kernel. Several reconstruction paths can be combined:

Latency model: when resurrection is viable

The core performance concern: decode steps are fast and memory-bound. If a resurrected token's DMA fetch isn't hidden, it stalls execution. The viability condition is:

At decode batch size = 1, compute time is short (~10–50μs on A100), making latency hiding difficult. GhostKV's primary target is batch size > 1, where compute per step is proportionally longer. At batch = 8 with 2% resurrection rate (~2,560 tokens in 128K context), a pipelined DMA over 2,560 tokens × 4KB = ~10MB at 900 GB/s HBM takes ~11μs — overlappable with attention compute. Predictive prefetch adds another buffer: scanning sketches for step T+1 while step T's attention runs.

When and where ghostification happens

Converting a token from warm KV to ghost state requires computing its sketch, assigning an anchor cluster, and writing the residual fingerprint. This must not happen on the GPU's critical decode path. Three viable approaches:

Head-aware ghosting

Different attention heads specialize differently. Some heads are local and syntactic — they care only about nearby tokens and decay rapidly after a few hundred positions. Others are retrieval heads that carry long-range semantic dependencies across thousands of tokens.

GhostKV should not ghost all heads equally. Per-head ghost aggressiveness can be determined from profiling or learned during fine-tuning:

Memory-side attention filtering

GhostKV becomes especially compelling when the sketch-and-bound computation happens near memory — before data ever crosses the HBM or PCIe bandwidth bottleneck. Critically, this is not purely a future-hardware story: suitable substrates exist today.

How GhostKV differs from prior work

Several important systems address KV-cache efficiency. GhostKV is most closely related to eviction-based and sparse-attention methods, but differs in a fundamental way: it does not discard or evict tokens — it converts them into reconstructable witnesses.

Why this matters beyond compression

Compression reduces capacity pressure. GhostKV reduces movement pressure. These are different problems with different bottlenecks.

Complexity reduction

The bandwidth savings can be expressed compactly. Let N = total context tokens, d = KV dimension, q = quantization factor, H = hot tokens, R = resurrected tokens per step. The decode memory traffic scales as:

Where R ≪ N is the key. If 85% of tokens are cold and 2% of those are resurrected per step, then R ≈ 0.017N. GhostKV's traffic approaches O(H · d) in the best case — linear in the hot window, not the full context.

Limitations & honest failure modes

A strong concept note acknowledges where it could be wrong. GhostKV has four principal threats:

Simulation plan

GhostKV is pre-experimental. Here is the exact sequence of experiments needed to validate or falsify its core claims — ordered from cheapest to most involved.

Research questions worth pursuing

GhostKV is a pre-experimental concept. These are the hard questions that determine whether it works in practice:

The main criticism — and the answer

This is the right criticism, and it has three answers:

A secondary criticism is that the system adds engineering complexity — a lifecycle manager, a sketch index, async ghostification, and a resurrection DMA pipeline — on top of an already complex inference stack. This is fair. GhostKV is not a one-line patch. The bet is that the bandwidth savings at long-context scale justify the complexity, the same way FlashAttention's tile scheduling justified its complexity over naive attention.

A different mental model for AI memory

GhostKV suggests that future inference memory systems may stop treating all stored state as raw tensors waiting to be loaded. Instead, cold memory becomes probabilistic, reconstructable, and query-conditioned:

• The exact tensor is no longer always the primary object in the cache.
• Cold tokens store enough to make an elimination decision — not enough to participate in attention directly.
• Reconstruction is expensive and rare — elimination is cheap and common.
• Memory behaves like a smart index, not a passive buffer.

Whether this pans out empirically depends on the tightness of elimination bounds, anchor stability, and whether resurrection latency can be hidden. These are solvable engineering problems. The conceptual shift — from "compress and load all" to "prove irrelevance before loading" — seems worth exploring.

Related areas

GhostKV sits at the intersection of several active research threads. Prior work it builds on or connects to:

KV-cache compression: KIVI, WKVQuant, and KVSharer explore quantization and sharing of KV tensors. Sparse/selective attention: H2O, SnapKV, PyramidKV, and ScissorHands develop eviction policies for KV budgets. Approximate attention: Reformer (LSH attention), Longformer, BigBird, and Scatterbrain use structured sparsity. Approximate nearest-neighbor search: LSH, HNSW, and FAISS form the basis for sketch-based similarity estimation. Memory-side compute: PIM (Processing-in-Memory), CXL smart memory, and near-data processing are relevant hardware directions. Semantic compression: Memorizing Transformers and MemGPT treat long-horizon memory as a retrieval problem.

The distinctive claim in GhostKV: combining sketch-based elimination, conservative upper bounds, and lazy reconstruction into a single lifecycle that avoids memory movement for cold tokens rather than just reducing their stored size.