Coherent Fabrics:
The Memory Highways
Behind Agentic AI

Why coherent fabrics matter now

AI infrastructure is undergoing a fundamental shift — from compute-centric to memory-centric. The GPU is increasingly the thing that waits, not the thing that bottlenecks.

For years the dominant AI story was linear: more GPUs, more FLOPS, bigger models, faster training. That equation still holds for raw throughput, but it breaks down the moment you zoom out to the full inference loop. Once models grow large, context windows lengthen to hundreds of thousands of tokens, and agentic systems start calling tools, reading codebases, retrieving documents, and managing multi-step workflows — the bottleneck quietly migrates.

That is exactly the territory of coherent fabrics. They sit underneath CPUs, GPUs, memory expansion devices, SmartNICs, and accelerators, trying to turn fragmented memory islands into something that behaves like a single coordinated system.

# Old bottleneck
GPU compute → buy more H100s → solved

# New bottleneck
memory movement + synchronization + orchestration
→ buying more GPUs often makes it worse
  (more devices = more copies = more coherence traffic)

First: what is a "fabric"?

In hardware systems, a fabric is the communication network connecting compute and memory components. It is the highway system between chips, sockets, accelerators, and racks. Not a single bus — a structured mesh of links, switches, and protocols.

Examples of fabrics include PCIe, NVLink/NVSwitch, AMD Infinity Fabric, and CXL-based fabrics. The critical distinction is what each fabric carries. Some are pure transport — they move bytes. Others carry memory semantics: ownership, visibility, validity. A coherent fabric participates in memory consistency, not just packet delivery.

Then: what does "coherent" mean?

Coherence is about making sure that when multiple processors cache the same memory location, they do not silently disagree about what the current value is.

Every modern CPU caches memory in L1/L2/L3 to avoid slow DRAM accesses. When you have one CPU, there's no conflict. When you have two CPUs sharing RAM, both can cache the same address simultaneously — and one can modify it without the other knowing. Same problem, amplified, when GPUs and accelerators enter the picture.

Modern processors use cache coherence protocols to track the state of every cache line across all devices. The famous family of protocols is MESI and its variants. Before getting to those, note the key insight:

Coherence = hardware-assisted agreement about the latest memory state

# Without it:
programmer must:  flush → copy → barrier → invalidate → synchronize
# every handoff between devices is a bespoke software dance

# With it:
hardware tracks who has which copy
# and automatically invalidates or updates stale entries

The MESI protocol: how hardware tracks ownership

MESI is the dominant family of cache coherence protocols. The name is an acronym for the four states a cache line can be in. Understanding MESI gives you the mental model for every coherent fabric — even CXL and NVLink extensions ultimately build on these ideas.

MOESI adds a fifth state — Owned — allowing a cache to satisfy another cache's read request directly (cache-to-cache transfer) without going to main memory first. This matters enormously for multi-GPU systems where you want GPUs to share data without bouncing through CPU memory.

Non-coherent vs coherent systems

The clearest way to understand coherent fabrics is to see what changes when you remove the coherence guarantee.

# programmer manages everything
CPU RAM → copy → GPU VRAM
GPU writes result
GPU VRAM → copy → CPU RAM
CPU uses result

# plus manually:
flush · barrier · invalidate

# hardware manages consistency
CPU and GPU access same address
Fabric tracks ownership
Stale copies auto-invalidated

# programmer writes:
ptr[i] = value   // just works

Why AI systems care so much

AI used to be mostly about weights. Agentic AI is increasingly about state.

Model weights are large but relatively static during inference — they're loaded once and read repeatedly. The dynamic picture is far messier. Every step of an agentic loop creates and consumes state: prompts, retrieved documents, embeddings, tool outputs, intermediate plans, logs, and most importantly — KV cache.

In transformer inference, the KV cache stores key/value tensors for every token in the current context. Subsequent tokens attend over these without recomputing. With a 128k-token context on a large model, the KV cache can be 200–400 GB per session — easily larger than the weights themselves.

Once you have prefix sharing, you also need prefix routing — a scheduler that knows where each KV prefix lives and steers new requests toward it rather than migrating gigabytes of state. That is why coherent fabrics and KV routing are deeply coupled problems.

# Scale the problem
1 agent, 128k ctx  →  ~256 GB KV
100 agents, shared 100k prefix  →  25.6 TB naive  vs  256 GB + 100 × delta

# The fabric makes sharing possible
# The scheduler makes sharing practical

Real-world fabrics and their tradeoffs

No single fabric solves every layer of the problem. Different vendors attack different segments of the memory hierarchy.

CXL — Compute Express Link

CXL is an industry-backed cache-coherent interconnect built on top of PCIe physical infrastructure, backed by Intel, AMD, NVIDIA, Samsung, and others. It defines three sub-protocols that can be mixed:

NVLink / NVSwitch — NVIDIA's GPU fabric

NVLink is NVIDIA's proprietary high-bandwidth interconnect between GPUs (and between GPU and CPU on Grace Hopper). NVSwitch is the fabric chip that connects up to 576 GPUs in full all-to-all topology. The bandwidth numbers are staggering:

AMD Infinity Fabric

AMD's Infinity Fabric began as the die-to-die interconnect inside Zen CPUs and has evolved into a full system fabric connecting chiplets, compute dies, memory controllers, and I/O dies within a package and across sockets. In the CDNA3 GPU architecture (Instinct MI300), Infinity Fabric tightly integrates CPU and GPU dies with shared HBM, creating a genuinely unified memory space — the CPU and GPU share the same physical memory pool with hardware-managed coherence.

CCIX, OpenCAPI, Gen-Z, and the standards landscape

Several industry standards have attempted to generalize cache-coherent interconnects beyond any single vendor: CCIX (Cache Coherent Interconnect for Accelerators), OpenCAPI, and Gen-Z. Many of their ideas have been absorbed into CXL, which has emerged as the primary industry standard. The direction of the industry is clear: accelerators should participate as first-class citizens in the coherence domain, not merely sit behind slow copy boundaries.

How a coherent fabric actually behaves

At a high level, the fabric must answer a set of questions for every shared cache line in real time:

In small systems, coherence can be handled with snooping protocols — every device listens to every transaction on the bus and reacts. This works at low device counts but becomes untenable at scale because broadcast traffic grows with every device added. At large scale, directory-based coherence is standard: a central directory (or distributed directories) maintains metadata about who shares or owns each cache line, and only notifies the relevant parties.

# Snooping protocol (small systems)
device broadcasts:  "I'm writing to address 0x4A2F"
everyone listens and reacts
→ O(n) broadcast traffic per write  # doesn't scale

# Directory protocol (large systems)
device tells directory: "I want to write 0x4A2F"
directory checks sharer list: [CPU0, GPU3]
directory sends invalidation only to CPU0 and GPU3
→ targeted messages, scales to thousands of nodes

Power & danger: what coherence giveth

The false-sharing trap

Cache lines are typically 64 bytes. If two devices each write to different bytes within the same cache line, the hardware still treats this as a conflict and forces serialization — even though the writes don't logically overlap. This is false sharing, and it can silently destroy performance in ways that are nearly invisible without hardware counter instrumentation.

# False sharing example
struct AgentState {
  uint64_t counter_a;  // bytes 0–7  — Agent A writes this
  uint64_t counter_b;  // bytes 8–15 — Agent B writes this
  // Both on same 64-byte cache line!
};

# Hardware sees:
Agent A writes → invalidates Agent B's copy
Agent B writes → invalidates Agent A's copy
→ cache line bounces between devices constantly
# Fix: pad to separate cache lines
alignas(64) uint64_t counter_a;
alignas(64) uint64_t counter_b;

Coherent fabrics and KV-cache routing

KV-cache routing is the clearest AI-native application of everything we've discussed. It is where the hardware capabilities of coherent fabrics become a real scheduling and orchestration problem.

Suppose a long-running agent has built a 100k-token context. The KV cache for that context is resident on GPU 3. The next step is scheduled on GPU 7. Without fabric intelligence, the system faces a brutal choice:

A coherent fabric doesn't automatically solve KV routing — but it gives the scheduler better primitives: shared memory windows, peer access, coherent reads without explicit copies, and potentially a pooled memory tier where the shared prefix lives once and is readable by any GPU on the fabric.

# Without coherent fabric
request arrives for session_47 (KV on GPU 3)
scheduled on GPU 7
→ cudaMemcpy 256 GB  # PCIe: ~9s. NVLink: ~0.3s
→ execute decode
→ copy result back

# With coherent fabric + smart scheduler
request arrives for session_47
scheduler: KV affinity = GPU 3, pool_offset = 0x40000000
route to GPU 3  # or map CXL window to any GPU
GPU reads prefix coherently from pool  # no copy
→ execute decode  # latency: fabric access time, not copy time

# The policy question that remains:
move compute to KV   # works when KV is large, GPU has capacity
move KV to compute   # works when KV is small or NVLink is fast
share via pool       # works when many agents share prefix

The system architecture this points toward

The most interesting future AI servers may not look like "a CPU with attached memory and some GPUs." They may look like a memory fabric with compute engines attached — where the fabric is the primary coordination mechanism, and compute is a fungible resource dispatched to wherever the data already lives.

CPU       →  orchestration, policy, scheduling
GPU       →  tensor compute (dispatched to where KV lives)
SmartNIC  →  data movement, routing, compression offload
CXL pool  →  expanded shared coherent memory tier
Fabric    →  consistency + transport + ownership + routing