Memory Intent IR: The Missing Compiler Output

The missing artifact in the AI stack

Today, most AI compilers emit some combination of a lowered graph, fused kernels, local buffer allocation plans, communication schedules, and device placement hints. Those artifacts are necessary. They are no longer sufficient.

They tell the system what must run. They do not tell the system how the objects involved in that execution are expected to behave over time — which tensors are truly latency-critical, which objects have short but intense reuse windows, which buffers are safe to spill, which groups of objects are usually needed together, and which data should be pinned because eviction would create a disproportionate stall.

The runtime is left to infer this, after the fact, from access history alone. That is a structurally bad position. The compiler already had most of this information. It just never wrote it down.

The core idea

A Memory Intent IR is a compiler-emitted intermediate representation that describes expected memory behavior — not just initial allocation. It is a behavioral contract for objects: how they live, how they expire, how hot they are, what deadlines matter, what tiers are acceptable, what spill policies are permitted, and what future reuse the system should preserve.

How this differs from existing approaches

The easiest way to misread Memory Intent IR is to treat it as a prettier buffer allocation table. That is not what it is. Existing approaches each capture a slice of the problem. None assemble it into a first-class transferable artifact.

Why this matters now

1. Models have outgrown single-device memory

Once a model spills across HBM, host DRAM, CXL pools, and peer accelerators, the number of placement decisions that can go wrong grows rapidly. A recency policy that worked at single-GPU scale becomes a liability at rack scale.

2. KV cache has become a first-order memory management problem

Long contexts, prefix sharing, speculative decoding, and retrieval-augmented generation all push KV cache management into a domain that generic page replacement cannot handle correctly. A prefix-shared KV block may look cold to LRU yet be one of the highest-value objects in the system because dozens of decode continuations are about to depend on it.

3. Expert routing creates dynamic hotness

In MoE models, which expert weights are "hot" changes across decode steps based on routing decisions. The compiler can observe this structure statically and emit probability-weighted prefetch hints. The hardware has no access to this information without an intent channel.

4. Hardware controllers are getting smarter — but need structured input

If hardware-resident or near-hardware orchestration exists (as in MCOS-HFC), it needs structured input. Otherwise it becomes a faster blind heuristic engine — powerful machinery with no map.

What a Memory Intent IR could look like

Conceptually the IR is organized around three levels: object descriptors, phase descriptors, and policy overrides. Object descriptors define type, size, lifecycle, and base placement preferences. Phase descriptors capture execution stages that reshape urgency and locality. Policy overrides let phase-specific behavior modify base rules without redundancy.

// ── Object descriptors ──────────────────────────────────────

object layer_18_weight_group_A {
  class              = WEIGHT_TILE
  size_bytes         = 134_217_728          // 128 MB
  persistence_scope  = SESSION
  reuse_window       = HIGH                 // reused every layer pass
  preferred_tiers    = [SRAM, HBM]
  fallback_tiers     = [DRAM, CXL]
  spill_policy       = SEVERE_PRESSURE_ONLY
  multicast_eligible = true                // can fan-out to multiple accs
  deadline_class     = HARD
}

object kv_prefix_seg_P {
  class              = KV_BLOCK
  sharing_scope      = SEQUENCE_GROUP(17, 18, 21)  // shared by 3 seqs
  persistence_scope  = TOKEN_PROGRESS
  anti_evict_rule    = WHILE_FANOUT_GT_1   // protect while shared
  preferred_tiers    = [HBM, DRAM]
  deadline_class     = HARD
}

object decode_buf_D {
  class              = ACTIVATION_TEMP
  persistence_scope  = STEP               // expires after one step
  recompute_class    = CHEAP              // safe to recompute
  spill_policy       = ALLOW
  deadline_class     = SOFT
}

// ── Phase descriptors with overrides ────────────────────────

phase decode_step {
  override kv_prefix_seg_P {
    anti_evict_rule      = WHILE_FANOUT_GT_1
    prefetch_horizon_us  = 12
    preferred_tiers      = [HBM, DRAM]
  }
  override expert_weight_group_E3 {
    route_probability    = HIGH           // router hints this is likely
    replication_allowed  = true
    prefetch_horizon_us  = 20
  }
}

phase optimizer_update {
  override gradient_shard_G {
    persistence_scope    = UPDATE_WINDOW
    preferred_tiers      = [DRAM, CXL]   // keep off HBM during fwd pass
    anti_evict_rule      = UNTIL_ALLREDUCE_COMPLETE
  }
}

What new compiler passes this requires

Memory Intent IR is not just a runtime improvement. It implies new compiler passes that elevate existing analyses into a formal, transferable artifact. The compiler already performs many of the underlying analyses — it just does not emit the results in a consumable form.

In other words: the compiler stops thinking only like a kernel scheduler and starts thinking like a memory strategist — one that describes not just what to compute but what the data will need to do while that computation runs.

A concrete inference example

In LLM serving, the compiler and serving stack often know several useful things before runtime pressure appears. Weight groups for active layers will be reused. Prefix-shared KV segments have disproportionate value. Some activation buffers are short-lived and cheap to recompute. Expert routing may make a subset of weights hotter over the next few decode steps.

That means the system does not need clairvoyance. It needs a disciplined way to express what is already structurally known.

The example policy split is straightforward: protect shared prefix KV (hard anti-evict); keep high-reuse weights in HBM (hard deadline); allow cheap decode temporaries to spill or recompute (soft deadline, cheap recompute); and pre-stage likely-hot expert weights a few steps ahead (route-probability override in the decode phase descriptor).

Why hardware people should care

Without a structured intent channel, hardware memory orchestration is forced into two unsatisfying modes: fixed heuristics that only work for some workloads, or software-driven micromanagement that drags the control path back into microsecond territory and defeats the purpose of the hardware accelerator.

Memory Intent IR offers a cleaner architecture. The controller remains generic and fabric-agnostic. But it is no longer blind. It receives normalized object semantics at model load time and combines them with live telemetry — queue depth, bandwidth pressure, residency fill, hit history — to make placement and eviction decisions that are simultaneously fast and informed.

What this means for the AI engineer

This is not only for compiler researchers or hardware architects. If it becomes real in production systems, the downstream effect for AI engineers is practical and immediate.

Objections worth taking seriously

What this unlocks next

Once a Memory Intent IR exists as a real artifact, several practical directions open up immediately and naturally.

Compute graphs are incomplete

A compute graph tells you what must be computed. A Memory Intent IR tells you how the objects involved in that computation are expected to behave under pressure — which ones are fragile, which are resilient, which are time-sensitive, which are cheap to reproduce, and which ones, if evicted at the wrong moment, will cascade into a stall that propagates to everything downstream.

As long as AI systems emit only the first artifact, they leave one of the central determinants of performance underspecified. And underspecified systems get patched with heuristics — fragile, opaque heuristics that work until they don't.

The gap is real. The compiler already has the information. The question is only whether the ecosystem decides that writing it down is worth the effort. Given what bad memory policy costs at scale, the answer seems clear.