Hardware-Enforced On-Chip Memory Residency for Neural Network Inference Accelerators

The Memory Hierarchy and Why It Hurts

Every inference accelerator lives inside the same basic two-tier memory reality. On-chip volatile memory, typically SRAM, sits on the die and delivers the bandwidth the compute engines actually want. Off-chip memory, whether HBM, GDDR, or LPDDR, holds the full model but comes with a far more expensive byte-movement budget.

That asymmetry matters most during decode. A decode-step matrix-vector multiply has extremely low arithmetic intensity. In practical terms, that means performance is set less by how many operations the accelerator can issue than by how often it has to drag weights back from off-chip memory into SRAM.

Once arithmetic intensity falls that low, the accelerator becomes a machine for moving bytes. When identical tensors are reloaded token after token, the real cost of inference becomes bytes moved per token, not nominal peak FLOPS.

What Autoregressive Inference Actually Looks Like

Autoregressive generation repeatedly runs the model forward one token at a time. That means many of the most important tensors are not merely reused; they are reused at every single generation step. Attention projections, feed-forward blocks, embeddings, and hot KV structures are logically stable across the session even though they may far exceed on-chip capacity in aggregate.

Conventional replacement logic does not understand this pattern. It sees pressure, not permanence. It sees a large working set and evicts lines that, from the workload's perspective, are about to become valuable again almost immediately.

The accelerator is not bandwidth-bound only because the model is large. It is bandwidth-bound because the replacement policy cannot express that some data is logically permanent for the life of the request.

From the cache controller's perspective, these accesses resemble a stream of misses under relentless working-set pressure. There is no native mechanism that says, "this tensor should survive across the next 512 decode steps unless privileged software explicitly releases it."

The Existing Solutions and Why They Are Incomplete

The industry has accumulated several partial responses to the AI chip memory bottleneck. They are useful. They are often necessary. But they do not establish the specific invariant decode workloads want most: guaranteed on-chip residency.

The central distinction is between a hint and a contract. A hint asks the hardware to try. A hardware-enforced residency primitive changes the structure of victim selection itself so the hardware cannot silently break the promise.

A Crisp Analogy: Wired TLB Entries

Computer architecture has solved an analogous problem before. Translation Lookaside Buffers in MIPS, ARM, x86, and RISC-V families have long supported the concept of wired entries. Privileged software can mark an entry as non-evictable, and the replacement engine simply skips it. Not preferentially. Unconditionally.

The analogy is powerful because the important design properties carry over cleanly from translation residency to data residency.

That is the architectural move here: take the idea of a wired TLB entry and reinterpret it as a wired on-chip memory residency primitive for neural network inference accelerators.

The Wired On-Chip Residency Primitive

The proposal is intentionally minimal. It does not require a new accelerator category. It requires a stronger contract inside the memory subsystem.

Tag-array extension

Each on-chip tag entry receives a small extension. The crucial field is a WIRED bit, but the design is stronger when paired with tenant identity and generation metadata.

Standard tag entry:
┌────────┬───────┬────────────────────────────────┐
│ Valid  │ Dirty │       Address Tag (N bits)     │
└────────┴───────┴────────────────────────────────┘

Extended tag entry:
┌────────┬───────┬──────────────┬───────────┬───────┬──────┐
│ Valid  │ Dirty │ Address Tag  │   WIRED   │  TID  │ GEN  │
│        │       │  (N bits)    │  (1 bit)  │ (M b) │ (K b)│
└────────┴───────┴──────────────┴───────────┴───────┴──────┘

WIRED : If asserted, the line is excluded from all victim selection.
TID   : Associates the binding with the tenant or execution context.
GEN   : Generation counter used for O(1) invalidation without a tag scan.

This is architecture-independent. The same semantic idea applies to set-associative caches, direct-mapped structures, and explicitly managed on-chip data arrays, whether the implementation target is a GPU, NPU, TPU, FPGA, or dedicated ASIC.

BIND and RELEASE semantics

The software interface can be exposed as custom instructions, queue commands, memory-mapped control writes, or a driver API. What matters is that the semantics are explicit. BIND establishes the contract. RELEASE tears it down.

// Associate a wired on-chip region with off-chip backing storage.
// onchip_target : region identifier or explicit on-chip address
// offchip_addr  : physical address in HBM / GDDR / CXL / host memory
// flags         : PREFETCH | IMMUTABLE | SOFT_PIN

BIND(onchip_target, offchip_addr, size_bytes, tenant_id, flags)
  -> binding_handle
  | FAULT(QUOTA_EXCEEDED | ALIGN_ERROR)

// Contract:
// 1. Validate target and quota.
// 2. If PREFETCH: DMA from offchip_addr to onchip_target.
// 3. Assert WIRED=1, TID, GEN on all covered tag entries.
// 4. If IMMUTABLE: write-protect [offchip_addr, +size) in MPU.
// 5. Return opaque binding_handle.

RELEASE(binding_handle, flags)
  -> OK
  | FAULT(INVALID_HANDLE | PERMISSION_DENIED)

// flags: WRITEBACK | INVALIDATE
// Clears WIRED=0 and decrements per-tenant quota usage.

Replacement-controller exclusion

This is the line between a real primitive and a polite suggestion. Any line whose WIRED bit is asserted is excluded from victim selection unconditionally. It does not matter whether the replacement policy is LRU, pseudo-LRU, RRIP, random, or anything else. The controller cannot evict a wired line because the line is never admitted into the victim set in the first place.

Without that property, BIND is just another cache hint. With it, the primitive becomes a hardware guarantee.

Three Mechanisms Worth Looking at Closely

Immutable mode and coherence bypass

Model weights are read-only after load. If the runtime binds such a region and declares it IMMUTABLE, the architecture can suppress unnecessary coherence traffic to the on-chip copy while write-protecting the backing range in the memory protection unit. Any illegal write becomes a BINDING_VIOLATION.

That matters because it turns the on-chip copy into the authoritative copy for the duration of the binding and removes coherence bandwidth that would otherwise do no useful work.

Soft-pin and pre-eviction signaling

A hard wired guarantee is ideal for the most critical tensors, but it can become dangerous near capacity limits. If too much SRAM is permanently wired, the system can deadlock under pressure. A SOFT_PIN state creates a controlled compromise: retain aggressively, but if eviction becomes unavoidable, emit a RESIDENCY_LOSS signal before the eviction completes.

BIND(SOFT_PIN)                  BIND(no SOFT_PIN)
       |                                  |
       v                                  v
 ┌──────────────┐                 ┌─────────────────┐
 │  SOFT_WIRED  │                 │     WIRED       │
 │  Evictable   │                 │  Never evicted  │
 │  under       │                 └─────────────────┘
 │  pressure    │
 └──────┬───────┘
        │ pressure > threshold
        v
 ┌──────────────────────────────────────────┐
 │  RESIDENCY_LOSS emitted                  │
 │  · binding_handle                        │
 │  · estimated_eviction_ticks              │
 └──────┬───────────────────────────────────┘
        │ grace period (configurable N ticks)
   ┌────┴──────────────────────┐
   │ Runtime may respond:      │
   │  · Re-BIND to new region  │
   │  · Schedule reload        │
   │  · Accept eviction        │
   └────┬──────────────────────┘
        │ grace period expires
        v
    EVICTED -> normal replacement eligibility

That is the difference between a system that stalls catastrophically and one that degrades gracefully.

Generation-counter invalidation

In multi-tenant inference systems, scanning the full tag array to clear one terminated tenant's bindings is expensive. Generation counters avoid that. Each tenant has a current generation in the quota register file. Each wired line records the generation that created it. If the numbers stop matching, the controller immediately treats the line as effectively unwired.

Invalidate the tenant by incrementing one register. No global scan. No long cleanup sweep. One architectural write.

The Runtime Arbiter

Hardware enforcement does not replace software policy. It lets software make decisions that will actually stick. A residency arbiter can decide which tensors, KV blocks, or decode-critical state deserve the scarce wired budget.

At model load or phase transitions, the arbiter can rank candidate bindings by expected byte savings over the rest of the session.

// For each candidate binding C:

score(C) = reuse_count(C) * bytes_saved_per_reuse(C)
           -----------------------------------------
              bind_cost(C) + quota_cost(C)

// Inputs:
// · reuse_count : estimated remaining accesses this session
// · bytes_saved : tensor size * off-chip bandwidth cost
// · bind_cost   : DMA prefetch latency, amortized
// · quota_cost  : bytes consumed from the tenant's wired budget

// Selection:
// Greedy knapsack over quota budget, descending score.
// IMMUTABLE is applied automatically to read-only weight tensors.
// SOFT_PIN is used for lower-criticality candidates near quota limits.

The same arbiter model can run across GPU, NPU, TPU, FPGA, or ASIC targets because it depends on the abstract interface, not on one vendor's cache-control dialect.

Why This Must Live in Hardware, Not Merely in Software Hints

Software can schedule. Compilers can tile. Runtimes can prefetch. Those are useful, but they remain approximations when the memory system itself is still allowed to discard the very data the runtime intends to preserve.

Scratchpad schemes usually guarantee residency only inside a single kernel lifetime. Streaming DMA pipelines deliberately move tiles through transient buffers. Those are not the same thing as a region that survives across arbitrary numbers of launches until an explicit release.

The boundary conditions are where the distinction matters most: cross-tenant pressure, phase transitions, runtime interference, or an unexpected control path. A heuristic can fail exactly when conditions get interesting. A hardware invariant holds even then.

The bandwidth arithmetic is brutal and simple. If a 7B-parameter model emits a 512-token sequence, and the dominant cost is repeatedly dragging the same weight footprint over the off-chip link, then every tensor that remains resident on chip meaningfully lowers bytes moved per token. That reclaimed bandwidth can go to KV cache traffic, speculative work, or the next request in line.

Why This Matters for Real AI Chips

Inference efficiency is increasingly about memory traffic discipline, not just raw matrix throughput. The accelerators that win over the next generation of LLM deployment are not only the ones that compute quickly. They are the ones that stop paying for the same bytes twice.

That is why the distinction between a cache hint and hardware-enforced memory residency matters so much. One asks politely for locality. The other changes the machine so locality becomes part of the architecture. In a world where real AI chips are increasingly constrained by bytes moved per token, that shift is not cosmetic. It is foundational.