SRAM-Based Linux Inference Servers: A Kernel-Level Latency Autopsy

1. Why SRAM-Based Inference Feels Different

In traditional inference systems, much of the performance variance is hidden inside memory hierarchy behavior: cache misses, DRAM access patterns, HBM pressure, contention, kernel launch overhead, queueing, and runtime scheduling. In an SRAM-heavy design, more of the working set is moved into fast on-chip memory, and the execution plan is often shaped more aggressively by the compiler or runtime.

2. The End-to-End Request Lifecycle

The inference dataplane may be SRAM-powered, but the request lifecycle is still host-driven. A user request has to be admitted, batched or routed, copied or referenced, submitted to a device queue, completed, and delivered back to userspace.

Userspace inference request
  → syscall / io_uring
  → kernel submission path
  → driver + DMA programming
  → device execution             // most deterministic part
  → interrupt or polling completion
  → softirq / completion handling
  → scheduler wakeup
  → userspace continuation

3. Submission Path: Where the Request Enters the Kernel

The submit path is often underestimated. Before an accelerator can read request data safely, the host has to ensure that memory is stable, mapped, and visible to the device. That is where page pinning, IOMMU mapping, descriptor construction, and doorbell writes appear.

io_uring_enter()
  → __do_sys_io_uring_enter()
    → io_submit_sqes()
      → io_issue_sqe()
        → driver submit()

Driver submission:
  → pin_user_pages()        // formerly discussed as get_user_pages()
  → dma_map_sg()            // IOMMU translation + DMA visibility
  → build DMA descriptors
  → write device doorbell   // PCIe MMIO

Submit-path costs that show up in P99

• Page pinning: page-table walks, page references, and potential TLB/cache disturbance.
• IOMMU mapping: address translation setup for the device view of memory.
• Descriptor construction: CPU cacheline activity and queue ownership transitions.
• MMIO doorbell: posted writes, ordering rules, and PCIe interaction.
• Queue contention: multiple submitters fighting over shared rings or locks.

4. Completion Path: IRQ → SoftIRQ → Wakeup → Schedule

The completion path is the latency autopsy. People often focus on the interrupt handler, but the ISR itself is usually designed to do minimal work. The expensive and variable work is deferred: softirq processing, completion queue handling, waking the target task, and actually getting that task scheduled.

Hardware completion event

→ do_IRQ()
  → irq_enter()
  → handle_irq_event()
    → driver ISR
       // acknowledge device, collect completion metadata,
       // defer heavy processing
  → irq_exit()
    → __raise_softirq_irqoff()

→ __do_softirq()
  → net_rx_action() / blk_done_softirq() / driver completion handler
  → update completion queue
  → mark userspace-visible completion

→ try_to_wake_up()
  → ttwu_queue()
  → sched_ttwu_pending()
  → enqueue task on runqueue

→ schedule()
  → pick_next_task_fair()
  → context_switch()

→ userspace resumes

Why softirq is often the real trap

Softirq is a deferred execution mechanism. It is fast when the system is quiet, but it can become a batching point under load. If a CPU is handling a burst of completions, packets, or block I/O events, the target inference thread may not see completion immediately. The hardware finished. The kernel did not finish delivering that fact to userspace.

5. Microsecond-Level Latency Breakdown

Exact numbers depend on CPU generation, kernel version, BIOS settings, IOMMU mode, NUMA placement, interrupt moderation, driver behavior, and load. But the shape of the problem is consistent: Linux overhead can be in the same order of magnitude as the accelerator execution window.

The uncomfortable conclusion: SRAM execution can be smaller than the scheduler delay alone.

6. Scheduler Effects: The Silent Killer

The scheduler is optimized for fairness, utilization, and general-purpose behavior. Inference serving cares about something narrower: predictable wakeup and bounded tail latency. Those goals often conflict.

try_to_wake_up()
  → task becomes runnable
  → task is enqueued
  → scheduler chooses when it actually runs
  → context_switch()
  → userspace sees completion

Scheduler-driven latency mechanisms

• Runqueue contention: the completion thread is runnable but waits behind other work.
• CPU migration: the task resumes on a different CPU and loses cache warmth.
• Preemption windows: kernel or userspace code may delay when the target can run.
• Tick behavior: timer and tick configuration can affect latency distribution.
• Fairness vs latency: CFS/EEVDF fairness is not the same as inference deadline control.

7. Interrupt Mode vs Polling Mode

Interrupts are efficient when work is sparse. Polling is wasteful but predictable when work is constant. SRAM inference servers often operate in a regime where burning a CPU core to protect P99 can be a rational trade.

Dedicated polling loop:
  bind thread to isolated CPU
  pre-register buffers
  submit work through shared ring
  while service is active:
      poll completion queue
      process completions immediately
      submit next ready batch

This is why the CPU should not be treated as a passive host. In these servers, the CPU is the timing controller, queue manager, and completion delivery mechanism for the accelerator.

8. Tail at Scale: Why Small Delays Explode at P99

The post earlier said variance compounds. More precisely, the server’s tail latency is not determined by one component. It is determined by the slowest unlucky component in the chain. If a request crosses ten stages, and each stage has a small probability of a bad delay, the chance that at least one stage hits a bad delay is much larger than the chance of one isolated delay.

P50 path:
  warm thread + no GUP surprise + clean IRQ + empty runqueue

P99 path:
  GUP slow path + IOMMU churn + IRQ delay + softirq backlog + scheduler wait

If deterministic compute removes the big obvious source of variance, the remaining kernel micro-delays become the new product experience.

9. The CPU’s Responsibility in an SRAM-Based Inference Server

In the GPU-era mental model, the CPU “feeds” the accelerator. In the SRAM-inference mental model, that framing is too weak. The CPU is the control-plane engine that determines whether the deterministic dataplane is actually reachable at low latency.

This is the heart of the argument: the CPU is no longer just a general-purpose host. It is the inference control plane.

10. Problem → Mitigation Map

This post is not yet a full redesign proposal, but the mitigation map is already visible. Each kernel cost points toward a specific systems technique.

11. Architectural Conclusion

SRAM-based inference changes the system bottleneck by making accelerator execution less mysterious. That is a good thing, but it means the host can no longer hide behind the device.

Total latency =
    Linux submission path
  + DMA and queueing
  + deterministic SRAM execution
  + Linux completion path
  + scheduler wakeup

The bottleneck did not disappear. It moved into the Linux control plane.

12. Final Statement

The next phase of inference performance will not be won only by faster dataplanes. It will be won by making the CPU, kernel, driver, DMA, and completion path behave like a deterministic inference pipeline instead of a generic best-effort operating system path.