Patching
blk-mq for
KV-Cache-Aware
NVMe Scheduling
Inside Kairo — an RFC/POC that patches mq-deadline and blk-mq to prioritise
decode reads over prefill writes at the Linux block layer — decode p99 latency on generic NVMe SSDs, without bypassing the kernel.
Every large language model running in production today is quietly waging a war against physics. The attention mechanism at the heart of transformer inference demands that the model remember everything it has ever seen in a conversation — every token, every layer, every head — stored as a dense matrix of floating-point numbers called the KV cache. As context windows stretch from thousands to hundreds of thousands of tokens, that cache metastasises: a single 70B-parameter model serving a 128K-token conversation needs dozens of gigabytes of fast memory just to exist in a conversational state.
GPU HBM — the precious high-bandwidth memory bonded directly to the silicon — runs out first. CPU DRAM fills next. And then, inevitably, the KV cache spills onto NVMe SSDs: the fastest persistent storage available, but still orders of magnitude slower than either. At that point, every token the model generates depends on a storage read completing fast enough that the human on the other end of the chat box doesn't notice they're waiting. The p99 latency of a random NVMe read — typically 50–200 microseconds on a good drive, 300–500µs under mixed-workload pressure — becomes, quietly, the bottleneck of the entire inference pipeline.
This is the problem that Kairo is probing. Not from the comfortable vantage of a user-space inference framework, but from inside the Linux kernel itself — patching the block I/O scheduler to understand that not all NVMe reads are created equal, and that a decode read completing a user-facing token generation is categorically different from a background prefill write or a cache eviction discard. This is Part I of a series examining that bet: what it means, how it fits into the broader landscape of the KV-cache I/O problem, and whether the kernel is the right place to solve it.
The KV Cache I/O
Crisis, Explained
Before understanding why Kairo exists, it's worth grounding ourselves in the mechanics of what KV cache I/O actually looks like to a storage device. The transformer attention mechanism, for every layer and every attention head, produces Key and Value tensors for each input token. During prefill — processing the entire prompt — these get computed and written to the cache. During decode — the autoregressive generation of each output token, one at a time — the model reads back all previously cached K/V tensors to compute attention over the full context, then writes the new token's K/V pair.
This creates four structurally distinct I/O classes that co-exist on the same device simultaneously:
The brutally simple observation at the heart of this entire research space: a standard Linux
I/O scheduler — whether it's mq-deadline, bfq, or none —
sees all four of these as undifferentiated block reads and writes. It optimises for throughput,
for fairness, for latency in aggregate. But it has absolutely no idea that the read in position
47 of its dispatch queue is gating a user staring at a blinking cursor, while the write in
position 12 is a leisurely background prefill that can wait another 50 milliseconds without
anyone noticing.
The Access Pattern Fingerprint
KV-cache I/O has a very specific signature that distinguishes it from ordinary workloads. Research characterising actual I/O traces from DeepSpeed and FlexGen found that KV cache offloading is dominated by 128 KiB read and write requests — far larger than the 4K random I/O of databases, and distinct from the sequential streaming pattern of video or backup. It's large-block, partially-random (offset by session, layer, and head), with a strong write-once/read-many temporal skew: data gets written once during prefill and potentially read dozens of times during a long decode session.
This access pattern has an important implication: the workload is inherently asymmetric, and optimising for aggregate throughput is the wrong objective function. What matters is tail latency for a specific class of reads under pressure from a different class of writes. That's a fundamentally different optimisation target than anything the generic block layer was designed to serve.
Why This Matters Now
Three trends are colliding to make this acutely relevant in 2025–2026. First, context windows are expanding aggressively — from 8K tokens two years ago to 128K, 200K, and even 1M tokens today — which scales KV cache size superlinearly. Second, the economics of inference are pushing operators toward higher GPU utilisation and lower memory reservation per request, which means more KV cache spills to SSD rather than holding everything in HBM or DRAM. Third, the expectation of low-latency, streaming token generation has become a baseline user expectation — a 300ms TTFT (time to first token) is acceptable; a 2-second stall mid-generation while a stale prefill write clogs the scheduler is not.
How Everyone Else
Is Attacking This Problem
Kairo is not operating in a vacuum. The KV-cache storage problem has become one of the most active areas in systems research in the past 18 months, and a rich ecosystem of approaches has emerged — all attacking the same fundamental bottleneck from different angles. Understanding this landscape is essential to appreciating where Kairo's bet is unconventional.
The overwhelming consensus in current research is to solve this problem above the kernel — in inference frameworks, in GPU-side software, in user-space scheduling layers. The kernel block scheduler is largely treated as a fixed, dumb substrate to be routed around rather than modified. Kairo's thesis is that this consensus may be leaving significant gains on the table.
Notice what all of the above share: they operate at the inference framework, GPU driver, hardware, or bypass-the-kernel level. None of them ask the question Kairo is asking: what if the Linux block I/O scheduler itself understood the semantic difference between a decode read and a prefill write?
Kairo: Kernel AI Runtime I/O
Kairo — Kernel AI Runtime I/O — is a Linux kernel RFC/POC series authored by
Manish Klach, exploring a narrowly defined but consequential hypothesis: that block-layer changes
to mq-deadline and blk-mq can meaningfully improve generic NVMe SSD
behaviour for AI KV-cache workloads, specifically by prioritising decode reads over prefill
writes and eviction discards at the scheduler level.
It is not (yet) an LKML submission. It is not a production system. It is an honest, structured research scaffold — nine kernel patches ranging from implemented to scaffolded, a pthread-based C benchmark harness, and an unusually candid set of status-tracking documents that tell you exactly how much of it has been validated. The ambition is real; the current validation state is not yet complete. That honesty is itself notable.
The Architecture
Kairo's design is a four-layer stack, from AI runtime at the top to generic NVMe hardware at the bottom. Each layer has a specific role in the decode-priority pipeline:
The critical design decision is that Kairo carries the semantic classification all the way down
from the AI runtime to the NVMe command queue, rather than letting it dissolve at the user-space
boundary. A decode read enters as an RT prio 0 request, gets classified as
KAIRO_DECODE_READ in the hint path, and is treated as decode-critical through
the scheduler dispatch. A prefill write enters as a BE prio 7 request, is
demoted accordingly, and does not compete on equal terms with the decode read.
The Benchmark Harness
The benchmark — bench/kairo_bench.c — is a pthread-based C program that
synthesises the four I/O classes using standard Linux mechanisms, without requiring a real
inference framework. It uses pread() and pwrite() with
O_DIRECT, aligned buffers via posix_memalign(), and per-thread
ioprio_set() classification:
/* ioprio → KAIRO request class mapping */
RT prio 0 read → KAIRO_DECODE_READ /* user-facing, latency critical */
RT prio 1 read → KAIRO_PREFETCH_READ /* speculative, high priority */
BE prio 7 write → KAIRO_PREFILL_WRITE /* background, latency tolerant */
discard → KAIRO_EVICT /* lowest priority, deferrable */
The primary success metric is decode_p99_us — the 99th percentile latency
of decode reads, measured under simultaneous mixed-workload pressure from all four I/O
classes. This is the right metric: it directly models what a user experiences when the
decode path stalls. Secondary metrics include decode_p95_us,
decode_avg_us, decode_read_MBps, write throughput, and
starvation behaviour (whether prefill writes get permanently starved under decode pressure).
The Nine-Patch
Kernel Series
Kairo's kernel work is structured as nine RFC patches against Linux 6.8.x — a series that spans from concrete, implemented changes to scaffolded designs awaiting full kernel integration. Understanding the distinction between the two is important: the implemented patches have actual kernel code changes; the scaffolded patches define the interface and semantics of a change without yet fully plumbing it through all necessary kernel subsystems.
The Unvalidated Middle
The honest picture — and Kairo's own status docs make this plain — is that the sysfs counters
(patch 0009), the io_uring hint path (patch 0003), and the full benchmark run-to-results
pipeline are not yet validated on real hardware. The tested_kernel_matrix.md
shows boot tests, sysfs counter visibility, and benchmark runs still marked as pending for
most patches. This doesn't undermine the design — it accurately reflects where a solo
research project sits at v0.1.0. What it does mean is that the most important next experiment
is running run_ab_experiment.sh on a real NVMe drive with the patched kernel
and committing actual decode_p99_us numbers into the results/
directory.
The Kernel Bet:
Is This the Right Layer?
This is the question worth sitting with. Every major production approach — Tutti, LMCache, Dual-Blade, vLLM — has converged on solving the KV-cache I/O problem above the kernel, or by routing around the kernel scheduler entirely. Why does Kairo believe the block scheduler is worth patching?
The argument has several threads:
Thread 1: Semantic Loss at the Kernel Boundary
When an inference framework like vLLM makes a storage decision — "this is a decode read,
this is a prefill write" — that semantic information is lost the moment it crosses into the
kernel via a standard read() or even an io_uring submission without
hint plumbing. The kernel scheduler sees two reads; it cannot know that one is user-facing
and one is speculative. The only current mechanism to carry priority information into the
kernel is ioprio, which is a blunt instrument designed for process-level
fairness, not workload-class semantics.
Kairo's argument: if you carry the semantic classification all the way to the scheduler, the scheduler can make better decisions. The alternative — bypassing the scheduler entirely via NVMe-direct paths — trades kernel-managed safety and portability for performance, and it's not available in every deployment environment.
Thread 2: The Portability Advantage
Tutti requires GPU io_uring and specific hardware capabilities. Dual-Blade requires NVMe-direct access and specific FTL assumptions. Purpose-built inference SSDs require specific hardware procurement. Kairo's target is explicitly generic NVMe SSDs — the drives you already have in your inference server. If a kernel patch series can improve KV-cache decode latency on commodity NVMe without requiring hardware changes, the deployment surface is dramatically larger.
| Approach | Layer | Hardware Req. | Kernel Changes | Portability |
|---|---|---|---|---|
| Tutti | GPU storage stack | GPU direct capable | GPU io_uring (new) | Limited |
| KVDrive | Inference framework | None | None | High |
| Dual-Blade | User-space NVMe-direct | NVMe namespace access | None (bypasses kernel) | Medium |
| vLLM/LMCache | Inference framework | None | None | High |
| Inference SSDs | Hardware firmware | Specific SSD models | None | Low (specialised) |
| Kairo | Linux block layer | Generic NVMe | mq-deadline + blk-mq | Broad (if merged) |
Thread 3: The Composability Argument
A kernel-level solution doesn't compete with framework-level solutions — it composes with them. If vLLM or LMCache is already making smart tiering decisions, a Kairo-patched kernel would still improve the latency of the NVMe reads those frameworks generate, regardless of which framework is above it. A scheduler that understands decode priority helps whether the caller is a research framework or a production system.
The Counterarguments
Fairness requires acknowledging the counterarguments, because they're real:
Kernel patch complexity is high. A scheduler change that goes wrong can cause system-wide I/O starvation, priority inversion, or worse. The Linux block layer maintainers have extremely high standards for correctness, and any change touching mq-deadline dispatch logic needs exhaustive testing across workload types, not just the AI inference case.
ioprio is limited. The current hint mechanism (ioprio + the benchmark's RT priority classification) is a proxy for the richer semantic classification Kairo wants. Until io_uring hint plumbing (patch 0003) is fully implemented and the classification (patch 0002) is validated end-to-end, the system is not yet carrying the right information to the scheduler.
The problem may be solvable in user-space. If Tutti can achieve DRAM-like performance by going around the kernel entirely, and KVDrive can achieve good results through layout and tiering alone, the marginal value of a kernel patch over a well-optimised user-space solution needs to be demonstrated with numbers — which Kairo doesn't yet have.
An Honest Assessment
of Where Kairo Stands
At v0.1.0, released June 2026, Kairo is a documented plan with a working benchmark harness and two implemented kernel patches. That's not a slight — it's an accurate description of what a research scaffold at this stage looks like. The important question is how to read it.
What Is Real
The C benchmark harness in bench/kairo_bench.c compiles and runs. The ioprio
classification and per-thread worker model are correct. The two implemented patches
(0001 mq-deadline decode priority and 0004 large-block coalescing)
have real kernel code changes that should apply cleanly to Linux 6.8.x. The architecture
diagram is well-considered and matches the patch series structure. The primary metric
(decode_p99_us) is the right thing to measure.
What Is Not Yet Validated
The results/ directory is empty. The tested_kernel_matrix.md shows
boot tests, sysfs counter visibility, and benchmark runs as pending for most patches. The
sysfs debug interface (patch 0009) — the essential tool for confirming the kernel path is
actually exercised during a benchmark run — is scaffolded but not validated. Without those
sysfs counters moving, you cannot know whether a given benchmark result reflects the Kairo
scheduler path or the unpatched baseline.
What Makes It Notable Anyway
Independently of validation status, the framing of Kairo's hypothesis is
underexplored in the literature. Nobody else is trying to put KV-cache priority semantics
inside mq-deadline. The closest thing in the academic literature (Dual-Blade)
explicitly frames the kernel scheduler as an obstacle to route around. The closest thing
in the kernel itself is ioprio, which is process-granularity fairness, not workload-class
scheduling. The design space Kairo is probing — what does it look like if the scheduler
knows about AI inference? — is genuinely unexplored at the implementation level.
docs/tested_kernel_matrix.md and
docs/full_architecture_status.md. The explicit implemented vs
scaffolded labelling, and the pending rows in the validation matrix, are more
credible than the typical research repo that presents scaffolded code as working results.
That intellectual honesty is a feature, not a limitation.
What Comes Next:
The Validation Imperative
The path from "documented research scaffold" to "validated POC" requires one specific sequence of experiments, in order. Not more patches, not more architecture documentation — actual numbers from actual hardware.
First: boot a patched 6.8.x kernel with patches 0001 + 0002 + 0009 applied and confirm
the sysfs counters are visible under /sys/block/nvme0n1/kairo/. Without this
step, no subsequent benchmark result is interpretable.
Second: run run_ab_experiment.sh with the baseline kernel and then with the
patched kernel, on the same NVMe device, with the same workload mix. Record
decode_p99_us in both runs. Commit the output to results/.
Third: check for write starvation — that prefill writes under sustained decode pressure are merely demoted, not permanently starved. This is the correctness gate that any LKML reviewer would scrutinise first.
If those three steps produce a statistically meaningful reduction in decode_p99_us
without starvation, Kairo has made its argument. If the numbers are flat, that's also
a research result — it would suggest the bottleneck isn't the scheduler at all, and
the field's consensus about bypassing the kernel may be empirically correct.
Either way, the experiment is worth running. Which is what Part II of this series will examine: the full validation methodology, what the numbers should look like if the hypothesis is correct, and how to interpret them when they arrive.