Recap: What Kairo Is Trying To Solve
If you read Part 1, you know the setup. Kairo — Kernel AI Runtime I/O — is an internal Linux RFC/POC that asks a deceptively simple question: what if the block layer actually knew it was serving an LLM?
Today's Linux storage stack treats a decode read (the latency-critical I/O that feeds the next token to the GPU) identically to a prefill write (background cache population). Under mixed I/O pressure, decode p99 latency blows out and the model stalls — not because the SSD is slow, but because the scheduler doesn't know which request matters most.
AI inference creates four distinct I/O classes with wildly different latency sensitivity. Kairo exposes that structure to the Linux block layer so the scheduler can act on it.
The Four I/O Classes
Kairo's entire design stems from a taxonomy that most I/O schedulers ignore. KV-cache traffic is large-block, read-dominant, and session-scoped — but not all reads are created equal.
Feeds the active inference step. Every microsecond matters. Must dispatch ahead of everything else.
Anticipatory cache load. Deadline-sensitive but can yield briefly to decode without harming quality.
Background KV-cache population. Throughput-oriented. Demoted under read pressure.
Cache cleanup and punch-hole operations. Absolute lowest priority. Can wait.
In Kairo's user-space header, include/kairo_hints.h, these map
directly to ioprio classes that flow down into the block layer.
The current implementation uses IOPRIO_CLASS_RT at priorities 0
and 1 for reads, and IOPRIO_CLASS_BE at priority 7 for prefill
writes. Eviction rides the discard path or a BE prio 6 fallback.
The Four-Layer Stack
Kairo inserts a new conceptual layer between the AI runtime and the generic NVMe backend. Here's what that looks like:
┌─────────────────────────────────────────────────────┐ │ AI Runtime / Synthetic Benchmark │ │ · decode reads · prefetch reads │ │ · prefill writes · eviction / discard │ └─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐ │ User-Space Hint Path │ │ · io_uring + O_DIRECT + registered buffers │ │ · ioprio / model / session / lifetime hints │ │ · kairo_hints.h defines RWF_KAIRO_* flags │ └─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐ │ Kairo Block Layer │ │ · request classification (patch 0002 / 0010) │ │ · decode-critical fast path (patch 0001) │ │ · prefetch-aware scheduling (patch 0005) │ │ · prefill-write demotion (patch 0011) │ │ · large-block coalescing (patch 0004 / 0015) │ │ · placement / lifetime metadata (patch 0007) │ │ · BPF programmable dispatch (patch 0016) │ └─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐ │ Generic NVMe Backend │ │ · mq-deadline extensions │ │ · blk-mq metadata │ │ · optional ZNS / Streams / FDP mapping │ └─────────────────────────────────────────────────────┘
The beauty of this design is that each layer degrades gracefully. NVMe
Streams/FDP/ZNS hooks are feature-detected and fall back to no-ops on standard
SSDs. The ioprio signaling path works on any kernel running
mq-deadline.
The Patch Series, Explained
As of v0.1.0, Kairo carries 30 patches in kernel/patches/,
plus a foundation compile-targeted subset for Linux 6.8.x. Here's the full
picture, grouped by theme.
Foundation: Scheduling Core (Patches 0001–0005)
The primary patch. Modifies mq-deadline to dispatch
decode-classified reads ahead of all other traffic. This is the core
proof point: reduce decode_p99_us under mixed write pressure.
Introduces the five Kairo I/O classes and merge-instrumentation flags
in blk-mq headers. Patch 0010 hardens this with real
ioprio-to-class conversion at bio-to-request time, replacing the deferred
classification stub from 0002.
Propagates Kairo intent from user-space through kiocb
into conceptual bio/request metadata. Enables future per-IO classification
via IORING_SQE_KAIRO_CLASS (patch 0014).
KV-cache reads are large. Patch 0004 adds merge-bias helpers and
per-request merge flags. Patch 0015 fills in the real
kairo_attempt_forced_merge() implementation with safety
checks, bridging the scaffold to a working prototype.
Gives prefetch reads their own urgency deadline, distinct from both decode and write traffic. Prevents prefetch from being treated as best-effort while still yielding to decode.
Semantics & Placement (Patches 0006–0008)
These patches move Kairo beyond scheduling hints into richer semantic metadata.
| Patch | Subsystem | What It Does |
|---|---|---|
| 0006 | fs · mm · block | Ephemeral and recomputable semantics. KV-cache data that can be regenerated on eviction gets tagged as KAIRO_RWF_RECOMPUTE, enabling smarter eviction decisions downstream. |
| 0007 | blk-mq · blk_types | Model/session/lifetime metadata. Each request can carry model_id, session_id, cache_pool_id, and a lifetime_class (short / session / model / persistent). |
| 0008 / 7.5 | blk_types · NVMe host | Generic backend mapping scaffold. Introduces kairo_backend_class and kairo_backend_caps. Feature-detected hooks for NVMe Streams, FDP, and ZNS — all currently no-ops with safe fallback. |
Reliability: Anti-Starvation & Tag Reservation (Patches 0011–0012)
Prioritizing decode reads aggressively risks starving writes entirely. Patches 0011 and 0012 address this head-on.
Indefinite deferral of prefill writes under sustained decode pressure would
eventually deadlock inference. Patch 0011 adds a per-write expiry deadline
via the kairo_write_deadline_ms sysfs tunable. Patch 0012
reserves 1/8 of hardware queue tags specifically for decode reads, preventing
tag starvation upstream of the scheduler entirely.
Performance: O(1) Dispatch & BPF (Patches 0013, 0016)
The original mq-deadline FIFO scan under spinlock is O(n). Patch 0013 replaces
this with per-priority dedicated FIFO lists for decode and prefetch — O(1)
dispatch at any queue depth. Patch 0016 takes this further, adding a
BPF_PROG_TYPE_KAIRO_SCHED hook for fully programmable dispatch
arbitration, with additive fallback to static logic when no BPF program is loaded.
Advanced Features: Fairness, Admission, Heatmap (Patches 0020, 0029, 0030)
The later patches in the series push Kairo toward multi-tenant inference scenarios and smarter eviction policy.
Credit-based decode scheduling for multi-tenant AI inference. Each model and session gets a decode credit budget with periodic refill, noisy-session detection, and five sysfs tunables.
A blk-cgroup policy scaffold for AI inference containers, with per-class weights and per-entity stats. Cgroup interface files are documented with implementation deferred pending upstream feedback.
Introduces IORING_REGISTER_KAIRO_KV_REGION opcodes (42, 43)
so runtimes can register named memory regions with type, model/session IDs,
and lifetime class — flowing into the scheduler as richer placement hints.
An eviction-class model with a scoring policy that penalizes eviction of decode-hot, non-recomputable data. 10 sysfs eviction counters for observability.
A 1024-entry fixed-array heatmap tracking per-region access frequency and recency. Heat scoring uses 4 weights plus age decay. 9 sysfs counters.
A KV-cache admission scaffold deciding which objects merit flash storage. 9 admission decisions, 8 config fields, 7 priority-ordered policy rules.
The kairo_hints.h API
The user-space contract lives in include/kairo_hints.h — 263 lines
of carefully annotated C that defines the full Kairo hint surface. A few key
excerpts illustrate the design philosophy.
I/O Classification Flags
/* ioprio mapping (current implementation) */
#define KAIRO_CLASS_DECODE_READ 0 /* → IOPRIO_CLASS_RT, prio 0 */
#define KAIRO_CLASS_PREFETCH_READ 1 /* → IOPRIO_CLASS_RT, prio 1 */
#define KAIRO_CLASS_PREFILL_WRITE 2 /* → IOPRIO_CLASS_BE, prio 7 */
#define KAIRO_CLASS_EVICT 3 /* → discard / BE prio 6 fallback */
/* RWF flags for future io_uring path */
#define KAIRO_RWF_DECODE (1ULL << 28)
#define KAIRO_RWF_PREFETCH (1ULL << 29)
#define KAIRO_RWF_PREFILL (1ULL << 30)
#define KAIRO_RWF_RECOMPUTE (1ULL << 31)Placement Metadata
struct kairo_user_placement_hint {
uint32_t model_id;
uint32_t session_id;
uint32_t cache_pool_id;
uint32_t placement_group;
uint32_t lifetime_class; /* short / session / model / persistent */
uint32_t flags;
};KV Region Hints
struct kairo_user_kv_region_hint {
uint32_t region_id;
uint32_t region_type; /* decode / prefetch / session / model / recomputable */
uint32_t model_id;
uint32_t session_id;
uint64_t file_offset;
uint64_t length;
uint32_t lifetime_class;
uint32_t flags;
};
The header is deliberately annotated as a local RFC/POC — not a proposed stable
UAPI. This is important: Kairo is doing kernel-space research, not trying to land
patches upstream yet. The annotations like COMPILE-TARGET,
CONCEPTUAL-HOOK, and VERSION-SENSITIVE in the kernel
patches themselves make the maturity of each hook point explicit.
Observability Stack
A key design principle throughout Kairo is that every code path must be observable. Patches 0009 and 0017 build out a comprehensive observability layer.
Sysfs Counters (Patch 0009)
Patch 0009 adds debugfs counters that prove the Kairo code paths are actually
executing — dispatch counts, merge instrumentation, request-size histograms,
placement/lifetime counters, and backend mapping outcomes. The validation script
scripts/validate_kairo_runtime.sh uses these to confirm that
Kairo counters move under the synthetic KV-cache workload.
Kernel Tracepoints (Patch 0017)
Stage 8 adds 9 TRACE_EVENT definitions in
include/trace/events/kairo.h, covering the full request lifecycle:
classification, scheduler decisions, dispatch, demotion, merge, semantic flags,
placement metadata, and backend mapping. Companion bpftrace scripts
(scripts/bpftrace/kairo_*.bt) make these immediately useful for
profiling.
Adaptive Latency Controller (Patch 0018)
Patch 0018 closes the loop with an adaptive controller that observes
decode_p99_us in real time and adjusts decode/prefetch budgets
accordingly. Three modes: OFF, OBSERVE, and
ADAPTIVE. The decode latency histogram from patch 0023 feeds
this controller with 10 buckets from 0–10 µs to >5 ms.
The Benchmark Harness
Kairo's benchmark (bench/kairo_bench.c) is a pthread-based
synthetic workload designed to mimic KV-cache I/O patterns on a real NVMe
device. It uses pread() and pwrite() with
O_DIRECT, aligned buffers via posix_memalign(),
and per-thread ioprio classification.
Build and run a full A/B experiment comparing baseline vs Kairo in three commands:
# Build
gcc -O2 -Wall -pthread -Iinclude -o kairo_bench bench/kairo_bench.c
# Baseline
./scripts/run_baseline.sh /mnt/nvme/kairo.test nvme0n1
# Kairo POC
./scripts/set_mq_deadline.sh nvme0n1
./scripts/run_kairo_poc.sh /mnt/nvme/kairo.test nvme0n1
The A/B runner (scripts/run_ab_experiment.sh) compares both on
the same device and saves structured results. A multi-session runner
(scripts/run_multisession_experiment.sh) stresses model/session
fan-out scenarios.
What Kairo Measures
The primary validation metric is decode_p99_us — the 99th-percentile
decode read latency under mixed prefill-write pressure. The constraint is equally
important: write throughput must not collapse.
(primary target)
(secondary)
(baseline ref)
(throughput)
(must not collapse)
What Remains Open
Kairo's documentation is refreshingly honest about what hasn't been settled.
The docs/patch_series.md "What Remains Open" section lists the
genuine design questions — not marketing hedges:
Does merge bias belong in generic blk-merge or only in
Kairo-classified paths? Should RWF_KAIRO_* stay RFC-only?
Does the generic kairo_backend_class abstraction map usefully
to real NVMe Streams/FDP/ZNS capabilities? Does physical placement via
backend hooks give measurable improvement over software-only grouping?
These are unanswered — and Kairo's benchmark harness is exactly how they
will be answered.
What's Next for Kairo
The validation roadmap is tracked in docs/tested_kernel_matrix.md
and docs/full_architecture_status.md. The near-term focus is on
three things: proving patch 0001 builds and applies cleanly on Linux 6.8.x,
validating that decode_p99_us improves measurably in the synthetic
A/B experiment, and wiring real NVMe feature detection into the backend mapping
scaffold.
The longer arc — blk-cgroup AI policy, programmable BPF dispatch, KV admission control — builds the case for a fundamentally smarter storage substrate for inference infrastructure. Whether any of this lands upstream is an open question that depends on the benchmark data making a compelling argument.
Storage is the forgotten bottleneck in AI inference. CPUs, GPUs, and memory get all the attention. Kairo is betting that a relatively small number of targeted kernel patches can meaningfully reduce decode tail latency — on commodity NVMe SSDs that every inference deployment already has.