Flash is capacity.
DRAM is prediction.
SRAM is latency.

The Idea

Treat inference memory access as a schedule, not a guess

Most discussions of "cheap memory for AI inference" start and end with SSD caching — swap in flash, accept the latency hit, move on. This repo argues that's the wrong framing entirely for transformer inference specifically, because transformer inference isn't a random-access workload the way a database or a game engine is.

Layer execution order is fixed. Weight access can be linearized into predictable tile sequences. Even KV cache access, while not perfectly uniform, has locality and "temperature" — recently used tokens matter more than old ones. MoE routing history can hint at which experts get called next. None of this is a novel observation on its own, but the repo's contribution is organizing an entire memory architecture around exploiting it: a compiler that schedules based on traced access patterns, a runtime that predicts and prefetches ahead of compute, and a strict rule that flash is never allowed to sit on the synchronous decode path.

A synchronous flash read during token generation isn't treated as an ordinary cache miss — it's treated as a scheduling failure the system should have prevented.

Architecture

Three tiers, one direction of hiding

The fabric is a strict hierarchy. Each tier has a distinct job, and — this is the part that matters — a distinct relationship to time: SRAM is where compute happens right now, DRAM is where the system stages what it predicts compute will need soon, and flash is where everything else waits, streamed in bulk, never on demand.

The reframing of the SSD access pattern is the whole trick: instead of random synchronous reads triggered by a compute stall, the system does large sequential asynchronous prefetches into DRAM ahead of time, based on trace-guided prediction. The visible path compute sees is short: SRAM → compute → output token. Everything involving the SSD happens off to the side, hidden behind however much lookahead the predictor can buy.

What's Actually In The Repo

More scaffolding than most solo research repos ever build

What separates this from a README-only concept pitch is that it ships a runnable simulator path alongside the design docs — run_sim.py, DRAM capacity sweeps, lookahead sweeps, and a deployment economics model, each with checked-in CSV/JSON results. The documentation set underneath it is unusually deep for a project at this stage:

The Non-Goals section is worth calling out specifically: it explicitly disclaims that SSD latency equals DRAM latency, that every workload can hit HBM-class performance, and that prediction misses are free. That kind of self-limiting honesty is rare, and it's a good sign for how the rest of the project is likely to be reasoned about.

Assessment

Where it stands today

Path To A 10

What would close the gap

1. Add a real Quickstart. Install command, one script invocation, and a pasted sample output block, placed before the architecture diagram.
2. Surface one headline number in the README. Pull the best result out of the DRAM capacity sweep or deployment economics model and put it in a table or chart, not just a link.
3. Add a Related Work section. Position this explicitly against vLLM/PagedAttention, FlexGen, and ZeRO-Infinity-style offloading — say in one line each what's actually different.
4. Wire up CI. A GitHub Actions workflow running the test suite and a simulator smoke test, plus build/license/version badges at the top of the README.
5. Render the real diagrams. The diagrams/ folder exists — use its output in the README instead of ASCII boxes.
6. Add a maturity badge. One line near the top: "research prototype — simulator only, no hardware backend yet" — sets expectations before the vision sections do.
7. Add CONTRIBUTING.md and issue templates. Lower the barrier for the first outside contributor to show up.
8. Rebalance the README. Move commercialization and big-vision framing below the runnable evidence so the engineering leads.