A look inside ExpertMesh, an open‑source project that trades a proprietary 64‑GPU scale‑up domain for refurbished AMD cards, a fixed home for every expert, and a data plane honest enough to say what it hasn't proven yet.
Serving a trillion‑parameter sparse mixture‑of‑experts model usually means one of two expensive answers: buy an enormous HBM supernode, or stream terabytes of weights across CXL fast enough to keep GPUs fed. ExpertMesh asks a third question — what if each quantized expert just stayed put on one inexpensive card, permanently, and the network only had to carry the small activations routed to it?
That single reframing is the whole architecture. Experts are stationary. The network moves tokens, not tensors. And the hardware underneath is deliberately unglamorous: refurbished AMD Instinct MI210 cards and used EPYC servers, not a vendor's flagship scale‑up rack.
Every decode step follows the same rail. A continuous‑batching scheduler feeds attention and a top‑k router, which hands routed activations to a dispatcher. That dispatcher checks who owns each expert: local traffic rides the in‑node Infinity Fabric hive, remote traffic goes out over RoCE. Either way it lands on a stationary, packed‑INT4 expert, and the results come back through a weighted top‑k reduction.
Local xGMI is preferred over the remote RoCE fabric wherever routing allows it — the scheduling layer is explicitly topology‑aware, and hot experts get replicated only after routing traces demonstrate real skew, not by default.
The Phase 1 reference system is a concrete, priced bill of materials, not a hand‑wave. Eight refurbished four‑GPU EPYC nodes, thirty‑two MI210 accelerators, dual‑port 100/200 GbE RoCE per node.
| Compute nodes | 8 × refurbished 4‑GPU EPYC systems |
| Accelerators | 32 × AMD Instinct MI210, 64 GB |
| Aggregate HBM | 2 TiB |
| Aggregate HBM bandwidth | 51.2 TB/s theoretical |
| Inter‑node fabric | Dual 100/200 GbE RoCE per node |
| GPU board power | 9.6 kW |
| Active‑weight planning range | 30–50 GB/token |
| Estimated acquisition cost | $135k – $235k |
| Bandwidth roofline | 1,024 – 1,707 aggregate token/s |
The example sizing target is Kimi K3 — roughly 2.8T parameters, 896 experts, top‑16 routing, packed 4‑bit weights — used as a demanding stress case for the planner, not a claim of compatibility. That distinction matters more than it sounds: this codebase is built to be told which model it's sizing for, rather than assuming one.
The most unusual thing about ExpertMesh isn't the architecture — plenty of MoE serving projects propose expert placement schemes. It's that the project separates, in public, what's been verified from what's still an assumption, and refuses to let one borrow credibility from the other.
expertmesh-plan and expertmesh-roofline CLIsPerformance claims are listed as "intentionally withheld until measured." That's the line that separates this from most trillion‑parameter‑anything repos on GitHub.
Rather than shipping a model integration and hoping it holds up, ExpertMesh defines a fixed sequence of release gates that any target checkpoint — Kimi K2, K3, or otherwise — has to clear in order, before the label "compatible" is allowed to appear anywhere.
Until every gate closes, the shipped config is labeled exactly for what it is: configs/phase1-mi210.json, an assumption profile.
This is the part most "hardware‑aware" AI infrastructure repos don't offer: a Phase 1 you can actually exercise on a laptop, end to end, planner through a live chat‑completions endpoint.
# plan and check the roofline math python -m venv .venv source .venv/bin/activate pip install -e ".[dev]" pytest -q expertmesh-plan configs/phase1-mi210.json expertmesh-roofline configs/phase1-mi210.json # spin up a tiny deterministic MoE end to end expertmesh-checkpoint .demo --workers 2 expertmesh-serve .demo --workers 2 # → POST /v1/chat/completions on :8000
The generated model here is a plumbing fixture, not a useful language model — it exists to prove the coordinator/worker path, checkpoint sharding, and serving loop actually connect end to end.
ExpertMesh is early — one contributor, ten commits, a single star as of this writing — and its central bet, that commodity AMD hardware plus a stationary‑expert data plane can approach trillion‑parameter serving at a fraction of the usual cost, is still unproven on real silicon. The HIP kernel needs a ROCm runner it hasn't had yet. No checkpoint has cleared the release gates.
But the engineering posture is exactly right for a project at this stage: a real reference implementation you can run today, a cost model you can inspect line by line, and a validation table that says "not measured" instead of guessing. That's worth more, long‑term, than a flashy benchmark nobody can reproduce.