KTransformersone model, two machines

A dense model uses nearly every parameter for every token. An MoE model may own hundreds of expert feed-forward networks but route each token through only a small top-k subset. Total weight capacity is enormous; active computation is much smaller. That gap creates an opportunity unavailable to a similarly sized dense model.

KTransformers keeps latency-sensitive, repeatedly reused operators—attention, routing, normalization, dense/shared paths, and often the KV cache—on GPU. The much larger routed-expert pool lives in CPU DRAM and is computed by CPU kernels. The system transfers compact activations and partial results across the device boundary rather than loading an expert’s weights into VRAM every time it is selected.

The result is not GPU-class speed from cheap RAM. It is a deliberate trade: accept slower expert computation and sophisticated synchronization so a workstation can execute a model whose total weights would otherwise require a multi-GPU server.

The paper’s machine measured 220 GB/s within a socket and 125 GB/s across sockets. A NUMA-oblivious DeepSeek-V3 layer took 6.9 ms on one socket and only improved to 5.8 ms on both—a weak 16% gain because remote access consumed the theoretical parallelism.

KTransformers uses NUMA-aware tensor parallelism: every expert matrix is sliced across sockets rather than assigning whole experts to nodes. Each socket stores and computes its local slice; a lightweight reduction combines partial outputs. This balances work even when the router selects an uneven set of experts and avoids almost all remote weight traffic.

The reported breakdown attributes up to 1.63× decode improvement and up to 1.22× prefill improvement to NUMA-aware tensor parallelism. Decode benefits more because it is particularly memory-bandwidth-bound.

Short prefill stays heterogeneous: CPU experts and GPU operators cooperate. Beyond a configurable token threshold, layerwise prefill chunks the prompt, stages one MoE layer’s full working expert set into GPU memory, runs that layer for the chunk, then proceeds layer by layer.

The pipeline overlaps weight-format conversion into pinned memory, PCIe transfer, and GPU repacking at expert granularity. Double buffering and DDIO-aware adjacency aim to keep converted expert chunks hot in last-level cache so PCIe reads do not force a full DRAM round trip.

This deliberately reverses the normal computation-offload choice because a long token batch gives the GPU enough reuse to justify moving weights. The bottleneck moves from CPU expert compute toward PCIe bandwidth and GPU math. Extra VRAM includes roughly one full MoE layer working set plus temporary buffers that grow with chunk size.

Current documented inference methods include native BF16, FP8, per-channel FP8, RAWINT4, GPTQ INT4, converted AMX INT8/INT4, LLAMAFILE/GGUF, and a model-specific MXFP4 path. A checkpoint labeled INT4 may still be incompatible if its packing, scale granularity, expert layout, or CPU kernel expectation differs.

Native-format paths can let CPU and GPU share one semantic weight representation; converted AMX paths preprocess expert weights for the tile-oriented CPU backend. Quantization reduces DRAM capacity and bandwidth pressure but introduces format conversion, accuracy, and kernel-support questions.

The support matrix itself uses statuses such as Current, Needs smoke, Legacy, Historical, or narrow. That is healthy documentation: rapid model support is not the same as broad production validation. Reproduce the exact tuple before trusting a throughput number.

The GPU handles attention, shared paths, and trainable adapter capacity; CPU AMX backends execute the frozen or mostly frozen expert base using BF16, INT8, or INT4 prepared weights. This reduces the VRAM cliff enough to make workstation-scale adapter training conceivable.

Training is not simply inference with gradients. Activations, adapter gradients, optimizer state, checkpointing, data pipeline, and backward communication add memory and bandwidth. KTransformers v0.6.1 reports 6–12× performance over a ZeRO-Offload baseline and roughly half the CPU memory of the previous KT SFT path in its benchmark settings, but those results are tied to exact models, context, LoRA configuration, hardware, and baseline.

The current docs explicitly mark old Kimi SFT and DPO pages as historical or unconfirmed. The supported public path is LLaMA-Factory YAML plus use_kt: true, an Accelerate KT configuration, and the ktransformers[sft] package.

DRAM supplies capacity for routed expert weights. CPU matrix extensions and cache-aware layouts turn selected experts into useful throughput. GPU HBM and tensor cores retain the attention path, KV cache, shared experts, and dense operators. NUMA-aware slicing keeps each socket local. CUDA Graph callbacks reduce control overhead. Optional Expert Deferral changes dependency timing to expose still more parallelism.

The design works because sparse MoE models separate capacity from active computation. It is most compelling for low-concurrency local or private inference where owning the entire model matters more than matching an all-GPU cluster’s latency.

It is also a reminder that “can run” and “runs well” are different claims. A few hundred gigabytes of RAM may satisfy capacity while insufficient memory channels, weak CPU ISA support, poor NUMA placement, or a slow PCIe path makes the experience impractical. KTransformers raises the ceiling; the whole workstation determines where you land.