Search essays
Current essays
· 23 min read
Chunked Prefill: Ending the Great Stall
How chunked prefill killed the decode stall: Sarathi's token-budget scheduler, stall-free batching, and the arithmetic of mixing prefills with decodes in one batch.
· 24 min read
PagedAttention, Internals First: Blocks, Tables, and Copy-on-Write
A bottom-up tour of PagedAttention: block tables, the fragmentation math that motivated them, copy-on-write sharing across beams and requests, and why block size 16 isn't arbitrary.
· 21 min read
TP vs PP vs EP vs DP: The Communication Math That Picks Your Parallelism
Four ways to split a model across GPUs, each with its own communication tax. The all-reduce bytes, pipeline bubbles, and all-to-all volumes — worked through for a 70B model on 8 GPUs.
· 24 min read
Quantization for Serving: A Roofline View of FP8, FP4, and AWQ
Weight-only vs weight+activation quantization through the roofline lens: what AWQ, FP8, and NVFP4 each buy you in bytes-per-token, and where the accuracy cliffs actually are.
· 22 min read
The CPU Is the Bottleneck: CUDA Graphs, torch.compile, and the Overhead Wall
At small batch sizes the GPU starves waiting on the CPU. Kernel launch overhead, Python scheduling, and graph capture — the full anatomy of the overhead wall and how serving stacks climb it.
· 23 min read
The Radix Tree That Serves the Fleet: Prefix Caching at Scale
System prompts, few-shot examples, and multi-turn chats repeat the same tokens millions of times. RadixAttention turns that redundancy into free prefill — here's the tree, the router, and the hit-rate math.
· 24 min read
Fitting a Million Tokens: KV Compression, Eviction, and Offload
A million tokens of KV cache doesn't fit in HBM — so you compress it, evict it, or move it. KIVI quantization, attention sinks, and the prefetch pipelines that make offload actually work.
· 25 min read
Grammars at Token Speed: How Constrained Decoding Really Works
JSON mode isn't magic — it's a pushdown automaton riding shotgun on every decode step. Token masks, grammar compilation, and XGrammar's trick for making it nearly free.
· 23 min read
The $/MToken Model: Inference Economics from First Principles
What does a million tokens actually cost to serve? GPU amortization, power, utilization math, and why the providers charging $0.10/MToken aren't losing money — a full cost model.
· 24 min read
Cold Starts: The First-Token Problem Nobody Benchmarks
Before the first token, 140GB of weights has to cross PCIe. Safetensors mmap, sharded loading, RDMA broadcast, and graph capture — the full anatomy of a cold start and how to kill it.
· 14 min read
The Enterprise Sovereign AI Stack
Why durable enterprise AI comes from combining open weights, proprietary data, local compute, and a governed runtime into one controllable system.
· 19 min read
The Reward Is Code. The Experiment Is a System.
A technical walkthrough of zero-SFT GRPO on Countdown: exact rewards, leakage-resistant evaluation, pass@k, the Qwen text-loader fix, and real trainer acceptance tests.
· 15 min read
Three Loops, Three Different Kinds of Progress
A detailed guide to looped transformers, RL, and recursive self-improvement: what changes, what persists, how tools evolve, and how to measure real progress.
· 9 min read
Eight GPUs, Two Workloads, One Fair Scheduler
The hard part is not allocating GPUs. It is knowing when to stop: admission gates, drain phases, gang scheduling, and a working starvation-free implementation.
· 24 min read
Prompts → Agents → Harnesses → Loops → Graphs
The model proposes. The harness governs. The loop learns. The graph scales—a detailed systems guide to agentic inference and safely self-improving runtimes.
· 18 min read
Inference Engineering Is the Art of Keeping Tokens Moving
From acquiring GPUs to serving engines, replicas, cache-aware routing, shared KV tiers, and gateways across heterogeneous compute—the complete production inference control loop.
· 14 min read
The Millisecond Is a Systems Problem
Inside WarpStream TRT: how fixed memory, fused CUDA preprocessing, lane-specific TensorRT contexts, CUDA Graph replay, and device-side Top-K turn small-model inference into an auditable systems pipeline.
· 9 min read
The NVMe Trick, the 85% Margin, and What's Actually Verified
A technical explainer on DeepSeek’s SSD-backed KV caching, the public 3FS architecture underneath it, and why the viral 85% margin claim is much shakier than the engineering story.
· 10 min read
Stationary Experts
A technical review of ExpertMesh, where stationary INT4 experts on cheap AMD hardware challenge the default trillion-parameter serving playbook with unusual validation discipline.
· 10 min read
K3 Mini
Why a real K3 Mini would be a smaller KDA-style student distilled from a 2.8T teacher, not an absurdly compressed giant model, and where the actual product opportunity lives.
· 9 min read
Could Kimi Linear Run on a CPU?
Why Kimi Linear is unusually plausible as a CPU-only model after quantization, and why the real blocker is still the missing runtime stack for KDA, MLA, and MoE kernels.
· 8 min read
Kernel Families
A systems explainer on why reusable kernels are really reusable families that track architectural compatibility, not one universal binary that accelerates every model.
· 9 min read
The Cloning Problem
What open weights actually give away: not just cheaper inference, but a pretrained asset that can be quantized, fine-tuned, distilled, and reverse-engineered into entire families of competitors.
· 24 min read
Inside NVIDIA Vera Rubin
The complete hardware supply chain and investment map behind NVIDIA’s next AI platform: HBM4, foundry, optics, power, racks, CPUs and networking—plus consensus growth, prices and forward valuation.
· 20 min read
Building a CPU Server for Kimi K3
A practical blueprint for making a 2.8T MoE model local: weight-residency math, 3 TB of ECC DRAM, 24 memory channels, AMX and AVX-512, NVMe, NUMA, optional GPU offload, power, cost, and a prudent post-release buying gate.
· 18 min read
Kimi Linear, Decoded
An equation-first explanation of how KDA ages memory, erases a wrong prediction, writes the delta correction, retrieves with a query, and combines three recurrent layers with one global MLA layer.
· 24 min read
Open Weights to Tokens per Second
What open weights contain, how inference turns tensors into tokens, which hardware controls performance, and how quantization, packing, placement, caching, batching, and kernels optimize the serving path.
· 22 min read
KTransformers
How giant sparse models split across GPU and CPU: computation offload, AMX/AVX expert kernels, CUDA Graph coordination, NUMA, Expert Deferral, layerwise prefill, precision, and hardware sizing.
· 19 min read
MuonClip Optimizer
How Muon’s Newton–Schulz-orthogonalized momentum improves token efficiency, why attention logits exploded at scale, and how per-head QK-Clip stabilized Kimi K2’s 15.5T-token run.
· 17 min read
Logits — From Model Scores to Tokens
How raw vocabulary scores become cross-entropy gradients, sampled tokens, constrained outputs, calibration traps, and a distributed systems problem at large vocabulary scale.
· 17 min read
Attention Residuals
How AttnRes replaces fixed residual accumulation with learned retrieval across depth—and how Block AttnRes makes the state, pipeline communication, and inference overhead practical.
· 15 min read
The Model’s View of Memory
A model-centric field guide to GPU memory, HBM, CPU DRAM, and NVMe—and how weights, KV cache, prefill, decode, and offloading turn capacity into data movement.
· 14 min read
Kimi Delta Attention — A KDA Deep Dive
A model-to-kernel explanation of KDA’s delta-rule recurrent memory, channel-wise decay, hybrid KDA/MLA design, KV-cache payoff, prefix reuse, kernel paths, and tradeoffs.
· 17 min read
Inside Kimi K3’s Architectural Stack
A systems analysis of KDA, Attention Residuals, Stable LatentMoE, quantile-balanced routing, and how their combined payoffs support sparse 2.8T-scale intelligence.
· 17 min read
WideEP for Kimi K3
How 896 resident experts turn inference into a placement, token-dispatch, collective-communication, load-balance, latency, and failure-domain problem.
· 10 min read
MXFP4 Quantization
A rigorous guide to the open microscaling format: E2M1 elements, E8M0 block scales, W4A4 economics, quantization error, Kimi K3’s use, and comparisons with INT4, FP8, BF16, and NVFP4.
· 18 min read
The Kimi K3 Paradox
A research-backed analysis of how KDA can shrink KV-cache pressure while K3's 2.8-trillion-parameter sparse model increases the value of rack-scale HBM, accelerator, and networking capacity.
· 11 min read
The Memory Mirage
An architectural explainer on what people usually mean when they say HBM “isn’t needed”: training still wants premium memory, while long-run inference may drift toward a small hot tier backed by larger, cheaper capacity.
· 10 min read
Before It Becomes RTL: Inside OpenFlash Controller Lab
A systems essay on OpenFlash Controller Lab, where SSD controller behavior is validated in seeded simulation and Linux/QEMU plumbing before firmware and RTL work begins.
· 10 min read
The Scheduler That Has to Earn Its Keep
Notes on rl-inference-scheduler, where a learned policy has to honestly beat FIFO, SJF, and priority queues on the messy tradeoffs inside LLM serving.
· 12 min read
Flash Is Capacity. DRAM Is Prediction. SRAM Is Latency.
A review of flash-sram-dram-inference-fabric — a solo research project betting that AI inference memory access patterns are predictable enough to turn cheap NVMe flash into a hidden capacity tier.
· 18 min read
LPDDR & HBM — The Complete Signal-to-Silicon Teardown
A deep technical teardown of LPDDR and HBM, from signaling and PHY training to TSV stacks, interposers, HBM4 packaging, and the memory roadmap shaping AI systems.
· 9 min read
The Compute-to-I/O Area Ratio
A systems essay on why MAC arrays often occupy less die area than the DDR PHYs and high-speed SerDes blocks required to feed them.
· 11 min read
The Blackwell Architectural Split — NVFP4 Attention on Consumer Silicon
A deep dive into the SM100 vs SM12x split in Blackwell, TMEM removal on consumer GPUs, NVFP4 attention bottlenecks, and the open-source workaround path.
· 13 min read
CPUOpt-Kernel: The Linux CPU Tool That Actually Thinks — Deep Dive
A deep dive into CPUOpt-Kernel's capability maps, preview-and-rollback model, workload-aware policy engine, and the systems philosophy behind safe Linux CPU control.
· 16 min read
CPUOpt-Kernel: Architecture-Aware CPU Performance for Linux
A systems essay on CPUOpt-Kernel, mapping Linux CPU policy across Intel HWP, AMD CPPC, ARM SCMI, thermal telemetry, and workload-aware performance control.
· 14 min read
Kairo — Part 2: Inside the Kernel Patch Stack
Part 2 of the Kairo series, walking through the kernel patch stack, user-space hint surface, observability path, benchmark harness, and validation strategy for KV-cache-aware NVMe scheduling.
· 16 min read
Model Distillation — A Deep Dive
A deep technical explainer on knowledge distillation, from temperature-scaled soft targets and loss design to feature matching, deployment tradeoffs, and modern LLM distillation workflows.
· 15 min read
Kairo: Patching blk-mq for KV-Cache-Aware NVMe Scheduling — Part I
Part I of a technical essay on Kairo, a kernel RFC/POC exploring KV-cache-aware NVMe scheduling inside blk-mq and mq-deadline rather than routing around the kernel in user space.
· 14 min read
GLM-5.2: Why Long-Horizon AI Is Becoming a Systems Problem
A systems-level essay on GLM-5.2, from 1M-token serving and IndexShare sparse attention to speculative decode, RL anti-hacking, and long-horizon agentic workloads.
· 12 min read
No Kernel Ships Until It Can Prove It's Right
Field notes on mlx-metal-kernels: a correctness-first Apple Silicon kernel lab for MLX, Metal decode paths, quantized matvec work, and machine-local benchmarking discipline.
· 10 min read
Warp & Thread Groups: Field Notes on GPU Execution
A practical field guide to GPU warps, wavefronts, thread groups, divergence, and shared memory, with diagrams that map software threads to real execution lanes.
· 15 min read
Part 2: KV Cache Is Not Anonymous Memory
Part 2 of the KV scheduler series, expanding the kernel-facing architecture, trace adapters, and intent-aware scheduling path behind kv_deadline_scheduler.
· 17 min read
KV Cache Isn't Anonymous Memory
A systems essay on deadline-aware KV scheduling, the MemoryIntent schema, and why inference cache state cannot be managed like generic anonymous memory.
· 25 min read
Writing GPU Kernels for Apple Silicon with MLX
A deep technical guide to MLX, Metal Shading Language, unified memory, fused kernels, and performance tuning on Apple Silicon GPUs.
· 16 min read
From 1,000 Tokens per Second to Decisions per Second
A systems essay on why MiMo's speed milestone matters, and why the next frontier is decisions per second rather than raw token throughput.
· 18 min read
The Next Inference Bottleneck Is Not FLOPs. It Is the Shape of the Decode Loop.
A systems essay on why the real inference constraint is the structure of the decode loop itself, spanning FP4, MoE, SWA, speculative decoding, and persistent kernels.
· 18 min read
The Great Light Migration: Optical Interconnect Evolution
A technical guide to the five-step optical integration journey from pluggable optics to on-package photonics, and why AI is accelerating every stage of that migration.
· 18 min read
Adaptive Speculative Block Sizing for XL-Persistent-Kernel
A systems essay on making speculative decode self-tuning inside a persistent mega-kernel, using acceptance-rate feedback to adapt block size on-device with zero host round-trips.
· 16 min read
The Race to Put Light Inside the Chip: LPO, NPO & CPO Explained
A technical guide to pluggable optics, LPO, NPO, and copackaged optics in the AI data center, and why the industry is pushing light closer and closer to the switch ASIC.
· 17 min read
KV-Direct: Why Distributed KV Cache Needs a Linux Fast Path
A systems essay on why distributed KV cache turns Linux into the next bottleneck, and what a batched, deadline-aware restore path should look like across NVMe, RDMA, DRAM, and GPU memory.
· 15 min read
The Chip That Cools Itself: Inside the Race to Tame HBM's Heat Crisis
A deep dive into stacked-memory thermals, HBM5 cooling architectures, and how SK hynix, Samsung, and Micron are trying to move heat out from inside the package before AI server power density outruns external cooling.
· 12 min read
NIXL, NCCL, and UCX: What Each One Does and Why Host Memory Can Blow Up First
A technical explainer on abstraction boundaries in GPU communication: what NIXL owns, what NCCL solves, where UCX sits underneath, and why host-side registration and pinning can become the first memory failure domain.
· 18 min read
The $1 Component That Every AI Server Desperately Needs
A technical deep dive into multilayer ceramic capacitors in AI servers: voltage stabilization, GPU power integrity, supply concentration, failure modes, and why tiny passive parts quietly decide whether advanced compute stays upright.
· 25 min read
Beyond Memory Counters: Observing the Hidden Subsystems Under RDMA Pressure
A systems guide to the layers ordinary memory dashboards miss: PCIe AER, IOMMU faults, NIC transport counters, softirq squeeze, NUMA drift, GPU-side corroboration, and the failure chains that connect them.
· 20 min read
When the Kernel Loses Its Mind: Panics, OOM Killer & Memory Mayhem
A deep guide to Linux kernel panics, oopses, the OOM killer, soft lockups, hung tasks, and production debugging — how the kernel fails, why it fails, and how to harden systems.
· 15 min read
Gemma 4 on 16GB: The Definitive Local AI Guide
A practical guide to running Gemma 4 locally on a 16 GB machine: model selection, memory math, quantization, Ollama, llama.cpp, MLX, vLLM, multimodal support, and fine-tuning.
· 16 min read
The Hidden Cost of KV Cache Compression
A systems deep dive into KV cache compression: eviction, quantization, low-rank methods, latency tradeoffs, and why long-horizon agentic systems make compression failures much more dangerous.
· 18 min read
SambaNova Disaggregated Inference: The Architecture Reinventing How AI Thinks at Scale
A systems-level guide to SambaNova's heterogeneous inference stack: prefill on GPUs, decode on RDUs, orchestration on Xeon 6, and why agentic AI makes the split increasingly compelling.
· 14 min read
Solid-State Transformers and the 800V AI Data Center
A polished systems essay on why solid-state transformers and 800V DC are becoming serious AI-facility architecture questions: lower current, fewer conversion stages, faster control loops, and tougher protection tradeoffs.
· 20 min read
C to RTL: Co-Simulating an RL Inference Control Plane
A deep dive on bridging a C and C/CUDA runtime to a Verilator-backed RTL control plane: descriptor contracts, cycle-accurate tests, and the road to a 64-byte physical layout.
· 18 min read
From Runtime Rings to RTL: Hardware Queue Engine for RL Inference
A systems-level deep dive on turning an RL inference control plane into RTL: descriptor rings, MMIO doorbells, worker FSMs, completion queues, and Verilator co-simulation.
· 20 min read
From Doorbells to Rollouts — GB300 RL Runtime v0.2
How gb300-rl-runtime v0.2 built a measurable C/CUDA runtime skeleton for RL inference: rollout state machines, SPSC queues, COW prefix KV, reward handoff, tracing, and benchmarks.
· 25 min read
Close-to-Metal RL Inference at GB300 Scale
A deep systems-level reference on close-to-metal RL inference at GB300 scale: persistent decode workers, hugepage-backed KV arenas, GPU/NIC command rings, NUMA locality, TLB/IOMMU reduction, cache coherency, and async reward handoff — with full C code.
· 18 min read
GPU-to-GPU Hybrid Bonding: From Multi-GPU Communication to Multi-Die GPU Fabrics
A systems-level analysis of GPU-to-GPU hybrid bonding, advanced packaging, chiplet GPUs, AI accelerator fabrics, and the future of multi-die GPU architecture — why the next scaling curve depends less on raw FLOPs and more on how cheaply silicon can talk to silicon.
· 18 min read
Intent Attention Kernel: Intent-Aware KV Execution for Agentic Long-Context Inference
A systems-level essay on why long-context attention should not pretend every KV block is equally useful — semantic block metadata, IntentQuant-KV precision policies, and what runtime-intent-aware KV execution looks like in the long-context regime.
· 25 min read
MLCC: The Invisible Backbone of Modern Electronics
A complete ground-up guide to the world's most essential passive component — from ceramic dielectric physics and manufacturing to a $22B market shaped by AI servers, electric vehicles, and geopolitical supply shocks.
· 13 min read
SK Hynix's iHBM: Inside the Technology Redefining AI Memory Cooling
A systems-level analysis of SK Hynix iHBM, HBM thermal bottlenecks, MR-MUF packaging, HBM5 power density, and why memory cooling is becoming strategic.
· 10 min read
NVIDIA Vera and the Control Plane of the AI Factory
A systems-level analysis of NVIDIA Vera, Rubin, and why the AI factory is becoming an orchestration and memory-movement problem, not just a GPU story.
· 14 min read
Huawei's Tau Scaling Law and the Beginning of Distance-Centric Computing
A systems-level analysis of Huawei's Tau Scaling Law, LogicFolding, distance-centric computing, verification complexity, and why test and packaging infrastructure may matter more.
· 8 min read
What Is a Token? The Atom of AI Language
A clear explainer of what tokens are, how tokenization works, why context windows and pricing depend on them, and why they shape model behavior.
· 12 min read
Disaggregated Memory for Long-Context AI Systems
A vendor-neutral reference architecture for standalone DRAM racks serving long-context AI systems, from fabric topology and KV-cache offload to failure modes and deployment criteria.
· 18 min read
KV ASIC Part 3: The Chip
A deep technical architecture essay opening up the proposed KV infrastructure ASIC: floorplan, DMA engines, metadata SRAM, compression, prefetching, timing paths, and cluster topology.
· 15 min read
Why KV Cache Needs Dedicated Infrastructure
A systems architecture essay on why KV cache has become first-class infrastructure for long-context inference, and why generic memory systems are not enough.
· 11 min read
GQA and MQA: Attention, KV Cache, and Faster LLM Inference
A ground-up guide to Multi-Query and Grouped-Query Attention, why they shrink KV cache memory, and how they make long-context LLM inference cheaper and faster.
· 16 min read
CXL and the Search for a New Memory Tier in AI Inference
A systems essay on where CXL fits in the AI memory hierarchy, from GPU HBM and warm KV cache tiers to CPU DRAM, NVMe, and long-context serving pressure.
· 20 min read
Beyond CXL: The Case for a KV-Aware Memory ASIC
A systems architecture essay on why generic CXL memory is the wrong abstraction for transformer inference, and how semantic KV-aware memory control could reduce HBM pressure.
· 20 min read
Retimers: The Hidden Signal Infrastructure of the AI Chip Era
A systems-level guide to why retimers matter in AI servers, from PCIe 5/6 and CXL signal integrity to vendor positioning, active cables, optics, and the bear case.
· 22 min read
Part 2: How Memory Timings Are Actually Configured in Modern Systems
Part 2 extends the memory-timings deep dive into inference performance: Linux runtime tuning, hugepages, NUMA binding, PCIe topology, GPU placement, and verification commands.
· 18 min read
How Memory Timings Are Actually Configured in Modern Systems
A practical deep dive into how modern systems configure memory timings across BIOS, firmware training, Linux runtime tuning, NUMA placement, CXL tiering, and HBM locality.
· 10 min read
The Rise of AI Kernel Engineers
Autonomous agents are starting to design, benchmark, debug, and optimize the low-level CUDA and Triton kernels that determine real AI system throughput.
· 19 min read
How Chips Actually Work: From Logic Gates to GPUs, ASICs, TPUs, FPGAs, Accelerators, and the Brain
A ground-up guide to how transistors become logic, logic becomes stateful machines, and architecture diverges into CPUs, GPUs, ASICs, TPUs, FPGAs, AI accelerators, and brain-like systems.
· 8 min read
Zhenwu M890: Why Alibaba Bet on Memory Fabrics, Not Just FLOPS
A systems architecture analysis of Alibaba Zhenwu M890, explaining why AI infrastructure is shifting from tensor FLOPS to memory orchestration, HBM, KV cache, and rack-scale fabrics.
· 21 min read
First Silicon Is Not the Finish Line: Linux Board Bring-Up for x86, Arm, and Accelerators
A systems engineering guide to Linux board bring-up for x86, Arm, and accelerators, covering firmware, ACPI, Device Tree, drivers, DMA, IOMMU, runtimes, and debugging.
· 12 min read
HBM, DDR, SODIMM and the AI Memory Wall
A systems-level deep dive into HBM, DDR5, SODIMM, LPDDR, CXL, KV-cache pressure, packaging IP, and the future AI memory wall.
· 9 min read
The Quadrillion-Token Era Has Arrived
A systems deep dive into the quadrillion-token era, KV-cache memory pressure, prefill/decode disaggregation, token warehouses, and the AI memory wall.
· 16 min read
Disaggregated Memory — The Architecture Reshaping AI Infrastructure
A technical deep dive into disaggregated memory for AI infrastructure, covering CXL, NVLink, memory pooling, vendor ecosystems, latency, and data-center design.
· 16 min read
Why Co-Packaged Optics Is Crucial in Semiconductors
A technical essay on why co-packaged optics matters for AI semiconductors, bandwidth density, switch power, optical I/O, and data-center fabric scaling.
· 17 min read
Pretraining Scaling: The Engine Behind Modern AI
A deep technical explainer on scaling laws, Chinchilla compute-optimal training, data limits, inference economics, and why modern AI progress was predictable enough to plan around.
· 25 min read
GhostKV: Attention Without Storing Full KV
A technical concept essay on query-time bounded elimination, semantic witnesses, and reconstruction-first KV memory for long-context transformer inference.
· 22 min read
The Memory Wall Is Moving
A deep technical essay on why AI inference is shifting from a compute bottleneck to a memory-bandwidth and orchestration problem.
· 16 min read
The SRAM Insight: How I Built a Patent Around AI's Oldest Memory Habit
A patent-backed essay on attention-sink-aware SRAM placement, explaining how sink-token behavior can reroute the hottest KV state into faster on-chip memory.
· 17 min read
SRAM as the Specialist: Heterogeneous Inference and the Decode Bottleneck
A deep technical essay on why SRAM-heavy accelerators fit decode-side FFN and MoE paths, and how heterogeneous inference reframes the AI chip memory debate.
· 18 min read
MFU, BF16, FP8 and AI Numeric Formats: The Complete Guide
A complete guide to model flop utilization, low-precision formats, tensor-core throughput, and how to read AI training efficiency claims without getting fooled by the denominator.
· 19 min read
The Ideal PagedAttention Stack: Hardware + Software for Long-Context Inference
A systems essay on KV paging, FlashAttention-style tiling, SRAM-sized decompression windows, cold-memory prefetch, and the hardware long-context inference really wants.
· 17 min read
Four Things Nobody Has Written About FlashDecode
An original technical analysis of FlashDecode covering kernel crossover math, reduction inversion, CXL asymmetry, and runtime switching for long-context inference.
· 19 min read
KV Cache, Transformer Memory, and Why TurboQuant Matters
A deep technical essay on KV cache growth, tiled attention, outliers, and why long-context inference is increasingly a memory systems problem rather than just a compute problem.
· 22 min read
The Compiler Becomes the Memory Scheduler
Why runtime dynamism is becoming too expensive for AI inference, and how compilers are evolving into full memory traffic planners across DMA, tensor placement, and deterministic execution windows.
· 18 min read
The Next AI Bottleneck Isn't FLOPs. It's Attention Memory.
A systems-level deep dive into KV cache growth, attention memory, and why the next AI architecture war is rapidly becoming a memory war.
· 20 min read
Coherent Fabrics: The Memory Highways Behind Agentic AI
A deep technical essay on coherent fabrics, CXL, NVLink, cache coherence, and why shared-state coordination is becoming the hardware bottleneck for agentic AI systems.
· 18 min read
The Memory Wall Is the New Compute Wall
A deep technical essay on how CXL, coherent memory, SmartNICs, GPUDirect Storage, unified memory, and KV-cache routing all converge on the same core AI infrastructure problem: moving state efficiently.
· 14 min read
Host RAM vs HBM for Inference: What Really Lives Where
Device-local KV, host spill, pinned buffers, DMA paths, and the physical constraints that determine what your serving stack should put where — and why getting it wrong costs 10× on tail latency.
· 18 min read
The Memory Wall — How AI Infrastructure Is Being Rewired From the Ground Up
A technical deep-dive into how CXL, SmartNICs, GPUDirect Storage, and unified memory are solving the memory movement bottleneck in modern AI systems.
· 18 min read
KV Cache Is a Memory System
Attention indexing, VRAM, HBM, huge pages, TLB pressure, and pinned memory explained from first principles for modern inference stacks.
· 12 min read
Why AI Inference Needs a Weight Delivery Architecture
A public-facing note on Indian patent application no. 202641059412 and the case for treating model-weight delivery as its own inference architecture problem.
· 16 min read
Why LLM Prefill and Decode Suffer from Cache Misses, Branch Stalls, and Pipeline Bubbles on CPUs
A systems-level explainer on how cache misses, branch stalls, and pipeline bubbles shape CPU inference performance in transformer prefill and decode.
· 21 min read
/dev/mem_hint: A Kernel Control Plane for AI Memory Systems
A deep technical primer on /dev/mem_hint, the kernel-mediated hint path linking AI runtimes, PMU classifiers, MSR/MMIO/CXL conduits, and memory PHY policy engines.
· 14 min read
Teaching Computers to Remember Smarter
A patent essay on software-defined, workload-aware adaptive memory signaling for AI systems, letting runtimes tune memory timing, power, and latency by phase.
· 8 min read
HBM Is Not a Cache: What High-Bandwidth Memory Actually Does in AI Systems
A technical essay on why HBM is not a magical cache, but a scarce local working tier whose value depends on residency policy, bandwidth budgeting, and orchestration.
· 7 min read
MoE Is a Networking Problem Wearing a Model Costume
A systems essay on why sparse expert routing becomes a fabric, transport, and queueing problem long before it becomes a pure model-efficiency win.
· 6 min read
Power Is Becoming a Scheduling Constraint
A technical essay on why rack power, thermal headroom, and electrical state are becoming live runtime inputs for admission, placement, and useful-throughput control.
· 13 min read
Inference Is a Queueing System: Why AI Serving Lives or Dies by Arrival Curves, Batching, and Tail Latency
A systems essay on why serving quality is shaped by queue depth, prefill/decode asymmetry, batching policy, and memory-driven service-time variance.
· 11 min read
The Scheduler Is the Product: Why AI Infrastructure Moats Are Becoming Policy Engines
A technical essay on why the deepest AI infrastructure advantage is moving into placement, admission, routing, and state-management policy.
· 11 min read
The Real Cost of a Miss: Unifying Cache Misses, KV Misses, Fabric Misses, and Remote Fetch in AI Systems
A memory-systems essay on how misses really behave in AI infrastructure, from HBM and KV locality to fabric fetch and byte-movement economics.
· 12 min read
Rambus: The Hidden Bottleneck Bet in the AI Memory Supercycle
A deep technical essay on Rambus, from MRDIMM and interface chips to CXL, security IP, and why AI infrastructure is becoming memory-interface bound.
· 38 min read
DeepSeek V4: The Model That Refuses to Stop Shocking the World
A technical essay on DeepSeek V4, from hybrid attention and million-token context to pricing, Huawei inference infrastructure, and the geopolitics of open weights.
· 36 min read
Intel 18A & 14A: The Nodes That Could Rewrite Chip History
A technical essay on Intel 18A and 14A, from RibbonFET and PowerVia to High-NA EUV, foundry economics, packaging, and the race with TSMC.
· 29 min read
ARM Data Center CPU Report: Comprehensive Technical Analysis
A technical deep dive into Graviton4, Axion, Cobalt 100, AmpereOne, Grace, and Qualcomm across architecture, benchmarks, software, and platform tradeoffs.
· 7 min read
Rambus SOCAMM2: The Modular LPDDR Memory Layer for AI Servers
A systems-level primer on SOCAMM2, LPDDR5X server modules, Rambus support silicon, and where modular memory fits in the AI server hierarchy.
· 20 min read
High-Power Laser Arrays for Co-Packaged Optics: The Hidden Power Plant of AI Interconnects
A systems primer on high-power laser arrays, external laser sources, silicon photonics, packaging, and why CPO optics are becoming central to AI data-center networks.
· 22 min read
The Substrate Layer: What Sits Between the Chip and Everything Else
A deep dive into semiconductor substrates, from ABF and BT to ceramic, glass, fan-out, and the advanced packaging stack underneath modern AI chips.
· 18 min read
The Brains Behind AI Storage, v2: Smart Controllers, Linux Drivers, and the Full Path from read() to GPU
A deep technical primer on smart storage controllers, Linux drivers, NVMe queues, DMA, IOMMU, and the full AI server path from read() to GPU HBM.
· 15 min read
AI Storage Primer: From NAND Physics to GPUDirect and the Memory Wall
A technical primer on AI storage, from NAND and SSD controllers to GPUDirect Storage, HDD economics, HBF, and the memory wall in modern AI systems.
· 17 min read
Vera, Venice, AGI, Clearwater: The Coming Wave of AI CPUs
A technical essay on the next wave of AI CPUs: Intel Clearwater Forest, AMD Venice, Arm AGI, NVIDIA Vera, memory coherency, power, and host orchestration.
· 18 min read
The CPU Is Back: Why AI Broke the GPU-Only Illusion
A technical essay on why CPUs are becoming central again in AI infrastructure: memory movement, GPU orchestration, agentic workloads, power, and runtime control.
· 18 min read
The AI Power Delivery Stack: How Modern Accelerators Are Really Powered, and What Each Vendor Actually Does
A technical guide to the AI power delivery stack from grid to chip, mapping Vicor, MPS, TI, ADI, Renesas, Infineon, Delta, Flex, Murata, Bel, Navitas, and Wolfspeed.
· 22 min read
The 800V DC Era Is Here — and It's Rewriting Who Builds AI Power
A technical essay on why NVIDIA and AI data centers are moving toward 800V DC power, and how SiC and GaN semiconductors reshape the infrastructure stack.
· 16 min read
The Next-Generation AI Chip: Inside the Logic Bridge, Realistic Floorplans, and the Rise of On-Package NUMA
A technical essay on next-generation AI chip floorplans, logic bridges, HBM expansion, on-package NUMA, SRAM locality, and where optical interconnects fit.
· 18 min read
The Next-Generation AI Chip: From Flat HBM to an On-Package NUMA Fabric
A technical architecture essay on how AI accelerators may evolve from flat HBM layouts into topology-aware on-package NUMA fabrics with chiplets, SRAM layers, and optical links.
· 18 min read
NVLink Switch Is Not NVLink: The Scale-Up Fabric Architecture Nobody Fully Explains
A systems-level explanation of NVLink Switch, NVL72 scale-up fabrics, GPU coherence boundaries, and where scale-up ends and scale-out begins.
· 20 min read
800G Ethernet vs. InfiniBand: The AI Scale-Out Decision Nobody Documents Honestly
A technical comparison of 800G Ethernet and InfiniBand for AI scale-out, covering congestion control, collectives, latency, cost, and operational tradeoffs.
· 19 min read
CXL Is Three Protocols in a Trenchcoat: What .io, .mem, and .cache Actually Do
A ground-up explanation of CXL.io, CXL.mem, and CXL.cache, and why CXL is really a family of protocols for coherent memory expansion.
· 21 min read
NCCL Internals: The Collective Communications Layer Nobody Reads — But Everyone Depends On
A deep technical walkthrough of NCCL internals, collective communication, rings, trees, channels, topology awareness, and why AI clusters depend on it.
· 16 min read
The Cost of Testing AI Chips Is Exploding
Why wafer probe, burn-in, ATE, high-speed I/O validation, thermal control, and advanced packaging are turning AI chip test into a first-order cost constraint.
· 18 min read
The Hidden Backbone of AI Hardware: How Chips Are Tested Before They Power the World
A practical map of wafer probe, burn-in, ATE, high-speed I/O validation, handlers, and system-level test for modern AI chips.
· 20 min read
The Compiler Is the New Kernel: Why MLIR/Triton/XLA Are the Most Underrated Layer in AI Infrastructure
Why MLIR, Triton, and XLA are becoming the control layer between AI hardware, kernel generation, memory policy, and inference efficiency.
· 22 min read
Fat-Tree Is Lying to You: Network Topology as a First-Class Inference Constraint
Why fat-tree assumptions break under MoE routing, prefill-decode disaggregation, KV transfer, and topology-aware inference scheduling.
· 24 min read
The Inference Unit Economics Time Bomb: Why $/Token Will Collapse (and What Survives)
A first-principles look at token pricing, the physical memory-and-cooling cost floor, and which AI infrastructure moats survive.
· 35 min read
Co-Packaged Optics (CPO): The End of Pluggable Transceivers — A Ground-Up Guide
A ground-up technical guide to co-packaged optics, pluggable transceiver limits, silicon photonics, and AI data-center fabric power.
· 12 min read
The Machines That Build the Machines Pt. 3 — Beyond Lithography
A technical essay on the semiconductor manufacturing stack beyond lithography: deposition, etch, CMP, implant, metrology, clean, and subsystems.
· 12 min read
Inside ASML: The Product Line That Prints the Future (and Why EUV Took 30 Years)
A technical deep dive into ASML lithography systems, from DUV and immersion to Low-NA EUV, High-NA EUV, and why manufacturable EUV took three decades.
· 28 min read
The Machines That Build the Machines: Inside Lam, AMAT, ASML, KLA and TSMC's Node Race
A technical deep dive into the equipment stack behind advanced nodes: how Lam, Applied Materials, ASML, KLA, and TSMC fit together from etch and deposition to EUV and yield.
· 6 min read
GPU-to-CPU Deployment Ratios in Modern Server Deployments
GPU:CPU planning is becoming workload-specific again. For long-context, retrieval-heavy, and orchestration-heavy inference, CPU capacity, memory bandwidth,...
· 10 min read
Quantum Series #5: Quantum Simulation — Why Chemistry and Pharma Actually Care
From a 70-year algorithm to medical AI: how quantum computing flips the resource game for molecules, materials, and drugs
· 12 min read
Gradient Checkpointing Is a Memory Problem: Training Efficiency From First Principles
The inference memory essays in this series talk about KV cache, weight residency, and HBM pressure. Training has its own version of all three — and the...
· 13 min read
The Router Tax: What Nobody Accounts For in MoE Inference Cost
Mixture-of-Experts models are sold as a 10× efficiency win: 10× more parameters, same per-token compute. That math is correct. But it omits the router — the...
· 14 min read
Weight Streaming: Why Your Model's Weights Are the Real Decode Bottleneck
The KV cache gets all the attention. But for large models at low batch sizes, loading transformer layer weights from HBM dominates decode latency just as...
· 13 min read
Quantum Series #4: Why VQE/QAOA Don't Scale Yet — The Barren Plateau Problem
A thesis-driven essay on why VQE and QAOA hit barren plateaus, routing overhead, and noise walls long before meaningful NISQ advantage.
· 9 min read
Quantum Series #3: Quantum Error Correction — Why 1 Good Qubit Costs 1,000 Bad Ones
A technical essay on surface codes, fault-tolerance thresholds, and the brutal redundancy required to turn noisy physical qubits into useful logical ones.
· 11 min read
Quantum Computing and Cryptography: What Breaks, What Survives, What Comes Next
A technical essay on which cryptographic systems quantum computers actually threaten, which survive, and how post-quantum migration should be understood.
· 12 min read
Quantum Computing: Structured Interference at Scale
A technical essay on quantum computing as structured interference, covering superposition, entanglement, amplitude amplification, error correction, and where speedups actually come from.
· 10 min read
LLM Architecture Survey: From Transformers to Modern Variants
A technical survey of large language model architectures covering transformer backbones, attention variants, sparsity, MoE, normalization, positional encoding, and systems tradeoffs.
· 15 min read
Quantum Computing: A Ground-Up Technical Primer
A ground-up technical primer on quantum computing covering qubits, gates, entanglement, circuits, error correction, and the path from theory to useful machines.
· 17 min read
The Whole Stack: Everything Between a Transistor and a Token
A synthesis essay connecting transistor physics, memory hierarchies, inference systems, agents, and economics into one view of the AI stack.
· 20 min read
Sparse MQA + Fused MoE Mega Kernel + Hyper-connections: The Trifecta Behind Modern Frontier Models
A technical essay on three systems ideas shaping frontier models: sparse MQA, fused MoE mega kernels, and hyper-connections as a throughput and efficiency stack.
· 14 min read
The PCIe Tax: How Host-Staged Networking Steals Half Your GPU Bandwidth
A technical essay on the host-side copy tax in inference systems, why PCIe crossings burn usable bandwidth, and how GPUDirect RDMA plus DPUs recover MoE throughput.
· 16 min read
HBM Explained: How High Bandwidth Memory Is Actually Built
A technical essay on how HBM is physically constructed, from stacked DRAM dies and TSVs to silicon interposers, and why packaging physics sets the AI bandwidth ceiling.
· 17 min read
Inside the GPU: SMs, Warps, Tensor Cores, and Why the Architecture Looks the Way It Does
A technical essay unpacking SMs, warps, tensor cores, register files, and why GPU design choices make sense for parallel workloads but strain during autoregressive decode.
· 17 min read
What "3nm" Actually Means: Process Nodes, Transistor Physics, and Why Scaling Is Slowing
A technical essay on what modern node names really mean, how FinFET and GAA devices extend scaling, and why Moore's Law is slowing for AI chip design.
· 12 min read
KV Fabrics: Treating Context as a Distributed Filesystem
A technical essay on turning KV cache into rack-scale infrastructure across HBM, CXL DRAM, and NVMe, with DPUs acting as the metadata control plane.
· 22 min read
The Memory Bottleneck: Why Inference Speed Is a Memory Problem, Not a Compute Problem
A technical essay on why AI inference latency is usually constrained by memory bandwidth and hierarchy design, not raw arithmetic throughput.
· 12 min read
The DPU as Agent Memory Controller: Offloading Orchestration
Why the 78% CPU / 31% GPU problem in agentic inference isn't a software bug — it's an architecture problem. Using a BlueField-3 DPU to bypass the host and feed the GPU directly.
· 12 min read
Why Agentic AI Is a CPU and DRAM Problem, Not Just a GPU Problem
The center of gravity for AI inference is shifting from dense matmuls to stateful orchestration, context movement, and memory capacity. For three years, "AI scaling" meant buy more GPUs. Agentic AI breaks that model.
· 8 min read
RISC vs CISC: Where Complexity Lives
RISC and CISC aren't processor brands — they're philosophies. One says make the hardware simple and push work to software. The other says make the hardware clever to make software easier. Modern chips use both ideas, but understanding the trade-off explains almost everything about why your phone's processor looks nothing like your laptop's from 1995.
· 23 min read
The Agent Topology Problem: Inference Scheduling Across Model Boundaries
Agentic AI workloads — orchestrators calling planners calling retrievers calling verifiers — are not well-served by inference infrastructure designed for isolated single-model requests.
· 25 min read
The True Cost of a Token: Inference Economics from Silicon to SLO
OpenAI, Anthropic, Google, and a dozen open-source providers all quote $/1M-token prices. Nobody explains what those numbers are made of. This essay derives the cost of an output token from first principles.
· 12 min read
Multi-Tenant KV Fabrics: Bandwidth Contracts for Shared LLM Memory Systems
KV cache bandwidth must be partitioned like network bandwidth. With compiler contracts, token buckets in hardware, and page coloring, multi-tenant tail latency improves 3.6x and throughput improves 28 to 31 percent.
· 14 min read
The Power Contract SLO: Making Joules-per-Token Schedulable
Power is not a limit to hit and then throttle. Power is a budget to allocate. By making joules-per-token a first class SLO, we can turn blind firmware throttling into global, proactive contract scheduling.
· 6 min read
Why AI Servers Need a Memory Fabric, Not Just a Faster Bus
The physical limits of the motherboard have been reached. HBM is fast, but the moment an AI workload spills to host memory over PCIe, performance falls off a 75x cliff. Here is why CXL memory disaggregation is dismantling the definition of a server.
· 9 min read
Floating Point in AI: The Hidden Numbers Running Your Models
AI is powered by billions of approximate numbers. Once you understand floating point constraints—from fp32 to fp8—a lot of AI hardware design, quantization, and training stability suddenly clicks.
· 44 min read
The Semiconductor Ecosystem: Every Layer, Every Player, Every Dependency
A comprehensive guide to the global semiconductor supply chain — from EDA tools and chip design through foundries, equipment, materials, memory, packaging, CPO, and test. 100+ companies across 7 countries, with detailed chokepoint analysis.
· 7 min read
Software-Managed Memory Is the TPU's Real Advantage — and Its Hardest Abstraction
No hardware caches. No L1/L2. The TPU exposes raw scratchpad SRAM to the XLA compiler — and that changes everything about memory determinism vs. generality.
· 7 min read
Optical Circuit Switching and the 3D Torus: Why TPU Pods Are a Different Kind of Supercomputer
GPU clusters route packets through switches. TPU pods route light through MEMS mirrors. The physical network reshapes itself to match the computation.
· 6 min read
Why TPUs Have Less HBM Than GPUs — and Why That's a Design Choice, Not a Limitation
NVIDIA bets on fat nodes with maximum per-GPU memory. Google bets on thin nodes with maximum inter-chip bandwidth. Both are rational — they optimize for different cost functions.
· 7 min read
NAND Flash Is the Invisible Backbone of Every AI Cluster
The GPU gets the credit. The SSD does the lifting. A systems essay on the five roles NAND flash plays — from checkpoint absorption to KV-cache offloading and weight staging.
· 7 min read
High Bandwidth Flash: The Missing Tier Between HBM and NVMe in AI Inference
8–16× HBM capacity, within 2.2% of HBM read performance. How NAND-based HBF is designed to close the 300× bandwidth gap in the AI memory hierarchy.
· 7 min read
The Storage Geometry of a 100,000-GPU Cluster: How Much NAND Does AI Actually Need?
A bottom-up accounting of flash capacity in hyperscale AI — from 15 TB per node to 3 exabytes per cluster. The numbers, the form factors, and the supply-chain pressure.
· 8 min read
The Scheduling Tax of Multi-Tenant Inference: Why Fairness and Throughput Are a Memory Problem
When a 128K-context request shares a GPU with forty 2K requests, the scheduler sees equal slots. The memory system sees a 64× asymmetry — and the minnows pay the price.
· 7 min read
Inference Batch Geometry: Why the Shape of a Batch Determines Its Memory Cost More Than Its Size
Batch size is a number. Batch geometry is a distribution. Two batch-32 configurations can differ by 10× in memory cost — and the memory system sees the distribution.
· 9 min read
The Recompute-vs-Transfer Frontier: When Redoing Work Is Cheaper Than Moving Bytes
For some KV pages, recomputing attention from scratch costs less than fetching from host RAM. A systems essay on the fourth action in eviction policy design.
· 15 min read
KV Hierarchy Lab: Regret-Aware Eviction and Trace-Driven Policy Evaluation
A trace-driven research harness for studying KV-cache eviction, quantization, and prefetch tradeoffs — featuring a regret-aware eviction policy with multi-seed ablation results.
· 29 min read
The Power Stack: How AI Scale-Out Gets Its Electricity — From Grid to GPU
A technical and industrial essay on the full power ecosystem behind AI data centers, from generation and transmission to UPS, switchgear, and GPU delivery.
· 23 min read
Why Vera Rubin Changes Everything About Cooling: 45°C Supply Temperature, Fan-Free Trays, and the Infrastructure Behind Them
A systems essay on how Vera Rubin NVL72 changes AI data center cooling with 45°C supply temperatures, fan-free trays, hose-free design, and liquid-cooled busbars.
· 22 min read
The Cooling Stack Is the New Critical Path: How Blackwell GB300 NVL72 Racks Manage 142 kW
A systems essay on the five-layer direct liquid cooling architecture of Blackwell GB300 NVL72 racks and the suppliers behind each thermal layer.
· 13 min read
vLLM Internals: Where to Actually Cut Batch Size When the GPU is Melting
A systems essay on where thermal control actually belongs inside vLLM, from scheduler decisions and swap behavior to tensor-parallel batch cutting.
· 22 min read
Why HBM Thermal Throttling Is Silent: Reading the Tea Leaves in nvidia-smi
A technical guide to detecting silent HBM thermal throttling on H100 and H200 clusters when standard GPU temperature dashboards look deceptively healthy.
· 18 min read
Thermal Debt Is a Memory Problem — How Hot Dies Throttle Your KV Prefetch
A systems essay on HBM thermal telemetry, KV fetch stalls, and why hot memory dies turn thermal debt into an inference scheduling problem.
· 19 min read
CPO Isn’t About Power. It’s About Making Memory Disaggregation Schedulable
A systems essay on why co-packaged optics matters because it makes disaggregated memory and expert movement schedulable under tight latency budgets.
· 11 min read
HBM Throttling-Safe KV Admission and ReuseNet for LLM Inference
A technical essay on thermal-safe KV admission, HBM backpressure, reuse prediction, and production-serving policy design for H100 and H200 inference clusters.
· 30 min read
The Photonics Stack: Who Builds What for AI Networking — Part 1
A technical primer mapping the AI optical networking stack across fiber, lasers, transceivers, DSPs, switches, and test infrastructure through the companies building each layer.
· 17 min read
The Attention Sink Problem: Why Transformer Inference Wastes More Memory Than You Think
A systems essay on attention sink tokens, structural KV cache waste, and why long-context serving needs memory-policy-aware treatment of hot-but-low-utility tokens.
· 19 min read
Thermal Debt in AI Clusters: The Silent Degradation Loop Nobody Is Measuring
A systems essay on how dense GPU racks accumulate thermal debt, why point-in-time observability misses it, and what thermally-aware control planes should measure.
· 9 min read
TechDemoForge — A Local-First Engine for Turning Technical Docs into Demo Videos
A detailed product and architecture essay on TechDemoForge, its workflow, repo structure, and why local-first technical demo generation is useful.
· 17 min read
Prefill-Decode Disaggregation: Why the Next Big Inference Architecture Splits the Job in Two
A systems essay on why prefill and decode should be split across different hardware pools and why the real engineering challenge becomes memory orchestration.
· 17 min read
Speculative Decoding Is a Memory Problem
A systems essay on draft/verify KV pressure, rollback fragmentation, and why speculation pays off only when memory policy is designed for it.
· 8 min read
The Next Frontier in Long-Context Inference: Memory-Orchestrated Sparse Serving
A systems essay on sparse attention serving, hierarchical KV residency, predictive prefetch, and why long-context wins increasingly come from memory policy.
· 8 min read
Scale-Out Was Yesterday. Scale-Up Optics Is the Next Battle
A systems-first essay on why the next optical contest in AI infrastructure is shifting inward toward dense rack-scale and scale-up fabrics.
· 6 min read
InP vs Silicon Photonics vs VCSEL: The Materials Stack Behind AI Networking
A deeper systems-first essay on why AI networking will be built from a layered materials stack rather than a single optical winner.
· 7 min read
The Real AI Bottleneck Is Moving From Compute to Interconnect Power Density
A deeper systems essay on why the next constraint is the power, heat, and topology cost of moving bits across clusters.
· 15 min read
Photonics Is No Longer a Component Story — It Is Becoming the Operating System of AI Clusters
A systems essay on how optics is moving into the scheduler, topology planner, reliability model, and power logic of next-generation AI infrastructure.
· 15 min read
CPO, LPO, DSP, and VCSEL: What Actually Matters for AI Infrastructure
A practical hardware-focused companion essay on optical architectures, power budgets, serviceability, and the real tradeoffs hyperscalers optimize for.
· 12 min read
The Memory Scheduler Is the New Critical Path in AI Inference
A systems essay on explicit data movement, KV cache management, tiered memory placement, DMA orchestration, and why scheduler quality now directly determines inference efficiency.
· 18 min read
Why Cache Coherency Is the Wrong Default for AI Machines
A long-form systems argument for selective coherency, explicit tensor movement, CXL.mem over universal coherence, and schedule-first design in large-model infrastructure.
· 16 min read
When VRAM Stops Being a Weight Warehouse
A systems primer on HBM as a bounded working set, KV cache dominance, PagedAttention, weight offload, and why scheduled weight streaming is the next step in inference architecture.
· 11 min read
When 128 TB Stops Feeling Infinite: Address Space, mmap, and the Quiet Limits of Modern Systems
A systems essay on 48-bit virtual addressing, mmap-heavy designs, storage density, 5-level paging, and why explicit data orchestration becomes the long-term answer.
· 16 min read
Memory Intent IR: Why AI Compilers Must Emit Memory Plans
A systems essay on compiler-emitted memory intent, object semantics, workload phases, reuse confidence, and why hardware orchestration needs structured plans instead of blind guesses.
· 14 min read
MCOS-HFC: A Hardware Fabric Controller for Memory-Centric AI Systems
A technical essay on explicit memory intent, residency maps, regret-aware eviction, recomputation-vs-transfer arbitration, atomic doorbells, and DPU embodiments for AI memory fabrics.
· 6 min read
MCOS Must Live in Hardware: From JBOD to Intelligent AI Memory Fabrics
A systems essay on why software-only memory orchestration hits a ceiling and why the real future is hardware-resident movement control near the fabric, the tiers, and the accelerators.
· 8 min read
MCOS: A Memory-Centric Operating System for the Future of AI Systems
A technical essay on memory placement, movement, residency, reuse, admission, and eviction as first-class scheduling decisions rather than passive implementation details.
· 8 min read
Why "Disk → RDMA → GPU" Is Still Fragmented Today
A technical essay on why local direct-storage acceleration and network direct-memory acceleration still do not add up to one universal GPU-native end-to-end storage fabric.
· 8 min read
From SSD to GPU to SRAM: Why the Last Bottleneck Is Now On-Chip
A technical essay on why eliminating host-side bounce buffers shifts the real bottleneck inward, toward deterministic HBM↔SRAM orchestration inside the accelerator.
· 16 min read
Bounce Buffers: The Hidden Tax on Modern AI Systems
A technical essay on hidden staging buffers, GPUDirect-era dataflow, and why eliminating unnecessary copies matters across storage, network, memory, and accelerator paths.
· 10 min read
RDMA in the Age of AI: Zero-Copy, KV Cache Transfer, and the New Glue Layer
A detailed technical essay on RDMA, zero-copy realities, and why "RDMA exists" still does not mean true end-to-end zero-copy in disaggregated inference.
· 7 min read
Seam Orchestrator: Workload-Aware KV Routing in Disaggregated Inference
A technical essay on policy above transport for KV movement, workload-aware admissibility, swappable glue layers, Scenario E and F, and experiment-backed results.
· 12 min read
AI Cluster Reliability Beyond Fault-Tolerant Parallelism
A technical essay on gray failures, checkpoint economics, cooling-compute seams, and seam-aware control planes for modern AI cluster reliability.
· 10 min read
The Next AI Cluster Failure Won’t Look Like a GPU Failure
A technical essay on seam failures across facilities, fabrics, heterogeneous inference pools, checkpoint economics, and why component dashboards miss the most expensive cluster incidents.
· 5 min read
Adaptive Compiler–Runtime Power Contract for Energy-Optimal Edge Inference
A technical explainer on authenticated power contracts, alternative execution plans, safe switching boundaries, and runtime enforcement for edge inference systems.
· 4 min read
Deterministic Memory-Orchestrated Inference Using DMA and Bounded On-Chip Buffers
A technical deep dive into compiler-scheduled DMA, explicit fences, bounded SRAM, and why deterministic buffer legality changes the inference memory system.
· 4 min read
SLA-Constrained Energy-Aware Inference Scheduling on ARM Edge Systems
A technical essay on latency-aware policy selection across model variants, DMA strategy, memory residency, and accelerator performance-state control on edge systems.
· 8 min read
Predictive Weight Orchestration: Runtime Control for Multi-Tier Weight Residency
A technical essay on HBM pressure, predictive multi-tier placement, precision state transitions, MoE router-history signals, and bandwidth-aware runtime scheduling.
· 8 min read
Mobile Agent Control: A Vendor-Neutral Control Plane for Terminal-Native Coding Agents
A systems essay on Android-first operations, FastAPI supervision, runtime adapters, websocket telemetry, and safe control of local coding agents across machines.
· 5 min read
Introducing ChromeLens: Systems-Grade Web Performance Telemetry
An introduction to ChromeLens, deterministic CDP tracing, interactive flow profiling, and the hydration penalty behind complex modern web applications.
· 11 min read
What Bigger L2 Actually Buys You
A detailed technical essay on larger L2 caches in AI systems, miss-rate reduction, average access time, bandwidth relief, energy tradeoffs, and where bigger L2 stops being enough.
· 12 min read
Why AI Needs a New Memory Hierarchy, Not Just Bigger Caches
A detailed technical essay on AI-native residency fabrics, class-aware memory, weights, KV cache, experts, and why bigger generic caches are not the full answer.
· 13 min read
SRMIC-X1: Rethinking the Memory Hierarchy for LLM Decode
A technical essay on residency-first decode acceleration, distributed on-package SRAM, HRM/HBM crossover behavior, and why autoregressive inference is fundamentally memory-bound.
· 9 min read
How to Measure GPU Underutilization on NVIDIA H100 and H200
A practical systems essay on rolling-window low-utilization metrics, sampled idle behavior, power telemetry, and what point-in-time GPU utilization misses.
· 7 min read
Schwab Portfolio Tools and the Case for Local, Practical Portfolio Infrastructure
A technical essay on privacy-first local portfolio tooling, Schwab CSV analysis, options workflows, IV crush scenarios, and practical risk reporting without platform theater.
· 8 min read
vOrchestrate and the Case for Controller-Centric Memory Policy in LLM Inference
A technical essay on dynamic multi-tier weight residency orchestration across HBM, DRAM, and NVMe, with scoring, guardrails, state transitions, and simulation-first evaluation.
· 8 min read
MHC Atlas OS and the Case for Explainable Structure-Guided Prioritization
A technical essay on runtime-agnostic, policy-governed, structure-guided experimental prioritization using AlphaFold-derived data, explainable scoring, and decision memory.
· 7 min read
Long-Context Inference Needs Better Memory Policy, Not Just More Memory
A technical essay on predictive context region orchestration, region-level attention and residency control, speculative promotion, reversible demotion, and coherence-aware long-context inference.
· 8 min read
The Real Tax in AI Systems Is Moving Bytes
A technical essay on bandwidth amplification, repeated refill across the hierarchy, and why better AI machines need stronger movement discipline, not just more compute and memory.
· 13 min read
Hardware-Enforced On-Chip Memory Residency for Neural Network Inference Accelerators
A deep technical essay on the bandwidth cost of repeatedly reloading hot weights during autoregressive inference, and why a wired on-chip residency primitive changes the machine rather than merely nudging the policy.
· 6 min read
gemma4-wdc: A middleware layer that stops local agents from doing the same work twice
A laptop-first systems write-up on shared execution units, bounded admission windows, and why local multi-agent workflows waste more backend work than most people realize.
· 23 min read
HBM Fragmentation Guard: Confidence-Gated Residency Control for AI Accelerators
A systems essay on why LRU is the wrong default for HBM residency under LLM serving pressure, and how confidence gating, thrash budgets, and safe-window compaction change allocator behavior.