Blog

Engineering writing on inference serving, scheduler behavior, and GPU kernel paths.

Inference2026-08-3112 min

Dynamo's KV Router: Routing on What the Fleet Already Cached

A fleet's replicas are not interchangeable — each one holds a different slice of KV cache, and routing to the wrong one turns a cache hit into a full prefill. NVIDIA Dynamo's KV router answers that with one number per worker: a cost combining how much of the prompt that worker already has against how loaded it is. These are my notes on the parts that produce those two numbers — block hashes and KV events, the global prefix index, the tiered block manager behind it — and on the cost function that spends them.

Distributed inference Serving Routing KV cache Dynamo

Inference2026-07-1613 min

How NVIDIA Dynamo Runs an Inference Fleet

Past a certain load one GPU node isn't enough, and a bigger GPU won't save you — the fix is a fleet. NVIDIA Dynamo is an open framework for running one, and its design is the clearest overview I've found of the whole problem: three planes — request, control, storage & events — and three control loops that keep the fleet fast, right-sized, and alive. These are my notes on the parts a Dynamo fleet is built from, how it manages them, and the one metric — goodput — that judges the result.

Distributed inference Serving Prefill/decode Dynamo

Inference2026-07-1311 min

Scheduling: How Continuous Batching and Paged Attention Fill a GPU

Orca reported 36.9× the throughput of FasterTransformer at the same latency without touching the model's math; a year later vLLM's profiling found as little as a fifth of KV-cache memory holding actual tokens. Two papers, two mechanisms — re-form the batch every iteration, page the cache in 16-token blocks — and everything else an LLM scheduler does today is policy.

Scheduling Continuous batching PagedAttention KV cache

Inference2026-07-0713 min

MoE: How the Mixture of Experts Routes a Token

The router in DeepSeek-V3 is a 7,168 × 256 matrix — 1.8M parameters picking 8 of 256 experts for every token, in a layer of 11.3B. This post covers what the MLP post's one MoE paragraph skipped: how the router works, what the sparsity buys, and the recipe the mid-2026 frontier — DeepSeek-V4, GLM-5.2, Kimi K2.6 — converged on.

MoE Routing Transformer

Inference2026-07-0412 min

MLP: Why Transformers Need the Multilayer Perceptron

In a Llama 2 70B layer the attention projections hold 151M parameters and the MLP next to them holds 704.6M — 82% of the layer, roughly 80% of the model, and most of the bytes decode streams for every token. This post builds the block the attention posts walked past: two matmuls and a gate, why the transformer collapses without it, and where the frontier is taking it.

Transformer MLP FFN SwiGLU Activation functions

Inference2026-07-0213 min

Attention, and How to Shrink It

The KV cache is num_layers × num_heads × 2 × seq_len × d_head × bytes, and every way to shrink attention attacks one of those terms. MQA and GQA share keys across heads, MLA compresses them to a latent that beats full attention on quality, FlashAttention computes the exact same thing with a fraction of the memory traffic, and the frontier goes after the rest — layers, sparsity, and the quadratic itself.

Attention KV cache GQA MLA FlashAttention

Inference2026-06-2912 min

Attention, from First Principles

Attention is the one operation that lets a token look back at the whole sequence — and it is also where the KV cache comes from. This post derives scaled dot-product attention from the ground up, then watches the cache, and the memory wall from the last post, fall straight out of the math. The first of two; the next is about making it cheap.

Attention Self-attention KV cache Transformer

Inference2026-06-249 min

LLM Inference: The Life of a Request

A serving model does two very different kinds of work for every request: it reads the prompt, then writes a reply. Those two phases — prefill and decode — have opposite bottlenecks, and almost every confusing thing about inference performance comes from treating them as one. This is the mental model the rest of my inference writing builds on.

Transformer Prefill/decode KV cache