KV cache internals: PagedAttention, RadixAttention, speculative decoding
Inference as prefill plus decode, why generation needs a KV cache and training doesn’t, a cache in PyTorch, how big it gets and how MLA shrinks it, where decode step times come from, continuous batching and PagedAttention with a toy vLLM you can run on a CPU, how SGLang’s RadixAttention differs, and how speculative decoding verifies a draft in one pass.