Parallelism for LLM training, with numbers you can check by hand
Data, ZeRO, tensor, sequence, context, pipeline and expert parallelism, each worked through with a tiny example: what gets split, what moves between GPUs, and when to use which.
Posts
Hi, this is Xuewen. I’m documenting my learning notes in this blog since 2026.