Parallelism for LLM training, with numbers you can check by hand
Data, ZeRO, tensor, sequence, context, pipeline and expert parallelism, each worked through with a tiny example: what gets split, what moves between GPUs, and when to use which.
Data, ZeRO, tensor, sequence, context, pipeline and expert parallelism, each worked through with a tiny example: what gets split, what moves between GPUs, and when to use which.