I’m a Senior Staff Research Engineer at Crusoe AI, working on large language models, multimodal systems, and large-scale training. This site is where I share my notes, blog posts, and publications.
Parallelism for LLM training, with numbers you can check by hand
Data, ZeRO, tensor, sequence, context, pipeline and expert parallelism, each worked through with a tiny example: what gets split, what moves between GPUs, and when to use which.