Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
姑且算作 Megatron LM v2，因为是晚一年发表的，粗略过一下，因为都是同样的 motivation 和 background 等等
Paper Reading: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Paper Reading: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Megatron-LM
Nvidia 开源的 Megatron-LM 大模型训练框架
Paper Reading: PipeDream: Generalized Pipeline Parallelism for DNN Training [SOSP2019]
Paper Reading: From Cloud Computing to Sky Computing
From Cloud Computing to Sky Computing
21 年 UCB 提出了 Sky Computing 的概念，后续也有一篇 35 页的 The Sky Above The Clouds