Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe
M Huang, C Wang, C Ma, Y Zhang, P Zhang… - arXiv preprint arXiv …, 2024 - arxiv.org
M Huang, C Wang, C Ma, Y Zhang, P Zhang, L Yu
arXiv preprint arXiv:2401.02088, 2024•arxiv.orgPipeline parallelism is an essential technique in the training of large-scale Transformer
models. However, it suffers from imbalanced memory consumption, leading to insufficient
memory utilization. The BPipe technique was proposed to address this issue and has
proven effective in the GPT-3 model. Nevertheless, our experiments have not yielded similar
benefits for LLaMA training. Additionally, BPipe only yields negligible benefits for GPT-3
training when applying flash attention. We analyze the underlying causes of the divergent …
models. However, it suffers from imbalanced memory consumption, leading to insufficient
memory utilization. The BPipe technique was proposed to address this issue and has
proven effective in the GPT-3 model. Nevertheless, our experiments have not yielded similar
benefits for LLaMA training. Additionally, BPipe only yields negligible benefits for GPT-3
training when applying flash attention. We analyze the underlying causes of the divergent …
Pipeline parallelism is an essential technique in the training of large-scale Transformer models. However, it suffers from imbalanced memory consumption, leading to insufficient memory utilization. The BPipe technique was proposed to address this issue and has proven effective in the GPT-3 model. Nevertheless, our experiments have not yielded similar benefits for LLaMA training. Additionally, BPipe only yields negligible benefits for GPT-3 training when applying flash attention. We analyze the underlying causes of the divergent performance of BPipe on GPT-3 and LLaMA. Furthermore, we introduce a novel method to estimate the performance of BPipe.
arxiv.org
Showing the best result for this search. See all results