Abstract:This paper presents a low-cost network architecture for training large language models (LLMs) at hyperscale. We study the optimal parallelization strategy of LLMs and propose a novel datacenter network design tailored to LLM's unique communication pattern. We show that LLM training generates sparse communication patterns in the network and, therefore, does not require any-to-any full-bisection network to complete efficiently. As a result, our design eliminates the spine layer in traditional GPU clusters. We name this design a Rail-only network and demonstrate that it achieves the same training performance while reducing the network cost by 38% to 77% and network power consumption by 37% to 75% compared to a conventional GPU datacenter. Our architecture also supports Mixture-of-Expert (MoE) models with all-to-all communication through forwarding, with only 8.2% to 11.2% completion time overhead for all-to-all traffic. We study the failure robustness of Rail-only networks and provide insights into the performance impact of different network and training parameters.
Submission history
From: Weiyang Wang [view email]
[v1]
Sat, 22 Jul 2023 21:18:41 UTC (1,204 KB)
[v2]
Sat, 21 Oct 2023 03:15:48 UTC (1,146 KB)
[v3]
Wed, 1 Nov 2023 15:12:57 UTC (1,771 KB)
[v4]
Sat, 27 Jul 2024 20:59:24 UTC (1,088 KB)
[v5]
Sun, 15 Sep 2024 23:04:30 UTC (1,260 KB)