Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

TL;DR

This paper empirically investigates large-scale distributed training of neural networks, revealing that beyond a certain point, communication overhead and diminishing returns limit the efficiency of scaling models and hardware resources.

Contribution

It provides a comprehensive empirical analysis of hardware configurations and parallelization strategies, highlighting their impact on large-scale model training efficiency.

Findings

Communication overhead affects parallelization strategy effectiveness.

Diminishing returns occur when scaling hardware resources.

Proper optimization does not eliminate diminishing returns.

Abstract

Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To develop the exceedingly large networks required in modern applications, such as large language models (LLMs), model training is distributed across tens of thousands of hardware accelerators (e.g. GPUs), requiring orchestration of computation and communication across large computing clusters. In this work, we demonstrate that careful consideration of hardware configuration and parallelization strategy is critical for effective (i.e. compute- and cost-efficient) scaling of model size, training data, and total computation. We conduct an extensive empirical study of the performance of large-scale LLM training workloads across model size, hardware configurations, and distributed parallelization strategies. We demonstrate that: (1) beyond certain scales, overhead incurred from certain distributed communication strategies leads parallelization strategies previously thought to be sub-optimal in fact become preferable; and (2) scaling the total number of accelerators for large model training quickly yields diminishing returns even when hardware and parallelization strategies are properly optimized, implying poor marginal performance per additional unit of power or GPU-hour.