Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide

TL;DR

This paper systematically analyzes hybrid parallelism strategies for large language models, combining theoretical insights, empirical case studies, and design guidelines to optimize distributed training and inference.

Contribution

It provides a comprehensive review, mathematical formulations, and case studies on hybrid parallelism, offering a systematic methodology for designing efficient distributed LLM systems.

Findings

Hybrid parallel strategies improve communication efficiency.

Automated search methods optimize parallelism configurations.

Empirical case studies guide practical implementation choices.

Abstract

With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient training and inference. While existing surveys provide descriptive overviews of these techniques, systematic analysis of their benefits and trade offs and how such insights can inform principled methodology for designing optimal distributed systems remain limited. This paper offers a comprehensive review of collective operations and distributed parallel strategies, complemented by mathematical formulations to deepen theoretical understanding. We further examine hybrid parallelization designs, emphasizing communication computation overlap across different stages of model deployment, including both training and inference. Recent advances in automated search for optimal hybrid parallelization strategies using cost models are also discussed. Moreover, we present case studies with mainstream architecture categories to reveal empirical insights to guide researchers and practitioners in parallelism strategy selection. Finally, we highlight open challenges and limitations of current LLM training paradigms and outline promising directions for the next generation of large scale model development.