Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

TL;DR

Optimus is a system that significantly speeds up large-scale multimodal LLM training by optimizing bubble scheduling and parallel plans, reducing GPU idle time and training duration.

Contribution

It introduces a novel bubble scheduling algorithm and parallel plan search to minimize GPU bubbles during MLLM training, improving efficiency.

Findings

Achieves 20.5%-21.3% faster training on ViT-22B and GPT-175B models.

Effectively reduces GPU idle time caused by heterogeneous data dependencies.

Demonstrates scalability on 3072 GPUs in a production environment.

Abstract

Multimodal large language models (MLLMs) have extended the success of large language models (LLMs) to multiple data types, such as image, text and audio, achieving significant performance in various domains, including multimodal translation, visual question answering and content generation. Nonetheless, existing systems are inefficient to train MLLMs due to substantial GPU bubbles caused by the heterogeneous modality models and complex data dependencies in 3D parallelism. This paper proposes Optimus, a distributed MLLM training system that reduces end-to-end MLLM training time. Optimus is based on our principled analysis that scheduling the encoder computation within the LLM bubbles can reduce bubbles in MLLM training. To make scheduling encoder computation possible for all GPUs, Optimus searches the separate parallel plans for encoder and LLM, and adopts a bubble scheduling algorithm to enable exploiting LLM bubbles without breaking the original data dependencies in the MLLM model architecture. We further decompose encoder layer computation into a series of kernels, and analyze the common bubble pattern of 3D parallelism to carefully optimize the sub-millisecond bubble scheduling, minimizing the overall training time. Our experiments in a production cluster show that Optimus accelerates MLLM training by 20.5%-21.3% with ViT-22B and GPT-175B model over 3072 GPUs compared to baselines.