Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing

TL;DR

This paper introduces Apollo, a system that improves multimodal model training efficiency by deploying multiple modules on GPUs using spatial-temporal multiplexing, achieving up to 1.31x speedup.

Contribution

It presents a novel spatial-temporal multiplexing approach and a flexible execution engine for efficient multimodal model training.

Findings

Achieves up to 1.31x training speedup on popular MMs.

Develops a performance model to estimate execution time under different resource plans.

Uses heuristics to derive high-quality deployment plans efficiently.

Abstract

With the wide adoption of Multimodal Models (MMs) in real-world scenarios, it is significant to efficiently train emerging MMs that exhibit increasingly complex module architectures. For MM deployment, existing works allocate a GPU to only one MM module in a temporal-multiplexing manner; this compromises training efficiency because a single module often fails to achieve high GPU utilization. To improve GPU utilization and enable efficient MM training, we propose deploying MMs in a temporal-spatial multiplexing manner, allowing multiple MM modules to colocate on a GPU with well-controlled resource quotas. In this paper, we propose Apollo, an efficient MM training system that applies temporal-spatial multiplexing. We first develop a flexible and lightweight execution engine that supports MM training with arbitrary resource quotas, and then build a comprehensive and accurate performance model to estimate module execution time under different allocation plans. With the performance model, we further adopt effective heuristics to derive high-quality MM deployment plans efficiently. Testbed experiments confirm that Apollo effectively improves the training efficiency of popular MMs, with a training speedup of up to 1.31x.