CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

TL;DR

CuMo introduces a scalable multimodal LLM architecture that integrates mixture-of-experts in both vision and language components, achieving superior performance with minimal inference overhead by leveraging pre-training and expert balancing techniques.

Contribution

The paper proposes CuMo, a novel multimodal LLM framework that incorporates co-upcycled mixture-of-experts in vision and language modules, improving scalability and performance efficiently.

Findings

Outperforms state-of-the-art multimodal LLMs on VQA and visual instruction benchmarks.

Uses open-source datasets for training, ensuring accessibility and reproducibility.

Achieves high performance with minimal additional inference costs.

Abstract

Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.