Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts

TL;DR

Awaker2.5-VL introduces a sparse Mixture of Experts architecture with low-rank adaptations to improve multi-task performance in Multimodal Large Language Models, effectively addressing multi-task conflicts and enhancing training efficiency.

Contribution

The paper presents a novel MoE-based architecture with LoRA for scalable, multi-task MLLMs, mitigating multi-task conflicts and improving training and inference speed.

Findings

Effective multi-task learning across diverse tasks

Improved performance on multiple benchmarks

Faster training and inference with LoRA experts

Abstract

As the research of Multimodal Large Language Models (MLLMs) becomes popular, an advancing MLLM model is typically required to handle various textual and visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for real-world applications. However, due to the significant differences in representation and distribution among data from various tasks, simply mixing data of all tasks together leads to the well-known``multi-task conflict" issue, resulting in performance degradation across various tasks. To address this issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture suitable for MLLM, which acquires the multi-task capabilities through multiple sparsely activated experts. To speed up the training and inference of Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation (LoRA) structure. Extensive experiments on multiple latest benchmarks demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are released in our Project Page: https://github.com/MetabrainAGI/Awaker.